2024-03-22T01:13:47.276 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-22T01:13:47.276 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --archive /home/teuthworker/archive/pdonnell-2024-03-22_01:06:30-fs-wip-batrick-testing-20240321.172515-distro-default-smithi/7615721 --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/ffsb}} --name pdonnell-2024-03-22_01:06:30-fs-wip-batrick-testing-20240321.172515-distro-default-smithi --owner scheduled_pdonnell@teuthology -- /home/teuthworker/archive/pdonnell-2024-03-22_01:06:30-fs-wip-batrick-testing-20240321.172515-distro-default-smithi/7615721/orig.config.yaml 2024-03-22T01:13:47.340 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-22T01:13:47.431 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-03-22_01:06:30-fs-wip-batrick-testing-20240321.172515-distro-default-smithi/7615721 branch: wip-batrick-testing-20240321.172515 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/ffsb}} email: pdonnell@redhat.com first_in_suite: false job_id: '7615721' kernel: client: sha1: distro kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true name: pdonnell-2024-03-22_01:06:30-fs-wip-batrick-testing-20240321.172515-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-batrick-testing-20240321.172515 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 1 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 filestore flush min: 0 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd heartbeat grace: 60 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - slow metadata IO - SLOW_OPS - slow request - SLOW_OPS - slow request sha1: c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e subvols: create: 2 subvol_options: --namespace-isolated --size 25000000000 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount - mds.dir_split install: ceph: flavor: default sha1: c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=secure - nowsync syntax: v2 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-batrick-testing-20240321.172515 sha1: c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e owner: scheduled_pdonnell@teuthology priority: 85 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 316 sha1: c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e sleep_before_teardown: 0 subset: 420/512 suite: fs suite_branch: wip-batrick-testing-20240321.172515 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi072.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKo+S2CXMsjpVcPBFgBBxourTVomk7ZnFioj/wZwp2FQnk+p+7YpjlIcrH0cdg8qzENQcY7AY/FnSyIKX1KDZzE= smithi089.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLWzifK/OZv43YSON/Qai1svwB4TjTJhvPiWpjHja5YIoIpzEAE5MhByrLzxAxpJXLlwFKb3r5VESzqMTnqdIrI= smithi098.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKWQYuZbn6ocySjHj2r0rq7Kuu00Mey6QK1jgdp35wO6acMUarg2s3WLQn35c9XdEpKW6IZmQNfpVpOPnNZxD74= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - suites/ffsb.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-22_01:06:30 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-22T01:13:47.431 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/qa; will attempt to use it 2024-03-22T01:13:47.432 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/qa/tasks 2024-03-22T01:13:47.432 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-22T01:13:47.434 INFO:teuthology.task.internal:Checking packages... 2024-03-22T01:13:47.456 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e' 2024-03-22T01:13:47.457 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-22T01:13:47.457 INFO:teuthology.packaging:ref: None 2024-03-22T01:13:47.457 INFO:teuthology.packaging:tag: None 2024-03-22T01:13:47.457 INFO:teuthology.packaging:branch: wip-batrick-testing-20240321.172515 2024-03-22T01:13:47.457 INFO:teuthology.packaging:sha1: c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e 2024-03-22T01:13:47.457 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-batrick-testing-20240321.172515 2024-03-22T01:13:47.675 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2344.gc3a04c0e 2024-03-22T01:13:47.677 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-22T01:13:47.684 INFO:teuthology.task.internal:no buildpackages task found 2024-03-22T01:13:47.684 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-22T01:13:47.695 INFO:teuthology.task.internal:Saving configuration 2024-03-22T01:13:47.716 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-22T01:13:47.726 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-22T01:13:47.748 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi072.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-22_01:06:30-fs-wip-batrick-testing-20240321.172515-distro-default-smithi/7615721', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-22 01:08:20.963144', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKo+S2CXMsjpVcPBFgBBxourTVomk7ZnFioj/wZwp2FQnk+p+7YpjlIcrH0cdg8qzENQcY7AY/FnSyIKX1KDZzE='} 2024-03-22T01:13:47.768 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi089.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-22_01:06:30-fs-wip-batrick-testing-20240321.172515-distro-default-smithi/7615721', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-22 01:08:20.964332', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLWzifK/OZv43YSON/Qai1svwB4TjTJhvPiWpjHja5YIoIpzEAE5MhByrLzxAxpJXLlwFKb3r5VESzqMTnqdIrI='} 2024-03-22T01:13:47.787 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi098.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-22_01:06:30-fs-wip-batrick-testing-20240321.172515-distro-default-smithi/7615721', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-22 01:08:20.961732', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKWQYuZbn6ocySjHj2r0rq7Kuu00Mey6QK1jgdp35wO6acMUarg2s3WLQn35c9XdEpKW6IZmQNfpVpOPnNZxD74='} 2024-03-22T01:13:47.787 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-22T01:13:47.794 INFO:teuthology.task.internal:roles: ubuntu@smithi072.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-03-22T01:13:47.794 INFO:teuthology.task.internal:roles: ubuntu@smithi089.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-03-22T01:13:47.794 INFO:teuthology.task.internal:roles: ubuntu@smithi098.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-03-22T01:13:47.794 INFO:teuthology.run_tasks:Running task console_log... 2024-03-22T01:13:47.892 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f2773fe6160>, signals=[15]) 2024-03-22T01:13:47.892 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-22T01:13:47.898 INFO:teuthology.task.internal:Opening connections... 2024-03-22T01:13:47.898 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi072.front.sepia.ceph.com 2024-03-22T01:13:47.901 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-22T01:13:47.989 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi089.front.sepia.ceph.com 2024-03-22T01:13:47.989 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-22T01:13:48.078 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi098.front.sepia.ceph.com 2024-03-22T01:13:48.079 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-22T01:13:48.167 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-22T01:13:48.172 DEBUG:teuthology.orchestra.run.smithi072:> uname -m 2024-03-22T01:13:48.190 INFO:teuthology.orchestra.run.smithi072.stdout:x86_64 2024-03-22T01:13:48.190 DEBUG:teuthology.orchestra.run.smithi072:> cat /etc/os-release 2024-03-22T01:13:48.247 INFO:teuthology.orchestra.run.smithi072.stdout:NAME="CentOS Stream" 2024-03-22T01:13:48.248 INFO:teuthology.orchestra.run.smithi072.stdout:VERSION="9" 2024-03-22T01:13:48.248 INFO:teuthology.orchestra.run.smithi072.stdout:ID="centos" 2024-03-22T01:13:48.248 INFO:teuthology.orchestra.run.smithi072.stdout:ID_LIKE="rhel fedora" 2024-03-22T01:13:48.248 INFO:teuthology.orchestra.run.smithi072.stdout:VERSION_ID="9" 2024-03-22T01:13:48.248 INFO:teuthology.orchestra.run.smithi072.stdout:PLATFORM_ID="platform:el9" 2024-03-22T01:13:48.248 INFO:teuthology.orchestra.run.smithi072.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-22T01:13:48.248 INFO:teuthology.orchestra.run.smithi072.stdout:ANSI_COLOR="0;31" 2024-03-22T01:13:48.248 INFO:teuthology.orchestra.run.smithi072.stdout:LOGO="fedora-logo-icon" 2024-03-22T01:13:48.248 INFO:teuthology.orchestra.run.smithi072.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-22T01:13:48.248 INFO:teuthology.orchestra.run.smithi072.stdout:HOME_URL="https://centos.org/" 2024-03-22T01:13:48.248 INFO:teuthology.orchestra.run.smithi072.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-22T01:13:48.248 INFO:teuthology.orchestra.run.smithi072.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-22T01:13:48.248 INFO:teuthology.orchestra.run.smithi072.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-22T01:13:48.249 INFO:teuthology.lock.ops:Updating smithi072.front.sepia.ceph.com on lock server 2024-03-22T01:13:48.272 DEBUG:teuthology.orchestra.run.smithi089:> uname -m 2024-03-22T01:13:48.290 INFO:teuthology.orchestra.run.smithi089.stdout:x86_64 2024-03-22T01:13:48.290 DEBUG:teuthology.orchestra.run.smithi089:> cat /etc/os-release 2024-03-22T01:13:48.347 INFO:teuthology.orchestra.run.smithi089.stdout:NAME="CentOS Stream" 2024-03-22T01:13:48.347 INFO:teuthology.orchestra.run.smithi089.stdout:VERSION="9" 2024-03-22T01:13:48.347 INFO:teuthology.orchestra.run.smithi089.stdout:ID="centos" 2024-03-22T01:13:48.347 INFO:teuthology.orchestra.run.smithi089.stdout:ID_LIKE="rhel fedora" 2024-03-22T01:13:48.347 INFO:teuthology.orchestra.run.smithi089.stdout:VERSION_ID="9" 2024-03-22T01:13:48.347 INFO:teuthology.orchestra.run.smithi089.stdout:PLATFORM_ID="platform:el9" 2024-03-22T01:13:48.347 INFO:teuthology.orchestra.run.smithi089.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-22T01:13:48.347 INFO:teuthology.orchestra.run.smithi089.stdout:ANSI_COLOR="0;31" 2024-03-22T01:13:48.347 INFO:teuthology.orchestra.run.smithi089.stdout:LOGO="fedora-logo-icon" 2024-03-22T01:13:48.347 INFO:teuthology.orchestra.run.smithi089.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-22T01:13:48.348 INFO:teuthology.orchestra.run.smithi089.stdout:HOME_URL="https://centos.org/" 2024-03-22T01:13:48.348 INFO:teuthology.orchestra.run.smithi089.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-22T01:13:48.348 INFO:teuthology.orchestra.run.smithi089.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-22T01:13:48.348 INFO:teuthology.orchestra.run.smithi089.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-22T01:13:48.348 INFO:teuthology.lock.ops:Updating smithi089.front.sepia.ceph.com on lock server 2024-03-22T01:13:48.372 DEBUG:teuthology.orchestra.run.smithi098:> uname -m 2024-03-22T01:13:48.390 INFO:teuthology.orchestra.run.smithi098.stdout:x86_64 2024-03-22T01:13:48.390 DEBUG:teuthology.orchestra.run.smithi098:> cat /etc/os-release 2024-03-22T01:13:48.448 INFO:teuthology.orchestra.run.smithi098.stdout:NAME="CentOS Stream" 2024-03-22T01:13:48.448 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION="9" 2024-03-22T01:13:48.448 INFO:teuthology.orchestra.run.smithi098.stdout:ID="centos" 2024-03-22T01:13:48.448 INFO:teuthology.orchestra.run.smithi098.stdout:ID_LIKE="rhel fedora" 2024-03-22T01:13:48.448 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION_ID="9" 2024-03-22T01:13:48.448 INFO:teuthology.orchestra.run.smithi098.stdout:PLATFORM_ID="platform:el9" 2024-03-22T01:13:48.448 INFO:teuthology.orchestra.run.smithi098.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-22T01:13:48.448 INFO:teuthology.orchestra.run.smithi098.stdout:ANSI_COLOR="0;31" 2024-03-22T01:13:48.448 INFO:teuthology.orchestra.run.smithi098.stdout:LOGO="fedora-logo-icon" 2024-03-22T01:13:48.449 INFO:teuthology.orchestra.run.smithi098.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-22T01:13:48.449 INFO:teuthology.orchestra.run.smithi098.stdout:HOME_URL="https://centos.org/" 2024-03-22T01:13:48.449 INFO:teuthology.orchestra.run.smithi098.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-22T01:13:48.449 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-22T01:13:48.449 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-22T01:13:48.449 INFO:teuthology.lock.ops:Updating smithi098.front.sepia.ceph.com on lock server 2024-03-22T01:13:48.472 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-22T01:13:48.482 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-22T01:13:48.487 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-22T01:13:48.488 DEBUG:teuthology.orchestra.run.smithi072:> test '!' -e /home/ubuntu/cephtest 2024-03-22T01:13:48.490 DEBUG:teuthology.orchestra.run.smithi089:> test '!' -e /home/ubuntu/cephtest 2024-03-22T01:13:48.493 DEBUG:teuthology.orchestra.run.smithi098:> test '!' -e /home/ubuntu/cephtest 2024-03-22T01:13:48.509 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-22T01:13:48.514 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-22T01:13:48.514 DEBUG:teuthology.orchestra.run.smithi072:> test -z $(ls -A /var/lib/ceph) 2024-03-22T01:13:48.547 DEBUG:teuthology.orchestra.run.smithi089:> test -z $(ls -A /var/lib/ceph) 2024-03-22T01:13:48.549 DEBUG:teuthology.orchestra.run.smithi098:> test -z $(ls -A /var/lib/ceph) 2024-03-22T01:13:48.650 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-22T01:13:48.720 INFO:teuthology.run_tasks:Running task kernel... 2024-03-22T01:13:48.733 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2024-03-22T01:13:48.733 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-03-22T01:13:48.733 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-03-22T01:13:48.733 DEBUG:teuthology.orchestra.run.smithi072:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-22T01:13:48.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-22T01:13:48.750 DEBUG:teuthology.orchestra.run.smithi072:> uname -r 2024-03-22T01:13:48.807 INFO:teuthology.orchestra.run.smithi072.stdout:5.14.0-361.el9.x86_64 2024-03-22T01:13:48.807 INFO:teuthology.task.kernel:Running kernel on smithi072: 5.14.0-361.el9.x86_64 2024-03-22T01:13:48.807 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum install -y kernel 2024-03-22T01:13:52.266 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS-9 - AppStream 7.1 MB/s | 6.7 MB 00:00 2024-03-22T01:13:54.000 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS-9 - BaseOS 3.2 MB/s | 2.0 MB 00:00 2024-03-22T01:13:55.593 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - BaseOS 6.6 MB/s | 8.0 MB 00:01 2024-03-22T01:13:57.529 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-03-22T01:14:02.505 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - CRB 4.2 MB/s | 6.1 MB 00:01 2024-03-22T01:14:04.899 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - Extras packages 12 kB/s | 16 kB 00:01 2024-03-22T01:14:07.114 INFO:teuthology.orchestra.run.smithi072.stdout:Extra Packages for Enterprise Linux 9.9 MB/s | 21 MB 00:02 2024-03-22T01:14:11.962 INFO:teuthology.orchestra.run.smithi072.stdout:lab-extras 40 kB/s | 1.7 kB 00:00 2024-03-22T01:14:14.069 INFO:teuthology.orchestra.run.smithi072.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-22T01:14:14.070 INFO:teuthology.orchestra.run.smithi072.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-22T01:14:14.201 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-03-22T01:14:14.216 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-22T01:14:14.216 INFO:teuthology.orchestra.run.smithi072.stdout: Package Architecture Version Repository Size 2024-03-22T01:14:14.217 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-22T01:14:14.217 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2024-03-22T01:14:14.217 INFO:teuthology.orchestra.run.smithi072.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-22T01:14:14.217 INFO:teuthology.orchestra.run.smithi072.stdout:Installing dependencies: 2024-03-22T01:14:14.217 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-22T01:14:14.217 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-22T01:14:14.217 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-22T01:14:14.217 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:14:14.217 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2024-03-22T01:14:14.217 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-22T01:14:14.217 INFO:teuthology.orchestra.run.smithi072.stdout:Install 4 Packages 2024-03-22T01:14:14.217 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:14:14.218 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 76 M 2024-03-22T01:14:14.218 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 123 M 2024-03-22T01:14:14.218 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2024-03-22T01:14:15.057 INFO:teuthology.orchestra.run.smithi072.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 63 kB/s | 44 kB 00:00 2024-03-22T01:14:37.691 INFO:teuthology.orchestra.run.smithi072.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 654 kB/s | 15 MB 00:23 2024-03-22T01:14:53.898 INFO:teuthology.orchestra.run.smithi072.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 724 kB/s | 27 MB 00:38 2024-03-22T01:15:01.765 INFO:teuthology.orchestra.run.smithi072.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 722 kB/s | 33 MB 00:47 2024-03-22T01:15:01.765 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-03-22T01:15:01.766 INFO:teuthology.orchestra.run.smithi072.stdout:Total 1.6 MB/s | 76 MB 00:47 2024-03-22T01:15:02.430 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2024-03-22T01:15:02.810 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2024-03-22T01:15:02.811 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2024-03-22T01:15:03.812 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2024-03-22T01:15:03.812 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2024-03-22T01:15:06.258 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2024-03-22T01:15:06.916 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-22T01:15:07.020 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-22T01:15:09.766 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-22T01:15:09.947 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-22T01:15:14.695 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-22T01:15:14.735 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-22T01:15:19.474 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-22T01:15:51.041 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-22T01:15:51.042 INFO:teuthology.orchestra.run.smithi072.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-22T01:15:51.042 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:15:51.086 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-22T01:15:51.507 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-22T01:15:51.507 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-22T01:15:51.507 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-22T01:15:51.507 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-22T01:15:51.962 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-22T01:15:51.962 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:15:51.962 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2024-03-22T01:15:51.962 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-22T01:15:51.963 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-22T01:15:51.963 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-22T01:15:51.963 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-22T01:15:51.963 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:15:51.963 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-03-22T01:15:52.404 DEBUG:teuthology.orchestra.run.smithi072:> rpm -q kernel | sort -rV | head -n 1 2024-03-22T01:15:52.529 INFO:teuthology.orchestra.run.smithi072.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-22T01:15:52.530 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-22T01:15:52.530 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-22T01:15:52.530 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-22T01:15:52.530 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-22T01:15:52.530 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2024-03-22T01:15:52.530 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi072.front.sepia.ceph.com, path=None, version=distro) 2024-03-22T01:15:52.530 DEBUG:teuthology.orchestra.run.smithi072:> rpm -q kernel | sort -rV | head -n 1 2024-03-22T01:15:52.610 INFO:teuthology.orchestra.run.smithi072.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-22T01:15:52.610 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-22T01:15:52.610 DEBUG:teuthology.orchestra.run.smithi072:> sudo rpm -qi grub2-tools 2024-03-22T01:15:52.649 INFO:teuthology.orchestra.run.smithi072.stdout:Name : grub2-tools 2024-03-22T01:15:52.649 INFO:teuthology.orchestra.run.smithi072.stdout:Epoch : 1 2024-03-22T01:15:52.649 INFO:teuthology.orchestra.run.smithi072.stdout:Version : 2.06 2024-03-22T01:15:52.649 INFO:teuthology.orchestra.run.smithi072.stdout:Release : 68.el9 2024-03-22T01:15:52.650 INFO:teuthology.orchestra.run.smithi072.stdout:Architecture: x86_64 2024-03-22T01:15:52.650 INFO:teuthology.orchestra.run.smithi072.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-22T01:15:52.650 INFO:teuthology.orchestra.run.smithi072.stdout:Group : Unspecified 2024-03-22T01:15:52.650 INFO:teuthology.orchestra.run.smithi072.stdout:Size : 8264254 2024-03-22T01:15:52.650 INFO:teuthology.orchestra.run.smithi072.stdout:License : GPLv3+ 2024-03-22T01:15:52.650 INFO:teuthology.orchestra.run.smithi072.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-22T01:15:52.650 INFO:teuthology.orchestra.run.smithi072.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-22T01:15:52.650 INFO:teuthology.orchestra.run.smithi072.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-22T01:15:52.650 INFO:teuthology.orchestra.run.smithi072.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-22T01:15:52.650 INFO:teuthology.orchestra.run.smithi072.stdout:Packager : builder@centos.org 2024-03-22T01:15:52.650 INFO:teuthology.orchestra.run.smithi072.stdout:Vendor : CentOS 2024-03-22T01:15:52.651 INFO:teuthology.orchestra.run.smithi072.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-22T01:15:52.651 INFO:teuthology.orchestra.run.smithi072.stdout:Summary : Support tools for GRUB. 2024-03-22T01:15:52.651 INFO:teuthology.orchestra.run.smithi072.stdout:Description : 2024-03-22T01:15:52.651 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:15:52.651 INFO:teuthology.orchestra.run.smithi072.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-22T01:15:52.651 INFO:teuthology.orchestra.run.smithi072.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-22T01:15:52.651 INFO:teuthology.orchestra.run.smithi072.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-22T01:15:52.651 INFO:teuthology.orchestra.run.smithi072.stdout:hardware devices. 2024-03-22T01:15:52.651 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:15:52.651 INFO:teuthology.orchestra.run.smithi072.stdout:This subpackage provides tools for support of all platforms. 2024-03-22T01:15:52.652 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-22T01:15:52.653 INFO:teuthology.task.kernel:Updating grub on smithi072 to boot 5.14.0-430.el9.x86_64 2024-03-22T01:15:52.653 DEBUG:teuthology.orchestra.run.smithi072:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-22T01:15:53.310 INFO:teuthology.orchestra.run.smithi072.stderr:Generating grub configuration file ... 2024-03-22T01:15:54.820 INFO:teuthology.orchestra.run.smithi072.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-22T01:15:54.846 INFO:teuthology.orchestra.run.smithi072.stderr:done 2024-03-22T01:15:54.848 DEBUG:teuthology.orchestra.run.smithi072:> mktemp 2024-03-22T01:15:54.883 INFO:teuthology.orchestra.run.smithi072.stdout:/tmp/tmp.fRJptd5myD 2024-03-22T01:15:54.883 DEBUG:teuthology.orchestra.run.smithi072:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.fRJptd5myD 2024-03-22T01:15:54.951 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 0666 /tmp/tmp.fRJptd5myD 2024-03-22T01:15:55.094 DEBUG:teuthology.orchestra.remote:smithi072:/tmp/tmp.fRJptd5myD is 6KB 2024-03-22T01:15:55.143 DEBUG:teuthology.orchestra.run.smithi072:> rm -fr /tmp/tmp.fRJptd5myD 2024-03-22T01:15:55.157 DEBUG:teuthology.orchestra.run.smithi072:> sudo /bin/ls /boot/loader/entries || true 2024-03-22T01:15:55.222 INFO:teuthology.orchestra.run.smithi072.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-22T01:15:55.222 INFO:teuthology.orchestra.run.smithi072.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-22T01:15:55.222 INFO:teuthology.orchestra.run.smithi072.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-22T01:15:55.222 INFO:teuthology.orchestra.run.smithi072.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-22T01:15:55.224 DEBUG:teuthology.orchestra.run.smithi072:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-22T01:15:55.375 DEBUG:teuthology.orchestra.run.smithi072:> sudo shutdown -r now 2024-03-22T01:16:25.378 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-03-22T01:16:25.379 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2024-03-22T01:16:25.380 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-22T01:17:00.062 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.72 2024-03-22T01:17:09.064 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2024-03-22T01:17:09.065 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-22T01:17:27.454 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.72 2024-03-22T01:17:39.455 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2024-03-22T01:17:39.456 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-22T01:17:42.531 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.72 2024-03-22T01:17:57.531 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2024-03-22T01:17:57.532 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-22T01:17:58.002 DEBUG:teuthology.orchestra.run.smithi072:> true 2024-03-22T01:17:58.472 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi072.front.sepia.ceph.com' 2024-03-22T01:17:58.472 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.14.0-430.el9.x86_64"... 2024-03-22T01:17:58.473 DEBUG:teuthology.orchestra.run.smithi072:> uname -r 2024-03-22T01:17:58.489 INFO:teuthology.orchestra.run.smithi072.stdout:5.14.0-430.el9.x86_64 2024-03-22T01:17:58.489 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-22T01:17:58.489 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-22T01:17:58.489 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-22T01:17:59.490 DEBUG:teuthology.parallel:result is None 2024-03-22T01:17:59.490 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-22T01:17:59.498 INFO:teuthology.task.internal:Creating test directory... 2024-03-22T01:17:59.498 DEBUG:teuthology.orchestra.run.smithi072:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-22T01:17:59.501 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-22T01:17:59.504 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-22T01:17:59.522 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-22T01:17:59.528 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-22T01:17:59.535 INFO:teuthology.task.internal:Creating archive directory... 2024-03-22T01:17:59.535 DEBUG:teuthology.orchestra.run.smithi072:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-22T01:17:59.559 DEBUG:teuthology.orchestra.run.smithi089:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-22T01:17:59.562 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-22T01:17:59.595 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-22T01:17:59.602 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-22T01:17:59.602 DEBUG:teuthology.orchestra.run.smithi089:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-22T01:17:59.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-22T01:17:59.646 DEBUG:teuthology.orchestra.run.smithi098:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-22T01:17:59.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-22T01:17:59.662 DEBUG:teuthology.orchestra.run.smithi072:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-22T01:17:59.664 DEBUG:teuthology.orchestra.run.smithi089:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-22T01:17:59.689 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-22T01:17:59.729 INFO:teuthology.orchestra.run.smithi089.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-22T01:17:59.736 INFO:teuthology.orchestra.run.smithi072.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-22T01:17:59.737 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-22T01:17:59.747 INFO:teuthology.orchestra.run.smithi089.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-22T01:17:59.761 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-22T01:17:59.789 INFO:teuthology.orchestra.run.smithi072.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-22T01:17:59.794 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-22T01:17:59.802 INFO:teuthology.task.internal:Configuring sudo... 2024-03-22T01:17:59.802 DEBUG:teuthology.orchestra.run.smithi072:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-22T01:17:59.804 DEBUG:teuthology.orchestra.run.smithi089:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-22T01:17:59.807 DEBUG:teuthology.orchestra.run.smithi098:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-22T01:17:59.835 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-22T01:17:59.902 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-22T01:17:59.903 DEBUG:teuthology.orchestra.run.smithi072:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-22T01:17:59.905 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-22T01:17:59.908 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-22T01:17:59.923 DEBUG:teuthology.orchestra.run.smithi072:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-22T01:17:59.993 DEBUG:teuthology.orchestra.run.smithi072:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-22T01:18:00.064 DEBUG:teuthology.orchestra.run.smithi072:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-22T01:18:00.102 DEBUG:teuthology.orchestra.run.smithi072:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-22T01:18:00.150 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-22T01:18:00.150 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-22T01:18:00.217 DEBUG:teuthology.orchestra.run.smithi089:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-22T01:18:00.255 DEBUG:teuthology.orchestra.run.smithi089:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-22T01:18:00.310 DEBUG:teuthology.orchestra.run.smithi089:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-22T01:18:00.388 DEBUG:teuthology.orchestra.run.smithi089:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-22T01:18:00.439 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-22T01:18:00.439 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-22T01:18:00.507 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-22T01:18:00.541 DEBUG:teuthology.orchestra.run.smithi098:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-22T01:18:00.609 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-22T01:18:00.689 DEBUG:teuthology.orchestra.run.smithi098:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-22T01:18:00.736 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-22T01:18:00.736 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-22T01:18:00.805 DEBUG:teuthology.orchestra.run.smithi072:> sudo service rsyslog restart 2024-03-22T01:18:00.807 DEBUG:teuthology.orchestra.run.smithi089:> sudo service rsyslog restart 2024-03-22T01:18:00.810 DEBUG:teuthology.orchestra.run.smithi098:> sudo service rsyslog restart 2024-03-22T01:18:00.850 INFO:teuthology.orchestra.run.smithi072.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-22T01:18:00.856 INFO:teuthology.orchestra.run.smithi089.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-22T01:18:00.879 INFO:teuthology.orchestra.run.smithi098.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-22T01:18:01.364 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-22T01:18:01.372 INFO:teuthology.task.internal:Starting timer... 2024-03-22T01:18:01.372 INFO:teuthology.run_tasks:Running task pcp... 2024-03-22T01:18:01.383 INFO:teuthology.run_tasks:Running task selinux... 2024-03-22T01:18:01.391 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-03-22T01:18:01.392 DEBUG:teuthology.orchestra.run.smithi072:> sudo service auditd rotate 2024-03-22T01:18:01.494 INFO:teuthology.orchestra.run.smithi072.stdout:Rotating logs: 2024-03-22T01:18:01.495 DEBUG:teuthology.orchestra.run.smithi089:> sudo service auditd rotate 2024-03-22T01:18:01.551 INFO:teuthology.orchestra.run.smithi089.stdout:Rotating logs: 2024-03-22T01:18:01.553 DEBUG:teuthology.orchestra.run.smithi098:> sudo service auditd rotate 2024-03-22T01:18:01.630 INFO:teuthology.orchestra.run.smithi098.stdout:Rotating logs: 2024-03-22T01:18:01.632 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-22T01:18:01.633 DEBUG:teuthology.orchestra.run.smithi072:> /usr/sbin/getenforce 2024-03-22T01:18:01.654 INFO:teuthology.orchestra.run.smithi072.stdout:Permissive 2024-03-22T01:18:01.654 DEBUG:teuthology.orchestra.run.smithi089:> /usr/sbin/getenforce 2024-03-22T01:18:01.674 INFO:teuthology.orchestra.run.smithi089.stdout:Permissive 2024-03-22T01:18:01.674 DEBUG:teuthology.orchestra.run.smithi098:> /usr/sbin/getenforce 2024-03-22T01:18:01.694 INFO:teuthology.orchestra.run.smithi098.stdout:Permissive 2024-03-22T01:18:01.694 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi072.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi089.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi098.front.sepia.ceph.com': 'permissive'} 2024-03-22T01:18:01.694 DEBUG:teuthology.orchestra.run.smithi072:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-22T01:18:01.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-22T01:18:01.724 DEBUG:teuthology.orchestra.run.smithi089:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-22T01:18:01.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-22T01:18:01.755 DEBUG:teuthology.orchestra.run.smithi098:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-22T01:18:01.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-22T01:18:01.782 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-22T01:18:01.782 DEBUG:teuthology.orchestra.run.smithi072:> sudo /usr/sbin/setenforce permissive 2024-03-22T01:18:01.815 DEBUG:teuthology.orchestra.run.smithi089:> sudo /usr/sbin/setenforce permissive 2024-03-22T01:18:01.850 DEBUG:teuthology.orchestra.run.smithi098:> sudo /usr/sbin/setenforce permissive 2024-03-22T01:18:01.888 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-22T01:18:01.897 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-22T01:18:01.903 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-22T01:18:02.009 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-22T01:18:02.024 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-22T01:18:02.026 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi072.front.sepia.ceph.com,smithi089.front.sepia.ceph.com,smithi098.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-22T01:22:52.851 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi072.front.sepia.ceph.com'), Remote(name='ubuntu@smithi089.front.sepia.ceph.com'), Remote(name='ubuntu@smithi098.front.sepia.ceph.com')] 2024-03-22T01:22:52.852 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2024-03-22T01:22:52.853 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-22T01:22:52.949 DEBUG:teuthology.orchestra.run.smithi072:> true 2024-03-22T01:22:53.034 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi072.front.sepia.ceph.com' 2024-03-22T01:22:53.034 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2024-03-22T01:22:53.035 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-22T01:22:53.123 DEBUG:teuthology.orchestra.run.smithi089:> true 2024-03-22T01:22:53.189 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi089.front.sepia.ceph.com' 2024-03-22T01:22:53.189 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-03-22T01:22:53.190 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-22T01:22:53.277 DEBUG:teuthology.orchestra.run.smithi098:> true 2024-03-22T01:22:53.350 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-03-22T01:22:53.350 INFO:teuthology.run_tasks:Running task clock... 2024-03-22T01:22:53.370 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-22T01:22:53.371 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-22T01:22:53.371 DEBUG:teuthology.orchestra.run.smithi072:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-22T01:22:53.374 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-22T01:22:53.374 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-22T01:22:53.376 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-22T01:22:53.376 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-22T01:22:53.403 INFO:teuthology.orchestra.run.smithi072.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-22T01:22:53.405 INFO:teuthology.orchestra.run.smithi089.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-22T01:22:53.420 INFO:teuthology.orchestra.run.smithi072.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-22T01:22:53.422 INFO:teuthology.orchestra.run.smithi089.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-22T01:22:53.423 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-22T01:22:53.441 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-22T01:22:53.450 INFO:teuthology.orchestra.run.smithi072.stderr:sudo: ntpd: command not found 2024-03-22T01:22:53.452 INFO:teuthology.orchestra.run.smithi089.stderr:sudo: ntpd: command not found 2024-03-22T01:22:53.464 INFO:teuthology.orchestra.run.smithi072.stdout:506 Cannot talk to daemon 2024-03-22T01:22:53.465 INFO:teuthology.orchestra.run.smithi089.stdout:506 Cannot talk to daemon 2024-03-22T01:22:53.471 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: ntpd: command not found 2024-03-22T01:22:53.480 INFO:teuthology.orchestra.run.smithi072.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-22T01:22:53.482 INFO:teuthology.orchestra.run.smithi089.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-22T01:22:53.485 INFO:teuthology.orchestra.run.smithi098.stdout:506 Cannot talk to daemon 2024-03-22T01:22:53.496 INFO:teuthology.orchestra.run.smithi072.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-22T01:22:53.498 INFO:teuthology.orchestra.run.smithi089.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-22T01:22:53.501 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-22T01:22:53.518 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-22T01:22:53.542 INFO:teuthology.orchestra.run.smithi089.stderr:bash: line 1: ntpq: command not found 2024-03-22T01:22:53.545 INFO:teuthology.orchestra.run.smithi089.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-22T01:22:53.545 INFO:teuthology.orchestra.run.smithi089.stdout:=============================================================================== 2024-03-22T01:22:53.545 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-22T01:22:53.545 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-22T01:22:53.545 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-22T01:22:53.546 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-22T01:22:53.550 INFO:teuthology.orchestra.run.smithi072.stderr:bash: line 1: ntpq: command not found 2024-03-22T01:22:53.554 INFO:teuthology.orchestra.run.smithi072.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-22T01:22:53.554 INFO:teuthology.orchestra.run.smithi072.stdout:=============================================================================== 2024-03-22T01:22:53.554 INFO:teuthology.orchestra.run.smithi072.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-22T01:22:53.554 INFO:teuthology.orchestra.run.smithi072.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-22T01:22:53.554 INFO:teuthology.orchestra.run.smithi072.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-22T01:22:53.554 INFO:teuthology.orchestra.run.smithi072.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-22T01:22:53.561 INFO:teuthology.orchestra.run.smithi098.stderr:bash: line 1: ntpq: command not found 2024-03-22T01:22:53.564 INFO:teuthology.orchestra.run.smithi098.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-22T01:22:53.564 INFO:teuthology.orchestra.run.smithi098.stdout:=============================================================================== 2024-03-22T01:22:53.565 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-22T01:22:53.565 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-22T01:22:53.565 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-22T01:22:53.565 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-22T01:22:53.566 INFO:teuthology.run_tasks:Running task install... 2024-03-22T01:22:53.577 DEBUG:teuthology.task.install:project ceph 2024-03-22T01:22:53.577 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e'}} 2024-03-22T01:22:53.577 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e'} 2024-03-22T01:22:53.577 INFO:teuthology.task.install:Using flavor: default 2024-03-22T01:22:53.583 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-22T01:22:53.583 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-22T01:22:53.584 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e', 'tag': None, 'wait_for_package': False} 2024-03-22T01:22:53.584 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e 2024-03-22T01:22:53.586 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e', 'tag': None, 'wait_for_package': False} 2024-03-22T01:22:53.586 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e 2024-03-22T01:22:53.587 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e', 'tag': None, 'wait_for_package': False} 2024-03-22T01:22:53.587 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e 2024-03-22T01:22:53.726 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-batrick-testing-20240321.172515/c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/centos/9/flavors/default/ 2024-03-22T01:22:53.726 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2344.gc3a04c0e 2024-03-22T01:22:53.732 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-batrick-testing-20240321.172515/c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/centos/9/flavors/default/ 2024-03-22T01:22:53.732 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2344.gc3a04c0e 2024-03-22T01:22:53.803 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-batrick-testing-20240321.172515/c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/centos/9/flavors/default/ 2024-03-22T01:22:53.803 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2344.gc3a04c0e 2024-03-22T01:22:53.900 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-batrick-testing-20240321.172515/c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-batrick-testing-20240321.172515/c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-batrick-testing-20240321.172515/c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-22T01:22:53.900 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-22T01:22:53.900 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-22T01:22:53.906 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-batrick-testing-20240321.172515/c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-batrick-testing-20240321.172515/c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-batrick-testing-20240321.172515/c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-22T01:22:53.907 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-22T01:22:53.907 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-22T01:22:53.930 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-22T01:22:53.930 DEBUG:teuthology.orchestra.run.smithi072:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-22T01:22:53.934 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-22T01:22:53.934 DEBUG:teuthology.orchestra.run.smithi098:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-22T01:22:53.974 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-batrick-testing-20240321.172515/c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-batrick-testing-20240321.172515/c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-batrick-testing-20240321.172515/c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-22T01:22:53.974 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-22T01:22:53.974 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-22T01:22:53.999 DEBUG:teuthology.orchestra.run.smithi072:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-22T01:22:54.004 DEBUG:teuthology.orchestra.run.smithi098:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-22T01:22:54.005 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-22T01:22:54.005 DEBUG:teuthology.orchestra.run.smithi089:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-22T01:22:54.073 DEBUG:teuthology.orchestra.run.smithi089:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-22T01:22:54.078 DEBUG:teuthology.orchestra.run.smithi072:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-22T01:22:54.082 DEBUG:teuthology.orchestra.run.smithi098:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-22T01:22:54.144 INFO:teuthology.orchestra.run.smithi072.stdout:check_obsoletes = 1 2024-03-22T01:22:54.146 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum clean all 2024-03-22T01:22:54.148 INFO:teuthology.orchestra.run.smithi098.stdout:check_obsoletes = 1 2024-03-22T01:22:54.150 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum clean all 2024-03-22T01:22:54.154 DEBUG:teuthology.orchestra.run.smithi089:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-22T01:22:54.220 INFO:teuthology.orchestra.run.smithi089.stdout:check_obsoletes = 1 2024-03-22T01:22:54.221 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum clean all 2024-03-22T01:22:54.383 INFO:teuthology.orchestra.run.smithi098.stdout:74 files removed 2024-03-22T01:22:54.389 INFO:teuthology.orchestra.run.smithi072.stdout:74 files removed 2024-03-22T01:22:54.410 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-22T01:22:54.415 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-22T01:22:54.460 INFO:teuthology.orchestra.run.smithi089.stdout:74 files removed 2024-03-22T01:22:54.489 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-22T01:22:55.123 INFO:teuthology.orchestra.run.smithi072.stdout:ceph packages for x86_64 202 kB/s | 83 kB 00:00 2024-03-22T01:22:55.149 INFO:teuthology.orchestra.run.smithi098.stdout:ceph packages for x86_64 185 kB/s | 83 kB 00:00 2024-03-22T01:22:55.202 INFO:teuthology.orchestra.run.smithi089.stdout:ceph packages for x86_64 198 kB/s | 83 kB 00:00 2024-03-22T01:22:55.440 INFO:teuthology.orchestra.run.smithi072.stdout:ceph noarch packages 43 kB/s | 12 kB 00:00 2024-03-22T01:22:55.516 INFO:teuthology.orchestra.run.smithi098.stdout:ceph noarch packages 35 kB/s | 12 kB 00:00 2024-03-22T01:22:55.519 INFO:teuthology.orchestra.run.smithi089.stdout:ceph noarch packages 42 kB/s | 12 kB 00:00 2024-03-22T01:22:55.750 INFO:teuthology.orchestra.run.smithi072.stdout:ceph source packages 7.0 kB/s | 1.9 kB 00:00 2024-03-22T01:22:55.859 INFO:teuthology.orchestra.run.smithi098.stdout:ceph source packages 6.1 kB/s | 1.9 kB 00:00 2024-03-22T01:22:55.903 INFO:teuthology.orchestra.run.smithi089.stdout:ceph source packages 5.4 kB/s | 1.9 kB 00:00 2024-03-22T01:22:56.188 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-03-22T01:22:56.305 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-22T01:22:56.353 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-22T01:22:57.856 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS-9 - BaseOS 3.6 MB/s | 2.0 MB 00:00 2024-03-22T01:22:58.053 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS-9 - BaseOS 3.5 MB/s | 2.0 MB 00:00 2024-03-22T01:22:58.089 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS-9 - BaseOS 3.0 MB/s | 2.0 MB 00:00 2024-03-22T01:22:58.657 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.0 MB 00:00 2024-03-22T01:23:00.575 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-03-22T01:23:00.838 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - BaseOS 3.3 MB/s | 8.0 MB 00:02 2024-03-22T01:23:04.657 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - CRB 9.3 MB/s | 6.1 MB 00:00 2024-03-22T01:23:06.315 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - Extras packages 30 kB/s | 16 kB 00:00 2024-03-22T01:23:07.264 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - BaseOS 926 kB/s | 8.0 MB 00:08 2024-03-22T01:23:08.429 INFO:teuthology.orchestra.run.smithi072.stdout:Extra Packages for Enterprise Linux 10 MB/s | 21 MB 00:02 2024-03-22T01:23:10.765 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - AppStream 7.6 MB/s | 19 MB 00:02 2024-03-22T01:23:13.286 INFO:teuthology.orchestra.run.smithi072.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-03-22T01:23:15.287 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - CRB 5.6 MB/s | 6.1 MB 00:01 2024-03-22T01:23:15.761 INFO:teuthology.orchestra.run.smithi072.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-22T01:23:15.762 INFO:teuthology.orchestra.run.smithi072.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-22T01:23:15.777 INFO:teuthology.orchestra.run.smithi072.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-22T01:23:15.778 INFO:teuthology.orchestra.run.smithi072.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-22T01:23:15.779 INFO:teuthology.orchestra.run.smithi072.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-22T01:23:15.780 INFO:teuthology.orchestra.run.smithi072.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-22T01:23:15.782 INFO:teuthology.orchestra.run.smithi072.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-22T01:23:15.785 INFO:teuthology.orchestra.run.smithi072.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-22T01:23:15.786 INFO:teuthology.orchestra.run.smithi072.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-22T01:23:15.866 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-03-22T01:23:15.895 INFO:teuthology.orchestra.run.smithi072.stdout:============================================================================================ 2024-03-22T01:23:15.896 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repository Size 2024-03-22T01:23:15.896 INFO:teuthology.orchestra.run.smithi072.stdout:============================================================================================ 2024-03-22T01:23:15.896 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2024-03-22T01:23:15.896 INFO:teuthology.orchestra.run.smithi072.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-22T01:23:15.896 INFO:teuthology.orchestra.run.smithi072.stdout: ceph x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 6.6 k 2024-03-22T01:23:15.896 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-base x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 5.4 M 2024-03-22T01:23:15.896 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-fuse x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 890 k 2024-03-22T01:23:15.896 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 145 k 2024-03-22T01:23:15.896 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 1.5 M 2024-03-22T01:23:15.896 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2344.gc3a04c0e.el9 ceph-noarch 140 k 2024-03-22T01:23:15.896 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2344.gc3a04c0e.el9 ceph-noarch 3.5 M 2024-03-22T01:23:15.896 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2344.gc3a04c0e.el9 ceph-noarch 7.4 M 2024-03-22T01:23:15.897 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-rook noarch 2:19.0.0-2344.gc3a04c0e.el9 ceph-noarch 49 k 2024-03-22T01:23:15.897 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-radosgw x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 9.8 M 2024-03-22T01:23:15.897 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-test x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 48 M 2024-03-22T01:23:15.897 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-volume noarch 2:19.0.0-2344.gc3a04c0e.el9 ceph-noarch 264 k 2024-03-22T01:23:15.897 INFO:teuthology.orchestra.run.smithi072.stdout: cephadm noarch 2:19.0.0-2344.gc3a04c0e.el9 ceph-noarch 750 k 2024-03-22T01:23:15.897 INFO:teuthology.orchestra.run.smithi072.stdout: cephfs-mirror x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 222 k 2024-03-22T01:23:15.897 INFO:teuthology.orchestra.run.smithi072.stdout: cephfs-shell x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 27 k 2024-03-22T01:23:15.897 INFO:teuthology.orchestra.run.smithi072.stdout: cephfs-top noarch 2:19.0.0-2344.gc3a04c0e.el9 ceph-noarch 18 k 2024-03-22T01:23:15.897 INFO:teuthology.orchestra.run.smithi072.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-22T01:23:15.897 INFO:teuthology.orchestra.run.smithi072.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-22T01:23:15.897 INFO:teuthology.orchestra.run.smithi072.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-22T01:23:15.897 INFO:teuthology.orchestra.run.smithi072.stdout: libcephfs-devel x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 32 k 2024-03-22T01:23:15.898 INFO:teuthology.orchestra.run.smithi072.stdout: libcephfs2 x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 747 k 2024-03-22T01:23:15.898 INFO:teuthology.orchestra.run.smithi072.stdout: librados-devel x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 127 k 2024-03-22T01:23:15.898 INFO:teuthology.orchestra.run.smithi072.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-22T01:23:15.898 INFO:teuthology.orchestra.run.smithi072.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-22T01:23:15.898 INFO:teuthology.orchestra.run.smithi072.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-22T01:23:15.898 INFO:teuthology.orchestra.run.smithi072.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-22T01:23:15.898 INFO:teuthology.orchestra.run.smithi072.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-22T01:23:15.898 INFO:teuthology.orchestra.run.smithi072.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-22T01:23:15.898 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cephfs x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 161 k 2024-03-22T01:23:15.902 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rados x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 321 k 2024-03-22T01:23:15.902 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rbd x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 298 k 2024-03-22T01:23:15.902 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rgw x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 99 k 2024-03-22T01:23:15.902 INFO:teuthology.orchestra.run.smithi072.stdout: rbd-fuse x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 85 k 2024-03-22T01:23:15.902 INFO:teuthology.orchestra.run.smithi072.stdout: rbd-mirror x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 3.1 M 2024-03-22T01:23:15.902 INFO:teuthology.orchestra.run.smithi072.stdout: rbd-nbd x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 170 k 2024-03-22T01:23:15.902 INFO:teuthology.orchestra.run.smithi072.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-22T01:23:15.903 INFO:teuthology.orchestra.run.smithi072.stdout:Upgrading: 2024-03-22T01:23:15.903 INFO:teuthology.orchestra.run.smithi072.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-22T01:23:15.903 INFO:teuthology.orchestra.run.smithi072.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-22T01:23:15.903 INFO:teuthology.orchestra.run.smithi072.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-22T01:23:15.903 INFO:teuthology.orchestra.run.smithi072.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-22T01:23:15.903 INFO:teuthology.orchestra.run.smithi072.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-22T01:23:15.903 INFO:teuthology.orchestra.run.smithi072.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-22T01:23:15.903 INFO:teuthology.orchestra.run.smithi072.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-22T01:23:15.904 INFO:teuthology.orchestra.run.smithi072.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-22T01:23:15.904 INFO:teuthology.orchestra.run.smithi072.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-22T01:23:15.904 INFO:teuthology.orchestra.run.smithi072.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-22T01:23:15.904 INFO:teuthology.orchestra.run.smithi072.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-22T01:23:15.904 INFO:teuthology.orchestra.run.smithi072.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-22T01:23:15.904 INFO:teuthology.orchestra.run.smithi072.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-22T01:23:15.904 INFO:teuthology.orchestra.run.smithi072.stdout: librados2 x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 3.4 M 2024-03-22T01:23:15.904 INFO:teuthology.orchestra.run.smithi072.stdout: librbd1 x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 3.1 M 2024-03-22T01:23:15.904 INFO:teuthology.orchestra.run.smithi072.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-22T01:23:15.904 INFO:teuthology.orchestra.run.smithi072.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-22T01:23:15.904 INFO:teuthology.orchestra.run.smithi072.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-22T01:23:15.904 INFO:teuthology.orchestra.run.smithi072.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-22T01:23:15.904 INFO:teuthology.orchestra.run.smithi072.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-22T01:23:15.905 INFO:teuthology.orchestra.run.smithi072.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-22T01:23:15.905 INFO:teuthology.orchestra.run.smithi072.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-22T01:23:15.905 INFO:teuthology.orchestra.run.smithi072.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-22T01:23:15.905 INFO:teuthology.orchestra.run.smithi072.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-22T01:23:15.905 INFO:teuthology.orchestra.run.smithi072.stdout:Installing dependencies: 2024-03-22T01:23:15.905 INFO:teuthology.orchestra.run.smithi072.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-22T01:23:15.905 INFO:teuthology.orchestra.run.smithi072.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-22T01:23:15.905 INFO:teuthology.orchestra.run.smithi072.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-22T01:23:15.905 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-common x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 21 M 2024-03-22T01:23:15.905 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2344.gc3a04c0e.el9 ceph-noarch 34 k 2024-03-22T01:23:15.905 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mds x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 2.4 M 2024-03-22T01:23:15.905 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2344.gc3a04c0e.el9 ceph-noarch 248 k 2024-03-22T01:23:15.905 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mon x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 4.7 M 2024-03-22T01:23:15.906 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-osd x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 17 M 2024-03-22T01:23:15.906 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2344.gc3a04c0e.el9 ceph-noarch 16 k 2024-03-22T01:23:15.906 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-selinux x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 25 k 2024-03-22T01:23:15.906 INFO:teuthology.orchestra.run.smithi072.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-22T01:23:15.906 INFO:teuthology.orchestra.run.smithi072.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-22T01:23:15.906 INFO:teuthology.orchestra.run.smithi072.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-22T01:23:15.906 INFO:teuthology.orchestra.run.smithi072.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-22T01:23:15.906 INFO:teuthology.orchestra.run.smithi072.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-22T01:23:15.906 INFO:teuthology.orchestra.run.smithi072.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-22T01:23:15.906 INFO:teuthology.orchestra.run.smithi072.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-22T01:23:15.906 INFO:teuthology.orchestra.run.smithi072.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-22T01:23:15.906 INFO:teuthology.orchestra.run.smithi072.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-22T01:23:15.906 INFO:teuthology.orchestra.run.smithi072.stdout: libcephsqlite x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 163 k 2024-03-22T01:23:15.906 INFO:teuthology.orchestra.run.smithi072.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-22T01:23:15.907 INFO:teuthology.orchestra.run.smithi072.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-22T01:23:15.907 INFO:teuthology.orchestra.run.smithi072.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-22T01:23:15.907 INFO:teuthology.orchestra.run.smithi072.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-22T01:23:15.907 INFO:teuthology.orchestra.run.smithi072.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-22T01:23:15.907 INFO:teuthology.orchestra.run.smithi072.stdout: libradosstriper1 x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 501 k 2024-03-22T01:23:15.907 INFO:teuthology.orchestra.run.smithi072.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-22T01:23:15.907 INFO:teuthology.orchestra.run.smithi072.stdout: librgw2 x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 5.1 M 2024-03-22T01:23:15.907 INFO:teuthology.orchestra.run.smithi072.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-22T01:23:15.907 INFO:teuthology.orchestra.run.smithi072.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-22T01:23:15.907 INFO:teuthology.orchestra.run.smithi072.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-22T01:23:15.907 INFO:teuthology.orchestra.run.smithi072.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-22T01:23:15.907 INFO:teuthology.orchestra.run.smithi072.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-22T01:23:15.907 INFO:teuthology.orchestra.run.smithi072.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-22T01:23:15.907 INFO:teuthology.orchestra.run.smithi072.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-22T01:23:15.907 INFO:teuthology.orchestra.run.smithi072.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-22T01:23:15.908 INFO:teuthology.orchestra.run.smithi072.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-22T01:23:15.908 INFO:teuthology.orchestra.run.smithi072.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-22T01:23:15.908 INFO:teuthology.orchestra.run.smithi072.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-22T01:23:15.908 INFO:teuthology.orchestra.run.smithi072.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-22T01:23:15.908 INFO:teuthology.orchestra.run.smithi072.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-22T01:23:15.908 INFO:teuthology.orchestra.run.smithi072.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-22T01:23:15.908 INFO:teuthology.orchestra.run.smithi072.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-22T01:23:15.908 INFO:teuthology.orchestra.run.smithi072.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-22T01:23:15.908 INFO:teuthology.orchestra.run.smithi072.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-22T01:23:15.908 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-22T01:23:15.908 INFO:teuthology.orchestra.run.smithi072.stdout: python3-ceph-argparse x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 45 k 2024-03-22T01:23:15.908 INFO:teuthology.orchestra.run.smithi072.stdout: python3-ceph-common x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 131 k 2024-03-22T01:23:15.908 INFO:teuthology.orchestra.run.smithi072.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-22T01:23:15.909 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-22T01:23:15.909 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-22T01:23:15.909 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-22T01:23:15.909 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-22T01:23:15.909 INFO:teuthology.orchestra.run.smithi072.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-22T01:23:15.909 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-22T01:23:15.909 INFO:teuthology.orchestra.run.smithi072.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-22T01:23:15.909 INFO:teuthology.orchestra.run.smithi072.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-22T01:23:15.909 INFO:teuthology.orchestra.run.smithi072.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-22T01:23:15.909 INFO:teuthology.orchestra.run.smithi072.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-22T01:23:15.909 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-22T01:23:15.909 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-22T01:23:15.909 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-22T01:23:15.909 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-22T01:23:15.909 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-22T01:23:15.910 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-22T01:23:15.910 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-22T01:23:15.910 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-22T01:23:15.910 INFO:teuthology.orchestra.run.smithi072.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-22T01:23:15.910 INFO:teuthology.orchestra.run.smithi072.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-22T01:23:15.910 INFO:teuthology.orchestra.run.smithi072.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-22T01:23:15.910 INFO:teuthology.orchestra.run.smithi072.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-22T01:23:15.910 INFO:teuthology.orchestra.run.smithi072.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-22T01:23:15.910 INFO:teuthology.orchestra.run.smithi072.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-22T01:23:15.910 INFO:teuthology.orchestra.run.smithi072.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-22T01:23:15.910 INFO:teuthology.orchestra.run.smithi072.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-22T01:23:15.910 INFO:teuthology.orchestra.run.smithi072.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-22T01:23:15.910 INFO:teuthology.orchestra.run.smithi072.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-22T01:23:15.911 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-22T01:23:15.911 INFO:teuthology.orchestra.run.smithi072.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-22T01:23:15.911 INFO:teuthology.orchestra.run.smithi072.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-22T01:23:15.911 INFO:teuthology.orchestra.run.smithi072.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-22T01:23:15.911 INFO:teuthology.orchestra.run.smithi072.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-22T01:23:15.911 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-22T01:23:15.911 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-22T01:23:15.911 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-22T01:23:15.911 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-22T01:23:15.911 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-22T01:23:15.911 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-22T01:23:15.911 INFO:teuthology.orchestra.run.smithi072.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-22T01:23:15.911 INFO:teuthology.orchestra.run.smithi072.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-22T01:23:15.911 INFO:teuthology.orchestra.run.smithi072.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-22T01:23:15.912 INFO:teuthology.orchestra.run.smithi072.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-22T01:23:15.912 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-22T01:23:15.912 INFO:teuthology.orchestra.run.smithi072.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-22T01:23:15.912 INFO:teuthology.orchestra.run.smithi072.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-22T01:23:15.912 INFO:teuthology.orchestra.run.smithi072.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-22T01:23:15.912 INFO:teuthology.orchestra.run.smithi072.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-22T01:23:15.912 INFO:teuthology.orchestra.run.smithi072.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-22T01:23:15.912 INFO:teuthology.orchestra.run.smithi072.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-22T01:23:15.912 INFO:teuthology.orchestra.run.smithi072.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-22T01:23:15.912 INFO:teuthology.orchestra.run.smithi072.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-22T01:23:15.912 INFO:teuthology.orchestra.run.smithi072.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-22T01:23:15.912 INFO:teuthology.orchestra.run.smithi072.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-22T01:23:15.912 INFO:teuthology.orchestra.run.smithi072.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-22T01:23:15.912 INFO:teuthology.orchestra.run.smithi072.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-22T01:23:15.912 INFO:teuthology.orchestra.run.smithi072.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-22T01:23:15.913 INFO:teuthology.orchestra.run.smithi072.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-22T01:23:15.913 INFO:teuthology.orchestra.run.smithi072.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-22T01:23:15.913 INFO:teuthology.orchestra.run.smithi072.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-22T01:23:15.913 INFO:teuthology.orchestra.run.smithi072.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-22T01:23:15.913 INFO:teuthology.orchestra.run.smithi072.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-22T01:23:15.913 INFO:teuthology.orchestra.run.smithi072.stdout:Installing weak dependencies: 2024-03-22T01:23:15.913 INFO:teuthology.orchestra.run.smithi072.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-22T01:23:15.913 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:23:15.913 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2024-03-22T01:23:15.913 INFO:teuthology.orchestra.run.smithi072.stdout:============================================================================================ 2024-03-22T01:23:15.913 INFO:teuthology.orchestra.run.smithi072.stdout:Install 150 Packages 2024-03-22T01:23:15.913 INFO:teuthology.orchestra.run.smithi072.stdout:Upgrade 24 Packages 2024-03-22T01:23:15.913 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:23:15.914 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 226 M 2024-03-22T01:23:15.914 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2024-03-22T01:23:16.621 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - Extras packages 79 kB/s | 16 kB 00:00 2024-03-22T01:23:16.989 INFO:teuthology.orchestra.run.smithi072.stdout:(1/174): ceph-19.0.0-2344.gc3a04c0e.el9.x86_64. 18 kB/s | 6.6 kB 00:00 2024-03-22T01:23:17.642 INFO:teuthology.orchestra.run.smithi098.stdout:Extra Packages for Enterprise Linux 23 MB/s | 21 MB 00:00 2024-03-22T01:23:19.382 INFO:teuthology.orchestra.run.smithi072.stdout:(2/174): ceph-base-19.0.0-2344.gc3a04c0e.el9.x8 1.9 MB/s | 5.4 MB 00:02 2024-03-22T01:23:19.457 INFO:teuthology.orchestra.run.smithi072.stdout:(3/174): ceph-immutable-object-cache-19.0.0-234 1.9 MB/s | 145 kB 00:00 2024-03-22T01:23:19.924 INFO:teuthology.orchestra.run.smithi072.stdout:(4/174): ceph-mds-19.0.0-2344.gc3a04c0e.el9.x86 5.1 MB/s | 2.4 MB 00:00 2024-03-22T01:23:20.025 INFO:teuthology.orchestra.run.smithi072.stdout:(5/174): ceph-fuse-19.0.0-2344.gc3a04c0e.el9.x8 293 kB/s | 890 kB 00:03 2024-03-22T01:23:20.258 INFO:teuthology.orchestra.run.smithi072.stdout:(6/174): ceph-mgr-19.0.0-2344.gc3a04c0e.el9.x86 4.5 MB/s | 1.5 MB 00:00 2024-03-22T01:23:22.209 INFO:teuthology.orchestra.run.smithi072.stdout:(7/174): ceph-osd-19.0.0-2344.gc3a04c0e.el9.x86 8.6 MB/s | 17 MB 00:01 2024-03-22T01:23:22.490 INFO:teuthology.orchestra.run.smithi098.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-22T01:23:23.302 INFO:teuthology.orchestra.run.smithi072.stdout:(8/174): ceph-common-19.0.0-2344.gc3a04c0e.el9. 3.1 MB/s | 21 MB 00:06 2024-03-22T01:23:23.486 INFO:teuthology.orchestra.run.smithi072.stdout:(9/174): ceph-radosgw-19.0.0-2344.gc3a04c0e.el9 7.7 MB/s | 9.8 MB 00:01 2024-03-22T01:23:23.512 INFO:teuthology.orchestra.run.smithi072.stdout:(10/174): ceph-selinux-19.0.0-2344.gc3a04c0e.el 120 kB/s | 25 kB 00:00 2024-03-22T01:23:23.579 INFO:teuthology.orchestra.run.smithi072.stdout:(11/174): cephfs-mirror-19.0.0-2344.gc3a04c0e.e 3.2 MB/s | 222 kB 00:00 2024-03-22T01:23:23.629 INFO:teuthology.orchestra.run.smithi072.stdout:(12/174): cephfs-shell-19.0.0-2344.gc3a04c0e.el 539 kB/s | 27 kB 00:00 2024-03-22T01:23:23.688 INFO:teuthology.orchestra.run.smithi072.stdout:(13/174): libcephfs-devel-19.0.0-2344.gc3a04c0e 542 kB/s | 32 kB 00:00 2024-03-22T01:23:23.814 INFO:teuthology.orchestra.run.smithi072.stdout:(14/174): libcephfs2-19.0.0-2344.gc3a04c0e.el9. 5.8 MB/s | 747 kB 00:00 2024-03-22T01:23:23.864 INFO:teuthology.orchestra.run.smithi072.stdout:(15/174): libcephsqlite-19.0.0-2344.gc3a04c0e.e 3.2 MB/s | 163 kB 00:00 2024-03-22T01:23:23.915 INFO:teuthology.orchestra.run.smithi072.stdout:(16/174): librados-devel-19.0.0-2344.gc3a04c0e. 2.5 MB/s | 127 kB 00:00 2024-03-22T01:23:24.015 INFO:teuthology.orchestra.run.smithi072.stdout:(17/174): libradosstriper1-19.0.0-2344.gc3a04c0 4.9 MB/s | 501 kB 00:00 2024-03-22T01:23:24.666 INFO:teuthology.orchestra.run.smithi072.stdout:(18/174): librgw2-19.0.0-2344.gc3a04c0e.el9.x86 7.9 MB/s | 5.1 MB 00:00 2024-03-22T01:23:24.724 INFO:teuthology.orchestra.run.smithi072.stdout:(19/174): python3-ceph-argparse-19.0.0-2344.gc3 776 kB/s | 45 kB 00:00 2024-03-22T01:23:24.784 INFO:teuthology.orchestra.run.smithi072.stdout:(20/174): python3-ceph-common-19.0.0-2344.gc3a0 2.2 MB/s | 131 kB 00:00 2024-03-22T01:23:24.792 INFO:teuthology.orchestra.run.smithi098.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-22T01:23:24.793 INFO:teuthology.orchestra.run.smithi098.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-22T01:23:24.805 INFO:teuthology.orchestra.run.smithi098.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-22T01:23:24.805 INFO:teuthology.orchestra.run.smithi098.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-22T01:23:24.806 INFO:teuthology.orchestra.run.smithi098.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-22T01:23:24.807 INFO:teuthology.orchestra.run.smithi098.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-22T01:23:24.809 INFO:teuthology.orchestra.run.smithi098.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-22T01:23:24.811 INFO:teuthology.orchestra.run.smithi098.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-22T01:23:24.812 INFO:teuthology.orchestra.run.smithi098.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-22T01:23:24.834 INFO:teuthology.orchestra.run.smithi072.stdout:(21/174): python3-cephfs-19.0.0-2344.gc3a04c0e. 3.1 MB/s | 161 kB 00:00 2024-03-22T01:23:24.891 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-22T01:23:24.913 INFO:teuthology.orchestra.run.smithi098.stdout:============================================================================================ 2024-03-22T01:23:24.913 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-03-22T01:23:24.913 INFO:teuthology.orchestra.run.smithi098.stdout:============================================================================================ 2024-03-22T01:23:24.913 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-22T01:23:24.913 INFO:teuthology.orchestra.run.smithi098.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-22T01:23:24.913 INFO:teuthology.orchestra.run.smithi098.stdout: ceph x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 6.6 k 2024-03-22T01:23:24.913 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-base x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 5.4 M 2024-03-22T01:23:24.913 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-fuse x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 890 k 2024-03-22T01:23:24.914 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 145 k 2024-03-22T01:23:24.914 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 1.5 M 2024-03-22T01:23:24.914 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2344.gc3a04c0e.el9 ceph-noarch 140 k 2024-03-22T01:23:24.914 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2344.gc3a04c0e.el9 ceph-noarch 3.5 M 2024-03-22T01:23:24.914 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2344.gc3a04c0e.el9 ceph-noarch 7.4 M 2024-03-22T01:23:24.914 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-rook noarch 2:19.0.0-2344.gc3a04c0e.el9 ceph-noarch 49 k 2024-03-22T01:23:24.914 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-radosgw x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 9.8 M 2024-03-22T01:23:24.914 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-test x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 48 M 2024-03-22T01:23:24.914 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-volume noarch 2:19.0.0-2344.gc3a04c0e.el9 ceph-noarch 264 k 2024-03-22T01:23:24.914 INFO:teuthology.orchestra.run.smithi098.stdout: cephadm noarch 2:19.0.0-2344.gc3a04c0e.el9 ceph-noarch 750 k 2024-03-22T01:23:24.914 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-mirror x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 222 k 2024-03-22T01:23:24.914 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-shell x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 27 k 2024-03-22T01:23:24.914 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-top noarch 2:19.0.0-2344.gc3a04c0e.el9 ceph-noarch 18 k 2024-03-22T01:23:24.914 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-22T01:23:24.915 INFO:teuthology.orchestra.run.smithi098.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-22T01:23:24.915 INFO:teuthology.orchestra.run.smithi098.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-22T01:23:24.915 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs-devel x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 32 k 2024-03-22T01:23:24.915 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs2 x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 747 k 2024-03-22T01:23:24.915 INFO:teuthology.orchestra.run.smithi098.stdout: librados-devel x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 127 k 2024-03-22T01:23:24.915 INFO:teuthology.orchestra.run.smithi098.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-22T01:23:24.915 INFO:teuthology.orchestra.run.smithi098.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-22T01:23:24.915 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-22T01:23:24.915 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-22T01:23:24.915 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-22T01:23:24.915 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-22T01:23:24.915 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cephfs x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 161 k 2024-03-22T01:23:24.915 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rados x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 321 k 2024-03-22T01:23:24.916 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rbd x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 298 k 2024-03-22T01:23:24.916 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rgw x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 99 k 2024-03-22T01:23:24.916 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-fuse x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 85 k 2024-03-22T01:23:24.916 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-mirror x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 3.1 M 2024-03-22T01:23:24.916 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-nbd x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 170 k 2024-03-22T01:23:24.916 INFO:teuthology.orchestra.run.smithi098.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-22T01:23:24.916 INFO:teuthology.orchestra.run.smithi098.stdout:Upgrading: 2024-03-22T01:23:24.916 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-22T01:23:24.916 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-22T01:23:24.916 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-22T01:23:24.916 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-22T01:23:24.916 INFO:teuthology.orchestra.run.smithi098.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-22T01:23:24.917 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-22T01:23:24.917 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-22T01:23:24.917 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-22T01:23:24.917 INFO:teuthology.orchestra.run.smithi098.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-22T01:23:24.917 INFO:teuthology.orchestra.run.smithi098.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-22T01:23:24.917 INFO:teuthology.orchestra.run.smithi098.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-22T01:23:24.917 INFO:teuthology.orchestra.run.smithi098.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-22T01:23:24.917 INFO:teuthology.orchestra.run.smithi098.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-22T01:23:24.917 INFO:teuthology.orchestra.run.smithi098.stdout: librados2 x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 3.4 M 2024-03-22T01:23:24.917 INFO:teuthology.orchestra.run.smithi098.stdout: librbd1 x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 3.1 M 2024-03-22T01:23:24.917 INFO:teuthology.orchestra.run.smithi098.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-22T01:23:24.917 INFO:teuthology.orchestra.run.smithi098.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-22T01:23:24.917 INFO:teuthology.orchestra.run.smithi098.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-22T01:23:24.917 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-22T01:23:24.918 INFO:teuthology.orchestra.run.smithi098.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-22T01:23:24.918 INFO:teuthology.orchestra.run.smithi098.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-22T01:23:24.918 INFO:teuthology.orchestra.run.smithi098.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-22T01:23:24.918 INFO:teuthology.orchestra.run.smithi098.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-22T01:23:24.918 INFO:teuthology.orchestra.run.smithi098.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-22T01:23:24.918 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-03-22T01:23:24.918 INFO:teuthology.orchestra.run.smithi098.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-22T01:23:24.918 INFO:teuthology.orchestra.run.smithi098.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-22T01:23:24.918 INFO:teuthology.orchestra.run.smithi098.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-22T01:23:24.918 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-common x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 21 M 2024-03-22T01:23:24.918 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2344.gc3a04c0e.el9 ceph-noarch 34 k 2024-03-22T01:23:24.918 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mds x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 2.4 M 2024-03-22T01:23:24.919 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2344.gc3a04c0e.el9 ceph-noarch 248 k 2024-03-22T01:23:24.919 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mon x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 4.7 M 2024-03-22T01:23:24.919 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-osd x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 17 M 2024-03-22T01:23:24.919 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2344.gc3a04c0e.el9 ceph-noarch 16 k 2024-03-22T01:23:24.919 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-selinux x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 25 k 2024-03-22T01:23:24.919 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-22T01:23:24.919 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-22T01:23:24.919 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-22T01:23:24.919 INFO:teuthology.orchestra.run.smithi098.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-22T01:23:24.919 INFO:teuthology.orchestra.run.smithi098.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-22T01:23:24.919 INFO:teuthology.orchestra.run.smithi098.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-22T01:23:24.919 INFO:teuthology.orchestra.run.smithi098.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-22T01:23:24.919 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-22T01:23:24.920 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-22T01:23:24.920 INFO:teuthology.orchestra.run.smithi098.stdout: libcephsqlite x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 163 k 2024-03-22T01:23:24.920 INFO:teuthology.orchestra.run.smithi098.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-22T01:23:24.920 INFO:teuthology.orchestra.run.smithi098.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-22T01:23:24.920 INFO:teuthology.orchestra.run.smithi098.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-22T01:23:24.920 INFO:teuthology.orchestra.run.smithi098.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-22T01:23:24.920 INFO:teuthology.orchestra.run.smithi098.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-22T01:23:24.920 INFO:teuthology.orchestra.run.smithi098.stdout: libradosstriper1 x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 501 k 2024-03-22T01:23:24.920 INFO:teuthology.orchestra.run.smithi098.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-22T01:23:24.920 INFO:teuthology.orchestra.run.smithi098.stdout: librgw2 x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 5.1 M 2024-03-22T01:23:24.920 INFO:teuthology.orchestra.run.smithi098.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-22T01:23:24.920 INFO:teuthology.orchestra.run.smithi098.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-22T01:23:24.920 INFO:teuthology.orchestra.run.smithi098.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-22T01:23:24.920 INFO:teuthology.orchestra.run.smithi098.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-22T01:23:24.921 INFO:teuthology.orchestra.run.smithi098.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-22T01:23:24.921 INFO:teuthology.orchestra.run.smithi098.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-22T01:23:24.921 INFO:teuthology.orchestra.run.smithi098.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-22T01:23:24.921 INFO:teuthology.orchestra.run.smithi098.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-22T01:23:24.921 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-22T01:23:24.921 INFO:teuthology.orchestra.run.smithi098.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-22T01:23:24.921 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-22T01:23:24.921 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-22T01:23:24.921 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-22T01:23:24.921 INFO:teuthology.orchestra.run.smithi098.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-22T01:23:24.921 INFO:teuthology.orchestra.run.smithi098.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-22T01:23:24.921 INFO:teuthology.orchestra.run.smithi098.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-22T01:23:24.921 INFO:teuthology.orchestra.run.smithi098.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-22T01:23:24.921 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-22T01:23:24.922 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-argparse x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 45 k 2024-03-22T01:23:24.922 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-common x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 131 k 2024-03-22T01:23:24.922 INFO:teuthology.orchestra.run.smithi098.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-22T01:23:24.922 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-22T01:23:24.922 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-22T01:23:24.922 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-22T01:23:24.922 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-22T01:23:24.922 INFO:teuthology.orchestra.run.smithi098.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-22T01:23:24.922 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-22T01:23:24.922 INFO:teuthology.orchestra.run.smithi098.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-22T01:23:24.922 INFO:teuthology.orchestra.run.smithi098.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-22T01:23:24.922 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-22T01:23:24.922 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-22T01:23:24.923 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-22T01:23:24.923 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-22T01:23:24.923 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-22T01:23:24.923 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-22T01:23:24.923 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-22T01:23:24.923 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-22T01:23:24.923 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-22T01:23:24.923 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-22T01:23:24.923 INFO:teuthology.orchestra.run.smithi098.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-22T01:23:24.923 INFO:teuthology.orchestra.run.smithi098.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-22T01:23:24.923 INFO:teuthology.orchestra.run.smithi098.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-22T01:23:24.923 INFO:teuthology.orchestra.run.smithi098.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-22T01:23:24.923 INFO:teuthology.orchestra.run.smithi098.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-22T01:23:24.924 INFO:teuthology.orchestra.run.smithi098.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-22T01:23:24.924 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-22T01:23:24.924 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-22T01:23:24.924 INFO:teuthology.orchestra.run.smithi098.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-22T01:23:24.924 INFO:teuthology.orchestra.run.smithi098.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-22T01:23:24.924 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-22T01:23:24.924 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-22T01:23:24.924 INFO:teuthology.orchestra.run.smithi098.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-22T01:23:24.924 INFO:teuthology.orchestra.run.smithi098.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-22T01:23:24.924 INFO:teuthology.orchestra.run.smithi098.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-22T01:23:24.924 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-22T01:23:24.924 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-22T01:23:24.924 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-22T01:23:24.924 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-22T01:23:24.925 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-22T01:23:24.925 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-22T01:23:24.925 INFO:teuthology.orchestra.run.smithi098.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-22T01:23:24.925 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-22T01:23:24.925 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-22T01:23:24.925 INFO:teuthology.orchestra.run.smithi098.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-22T01:23:24.925 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-22T01:23:24.925 INFO:teuthology.orchestra.run.smithi098.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-22T01:23:24.925 INFO:teuthology.orchestra.run.smithi098.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-22T01:23:24.925 INFO:teuthology.orchestra.run.smithi098.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-22T01:23:24.925 INFO:teuthology.orchestra.run.smithi098.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-22T01:23:24.925 INFO:teuthology.orchestra.run.smithi098.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-22T01:23:24.925 INFO:teuthology.orchestra.run.smithi098.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-22T01:23:24.925 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-22T01:23:24.926 INFO:teuthology.orchestra.run.smithi098.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-22T01:23:24.926 INFO:teuthology.orchestra.run.smithi098.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-22T01:23:24.926 INFO:teuthology.orchestra.run.smithi098.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-22T01:23:24.926 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-22T01:23:24.926 INFO:teuthology.orchestra.run.smithi098.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-22T01:23:24.926 INFO:teuthology.orchestra.run.smithi098.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-22T01:23:24.926 INFO:teuthology.orchestra.run.smithi098.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-22T01:23:24.926 INFO:teuthology.orchestra.run.smithi098.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-22T01:23:24.926 INFO:teuthology.orchestra.run.smithi098.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-22T01:23:24.926 INFO:teuthology.orchestra.run.smithi098.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-22T01:23:24.926 INFO:teuthology.orchestra.run.smithi098.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-22T01:23:24.926 INFO:teuthology.orchestra.run.smithi098.stdout:Installing weak dependencies: 2024-03-22T01:23:24.926 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-22T01:23:24.926 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-22T01:23:24.927 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-22T01:23:24.927 INFO:teuthology.orchestra.run.smithi098.stdout:============================================================================================ 2024-03-22T01:23:24.927 INFO:teuthology.orchestra.run.smithi098.stdout:Install 150 Packages 2024-03-22T01:23:24.927 INFO:teuthology.orchestra.run.smithi098.stdout:Upgrade 24 Packages 2024-03-22T01:23:24.927 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-22T01:23:24.927 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 226 M 2024-03-22T01:23:24.927 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-22T01:23:24.943 INFO:teuthology.orchestra.run.smithi072.stdout:(22/174): ceph-mon-19.0.0-2344.gc3a04c0e.el9.x8 977 kB/s | 4.7 MB 00:04 2024-03-22T01:23:24.977 INFO:teuthology.orchestra.run.smithi072.stdout:(23/174): python3-rados-19.0.0-2344.gc3a04c0e.e 2.2 MB/s | 321 kB 00:00 2024-03-22T01:23:25.037 INFO:teuthology.orchestra.run.smithi072.stdout:(24/174): python3-rgw-19.0.0-2344.gc3a04c0e.el9 1.6 MB/s | 99 kB 00:00 2024-03-22T01:23:25.096 INFO:teuthology.orchestra.run.smithi072.stdout:(25/174): rbd-fuse-19.0.0-2344.gc3a04c0e.el9.x8 1.4 MB/s | 85 kB 00:00 2024-03-22T01:23:25.230 INFO:teuthology.orchestra.run.smithi072.stdout:(26/174): python3-rbd-19.0.0-2344.gc3a04c0e.el9 1.0 MB/s | 298 kB 00:00 2024-03-22T01:23:25.397 INFO:teuthology.orchestra.run.smithi072.stdout:(27/174): rbd-nbd-19.0.0-2344.gc3a04c0e.el9.x86 1.0 MB/s | 170 kB 00:00 2024-03-22T01:23:25.481 INFO:teuthology.orchestra.run.smithi072.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2344.g 406 kB/s | 34 kB 00:00 2024-03-22T01:23:25.533 INFO:teuthology.orchestra.run.smithi098.stdout:(1/174): ceph-19.0.0-2344.gc3a04c0e.el9.x86_64. 38 kB/s | 6.6 kB 00:00 2024-03-22T01:23:25.556 INFO:teuthology.orchestra.run.smithi072.stdout:(29/174): rbd-mirror-19.0.0-2344.gc3a04c0e.el9. 6.7 MB/s | 3.1 MB 00:00 2024-03-22T01:23:25.632 INFO:teuthology.orchestra.run.smithi072.stdout:(30/174): ceph-mgr-cephadm-19.0.0-2344.gc3a04c0 928 kB/s | 140 kB 00:00 2024-03-22T01:23:25.858 INFO:teuthology.orchestra.run.smithi098.stdout:(2/174): ceph-fuse-19.0.0-2344.gc3a04c0e.el9.x8 2.7 MB/s | 890 kB 00:00 2024-03-22T01:23:25.925 INFO:teuthology.orchestra.run.smithi098.stdout:(3/174): ceph-immutable-object-cache-19.0.0-234 2.1 MB/s | 145 kB 00:00 2024-03-22T01:23:26.224 INFO:teuthology.orchestra.run.smithi072.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2344.gc3a04 5.3 MB/s | 3.5 MB 00:00 2024-03-22T01:23:26.341 INFO:teuthology.orchestra.run.smithi072.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2344.gc3 2.1 MB/s | 248 kB 00:00 2024-03-22T01:23:26.417 INFO:teuthology.orchestra.run.smithi072.stdout:(33/174): ceph-mgr-rook-19.0.0-2344.gc3a04c0e.e 657 kB/s | 49 kB 00:00 2024-03-22T01:23:26.450 INFO:teuthology.orchestra.run.smithi098.stdout:(4/174): ceph-mds-19.0.0-2344.gc3a04c0e.el9.x86 4.5 MB/s | 2.4 MB 00:00 2024-03-22T01:23:26.476 INFO:teuthology.orchestra.run.smithi072.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2344.gc 281 kB/s | 16 kB 00:00 2024-03-22T01:23:26.550 INFO:teuthology.orchestra.run.smithi098.stdout:(5/174): ceph-base-19.0.0-2344.gc3a04c0e.el9.x8 4.5 MB/s | 5.4 MB 00:01 2024-03-22T01:23:26.560 INFO:teuthology.orchestra.run.smithi072.stdout:(35/174): ceph-volume-19.0.0-2344.gc3a04c0e.el9 3.1 MB/s | 264 kB 00:00 2024-03-22T01:23:26.776 INFO:teuthology.orchestra.run.smithi098.stdout:(6/174): ceph-mgr-19.0.0-2344.gc3a04c0e.el9.x86 4.6 MB/s | 1.5 MB 00:00 2024-03-22T01:23:26.827 INFO:teuthology.orchestra.run.smithi072.stdout:(36/174): cephadm-19.0.0-2344.gc3a04c0e.el9.noa 2.7 MB/s | 750 kB 00:00 2024-03-22T01:23:26.911 INFO:teuthology.orchestra.run.smithi072.stdout:(37/174): cephfs-top-19.0.0-2344.gc3a04c0e.el9. 218 kB/s | 18 kB 00:00 2024-03-22T01:23:27.195 INFO:teuthology.orchestra.run.smithi072.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 2.3 MB/s | 662 kB 00:00 2024-03-22T01:23:27.284 INFO:teuthology.orchestra.run.smithi098.stdout:(7/174): ceph-mon-19.0.0-2344.gc3a04c0e.el9.x86 6.4 MB/s | 4.7 MB 00:00 2024-03-22T01:23:27.330 INFO:teuthology.orchestra.run.smithi072.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 6.7 MB/s | 920 kB 00:00 2024-03-22T01:23:27.398 INFO:teuthology.orchestra.run.smithi072.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 1.5 MB/s | 106 kB 00:00 2024-03-22T01:23:27.682 INFO:teuthology.orchestra.run.smithi072.stdout:(41/174): flex-2.6.4-9.el9.x86_64.rpm 1.1 MB/s | 307 kB 00:00 2024-03-22T01:23:27.857 INFO:teuthology.orchestra.run.smithi072.stdout:(42/174): flexiblas-3.0.4-8.el9.x86_64.rpm 180 kB/s | 32 kB 00:00 2024-03-22T01:23:28.149 INFO:teuthology.orchestra.run.smithi072.stdout:(43/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 10 MB/s | 3.0 MB 00:00 2024-03-22T01:23:28.300 INFO:teuthology.orchestra.run.smithi072.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 114 kB/s | 17 kB 00:00 2024-03-22T01:23:28.334 INFO:teuthology.orchestra.run.smithi098.stdout:(8/174): ceph-common-19.0.0-2344.gc3a04c0e.el9. 6.9 MB/s | 21 MB 00:02 2024-03-22T01:23:28.385 INFO:teuthology.orchestra.run.smithi072.stdout:(45/174): libaio-devel-0.3.111-13.el9.x86_64.rp 120 kB/s | 10 kB 00:00 2024-03-22T01:23:28.401 INFO:teuthology.orchestra.run.smithi098.stdout:(9/174): ceph-selinux-19.0.0-2344.gc3a04c0e.el9 378 kB/s | 25 kB 00:00 2024-03-22T01:23:28.518 INFO:teuthology.orchestra.run.smithi098.stdout:(10/174): ceph-radosgw-19.0.0-2344.gc3a04c0e.el 8.0 MB/s | 9.8 MB 00:01 2024-03-22T01:23:28.569 INFO:teuthology.orchestra.run.smithi072.stdout:(46/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 868 kB/s | 159 kB 00:00 2024-03-22T01:23:28.594 INFO:teuthology.orchestra.run.smithi098.stdout:(11/174): cephfs-mirror-19.0.0-2344.gc3a04c0e.e 2.9 MB/s | 222 kB 00:00 2024-03-22T01:23:28.643 INFO:teuthology.orchestra.run.smithi072.stdout:(47/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 585 kB/s | 44 kB 00:00 2024-03-22T01:23:28.652 INFO:teuthology.orchestra.run.smithi098.stdout:(12/174): cephfs-shell-19.0.0-2344.gc3a04c0e.el 463 kB/s | 27 kB 00:00 2024-03-22T01:23:28.702 INFO:teuthology.orchestra.run.smithi072.stdout:(48/174): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-22T01:23:28.711 INFO:teuthology.orchestra.run.smithi098.stdout:(13/174): libcephfs-devel-19.0.0-2344.gc3a04c0e 542 kB/s | 32 kB 00:00 2024-03-22T01:23:28.861 INFO:teuthology.orchestra.run.smithi098.stdout:(14/174): libcephfs2-19.0.0-2344.gc3a04c0e.el9. 4.9 MB/s | 747 kB 00:00 2024-03-22T01:23:28.903 INFO:teuthology.orchestra.run.smithi072.stdout:(49/174): libtool-2.4.6-45.el9.x86_64.rpm 2.8 MB/s | 578 kB 00:00 2024-03-22T01:23:28.945 INFO:teuthology.orchestra.run.smithi098.stdout:(15/174): libcephsqlite-19.0.0-2344.gc3a04c0e.e 1.9 MB/s | 163 kB 00:00 2024-03-22T01:23:29.095 INFO:teuthology.orchestra.run.smithi072.stdout:(50/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 241 kB/s | 46 kB 00:00 2024-03-22T01:23:29.120 INFO:teuthology.orchestra.run.smithi098.stdout:(16/174): ceph-osd-19.0.0-2344.gc3a04c0e.el9.x8 7.2 MB/s | 17 MB 00:02 2024-03-22T01:23:29.145 INFO:teuthology.orchestra.run.smithi098.stdout:(17/174): librados-devel-19.0.0-2344.gc3a04c0e. 632 kB/s | 127 kB 00:00 2024-03-22T01:23:29.162 INFO:teuthology.orchestra.run.smithi072.stdout:(51/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.1 MB/s | 282 kB 00:00 2024-03-22T01:23:29.205 INFO:teuthology.orchestra.run.smithi072.stdout:(52/174): lua-5.4.4-4.el9.x86_64.rpm 4.4 MB/s | 187 kB 00:00 2024-03-22T01:23:29.238 INFO:teuthology.orchestra.run.smithi098.stdout:(18/174): libradosstriper1-19.0.0-2344.gc3a04c0 4.2 MB/s | 501 kB 00:00 2024-03-22T01:23:29.248 INFO:teuthology.orchestra.run.smithi072.stdout:(53/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.9 MB/s | 217 kB 00:00 2024-03-22T01:23:29.305 INFO:teuthology.orchestra.run.smithi098.stdout:(19/174): python3-ceph-argparse-19.0.0-2344.gc3 677 kB/s | 45 kB 00:00 2024-03-22T01:23:29.380 INFO:teuthology.orchestra.run.smithi098.stdout:(20/174): python3-ceph-common-19.0.0-2344.gc3a0 1.7 MB/s | 131 kB 00:00 2024-03-22T01:23:29.424 INFO:teuthology.orchestra.run.smithi072.stdout:(54/174): openblas-0.3.21-2.el9.x86_64.rpm 195 kB/s | 34 kB 00:00 2024-03-22T01:23:29.463 INFO:teuthology.orchestra.run.smithi098.stdout:(21/174): python3-cephfs-19.0.0-2344.gc3a04c0e. 1.9 MB/s | 161 kB 00:00 2024-03-22T01:23:29.563 INFO:teuthology.orchestra.run.smithi098.stdout:(22/174): python3-rados-19.0.0-2344.gc3a04c0e.e 3.1 MB/s | 321 kB 00:00 2024-03-22T01:23:29.664 INFO:teuthology.orchestra.run.smithi098.stdout:(23/174): python3-rbd-19.0.0-2344.gc3a04c0e.el9 2.9 MB/s | 298 kB 00:00 2024-03-22T01:23:29.748 INFO:teuthology.orchestra.run.smithi098.stdout:(24/174): python3-rgw-19.0.0-2344.gc3a04c0e.el9 1.2 MB/s | 99 kB 00:00 2024-03-22T01:23:29.800 INFO:teuthology.orchestra.run.smithi072.stdout:(55/174): openblas-openmp-0.3.21-2.el9.x86_64.r 13 MB/s | 4.7 MB 00:00 2024-03-22T01:23:29.848 INFO:teuthology.orchestra.run.smithi098.stdout:(25/174): librgw2-19.0.0-2344.gc3a04c0e.el9.x86 7.3 MB/s | 5.1 MB 00:00 2024-03-22T01:23:29.873 INFO:teuthology.orchestra.run.smithi098.stdout:(26/174): rbd-fuse-19.0.0-2344.gc3a04c0e.el9.x8 680 kB/s | 85 kB 00:00 2024-03-22T01:23:29.958 INFO:teuthology.orchestra.run.smithi072.stdout:(56/174): postgresql-13.11-1.el9.x86_64.rpm 9.6 MB/s | 1.5 MB 00:00 2024-03-22T01:23:29.965 INFO:teuthology.orchestra.run.smithi098.stdout:(27/174): rbd-nbd-19.0.0-2344.gc3a04c0e.el9.x86 1.8 MB/s | 170 kB 00:00 2024-03-22T01:23:30.026 INFO:teuthology.orchestra.run.smithi072.stdout:(57/174): postgresql-contrib-13.11-1.el9.x86_64 12 MB/s | 811 kB 00:00 2024-03-22T01:23:30.115 INFO:teuthology.orchestra.run.smithi098.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2344.g 226 kB/s | 34 kB 00:00 2024-03-22T01:23:30.208 INFO:teuthology.orchestra.run.smithi098.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2344.gc3a04c0 1.5 MB/s | 140 kB 00:00 2024-03-22T01:23:30.251 INFO:teuthology.orchestra.run.smithi072.stdout:(58/174): postgresql-private-libs-13.11-1.el9.x 594 kB/s | 134 kB 00:00 2024-03-22T01:23:30.316 INFO:teuthology.orchestra.run.smithi098.stdout:(30/174): rbd-mirror-19.0.0-2344.gc3a04c0e.el9. 6.6 MB/s | 3.1 MB 00:00 2024-03-22T01:23:30.469 INFO:teuthology.orchestra.run.smithi072.stdout:(59/174): postgresql-server-13.11-1.el9.x86_64. 26 MB/s | 5.7 MB 00:00 2024-03-22T01:23:30.528 INFO:teuthology.orchestra.run.smithi072.stdout:(60/174): protobuf-3.14.0-13.el9.x86_64.rpm 17 MB/s | 1.0 MB 00:00 2024-03-22T01:23:30.570 INFO:teuthology.orchestra.run.smithi072.stdout:(61/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.8 MB/s | 78 kB 00:00 2024-03-22T01:23:30.762 INFO:teuthology.orchestra.run.smithi072.stdout:(62/174): python3-babel-2.9.1-2.el9.noarch.rpm 30 MB/s | 5.8 MB 00:00 2024-03-22T01:23:30.791 INFO:teuthology.orchestra.run.smithi098.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2344.gc3a04 6.0 MB/s | 3.5 MB 00:00 2024-03-22T01:23:30.867 INFO:teuthology.orchestra.run.smithi098.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2344.gc3 3.2 MB/s | 248 kB 00:00 2024-03-22T01:23:30.942 INFO:teuthology.orchestra.run.smithi098.stdout:(33/174): ceph-mgr-rook-19.0.0-2344.gc3a04c0e.e 657 kB/s | 49 kB 00:00 2024-03-22T01:23:31.009 INFO:teuthology.orchestra.run.smithi098.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2344.gc 247 kB/s | 16 kB 00:00 2024-03-22T01:23:31.093 INFO:teuthology.orchestra.run.smithi098.stdout:(35/174): ceph-volume-19.0.0-2344.gc3a04c0e.el9 3.1 MB/s | 264 kB 00:00 2024-03-22T01:23:31.251 INFO:teuthology.orchestra.run.smithi098.stdout:(36/174): cephadm-19.0.0-2344.gc3a04c0e.el9.noa 4.6 MB/s | 750 kB 00:00 2024-03-22T01:23:31.327 INFO:teuthology.orchestra.run.smithi098.stdout:(37/174): cephfs-top-19.0.0-2344.gc3a04c0e.el9. 243 kB/s | 18 kB 00:00 2024-03-22T01:23:31.410 INFO:teuthology.orchestra.run.smithi098.stdout:(38/174): ceph-mgr-diskprediction-local-19.0.0- 6.8 MB/s | 7.4 MB 00:01 2024-03-22T01:23:31.488 INFO:teuthology.orchestra.run.smithi072.stdout:(63/174): ceph-test-19.0.0-2344.gc3a04c0e.el9.x 6.0 MB/s | 48 MB 00:08 2024-03-22T01:23:31.513 INFO:teuthology.orchestra.run.smithi072.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 32 kB/s | 24 kB 00:00 2024-03-22T01:23:31.539 INFO:teuthology.orchestra.run.smithi072.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 358 kB/s | 18 kB 00:00 2024-03-22T01:23:31.566 INFO:teuthology.orchestra.run.smithi072.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.7 MB/s | 144 kB 00:00 2024-03-22T01:23:31.569 INFO:teuthology.orchestra.run.smithi098.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 2.7 MB/s | 662 kB 00:00 2024-03-22T01:23:31.591 INFO:teuthology.orchestra.run.smithi072.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 609 kB/s | 32 kB 00:00 2024-03-22T01:23:31.603 INFO:teuthology.orchestra.run.smithi098.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 4.7 MB/s | 920 kB 00:00 2024-03-22T01:23:31.628 INFO:teuthology.orchestra.run.smithi098.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 106 kB 00:00 2024-03-22T01:23:31.661 INFO:teuthology.orchestra.run.smithi098.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 5.2 MB/s | 307 kB 00:00 2024-03-22T01:23:31.678 INFO:teuthology.orchestra.run.smithi098.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 628 kB/s | 32 kB 00:00 2024-03-22T01:23:31.734 INFO:teuthology.orchestra.run.smithi072.stdout:(68/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 29 MB/s | 4.8 MB 00:00 2024-03-22T01:23:31.759 INFO:teuthology.orchestra.run.smithi072.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.3 MB/s | 217 kB 00:00 2024-03-22T01:23:31.762 INFO:teuthology.orchestra.run.smithi098.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 30 MB/s | 3.0 MB 00:00 2024-03-22T01:23:31.785 INFO:teuthology.orchestra.run.smithi072.stdout:(70/174): python3-oauthlib-3.1.1-5.el9.noarch.r 3.4 MB/s | 174 kB 00:00 2024-03-22T01:23:31.796 INFO:teuthology.orchestra.run.smithi098.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 147 kB/s | 17 kB 00:00 2024-03-22T01:23:31.810 INFO:teuthology.orchestra.run.smithi072.stdout:(71/174): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 69 kB 00:00 2024-03-22T01:23:31.821 INFO:teuthology.orchestra.run.smithi098.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 171 kB/s | 10 kB 00:00 2024-03-22T01:23:31.836 INFO:teuthology.orchestra.run.smithi072.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 815 kB/s | 41 kB 00:00 2024-03-22T01:23:31.847 INFO:teuthology.orchestra.run.smithi098.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-22T01:23:31.870 INFO:teuthology.orchestra.run.smithi072.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 3.9 MB/s | 238 kB 00:00 2024-03-22T01:23:31.872 INFO:teuthology.orchestra.run.smithi098.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 881 kB/s | 44 kB 00:00 2024-03-22T01:23:31.895 INFO:teuthology.orchestra.run.smithi072.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.2 MB/s | 132 kB 00:00 2024-03-22T01:23:31.905 INFO:teuthology.orchestra.run.smithi098.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-22T01:23:31.922 INFO:teuthology.orchestra.run.smithi072.stdout:(75/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-22T01:23:31.943 INFO:teuthology.orchestra.run.smithi098.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 8.0 MB/s | 578 kB 00:00 2024-03-22T01:23:31.946 INFO:teuthology.orchestra.run.smithi072.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 937 kB/s | 47 kB 00:00 2024-03-22T01:23:31.969 INFO:teuthology.orchestra.run.smithi098.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 744 kB/s | 46 kB 00:00 2024-03-22T01:23:31.972 INFO:teuthology.orchestra.run.smithi072.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 844 kB/s | 43 kB 00:00 2024-03-22T01:23:31.994 INFO:teuthology.orchestra.run.smithi098.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-03-22T01:23:32.020 INFO:teuthology.orchestra.run.smithi098.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.6 MB/s | 187 kB 00:00 2024-03-22T01:23:32.028 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - AppStream 640 kB/s | 19 MB 00:30 2024-03-22T01:23:32.054 INFO:teuthology.orchestra.run.smithi098.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.6 MB/s | 217 kB 00:00 2024-03-22T01:23:32.079 INFO:teuthology.orchestra.run.smithi098.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 578 kB/s | 34 kB 00:00 2024-03-22T01:23:32.239 INFO:teuthology.orchestra.run.smithi072.stdout:(78/174): python3-toml-0.10.2-6.el9.noarch.rpm 146 kB/s | 39 kB 00:00 2024-03-22T01:23:32.246 INFO:teuthology.orchestra.run.smithi098.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 25 MB/s | 4.7 MB 00:00 2024-03-22T01:23:32.280 INFO:teuthology.orchestra.run.smithi098.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 7.6 MB/s | 1.5 MB 00:00 2024-03-22T01:23:32.322 INFO:teuthology.orchestra.run.smithi098.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 11 MB/s | 811 kB 00:00 2024-03-22T01:23:32.347 INFO:teuthology.orchestra.run.smithi098.stdout:(59/174): postgresql-private-libs-13.11-1.el9.x 2.0 MB/s | 134 kB 00:00 2024-03-22T01:23:32.506 INFO:teuthology.orchestra.run.smithi072.stdout:(79/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 28 MB/s | 16 MB 00:00 2024-03-22T01:23:32.540 INFO:teuthology.orchestra.run.smithi072.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 137 kB/s | 41 kB 00:00 2024-03-22T01:23:32.576 INFO:teuthology.orchestra.run.smithi072.stdout:(81/174): socat-1.7.4.1-5.el9.x86_64.rpm 4.3 MB/s | 300 kB 00:00 2024-03-22T01:23:32.601 INFO:teuthology.orchestra.run.smithi072.stdout:(82/174): uuid-1.6.2-55.el9.x86_64.rpm 939 kB/s | 56 kB 00:00 2024-03-22T01:23:32.627 INFO:teuthology.orchestra.run.smithi072.stdout:(83/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-22T01:23:32.652 INFO:teuthology.orchestra.run.smithi072.stdout:(84/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 872 kB/s | 44 kB 00:00 2024-03-22T01:23:32.678 INFO:teuthology.orchestra.run.smithi072.stdout:(85/174): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-03-22T01:23:32.712 INFO:teuthology.orchestra.run.smithi072.stdout:(86/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2024-03-22T01:23:32.762 INFO:teuthology.orchestra.run.smithi072.stdout:(87/174): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-03-22T01:23:32.788 INFO:teuthology.orchestra.run.smithi072.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 103 kB 00:00 2024-03-22T01:23:32.797 INFO:teuthology.orchestra.run.smithi098.stdout:(60/174): ceph-test-19.0.0-2344.gc3a04c0e.el9.x 11 MB/s | 48 MB 00:04 2024-03-22T01:23:32.822 INFO:teuthology.orchestra.run.smithi072.stdout:(89/174): python3-pycparser-2.20-6.el9.noarch.r 2.1 MB/s | 124 kB 00:00 2024-03-22T01:23:32.856 INFO:teuthology.orchestra.run.smithi072.stdout:(90/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.7 MB/s | 187 kB 00:00 2024-03-22T01:23:32.897 INFO:teuthology.orchestra.run.smithi098.stdout:(61/174): postgresql-server-13.11-1.el9.x86_64. 9.9 MB/s | 5.7 MB 00:00 2024-03-22T01:23:32.931 INFO:teuthology.orchestra.run.smithi098.stdout:(62/174): protobuf-3.14.0-13.el9.x86_64.rpm 1.7 MB/s | 1.0 MB 00:00 2024-03-22T01:23:32.956 INFO:teuthology.orchestra.run.smithi098.stdout:(63/174): python3-attrs-20.3.0-7.el9.noarch.rpm 499 kB/s | 78 kB 00:00 2024-03-22T01:23:33.123 INFO:teuthology.orchestra.run.smithi098.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 26 MB/s | 5.8 MB 00:00 2024-03-22T01:23:33.148 INFO:teuthology.orchestra.run.smithi098.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 129 kB/s | 24 kB 00:00 2024-03-22T01:23:33.174 INFO:teuthology.orchestra.run.smithi098.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 85 kB/s | 18 kB 00:00 2024-03-22T01:23:33.199 INFO:teuthology.orchestra.run.smithi098.stdout:(67/174): python3-mako-1.1.4-6.el9.noarch.rpm 1.9 MB/s | 144 kB 00:00 2024-03-22T01:23:33.224 INFO:teuthology.orchestra.run.smithi098.stdout:(68/174): python3-markupsafe-1.1.1-12.el9.x86_6 637 kB/s | 32 kB 00:00 2024-03-22T01:23:33.333 INFO:teuthology.orchestra.run.smithi098.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 30 MB/s | 4.8 MB 00:00 2024-03-22T01:23:33.358 INFO:teuthology.orchestra.run.smithi098.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.3 MB/s | 217 kB 00:00 2024-03-22T01:23:33.383 INFO:teuthology.orchestra.run.smithi098.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.1 MB/s | 174 kB 00:00 2024-03-22T01:23:33.409 INFO:teuthology.orchestra.run.smithi098.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 919 kB/s | 69 kB 00:00 2024-03-22T01:23:33.434 INFO:teuthology.orchestra.run.smithi098.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 825 kB/s | 41 kB 00:00 2024-03-22T01:23:33.459 INFO:teuthology.orchestra.run.smithi098.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 3.1 MB/s | 238 kB 00:00 2024-03-22T01:23:33.484 INFO:teuthology.orchestra.run.smithi098.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-22T01:23:33.510 INFO:teuthology.orchestra.run.smithi098.stdout:(76/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-22T01:23:33.535 INFO:teuthology.orchestra.run.smithi098.stdout:(77/174): python3-pytz-2021.1-5.el9.noarch.rpm 633 kB/s | 47 kB 00:00 2024-03-22T01:23:33.560 INFO:teuthology.orchestra.run.smithi098.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 572 kB/s | 43 kB 00:00 2024-03-22T01:23:33.602 INFO:teuthology.orchestra.run.smithi098.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 580 kB/s | 39 kB 00:00 2024-03-22T01:23:33.681 INFO:teuthology.orchestra.run.smithi072.stdout:(91/174): ceph-mgr-diskprediction-local-19.0.0- 940 kB/s | 7.4 MB 00:08 2024-03-22T01:23:33.727 INFO:teuthology.orchestra.run.smithi098.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 248 kB/s | 41 kB 00:00 2024-03-22T01:23:33.740 INFO:teuthology.orchestra.run.smithi072.stdout:(92/174): jq-1.6-16.el9.x86_64.rpm 204 kB/s | 187 kB 00:00 2024-03-22T01:23:33.778 INFO:teuthology.orchestra.run.smithi098.stdout:(81/174): socat-1.7.4.1-5.el9.x86_64.rpm 1.7 MB/s | 300 kB 00:00 2024-03-22T01:23:33.945 INFO:teuthology.orchestra.run.smithi098.stdout:(82/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 38 MB/s | 16 MB 00:00 2024-03-22T01:23:33.970 INFO:teuthology.orchestra.run.smithi098.stdout:(83/174): uuid-1.6.2-55.el9.x86_64.rpm 235 kB/s | 56 kB 00:00 2024-03-22T01:23:33.995 INFO:teuthology.orchestra.run.smithi098.stdout:(84/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 291 kB/s | 63 kB 00:00 2024-03-22T01:23:34.020 INFO:teuthology.orchestra.run.smithi098.stdout:(85/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 587 kB/s | 44 kB 00:00 2024-03-22T01:23:34.046 INFO:teuthology.orchestra.run.smithi098.stdout:(86/174): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-03-22T01:23:34.071 INFO:teuthology.orchestra.run.smithi098.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.1 MB/s | 241 kB 00:00 2024-03-22T01:23:34.113 INFO:teuthology.orchestra.run.smithi098.stdout:(88/174): python3-cryptography-36.0.1-4.el9.x86 12 MB/s | 1.1 MB 00:00 2024-03-22T01:23:34.139 INFO:teuthology.orchestra.run.smithi098.stdout:(89/174): python3-ply-3.11-14.el9.noarch.rpm 1.5 MB/s | 103 kB 00:00 2024-03-22T01:23:34.163 INFO:teuthology.orchestra.run.smithi098.stdout:(90/174): python3-pycparser-2.20-6.el9.noarch.r 1.3 MB/s | 124 kB 00:00 2024-03-22T01:23:34.189 INFO:teuthology.orchestra.run.smithi098.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.4 MB/s | 187 kB 00:00 2024-03-22T01:23:34.330 INFO:teuthology.orchestra.run.smithi098.stdout:(92/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 195 kB 00:00 2024-03-22T01:23:34.355 INFO:teuthology.orchestra.run.smithi098.stdout:(93/174): jq-1.6-16.el9.x86_64.rpm 977 kB/s | 187 kB 00:00 2024-03-22T01:23:34.381 INFO:teuthology.orchestra.run.smithi098.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-03-22T01:23:34.414 INFO:teuthology.orchestra.run.smithi098.stdout:(95/174): libgfortran-11.4.1-3.el9.x86_64.rpm 3.1 MB/s | 803 kB 00:00 2024-03-22T01:23:34.431 INFO:teuthology.orchestra.run.smithi098.stdout:(96/174): elfutils-libelf-devel-0.190-2.el9.x86 284 kB/s | 21 kB 00:00 2024-03-22T01:23:34.465 INFO:teuthology.orchestra.run.smithi098.stdout:(97/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 303 kB/s | 25 kB 00:00 2024-03-22T01:23:34.516 INFO:teuthology.orchestra.run.smithi072.stdout:(93/174): python3-requests-2.25.1-8.el9.noarch. 161 kB/s | 125 kB 00:00 2024-03-22T01:23:34.532 INFO:teuthology.orchestra.run.smithi098.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 2.4 MB/s | 245 kB 00:00 2024-03-22T01:23:34.558 INFO:teuthology.orchestra.run.smithi098.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 2.6 MB/s | 248 kB 00:00 2024-03-22T01:23:34.583 INFO:teuthology.orchestra.run.smithi098.stdout:(100/174): qatlib-23.11.0-1.el9.x86_64.rpm 4.2 MB/s | 216 kB 00:00 2024-03-22T01:23:34.591 INFO:teuthology.orchestra.run.smithi072.stdout:(94/174): libgfortran-11.4.1-3.el9.x86_64.rpm 463 kB/s | 803 kB 00:01 2024-03-22T01:23:34.608 INFO:teuthology.orchestra.run.smithi098.stdout:(101/174): qatlib-service-23.11.0-1.el9.x86_64. 693 kB/s | 35 kB 00:00 2024-03-22T01:23:34.625 INFO:teuthology.orchestra.run.smithi072.stdout:(95/174): libquadmath-11.4.1-3.el9.x86_64.rpm 207 kB/s | 195 kB 00:00 2024-03-22T01:23:34.633 INFO:teuthology.orchestra.run.smithi098.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 887 kB/s | 44 kB 00:00 2024-03-22T01:23:34.650 INFO:teuthology.orchestra.run.smithi098.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 2.2 MB/s | 94 kB 00:00 2024-03-22T01:23:34.685 INFO:teuthology.orchestra.run.smithi072.stdout:(96/174): elfutils-libelf-devel-0.190-2.el9.x86 127 kB/s | 21 kB 00:00 2024-03-22T01:23:34.718 INFO:teuthology.orchestra.run.smithi098.stdout:(104/174): openssl-devel-3.0.7-27.el9.x86_64.rp 14 MB/s | 4.1 MB 00:00 2024-03-22T01:23:34.728 INFO:teuthology.orchestra.run.smithi072.stdout:(97/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 187 kB/s | 25 kB 00:00 2024-03-22T01:23:34.870 INFO:teuthology.orchestra.run.smithi072.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 1.3 MB/s | 245 kB 00:00 2024-03-22T01:23:34.984 INFO:teuthology.orchestra.run.smithi098.stdout:(105/174): lua-devel-5.4.4-4.el9.x86_64.rpm 63 kB/s | 22 kB 00:00 2024-03-22T01:23:34.995 INFO:teuthology.orchestra.run.smithi072.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 928 kB/s | 248 kB 00:00 2024-03-22T01:23:35.021 INFO:teuthology.orchestra.run.smithi072.stdout:(100/174): qatlib-23.11.0-1.el9.x86_64.rpm 1.4 MB/s | 216 kB 00:00 2024-03-22T01:23:35.046 INFO:teuthology.orchestra.run.smithi072.stdout:(101/174): qatlib-service-23.11.0-1.el9.x86_64. 687 kB/s | 35 kB 00:00 2024-03-22T01:23:35.097 INFO:teuthology.orchestra.run.smithi072.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 588 kB/s | 44 kB 00:00 2024-03-22T01:23:35.181 INFO:teuthology.orchestra.run.smithi072.stdout:(103/174): lua-devel-5.4.4-4.el9.x86_64.rpm 267 kB/s | 22 kB 00:00 2024-03-22T01:23:35.206 INFO:teuthology.orchestra.run.smithi072.stdout:(104/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 587 kB/s | 94 kB 00:00 2024-03-22T01:23:35.518 INFO:teuthology.orchestra.run.smithi098.stdout:(106/174): gperftools-libs-2.9.1-2.el9.x86_64.r 579 kB/s | 309 kB 00:00 2024-03-22T01:23:35.557 INFO:teuthology.orchestra.run.smithi098.stdout:(107/174): protobuf-compiler-3.14.0-13.el9.x86_ 953 kB/s | 863 kB 00:00 2024-03-22T01:23:35.557 INFO:teuthology.orchestra.run.smithi072.stdout:(105/174): protobuf-compiler-3.14.0-13.el9.x86_ 2.2 MB/s | 863 kB 00:00 2024-03-22T01:23:35.591 INFO:teuthology.orchestra.run.smithi072.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.4 MB/s | 548 kB 00:00 2024-03-22T01:23:35.599 INFO:teuthology.orchestra.run.smithi098.stdout:(108/174): abseil-cpp-20211102.0-3.el9.x86_64.r 622 kB/s | 548 kB 00:00 2024-03-22T01:23:35.616 INFO:teuthology.orchestra.run.smithi098.stdout:(109/174): grpc-data-1.46.7-10.el9.noarch.rpm 200 kB/s | 19 kB 00:00 2024-03-22T01:23:35.633 INFO:teuthology.orchestra.run.smithi072.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 465 kB/s | 19 kB 00:00 2024-03-22T01:23:35.684 INFO:teuthology.orchestra.run.smithi072.stdout:(108/174): gperftools-libs-2.9.1-2.el9.x86_64.r 2.4 MB/s | 309 kB 00:00 2024-03-22T01:23:35.699 INFO:teuthology.orchestra.run.smithi098.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 259 kB/s | 26 kB 00:00 2024-03-22T01:23:35.725 INFO:teuthology.orchestra.run.smithi098.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 447 kB/s | 49 kB 00:00 2024-03-22T01:23:35.726 INFO:teuthology.orchestra.run.smithi072.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 618 kB/s | 26 kB 00:00 2024-03-22T01:23:35.768 INFO:teuthology.orchestra.run.smithi072.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 1.1 MB/s | 49 kB 00:00 2024-03-22T01:23:35.808 INFO:teuthology.orchestra.run.smithi098.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 621 kB/s | 67 kB 00:00 2024-03-22T01:23:35.834 INFO:teuthology.orchestra.run.smithi098.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 1.4 MB/s | 151 kB 00:00 2024-03-22T01:23:35.870 INFO:teuthology.orchestra.run.smithi072.stdout:(111/174): libarrow-9.0.0-10.el9.x86_64.rpm 19 MB/s | 4.4 MB 00:00 2024-03-22T01:23:35.896 INFO:teuthology.orchestra.run.smithi072.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 529 kB/s | 67 kB 00:00 2024-03-22T01:23:35.930 INFO:teuthology.orchestra.run.smithi072.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 2.5 MB/s | 151 kB 00:00 2024-03-22T01:23:35.989 INFO:teuthology.orchestra.run.smithi072.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 8.9 MB/s | 838 kB 00:00 2024-03-22T01:23:36.000 INFO:teuthology.orchestra.run.smithi098.stdout:(114/174): python3-asyncssh-2.13.2-5.el9.noarch 3.2 MB/s | 548 kB 00:00 2024-03-22T01:23:36.023 INFO:teuthology.orchestra.run.smithi072.stdout:(115/174): python3-asyncssh-2.13.2-5.el9.noarch 5.8 MB/s | 548 kB 00:00 2024-03-22T01:23:36.048 INFO:teuthology.orchestra.run.smithi072.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 736 kB/s | 43 kB 00:00 2024-03-22T01:23:36.074 INFO:teuthology.orchestra.run.smithi072.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 634 kB/s | 32 kB 00:00 2024-03-22T01:23:36.092 INFO:teuthology.orchestra.run.smithi098.stdout:(115/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 474 kB/s | 43 kB 00:00 2024-03-22T01:23:36.099 INFO:teuthology.orchestra.run.smithi072.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 296 kB/s | 15 kB 00:00 2024-03-22T01:23:36.125 INFO:teuthology.orchestra.run.smithi072.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 3.3 MB/s | 172 kB 00:00 2024-03-22T01:23:36.201 INFO:teuthology.orchestra.run.smithi098.stdout:(116/174): python3-cachetools-4.2.4-1.el9.noarc 297 kB/s | 32 kB 00:00 2024-03-22T01:23:36.208 INFO:teuthology.orchestra.run.smithi072.stdout:(120/174): openssl-devel-3.0.7-27.el9.x86_64.rp 2.6 MB/s | 4.1 MB 00:01 2024-03-22T01:23:36.237 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - CRB 8.1 MB/s | 6.1 MB 00:00 2024-03-22T01:23:36.242 INFO:teuthology.orchestra.run.smithi072.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 2.4 MB/s | 358 kB 00:00 2024-03-22T01:23:36.268 INFO:teuthology.orchestra.run.smithi072.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.1 MB/s | 303 kB 00:00 2024-03-22T01:23:36.293 INFO:teuthology.orchestra.run.smithi098.stdout:(117/174): python3-certifi-2021.10.8-2.el9.noar 163 kB/s | 15 kB 00:00 2024-03-22T01:23:36.294 INFO:teuthology.orchestra.run.smithi072.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 418 kB/s | 35 kB 00:00 2024-03-22T01:23:36.327 INFO:teuthology.orchestra.run.smithi072.stdout:(124/174): python3-google-auth-2.28.2-1.el9.noa 2.3 MB/s | 203 kB 00:00 2024-03-22T01:23:36.370 INFO:teuthology.orchestra.run.smithi072.stdout:(125/174): python3-jaraco-8.2.1-3.el9.noarch.rp 255 kB/s | 11 kB 00:00 2024-03-22T01:23:36.385 INFO:teuthology.orchestra.run.smithi098.stdout:(118/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.8 MB/s | 172 kB 00:00 2024-03-22T01:23:36.397 INFO:teuthology.orchestra.run.smithi072.stdout:(126/174): python3-grpcio-tools-1.46.7-10.el9.x 1.4 MB/s | 144 kB 00:00 2024-03-22T01:23:36.422 INFO:teuthology.orchestra.run.smithi072.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 339 kB/s | 18 kB 00:00 2024-03-22T01:23:36.452 INFO:teuthology.orchestra.run.smithi098.stdout:(119/174): libarrow-9.0.0-10.el9.x86_64.rpm 4.9 MB/s | 4.4 MB 00:00 2024-03-22T01:23:36.481 INFO:teuthology.orchestra.run.smithi072.stdout:(128/174): python3-grpcio-1.46.7-10.el9.x86_64. 9.6 MB/s | 2.0 MB 00:00 2024-03-22T01:23:36.485 INFO:teuthology.orchestra.run.smithi098.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 3.5 MB/s | 358 kB 00:00 2024-03-22T01:23:36.506 INFO:teuthology.orchestra.run.smithi072.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 212 kB/s | 23 kB 00:00 2024-03-22T01:23:36.510 INFO:teuthology.orchestra.run.smithi098.stdout:(121/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.2 MB/s | 838 kB 00:00 2024-03-22T01:23:36.532 INFO:teuthology.orchestra.run.smithi072.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 178 kB/s | 19 kB 00:00 2024-03-22T01:23:36.557 INFO:teuthology.orchestra.run.smithi072.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 388 kB/s | 20 kB 00:00 2024-03-22T01:23:36.583 INFO:teuthology.orchestra.run.smithi072.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 914 kB/s | 46 kB 00:00 2024-03-22T01:23:36.586 INFO:teuthology.orchestra.run.smithi098.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 351 kB/s | 35 kB 00:00 2024-03-22T01:23:36.608 INFO:teuthology.orchestra.run.smithi072.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 1.5 MB/s | 79 kB 00:00 2024-03-22T01:23:36.634 INFO:teuthology.orchestra.run.smithi072.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 1.1 MB/s | 58 kB 00:00 2024-03-22T01:23:36.668 INFO:teuthology.orchestra.run.smithi072.stdout:(135/174): python3-pecan-1.4.2-3.el9.noarch.rpm 4.5 MB/s | 272 kB 00:00 2024-03-22T01:23:36.693 INFO:teuthology.orchestra.run.smithi072.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 279 kB/s | 16 kB 00:00 2024-03-22T01:23:36.719 INFO:teuthology.orchestra.run.smithi072.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.7 MB/s | 90 kB 00:00 2024-03-22T01:23:36.744 INFO:teuthology.orchestra.run.smithi072.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 556 kB/s | 28 kB 00:00 2024-03-22T01:23:36.753 INFO:teuthology.orchestra.run.smithi098.stdout:(123/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.0 MB/s | 303 kB 00:00 2024-03-22T01:23:36.770 INFO:teuthology.orchestra.run.smithi072.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 606 kB/s | 31 kB 00:00 2024-03-22T01:23:36.837 INFO:teuthology.orchestra.run.smithi098.stdout:(124/174): python3-grpcio-1.46.7-10.el9.x86_64. 8.1 MB/s | 2.0 MB 00:00 2024-03-22T01:23:36.862 INFO:teuthology.orchestra.run.smithi098.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 1.3 MB/s | 144 kB 00:00 2024-03-22T01:23:36.887 INFO:teuthology.orchestra.run.smithi072.stdout:(140/174): python3-routes-2.5.1-5.el9.noarch.rp 1.3 MB/s | 188 kB 00:00 2024-03-22T01:23:36.929 INFO:teuthology.orchestra.run.smithi098.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 116 kB/s | 11 kB 00:00 2024-03-22T01:23:36.954 INFO:teuthology.orchestra.run.smithi098.stdout:(127/174): python3-google-auth-2.28.2-1.el9.noa 457 kB/s | 203 kB 00:00 2024-03-22T01:23:36.964 INFO:teuthology.orchestra.run.smithi072.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 304 kB/s | 59 kB 00:00 2024-03-22T01:23:36.979 INFO:teuthology.orchestra.run.smithi098.stdout:(128/174): python3-jaraco-classes-3.2.1-5.el9.n 151 kB/s | 18 kB 00:00 2024-03-22T01:23:36.993 INFO:teuthology.orchestra.run.smithi072.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 348 kB/s | 36 kB 00:00 2024-03-22T01:23:37.022 INFO:teuthology.orchestra.run.smithi098.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 251 kB/s | 23 kB 00:00 2024-03-22T01:23:37.033 INFO:teuthology.orchestra.run.smithi072.stdout:(143/174): python3-kubernetes-26.1.0-2.el9.noar 1.9 MB/s | 1.0 MB 00:00 2024-03-22T01:23:37.058 INFO:teuthology.orchestra.run.smithi072.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 548 kB/s | 51 kB 00:00 2024-03-22T01:23:37.072 INFO:teuthology.orchestra.run.smithi098.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 214 kB/s | 20 kB 00:00 2024-03-22T01:23:37.083 INFO:teuthology.orchestra.run.smithi072.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-03-22T01:23:37.117 INFO:teuthology.orchestra.run.smithi072.stdout:(146/174): python3-webob-1.8.7-6.el9.noarch.rpm 1.8 MB/s | 230 kB 00:00 2024-03-22T01:23:37.143 INFO:teuthology.orchestra.run.smithi072.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 338 kB/s | 20 kB 00:00 2024-03-22T01:23:37.168 INFO:teuthology.orchestra.run.smithi072.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 3.7 MB/s | 192 kB 00:00 2024-03-22T01:23:37.172 INFO:teuthology.orchestra.run.smithi098.stdout:(131/174): python3-logutils-0.3.5-21.el9.noarch 463 kB/s | 46 kB 00:00 2024-03-22T01:23:37.202 INFO:teuthology.orchestra.run.smithi072.stdout:(149/174): python3-werkzeug-2.0.3-3.el9.1.noarc 2.9 MB/s | 427 kB 00:00 2024-03-22T01:23:37.205 INFO:teuthology.orchestra.run.smithi098.stdout:(132/174): python3-kubernetes-26.1.0-2.el9.noar 5.6 MB/s | 1.0 MB 00:00 2024-03-22T01:23:37.231 INFO:teuthology.orchestra.run.smithi098.stdout:(133/174): python3-jaraco-functools-3.5.0-2.el9 70 kB/s | 19 kB 00:00 2024-03-22T01:23:37.261 INFO:teuthology.orchestra.run.smithi072.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-03-22T01:23:37.264 INFO:teuthology.orchestra.run.smithi098.stdout:(134/174): python3-more-itertools-8.12.0-2.el9. 855 kB/s | 79 kB 00:00 2024-03-22T01:23:37.298 INFO:teuthology.orchestra.run.smithi098.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 625 kB/s | 58 kB 00:00 2024-03-22T01:23:37.365 INFO:teuthology.orchestra.run.smithi098.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 164 kB/s | 16 kB 00:00 2024-03-22T01:23:37.390 INFO:teuthology.orchestra.run.smithi098.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 980 kB/s | 90 kB 00:00 2024-03-22T01:23:37.412 INFO:teuthology.orchestra.run.smithi072.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 2.1 MB/s | 321 kB 00:00 2024-03-22T01:23:37.457 INFO:teuthology.orchestra.run.smithi098.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 306 kB/s | 28 kB 00:00 2024-03-22T01:23:37.483 INFO:teuthology.orchestra.run.smithi098.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 334 kB/s | 31 kB 00:00 2024-03-22T01:23:37.558 INFO:teuthology.orchestra.run.smithi098.stdout:(140/174): python3-pecan-1.4.2-3.el9.noarch.rpm 833 kB/s | 272 kB 00:00 2024-03-22T01:23:37.583 INFO:teuthology.orchestra.run.smithi098.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 590 kB/s | 59 kB 00:00 2024-03-22T01:23:37.625 INFO:teuthology.orchestra.run.smithi098.stdout:(142/174): python3-routes-2.5.1-5.el9.noarch.rp 1.1 MB/s | 188 kB 00:00 2024-03-22T01:23:37.650 INFO:teuthology.orchestra.run.smithi098.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 387 kB/s | 36 kB 00:00 2024-03-22T01:23:37.676 INFO:teuthology.orchestra.run.smithi098.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 553 kB/s | 51 kB 00:00 2024-03-22T01:23:37.743 INFO:teuthology.orchestra.run.smithi098.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 976 kB/s | 90 kB 00:00 2024-03-22T01:23:37.785 INFO:teuthology.orchestra.run.smithi098.stdout:(146/174): python3-werkzeug-2.0.3-3.el9.1.noarc 3.8 MB/s | 427 kB 00:00 2024-03-22T01:23:37.810 INFO:teuthology.orchestra.run.smithi098.stdout:(147/174): python3-webob-1.8.7-6.el9.noarch.rpm 1.2 MB/s | 230 kB 00:00 2024-03-22T01:23:37.835 INFO:teuthology.orchestra.run.smithi098.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 217 kB/s | 20 kB 00:00 2024-03-22T01:23:37.877 INFO:teuthology.orchestra.run.smithi098.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 2.0 MB/s | 192 kB 00:00 2024-03-22T01:23:37.896 INFO:teuthology.orchestra.run.smithi072.stdout:(152/174): librbd1-19.0.0-2344.gc3a04c0e.el9.x8 4.5 MB/s | 3.1 MB 00:00 2024-03-22T01:23:37.929 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - Extras packages 27 kB/s | 16 kB 00:00 2024-03-22T01:23:38.469 INFO:teuthology.orchestra.run.smithi098.stdout:(150/174): librados2-19.0.0-2344.gc3a04c0e.el9. 5.4 MB/s | 3.4 MB 00:00 2024-03-22T01:23:38.511 INFO:teuthology.orchestra.run.smithi098.stdout:(151/174): thrift-0.15.0-2.el9.x86_64.rpm 2.3 MB/s | 1.6 MB 00:00 2024-03-22T01:23:38.620 INFO:teuthology.orchestra.run.smithi098.stdout:(152/174): xfsdump-3.1.12-4.el9.x86_64.rpm 2.1 MB/s | 321 kB 00:00 2024-03-22T01:23:38.738 INFO:teuthology.orchestra.run.smithi072.stdout:(153/174): librados2-19.0.0-2344.gc3a04c0e.el9. 2.2 MB/s | 3.4 MB 00:01 2024-03-22T01:23:38.762 INFO:teuthology.orchestra.run.smithi098.stdout:(153/174): librbd1-19.0.0-2344.gc3a04c0e.el9.x8 3.7 MB/s | 3.1 MB 00:00 2024-03-22T01:23:38.904 INFO:teuthology.orchestra.run.smithi098.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 297 kB/s | 42 kB 00:00 2024-03-22T01:23:38.987 INFO:teuthology.orchestra.run.smithi098.stdout:(155/174): NetworkManager-tui-1.47.2-1.el9.x86_ 2.9 MB/s | 248 kB 00:00 2024-03-22T01:23:39.029 INFO:teuthology.orchestra.run.smithi098.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 1.7 MB/s | 71 kB 00:00 2024-03-22T01:23:39.088 INFO:teuthology.orchestra.run.smithi098.stdout:(157/174): NetworkManager-libnm-1.47.2-1.el9.x8 3.9 MB/s | 1.8 MB 00:00 2024-03-22T01:23:39.113 INFO:teuthology.orchestra.run.smithi098.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 437 kB/s | 36 kB 00:00 2024-03-22T01:23:39.155 INFO:teuthology.orchestra.run.smithi098.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.8 MB/s | 192 kB 00:00 2024-03-22T01:23:39.180 INFO:teuthology.orchestra.run.smithi098.stdout:(160/174): elfutils-libs-0.190-2.el9.x86_64.rpm 3.7 MB/s | 256 kB 00:00 2024-03-22T01:23:39.205 INFO:teuthology.orchestra.run.smithi098.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 458 kB/s | 23 kB 00:00 2024-03-22T01:23:39.231 INFO:teuthology.orchestra.run.smithi098.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 2.1 MB/s | 108 kB 00:00 2024-03-22T01:23:39.256 INFO:teuthology.orchestra.run.smithi098.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 3.0 MB/s | 155 kB 00:00 2024-03-22T01:23:39.281 INFO:teuthology.orchestra.run.smithi098.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 2.6 MB/s | 136 kB 00:00 2024-03-22T01:23:39.323 INFO:teuthology.orchestra.run.smithi098.stdout:(165/174): NetworkManager-1.47.2-1.el9.x86_64.r 2.9 MB/s | 2.3 MB 00:00 2024-03-22T01:23:39.324 INFO:teuthology.orchestra.run.smithi089.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-03-22T01:23:39.348 INFO:teuthology.orchestra.run.smithi098.stdout:(166/174): libsmartcols-2.37.4-18.el9.x86_64.rp 692 kB/s | 64 kB 00:00 2024-03-22T01:23:39.366 INFO:teuthology.orchestra.run.smithi098.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 342 kB/s | 29 kB 00:00 2024-03-22T01:23:39.421 INFO:teuthology.orchestra.run.smithi072.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 62 kB/s | 42 kB 00:00 2024-03-22T01:23:39.591 INFO:teuthology.orchestra.run.smithi098.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 4.5 MB/s | 1.2 MB 00:00 2024-03-22T01:23:39.833 INFO:teuthology.orchestra.run.smithi098.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 1.9 MB/s | 465 kB 00:00 2024-03-22T01:23:39.874 INFO:teuthology.orchestra.run.smithi098.stdout:(170/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 4.1 MB/s | 2.2 MB 00:00 2024-03-22T01:23:39.892 INFO:teuthology.orchestra.run.smithi098.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 530 kB/s | 31 kB 00:00 2024-03-22T01:23:39.975 INFO:teuthology.orchestra.run.smithi098.stdout:(172/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 893 kB/s | 74 kB 00:00 2024-03-22T01:23:40.075 INFO:teuthology.orchestra.run.smithi098.stdout:(173/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 5.4 MB/s | 1.1 MB 00:00 2024-03-22T01:23:40.234 INFO:teuthology.orchestra.run.smithi098.stdout:(174/174): util-linux-2.37.4-18.el9.x86_64.rpm 2.6 MB/s | 2.3 MB 00:00 2024-03-22T01:23:40.238 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-22T01:23:40.238 INFO:teuthology.orchestra.run.smithi098.stdout:Total 15 MB/s | 226 MB 00:15 2024-03-22T01:23:40.572 INFO:teuthology.orchestra.run.smithi072.stdout:(155/174): NetworkManager-tui-1.47.2-1.el9.x86_ 216 kB/s | 248 kB 00:01 2024-03-22T01:23:40.701 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-22T01:23:40.968 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-22T01:23:40.968 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-22T01:23:41.073 INFO:teuthology.orchestra.run.smithi072.stdout:(156/174): NetworkManager-libnm-1.47.2-1.el9.x8 591 kB/s | 1.8 MB 00:03 2024-03-22T01:23:41.216 INFO:teuthology.orchestra.run.smithi072.stdout:(157/174): NetworkManager-1.47.2-1.el9.x86_64.r 621 kB/s | 2.3 MB 00:03 2024-03-22T01:23:41.266 INFO:teuthology.orchestra.run.smithi072.stdout:(158/174): acl-2.3.1-4.el9.x86_64.rpm 103 kB/s | 71 kB 00:00 2024-03-22T01:23:41.675 INFO:teuthology.orchestra.run.smithi072.stdout:(159/174): elfutils-debuginfod-client-0.190-2.e 60 kB/s | 36 kB 00:00 2024-03-22T01:23:42.126 INFO:teuthology.orchestra.run.smithi072.stdout:(160/174): elfutils-libelf-0.190-2.el9.x86_64.r 211 kB/s | 192 kB 00:00 2024-03-22T01:23:42.260 INFO:teuthology.orchestra.run.smithi072.stdout:(161/174): elfutils-libs-0.190-2.el9.x86_64.rpm 258 kB/s | 256 kB 00:00 2024-03-22T01:23:42.302 INFO:teuthology.orchestra.run.smithi072.stdout:(162/174): libacl-2.3.1-4.el9.x86_64.rpm 37 kB/s | 23 kB 00:00 2024-03-22T01:23:42.677 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-22T01:23:42.677 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-22T01:23:42.994 INFO:teuthology.orchestra.run.smithi072.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 125 kB/s | 108 kB 00:00 2024-03-22T01:23:43.186 INFO:teuthology.orchestra.run.smithi072.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 167 kB/s | 155 kB 00:00 2024-03-22T01:23:43.212 INFO:teuthology.orchestra.run.smithi072.stdout:(165/174): libmount-2.37.4-18.el9.x86_64.rpm 149 kB/s | 136 kB 00:00 2024-03-22T01:23:43.729 INFO:teuthology.orchestra.run.smithi072.stdout:(166/174): libsmartcols-2.37.4-18.el9.x86_64.rp 87 kB/s | 64 kB 00:00 2024-03-22T01:23:43.846 INFO:teuthology.orchestra.run.smithi072.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 44 kB/s | 29 kB 00:00 2024-03-22T01:23:44.238 INFO:teuthology.orchestra.run.smithi089.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-22T01:23:45.922 INFO:teuthology.orchestra.run.smithi072.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 457 kB/s | 1.2 MB 00:02 2024-03-22T01:23:46.567 INFO:teuthology.orchestra.run.smithi089.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-22T01:23:46.568 INFO:teuthology.orchestra.run.smithi089.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-22T01:23:46.581 INFO:teuthology.orchestra.run.smithi089.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-22T01:23:46.582 INFO:teuthology.orchestra.run.smithi089.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-22T01:23:46.583 INFO:teuthology.orchestra.run.smithi089.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-22T01:23:46.584 INFO:teuthology.orchestra.run.smithi089.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-22T01:23:46.586 INFO:teuthology.orchestra.run.smithi089.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-22T01:23:46.588 INFO:teuthology.orchestra.run.smithi089.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-22T01:23:46.589 INFO:teuthology.orchestra.run.smithi089.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-22T01:23:46.669 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-03-22T01:23:46.699 INFO:teuthology.orchestra.run.smithi089.stdout:============================================================================================ 2024-03-22T01:23:46.699 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-03-22T01:23:46.699 INFO:teuthology.orchestra.run.smithi089.stdout:============================================================================================ 2024-03-22T01:23:46.699 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-03-22T01:23:46.699 INFO:teuthology.orchestra.run.smithi089.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-22T01:23:46.699 INFO:teuthology.orchestra.run.smithi089.stdout: ceph x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 6.6 k 2024-03-22T01:23:46.699 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-base x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 5.4 M 2024-03-22T01:23:46.699 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-fuse x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 890 k 2024-03-22T01:23:46.700 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 145 k 2024-03-22T01:23:46.700 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 1.5 M 2024-03-22T01:23:46.700 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2344.gc3a04c0e.el9 ceph-noarch 140 k 2024-03-22T01:23:46.700 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2344.gc3a04c0e.el9 ceph-noarch 3.5 M 2024-03-22T01:23:46.700 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2344.gc3a04c0e.el9 ceph-noarch 7.4 M 2024-03-22T01:23:46.700 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-rook noarch 2:19.0.0-2344.gc3a04c0e.el9 ceph-noarch 49 k 2024-03-22T01:23:46.700 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-radosgw x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 9.8 M 2024-03-22T01:23:46.700 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-test x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 48 M 2024-03-22T01:23:46.700 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-volume noarch 2:19.0.0-2344.gc3a04c0e.el9 ceph-noarch 264 k 2024-03-22T01:23:46.700 INFO:teuthology.orchestra.run.smithi089.stdout: cephadm noarch 2:19.0.0-2344.gc3a04c0e.el9 ceph-noarch 750 k 2024-03-22T01:23:46.700 INFO:teuthology.orchestra.run.smithi089.stdout: cephfs-mirror x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 222 k 2024-03-22T01:23:46.700 INFO:teuthology.orchestra.run.smithi089.stdout: cephfs-shell x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 27 k 2024-03-22T01:23:46.700 INFO:teuthology.orchestra.run.smithi089.stdout: cephfs-top noarch 2:19.0.0-2344.gc3a04c0e.el9 ceph-noarch 18 k 2024-03-22T01:23:46.700 INFO:teuthology.orchestra.run.smithi089.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-22T01:23:46.701 INFO:teuthology.orchestra.run.smithi089.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-22T01:23:46.701 INFO:teuthology.orchestra.run.smithi089.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-22T01:23:46.701 INFO:teuthology.orchestra.run.smithi089.stdout: libcephfs-devel x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 32 k 2024-03-22T01:23:46.701 INFO:teuthology.orchestra.run.smithi089.stdout: libcephfs2 x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 747 k 2024-03-22T01:23:46.701 INFO:teuthology.orchestra.run.smithi089.stdout: librados-devel x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 127 k 2024-03-22T01:23:46.701 INFO:teuthology.orchestra.run.smithi089.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-22T01:23:46.701 INFO:teuthology.orchestra.run.smithi089.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-22T01:23:46.701 INFO:teuthology.orchestra.run.smithi089.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-22T01:23:46.701 INFO:teuthology.orchestra.run.smithi089.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-22T01:23:46.701 INFO:teuthology.orchestra.run.smithi089.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-22T01:23:46.701 INFO:teuthology.orchestra.run.smithi089.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-22T01:23:46.701 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cephfs x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 161 k 2024-03-22T01:23:46.701 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rados x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 321 k 2024-03-22T01:23:46.701 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rbd x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 298 k 2024-03-22T01:23:46.701 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rgw x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 99 k 2024-03-22T01:23:46.702 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-fuse x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 85 k 2024-03-22T01:23:46.702 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-mirror x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 3.1 M 2024-03-22T01:23:46.702 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-nbd x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 170 k 2024-03-22T01:23:46.702 INFO:teuthology.orchestra.run.smithi089.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-22T01:23:46.702 INFO:teuthology.orchestra.run.smithi089.stdout:Upgrading: 2024-03-22T01:23:46.702 INFO:teuthology.orchestra.run.smithi089.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-22T01:23:46.702 INFO:teuthology.orchestra.run.smithi089.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-22T01:23:46.702 INFO:teuthology.orchestra.run.smithi089.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-22T01:23:46.702 INFO:teuthology.orchestra.run.smithi089.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-22T01:23:46.702 INFO:teuthology.orchestra.run.smithi089.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-22T01:23:46.702 INFO:teuthology.orchestra.run.smithi089.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-22T01:23:46.702 INFO:teuthology.orchestra.run.smithi089.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-22T01:23:46.702 INFO:teuthology.orchestra.run.smithi089.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-22T01:23:46.703 INFO:teuthology.orchestra.run.smithi089.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-22T01:23:46.703 INFO:teuthology.orchestra.run.smithi089.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-22T01:23:46.703 INFO:teuthology.orchestra.run.smithi089.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-22T01:23:46.703 INFO:teuthology.orchestra.run.smithi089.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-22T01:23:46.703 INFO:teuthology.orchestra.run.smithi089.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-22T01:23:46.703 INFO:teuthology.orchestra.run.smithi089.stdout: librados2 x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 3.4 M 2024-03-22T01:23:46.703 INFO:teuthology.orchestra.run.smithi089.stdout: librbd1 x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 3.1 M 2024-03-22T01:23:46.703 INFO:teuthology.orchestra.run.smithi089.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-22T01:23:46.703 INFO:teuthology.orchestra.run.smithi089.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-22T01:23:46.703 INFO:teuthology.orchestra.run.smithi089.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-22T01:23:46.703 INFO:teuthology.orchestra.run.smithi089.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-22T01:23:46.703 INFO:teuthology.orchestra.run.smithi089.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-22T01:23:46.703 INFO:teuthology.orchestra.run.smithi089.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-22T01:23:46.703 INFO:teuthology.orchestra.run.smithi089.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-22T01:23:46.704 INFO:teuthology.orchestra.run.smithi089.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-22T01:23:46.704 INFO:teuthology.orchestra.run.smithi089.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-22T01:23:46.704 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-03-22T01:23:46.704 INFO:teuthology.orchestra.run.smithi089.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-22T01:23:46.704 INFO:teuthology.orchestra.run.smithi089.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-22T01:23:46.704 INFO:teuthology.orchestra.run.smithi089.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-22T01:23:46.704 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-common x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 21 M 2024-03-22T01:23:46.704 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2344.gc3a04c0e.el9 ceph-noarch 34 k 2024-03-22T01:23:46.704 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mds x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 2.4 M 2024-03-22T01:23:46.704 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2344.gc3a04c0e.el9 ceph-noarch 248 k 2024-03-22T01:23:46.704 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mon x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 4.7 M 2024-03-22T01:23:46.704 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-osd x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 17 M 2024-03-22T01:23:46.704 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2344.gc3a04c0e.el9 ceph-noarch 16 k 2024-03-22T01:23:46.704 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-selinux x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 25 k 2024-03-22T01:23:46.704 INFO:teuthology.orchestra.run.smithi089.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-22T01:23:46.705 INFO:teuthology.orchestra.run.smithi089.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-22T01:23:46.705 INFO:teuthology.orchestra.run.smithi089.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-22T01:23:46.705 INFO:teuthology.orchestra.run.smithi089.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-22T01:23:46.705 INFO:teuthology.orchestra.run.smithi089.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-22T01:23:46.705 INFO:teuthology.orchestra.run.smithi089.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-22T01:23:46.705 INFO:teuthology.orchestra.run.smithi089.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-22T01:23:46.705 INFO:teuthology.orchestra.run.smithi089.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-22T01:23:46.705 INFO:teuthology.orchestra.run.smithi089.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-22T01:23:46.705 INFO:teuthology.orchestra.run.smithi089.stdout: libcephsqlite x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 163 k 2024-03-22T01:23:46.705 INFO:teuthology.orchestra.run.smithi089.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-22T01:23:46.705 INFO:teuthology.orchestra.run.smithi089.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-22T01:23:46.705 INFO:teuthology.orchestra.run.smithi089.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-22T01:23:46.705 INFO:teuthology.orchestra.run.smithi089.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-22T01:23:46.706 INFO:teuthology.orchestra.run.smithi089.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-22T01:23:46.706 INFO:teuthology.orchestra.run.smithi089.stdout: libradosstriper1 x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 501 k 2024-03-22T01:23:46.706 INFO:teuthology.orchestra.run.smithi089.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-22T01:23:46.706 INFO:teuthology.orchestra.run.smithi089.stdout: librgw2 x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 5.1 M 2024-03-22T01:23:46.706 INFO:teuthology.orchestra.run.smithi089.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-22T01:23:46.706 INFO:teuthology.orchestra.run.smithi089.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-22T01:23:46.706 INFO:teuthology.orchestra.run.smithi089.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-22T01:23:46.706 INFO:teuthology.orchestra.run.smithi089.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-22T01:23:46.706 INFO:teuthology.orchestra.run.smithi089.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-22T01:23:46.706 INFO:teuthology.orchestra.run.smithi089.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-22T01:23:46.706 INFO:teuthology.orchestra.run.smithi089.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-22T01:23:46.706 INFO:teuthology.orchestra.run.smithi089.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-22T01:23:46.707 INFO:teuthology.orchestra.run.smithi089.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-22T01:23:46.707 INFO:teuthology.orchestra.run.smithi089.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-22T01:23:46.707 INFO:teuthology.orchestra.run.smithi089.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-22T01:23:46.707 INFO:teuthology.orchestra.run.smithi089.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-22T01:23:46.707 INFO:teuthology.orchestra.run.smithi089.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-22T01:23:46.707 INFO:teuthology.orchestra.run.smithi089.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-22T01:23:46.707 INFO:teuthology.orchestra.run.smithi089.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-22T01:23:46.707 INFO:teuthology.orchestra.run.smithi089.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-22T01:23:46.707 INFO:teuthology.orchestra.run.smithi089.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-22T01:23:46.707 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-22T01:23:46.707 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ceph-argparse x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 45 k 2024-03-22T01:23:46.707 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ceph-common x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 131 k 2024-03-22T01:23:46.707 INFO:teuthology.orchestra.run.smithi089.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-22T01:23:46.707 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-22T01:23:46.707 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-22T01:23:46.708 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-22T01:23:46.708 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-22T01:23:46.708 INFO:teuthology.orchestra.run.smithi089.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-22T01:23:46.708 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-22T01:23:46.708 INFO:teuthology.orchestra.run.smithi089.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-22T01:23:46.708 INFO:teuthology.orchestra.run.smithi089.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-22T01:23:46.708 INFO:teuthology.orchestra.run.smithi089.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-22T01:23:46.708 INFO:teuthology.orchestra.run.smithi089.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-22T01:23:46.708 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-22T01:23:46.708 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-22T01:23:46.708 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-22T01:23:46.708 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-22T01:23:46.708 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-22T01:23:46.708 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-22T01:23:46.709 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-22T01:23:46.709 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-22T01:23:46.709 INFO:teuthology.orchestra.run.smithi089.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-22T01:23:46.709 INFO:teuthology.orchestra.run.smithi089.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-22T01:23:46.709 INFO:teuthology.orchestra.run.smithi089.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-22T01:23:46.709 INFO:teuthology.orchestra.run.smithi089.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-22T01:23:46.709 INFO:teuthology.orchestra.run.smithi089.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-22T01:23:46.709 INFO:teuthology.orchestra.run.smithi089.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-22T01:23:46.709 INFO:teuthology.orchestra.run.smithi089.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-22T01:23:46.709 INFO:teuthology.orchestra.run.smithi089.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-22T01:23:46.709 INFO:teuthology.orchestra.run.smithi089.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-22T01:23:46.709 INFO:teuthology.orchestra.run.smithi089.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-22T01:23:46.709 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-22T01:23:46.709 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-22T01:23:46.709 INFO:teuthology.orchestra.run.smithi089.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-22T01:23:46.710 INFO:teuthology.orchestra.run.smithi089.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-22T01:23:46.710 INFO:teuthology.orchestra.run.smithi089.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-22T01:23:46.710 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-22T01:23:46.710 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-22T01:23:46.710 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-22T01:23:46.710 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-22T01:23:46.710 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-22T01:23:46.710 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-22T01:23:46.710 INFO:teuthology.orchestra.run.smithi089.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-22T01:23:46.710 INFO:teuthology.orchestra.run.smithi089.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-22T01:23:46.710 INFO:teuthology.orchestra.run.smithi089.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-22T01:23:46.710 INFO:teuthology.orchestra.run.smithi089.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-22T01:23:46.710 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-22T01:23:46.710 INFO:teuthology.orchestra.run.smithi089.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-22T01:23:46.710 INFO:teuthology.orchestra.run.smithi089.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-22T01:23:46.711 INFO:teuthology.orchestra.run.smithi089.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-22T01:23:46.711 INFO:teuthology.orchestra.run.smithi089.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-22T01:23:46.711 INFO:teuthology.orchestra.run.smithi089.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-22T01:23:46.711 INFO:teuthology.orchestra.run.smithi089.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-22T01:23:46.711 INFO:teuthology.orchestra.run.smithi089.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-22T01:23:46.711 INFO:teuthology.orchestra.run.smithi089.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-22T01:23:46.711 INFO:teuthology.orchestra.run.smithi089.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-22T01:23:46.711 INFO:teuthology.orchestra.run.smithi089.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-22T01:23:46.711 INFO:teuthology.orchestra.run.smithi089.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-22T01:23:46.711 INFO:teuthology.orchestra.run.smithi089.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-22T01:23:46.711 INFO:teuthology.orchestra.run.smithi089.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-22T01:23:46.711 INFO:teuthology.orchestra.run.smithi089.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-22T01:23:46.711 INFO:teuthology.orchestra.run.smithi089.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-22T01:23:46.711 INFO:teuthology.orchestra.run.smithi089.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-22T01:23:46.712 INFO:teuthology.orchestra.run.smithi089.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-22T01:23:46.712 INFO:teuthology.orchestra.run.smithi089.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-22T01:23:46.712 INFO:teuthology.orchestra.run.smithi089.stdout:Installing weak dependencies: 2024-03-22T01:23:46.712 INFO:teuthology.orchestra.run.smithi089.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-22T01:23:46.712 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-22T01:23:46.712 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-03-22T01:23:46.712 INFO:teuthology.orchestra.run.smithi089.stdout:============================================================================================ 2024-03-22T01:23:46.712 INFO:teuthology.orchestra.run.smithi089.stdout:Install 150 Packages 2024-03-22T01:23:46.712 INFO:teuthology.orchestra.run.smithi089.stdout:Upgrade 24 Packages 2024-03-22T01:23:46.712 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-22T01:23:46.713 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 226 M 2024-03-22T01:23:46.713 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-03-22T01:23:47.114 INFO:teuthology.orchestra.run.smithi072.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 390 kB/s | 465 kB 00:01 2024-03-22T01:23:47.448 INFO:teuthology.orchestra.run.smithi072.stdout:(170/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 594 kB/s | 2.2 MB 00:03 2024-03-22T01:23:47.515 INFO:teuthology.orchestra.run.smithi089.stdout:(1/174): ceph-19.0.0-2344.gc3a04c0e.el9.x86_64. 39 kB/s | 6.6 kB 00:00 2024-03-22T01:23:47.615 INFO:teuthology.orchestra.run.smithi072.stdout:(171/174): util-linux-2.37.4-18.el9.x86_64.rpm 620 kB/s | 2.3 MB 00:03 2024-03-22T01:23:47.699 INFO:teuthology.orchestra.run.smithi089.stdout:(2/174): ceph-fuse-19.0.0-2344.gc3a04c0e.el9.x8 4.7 MB/s | 890 kB 00:00 2024-03-22T01:23:47.725 INFO:teuthology.orchestra.run.smithi072.stdout:(172/174): util-linux-user-2.37.4-18.el9.x86_64 51 kB/s | 31 kB 00:00 2024-03-22T01:23:47.750 INFO:teuthology.orchestra.run.smithi072.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 551 kB/s | 74 kB 00:00 2024-03-22T01:23:47.757 INFO:teuthology.orchestra.run.smithi089.stdout:(3/174): ceph-immutable-object-cache-19.0.0-234 2.4 MB/s | 145 kB 00:00 2024-03-22T01:23:48.049 INFO:teuthology.orchestra.run.smithi089.stdout:(4/174): ceph-mds-19.0.0-2344.gc3a04c0e.el9.x86 8.1 MB/s | 2.4 MB 00:00 2024-03-22T01:23:48.116 INFO:teuthology.orchestra.run.smithi089.stdout:(5/174): ceph-base-19.0.0-2344.gc3a04c0e.el9.x8 7.0 MB/s | 5.4 MB 00:00 2024-03-22T01:23:48.250 INFO:teuthology.orchestra.run.smithi089.stdout:(6/174): ceph-mgr-19.0.0-2344.gc3a04c0e.el9.x86 7.5 MB/s | 1.5 MB 00:00 2024-03-22T01:23:48.542 INFO:teuthology.orchestra.run.smithi089.stdout:(7/174): ceph-mon-19.0.0-2344.gc3a04c0e.el9.x86 11 MB/s | 4.7 MB 00:00 2024-03-22T01:23:49.459 INFO:teuthology.orchestra.run.smithi072.stdout:(174/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 547 kB/s | 1.1 MB 00:02 2024-03-22T01:23:49.463 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-03-22T01:23:49.463 INFO:teuthology.orchestra.run.smithi072.stdout:Total 6.7 MB/s | 226 MB 00:33 2024-03-22T01:23:49.509 INFO:teuthology.orchestra.run.smithi089.stdout:(8/174): ceph-radosgw-19.0.0-2344.gc3a04c0e.el9 10 MB/s | 9.8 MB 00:00 2024-03-22T01:23:49.567 INFO:teuthology.orchestra.run.smithi089.stdout:(9/174): ceph-selinux-19.0.0-2344.gc3a04c0e.el9 431 kB/s | 25 kB 00:00 2024-03-22T01:23:49.704 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-22T01:23:49.796 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-22T01:23:49.831 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-22T01:23:49.870 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-22T01:23:49.879 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-22T01:23:49.919 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-22T01:23:49.928 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2024-03-22T01:23:49.942 INFO:teuthology.orchestra.run.smithi089.stdout:(10/174): ceph-osd-19.0.0-2344.gc3a04c0e.el9.x8 10 MB/s | 17 MB 00:01 2024-03-22T01:23:49.973 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-22T01:23:50.006 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-22T01:23:50.026 INFO:teuthology.orchestra.run.smithi089.stdout:(11/174): cephfs-mirror-19.0.0-2344.gc3a04c0e.e 2.6 MB/s | 222 kB 00:00 2024-03-22T01:23:50.032 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-22T01:23:50.074 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-22T01:23:50.085 INFO:teuthology.orchestra.run.smithi089.stdout:(12/174): cephfs-shell-19.0.0-2344.gc3a04c0e.el 464 kB/s | 27 kB 00:00 2024-03-22T01:23:50.143 INFO:teuthology.orchestra.run.smithi089.stdout:(13/174): libcephfs-devel-19.0.0-2344.gc3a04c0e 545 kB/s | 32 kB 00:00 2024-03-22T01:23:50.203 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2024-03-22T01:23:50.203 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2024-03-22T01:23:50.285 INFO:teuthology.orchestra.run.smithi089.stdout:(14/174): libcephfs2-19.0.0-2344.gc3a04c0e.el9. 5.2 MB/s | 747 kB 00:00 2024-03-22T01:23:50.360 INFO:teuthology.orchestra.run.smithi089.stdout:(15/174): libcephsqlite-19.0.0-2344.gc3a04c0e.e 2.1 MB/s | 163 kB 00:00 2024-03-22T01:23:50.394 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-22T01:23:50.458 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : librados2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 10/198 2024-03-22T01:23:50.569 INFO:teuthology.orchestra.run.smithi089.stdout:(16/174): ceph-common-19.0.0-2344.gc3a04c0e.el9 6.4 MB/s | 21 MB 00:03 2024-03-22T01:23:50.571 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 10/198 2024-03-22T01:23:50.594 INFO:teuthology.orchestra.run.smithi089.stdout:(17/174): librados-devel-19.0.0-2344.gc3a04c0e. 542 kB/s | 127 kB 00:00 2024-03-22T01:23:50.601 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephfs2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 11/198 2024-03-22T01:23:50.692 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libcephfs2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 11/198 2024-03-22T01:23:50.703 INFO:teuthology.orchestra.run.smithi089.stdout:(18/174): libradosstriper1-19.0.0-2344.gc3a04c0 3.7 MB/s | 501 kB 00:00 2024-03-22T01:23:50.722 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rados-2:19.0.0-2344.gc3a04c0e.el9.x86_64 12/198 2024-03-22T01:23:50.767 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-22T01:23:50.803 INFO:teuthology.orchestra.run.smithi089.stdout:(19/174): python3-ceph-argparse-19.0.0-2344.gc3 452 kB/s | 45 kB 00:00 2024-03-22T01:23:50.912 INFO:teuthology.orchestra.run.smithi089.stdout:(20/174): python3-ceph-common-19.0.0-2344.gc3a0 1.2 MB/s | 131 kB 00:00 2024-03-22T01:23:50.945 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-22T01:23:50.995 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-22T01:23:51.012 INFO:teuthology.orchestra.run.smithi089.stdout:(21/174): python3-cephfs-19.0.0-2344.gc3a04c0e. 1.6 MB/s | 161 kB 00:00 2024-03-22T01:23:51.023 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephsqlite-2:19.0.0-2344.gc3a04c0e.el9.x86_64 16/198 2024-03-22T01:23:51.088 INFO:teuthology.orchestra.run.smithi089.stdout:(22/174): librgw2-19.0.0-2344.gc3a04c0e.el9.x86 10 MB/s | 5.1 MB 00:00 2024-03-22T01:23:51.109 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2344.gc3a04c0e.el9.x86_64 16/198 2024-03-22T01:23:51.112 INFO:teuthology.orchestra.run.smithi089.stdout:(23/174): python3-rados-19.0.0-2344.gc3a04c0e.e 3.1 MB/s | 321 kB 00:00 2024-03-22T01:23:51.132 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libradosstriper1-2:19.0.0-2344.gc3a04c0e.el9.x86 17/198 2024-03-22T01:23:51.163 INFO:teuthology.orchestra.run.smithi089.stdout:(24/174): python3-rbd-19.0.0-2344.gc3a04c0e.el9 3.9 MB/s | 298 kB 00:00 2024-03-22T01:23:51.196 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2344.gc3a04c0e.el9.x86 17/198 2024-03-22T01:23:51.197 INFO:teuthology.orchestra.run.smithi089.stdout:(25/174): python3-rgw-19.0.0-2344.gc3a04c0e.el9 1.2 MB/s | 99 kB 00:00 2024-03-22T01:23:51.230 INFO:teuthology.orchestra.run.smithi089.stdout:(26/174): rbd-fuse-19.0.0-2344.gc3a04c0e.el9.x8 1.2 MB/s | 85 kB 00:00 2024-03-22T01:23:51.236 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-22T01:23:51.259 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-22T01:23:51.274 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-22T01:23:51.314 INFO:teuthology.orchestra.run.smithi089.stdout:(27/174): rbd-nbd-19.0.0-2344.gc3a04c0e.el9.x86 2.0 MB/s | 170 kB 00:00 2024-03-22T01:23:51.381 INFO:teuthology.orchestra.run.smithi089.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2344.g 508 kB/s | 34 kB 00:00 2024-03-22T01:23:51.459 INFO:teuthology.orchestra.run.smithi089.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2344.gc3a04c0 1.8 MB/s | 140 kB 00:00 2024-03-22T01:23:51.467 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-22T01:23:51.498 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-22T01:23:51.539 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-22T01:23:51.634 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-22T01:23:51.648 INFO:teuthology.orchestra.run.smithi089.stdout:(30/174): rbd-mirror-19.0.0-2344.gc3a04c0e.el9. 6.8 MB/s | 3.1 MB 00:00 2024-03-22T01:23:51.724 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-22T01:23:51.768 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-22T01:23:51.845 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-22T01:23:51.887 INFO:teuthology.orchestra.run.smithi089.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2344.gc3a04 8.2 MB/s | 3.5 MB 00:00 2024-03-22T01:23:51.894 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-22T01:23:51.971 INFO:teuthology.orchestra.run.smithi089.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2344.gc3 2.9 MB/s | 248 kB 00:00 2024-03-22T01:23:52.046 INFO:teuthology.orchestra.run.smithi089.stdout:(33/174): ceph-mgr-rook-19.0.0-2344.gc3a04c0e.e 659 kB/s | 49 kB 00:00 2024-03-22T01:23:52.113 INFO:teuthology.orchestra.run.smithi089.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2344.gc 246 kB/s | 16 kB 00:00 2024-03-22T01:23:52.170 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-22T01:23:52.214 INFO:teuthology.orchestra.run.smithi089.stdout:(35/174): ceph-volume-19.0.0-2344.gc3a04c0e.el9 2.6 MB/s | 264 kB 00:00 2024-03-22T01:23:52.330 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2024-03-22T01:23:52.331 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2024-03-22T01:23:52.372 INFO:teuthology.orchestra.run.smithi089.stdout:(36/174): cephadm-19.0.0-2344.gc3a04c0e.el9.noa 4.6 MB/s | 750 kB 00:00 2024-03-22T01:23:52.448 INFO:teuthology.orchestra.run.smithi089.stdout:(37/174): cephfs-top-19.0.0-2344.gc3a04c0e.el9. 243 kB/s | 18 kB 00:00 2024-03-22T01:23:52.474 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-22T01:23:52.564 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-22T01:23:52.599 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-22T01:23:52.626 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-22T01:23:52.662 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-22T01:23:52.736 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-22T01:23:52.790 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-22T01:23:52.829 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-22T01:23:52.900 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-22T01:23:52.907 INFO:teuthology.orchestra.run.smithi089.stdout:(38/174): ceph-test-19.0.0-2344.gc3a04c0e.el9.x 14 MB/s | 48 MB 00:03 2024-03-22T01:23:52.935 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-22T01:23:52.979 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-22T01:23:52.998 INFO:teuthology.orchestra.run.smithi089.stdout:(39/174): ceph-mgr-diskprediction-local-19.0.0- 5.5 MB/s | 7.4 MB 00:01 2024-03-22T01:23:53.012 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-22T01:23:53.040 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-22T01:23:53.082 INFO:teuthology.orchestra.run.smithi089.stdout:(40/174): automake-1.16.2-8.el9.noarch.rpm 1.0 MB/s | 662 kB 00:00 2024-03-22T01:23:53.109 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-22T01:23:53.116 INFO:teuthology.orchestra.run.smithi089.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 908 kB/s | 106 kB 00:00 2024-03-22T01:23:53.141 INFO:teuthology.orchestra.run.smithi089.stdout:(42/174): bison-3.7.4-5.el9.x86_64.rpm 3.9 MB/s | 920 kB 00:00 2024-03-22T01:23:53.145 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ceph-common-2:19.0.0-2344.gc3a04c0e.el9. 39/198 2024-03-22T01:23:53.174 INFO:teuthology.orchestra.run.smithi089.stdout:(43/174): flex-2.6.4-9.el9.x86_64.rpm 3.3 MB/s | 307 kB 00:00 2024-03-22T01:23:53.180 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ceph-argparse-2:19.0.0-2344.gc3a04c0e.el 40/198 2024-03-22T01:23:53.199 INFO:teuthology.orchestra.run.smithi089.stdout:(44/174): flexiblas-3.0.4-8.el9.x86_64.rpm 381 kB/s | 32 kB 00:00 2024-03-22T01:23:53.208 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cephfs-2:19.0.0-2344.gc3a04c0e.el9.x86_6 41/198 2024-03-22T01:23:53.225 INFO:teuthology.orchestra.run.smithi089.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 343 kB/s | 17 kB 00:00 2024-03-22T01:23:53.275 INFO:teuthology.orchestra.run.smithi089.stdout:(46/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 30 MB/s | 3.0 MB 00:00 2024-03-22T01:23:53.279 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-22T01:23:53.300 INFO:teuthology.orchestra.run.smithi089.stdout:(47/174): libaio-devel-0.3.111-13.el9.x86_64.rp 100 kB/s | 10 kB 00:00 2024-03-22T01:23:53.323 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-22T01:23:53.334 INFO:teuthology.orchestra.run.smithi089.stdout:(48/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.5 MB/s | 159 kB 00:00 2024-03-22T01:23:53.359 INFO:teuthology.orchestra.run.smithi089.stdout:(49/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 757 kB/s | 44 kB 00:00 2024-03-22T01:23:53.393 INFO:teuthology.orchestra.run.smithi089.stdout:(50/174): librdkafka-1.6.1-102.el9.x86_64.rpm 7.0 MB/s | 661 kB 00:00 2024-03-22T01:23:53.398 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-22T01:23:53.418 INFO:teuthology.orchestra.run.smithi089.stdout:(51/174): libtool-2.4.6-45.el9.x86_64.rpm 6.8 MB/s | 578 kB 00:00 2024-03-22T01:23:53.444 INFO:teuthology.orchestra.run.smithi089.stdout:(52/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 946 kB/s | 46 kB 00:00 2024-03-22T01:23:53.469 INFO:teuthology.orchestra.run.smithi089.stdout:(53/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-03-22T01:23:53.494 INFO:teuthology.orchestra.run.smithi089.stdout:(54/174): lua-5.4.4-4.el9.x86_64.rpm 2.4 MB/s | 187 kB 00:00 2024-03-22T01:23:53.511 INFO:teuthology.orchestra.run.smithi089.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 825 kB/s | 34 kB 00:00 2024-03-22T01:23:53.529 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-22T01:23:53.536 INFO:teuthology.orchestra.run.smithi089.stdout:(56/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.2 MB/s | 217 kB 00:00 2024-03-22T01:23:53.584 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-22T01:23:53.628 INFO:teuthology.orchestra.run.smithi089.stdout:(57/174): openblas-openmp-0.3.21-2.el9.x86_64.r 35 MB/s | 4.7 MB 00:00 2024-03-22T01:23:53.670 INFO:teuthology.orchestra.run.smithi089.stdout:(58/174): postgresql-13.11-1.el9.x86_64.rpm 9.7 MB/s | 1.5 MB 00:00 2024-03-22T01:23:53.704 INFO:teuthology.orchestra.run.smithi089.stdout:(59/174): postgresql-contrib-13.11-1.el9.x86_64 4.8 MB/s | 811 kB 00:00 2024-03-22T01:23:53.729 INFO:teuthology.orchestra.run.smithi089.stdout:(60/174): postgresql-private-libs-13.11-1.el9.x 1.3 MB/s | 134 kB 00:00 2024-03-22T01:23:53.813 INFO:teuthology.orchestra.run.smithi089.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 9.3 MB/s | 1.0 MB 00:00 2024-03-22T01:23:53.946 INFO:teuthology.orchestra.run.smithi089.stdout:(62/174): postgresql-server-13.11-1.el9.x86_64. 23 MB/s | 5.7 MB 00:00 2024-03-22T01:23:53.971 INFO:teuthology.orchestra.run.smithi089.stdout:(63/174): python3-attrs-20.3.0-7.el9.noarch.rpm 323 kB/s | 78 kB 00:00 2024-03-22T01:23:54.072 INFO:teuthology.orchestra.run.smithi089.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 23 MB/s | 5.8 MB 00:00 2024-03-22T01:23:54.097 INFO:teuthology.orchestra.run.smithi089.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 199 kB/s | 24 kB 00:00 2024-03-22T01:23:54.100 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-22T01:23:54.122 INFO:teuthology.orchestra.run.smithi089.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 122 kB/s | 18 kB 00:00 2024-03-22T01:23:54.156 INFO:teuthology.orchestra.run.smithi089.stdout:(67/174): python3-mako-1.1.4-6.el9.noarch.rpm 1.7 MB/s | 144 kB 00:00 2024-03-22T01:23:54.181 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-22T01:23:54.181 INFO:teuthology.orchestra.run.smithi089.stdout:(68/174): python3-markupsafe-1.1.1-12.el9.x86_6 542 kB/s | 32 kB 00:00 2024-03-22T01:23:54.270 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-22T01:23:54.273 INFO:teuthology.orchestra.run.smithi089.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 32 MB/s | 4.8 MB 00:00 2024-03-22T01:23:54.298 INFO:teuthology.orchestra.run.smithi089.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.5 MB/s | 217 kB 00:00 2024-03-22T01:23:54.316 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-22T01:23:54.324 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-22T01:23:54.332 INFO:teuthology.orchestra.run.smithi089.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.1 MB/s | 174 kB 00:00 2024-03-22T01:23:54.356 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-22T01:23:54.357 INFO:teuthology.orchestra.run.smithi089.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 825 kB/s | 69 kB 00:00 2024-03-22T01:23:54.383 INFO:teuthology.orchestra.run.smithi089.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 826 kB/s | 41 kB 00:00 2024-03-22T01:23:54.405 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-22T01:23:54.408 INFO:teuthology.orchestra.run.smithi089.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 3.1 MB/s | 238 kB 00:00 2024-03-22T01:23:54.433 INFO:teuthology.orchestra.run.smithi089.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-22T01:23:54.452 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librados-devel-2:19.0.0-2344.gc3a04c0e.el9.x86_6 53/198 2024-03-22T01:23:54.458 INFO:teuthology.orchestra.run.smithi089.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 951 kB/s | 47 kB 00:00 2024-03-22T01:23:54.481 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-22T01:23:54.492 INFO:teuthology.orchestra.run.smithi089.stdout:(77/174): python3-pyasn1-modules-0.4.8-6.el9.no 2.5 MB/s | 211 kB 00:00 2024-03-22T01:23:54.517 INFO:teuthology.orchestra.run.smithi089.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 512 kB/s | 43 kB 00:00 2024-03-22T01:23:54.706 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-22T01:23:54.809 INFO:teuthology.orchestra.run.smithi089.stdout:(79/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 45 MB/s | 16 MB 00:00 2024-03-22T01:23:54.834 INFO:teuthology.orchestra.run.smithi089.stdout:(80/174): python3-toml-0.10.2-6.el9.noarch.rpm 123 kB/s | 39 kB 00:00 2024-03-22T01:23:54.859 INFO:teuthology.orchestra.run.smithi089.stdout:(81/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 120 kB/s | 41 kB 00:00 2024-03-22T01:23:54.864 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-22T01:23:54.885 INFO:teuthology.orchestra.run.smithi089.stdout:(82/174): socat-1.7.4.1-5.el9.x86_64.rpm 3.9 MB/s | 300 kB 00:00 2024-03-22T01:23:54.910 INFO:teuthology.orchestra.run.smithi089.stdout:(83/174): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-03-22T01:23:54.934 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-22T01:23:54.960 INFO:teuthology.orchestra.run.smithi089.stdout:(84/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 626 kB/s | 63 kB 00:00 2024-03-22T01:23:54.978 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-22T01:23:54.985 INFO:teuthology.orchestra.run.smithi089.stdout:(85/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 439 kB/s | 44 kB 00:00 2024-03-22T01:23:55.022 INFO:teuthology.orchestra.run.smithi089.stdout:(86/174): fuse-2.9.9-15.el9.x86_64.rpm 1.3 MB/s | 79 kB 00:00 2024-03-22T01:23:55.046 INFO:teuthology.orchestra.run.smithi089.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.8 MB/s | 241 kB 00:00 2024-03-22T01:23:55.088 INFO:teuthology.orchestra.run.smithi089.stdout:(88/174): python3-cryptography-36.0.1-4.el9.x86 11 MB/s | 1.1 MB 00:00 2024-03-22T01:23:55.113 INFO:teuthology.orchestra.run.smithi089.stdout:(89/174): python3-ply-3.11-14.el9.noarch.rpm 1.5 MB/s | 103 kB 00:00 2024-03-22T01:23:55.138 INFO:teuthology.orchestra.run.smithi089.stdout:(90/174): python3-pycparser-2.20-6.el9.noarch.r 1.3 MB/s | 124 kB 00:00 2024-03-22T01:23:55.164 INFO:teuthology.orchestra.run.smithi089.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.4 MB/s | 187 kB 00:00 2024-03-22T01:23:55.381 INFO:teuthology.orchestra.run.smithi089.stdout:(92/174): jq-1.6-16.el9.x86_64.rpm 774 kB/s | 187 kB 00:00 2024-03-22T01:23:55.383 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-22T01:23:55.439 INFO:teuthology.orchestra.run.smithi089.stdout:(93/174): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-03-22T01:23:55.449 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-22T01:23:55.464 INFO:teuthology.orchestra.run.smithi089.stdout:(94/174): libquadmath-11.4.1-3.el9.x86_64.rpm 650 kB/s | 195 kB 00:00 2024-03-22T01:23:55.505 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-22T01:23:55.506 INFO:teuthology.orchestra.run.smithi089.stdout:(95/174): libgfortran-11.4.1-3.el9.x86_64.rpm 2.1 MB/s | 803 kB 00:00 2024-03-22T01:23:55.573 INFO:teuthology.orchestra.run.smithi089.stdout:(96/174): elfutils-libelf-devel-0.190-2.el9.x86 160 kB/s | 21 kB 00:00 2024-03-22T01:23:55.578 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-22T01:23:55.598 INFO:teuthology.orchestra.run.smithi089.stdout:(97/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 190 kB/s | 25 kB 00:00 2024-03-22T01:23:55.616 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-22T01:23:55.653 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-22T01:23:55.699 INFO:teuthology.orchestra.run.smithi089.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 2.4 MB/s | 245 kB 00:00 2024-03-22T01:23:55.699 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-22T01:23:55.724 INFO:teuthology.orchestra.run.smithi089.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 1.9 MB/s | 248 kB 00:00 2024-03-22T01:23:55.739 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-22T01:23:55.758 INFO:teuthology.orchestra.run.smithi089.stdout:(100/174): qatlib-23.11.0-1.el9.x86_64.rpm 3.6 MB/s | 216 kB 00:00 2024-03-22T01:23:55.781 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-22T01:23:55.783 INFO:teuthology.orchestra.run.smithi089.stdout:(101/174): qatlib-service-23.11.0-1.el9.x86_64. 594 kB/s | 35 kB 00:00 2024-03-22T01:23:55.808 INFO:teuthology.orchestra.run.smithi089.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 887 kB/s | 44 kB 00:00 2024-03-22T01:23:55.824 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-22T01:23:55.834 INFO:teuthology.orchestra.run.smithi089.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.8 MB/s | 94 kB 00:00 2024-03-22T01:23:55.909 INFO:teuthology.orchestra.run.smithi089.stdout:(104/174): openssl-devel-3.0.7-27.el9.x86_64.rp 10 MB/s | 4.1 MB 00:00 2024-03-22T01:23:55.934 INFO:teuthology.orchestra.run.smithi089.stdout:(105/174): lua-devel-5.4.4-4.el9.x86_64.rpm 178 kB/s | 22 kB 00:00 2024-03-22T01:23:55.973 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-22T01:23:56.010 INFO:teuthology.orchestra.run.smithi089.stdout:(106/174): protobuf-compiler-3.14.0-13.el9.x86_ 4.8 MB/s | 863 kB 00:00 2024-03-22T01:23:56.021 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-22T01:23:56.052 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-22T01:23:56.093 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-22T01:23:56.188 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-22T01:23:56.227 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-22T01:23:56.260 INFO:teuthology.orchestra.run.smithi089.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 78 kB/s | 19 kB 00:00 2024-03-22T01:23:56.267 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-22T01:23:56.293 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-22T01:23:56.317 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-22T01:23:56.389 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-22T01:23:56.460 INFO:teuthology.orchestra.run.smithi089.stdout:(108/174): gperftools-libs-2.9.1-2.el9.x86_64.r 588 kB/s | 309 kB 00:00 2024-03-22T01:23:56.510 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-22T01:23:56.527 INFO:teuthology.orchestra.run.smithi089.stdout:(109/174): abseil-cpp-20211102.0-3.el9.x86_64.r 887 kB/s | 548 kB 00:00 2024-03-22T01:23:56.553 INFO:teuthology.orchestra.run.smithi089.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 281 kB/s | 26 kB 00:00 2024-03-22T01:23:56.609 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-22T01:23:56.628 INFO:teuthology.orchestra.run.smithi089.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 485 kB/s | 49 kB 00:00 2024-03-22T01:23:56.653 INFO:teuthology.orchestra.run.smithi089.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 672 kB/s | 67 kB 00:00 2024-03-22T01:23:56.667 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-22T01:23:56.716 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-22T01:23:56.745 INFO:teuthology.orchestra.run.smithi089.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 1.3 MB/s | 151 kB 00:00 2024-03-22T01:23:56.771 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-22T01:23:56.903 INFO:teuthology.orchestra.run.smithi089.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.3 MB/s | 838 kB 00:00 2024-03-22T01:23:56.937 INFO:teuthology.orchestra.run.smithi089.stdout:(115/174): python3-asyncssh-2.13.2-5.el9.noarch 2.8 MB/s | 548 kB 00:00 2024-03-22T01:23:56.953 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-22T01:23:56.997 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-22T01:23:57.004 INFO:teuthology.orchestra.run.smithi089.stdout:(116/174): libarrow-9.0.0-10.el9.x86_64.rpm 5.9 MB/s | 4.4 MB 00:00 2024-03-22T01:23:57.029 INFO:teuthology.orchestra.run.smithi089.stdout:(117/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 346 kB/s | 43 kB 00:00 2024-03-22T01:23:57.058 INFO:teuthology.orchestra.run.smithi089.stdout:(118/174): python3-cachetools-4.2.4-1.el9.noarc 274 kB/s | 32 kB 00:00 2024-03-22T01:23:57.097 INFO:teuthology.orchestra.run.smithi089.stdout:(119/174): python3-certifi-2021.10.8-2.el9.noar 163 kB/s | 15 kB 00:00 2024-03-22T01:23:57.133 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-22T01:23:57.139 INFO:teuthology.orchestra.run.smithi089.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.5 MB/s | 172 kB 00:00 2024-03-22T01:23:57.172 INFO:teuthology.orchestra.run.smithi089.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 3.0 MB/s | 358 kB 00:00 2024-03-22T01:23:57.178 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-22T01:23:57.210 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-22T01:23:57.231 INFO:teuthology.orchestra.run.smithi089.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.2 MB/s | 303 kB 00:00 2024-03-22T01:23:57.241 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-22T01:23:57.256 INFO:teuthology.orchestra.run.smithi089.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 301 kB/s | 35 kB 00:00 2024-03-22T01:23:57.270 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-22T01:23:57.281 INFO:teuthology.orchestra.run.smithi089.stdout:(124/174): python3-google-auth-2.28.2-1.el9.noa 1.8 MB/s | 203 kB 00:00 2024-03-22T01:23:57.304 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-22T01:23:57.343 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-22T01:23:57.374 INFO:teuthology.orchestra.run.smithi089.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 1.2 MB/s | 144 kB 00:00 2024-03-22T01:23:57.374 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-22T01:23:57.399 INFO:teuthology.orchestra.run.smithi089.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 91 kB/s | 11 kB 00:00 2024-03-22T01:23:57.404 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-22T01:23:57.457 INFO:teuthology.orchestra.run.smithi089.stdout:(127/174): python3-grpcio-1.46.7-10.el9.x86_64. 9.0 MB/s | 2.0 MB 00:00 2024-03-22T01:23:57.480 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-22T01:23:57.499 INFO:teuthology.orchestra.run.smithi089.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 231 kB/s | 23 kB 00:00 2024-03-22T01:23:57.515 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-22T01:23:57.525 INFO:teuthology.orchestra.run.smithi089.stdout:(129/174): python3-jaraco-classes-3.2.1-5.el9.n 117 kB/s | 18 kB 00:00 2024-03-22T01:23:57.548 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-22T01:23:57.550 INFO:teuthology.orchestra.run.smithi089.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 210 kB/s | 19 kB 00:00 2024-03-22T01:23:57.589 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-22T01:23:57.592 INFO:teuthology.orchestra.run.smithi089.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 213 kB/s | 20 kB 00:00 2024-03-22T01:23:57.624 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-22T01:23:57.642 INFO:teuthology.orchestra.run.smithi089.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 503 kB/s | 46 kB 00:00 2024-03-22T01:23:57.692 INFO:teuthology.orchestra.run.smithi089.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 787 kB/s | 79 kB 00:00 2024-03-22T01:23:57.713 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-22T01:23:57.726 INFO:teuthology.orchestra.run.smithi089.stdout:(134/174): python3-kubernetes-26.1.0-2.el9.noar 5.1 MB/s | 1.0 MB 00:00 2024-03-22T01:23:57.751 INFO:teuthology.orchestra.run.smithi089.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 530 kB/s | 58 kB 00:00 2024-03-22T01:23:57.794 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-22T01:23:57.801 INFO:teuthology.orchestra.run.smithi089.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-03-22T01:23:57.827 INFO:teuthology.orchestra.run.smithi089.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 163 kB/s | 16 kB 00:00 2024-03-22T01:23:57.839 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-22T01:23:57.852 INFO:teuthology.orchestra.run.smithi089.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 891 kB/s | 90 kB 00:00 2024-03-22T01:23:57.879 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-22T01:23:57.926 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-22T01:23:57.952 INFO:teuthology.orchestra.run.smithi089.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 187 kB/s | 28 kB 00:00 2024-03-22T01:23:57.978 INFO:teuthology.orchestra.run.smithi089.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 204 kB/s | 31 kB 00:00 2024-03-22T01:23:58.015 INFO:teuthology.orchestra.run.smithi089.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 1.1 MB/s | 188 kB 00:00 2024-03-22T01:23:58.022 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-22T01:23:58.053 INFO:teuthology.orchestra.run.smithi089.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 588 kB/s | 59 kB 00:00 2024-03-22T01:23:58.072 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-22T01:23:58.096 INFO:teuthology.orchestra.run.smithi089.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 305 kB/s | 36 kB 00:00 2024-03-22T01:23:58.121 INFO:teuthology.orchestra.run.smithi089.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 481 kB/s | 51 kB 00:00 2024-03-22T01:23:58.171 INFO:teuthology.orchestra.run.smithi089.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2024-03-22T01:23:58.205 INFO:teuthology.orchestra.run.smithi089.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 825 kB/s | 90 kB 00:00 2024-03-22T01:23:58.238 INFO:teuthology.orchestra.run.smithi089.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 3.6 MB/s | 427 kB 00:00 2024-03-22T01:23:58.263 INFO:teuthology.orchestra.run.smithi089.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 217 kB/s | 20 kB 00:00 2024-03-22T01:23:58.305 INFO:teuthology.orchestra.run.smithi089.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-03-22T01:23:58.381 INFO:teuthology.orchestra.run.smithi089.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-03-22T01:23:58.539 INFO:teuthology.orchestra.run.smithi089.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 2.4 MB/s | 321 kB 00:00 2024-03-22T01:23:58.706 INFO:teuthology.orchestra.run.smithi089.stdout:(152/174): librados2-19.0.0-2344.gc3a04c0e.el9. 7.7 MB/s | 3.4 MB 00:00 2024-03-22T01:23:58.799 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-22T01:23:58.844 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-22T01:23:58.856 INFO:teuthology.orchestra.run.smithi089.stdout:(153/174): NetworkManager-1.47.2-1.el9.x86_64.r 7.3 MB/s | 2.3 MB 00:00 2024-03-22T01:23:58.878 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-22T01:23:58.924 INFO:teuthology.orchestra.run.smithi089.stdout:(154/174): NetworkManager-libnm-1.47.2-1.el9.x8 8.5 MB/s | 1.8 MB 00:00 2024-03-22T01:23:58.926 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-22T01:23:58.949 INFO:teuthology.orchestra.run.smithi089.stdout:(155/174): NetworkManager-team-1.47.2-1.el9.x86 457 kB/s | 42 kB 00:00 2024-03-22T01:23:59.006 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-22T01:23:59.008 INFO:teuthology.orchestra.run.smithi089.stdout:(156/174): NetworkManager-tui-1.47.2-1.el9.x86_ 2.9 MB/s | 248 kB 00:00 2024-03-22T01:23:59.041 INFO:teuthology.orchestra.run.smithi089.stdout:(157/174): acl-2.3.1-4.el9.x86_64.rpm 776 kB/s | 71 kB 00:00 2024-03-22T01:23:59.045 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-22T01:23:59.083 INFO:teuthology.orchestra.run.smithi089.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 485 kB/s | 36 kB 00:00 2024-03-22T01:23:59.117 INFO:teuthology.orchestra.run.smithi089.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.5 MB/s | 192 kB 00:00 2024-03-22T01:23:59.167 INFO:teuthology.orchestra.run.smithi089.stdout:(160/174): elfutils-libs-0.190-2.el9.x86_64.rpm 3.0 MB/s | 256 kB 00:00 2024-03-22T01:23:59.192 INFO:teuthology.orchestra.run.smithi089.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 305 kB/s | 23 kB 00:00 2024-03-22T01:23:59.223 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-22T01:23:59.234 INFO:teuthology.orchestra.run.smithi089.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.6 MB/s | 108 kB 00:00 2024-03-22T01:23:59.252 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-22T01:23:59.259 INFO:teuthology.orchestra.run.smithi089.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 2.3 MB/s | 155 kB 00:00 2024-03-22T01:23:59.301 INFO:teuthology.orchestra.run.smithi089.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 2.0 MB/s | 136 kB 00:00 2024-03-22T01:23:59.327 INFO:teuthology.orchestra.run.smithi089.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 952 kB/s | 64 kB 00:00 2024-03-22T01:23:59.359 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-22T01:23:59.404 INFO:teuthology.orchestra.run.smithi089.stdout:(166/174): librbd1-19.0.0-2344.gc3a04c0e.el9.x8 2.9 MB/s | 3.1 MB 00:01 2024-03-22T01:23:59.430 INFO:teuthology.orchestra.run.smithi089.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 224 kB/s | 29 kB 00:00 2024-03-22T01:23:59.480 INFO:teuthology.orchestra.run.smithi089.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 7.9 MB/s | 1.2 MB 00:00 2024-03-22T01:23:59.518 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2024-03-22T01:23:59.597 INFO:teuthology.orchestra.run.smithi089.stdout:(169/174): util-linux-2.37.4-18.el9.x86_64.rpm 14 MB/s | 2.3 MB 00:00 2024-03-22T01:23:59.625 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-22T01:23:59.639 INFO:teuthology.orchestra.run.smithi089.stdout:(170/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 9.2 MB/s | 2.2 MB 00:00 2024-03-22T01:23:59.673 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-22T01:23:59.681 INFO:teuthology.orchestra.run.smithi089.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 377 kB/s | 31 kB 00:00 2024-03-22T01:23:59.711 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-22T01:23:59.720 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-22T01:23:59.723 INFO:teuthology.orchestra.run.smithi089.stdout:(172/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 13 MB/s | 1.1 MB 00:00 2024-03-22T01:23:59.762 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-22T01:23:59.773 INFO:teuthology.orchestra.run.smithi089.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 808 kB/s | 74 kB 00:00 2024-03-22T01:23:59.807 INFO:teuthology.orchestra.run.smithi089.stdout:(174/174): util-linux-core-2.37.4-18.el9.x86_64 1.4 MB/s | 465 kB 00:00 2024-03-22T01:23:59.812 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-03-22T01:23:59.813 INFO:teuthology.orchestra.run.smithi089.stdout:Total 17 MB/s | 226 MB 00:13 2024-03-22T01:23:59.826 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-22T01:23:59.869 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-22T01:23:59.903 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-22T01:23:59.958 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-22T01:23:59.995 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-22T01:24:00.107 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-22T01:24:00.266 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-03-22T01:24:00.281 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-22T01:24:00.415 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : librados2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 10/198 2024-03-22T01:24:00.538 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-03-22T01:24:00.538 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-03-22T01:24:00.539 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: librados2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 10/198 2024-03-22T01:24:00.568 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libcephfs2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 11/198 2024-03-22T01:24:00.660 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: libcephfs2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 11/198 2024-03-22T01:24:00.707 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-rados-2:19.0.0-2344.gc3a04c0e.el9.x86_64 12/198 2024-03-22T01:24:00.764 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-22T01:24:00.948 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-22T01:24:01.009 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-22T01:24:01.038 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libcephsqlite-2:19.0.0-2344.gc3a04c0e.el9.x86_64 16/198 2024-03-22T01:24:01.142 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2344.gc3a04c0e.el9.x86_64 16/198 2024-03-22T01:24:01.173 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libradosstriper1-2:19.0.0-2344.gc3a04c0e.el9.x86 17/198 2024-03-22T01:24:01.239 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2344.gc3a04c0e.el9.x86 17/198 2024-03-22T01:24:01.283 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-22T01:24:01.310 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-22T01:24:01.320 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-22T01:24:01.325 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-22T01:24:01.395 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-22T01:24:01.426 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-22T01:24:01.457 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-22T01:24:01.485 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-22T01:24:01.510 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-22T01:24:01.542 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-22T01:24:01.584 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-22T01:24:01.799 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-22T01:24:01.832 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : librbd1-2:19.0.0-2344.gc3a04c0e.el9.x86_64 124/198 2024-03-22T01:24:01.922 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librbd1-2:19.0.0-2344.gc3a04c0e.el9.x86_64 124/198 2024-03-22T01:24:01.953 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rbd-2:19.0.0-2344.gc3a04c0e.el9.x86_64 125/198 2024-03-22T01:24:01.979 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-22T01:24:01.990 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-22T01:24:02.055 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-22T01:24:02.125 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-22T01:24:02.254 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-03-22T01:24:02.254 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-03-22T01:24:02.299 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-22T01:24:02.353 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-22T01:24:02.456 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-22T01:24:02.484 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librgw2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 128/198 2024-03-22T01:24:02.556 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librgw2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 128/198 2024-03-22T01:24:02.572 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-22T01:24:02.578 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rgw-2:19.0.0-2344.gc3a04c0e.el9.x86_64 129/198 2024-03-22T01:24:02.676 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-22T01:24:02.763 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-22T01:24:02.803 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-22T01:24:02.830 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-22T01:24:02.874 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-22T01:24:02.982 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-22T01:24:03.037 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-22T01:24:03.077 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-22T01:24:03.141 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-22T01:24:03.186 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-22T01:24:03.230 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-22T01:24:03.264 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-22T01:24:03.300 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-22T01:24:03.378 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-22T01:24:03.416 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-ceph-common-2:19.0.0-2344.gc3a04c0e.el9. 39/198 2024-03-22T01:24:03.460 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-ceph-argparse-2:19.0.0-2344.gc3a04c0e.el 40/198 2024-03-22T01:24:03.489 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cephfs-2:19.0.0-2344.gc3a04c0e.el9.x86_6 41/198 2024-03-22T01:24:03.558 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-22T01:24:03.613 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-22T01:24:03.687 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-22T01:24:03.830 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-22T01:24:03.885 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-22T01:24:04.413 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-22T01:24:04.500 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-22T01:24:04.587 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-22T01:24:04.642 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-22T01:24:04.649 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-22T01:24:04.692 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-22T01:24:04.740 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-22T01:24:04.780 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : librados-devel-2:19.0.0-2344.gc3a04c0e.el9.x86_6 53/198 2024-03-22T01:24:04.820 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-22T01:24:04.837 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-common-2:19.0.0-2344.gc3a04c0e.el9.x86_64 130/198 2024-03-22T01:24:04.867 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-common-2:19.0.0-2344.gc3a04c0e.el9.x86_64 130/198 2024-03-22T01:24:05.081 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-22T01:24:05.307 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-22T01:24:05.391 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-22T01:24:05.407 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-common-2:19.0.0-2344.gc3a04c0e.el9.x86_64 130/198 2024-03-22T01:24:05.436 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-22T01:24:05.445 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-base-2:19.0.0-2344.gc3a04c0e.el9.x86_64 131/198 2024-03-22T01:24:05.526 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-base-2:19.0.0-2344.gc3a04c0e.el9.x86_64 131/198 2024-03-22T01:24:05.526 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-22T01:24:05.526 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-22T01:24:05.526 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-22T01:24:05.554 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-selinux-2:19.0.0-2344.gc3a04c0e.el9.x86_64 132/198 2024-03-22T01:24:05.572 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-22T01:24:05.618 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-22T01:24:05.673 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-22T01:24:05.755 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-22T01:24:05.795 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-22T01:24:05.832 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-22T01:24:05.879 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-22T01:24:05.924 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-22T01:24:05.969 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-22T01:24:06.021 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-22T01:24:06.154 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-22T01:24:06.204 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-22T01:24:06.244 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-22T01:24:06.286 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-22T01:24:06.382 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-22T01:24:06.421 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-22T01:24:06.462 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-22T01:24:06.495 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-22T01:24:06.522 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-22T01:24:06.612 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-22T01:24:06.743 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-22T01:24:06.853 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-22T01:24:06.905 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-22T01:24:06.962 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-22T01:24:07.027 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-22T01:24:07.225 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-22T01:24:07.281 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-22T01:24:07.417 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-22T01:24:07.456 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-22T01:24:07.497 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-22T01:24:07.530 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-22T01:24:07.567 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-22T01:24:07.601 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-22T01:24:07.641 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-22T01:24:07.673 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-22T01:24:07.714 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-22T01:24:07.788 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-22T01:24:07.826 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-22T01:24:07.859 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-22T01:24:07.893 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-22T01:24:07.928 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-22T01:24:08.017 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-22T01:24:08.133 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-22T01:24:08.186 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-22T01:24:08.220 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-22T01:24:08.267 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-22T01:24:08.374 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-22T01:24:08.431 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-22T01:24:09.152 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-22T01:24:09.196 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-03-22T01:24:09.210 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-22T01:24:09.245 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-22T01:24:09.293 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-22T01:24:09.327 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-22T01:24:09.362 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-22T01:24:09.374 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-22T01:24:09.401 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-22T01:24:09.410 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-22T01:24:09.414 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-22T01:24:09.450 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-22T01:24:09.504 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-22T01:24:09.538 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-22T01:24:09.572 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-22T01:24:09.591 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-22T01:24:09.621 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-22T01:24:09.622 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-22T01:24:09.700 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-22T01:24:09.950 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-22T01:24:10.007 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : librados2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 10/198 2024-03-22T01:24:10.118 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librados2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 10/198 2024-03-22T01:24:10.142 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libcephfs2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 11/198 2024-03-22T01:24:10.233 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libcephfs2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 11/198 2024-03-22T01:24:10.262 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-rados-2:19.0.0-2344.gc3a04c0e.el9.x86_64 12/198 2024-03-22T01:24:10.298 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-22T01:24:10.338 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-22T01:24:10.459 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-22T01:24:10.481 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-22T01:24:10.537 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-22T01:24:10.564 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libcephsqlite-2:19.0.0-2344.gc3a04c0e.el9.x86_64 16/198 2024-03-22T01:24:10.651 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2344.gc3a04c0e.el9.x86_64 16/198 2024-03-22T01:24:10.674 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libradosstriper1-2:19.0.0-2344.gc3a04c0e.el9.x86 17/198 2024-03-22T01:24:10.738 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2344.gc3a04c0e.el9.x86 17/198 2024-03-22T01:24:10.777 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-22T01:24:10.801 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-22T01:24:10.816 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-22T01:24:10.958 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-22T01:24:10.990 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-22T01:24:11.030 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-22T01:24:11.136 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-22T01:24:11.241 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-22T01:24:11.285 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-22T01:24:11.361 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-22T01:24:11.420 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-22T01:24:11.624 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-22T01:24:11.683 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-22T01:24:11.734 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-22T01:24:11.780 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-22T01:24:11.811 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-22T01:24:11.828 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-22T01:24:11.852 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-22T01:24:11.867 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-22T01:24:11.888 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-22T01:24:11.893 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-22T01:24:11.929 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-22T01:24:12.002 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-22T01:24:12.065 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-22T01:24:12.112 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-22T01:24:12.175 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-22T01:24:12.199 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-22T01:24:12.214 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-22T01:24:12.229 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : librbd1-2:19.0.0-2344.gc3a04c0e.el9.x86_64 124/198 2024-03-22T01:24:12.255 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-22T01:24:12.289 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-22T01:24:12.316 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-22T01:24:12.320 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: librbd1-2:19.0.0-2344.gc3a04c0e.el9.x86_64 124/198 2024-03-22T01:24:12.352 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-rbd-2:19.0.0-2344.gc3a04c0e.el9.x86_64 125/198 2024-03-22T01:24:12.386 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-22T01:24:12.397 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-22T01:24:12.423 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-ceph-common-2:19.0.0-2344.gc3a04c0e.el9. 39/198 2024-03-22T01:24:12.464 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-ceph-argparse-2:19.0.0-2344.gc3a04c0e.el 40/198 2024-03-22T01:24:12.493 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cephfs-2:19.0.0-2344.gc3a04c0e.el9.x86_6 41/198 2024-03-22T01:24:12.564 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-22T01:24:12.616 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-22T01:24:12.689 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-22T01:24:12.822 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-22T01:24:12.854 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-22T01:24:12.876 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-22T01:24:12.885 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : librgw2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 128/198 2024-03-22T01:24:12.947 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: librgw2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 128/198 2024-03-22T01:24:12.981 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-rgw-2:19.0.0-2344.gc3a04c0e.el9.x86_64 129/198 2024-03-22T01:24:13.394 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-22T01:24:13.474 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-22T01:24:13.560 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-22T01:24:13.610 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-22T01:24:13.618 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-22T01:24:13.651 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-22T01:24:13.699 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-22T01:24:13.746 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : librados-devel-2:19.0.0-2344.gc3a04c0e.el9.x86_6 53/198 2024-03-22T01:24:13.784 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-22T01:24:14.029 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-22T01:24:14.193 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-22T01:24:14.288 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-22T01:24:14.333 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-22T01:24:14.714 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-22T01:24:14.761 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-22T01:24:14.816 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-22T01:24:14.889 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-22T01:24:14.928 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-22T01:24:14.965 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-22T01:24:15.012 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-22T01:24:15.048 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-22T01:24:15.091 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-22T01:24:15.134 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-22T01:24:15.214 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-common-2:19.0.0-2344.gc3a04c0e.el9.x86_64 130/198 2024-03-22T01:24:15.259 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-common-2:19.0.0-2344.gc3a04c0e.el9.x86_64 130/198 2024-03-22T01:24:15.267 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-22T01:24:15.314 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-22T01:24:15.345 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-22T01:24:15.386 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-22T01:24:15.479 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-22T01:24:15.520 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-22T01:24:15.560 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-22T01:24:15.594 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-22T01:24:15.619 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-22T01:24:15.690 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-22T01:24:15.794 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-common-2:19.0.0-2344.gc3a04c0e.el9.x86_64 130/198 2024-03-22T01:24:15.819 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-22T01:24:15.841 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-base-2:19.0.0-2344.gc3a04c0e.el9.x86_64 131/198 2024-03-22T01:24:15.927 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-22T01:24:15.936 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-base-2:19.0.0-2344.gc3a04c0e.el9.x86_64 131/198 2024-03-22T01:24:15.936 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-22T01:24:15.936 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-22T01:24:15.937 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:24:15.970 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-22T01:24:15.977 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-selinux-2:19.0.0-2344.gc3a04c0e.el9.x86_64 132/198 2024-03-22T01:24:16.018 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-22T01:24:16.074 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-22T01:24:16.264 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-22T01:24:16.316 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-22T01:24:16.443 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-22T01:24:16.490 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-22T01:24:16.530 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-22T01:24:16.561 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-22T01:24:16.589 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-22T01:24:16.624 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-22T01:24:16.662 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-22T01:24:16.694 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-22T01:24:16.724 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-22T01:24:16.799 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-22T01:24:16.835 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-22T01:24:16.868 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-22T01:24:16.918 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-22T01:24:16.953 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-22T01:24:17.050 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-22T01:24:17.140 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-22T01:24:17.184 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-22T01:24:17.216 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-22T01:24:17.263 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-22T01:24:17.367 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-22T01:24:17.425 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-22T01:24:17.566 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2344.gc3a04c0e.el9.x86_64 132/198 2024-03-22T01:24:17.567 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /sys 2024-03-22T01:24:17.567 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /proc 2024-03-22T01:24:17.567 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /mnt 2024-03-22T01:24:17.567 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /var/tmp 2024-03-22T01:24:17.567 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /home 2024-03-22T01:24:17.567 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /root 2024-03-22T01:24:17.567 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /tmp 2024-03-22T01:24:17.567 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-22T01:24:17.730 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-volume-2:19.0.0-2344.gc3a04c0e.el9.noarch 133/198 2024-03-22T01:24:17.747 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-volume-2:19.0.0-2344.gc3a04c0e.el9.noarch 133/198 2024-03-22T01:24:17.747 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T01:24:17.747 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-22T01:24:17.747 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-22T01:24:18.146 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-22T01:24:18.199 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-22T01:24:18.233 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-22T01:24:18.281 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-22T01:24:18.367 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-22T01:24:18.417 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-22T01:24:18.596 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-22T01:24:18.834 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-22T01:24:18.951 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-22T01:24:19.225 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-osd-2:19.0.0-2344.gc3a04c0e.el9.x86_64 134/198 2024-03-22T01:24:19.250 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-osd-2:19.0.0-2344.gc3a04c0e.el9.x86_64 134/198 2024-03-22T01:24:19.251 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T01:24:19.251 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-22T01:24:19.251 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-22T01:24:19.251 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-22T01:24:19.251 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-22T01:24:19.476 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mds-2:19.0.0-2344.gc3a04c0e.el9.x86_64 135/198 2024-03-22T01:24:19.499 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mds-2:19.0.0-2344.gc3a04c0e.el9.x86_64 135/198 2024-03-22T01:24:19.499 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T01:24:19.499 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-22T01:24:19.499 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-22T01:24:19.499 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-22T01:24:19.500 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-22T01:24:19.582 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-22T01:24:19.695 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-22T01:24:19.935 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mon-2:19.0.0-2344.gc3a04c0e.el9.x86_64 136/198 2024-03-22T01:24:19.957 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mon-2:19.0.0-2344.gc3a04c0e.el9.x86_64 136/198 2024-03-22T01:24:19.958 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T01:24:19.958 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-22T01:24:19.958 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-22T01:24:19.958 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-22T01:24:19.958 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-22T01:24:20.053 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-22T01:24:20.421 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: cephadm-2:19.0.0-2344.gc3a04c0e.el9.noarch 138/198 2024-03-22T01:24:20.452 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephadm-2:19.0.0-2344.gc3a04c0e.el9.noarch 138/198 2024-03-22T01:24:20.484 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2344.gc3a04c0e.e 139/198 2024-03-22T01:24:20.544 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2344.gc3a04c0e. 140/198 2024-03-22T01:24:20.575 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2344.gc3a04c0e.el9.noa 141/198 2024-03-22T01:24:20.903 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-22T01:24:20.993 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-22T01:24:21.024 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-22T01:24:21.055 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-22T01:24:21.083 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-22T01:24:21.395 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-22T01:24:21.422 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : librbd1-2:19.0.0-2344.gc3a04c0e.el9.x86_64 124/198 2024-03-22T01:24:21.495 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librbd1-2:19.0.0-2344.gc3a04c0e.el9.x86_64 124/198 2024-03-22T01:24:21.527 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-rbd-2:19.0.0-2344.gc3a04c0e.el9.x86_64 125/198 2024-03-22T01:24:21.562 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-22T01:24:21.605 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2344.gc3a04c0e.el9.noa 141/198 2024-03-22T01:24:21.644 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2344.gc3a04c0e.el9.n 142/198 2024-03-22T01:24:22.031 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-22T01:24:22.059 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : librgw2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 128/198 2024-03-22T01:24:22.112 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librgw2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 128/198 2024-03-22T01:24:22.135 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-rgw-2:19.0.0-2344.gc3a04c0e.el9.x86_64 129/198 2024-03-22T01:24:22.604 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2344.gc3a04c0e.el9.n 142/198 2024-03-22T01:24:22.636 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2344.gc3a 143/198 2024-03-22T01:24:22.734 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2344.gc3a 143/198 2024-03-22T01:24:22.906 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2344.gc3a04c0e.el 144/198 2024-03-22T01:24:22.938 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-2:19.0.0-2344.gc3a04c0e.el9.x86_64 145/198 2024-03-22T01:24:22.961 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2344.gc3a04c0e.el9.x86_64 145/198 2024-03-22T01:24:22.961 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T01:24:22.961 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-22T01:24:22.961 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-22T01:24:22.961 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-22T01:24:22.961 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-22T01:24:23.005 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-rook-2:19.0.0-2344.gc3a04c0e.el9.noarch 146/198 2024-03-22T01:24:23.016 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2344.gc3a04c0e.el9.noarch 146/198 2024-03-22T01:24:23.062 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-2:19.0.0-2344.gc3a04c0e.el9.x86_64 147/198 2024-03-22T01:24:23.933 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-22T01:24:23.966 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-radosgw-2:19.0.0-2344.gc3a04c0e.el9.x86_64 149/198 2024-03-22T01:24:23.988 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2344.gc3a04c0e.el9.x86_64 149/198 2024-03-22T01:24:23.989 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T01:24:23.989 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-22T01:24:23.989 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-22T01:24:23.989 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-22T01:24:23.989 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-22T01:24:24.033 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2344.gc3a04 150/198 2024-03-22T01:24:24.055 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2344.gc3a04 150/198 2024-03-22T01:24:24.055 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T01:24:24.055 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-22T01:24:24.055 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-22T01:24:24.109 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephfs-mirror-2:19.0.0-2344.gc3a04c0e.el9.x86_64 151/198 2024-03-22T01:24:24.131 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2344.gc3a04c0e.el9.x86_64 151/198 2024-03-22T01:24:24.131 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T01:24:24.131 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-22T01:24:24.131 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-22T01:24:24.131 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-22T01:24:24.132 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-22T01:24:24.435 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-mirror-2:19.0.0-2344.gc3a04c0e.el9.x86_64 152/198 2024-03-22T01:24:24.457 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2344.gc3a04c0e.el9.x86_64 152/198 2024-03-22T01:24:24.457 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T01:24:24.457 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-22T01:24:24.457 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-22T01:24:24.457 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-22T01:24:24.457 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-22T01:24:24.467 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-common-2:19.0.0-2344.gc3a04c0e.el9.x86_64 130/198 2024-03-22T01:24:24.518 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-common-2:19.0.0-2344.gc3a04c0e.el9.x86_64 130/198 2024-03-22T01:24:25.052 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-common-2:19.0.0-2344.gc3a04c0e.el9.x86_64 130/198 2024-03-22T01:24:25.096 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-base-2:19.0.0-2344.gc3a04c0e.el9.x86_64 131/198 2024-03-22T01:24:25.177 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-base-2:19.0.0-2344.gc3a04c0e.el9.x86_64 131/198 2024-03-22T01:24:25.177 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-22T01:24:25.178 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-22T01:24:25.178 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-22T01:24:25.206 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-selinux-2:19.0.0-2344.gc3a04c0e.el9.x86_64 132/198 2024-03-22T01:24:28.898 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-test-2:19.0.0-2344.gc3a04c0e.el9.x86_64 153/198 2024-03-22T01:24:28.939 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-fuse-2:19.0.0-2344.gc3a04c0e.el9.x86_64 154/198 2024-03-22T01:24:28.969 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-nbd-2:19.0.0-2344.gc3a04c0e.el9.x86_64 155/198 2024-03-22T01:24:28.999 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-22T01:24:29.176 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephfs-shell-2:19.0.0-2344.gc3a04c0e.el9.x86_64 157/198 2024-03-22T01:24:29.296 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-22T01:24:29.336 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-fuse-2:19.0.0-2344.gc3a04c0e.el9.x86_64 159/198 2024-03-22T01:24:29.384 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-22T01:24:29.437 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-22T01:24:29.480 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-22T01:24:29.513 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephfs-devel-2:19.0.0-2344.gc3a04c0e.el9.x86_ 163/198 2024-03-22T01:24:29.545 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-22T01:24:29.574 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-22T01:24:29.620 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-22T01:24:29.664 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-22T01:24:29.690 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-22T01:24:30.266 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-22T01:24:30.305 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-22T01:24:30.324 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-22T01:24:31.134 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephfs-top-2:19.0.0-2344.gc3a04c0e.el9.noarch 170/198 2024-03-22T01:24:31.217 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-22T01:24:31.256 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2344.gc3a04c0e.el9.x86_64 132/198 2024-03-22T01:24:31.256 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /sys 2024-03-22T01:24:31.256 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /proc 2024-03-22T01:24:31.256 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /mnt 2024-03-22T01:24:31.256 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /var/tmp 2024-03-22T01:24:31.257 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /home 2024-03-22T01:24:31.257 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /root 2024-03-22T01:24:31.257 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /tmp 2024-03-22T01:24:31.257 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:24:31.257 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-22T01:24:31.337 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-22T01:24:31.371 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-22T01:24:31.396 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-22T01:24:31.421 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-22T01:24:31.433 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-volume-2:19.0.0-2344.gc3a04c0e.el9.noarch 133/198 2024-03-22T01:24:31.448 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-22T01:24:31.448 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-22T01:24:31.448 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-volume-2:19.0.0-2344.gc3a04c0e.el9.noarch 133/198 2024-03-22T01:24:31.448 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T01:24:31.448 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-22T01:24:31.448 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:24:31.544 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-22T01:24:31.574 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-22T01:24:31.574 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-22T01:24:31.658 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-22T01:24:31.684 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-22T01:24:31.718 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-22T01:24:31.744 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-22T01:24:31.760 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-22T01:24:31.793 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-22T01:24:31.833 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-22T01:24:31.865 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-22T01:24:31.890 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-22T01:24:31.915 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-22T01:24:31.941 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-22T01:24:31.966 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-22T01:24:31.972 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-22T01:24:31.972 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-22T01:24:32.153 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-22T01:24:32.153 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-22T01:24:32.236 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-22T01:24:32.260 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-22T01:24:32.292 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-22T01:24:32.320 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-22T01:24:32.353 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-22T01:24:32.417 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-22T01:24:32.941 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-osd-2:19.0.0-2344.gc3a04c0e.el9.x86_64 134/198 2024-03-22T01:24:32.966 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-osd-2:19.0.0-2344.gc3a04c0e.el9.x86_64 134/198 2024-03-22T01:24:32.967 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T01:24:32.967 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-22T01:24:32.967 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-22T01:24:32.967 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-22T01:24:32.967 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:24:33.192 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mds-2:19.0.0-2344.gc3a04c0e.el9.x86_64 135/198 2024-03-22T01:24:33.214 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mds-2:19.0.0-2344.gc3a04c0e.el9.x86_64 135/198 2024-03-22T01:24:33.214 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T01:24:33.214 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-22T01:24:33.215 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-22T01:24:33.215 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-22T01:24:33.215 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:24:33.668 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mon-2:19.0.0-2344.gc3a04c0e.el9.x86_64 136/198 2024-03-22T01:24:33.691 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mon-2:19.0.0-2344.gc3a04c0e.el9.x86_64 136/198 2024-03-22T01:24:33.691 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T01:24:33.691 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-22T01:24:33.691 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-22T01:24:33.691 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-22T01:24:33.691 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:24:33.788 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-22T01:24:34.210 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: cephadm-2:19.0.0-2344.gc3a04c0e.el9.noarch 138/198 2024-03-22T01:24:34.242 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : cephadm-2:19.0.0-2344.gc3a04c0e.el9.noarch 138/198 2024-03-22T01:24:34.283 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2344.gc3a04c0e.e 139/198 2024-03-22T01:24:34.345 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2344.gc3a04c0e. 140/198 2024-03-22T01:24:34.376 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2344.gc3a04c0e.el9.noa 141/198 2024-03-22T01:24:35.415 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2344.gc3a04c0e.el9.noa 141/198 2024-03-22T01:24:35.455 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2344.gc3a04c0e.el9.n 142/198 2024-03-22T01:24:36.421 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2344.gc3a04c0e.el9.n 142/198 2024-03-22T01:24:36.447 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2344.gc3a 143/198 2024-03-22T01:24:36.547 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2344.gc3a 143/198 2024-03-22T01:24:36.714 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2344.gc3a04c0e.el 144/198 2024-03-22T01:24:36.744 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mgr-2:19.0.0-2344.gc3a04c0e.el9.x86_64 145/198 2024-03-22T01:24:36.767 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2344.gc3a04c0e.el9.x86_64 145/198 2024-03-22T01:24:36.767 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T01:24:36.767 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-22T01:24:36.767 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-22T01:24:36.767 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-22T01:24:36.767 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:24:36.821 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mgr-rook-2:19.0.0-2344.gc3a04c0e.el9.noarch 146/198 2024-03-22T01:24:36.832 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2344.gc3a04c0e.el9.noarch 146/198 2024-03-22T01:24:36.885 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-2:19.0.0-2344.gc3a04c0e.el9.x86_64 147/198 2024-03-22T01:24:36.935 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2344.gc3a04c0e.el9.x86_64 132/198 2024-03-22T01:24:36.935 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /sys 2024-03-22T01:24:36.935 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /proc 2024-03-22T01:24:36.935 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /mnt 2024-03-22T01:24:36.935 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /var/tmp 2024-03-22T01:24:36.935 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /home 2024-03-22T01:24:36.935 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /root 2024-03-22T01:24:36.935 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /tmp 2024-03-22T01:24:36.935 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-22T01:24:37.097 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-volume-2:19.0.0-2344.gc3a04c0e.el9.noarch 133/198 2024-03-22T01:24:37.113 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-volume-2:19.0.0-2344.gc3a04c0e.el9.noarch 133/198 2024-03-22T01:24:37.113 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T01:24:37.113 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-22T01:24:37.113 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-22T01:24:37.766 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-22T01:24:37.800 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-radosgw-2:19.0.0-2344.gc3a04c0e.el9.x86_64 149/198 2024-03-22T01:24:37.823 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2344.gc3a04c0e.el9.x86_64 149/198 2024-03-22T01:24:37.823 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T01:24:37.823 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-22T01:24:37.823 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-22T01:24:37.823 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-22T01:24:37.823 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:24:37.878 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2344.gc3a04 150/198 2024-03-22T01:24:37.900 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2344.gc3a04 150/198 2024-03-22T01:24:37.900 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T01:24:37.900 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-22T01:24:37.900 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:24:37.954 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : cephfs-mirror-2:19.0.0-2344.gc3a04c0e.el9.x86_64 151/198 2024-03-22T01:24:37.976 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2344.gc3a04c0e.el9.x86_64 151/198 2024-03-22T01:24:37.977 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T01:24:37.977 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-22T01:24:37.977 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-22T01:24:37.977 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-22T01:24:37.977 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:24:38.077 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-22T01:24:38.077 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-2:19.0.0-2344.gc3a04c0e.el9.x86_64 1/198 2024-03-22T01:24:38.077 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-base-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2/198 2024-03-22T01:24:38.077 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-common-2:19.0.0-2344.gc3a04c0e.el9.x86_64 3/198 2024-03-22T01:24:38.077 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-fuse-2:19.0.0-2344.gc3a04c0e.el9.x86_64 4/198 2024-03-22T01:24:38.077 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2344.gc3a04 5/198 2024-03-22T01:24:38.077 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mds-2:19.0.0-2344.gc3a04c0e.el9.x86_64 6/198 2024-03-22T01:24:38.077 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-2:19.0.0-2344.gc3a04c0e.el9.x86_64 7/198 2024-03-22T01:24:38.077 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mon-2:19.0.0-2344.gc3a04c0e.el9.x86_64 8/198 2024-03-22T01:24:38.078 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-osd-2:19.0.0-2344.gc3a04c0e.el9.x86_64 9/198 2024-03-22T01:24:38.078 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-radosgw-2:19.0.0-2344.gc3a04c0e.el9.x86_64 10/198 2024-03-22T01:24:38.078 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-selinux-2:19.0.0-2344.gc3a04c0e.el9.x86_64 11/198 2024-03-22T01:24:38.078 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-test-2:19.0.0-2344.gc3a04c0e.el9.x86_64 12/198 2024-03-22T01:24:38.078 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephfs-mirror-2:19.0.0-2344.gc3a04c0e.el9.x86_64 13/198 2024-03-22T01:24:38.078 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephfs-shell-2:19.0.0-2344.gc3a04c0e.el9.x86_64 14/198 2024-03-22T01:24:38.078 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephfs-devel-2:19.0.0-2344.gc3a04c0e.el9.x86_ 15/198 2024-03-22T01:24:38.078 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephfs2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 16/198 2024-03-22T01:24:38.078 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephsqlite-2:19.0.0-2344.gc3a04c0e.el9.x86_64 17/198 2024-03-22T01:24:38.078 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados-devel-2:19.0.0-2344.gc3a04c0e.el9.x86_6 18/198 2024-03-22T01:24:38.079 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libradosstriper1-2:19.0.0-2344.gc3a04c0e.el9.x86 19/198 2024-03-22T01:24:38.079 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librgw2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 20/198 2024-03-22T01:24:38.079 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2344.gc3a04c0e.el 21/198 2024-03-22T01:24:38.079 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ceph-common-2:19.0.0-2344.gc3a04c0e.el9. 22/198 2024-03-22T01:24:38.079 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cephfs-2:19.0.0-2344.gc3a04c0e.el9.x86_6 23/198 2024-03-22T01:24:38.080 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rados-2:19.0.0-2344.gc3a04c0e.el9.x86_64 24/198 2024-03-22T01:24:38.080 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rbd-2:19.0.0-2344.gc3a04c0e.el9.x86_64 25/198 2024-03-22T01:24:38.080 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rgw-2:19.0.0-2344.gc3a04c0e.el9.x86_64 26/198 2024-03-22T01:24:38.080 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-fuse-2:19.0.0-2344.gc3a04c0e.el9.x86_64 27/198 2024-03-22T01:24:38.080 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-mirror-2:19.0.0-2344.gc3a04c0e.el9.x86_64 28/198 2024-03-22T01:24:38.080 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-nbd-2:19.0.0-2344.gc3a04c0e.el9.x86_64 29/198 2024-03-22T01:24:38.080 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2344.gc3a04c0e. 30/198 2024-03-22T01:24:38.080 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2344.gc3a04c0e.el9.noa 31/198 2024-03-22T01:24:38.082 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2344.gc3a04c0e.el9.n 32/198 2024-03-22T01:24:38.082 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2344.gc3a 33/198 2024-03-22T01:24:38.082 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2344.gc3a04c0e.el 34/198 2024-03-22T01:24:38.083 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2344.gc3a04c0e.el9.noarch 35/198 2024-03-22T01:24:38.083 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2344.gc3a04c0e.e 36/198 2024-03-22T01:24:38.083 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-volume-2:19.0.0-2344.gc3a04c0e.el9.noarch 37/198 2024-03-22T01:24:38.083 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephadm-2:19.0.0-2344.gc3a04c0e.el9.noarch 38/198 2024-03-22T01:24:38.083 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephfs-top-2:19.0.0-2344.gc3a04c0e.el9.noarch 39/198 2024-03-22T01:24:38.084 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-22T01:24:38.084 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-22T01:24:38.084 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-22T01:24:38.084 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-22T01:24:38.084 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-22T01:24:38.084 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-22T01:24:38.085 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-22T01:24:38.085 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-22T01:24:38.085 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-22T01:24:38.085 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-22T01:24:38.085 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-22T01:24:38.086 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-22T01:24:38.086 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-22T01:24:38.086 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-22T01:24:38.086 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-22T01:24:38.086 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-22T01:24:38.086 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-22T01:24:38.087 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-22T01:24:38.087 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-22T01:24:38.087 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-22T01:24:38.087 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-22T01:24:38.087 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-22T01:24:38.088 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-22T01:24:38.088 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-22T01:24:38.088 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-22T01:24:38.088 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-22T01:24:38.088 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-22T01:24:38.088 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-22T01:24:38.088 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-22T01:24:38.089 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-22T01:24:38.089 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-22T01:24:38.089 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-22T01:24:38.089 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-22T01:24:38.089 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-22T01:24:38.089 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-22T01:24:38.090 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-22T01:24:38.090 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-22T01:24:38.090 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-22T01:24:38.090 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-22T01:24:38.090 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-22T01:24:38.090 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-22T01:24:38.091 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-22T01:24:38.091 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-22T01:24:38.091 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-22T01:24:38.091 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-22T01:24:38.091 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-22T01:24:38.091 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-22T01:24:38.092 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-22T01:24:38.092 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-22T01:24:38.092 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-22T01:24:38.092 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-22T01:24:38.093 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-22T01:24:38.093 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-22T01:24:38.093 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-22T01:24:38.093 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-22T01:24:38.093 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-22T01:24:38.093 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-22T01:24:38.094 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-22T01:24:38.094 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-22T01:24:38.094 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-22T01:24:38.094 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-22T01:24:38.094 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-22T01:24:38.095 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-22T01:24:38.095 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-22T01:24:38.095 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-22T01:24:38.095 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-22T01:24:38.095 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-22T01:24:38.095 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-22T01:24:38.096 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-22T01:24:38.096 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-22T01:24:38.096 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-22T01:24:38.096 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-22T01:24:38.096 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-22T01:24:38.096 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-22T01:24:38.097 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-22T01:24:38.097 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-22T01:24:38.097 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-22T01:24:38.097 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-22T01:24:38.097 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-22T01:24:38.097 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-22T01:24:38.098 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-22T01:24:38.098 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-22T01:24:38.098 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-22T01:24:38.098 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-22T01:24:38.098 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-22T01:24:38.098 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-22T01:24:38.099 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-22T01:24:38.099 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-22T01:24:38.099 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-22T01:24:38.099 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-22T01:24:38.099 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-22T01:24:38.100 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-22T01:24:38.100 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-22T01:24:38.100 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-22T01:24:38.100 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-22T01:24:38.100 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-22T01:24:38.100 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-22T01:24:38.101 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-22T01:24:38.101 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-22T01:24:38.101 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-22T01:24:38.101 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-22T01:24:38.101 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-22T01:24:38.101 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-22T01:24:38.102 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-22T01:24:38.102 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-22T01:24:38.102 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-22T01:24:38.102 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-22T01:24:38.103 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-22T01:24:38.103 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-22T01:24:38.103 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-22T01:24:38.103 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-22T01:24:38.103 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 151/198 2024-03-22T01:24:38.103 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-22T01:24:38.103 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librbd1-2:19.0.0-2344.gc3a04c0e.el9.x86_64 153/198 2024-03-22T01:24:38.103 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-22T01:24:38.103 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-22T01:24:38.104 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-22T01:24:38.104 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-22T01:24:38.104 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-22T01:24:38.104 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-22T01:24:38.104 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-22T01:24:38.104 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-22T01:24:38.104 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-22T01:24:38.104 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-22T01:24:38.104 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-22T01:24:38.104 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-22T01:24:38.104 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-22T01:24:38.104 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-22T01:24:38.105 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-22T01:24:38.105 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-22T01:24:38.105 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-22T01:24:38.105 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-22T01:24:38.105 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-22T01:24:38.105 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-22T01:24:38.105 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-22T01:24:38.105 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-22T01:24:38.105 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-22T01:24:38.105 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-22T01:24:38.105 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-22T01:24:38.105 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-22T01:24:38.106 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-22T01:24:38.106 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-22T01:24:38.106 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-22T01:24:38.106 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-22T01:24:38.106 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-22T01:24:38.106 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-22T01:24:38.106 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-22T01:24:38.106 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-22T01:24:38.106 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-22T01:24:38.106 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-22T01:24:38.106 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-22T01:24:38.107 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-22T01:24:38.107 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-22T01:24:38.107 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-22T01:24:38.107 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-22T01:24:38.107 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-22T01:24:38.107 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-22T01:24:38.107 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-22T01:24:38.272 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : rbd-mirror-2:19.0.0-2344.gc3a04c0e.el9.x86_64 152/198 2024-03-22T01:24:38.294 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2344.gc3a04c0e.el9.x86_64 152/198 2024-03-22T01:24:38.294 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T01:24:38.294 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-22T01:24:38.294 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-22T01:24:38.295 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-22T01:24:38.295 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:24:38.584 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-osd-2:19.0.0-2344.gc3a04c0e.el9.x86_64 134/198 2024-03-22T01:24:38.616 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-osd-2:19.0.0-2344.gc3a04c0e.el9.x86_64 134/198 2024-03-22T01:24:38.616 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T01:24:38.616 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-22T01:24:38.616 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-22T01:24:38.617 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-22T01:24:38.617 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-22T01:24:38.845 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mds-2:19.0.0-2344.gc3a04c0e.el9.x86_64 135/198 2024-03-22T01:24:38.873 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mds-2:19.0.0-2344.gc3a04c0e.el9.x86_64 135/198 2024-03-22T01:24:38.874 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T01:24:38.874 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-22T01:24:38.874 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-22T01:24:38.874 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-22T01:24:38.874 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-22T01:24:39.321 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mon-2:19.0.0-2344.gc3a04c0e.el9.x86_64 136/198 2024-03-22T01:24:39.344 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mon-2:19.0.0-2344.gc3a04c0e.el9.x86_64 136/198 2024-03-22T01:24:39.344 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T01:24:39.344 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-22T01:24:39.344 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-22T01:24:39.344 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-22T01:24:39.344 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-22T01:24:39.432 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-22T01:24:39.809 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: cephadm-2:19.0.0-2344.gc3a04c0e.el9.noarch 138/198 2024-03-22T01:24:39.842 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : cephadm-2:19.0.0-2344.gc3a04c0e.el9.noarch 138/198 2024-03-22T01:24:39.871 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2344.gc3a04c0e.e 139/198 2024-03-22T01:24:39.931 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2344.gc3a04c0e. 140/198 2024-03-22T01:24:39.954 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2344.gc3a04c0e.el9.noa 141/198 2024-03-22T01:24:40.993 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2344.gc3a04c0e.el9.noa 141/198 2024-03-22T01:24:41.060 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2344.gc3a04c0e.el9.n 142/198 2024-03-22T01:24:41.201 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-22T01:24:41.202 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-22T01:24:41.202 INFO:teuthology.orchestra.run.smithi098.stdout:Upgraded: 2024-03-22T01:24:41.202 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-22T01:24:41.202 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-22T01:24:41.202 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-22T01:24:41.202 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-22T01:24:41.202 INFO:teuthology.orchestra.run.smithi098.stdout: acl-2.3.1-4.el9.x86_64 2024-03-22T01:24:41.202 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-22T01:24:41.202 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-22T01:24:41.202 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-22T01:24:41.202 INFO:teuthology.orchestra.run.smithi098.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-22T01:24:41.203 INFO:teuthology.orchestra.run.smithi098.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-22T01:24:41.203 INFO:teuthology.orchestra.run.smithi098.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-22T01:24:41.203 INFO:teuthology.orchestra.run.smithi098.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-22T01:24:41.203 INFO:teuthology.orchestra.run.smithi098.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-22T01:24:41.203 INFO:teuthology.orchestra.run.smithi098.stdout: librados2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:24:41.203 INFO:teuthology.orchestra.run.smithi098.stdout: librbd1-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:24:41.203 INFO:teuthology.orchestra.run.smithi098.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-22T01:24:41.203 INFO:teuthology.orchestra.run.smithi098.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-22T01:24:41.203 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-22T01:24:41.203 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-22T01:24:41.203 INFO:teuthology.orchestra.run.smithi098.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-22T01:24:41.203 INFO:teuthology.orchestra.run.smithi098.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-22T01:24:41.204 INFO:teuthology.orchestra.run.smithi098.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-22T01:24:41.204 INFO:teuthology.orchestra.run.smithi098.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-22T01:24:41.204 INFO:teuthology.orchestra.run.smithi098.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-22T01:24:41.204 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-22T01:24:41.204 INFO:teuthology.orchestra.run.smithi098.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-22T01:24:41.204 INFO:teuthology.orchestra.run.smithi098.stdout: automake-1.16.2-8.el9.noarch 2024-03-22T01:24:41.204 INFO:teuthology.orchestra.run.smithi098.stdout: bison-3.7.4-5.el9.x86_64 2024-03-22T01:24:41.204 INFO:teuthology.orchestra.run.smithi098.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-22T01:24:41.204 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:24:41.204 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-base-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:24:41.204 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-common-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:24:41.204 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-fuse-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:24:41.205 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-grafana-dashboards-2:19.0.0-2344.gc3a04c0e.el9.noarch 2024-03-22T01:24:41.205 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-immutable-object-cache-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:24:41.205 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mds-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:24:41.205 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:24:41.205 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-cephadm-2:19.0.0-2344.gc3a04c0e.el9.noarch 2024-03-22T01:24:41.205 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-dashboard-2:19.0.0-2344.gc3a04c0e.el9.noarch 2024-03-22T01:24:41.205 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2344.gc3a04c0e.el9.noarch 2024-03-22T01:24:41.205 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-modules-core-2:19.0.0-2344.gc3a04c0e.el9.noarch 2024-03-22T01:24:41.205 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-rook-2:19.0.0-2344.gc3a04c0e.el9.noarch 2024-03-22T01:24:41.205 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mon-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:24:41.205 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-osd-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:24:41.206 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-prometheus-alerts-2:19.0.0-2344.gc3a04c0e.el9.noarch 2024-03-22T01:24:41.206 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-radosgw-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:24:41.206 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-selinux-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:24:41.206 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-test-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:24:41.206 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-volume-2:19.0.0-2344.gc3a04c0e.el9.noarch 2024-03-22T01:24:41.206 INFO:teuthology.orchestra.run.smithi098.stdout: cephadm-2:19.0.0-2344.gc3a04c0e.el9.noarch 2024-03-22T01:24:41.206 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-mirror-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:24:41.206 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-shell-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:24:41.206 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-top-2:19.0.0-2344.gc3a04c0e.el9.noarch 2024-03-22T01:24:41.206 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-22T01:24:41.207 INFO:teuthology.orchestra.run.smithi098.stdout: flex-2.6.4-9.el9.x86_64 2024-03-22T01:24:41.207 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-22T01:24:41.207 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-22T01:24:41.207 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-22T01:24:41.207 INFO:teuthology.orchestra.run.smithi098.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-22T01:24:41.207 INFO:teuthology.orchestra.run.smithi098.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-22T01:24:41.207 INFO:teuthology.orchestra.run.smithi098.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-22T01:24:41.207 INFO:teuthology.orchestra.run.smithi098.stdout: jq-1.6-16.el9.x86_64 2024-03-22T01:24:41.207 INFO:teuthology.orchestra.run.smithi098.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-22T01:24:41.207 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-22T01:24:41.207 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-22T01:24:41.207 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs-devel-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:24:41.208 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:24:41.208 INFO:teuthology.orchestra.run.smithi098.stdout: libcephsqlite-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:24:41.208 INFO:teuthology.orchestra.run.smithi098.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-22T01:24:41.208 INFO:teuthology.orchestra.run.smithi098.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-22T01:24:41.208 INFO:teuthology.orchestra.run.smithi098.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-22T01:24:41.208 INFO:teuthology.orchestra.run.smithi098.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-22T01:24:41.208 INFO:teuthology.orchestra.run.smithi098.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-22T01:24:41.208 INFO:teuthology.orchestra.run.smithi098.stdout: librados-devel-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:24:41.208 INFO:teuthology.orchestra.run.smithi098.stdout: libradosstriper1-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:24:41.208 INFO:teuthology.orchestra.run.smithi098.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-22T01:24:41.208 INFO:teuthology.orchestra.run.smithi098.stdout: librgw2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:24:41.209 INFO:teuthology.orchestra.run.smithi098.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-22T01:24:41.209 INFO:teuthology.orchestra.run.smithi098.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-22T01:24:41.209 INFO:teuthology.orchestra.run.smithi098.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-22T01:24:41.209 INFO:teuthology.orchestra.run.smithi098.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-22T01:24:41.209 INFO:teuthology.orchestra.run.smithi098.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-22T01:24:41.209 INFO:teuthology.orchestra.run.smithi098.stdout: lua-5.4.4-4.el9.x86_64 2024-03-22T01:24:41.209 INFO:teuthology.orchestra.run.smithi098.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-22T01:24:41.209 INFO:teuthology.orchestra.run.smithi098.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-22T01:24:41.209 INFO:teuthology.orchestra.run.smithi098.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-22T01:24:41.209 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-22T01:24:41.209 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-22T01:24:41.209 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-22T01:24:41.209 INFO:teuthology.orchestra.run.smithi098.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-22T01:24:41.210 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-13.11-1.el9.x86_64 2024-03-22T01:24:41.210 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-22T01:24:41.210 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-22T01:24:41.210 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-22T01:24:41.210 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-22T01:24:41.210 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-22T01:24:41.210 INFO:teuthology.orchestra.run.smithi098.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-22T01:24:41.210 INFO:teuthology.orchestra.run.smithi098.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-22T01:24:41.210 INFO:teuthology.orchestra.run.smithi098.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-22T01:24:41.210 INFO:teuthology.orchestra.run.smithi098.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-22T01:24:41.210 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-22T01:24:41.210 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-argparse-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:24:41.211 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-common-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:24:41.211 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cephfs-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:24:41.211 INFO:teuthology.orchestra.run.smithi098.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-22T01:24:41.211 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-22T01:24:41.211 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-22T01:24:41.211 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-22T01:24:41.211 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-22T01:24:41.211 INFO:teuthology.orchestra.run.smithi098.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-22T01:24:41.211 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-22T01:24:41.211 INFO:teuthology.orchestra.run.smithi098.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-22T01:24:41.212 INFO:teuthology.orchestra.run.smithi098.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-22T01:24:41.212 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-22T01:24:41.212 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-22T01:24:41.212 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-22T01:24:41.212 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-22T01:24:41.212 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-22T01:24:41.212 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-22T01:24:41.212 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-22T01:24:41.212 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-22T01:24:41.212 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-22T01:24:41.212 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-22T01:24:41.212 INFO:teuthology.orchestra.run.smithi098.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-22T01:24:41.213 INFO:teuthology.orchestra.run.smithi098.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-22T01:24:41.213 INFO:teuthology.orchestra.run.smithi098.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-22T01:24:41.213 INFO:teuthology.orchestra.run.smithi098.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-22T01:24:41.213 INFO:teuthology.orchestra.run.smithi098.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-22T01:24:41.213 INFO:teuthology.orchestra.run.smithi098.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-22T01:24:41.213 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-22T01:24:41.213 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-22T01:24:41.213 INFO:teuthology.orchestra.run.smithi098.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-22T01:24:41.213 INFO:teuthology.orchestra.run.smithi098.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-22T01:24:41.213 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-22T01:24:41.213 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ply-3.11-14.el9.noarch 2024-03-22T01:24:41.213 INFO:teuthology.orchestra.run.smithi098.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-22T01:24:41.214 INFO:teuthology.orchestra.run.smithi098.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-22T01:24:41.214 INFO:teuthology.orchestra.run.smithi098.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-22T01:24:41.214 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-22T01:24:41.214 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-22T01:24:41.214 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-22T01:24:41.214 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-22T01:24:41.214 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-22T01:24:41.214 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-22T01:24:41.214 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rados-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:24:41.214 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rbd-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:24:41.214 INFO:teuthology.orchestra.run.smithi098.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-22T01:24:41.214 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-22T01:24:41.215 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-22T01:24:41.215 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rgw-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:24:41.215 INFO:teuthology.orchestra.run.smithi098.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-22T01:24:41.215 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-22T01:24:41.215 INFO:teuthology.orchestra.run.smithi098.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-22T01:24:41.215 INFO:teuthology.orchestra.run.smithi098.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-22T01:24:41.215 INFO:teuthology.orchestra.run.smithi098.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-22T01:24:41.215 INFO:teuthology.orchestra.run.smithi098.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-22T01:24:41.215 INFO:teuthology.orchestra.run.smithi098.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-22T01:24:41.215 INFO:teuthology.orchestra.run.smithi098.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-22T01:24:41.215 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-22T01:24:41.216 INFO:teuthology.orchestra.run.smithi098.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-22T01:24:41.216 INFO:teuthology.orchestra.run.smithi098.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-22T01:24:41.216 INFO:teuthology.orchestra.run.smithi098.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-22T01:24:41.216 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-22T01:24:41.216 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-22T01:24:41.216 INFO:teuthology.orchestra.run.smithi098.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-22T01:24:41.216 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-fuse-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:24:41.216 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-mirror-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:24:41.216 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-nbd-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:24:41.216 INFO:teuthology.orchestra.run.smithi098.stdout: re2-1:20211101-3.el9.x86_64 2024-03-22T01:24:41.216 INFO:teuthology.orchestra.run.smithi098.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-22T01:24:41.216 INFO:teuthology.orchestra.run.smithi098.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-22T01:24:41.217 INFO:teuthology.orchestra.run.smithi098.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-22T01:24:41.217 INFO:teuthology.orchestra.run.smithi098.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-22T01:24:41.217 INFO:teuthology.orchestra.run.smithi098.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-22T01:24:41.217 INFO:teuthology.orchestra.run.smithi098.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-22T01:24:41.217 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-22T01:24:41.218 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-22T01:24:41.640 DEBUG:teuthology.parallel:result is None 2024-03-22T01:24:42.032 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2344.gc3a04c0e.el9.n 142/198 2024-03-22T01:24:42.065 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2344.gc3a 143/198 2024-03-22T01:24:42.165 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2344.gc3a 143/198 2024-03-22T01:24:42.337 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2344.gc3a04c0e.el 144/198 2024-03-22T01:24:42.376 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-2:19.0.0-2344.gc3a04c0e.el9.x86_64 145/198 2024-03-22T01:24:42.398 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2344.gc3a04c0e.el9.x86_64 145/198 2024-03-22T01:24:42.398 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T01:24:42.398 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-22T01:24:42.398 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-22T01:24:42.398 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-22T01:24:42.399 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-22T01:24:42.443 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-rook-2:19.0.0-2344.gc3a04c0e.el9.noarch 146/198 2024-03-22T01:24:42.454 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2344.gc3a04c0e.el9.noarch 146/198 2024-03-22T01:24:42.499 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-2:19.0.0-2344.gc3a04c0e.el9.x86_64 147/198 2024-03-22T01:24:42.775 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-test-2:19.0.0-2344.gc3a04c0e.el9.x86_64 153/198 2024-03-22T01:24:42.817 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : rbd-fuse-2:19.0.0-2344.gc3a04c0e.el9.x86_64 154/198 2024-03-22T01:24:42.856 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : rbd-nbd-2:19.0.0-2344.gc3a04c0e.el9.x86_64 155/198 2024-03-22T01:24:42.894 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-22T01:24:43.074 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : cephfs-shell-2:19.0.0-2344.gc3a04c0e.el9.x86_64 157/198 2024-03-22T01:24:43.199 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-22T01:24:43.247 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-fuse-2:19.0.0-2344.gc3a04c0e.el9.x86_64 159/198 2024-03-22T01:24:43.310 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-22T01:24:43.366 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-22T01:24:43.376 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-22T01:24:43.402 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-22T01:24:43.412 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-radosgw-2:19.0.0-2344.gc3a04c0e.el9.x86_64 149/198 2024-03-22T01:24:43.434 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2344.gc3a04c0e.el9.x86_64 149/198 2024-03-22T01:24:43.434 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T01:24:43.434 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-22T01:24:43.434 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-22T01:24:43.434 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-22T01:24:43.434 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-22T01:24:43.437 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libcephfs-devel-2:19.0.0-2344.gc3a04c0e.el9.x86_ 163/198 2024-03-22T01:24:43.470 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-22T01:24:43.479 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2344.gc3a04 150/198 2024-03-22T01:24:43.501 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2344.gc3a04 150/198 2024-03-22T01:24:43.501 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T01:24:43.501 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-22T01:24:43.501 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-22T01:24:43.508 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-22T01:24:43.547 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : cephfs-mirror-2:19.0.0-2344.gc3a04c0e.el9.x86_64 151/198 2024-03-22T01:24:43.569 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2344.gc3a04c0e.el9.x86_64 151/198 2024-03-22T01:24:43.569 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T01:24:43.569 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-22T01:24:43.569 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-22T01:24:43.569 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-22T01:24:43.569 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-22T01:24:43.590 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-22T01:24:43.650 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-22T01:24:43.686 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-22T01:24:43.881 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : rbd-mirror-2:19.0.0-2344.gc3a04c0e.el9.x86_64 152/198 2024-03-22T01:24:43.903 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2344.gc3a04c0e.el9.x86_64 152/198 2024-03-22T01:24:43.903 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T01:24:43.903 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-22T01:24:43.903 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-22T01:24:43.904 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-22T01:24:43.904 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-22T01:24:44.333 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-22T01:24:44.375 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-22T01:24:44.395 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-22T01:24:45.217 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : cephfs-top-2:19.0.0-2344.gc3a04c0e.el9.noarch 170/198 2024-03-22T01:24:45.318 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-22T01:24:45.359 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-22T01:24:45.451 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-22T01:24:45.483 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-22T01:24:45.510 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-22T01:24:45.536 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-22T01:24:45.563 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-22T01:24:45.563 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-22T01:24:45.660 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-22T01:24:45.690 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-22T01:24:45.690 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-22T01:24:45.784 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-22T01:24:45.819 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-22T01:24:45.863 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-22T01:24:45.889 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-22T01:24:45.914 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-22T01:24:45.967 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-22T01:24:46.015 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-22T01:24:46.058 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-22T01:24:46.083 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-22T01:24:46.117 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-22T01:24:46.151 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-22T01:24:46.177 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-22T01:24:46.183 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-22T01:24:46.183 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-22T01:24:46.363 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-22T01:24:46.363 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-22T01:24:46.457 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-22T01:24:46.482 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-22T01:24:46.515 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-22T01:24:46.551 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-22T01:24:46.584 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-22T01:24:46.651 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-22T01:24:48.528 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-test-2:19.0.0-2344.gc3a04c0e.el9.x86_64 153/198 2024-03-22T01:24:48.569 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : rbd-fuse-2:19.0.0-2344.gc3a04c0e.el9.x86_64 154/198 2024-03-22T01:24:48.599 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : rbd-nbd-2:19.0.0-2344.gc3a04c0e.el9.x86_64 155/198 2024-03-22T01:24:48.627 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-22T01:24:48.804 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : cephfs-shell-2:19.0.0-2344.gc3a04c0e.el9.x86_64 157/198 2024-03-22T01:24:48.918 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-22T01:24:48.959 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-fuse-2:19.0.0-2344.gc3a04c0e.el9.x86_64 159/198 2024-03-22T01:24:48.998 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-22T01:24:49.076 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-22T01:24:49.119 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-22T01:24:49.160 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libcephfs-devel-2:19.0.0-2344.gc3a04c0e.el9.x86_ 163/198 2024-03-22T01:24:49.193 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-22T01:24:49.222 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-22T01:24:49.309 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-22T01:24:49.377 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-22T01:24:49.413 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-22T01:24:50.045 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-22T01:24:50.087 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-22T01:24:50.106 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-22T01:24:50.922 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : cephfs-top-2:19.0.0-2344.gc3a04c0e.el9.noarch 170/198 2024-03-22T01:24:51.000 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-22T01:24:51.041 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-22T01:24:51.121 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-22T01:24:51.163 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-22T01:24:51.188 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-22T01:24:51.213 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-22T01:24:51.240 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-22T01:24:51.240 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-22T01:24:51.328 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-22T01:24:51.357 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-22T01:24:51.358 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-22T01:24:51.434 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-22T01:24:51.460 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-22T01:24:51.502 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-22T01:24:51.536 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-22T01:24:51.561 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-22T01:24:51.594 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-22T01:24:51.634 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-22T01:24:51.666 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-22T01:24:51.691 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-22T01:24:51.725 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-22T01:24:51.751 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-22T01:24:51.776 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-22T01:24:51.782 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-22T01:24:51.783 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-22T01:24:51.962 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-22T01:24:51.963 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-22T01:24:52.054 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-22T01:24:52.079 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-22T01:24:52.110 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-22T01:24:52.138 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-22T01:24:52.172 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-22T01:24:52.243 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-22T01:24:56.467 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-22T01:24:56.467 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-2:19.0.0-2344.gc3a04c0e.el9.x86_64 1/198 2024-03-22T01:24:56.467 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-base-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2/198 2024-03-22T01:24:56.467 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-common-2:19.0.0-2344.gc3a04c0e.el9.x86_64 3/198 2024-03-22T01:24:56.467 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-fuse-2:19.0.0-2344.gc3a04c0e.el9.x86_64 4/198 2024-03-22T01:24:56.468 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2344.gc3a04 5/198 2024-03-22T01:24:56.468 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mds-2:19.0.0-2344.gc3a04c0e.el9.x86_64 6/198 2024-03-22T01:24:56.468 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mgr-2:19.0.0-2344.gc3a04c0e.el9.x86_64 7/198 2024-03-22T01:24:56.468 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mon-2:19.0.0-2344.gc3a04c0e.el9.x86_64 8/198 2024-03-22T01:24:56.468 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-osd-2:19.0.0-2344.gc3a04c0e.el9.x86_64 9/198 2024-03-22T01:24:56.468 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-radosgw-2:19.0.0-2344.gc3a04c0e.el9.x86_64 10/198 2024-03-22T01:24:56.468 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-selinux-2:19.0.0-2344.gc3a04c0e.el9.x86_64 11/198 2024-03-22T01:24:56.468 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-test-2:19.0.0-2344.gc3a04c0e.el9.x86_64 12/198 2024-03-22T01:24:56.468 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : cephfs-mirror-2:19.0.0-2344.gc3a04c0e.el9.x86_64 13/198 2024-03-22T01:24:56.469 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : cephfs-shell-2:19.0.0-2344.gc3a04c0e.el9.x86_64 14/198 2024-03-22T01:24:56.469 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libcephfs-devel-2:19.0.0-2344.gc3a04c0e.el9.x86_ 15/198 2024-03-22T01:24:56.469 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libcephfs2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 16/198 2024-03-22T01:24:56.469 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libcephsqlite-2:19.0.0-2344.gc3a04c0e.el9.x86_64 17/198 2024-03-22T01:24:56.469 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librados-devel-2:19.0.0-2344.gc3a04c0e.el9.x86_6 18/198 2024-03-22T01:24:56.469 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libradosstriper1-2:19.0.0-2344.gc3a04c0e.el9.x86 19/198 2024-03-22T01:24:56.469 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librgw2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 20/198 2024-03-22T01:24:56.469 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2344.gc3a04c0e.el 21/198 2024-03-22T01:24:56.469 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-ceph-common-2:19.0.0-2344.gc3a04c0e.el9. 22/198 2024-03-22T01:24:56.469 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cephfs-2:19.0.0-2344.gc3a04c0e.el9.x86_6 23/198 2024-03-22T01:24:56.470 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-rados-2:19.0.0-2344.gc3a04c0e.el9.x86_64 24/198 2024-03-22T01:24:56.470 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-rbd-2:19.0.0-2344.gc3a04c0e.el9.x86_64 25/198 2024-03-22T01:24:56.470 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-rgw-2:19.0.0-2344.gc3a04c0e.el9.x86_64 26/198 2024-03-22T01:24:56.470 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : rbd-fuse-2:19.0.0-2344.gc3a04c0e.el9.x86_64 27/198 2024-03-22T01:24:56.470 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : rbd-mirror-2:19.0.0-2344.gc3a04c0e.el9.x86_64 28/198 2024-03-22T01:24:56.488 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : rbd-nbd-2:19.0.0-2344.gc3a04c0e.el9.x86_64 29/198 2024-03-22T01:24:56.488 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2344.gc3a04c0e. 30/198 2024-03-22T01:24:56.488 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2344.gc3a04c0e.el9.noa 31/198 2024-03-22T01:24:56.488 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2344.gc3a04c0e.el9.n 32/198 2024-03-22T01:24:56.488 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2344.gc3a 33/198 2024-03-22T01:24:56.488 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2344.gc3a04c0e.el 34/198 2024-03-22T01:24:56.488 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2344.gc3a04c0e.el9.noarch 35/198 2024-03-22T01:24:56.489 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2344.gc3a04c0e.e 36/198 2024-03-22T01:24:56.489 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-volume-2:19.0.0-2344.gc3a04c0e.el9.noarch 37/198 2024-03-22T01:24:56.489 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : cephadm-2:19.0.0-2344.gc3a04c0e.el9.noarch 38/198 2024-03-22T01:24:56.489 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : cephfs-top-2:19.0.0-2344.gc3a04c0e.el9.noarch 39/198 2024-03-22T01:24:56.489 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-22T01:24:56.489 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-22T01:24:56.489 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-22T01:24:56.489 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-22T01:24:56.489 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-22T01:24:56.489 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-22T01:24:56.489 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-22T01:24:56.489 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-22T01:24:56.489 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-22T01:24:56.490 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-22T01:24:56.490 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-22T01:24:56.490 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-22T01:24:56.490 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-22T01:24:56.490 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-22T01:24:56.490 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-22T01:24:56.490 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-22T01:24:56.490 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-22T01:24:56.490 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-22T01:24:56.490 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-22T01:24:56.490 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-22T01:24:56.491 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-22T01:24:56.491 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-22T01:24:56.491 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-22T01:24:56.491 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-22T01:24:56.491 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-22T01:24:56.491 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-22T01:24:56.491 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-22T01:24:56.491 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-22T01:24:56.491 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-22T01:24:56.491 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-22T01:24:56.491 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-22T01:24:56.491 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-22T01:24:56.491 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-22T01:24:56.491 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-22T01:24:56.492 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-22T01:24:56.492 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-22T01:24:56.492 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-22T01:24:56.492 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-22T01:24:56.492 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-22T01:24:56.492 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-22T01:24:56.492 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-22T01:24:56.492 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-22T01:24:56.492 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-22T01:24:56.492 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-22T01:24:56.492 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-22T01:24:56.492 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-22T01:24:56.492 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-22T01:24:56.493 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-22T01:24:56.493 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-22T01:24:56.493 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-22T01:24:56.493 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-22T01:24:56.493 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-22T01:24:56.493 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-22T01:24:56.493 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-22T01:24:56.493 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-22T01:24:56.493 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-22T01:24:56.493 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-22T01:24:56.493 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-22T01:24:56.493 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-22T01:24:56.493 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-22T01:24:56.493 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-22T01:24:56.494 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-22T01:24:56.494 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-22T01:24:56.494 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-22T01:24:56.494 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-22T01:24:56.494 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-22T01:24:56.494 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-22T01:24:56.494 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-22T01:24:56.494 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-22T01:24:56.494 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-22T01:24:56.494 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-22T01:24:56.494 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-22T01:24:56.494 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-22T01:24:56.494 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-22T01:24:56.495 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-22T01:24:56.495 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-22T01:24:56.495 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-22T01:24:56.495 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-22T01:24:56.495 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-22T01:24:56.495 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-22T01:24:56.495 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-22T01:24:56.495 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-22T01:24:56.495 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-22T01:24:56.495 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-22T01:24:56.495 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-22T01:24:56.495 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-22T01:24:56.495 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-22T01:24:56.495 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-22T01:24:56.495 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-22T01:24:56.496 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-22T01:24:56.496 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-22T01:24:56.496 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-22T01:24:56.496 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-22T01:24:56.496 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-22T01:24:56.496 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-22T01:24:56.496 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-22T01:24:56.496 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-22T01:24:56.496 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-22T01:24:56.496 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-22T01:24:56.496 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-22T01:24:56.496 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-22T01:24:56.496 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-22T01:24:56.496 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-22T01:24:56.497 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-22T01:24:56.497 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-22T01:24:56.497 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-22T01:24:56.497 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-22T01:24:56.497 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-22T01:24:56.497 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-22T01:24:56.497 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-22T01:24:56.497 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-22T01:24:56.497 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librados2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 151/198 2024-03-22T01:24:56.497 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-22T01:24:56.497 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librbd1-2:19.0.0-2344.gc3a04c0e.el9.x86_64 153/198 2024-03-22T01:24:56.497 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-22T01:24:56.497 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-22T01:24:56.497 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-22T01:24:56.498 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-22T01:24:56.498 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-22T01:24:56.498 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-22T01:24:56.498 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-22T01:24:56.498 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-22T01:24:56.498 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-22T01:24:56.498 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-22T01:24:56.498 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-22T01:24:56.498 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-22T01:24:56.498 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-22T01:24:56.498 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-22T01:24:56.498 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-22T01:24:56.498 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-22T01:24:56.498 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-22T01:24:56.499 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-22T01:24:56.499 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-22T01:24:56.499 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-22T01:24:56.499 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-22T01:24:56.499 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-22T01:24:56.499 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-22T01:24:56.499 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-22T01:24:56.499 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-22T01:24:56.499 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-22T01:24:56.499 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-22T01:24:56.499 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-22T01:24:56.499 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-22T01:24:56.499 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-22T01:24:56.499 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-22T01:24:56.499 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-22T01:24:56.499 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-22T01:24:56.500 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-22T01:24:56.500 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-22T01:24:56.500 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-22T01:24:56.500 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-22T01:24:56.500 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-22T01:24:56.500 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-22T01:24:56.500 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-22T01:24:56.500 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-22T01:24:56.500 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-22T01:24:56.500 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-22T01:24:56.500 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-22T01:24:57.882 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-22T01:24:57.882 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-2:19.0.0-2344.gc3a04c0e.el9.x86_64 1/198 2024-03-22T01:24:57.882 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-base-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2/198 2024-03-22T01:24:57.882 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-common-2:19.0.0-2344.gc3a04c0e.el9.x86_64 3/198 2024-03-22T01:24:57.882 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-fuse-2:19.0.0-2344.gc3a04c0e.el9.x86_64 4/198 2024-03-22T01:24:57.882 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2344.gc3a04 5/198 2024-03-22T01:24:57.882 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mds-2:19.0.0-2344.gc3a04c0e.el9.x86_64 6/198 2024-03-22T01:24:57.883 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-2:19.0.0-2344.gc3a04c0e.el9.x86_64 7/198 2024-03-22T01:24:57.883 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mon-2:19.0.0-2344.gc3a04c0e.el9.x86_64 8/198 2024-03-22T01:24:57.883 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-osd-2:19.0.0-2344.gc3a04c0e.el9.x86_64 9/198 2024-03-22T01:24:57.886 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-radosgw-2:19.0.0-2344.gc3a04c0e.el9.x86_64 10/198 2024-03-22T01:24:57.886 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-selinux-2:19.0.0-2344.gc3a04c0e.el9.x86_64 11/198 2024-03-22T01:24:57.886 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-test-2:19.0.0-2344.gc3a04c0e.el9.x86_64 12/198 2024-03-22T01:24:57.886 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : cephfs-mirror-2:19.0.0-2344.gc3a04c0e.el9.x86_64 13/198 2024-03-22T01:24:57.886 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : cephfs-shell-2:19.0.0-2344.gc3a04c0e.el9.x86_64 14/198 2024-03-22T01:24:57.886 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libcephfs-devel-2:19.0.0-2344.gc3a04c0e.el9.x86_ 15/198 2024-03-22T01:24:57.886 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libcephfs2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 16/198 2024-03-22T01:24:57.886 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libcephsqlite-2:19.0.0-2344.gc3a04c0e.el9.x86_64 17/198 2024-03-22T01:24:57.886 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librados-devel-2:19.0.0-2344.gc3a04c0e.el9.x86_6 18/198 2024-03-22T01:24:57.886 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libradosstriper1-2:19.0.0-2344.gc3a04c0e.el9.x86 19/198 2024-03-22T01:24:57.886 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librgw2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 20/198 2024-03-22T01:24:57.887 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2344.gc3a04c0e.el 21/198 2024-03-22T01:24:57.887 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-ceph-common-2:19.0.0-2344.gc3a04c0e.el9. 22/198 2024-03-22T01:24:57.887 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cephfs-2:19.0.0-2344.gc3a04c0e.el9.x86_6 23/198 2024-03-22T01:24:57.887 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-rados-2:19.0.0-2344.gc3a04c0e.el9.x86_64 24/198 2024-03-22T01:24:57.887 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-rbd-2:19.0.0-2344.gc3a04c0e.el9.x86_64 25/198 2024-03-22T01:24:57.887 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-rgw-2:19.0.0-2344.gc3a04c0e.el9.x86_64 26/198 2024-03-22T01:24:57.887 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : rbd-fuse-2:19.0.0-2344.gc3a04c0e.el9.x86_64 27/198 2024-03-22T01:24:57.887 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : rbd-mirror-2:19.0.0-2344.gc3a04c0e.el9.x86_64 28/198 2024-03-22T01:24:57.887 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : rbd-nbd-2:19.0.0-2344.gc3a04c0e.el9.x86_64 29/198 2024-03-22T01:24:57.887 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2344.gc3a04c0e. 30/198 2024-03-22T01:24:57.887 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2344.gc3a04c0e.el9.noa 31/198 2024-03-22T01:24:57.887 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2344.gc3a04c0e.el9.n 32/198 2024-03-22T01:24:57.902 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2344.gc3a 33/198 2024-03-22T01:24:57.902 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2344.gc3a04c0e.el 34/198 2024-03-22T01:24:57.902 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2344.gc3a04c0e.el9.noarch 35/198 2024-03-22T01:24:57.902 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2344.gc3a04c0e.e 36/198 2024-03-22T01:24:57.903 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-volume-2:19.0.0-2344.gc3a04c0e.el9.noarch 37/198 2024-03-22T01:24:57.903 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : cephadm-2:19.0.0-2344.gc3a04c0e.el9.noarch 38/198 2024-03-22T01:24:57.903 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : cephfs-top-2:19.0.0-2344.gc3a04c0e.el9.noarch 39/198 2024-03-22T01:24:57.903 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-22T01:24:57.903 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-22T01:24:57.903 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-22T01:24:57.903 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-22T01:24:57.903 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-22T01:24:57.903 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-22T01:24:57.903 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-22T01:24:57.903 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-22T01:24:57.903 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-22T01:24:57.903 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-22T01:24:57.904 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-22T01:24:57.904 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-22T01:24:57.904 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-22T01:24:57.904 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-22T01:24:57.904 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-22T01:24:57.904 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-22T01:24:57.904 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-22T01:24:57.904 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-22T01:24:57.904 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-22T01:24:57.904 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-22T01:24:57.905 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-22T01:24:57.905 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-22T01:24:57.905 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-22T01:24:57.905 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-22T01:24:57.905 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-22T01:24:57.905 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-22T01:24:57.905 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-22T01:24:57.905 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-22T01:24:57.905 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-22T01:24:57.905 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-22T01:24:57.905 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-22T01:24:57.905 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-22T01:24:57.905 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-22T01:24:57.906 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-22T01:24:57.906 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-22T01:24:57.906 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-22T01:24:57.906 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-22T01:24:57.906 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-22T01:24:57.906 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-22T01:24:57.906 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-22T01:24:57.906 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-22T01:24:57.906 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-22T01:24:57.906 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-22T01:24:57.906 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-22T01:24:57.906 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-22T01:24:57.906 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-22T01:24:57.906 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-22T01:24:57.907 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-22T01:24:57.907 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-22T01:24:57.907 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-22T01:24:57.907 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-22T01:24:57.907 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-22T01:24:57.907 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-22T01:24:57.907 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-22T01:24:57.907 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-22T01:24:57.907 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-22T01:24:57.908 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-22T01:24:57.908 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-22T01:24:57.908 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-22T01:24:57.908 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-22T01:24:57.908 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-22T01:24:57.908 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-22T01:24:57.908 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-22T01:24:57.908 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-22T01:24:57.908 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-22T01:24:57.908 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-22T01:24:57.908 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-22T01:24:57.908 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-22T01:24:57.908 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-22T01:24:57.908 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-22T01:24:57.909 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-22T01:24:57.909 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-22T01:24:57.909 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-22T01:24:57.909 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-22T01:24:57.909 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-22T01:24:57.909 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-22T01:24:57.909 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-22T01:24:57.909 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-22T01:24:57.909 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-22T01:24:57.909 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-22T01:24:57.909 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-22T01:24:57.909 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-22T01:24:57.909 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-22T01:24:57.909 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-22T01:24:57.910 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-22T01:24:57.910 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-22T01:24:57.910 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-22T01:24:57.910 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-22T01:24:57.910 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-22T01:24:57.910 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-22T01:24:57.910 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-22T01:24:57.910 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-22T01:24:57.910 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-22T01:24:57.910 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-22T01:24:57.910 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-22T01:24:57.910 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-22T01:24:57.910 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-22T01:24:57.910 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-22T01:24:57.911 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-22T01:24:57.911 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-22T01:24:57.911 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-22T01:24:57.911 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-22T01:24:57.911 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-22T01:24:57.911 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-22T01:24:57.911 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-22T01:24:57.911 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-22T01:24:57.911 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-22T01:24:57.911 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-22T01:24:57.911 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-22T01:24:57.911 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-22T01:24:57.911 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-22T01:24:57.911 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librados2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 151/198 2024-03-22T01:24:57.912 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-22T01:24:57.912 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librbd1-2:19.0.0-2344.gc3a04c0e.el9.x86_64 153/198 2024-03-22T01:24:57.912 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-22T01:24:57.912 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-22T01:24:57.912 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-22T01:24:57.912 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-22T01:24:57.912 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-22T01:24:57.912 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-22T01:24:57.912 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-22T01:24:57.912 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-22T01:24:57.912 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-22T01:24:57.912 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-22T01:24:57.912 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-22T01:24:57.912 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-22T01:24:57.912 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-22T01:24:57.913 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-22T01:24:57.913 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-22T01:24:57.913 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-22T01:24:57.913 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-22T01:24:57.913 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-22T01:24:57.913 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-22T01:24:57.913 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-22T01:24:57.913 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-22T01:24:57.913 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-22T01:24:57.913 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-22T01:24:57.913 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-22T01:24:57.913 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-22T01:24:57.913 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-22T01:24:57.913 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-22T01:24:57.914 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-22T01:24:57.914 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-22T01:24:57.914 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-22T01:24:57.914 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-22T01:24:57.914 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-22T01:24:57.914 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-22T01:24:57.914 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-22T01:24:57.914 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-22T01:24:57.914 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-22T01:24:57.914 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-22T01:24:57.914 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-22T01:24:57.914 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-22T01:24:57.914 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-22T01:24:57.914 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-22T01:24:57.915 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-22T01:24:57.915 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-22T01:24:57.915 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-22T01:24:59.371 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-22T01:24:59.371 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:24:59.371 INFO:teuthology.orchestra.run.smithi072.stdout:Upgraded: 2024-03-22T01:24:59.372 INFO:teuthology.orchestra.run.smithi072.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-22T01:24:59.372 INFO:teuthology.orchestra.run.smithi072.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-22T01:24:59.372 INFO:teuthology.orchestra.run.smithi072.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-22T01:24:59.372 INFO:teuthology.orchestra.run.smithi072.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-22T01:24:59.372 INFO:teuthology.orchestra.run.smithi072.stdout: acl-2.3.1-4.el9.x86_64 2024-03-22T01:24:59.372 INFO:teuthology.orchestra.run.smithi072.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-22T01:24:59.372 INFO:teuthology.orchestra.run.smithi072.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-22T01:24:59.372 INFO:teuthology.orchestra.run.smithi072.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-22T01:24:59.372 INFO:teuthology.orchestra.run.smithi072.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-22T01:24:59.372 INFO:teuthology.orchestra.run.smithi072.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-22T01:24:59.372 INFO:teuthology.orchestra.run.smithi072.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-22T01:24:59.372 INFO:teuthology.orchestra.run.smithi072.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-22T01:24:59.372 INFO:teuthology.orchestra.run.smithi072.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-22T01:24:59.373 INFO:teuthology.orchestra.run.smithi072.stdout: librados2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:24:59.373 INFO:teuthology.orchestra.run.smithi072.stdout: librbd1-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:24:59.373 INFO:teuthology.orchestra.run.smithi072.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-22T01:24:59.373 INFO:teuthology.orchestra.run.smithi072.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-22T01:24:59.373 INFO:teuthology.orchestra.run.smithi072.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-22T01:24:59.373 INFO:teuthology.orchestra.run.smithi072.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-22T01:24:59.373 INFO:teuthology.orchestra.run.smithi072.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-22T01:24:59.374 INFO:teuthology.orchestra.run.smithi072.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-22T01:24:59.374 INFO:teuthology.orchestra.run.smithi072.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-22T01:24:59.374 INFO:teuthology.orchestra.run.smithi072.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-22T01:24:59.374 INFO:teuthology.orchestra.run.smithi072.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-22T01:24:59.374 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2024-03-22T01:24:59.374 INFO:teuthology.orchestra.run.smithi072.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-22T01:24:59.375 INFO:teuthology.orchestra.run.smithi072.stdout: automake-1.16.2-8.el9.noarch 2024-03-22T01:24:59.375 INFO:teuthology.orchestra.run.smithi072.stdout: bison-3.7.4-5.el9.x86_64 2024-03-22T01:24:59.375 INFO:teuthology.orchestra.run.smithi072.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-22T01:24:59.375 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:24:59.375 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-base-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:24:59.375 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-common-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:24:59.376 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-fuse-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:24:59.376 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-grafana-dashboards-2:19.0.0-2344.gc3a04c0e.el9.noarch 2024-03-22T01:24:59.376 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-immutable-object-cache-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:24:59.376 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mds-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:24:59.376 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:24:59.376 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-cephadm-2:19.0.0-2344.gc3a04c0e.el9.noarch 2024-03-22T01:24:59.377 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-dashboard-2:19.0.0-2344.gc3a04c0e.el9.noarch 2024-03-22T01:24:59.377 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2344.gc3a04c0e.el9.noarch 2024-03-22T01:24:59.377 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-modules-core-2:19.0.0-2344.gc3a04c0e.el9.noarch 2024-03-22T01:24:59.377 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-rook-2:19.0.0-2344.gc3a04c0e.el9.noarch 2024-03-22T01:24:59.377 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mon-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:24:59.377 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-osd-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:24:59.377 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-prometheus-alerts-2:19.0.0-2344.gc3a04c0e.el9.noarch 2024-03-22T01:24:59.378 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-radosgw-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:24:59.378 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-selinux-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:24:59.378 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-test-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:24:59.378 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-volume-2:19.0.0-2344.gc3a04c0e.el9.noarch 2024-03-22T01:24:59.378 INFO:teuthology.orchestra.run.smithi072.stdout: cephadm-2:19.0.0-2344.gc3a04c0e.el9.noarch 2024-03-22T01:24:59.378 INFO:teuthology.orchestra.run.smithi072.stdout: cephfs-mirror-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:24:59.379 INFO:teuthology.orchestra.run.smithi072.stdout: cephfs-shell-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:24:59.379 INFO:teuthology.orchestra.run.smithi072.stdout: cephfs-top-2:19.0.0-2344.gc3a04c0e.el9.noarch 2024-03-22T01:24:59.379 INFO:teuthology.orchestra.run.smithi072.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-22T01:24:59.379 INFO:teuthology.orchestra.run.smithi072.stdout: flex-2.6.4-9.el9.x86_64 2024-03-22T01:24:59.379 INFO:teuthology.orchestra.run.smithi072.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-22T01:24:59.379 INFO:teuthology.orchestra.run.smithi072.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-22T01:24:59.379 INFO:teuthology.orchestra.run.smithi072.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-22T01:24:59.379 INFO:teuthology.orchestra.run.smithi072.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-22T01:24:59.380 INFO:teuthology.orchestra.run.smithi072.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-22T01:24:59.380 INFO:teuthology.orchestra.run.smithi072.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-22T01:24:59.380 INFO:teuthology.orchestra.run.smithi072.stdout: jq-1.6-16.el9.x86_64 2024-03-22T01:24:59.380 INFO:teuthology.orchestra.run.smithi072.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-22T01:24:59.380 INFO:teuthology.orchestra.run.smithi072.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-22T01:24:59.380 INFO:teuthology.orchestra.run.smithi072.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-22T01:24:59.381 INFO:teuthology.orchestra.run.smithi072.stdout: libcephfs-devel-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:24:59.381 INFO:teuthology.orchestra.run.smithi072.stdout: libcephfs2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:24:59.381 INFO:teuthology.orchestra.run.smithi072.stdout: libcephsqlite-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:24:59.381 INFO:teuthology.orchestra.run.smithi072.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-22T01:24:59.381 INFO:teuthology.orchestra.run.smithi072.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-22T01:24:59.381 INFO:teuthology.orchestra.run.smithi072.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-22T01:24:59.382 INFO:teuthology.orchestra.run.smithi072.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-22T01:24:59.382 INFO:teuthology.orchestra.run.smithi072.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-22T01:24:59.382 INFO:teuthology.orchestra.run.smithi072.stdout: librados-devel-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:24:59.382 INFO:teuthology.orchestra.run.smithi072.stdout: libradosstriper1-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:24:59.382 INFO:teuthology.orchestra.run.smithi072.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-22T01:24:59.382 INFO:teuthology.orchestra.run.smithi072.stdout: librgw2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:24:59.383 INFO:teuthology.orchestra.run.smithi072.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-22T01:24:59.383 INFO:teuthology.orchestra.run.smithi072.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-22T01:24:59.383 INFO:teuthology.orchestra.run.smithi072.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-22T01:24:59.383 INFO:teuthology.orchestra.run.smithi072.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-22T01:24:59.383 INFO:teuthology.orchestra.run.smithi072.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-22T01:24:59.383 INFO:teuthology.orchestra.run.smithi072.stdout: lua-5.4.4-4.el9.x86_64 2024-03-22T01:24:59.383 INFO:teuthology.orchestra.run.smithi072.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-22T01:24:59.383 INFO:teuthology.orchestra.run.smithi072.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-22T01:24:59.383 INFO:teuthology.orchestra.run.smithi072.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-22T01:24:59.383 INFO:teuthology.orchestra.run.smithi072.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-22T01:24:59.383 INFO:teuthology.orchestra.run.smithi072.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-22T01:24:59.383 INFO:teuthology.orchestra.run.smithi072.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-22T01:24:59.384 INFO:teuthology.orchestra.run.smithi072.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-22T01:24:59.384 INFO:teuthology.orchestra.run.smithi072.stdout: postgresql-13.11-1.el9.x86_64 2024-03-22T01:24:59.384 INFO:teuthology.orchestra.run.smithi072.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-22T01:24:59.384 INFO:teuthology.orchestra.run.smithi072.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-22T01:24:59.384 INFO:teuthology.orchestra.run.smithi072.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-22T01:24:59.384 INFO:teuthology.orchestra.run.smithi072.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-22T01:24:59.384 INFO:teuthology.orchestra.run.smithi072.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-22T01:24:59.384 INFO:teuthology.orchestra.run.smithi072.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-22T01:24:59.384 INFO:teuthology.orchestra.run.smithi072.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-22T01:24:59.384 INFO:teuthology.orchestra.run.smithi072.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-22T01:24:59.384 INFO:teuthology.orchestra.run.smithi072.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-22T01:24:59.384 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-22T01:24:59.384 INFO:teuthology.orchestra.run.smithi072.stdout: python3-ceph-argparse-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:24:59.385 INFO:teuthology.orchestra.run.smithi072.stdout: python3-ceph-common-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:24:59.385 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cephfs-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:24:59.385 INFO:teuthology.orchestra.run.smithi072.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-22T01:24:59.385 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-22T01:24:59.385 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-22T01:24:59.385 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-22T01:24:59.385 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-22T01:24:59.385 INFO:teuthology.orchestra.run.smithi072.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-22T01:24:59.385 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-22T01:24:59.385 INFO:teuthology.orchestra.run.smithi072.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-22T01:24:59.385 INFO:teuthology.orchestra.run.smithi072.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-22T01:24:59.385 INFO:teuthology.orchestra.run.smithi072.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-22T01:24:59.385 INFO:teuthology.orchestra.run.smithi072.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-22T01:24:59.385 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-22T01:24:59.385 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-22T01:24:59.386 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-22T01:24:59.386 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-22T01:24:59.386 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-22T01:24:59.386 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-22T01:24:59.386 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-22T01:24:59.386 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-22T01:24:59.386 INFO:teuthology.orchestra.run.smithi072.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-22T01:24:59.386 INFO:teuthology.orchestra.run.smithi072.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-22T01:24:59.386 INFO:teuthology.orchestra.run.smithi072.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-22T01:24:59.386 INFO:teuthology.orchestra.run.smithi072.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-22T01:24:59.386 INFO:teuthology.orchestra.run.smithi072.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-22T01:24:59.386 INFO:teuthology.orchestra.run.smithi072.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-22T01:24:59.386 INFO:teuthology.orchestra.run.smithi072.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-22T01:24:59.387 INFO:teuthology.orchestra.run.smithi072.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-22T01:24:59.387 INFO:teuthology.orchestra.run.smithi072.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-22T01:24:59.387 INFO:teuthology.orchestra.run.smithi072.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-22T01:24:59.387 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-22T01:24:59.387 INFO:teuthology.orchestra.run.smithi072.stdout: python3-ply-3.11-14.el9.noarch 2024-03-22T01:24:59.387 INFO:teuthology.orchestra.run.smithi072.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-22T01:24:59.387 INFO:teuthology.orchestra.run.smithi072.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-22T01:24:59.387 INFO:teuthology.orchestra.run.smithi072.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-22T01:24:59.387 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-22T01:24:59.387 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-22T01:24:59.387 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-22T01:24:59.387 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-22T01:24:59.387 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-22T01:24:59.387 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-22T01:24:59.388 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rados-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:24:59.388 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rbd-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:24:59.388 INFO:teuthology.orchestra.run.smithi072.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-22T01:24:59.388 INFO:teuthology.orchestra.run.smithi072.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-22T01:24:59.388 INFO:teuthology.orchestra.run.smithi072.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-22T01:24:59.388 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rgw-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:24:59.388 INFO:teuthology.orchestra.run.smithi072.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-22T01:24:59.388 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-22T01:24:59.388 INFO:teuthology.orchestra.run.smithi072.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-22T01:24:59.388 INFO:teuthology.orchestra.run.smithi072.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-22T01:24:59.388 INFO:teuthology.orchestra.run.smithi072.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-22T01:24:59.388 INFO:teuthology.orchestra.run.smithi072.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-22T01:24:59.388 INFO:teuthology.orchestra.run.smithi072.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-22T01:24:59.389 INFO:teuthology.orchestra.run.smithi072.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-22T01:24:59.389 INFO:teuthology.orchestra.run.smithi072.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-22T01:24:59.389 INFO:teuthology.orchestra.run.smithi072.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-22T01:24:59.389 INFO:teuthology.orchestra.run.smithi072.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-22T01:24:59.389 INFO:teuthology.orchestra.run.smithi072.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-22T01:24:59.389 INFO:teuthology.orchestra.run.smithi072.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-22T01:24:59.389 INFO:teuthology.orchestra.run.smithi072.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-22T01:24:59.389 INFO:teuthology.orchestra.run.smithi072.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-22T01:24:59.389 INFO:teuthology.orchestra.run.smithi072.stdout: rbd-fuse-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:24:59.389 INFO:teuthology.orchestra.run.smithi072.stdout: rbd-mirror-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:24:59.390 INFO:teuthology.orchestra.run.smithi072.stdout: rbd-nbd-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:24:59.390 INFO:teuthology.orchestra.run.smithi072.stdout: re2-1:20211101-3.el9.x86_64 2024-03-22T01:24:59.390 INFO:teuthology.orchestra.run.smithi072.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-22T01:24:59.390 INFO:teuthology.orchestra.run.smithi072.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-22T01:24:59.390 INFO:teuthology.orchestra.run.smithi072.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-22T01:24:59.390 INFO:teuthology.orchestra.run.smithi072.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-22T01:24:59.390 INFO:teuthology.orchestra.run.smithi072.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-22T01:24:59.391 INFO:teuthology.orchestra.run.smithi072.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-22T01:24:59.391 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:24:59.391 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-03-22T01:25:00.004 DEBUG:teuthology.parallel:result is None 2024-03-22T01:25:01.072 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-22T01:25:01.073 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-22T01:25:01.073 INFO:teuthology.orchestra.run.smithi089.stdout:Upgraded: 2024-03-22T01:25:01.073 INFO:teuthology.orchestra.run.smithi089.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-22T01:25:01.073 INFO:teuthology.orchestra.run.smithi089.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-22T01:25:01.073 INFO:teuthology.orchestra.run.smithi089.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-22T01:25:01.073 INFO:teuthology.orchestra.run.smithi089.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-22T01:25:01.073 INFO:teuthology.orchestra.run.smithi089.stdout: acl-2.3.1-4.el9.x86_64 2024-03-22T01:25:01.073 INFO:teuthology.orchestra.run.smithi089.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-22T01:25:01.073 INFO:teuthology.orchestra.run.smithi089.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-22T01:25:01.073 INFO:teuthology.orchestra.run.smithi089.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-22T01:25:01.073 INFO:teuthology.orchestra.run.smithi089.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-22T01:25:01.074 INFO:teuthology.orchestra.run.smithi089.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-22T01:25:01.074 INFO:teuthology.orchestra.run.smithi089.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-22T01:25:01.074 INFO:teuthology.orchestra.run.smithi089.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-22T01:25:01.074 INFO:teuthology.orchestra.run.smithi089.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-22T01:25:01.074 INFO:teuthology.orchestra.run.smithi089.stdout: librados2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:25:01.074 INFO:teuthology.orchestra.run.smithi089.stdout: librbd1-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:25:01.074 INFO:teuthology.orchestra.run.smithi089.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-22T01:25:01.074 INFO:teuthology.orchestra.run.smithi089.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-22T01:25:01.074 INFO:teuthology.orchestra.run.smithi089.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-22T01:25:01.074 INFO:teuthology.orchestra.run.smithi089.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-22T01:25:01.074 INFO:teuthology.orchestra.run.smithi089.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-22T01:25:01.074 INFO:teuthology.orchestra.run.smithi089.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-22T01:25:01.074 INFO:teuthology.orchestra.run.smithi089.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-22T01:25:01.075 INFO:teuthology.orchestra.run.smithi089.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-22T01:25:01.075 INFO:teuthology.orchestra.run.smithi089.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-22T01:25:01.075 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-03-22T01:25:01.075 INFO:teuthology.orchestra.run.smithi089.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-22T01:25:01.075 INFO:teuthology.orchestra.run.smithi089.stdout: automake-1.16.2-8.el9.noarch 2024-03-22T01:25:01.075 INFO:teuthology.orchestra.run.smithi089.stdout: bison-3.7.4-5.el9.x86_64 2024-03-22T01:25:01.075 INFO:teuthology.orchestra.run.smithi089.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-22T01:25:01.075 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:25:01.075 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-base-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:25:01.075 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-common-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:25:01.075 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-fuse-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:25:01.075 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-grafana-dashboards-2:19.0.0-2344.gc3a04c0e.el9.noarch 2024-03-22T01:25:01.075 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-immutable-object-cache-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:25:01.076 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mds-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:25:01.076 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:25:01.076 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-cephadm-2:19.0.0-2344.gc3a04c0e.el9.noarch 2024-03-22T01:25:01.076 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-dashboard-2:19.0.0-2344.gc3a04c0e.el9.noarch 2024-03-22T01:25:01.076 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2344.gc3a04c0e.el9.noarch 2024-03-22T01:25:01.076 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-modules-core-2:19.0.0-2344.gc3a04c0e.el9.noarch 2024-03-22T01:25:01.076 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-rook-2:19.0.0-2344.gc3a04c0e.el9.noarch 2024-03-22T01:25:01.076 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mon-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:25:01.076 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-osd-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:25:01.076 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-prometheus-alerts-2:19.0.0-2344.gc3a04c0e.el9.noarch 2024-03-22T01:25:01.076 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-radosgw-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:25:01.076 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-selinux-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:25:01.076 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-test-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:25:01.077 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-volume-2:19.0.0-2344.gc3a04c0e.el9.noarch 2024-03-22T01:25:01.077 INFO:teuthology.orchestra.run.smithi089.stdout: cephadm-2:19.0.0-2344.gc3a04c0e.el9.noarch 2024-03-22T01:25:01.077 INFO:teuthology.orchestra.run.smithi089.stdout: cephfs-mirror-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:25:01.077 INFO:teuthology.orchestra.run.smithi089.stdout: cephfs-shell-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:25:01.077 INFO:teuthology.orchestra.run.smithi089.stdout: cephfs-top-2:19.0.0-2344.gc3a04c0e.el9.noarch 2024-03-22T01:25:01.077 INFO:teuthology.orchestra.run.smithi089.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-22T01:25:01.077 INFO:teuthology.orchestra.run.smithi089.stdout: flex-2.6.4-9.el9.x86_64 2024-03-22T01:25:01.077 INFO:teuthology.orchestra.run.smithi089.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-22T01:25:01.077 INFO:teuthology.orchestra.run.smithi089.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-22T01:25:01.077 INFO:teuthology.orchestra.run.smithi089.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-22T01:25:01.077 INFO:teuthology.orchestra.run.smithi089.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-22T01:25:01.077 INFO:teuthology.orchestra.run.smithi089.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-22T01:25:01.078 INFO:teuthology.orchestra.run.smithi089.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-22T01:25:01.078 INFO:teuthology.orchestra.run.smithi089.stdout: jq-1.6-16.el9.x86_64 2024-03-22T01:25:01.078 INFO:teuthology.orchestra.run.smithi089.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-22T01:25:01.078 INFO:teuthology.orchestra.run.smithi089.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-22T01:25:01.078 INFO:teuthology.orchestra.run.smithi089.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-22T01:25:01.078 INFO:teuthology.orchestra.run.smithi089.stdout: libcephfs-devel-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:25:01.078 INFO:teuthology.orchestra.run.smithi089.stdout: libcephfs2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:25:01.078 INFO:teuthology.orchestra.run.smithi089.stdout: libcephsqlite-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:25:01.078 INFO:teuthology.orchestra.run.smithi089.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-22T01:25:01.078 INFO:teuthology.orchestra.run.smithi089.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-22T01:25:01.078 INFO:teuthology.orchestra.run.smithi089.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-22T01:25:01.078 INFO:teuthology.orchestra.run.smithi089.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-22T01:25:01.078 INFO:teuthology.orchestra.run.smithi089.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-22T01:25:01.079 INFO:teuthology.orchestra.run.smithi089.stdout: librados-devel-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:25:01.079 INFO:teuthology.orchestra.run.smithi089.stdout: libradosstriper1-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:25:01.079 INFO:teuthology.orchestra.run.smithi089.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-22T01:25:01.079 INFO:teuthology.orchestra.run.smithi089.stdout: librgw2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:25:01.079 INFO:teuthology.orchestra.run.smithi089.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-22T01:25:01.079 INFO:teuthology.orchestra.run.smithi089.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-22T01:25:01.079 INFO:teuthology.orchestra.run.smithi089.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-22T01:25:01.079 INFO:teuthology.orchestra.run.smithi089.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-22T01:25:01.079 INFO:teuthology.orchestra.run.smithi089.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-22T01:25:01.079 INFO:teuthology.orchestra.run.smithi089.stdout: lua-5.4.4-4.el9.x86_64 2024-03-22T01:25:01.079 INFO:teuthology.orchestra.run.smithi089.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-22T01:25:01.079 INFO:teuthology.orchestra.run.smithi089.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-22T01:25:01.080 INFO:teuthology.orchestra.run.smithi089.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-22T01:25:01.080 INFO:teuthology.orchestra.run.smithi089.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-22T01:25:01.080 INFO:teuthology.orchestra.run.smithi089.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-22T01:25:01.080 INFO:teuthology.orchestra.run.smithi089.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-22T01:25:01.080 INFO:teuthology.orchestra.run.smithi089.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-22T01:25:01.080 INFO:teuthology.orchestra.run.smithi089.stdout: postgresql-13.11-1.el9.x86_64 2024-03-22T01:25:01.080 INFO:teuthology.orchestra.run.smithi089.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-22T01:25:01.080 INFO:teuthology.orchestra.run.smithi089.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-22T01:25:01.080 INFO:teuthology.orchestra.run.smithi089.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-22T01:25:01.080 INFO:teuthology.orchestra.run.smithi089.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-22T01:25:01.081 INFO:teuthology.orchestra.run.smithi089.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-22T01:25:01.081 INFO:teuthology.orchestra.run.smithi089.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-22T01:25:01.081 INFO:teuthology.orchestra.run.smithi089.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-22T01:25:01.081 INFO:teuthology.orchestra.run.smithi089.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-22T01:25:01.081 INFO:teuthology.orchestra.run.smithi089.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-22T01:25:01.081 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-22T01:25:01.081 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ceph-argparse-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:25:01.081 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ceph-common-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:25:01.081 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cephfs-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:25:01.081 INFO:teuthology.orchestra.run.smithi089.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-22T01:25:01.081 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-22T01:25:01.081 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-22T01:25:01.082 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-22T01:25:01.082 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-22T01:25:01.082 INFO:teuthology.orchestra.run.smithi089.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-22T01:25:01.082 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-22T01:25:01.082 INFO:teuthology.orchestra.run.smithi089.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-22T01:25:01.082 INFO:teuthology.orchestra.run.smithi089.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-22T01:25:01.082 INFO:teuthology.orchestra.run.smithi089.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-22T01:25:01.082 INFO:teuthology.orchestra.run.smithi089.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-22T01:25:01.082 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-22T01:25:01.082 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-22T01:25:01.082 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-22T01:25:01.082 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-22T01:25:01.083 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-22T01:25:01.083 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-22T01:25:01.083 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-22T01:25:01.083 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-22T01:25:01.083 INFO:teuthology.orchestra.run.smithi089.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-22T01:25:01.083 INFO:teuthology.orchestra.run.smithi089.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-22T01:25:01.083 INFO:teuthology.orchestra.run.smithi089.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-22T01:25:01.083 INFO:teuthology.orchestra.run.smithi089.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-22T01:25:01.083 INFO:teuthology.orchestra.run.smithi089.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-22T01:25:01.083 INFO:teuthology.orchestra.run.smithi089.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-22T01:25:01.083 INFO:teuthology.orchestra.run.smithi089.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-22T01:25:01.083 INFO:teuthology.orchestra.run.smithi089.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-22T01:25:01.085 INFO:teuthology.orchestra.run.smithi089.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-22T01:25:01.085 INFO:teuthology.orchestra.run.smithi089.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-22T01:25:01.085 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-22T01:25:01.085 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ply-3.11-14.el9.noarch 2024-03-22T01:25:01.085 INFO:teuthology.orchestra.run.smithi089.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-22T01:25:01.085 INFO:teuthology.orchestra.run.smithi089.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-22T01:25:01.085 INFO:teuthology.orchestra.run.smithi089.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-22T01:25:01.085 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-22T01:25:01.085 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-22T01:25:01.085 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-22T01:25:01.085 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-22T01:25:01.086 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-22T01:25:01.086 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-22T01:25:01.086 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rados-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:25:01.086 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rbd-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:25:01.086 INFO:teuthology.orchestra.run.smithi089.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-22T01:25:01.086 INFO:teuthology.orchestra.run.smithi089.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-22T01:25:01.086 INFO:teuthology.orchestra.run.smithi089.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-22T01:25:01.086 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rgw-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:25:01.086 INFO:teuthology.orchestra.run.smithi089.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-22T01:25:01.086 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-22T01:25:01.086 INFO:teuthology.orchestra.run.smithi089.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-22T01:25:01.086 INFO:teuthology.orchestra.run.smithi089.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-22T01:25:01.087 INFO:teuthology.orchestra.run.smithi089.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-22T01:25:01.087 INFO:teuthology.orchestra.run.smithi089.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-22T01:25:01.087 INFO:teuthology.orchestra.run.smithi089.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-22T01:25:01.087 INFO:teuthology.orchestra.run.smithi089.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-22T01:25:01.087 INFO:teuthology.orchestra.run.smithi089.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-22T01:25:01.087 INFO:teuthology.orchestra.run.smithi089.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-22T01:25:01.087 INFO:teuthology.orchestra.run.smithi089.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-22T01:25:01.087 INFO:teuthology.orchestra.run.smithi089.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-22T01:25:01.087 INFO:teuthology.orchestra.run.smithi089.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-22T01:25:01.087 INFO:teuthology.orchestra.run.smithi089.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-22T01:25:01.087 INFO:teuthology.orchestra.run.smithi089.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-22T01:25:01.087 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-fuse-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:25:01.088 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-mirror-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:25:01.088 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-nbd-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T01:25:01.088 INFO:teuthology.orchestra.run.smithi089.stdout: re2-1:20211101-3.el9.x86_64 2024-03-22T01:25:01.088 INFO:teuthology.orchestra.run.smithi089.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-22T01:25:01.088 INFO:teuthology.orchestra.run.smithi089.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-22T01:25:01.088 INFO:teuthology.orchestra.run.smithi089.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-22T01:25:01.088 INFO:teuthology.orchestra.run.smithi089.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-22T01:25:01.088 INFO:teuthology.orchestra.run.smithi089.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-22T01:25:01.089 INFO:teuthology.orchestra.run.smithi089.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-22T01:25:01.089 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-22T01:25:01.089 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-03-22T01:25:01.618 DEBUG:teuthology.parallel:result is None 2024-03-22T01:25:01.619 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e 2024-03-22T01:25:01.842 DEBUG:teuthology.orchestra.run.smithi072:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-22T01:25:01.868 INFO:teuthology.orchestra.run.smithi072.stdout:19.0.0-2344.gc3a04c0e.el9 2024-03-22T01:25:01.869 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2344.gc3a04c0e.el9 2024-03-22T01:25:01.869 INFO:teuthology.task.install:The correct ceph version 19.0.0-2344.gc3a04c0e is installed. 2024-03-22T01:25:01.871 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e 2024-03-22T01:25:02.090 DEBUG:teuthology.orchestra.run.smithi089:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-22T01:25:02.116 INFO:teuthology.orchestra.run.smithi089.stdout:19.0.0-2344.gc3a04c0e.el9 2024-03-22T01:25:02.117 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2344.gc3a04c0e.el9 2024-03-22T01:25:02.117 INFO:teuthology.task.install:The correct ceph version 19.0.0-2344.gc3a04c0e is installed. 2024-03-22T01:25:02.120 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e 2024-03-22T01:25:02.342 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-22T01:25:02.367 INFO:teuthology.orchestra.run.smithi098.stdout:19.0.0-2344.gc3a04c0e.el9 2024-03-22T01:25:02.367 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2344.gc3a04c0e.el9 2024-03-22T01:25:02.368 INFO:teuthology.task.install:The correct ceph version 19.0.0-2344.gc3a04c0e is installed. 2024-03-22T01:25:02.370 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-22T01:25:02.370 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-22T01:25:02.370 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-22T01:25:02.408 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-22T01:25:02.408 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-22T01:25:02.442 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-22T01:25:02.443 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-22T01:25:02.476 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-22T01:25:02.476 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-22T01:25:02.477 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/usr/bin/daemon-helper 2024-03-22T01:25:02.507 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-22T01:25:02.576 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-22T01:25:02.577 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/usr/bin/daemon-helper 2024-03-22T01:25:02.608 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-22T01:25:02.679 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-22T01:25:02.679 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/daemon-helper 2024-03-22T01:25:02.711 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-22T01:25:02.781 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-22T01:25:02.782 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-22T01:25:02.782 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-22T01:25:02.814 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-22T01:25:02.883 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-22T01:25:02.883 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-22T01:25:02.914 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-22T01:25:02.984 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-22T01:25:02.984 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-22T01:25:03.011 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-22T01:25:03.078 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-22T01:25:03.079 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-22T01:25:03.079 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/usr/bin/stdin-killer 2024-03-22T01:25:03.106 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-22T01:25:03.173 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-22T01:25:03.173 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/usr/bin/stdin-killer 2024-03-22T01:25:03.203 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-22T01:25:03.271 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-22T01:25:03.271 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/stdin-killer 2024-03-22T01:25:03.298 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-22T01:25:03.365 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-22T01:25:03.473 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 1, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'filestore flush min': 0, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd heartbeat grace': 60, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow metadata IO', 'SLOW_OPS', 'slow request', 'SLOW_OPS', 'slow request'], 'sha1': 'c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-03-22T01:25:03.473 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e 2024-03-22T01:25:03.474 INFO:tasks.cephadm:Cluster fsid is 022a99bc-e7eb-11ee-95cd-87774f69a715 2024-03-22T01:25:03.474 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-22T01:25:03.474 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.72', 'mon.b': '172.21.15.89', 'mon.c': '172.21.15.98'} 2024-03-22T01:25:03.474 INFO:tasks.cephadm:First mon is mon.a on smithi072 2024-03-22T01:25:03.474 INFO:tasks.cephadm:First mgr is x 2024-03-22T01:25:03.474 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-22T01:25:03.475 DEBUG:teuthology.orchestra.run.smithi072:> sudo hostname $(hostname -s) 2024-03-22T01:25:03.507 DEBUG:teuthology.orchestra.run.smithi089:> sudo hostname $(hostname -s) 2024-03-22T01:25:03.538 DEBUG:teuthology.orchestra.run.smithi098:> sudo hostname $(hostname -s) 2024-03-22T01:25:03.565 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-22T01:25:03.565 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e 2024-03-22T01:25:03.785 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77857/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2344-gc3a04c0e', 'node_name': '172.21.5.34+adami04', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2344.gc3a04c0e'}, 'url': 'https://2.chacra.ceph.com/r/ceph/wip-batrick-testing-20240321.172515/c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-21 21:00:27.602277', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-batrick-testing-20240321.172515', 'chacra_url': 'https://2.chacra.ceph.com/repos/ceph/wip-batrick-testing-20240321.172515/c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-03-22T01:25:04.001 INFO:tasks.util.chacra:got chacra host 2.chacra.ceph.com, ref wip-batrick-testing-20240321.172515, sha1 c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e 2024-03-22T01:25:04.002 INFO:tasks.cephadm:Discovered cachra url: https://2.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240321.172515/c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/centos/9/x86_64/flavors/default/cephadm 2024-03-22T01:25:04.003 DEBUG:teuthology.orchestra.run.smithi072:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240321.172515/c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-22T01:25:04.612 INFO:teuthology.orchestra.run.smithi072.stdout:-rw-r--r--. 1 ubuntu ubuntu 769354 Mar 22 01:25 /home/ubuntu/cephtest/cephadm 2024-03-22T01:25:04.612 DEBUG:teuthology.orchestra.run.smithi089:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240321.172515/c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-22T01:25:05.179 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r--. 1 ubuntu ubuntu 769354 Mar 22 01:25 /home/ubuntu/cephtest/cephadm 2024-03-22T01:25:05.179 DEBUG:teuthology.orchestra.run.smithi098:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240321.172515/c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-22T01:25:05.666 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r--. 1 ubuntu ubuntu 769354 Mar 22 01:25 /home/ubuntu/cephtest/cephadm 2024-03-22T01:25:05.667 DEBUG:teuthology.orchestra.run.smithi072:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-22T01:25:05.685 DEBUG:teuthology.orchestra.run.smithi089:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-22T01:25:05.704 DEBUG:teuthology.orchestra.run.smithi098:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-22T01:25:05.738 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e on all hosts... 2024-03-22T01:25:05.739 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e pull 2024-03-22T01:25:05.741 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e pull 2024-03-22T01:25:05.747 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e pull 2024-03-22T01:25:06.522 INFO:teuthology.orchestra.run.smithi098.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e... 2024-03-22T01:25:06.632 INFO:teuthology.orchestra.run.smithi089.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e... 2024-03-22T01:25:06.722 INFO:teuthology.orchestra.run.smithi072.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e... 2024-03-22T01:25:39.520 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-22T01:25:39.521 INFO:teuthology.orchestra.run.smithi072.stdout: "ceph_version": "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)", 2024-03-22T01:25:39.521 INFO:teuthology.orchestra.run.smithi072.stdout: "image_id": "9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0", 2024-03-22T01:25:39.521 INFO:teuthology.orchestra.run.smithi072.stdout: "repo_digests": [ 2024-03-22T01:25:39.521 INFO:teuthology.orchestra.run.smithi072.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0" 2024-03-22T01:25:39.521 INFO:teuthology.orchestra.run.smithi072.stdout: ] 2024-03-22T01:25:39.521 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-22T01:25:54.980 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-03-22T01:25:54.981 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph_version": "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)", 2024-03-22T01:25:54.981 INFO:teuthology.orchestra.run.smithi089.stdout: "image_id": "9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0", 2024-03-22T01:25:54.981 INFO:teuthology.orchestra.run.smithi089.stdout: "repo_digests": [ 2024-03-22T01:25:54.981 INFO:teuthology.orchestra.run.smithi089.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0" 2024-03-22T01:25:54.981 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-03-22T01:25:54.981 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-03-22T01:25:56.914 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-22T01:25:56.914 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_version": "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)", 2024-03-22T01:25:56.914 INFO:teuthology.orchestra.run.smithi098.stdout: "image_id": "9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0", 2024-03-22T01:25:56.914 INFO:teuthology.orchestra.run.smithi098.stdout: "repo_digests": [ 2024-03-22T01:25:56.914 INFO:teuthology.orchestra.run.smithi098.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0" 2024-03-22T01:25:56.914 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-03-22T01:25:56.914 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-22T01:25:56.943 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /etc/ceph 2024-03-22T01:25:56.978 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /etc/ceph 2024-03-22T01:25:57.013 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /etc/ceph 2024-03-22T01:25:57.043 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 777 /etc/ceph 2024-03-22T01:25:57.073 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 777 /etc/ceph 2024-03-22T01:25:57.105 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 777 /etc/ceph 2024-03-22T01:25:57.131 INFO:tasks.cephadm:Writing seed config... 2024-03-22T01:25:57.133 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-22T01:25:57.133 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-22T01:25:57.133 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-22T01:25:57.133 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-22T01:25:57.133 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-22T01:25:57.133 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-03-22T01:25:57.133 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-22T01:25:57.133 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-22T01:25:57.133 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-22T01:25:57.133 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-03-22T01:25:57.133 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-03-22T01:25:57.133 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-03-22T01:25:57.133 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-03-22T01:25:57.134 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-22T01:25:57.134 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-22T01:25:57.134 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-22T01:25:57.134 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-22T01:25:57.134 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-22T01:25:57.134 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-22T01:25:57.134 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-22T01:25:57.134 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-03-22T01:25:57.134 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-03-22T01:25:57.134 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-03-22T01:25:57.134 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-22T01:25:57.135 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-22T01:25:57.135 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-22T01:25:57.135 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-22T01:25:57.135 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-22T01:25:57.135 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-03-22T01:25:57.135 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-22T01:25:57.135 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-22T01:25:57.135 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-22T01:25:57.135 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-03-22T01:25:57.135 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-03-22T01:25:57.135 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-22T01:25:57.135 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-22T01:25:57.135 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-03-22T01:25:57.135 INFO:tasks.cephadm: override: [osd] filestore flush min = 0 2024-03-22T01:25:57.135 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-22T01:25:57.136 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-22T01:25:57.136 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-22T01:25:57.136 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-22T01:25:57.136 INFO:tasks.cephadm: override: [osd] osd heartbeat grace = 60 2024-03-22T01:25:57.136 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-22T01:25:57.136 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-22T01:25:57.136 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-22T01:25:57.136 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2024-03-22T01:25:57.137 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-22T01:25:57.137 DEBUG:teuthology.orchestra.run.smithi072:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-22T01:25:57.154 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 022a99bc-e7eb-11ee-95cd-87774f69a715 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 filestore flush min = 0 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd heartbeat grace = 60 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-22T01:25:57.154 DEBUG:teuthology.orchestra.run.smithi072:mon.a> sudo journalctl -f -n 0 -u ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mon.a.service 2024-03-22T01:25:57.196 DEBUG:teuthology.orchestra.run.smithi072:mgr.x> sudo journalctl -f -n 0 -u ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mgr.x.service 2024-03-22T01:25:57.239 INFO:tasks.cephadm:Bootstrapping... 2024-03-22T01:25:57.239 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e -v bootstrap --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.72 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-22T01:25:57.431 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-03-22T01:25:57.431 INFO:teuthology.orchestra.run.smithi072.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e', '-v', 'bootstrap', '--fsid', '022a99bc-e7eb-11ee-95cd-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.72', '--skip-admin-label'] 2024-03-22T01:25:57.431 INFO:teuthology.orchestra.run.smithi072.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-22T01:25:57.431 INFO:teuthology.orchestra.run.smithi072.stdout:Verifying podman|docker is present... 2024-03-22T01:25:57.457 INFO:teuthology.orchestra.run.smithi072.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-22T01:25:57.457 INFO:teuthology.orchestra.run.smithi072.stdout:Verifying lvm2 is present... 2024-03-22T01:25:57.457 INFO:teuthology.orchestra.run.smithi072.stdout:Verifying time synchronization is in place... 2024-03-22T01:25:57.464 INFO:teuthology.orchestra.run.smithi072.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-22T01:25:57.464 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-22T01:25:57.470 INFO:teuthology.orchestra.run.smithi072.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-22T01:25:57.470 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stdout inactive 2024-03-22T01:25:57.476 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stdout enabled 2024-03-22T01:25:57.482 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stdout active 2024-03-22T01:25:57.482 INFO:teuthology.orchestra.run.smithi072.stdout:Unit chronyd.service is enabled and running 2024-03-22T01:25:57.483 INFO:teuthology.orchestra.run.smithi072.stdout:Repeating the final host check... 2024-03-22T01:25:57.506 INFO:teuthology.orchestra.run.smithi072.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-22T01:25:57.506 INFO:teuthology.orchestra.run.smithi072.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-22T01:25:57.506 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl is present 2024-03-22T01:25:57.506 INFO:teuthology.orchestra.run.smithi072.stdout:lvcreate is present 2024-03-22T01:25:57.512 INFO:teuthology.orchestra.run.smithi072.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-22T01:25:57.512 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-22T01:25:57.518 INFO:teuthology.orchestra.run.smithi072.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-22T01:25:57.518 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stdout inactive 2024-03-22T01:25:57.524 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stdout enabled 2024-03-22T01:25:57.530 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stdout active 2024-03-22T01:25:57.530 INFO:teuthology.orchestra.run.smithi072.stdout:Unit chronyd.service is enabled and running 2024-03-22T01:25:57.530 INFO:teuthology.orchestra.run.smithi072.stdout:Host looks OK 2024-03-22T01:25:57.530 INFO:teuthology.orchestra.run.smithi072.stdout:Cluster fsid: 022a99bc-e7eb-11ee-95cd-87774f69a715 2024-03-22T01:25:57.530 INFO:teuthology.orchestra.run.smithi072.stdout:Acquiring lock 139749503890384 on /run/cephadm/022a99bc-e7eb-11ee-95cd-87774f69a715.lock 2024-03-22T01:25:57.530 INFO:teuthology.orchestra.run.smithi072.stdout:Lock 139749503890384 acquired on /run/cephadm/022a99bc-e7eb-11ee-95cd-87774f69a715.lock 2024-03-22T01:25:57.530 INFO:teuthology.orchestra.run.smithi072.stdout:Verifying IP 172.21.15.72 port 3300 ... 2024-03-22T01:25:57.531 INFO:teuthology.orchestra.run.smithi072.stdout:Verifying IP 172.21.15.72 port 6789 ... 2024-03-22T01:25:57.531 INFO:teuthology.orchestra.run.smithi072.stdout:Base mon IP(s) is [172.21.15.72:3300, 172.21.15.72:6789], mon addrv is [v2:172.21.15.72:3300,v1:172.21.15.72:6789] 2024-03-22T01:25:57.534 INFO:teuthology.orchestra.run.smithi072.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.72 metric 100 2024-03-22T01:25:57.534 INFO:teuthology.orchestra.run.smithi072.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.72 metric 100 2024-03-22T01:25:57.536 INFO:teuthology.orchestra.run.smithi072.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-22T01:25:57.536 INFO:teuthology.orchestra.run.smithi072.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-22T01:25:57.536 INFO:teuthology.orchestra.run.smithi072.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1792sec hoplimit 64 pref medium 2024-03-22T01:25:57.539 INFO:teuthology.orchestra.run.smithi072.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-22T01:25:57.539 INFO:teuthology.orchestra.run.smithi072.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-22T01:25:57.539 INFO:teuthology.orchestra.run.smithi072.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-22T01:25:57.539 INFO:teuthology.orchestra.run.smithi072.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-22T01:25:57.539 INFO:teuthology.orchestra.run.smithi072.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:b765/64 scope link 2024-03-22T01:25:57.539 INFO:teuthology.orchestra.run.smithi072.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-22T01:25:57.540 INFO:teuthology.orchestra.run.smithi072.stdout:Mon IP `172.21.15.72` is in CIDR network `172.21.0.0/20` 2024-03-22T01:25:57.540 INFO:teuthology.orchestra.run.smithi072.stdout:Mon IP `172.21.15.72` is in CIDR network `172.21.0.0/20` 2024-03-22T01:25:57.540 INFO:teuthology.orchestra.run.smithi072.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-22T01:25:57.540 INFO:teuthology.orchestra.run.smithi072.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-22T01:25:57.541 INFO:teuthology.orchestra.run.smithi072.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e... 2024-03-22T01:25:57.990 INFO:teuthology.orchestra.run.smithi072.stdout:/bin/podman: stdout 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 2024-03-22T01:25:57.991 INFO:teuthology.orchestra.run.smithi072.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e... 2024-03-22T01:25:57.991 INFO:teuthology.orchestra.run.smithi072.stdout:/bin/podman: stderr Getting image source signatures 2024-03-22T01:25:57.991 INFO:teuthology.orchestra.run.smithi072.stdout:/bin/podman: stderr Copying blob sha256:05236af4953a5298b5dfb26dfe4c3e6900902214dbe03ae2042572624f92ee73 2024-03-22T01:25:57.991 INFO:teuthology.orchestra.run.smithi072.stdout:/bin/podman: stderr Copying blob sha256:9516588f908cdb14b5e4062a6063db2d9291b355c3f32e2a99864608a4a7b649 2024-03-22T01:25:57.991 INFO:teuthology.orchestra.run.smithi072.stdout:/bin/podman: stderr Copying config sha256:9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 2024-03-22T01:25:57.991 INFO:teuthology.orchestra.run.smithi072.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-22T01:26:04.004 INFO:teuthology.orchestra.run.smithi072.stdout:ceph: stdout ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev) 2024-03-22T01:26:04.005 INFO:teuthology.orchestra.run.smithi072.stdout:Ceph version: ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev) 2024-03-22T01:26:04.005 INFO:teuthology.orchestra.run.smithi072.stdout:Extracting ceph user uid/gid from container image... 2024-03-22T01:26:17.870 INFO:teuthology.orchestra.run.smithi072.stdout:stat: stdout 167 167 2024-03-22T01:26:17.870 INFO:teuthology.orchestra.run.smithi072.stdout:Creating initial keys... 2024-03-22T01:26:18.774 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph-authtool: stdout AQA63vxljh/kCxAAAxvgFqsiTaz85VXXa2mrmg== 2024-03-22T01:26:19.627 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph-authtool: stdout AQA73vxly1jSBRAAU9avqjkYB0GGmx29WLY7Xw== 2024-03-22T01:26:20.507 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph-authtool: stdout AQA73vxljBkcOBAAbSS62KUDQBnf7l42TYf93A== 2024-03-22T01:26:20.508 INFO:teuthology.orchestra.run.smithi072.stdout:Creating initial monmap... 2024-03-22T01:26:21.498 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-22T01:26:21.498 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-22T01:26:21.498 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 022a99bc-e7eb-11ee-95cd-87774f69a715 2024-03-22T01:26:21.498 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-22T01:26:21.498 INFO:teuthology.orchestra.run.smithi072.stdout:monmaptool for a [v2:172.21.15.72:3300,v1:172.21.15.72:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-22T01:26:21.498 INFO:teuthology.orchestra.run.smithi072.stdout:setting min_mon_release = quincy 2024-03-22T01:26:21.498 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/monmaptool: set fsid to 022a99bc-e7eb-11ee-95cd-87774f69a715 2024-03-22T01:26:21.498 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-22T01:26:21.499 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:26:21.499 INFO:teuthology.orchestra.run.smithi072.stdout:Creating mon... 2024-03-22T01:26:22.471 INFO:teuthology.orchestra.run.smithi072.stdout:create mon.a on 2024-03-22T01:26:22.673 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-22T01:26:22.815 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-22T01:26:22.984 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-022a99bc-e7eb-11ee-95cd-87774f69a715.target → /etc/systemd/system/ceph-022a99bc-e7eb-11ee-95cd-87774f69a715.target. 2024-03-22T01:26:22.985 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-022a99bc-e7eb-11ee-95cd-87774f69a715.target → /etc/systemd/system/ceph-022a99bc-e7eb-11ee-95cd-87774f69a715.target. 2024-03-22T01:26:23.228 INFO:teuthology.orchestra.run.smithi072.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mon.a 2024-03-22T01:26:23.228 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stderr Failed to reset failed state of unit ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mon.a.service: Unit ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mon.a.service not loaded. 2024-03-22T01:26:23.386 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-022a99bc-e7eb-11ee-95cd-87774f69a715.target.wants/ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mon.a.service → /etc/systemd/system/ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@.service. 2024-03-22T01:26:23.891 INFO:teuthology.orchestra.run.smithi072.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-22T01:26:23.892 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stdout disabled 2024-03-22T01:26:23.900 INFO:teuthology.orchestra.run.smithi072.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-22T01:26:23.900 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stdout inactive 2024-03-22T01:26:23.900 INFO:teuthology.orchestra.run.smithi072.stdout:firewalld.service is not enabled 2024-03-22T01:26:23.900 INFO:teuthology.orchestra.run.smithi072.stdout:Not possible to enable service . firewalld.service is not available 2024-03-22T01:26:23.900 INFO:teuthology.orchestra.run.smithi072.stdout:Waiting for mon to start... 2024-03-22T01:26:23.900 INFO:teuthology.orchestra.run.smithi072.stdout:Waiting for mon... 2024-03-22T01:26:24.979 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout cluster: 2024-03-22T01:26:24.979 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout id: 022a99bc-e7eb-11ee-95cd-87774f69a715 2024-03-22T01:26:24.980 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-22T01:26:24.980 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout 2024-03-22T01:26:24.980 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout services: 2024-03-22T01:26:24.980 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.51485s) 2024-03-22T01:26:24.980 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-22T01:26:24.980 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-22T01:26:24.980 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout 2024-03-22T01:26:24.980 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout data: 2024-03-22T01:26:24.980 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-22T01:26:24.980 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-22T01:26:24.980 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-22T01:26:24.980 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout pgs: 2024-03-22T01:26:24.980 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout 2024-03-22T01:26:24.981 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:24.318+0000 7f3c12930700 1 Processor -- start 2024-03-22T01:26:24.981 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:24.318+0000 7f3c12930700 1 -- start start 2024-03-22T01:26:24.981 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:24.318+0000 7f3c12930700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3c0c1096b0 0x7f3c0c109ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:26:24.981 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:24.318+0000 7f3c12930700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f3c0c10a0a0 con 0x7f3c0c1096b0 2024-03-22T01:26:24.981 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:24.318+0000 7f3c0bfff700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3c0c1096b0 0x7f3c0c109ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:26:24.981 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:24.318+0000 7f3c0bfff700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3c0c1096b0 0x7f3c0c109ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:40722/0 (socket says 172.21.15.72:40722) 2024-03-22T01:26:24.981 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:24.318+0000 7f3c0bfff700 1 -- 172.21.15.72:0/145550101 learned_addr learned my addr 172.21.15.72:0/145550101 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:26:24.981 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:24.319+0000 7f3c0bfff700 1 -- 172.21.15.72:0/145550101 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c0c10a220 con 0x7f3c0c1096b0 2024-03-22T01:26:24.981 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:24.319+0000 7f3c0bfff700 1 --2- 172.21.15.72:0/145550101 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3c0c1096b0 0x7f3c0c109ad0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc0098a0 tx=0x7f3bfc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef57927cff4a16c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:26:24.981 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:24.319+0000 7f3c0affd700 1 -- 172.21.15.72:0/145550101 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3bfc004160 con 0x7f3c0c1096b0 2024-03-22T01:26:24.981 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:24.319+0000 7f3c0affd700 1 -- 172.21.15.72:0/145550101 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bfc004300 con 0x7f3c0c1096b0 2024-03-22T01:26:24.981 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:24.319+0000 7f3c0affd700 1 -- 172.21.15.72:0/145550101 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3bfc0036a0 con 0x7f3c0c1096b0 2024-03-22T01:26:24.981 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:24.319+0000 7f3c12930700 1 -- 172.21.15.72:0/145550101 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3c0c1096b0 msgr2=0x7f3c0c109ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:26:24.981 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:24.319+0000 7f3c12930700 1 --2- 172.21.15.72:0/145550101 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3c0c1096b0 0x7f3c0c109ad0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc0098a0 tx=0x7f3bfc009bb0 comp rx=0 tx=0).stop 2024-03-22T01:26:24.982 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:24.320+0000 7f3c12930700 1 -- 172.21.15.72:0/145550101 shutdown_connections 2024-03-22T01:26:24.982 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:24.320+0000 7f3c12930700 1 --2- 172.21.15.72:0/145550101 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3c0c1096b0 0x7f3c0c109ad0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:26:24.982 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:24.320+0000 7f3c12930700 1 -- 172.21.15.72:0/145550101 >> 172.21.15.72:0/145550101 conn(0x7f3c0c101d30 msgr2=0x7f3c0c104150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:26:24.982 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:24.320+0000 7f3c12930700 1 -- 172.21.15.72:0/145550101 shutdown_connections 2024-03-22T01:26:24.982 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:24.320+0000 7f3c12930700 1 -- 172.21.15.72:0/145550101 wait complete. 2024-03-22T01:26:24.982 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:24.320+0000 7f3c12930700 1 Processor -- start 2024-03-22T01:26:24.982 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:24.320+0000 7f3c12930700 1 -- start start 2024-03-22T01:26:24.982 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:24.320+0000 7f3c12930700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3c0c1096b0 0x7f3c0c108cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:26:24.983 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:24.320+0000 7f3c12930700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f3c0c106430 con 0x7f3c0c1096b0 2024-03-22T01:26:24.983 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:24.321+0000 7f3c0bfff700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3c0c1096b0 0x7f3c0c108cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:26:24.983 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:24.321+0000 7f3c0bfff700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3c0c1096b0 0x7f3c0c108cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:40736/0 (socket says 172.21.15.72:40736) 2024-03-22T01:26:24.983 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:24.321+0000 7f3c0bfff700 1 -- 172.21.15.72:0/822808621 learned_addr learned my addr 172.21.15.72:0/822808621 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:26:24.983 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:24.321+0000 7f3c0bfff700 1 -- 172.21.15.72:0/822808621 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c0c109210 con 0x7f3c0c1096b0 2024-03-22T01:26:24.983 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:24.321+0000 7f3c0bfff700 1 --2- 172.21.15.72:0/822808621 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3c0c1096b0 0x7f3c0c108cd0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc000c00 tx=0x7f3bfc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:26:24.983 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:24.321+0000 7f3c097fa700 1 -- 172.21.15.72:0/822808621 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3bfc003db0 con 0x7f3c0c1096b0 2024-03-22T01:26:24.983 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:24.321+0000 7f3c097fa700 1 -- 172.21.15.72:0/822808621 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bfc011630 con 0x7f3c0c1096b0 2024-03-22T01:26:24.983 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:24.321+0000 7f3c12930700 1 -- 172.21.15.72:0/822808621 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c0c107400 con 0x7f3c0c1096b0 2024-03-22T01:26:24.983 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:24.321+0000 7f3c097fa700 1 -- 172.21.15.72:0/822808621 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3bfc011950 con 0x7f3c0c1096b0 2024-03-22T01:26:24.983 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:24.321+0000 7f3c12930700 1 -- 172.21.15.72:0/822808621 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c0c107940 con 0x7f3c0c1096b0 2024-03-22T01:26:24.984 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:24.321+0000 7f3c097fa700 1 -- 172.21.15.72:0/822808621 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f3bfc011af0 con 0x7f3c0c1096b0 2024-03-22T01:26:24.984 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:24.321+0000 7f3c097fa700 1 -- 172.21.15.72:0/822808621 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f3bfc01c450 con 0x7f3c0c1096b0 2024-03-22T01:26:24.984 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:24.322+0000 7f3c12930700 1 -- 172.21.15.72:0/822808621 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c0c063cb0 con 0x7f3c0c1096b0 2024-03-22T01:26:24.984 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:24.324+0000 7f3c097fa700 1 -- 172.21.15.72:0/822808621 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f3bfc032020 con 0x7f3c0c1096b0 2024-03-22T01:26:24.984 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:24.394+0000 7f3c12930700 1 -- 172.21.15.72:0/822808621 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f3c0c194a30 con 0x7f3c0c1096b0 2024-03-22T01:26:24.984 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:24.394+0000 7f3c097fa700 1 -- 172.21.15.72:0/822808621 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+316 (secure 0 0 0) 0x7f3bfc004260 con 0x7f3c0c1096b0 2024-03-22T01:26:24.984 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:24.395+0000 7f3c12930700 1 -- 172.21.15.72:0/822808621 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3c0c1096b0 msgr2=0x7f3c0c108cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:26:24.984 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:24.395+0000 7f3c12930700 1 --2- 172.21.15.72:0/822808621 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3c0c1096b0 0x7f3c0c108cd0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc000c00 tx=0x7f3bfc004750 comp rx=0 tx=0).stop 2024-03-22T01:26:24.984 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:24.395+0000 7f3c12930700 1 -- 172.21.15.72:0/822808621 shutdown_connections 2024-03-22T01:26:24.984 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:24.395+0000 7f3c12930700 1 --2- 172.21.15.72:0/822808621 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3c0c1096b0 0x7f3c0c108cd0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:26:24.984 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:24.395+0000 7f3c12930700 1 -- 172.21.15.72:0/822808621 >> 172.21.15.72:0/822808621 conn(0x7f3c0c101d30 msgr2=0x7f3c0c18f980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:26:24.984 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:24.395+0000 7f3c12930700 1 -- 172.21.15.72:0/822808621 shutdown_connections 2024-03-22T01:26:24.985 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:24.395+0000 7f3c12930700 1 -- 172.21.15.72:0/822808621 wait complete. 2024-03-22T01:26:24.985 INFO:teuthology.orchestra.run.smithi072.stdout:mon is available 2024-03-22T01:26:24.985 INFO:teuthology.orchestra.run.smithi072.stdout:Assimilating anything we can from ceph.conf... 2024-03-22T01:26:25.994 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout 2024-03-22T01:26:25.994 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout [global] 2024-03-22T01:26:25.994 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout fsid = 022a99bc-e7eb-11ee-95cd-87774f69a715 2024-03-22T01:26:25.994 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.72:3300,v1:172.21.15.72:6789] 2024-03-22T01:26:25.994 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-22T01:26:25.994 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-22T01:26:25.994 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-22T01:26:25.994 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-22T01:26:25.994 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout 2024-03-22T01:26:25.994 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-22T01:26:25.994 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-22T01:26:25.995 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout 2024-03-22T01:26:25.995 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout [osd] 2024-03-22T01:26:25.995 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout filestore_flush_min = 0 2024-03-22T01:26:25.995 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-22T01:26:25.995 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-22T01:26:25.996 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-22T01:26:25.996 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-22T01:26:25.996 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-22T01:26:25.996 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:25.391+0000 7f8929d4a700 1 Processor -- start 2024-03-22T01:26:25.996 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:25.391+0000 7f8929d4a700 1 -- start start 2024-03-22T01:26:25.996 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:25.391+0000 7f8929d4a700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8924108170 0x7f8924108590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:26:25.996 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:25.391+0000 7f8929d4a700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f8924108b60 con 0x7f8924108170 2024-03-22T01:26:25.996 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:25.391+0000 7f89237fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8924108170 0x7f8924108590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:26:25.996 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:25.391+0000 7f89237fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8924108170 0x7f8924108590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:40748/0 (socket says 172.21.15.72:40748) 2024-03-22T01:26:25.996 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:25.391+0000 7f89237fe700 1 -- 172.21.15.72:0/3809897021 learned_addr learned my addr 172.21.15.72:0/3809897021 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:26:25.997 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:25.391+0000 7f89237fe700 1 -- 172.21.15.72:0/3809897021 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89241092d0 con 0x7f8924108170 2024-03-22T01:26:25.997 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:25.391+0000 7f89237fe700 1 --2- 172.21.15.72:0/3809897021 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8924108170 0x7f8924108590 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f89140098a0 tx=0x7f8914009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5c7e48055b99ce4 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:26:25.997 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:25.392+0000 7f89227fc700 1 -- 172.21.15.72:0/3809897021 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8914004160 con 0x7f8924108170 2024-03-22T01:26:25.998 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:25.392+0000 7f89227fc700 1 -- 172.21.15.72:0/3809897021 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8914004300 con 0x7f8924108170 2024-03-22T01:26:25.998 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:25.392+0000 7f89227fc700 1 -- 172.21.15.72:0/3809897021 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f89140036a0 con 0x7f8924108170 2024-03-22T01:26:25.998 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:25.392+0000 7f8929d4a700 1 -- 172.21.15.72:0/3809897021 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8924108170 msgr2=0x7f8924108590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:26:25.998 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:25.392+0000 7f8929d4a700 1 --2- 172.21.15.72:0/3809897021 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8924108170 0x7f8924108590 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f89140098a0 tx=0x7f8914009bb0 comp rx=0 tx=0).stop 2024-03-22T01:26:25.998 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:25.392+0000 7f8929d4a700 1 -- 172.21.15.72:0/3809897021 shutdown_connections 2024-03-22T01:26:25.998 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:25.392+0000 7f8929d4a700 1 --2- 172.21.15.72:0/3809897021 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8924108170 0x7f8924108590 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:26:25.998 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:25.392+0000 7f8929d4a700 1 -- 172.21.15.72:0/3809897021 >> 172.21.15.72:0/3809897021 conn(0x7f8924103750 msgr2=0x7f8924105b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:26:25.998 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:25.392+0000 7f8929d4a700 1 -- 172.21.15.72:0/3809897021 shutdown_connections 2024-03-22T01:26:25.998 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:25.392+0000 7f8929d4a700 1 -- 172.21.15.72:0/3809897021 wait complete. 2024-03-22T01:26:25.998 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:25.393+0000 7f8929d4a700 1 Processor -- start 2024-03-22T01:26:25.998 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:25.393+0000 7f8929d4a700 1 -- start start 2024-03-22T01:26:25.998 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:25.393+0000 7f8929d4a700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8924108170 0x7f892419b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:26:25.998 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:25.393+0000 7f8929d4a700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f8924109800 con 0x7f8924108170 2024-03-22T01:26:25.999 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:25.393+0000 7f89237fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8924108170 0x7f892419b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:26:25.999 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:25.393+0000 7f89237fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8924108170 0x7f892419b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:40762/0 (socket says 172.21.15.72:40762) 2024-03-22T01:26:25.999 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:25.393+0000 7f89237fe700 1 -- 172.21.15.72:0/2132273276 learned_addr learned my addr 172.21.15.72:0/2132273276 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:26:25.999 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:25.394+0000 7f89237fe700 1 -- 172.21.15.72:0/2132273276 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f892419bce0 con 0x7f8924108170 2024-03-22T01:26:25.999 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:25.394+0000 7f89237fe700 1 --2- 172.21.15.72:0/2132273276 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8924108170 0x7f892419b7a0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f8914000c00 tx=0x7f8914004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:26:25.999 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:25.394+0000 7f8920ff9700 1 -- 172.21.15.72:0/2132273276 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8914003db0 con 0x7f8924108170 2024-03-22T01:26:25.999 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:25.394+0000 7f8920ff9700 1 -- 172.21.15.72:0/2132273276 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8914011630 con 0x7f8924108170 2024-03-22T01:26:25.999 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:25.394+0000 7f8929d4a700 1 -- 172.21.15.72:0/2132273276 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f892419bf70 con 0x7f8924108170 2024-03-22T01:26:25.999 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:25.394+0000 7f8920ff9700 1 -- 172.21.15.72:0/2132273276 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8914011950 con 0x7f8924108170 2024-03-22T01:26:25.999 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:25.394+0000 7f8929d4a700 1 -- 172.21.15.72:0/2132273276 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f892419ec60 con 0x7f8924108170 2024-03-22T01:26:25.999 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:25.394+0000 7f8920ff9700 1 -- 172.21.15.72:0/2132273276 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f891401a460 con 0x7f8924108170 2024-03-22T01:26:25.999 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:25.394+0000 7f8920ff9700 1 -- 172.21.15.72:0/2132273276 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f891401b780 con 0x7f8924108170 2024-03-22T01:26:25.999 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:25.395+0000 7f8929d4a700 1 -- 172.21.15.72:0/2132273276 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8924063cb0 con 0x7f8924108170 2024-03-22T01:26:25.999 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:25.397+0000 7f8920ff9700 1 -- 172.21.15.72:0/2132273276 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f8914031020 con 0x7f8924108170 2024-03-22T01:26:26.000 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:25.464+0000 7f8929d4a700 1 -- 172.21.15.72:0/2132273276 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f892402e070 con 0x7f8924108170 2024-03-22T01:26:26.000 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:25.467+0000 7f8920ff9700 1 -- 172.21.15.72:0/2132273276 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+454 (secure 0 0 0) 0x7f891401b980 con 0x7f8924108170 2024-03-22T01:26:26.000 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:25.468+0000 7f8920ff9700 1 -- 172.21.15.72:0/2132273276 <== mon.0 v2:172.21.15.72:3300/0 8 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f891401ad00 con 0x7f8924108170 2024-03-22T01:26:26.000 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:25.468+0000 7f8929d4a700 1 -- 172.21.15.72:0/2132273276 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8924108170 msgr2=0x7f892419b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:26:26.000 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:25.468+0000 7f8929d4a700 1 --2- 172.21.15.72:0/2132273276 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8924108170 0x7f892419b7a0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f8914000c00 tx=0x7f8914004750 comp rx=0 tx=0).stop 2024-03-22T01:26:26.000 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:25.469+0000 7f8929d4a700 1 -- 172.21.15.72:0/2132273276 shutdown_connections 2024-03-22T01:26:26.000 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:25.469+0000 7f8929d4a700 1 --2- 172.21.15.72:0/2132273276 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8924108170 0x7f892419b7a0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:26:26.000 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:25.469+0000 7f8929d4a700 1 -- 172.21.15.72:0/2132273276 >> 172.21.15.72:0/2132273276 conn(0x7f8924103750 msgr2=0x7f8924192380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:26:26.000 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:25.469+0000 7f8929d4a700 1 -- 172.21.15.72:0/2132273276 shutdown_connections 2024-03-22T01:26:26.000 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:25.469+0000 7f8929d4a700 1 -- 172.21.15.72:0/2132273276 wait complete. 2024-03-22T01:26:26.000 INFO:teuthology.orchestra.run.smithi072.stdout:Generating new minimal ceph.conf... 2024-03-22T01:26:27.066 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:26 smithi072 ceph-mon[32039]: from='client.? 172.21.15.72:0/3374210518' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:26:27.073 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:26.429+0000 7f256561a700 1 Processor -- start 2024-03-22T01:26:27.073 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:26.429+0000 7f256561a700 1 -- start start 2024-03-22T01:26:27.073 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:26.429+0000 7f256561a700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2560107010 0x7f2560107430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:26:27.073 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:26.429+0000 7f256561a700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f2560107a00 con 0x7f2560107010 2024-03-22T01:26:27.073 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:26.430+0000 7f255effd700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2560107010 0x7f2560107430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:26:27.074 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:26.430+0000 7f255effd700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2560107010 0x7f2560107430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:40776/0 (socket says 172.21.15.72:40776) 2024-03-22T01:26:27.074 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:26.430+0000 7f255effd700 1 -- 172.21.15.72:0/3434742992 learned_addr learned my addr 172.21.15.72:0/3434742992 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:26:27.074 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:26.430+0000 7f255effd700 1 -- 172.21.15.72:0/3434742992 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2560108210 con 0x7f2560107010 2024-03-22T01:26:27.074 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:26.430+0000 7f255effd700 1 --2- 172.21.15.72:0/3434742992 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2560107010 0x7f2560107430 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f2558009e80 tx=0x7f255800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b26e86c6a2ed4d18 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:26:27.074 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:26.430+0000 7f255dffb700 1 -- 172.21.15.72:0/3434742992 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2558017070 con 0x7f2560107010 2024-03-22T01:26:27.074 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:26.430+0000 7f255dffb700 1 -- 172.21.15.72:0/3434742992 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f255800b8c0 con 0x7f2560107010 2024-03-22T01:26:27.074 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:26.430+0000 7f255dffb700 1 -- 172.21.15.72:0/3434742992 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f25580045c0 con 0x7f2560107010 2024-03-22T01:26:27.074 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:26.431+0000 7f256561a700 1 -- 172.21.15.72:0/3434742992 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2560107010 msgr2=0x7f2560107430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:26:27.074 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:26.431+0000 7f256561a700 1 --2- 172.21.15.72:0/3434742992 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2560107010 0x7f2560107430 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f2558009e80 tx=0x7f255800b330 comp rx=0 tx=0).stop 2024-03-22T01:26:27.074 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:26.431+0000 7f256561a700 1 -- 172.21.15.72:0/3434742992 shutdown_connections 2024-03-22T01:26:27.074 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:26.431+0000 7f256561a700 1 --2- 172.21.15.72:0/3434742992 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2560107010 0x7f2560107430 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:26:27.074 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:26.431+0000 7f256561a700 1 -- 172.21.15.72:0/3434742992 >> 172.21.15.72:0/3434742992 conn(0x7f2560102630 msgr2=0x7f2560104a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:26:27.075 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:26.431+0000 7f256561a700 1 -- 172.21.15.72:0/3434742992 shutdown_connections 2024-03-22T01:26:27.075 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:26.431+0000 7f256561a700 1 -- 172.21.15.72:0/3434742992 wait complete. 2024-03-22T01:26:27.075 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:26.431+0000 7f256561a700 1 Processor -- start 2024-03-22T01:26:27.075 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:26.432+0000 7f256561a700 1 -- start start 2024-03-22T01:26:27.075 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:26.432+0000 7f256561a700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2560107010 0x7f25601921c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:26:27.075 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:26.432+0000 7f256561a700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f256010b5c0 con 0x7f2560107010 2024-03-22T01:26:27.075 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:26.432+0000 7f255effd700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2560107010 0x7f25601921c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:26:27.075 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:26.432+0000 7f255effd700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2560107010 0x7f25601921c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:40790/0 (socket says 172.21.15.72:40790) 2024-03-22T01:26:27.075 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:26.432+0000 7f255effd700 1 -- 172.21.15.72:0/3374210518 learned_addr learned my addr 172.21.15.72:0/3374210518 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:26:27.075 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:26.432+0000 7f255effd700 1 -- 172.21.15.72:0/3374210518 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2560192700 con 0x7f2560107010 2024-03-22T01:26:27.075 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:26.432+0000 7f255effd700 1 --2- 172.21.15.72:0/3374210518 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2560107010 0x7f25601921c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2558009e50 tx=0x7f255800bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:26:27.075 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:26.432+0000 7f2547fff700 1 -- 172.21.15.72:0/3374210518 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2558020020 con 0x7f2560107010 2024-03-22T01:26:27.075 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:26.432+0000 7f2547fff700 1 -- 172.21.15.72:0/3374210518 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2558003b50 con 0x7f2560107010 2024-03-22T01:26:27.076 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:26.432+0000 7f256561a700 1 -- 172.21.15.72:0/3374210518 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25601908f0 con 0x7f2560107010 2024-03-22T01:26:27.076 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:26.432+0000 7f2547fff700 1 -- 172.21.15.72:0/3374210518 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2558017040 con 0x7f2560107010 2024-03-22T01:26:27.076 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:26.432+0000 7f256561a700 1 -- 172.21.15.72:0/3374210518 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2560190e30 con 0x7f2560107010 2024-03-22T01:26:27.076 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:26.433+0000 7f2547fff700 1 -- 172.21.15.72:0/3374210518 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f2558023020 con 0x7f2560107010 2024-03-22T01:26:27.076 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:26.433+0000 7f2547fff700 1 -- 172.21.15.72:0/3374210518 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f255800f050 con 0x7f2560107010 2024-03-22T01:26:27.076 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:26.433+0000 7f256561a700 1 -- 172.21.15.72:0/3374210518 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2560063cb0 con 0x7f2560107010 2024-03-22T01:26:27.076 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:26.436+0000 7f2547fff700 1 -- 172.21.15.72:0/3374210518 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f255802a020 con 0x7f2560107010 2024-03-22T01:26:27.076 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:26.502+0000 7f256561a700 1 -- 172.21.15.72:0/3374210518 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f2560195e00 con 0x7f2560107010 2024-03-22T01:26:27.076 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:26.502+0000 7f2547fff700 1 -- 172.21.15.72:0/3374210518 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7f2558003d00 con 0x7f2560107010 2024-03-22T01:26:27.076 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:26.503+0000 7f256561a700 1 -- 172.21.15.72:0/3374210518 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2560107010 msgr2=0x7f25601921c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:26:27.076 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:26.503+0000 7f256561a700 1 --2- 172.21.15.72:0/3374210518 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2560107010 0x7f25601921c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2558009e50 tx=0x7f255800bd30 comp rx=0 tx=0).stop 2024-03-22T01:26:27.076 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:26.503+0000 7f256561a700 1 -- 172.21.15.72:0/3374210518 shutdown_connections 2024-03-22T01:26:27.076 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:26.503+0000 7f256561a700 1 --2- 172.21.15.72:0/3374210518 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2560107010 0x7f25601921c0 secure :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2558009e50 tx=0x7f255800bd30 comp rx=0 tx=0).stop 2024-03-22T01:26:27.076 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:26.503+0000 7f256561a700 1 -- 172.21.15.72:0/3374210518 >> 172.21.15.72:0/3374210518 conn(0x7f2560102630 msgr2=0x7f2560103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:26:27.077 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:26.503+0000 7f256561a700 1 -- 172.21.15.72:0/3374210518 shutdown_connections 2024-03-22T01:26:27.077 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:26.503+0000 7f256561a700 1 -- 172.21.15.72:0/3374210518 wait complete. 2024-03-22T01:26:27.077 INFO:teuthology.orchestra.run.smithi072.stdout:Restarting the monitor... 2024-03-22T01:26:27.329 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:27 smithi072 systemd[1]: Stopping Ceph mon.a for 022a99bc-e7eb-11ee-95cd-87774f69a715... 2024-03-22T01:26:27.580 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:27 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mon-a[32035]: 2024-03-22T01:26:27.330+0000 7f6bf28c2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T01:26:27.580 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:27 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mon-a[32035]: 2024-03-22T01:26:27.330+0000 7f6bf28c2700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-22T01:26:27.580 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:27 smithi072 podman[32287]: 2024-03-22 01:26:27.361553937 +0000 UTC m=+0.122853240 container died 97a4005b4f99771ee7ec45e9cef75474f2ad88aeea0b2fa4d6671129830870ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mon-a, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=) 2024-03-22T01:26:27.867 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:27 smithi072 podman[32287]: 2024-03-22 01:26:27.682017448 +0000 UTC m=+0.443316737 container cleanup 97a4005b4f99771ee7ec45e9cef75474f2ad88aeea0b2fa4d6671129830870ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mon-a, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-22T01:26:27.867 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:27 smithi072 bash[32287]: ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mon-a 2024-03-22T01:26:28.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:27 smithi072 podman[32301]: 2024-03-22 01:26:27.869080958 +0000 UTC m=+0.510495338 container remove 97a4005b4f99771ee7ec45e9cef75474f2ad88aeea0b2fa4d6671129830870ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mon-a, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5) 2024-03-22T01:26:28.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 systemd[1]: ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-22T01:26:28.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 systemd[1]: Stopped Ceph mon.a for 022a99bc-e7eb-11ee-95cd-87774f69a715. 2024-03-22T01:26:28.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 systemd[1]: Starting Ceph mon.a for 022a99bc-e7eb-11ee-95cd-87774f69a715... 2024-03-22T01:26:28.543 INFO:teuthology.orchestra.run.smithi072.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-22T01:26:28.569 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 podman[32395]: 2024-03-22 01:26:28.295694424 +0000 UTC m=+0.095835657 container create c91280b7adc3cb78e5ecaa1c015a5fce371dddbe8c964b3c10de41107ae3c499 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-22T01:26:28.570 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 podman[32395]: 2024-03-22 01:26:28.216595001 +0000 UTC m=+0.016736196 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e 2024-03-22T01:26:28.570 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 podman[32395]: 2024-03-22 01:26:28.466153419 +0000 UTC m=+0.266294603 container init c91280b7adc3cb78e5ecaa1c015a5fce371dddbe8c964b3c10de41107ae3c499 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mon-a, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0) 2024-03-22T01:26:28.570 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 podman[32395]: 2024-03-22 01:26:28.468890595 +0000 UTC m=+0.269031781 container start c91280b7adc3cb78e5ecaa1c015a5fce371dddbe8c964b3c10de41107ae3c499 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD) 2024-03-22T01:26:28.570 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: set uid:gid to 167:167 (ceph:ceph) 2024-03-22T01:26:28.570 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev), process ceph-mon, pid 2 2024-03-22T01:26:28.570 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: pidfile_write: ignore empty --pid-file 2024-03-22T01:26:28.570 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: load: jerasure load: lrc 2024-03-22T01:26:28.570 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: RocksDB version: 7.9.2 2024-03-22T01:26:28.570 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Git sha 0 2024-03-22T01:26:28.570 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Compile date 2024-03-21 19:33:57 2024-03-22T01:26:28.570 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: DB SUMMARY 2024-03-22T01:26:28.571 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: DB Session ID: ST3VH6AW8BH4GJF0AC27 2024-03-22T01:26:28.571 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: CURRENT file: CURRENT 2024-03-22T01:26:28.571 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: IDENTITY file: IDENTITY 2024-03-22T01:26:28.571 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-22T01:26:28.571 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-22T01:26:28.571 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 104938 ; 2024-03-22T01:26:28.571 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.error_if_exists: 0 2024-03-22T01:26:28.571 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.create_if_missing: 0 2024-03-22T01:26:28.571 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.paranoid_checks: 1 2024-03-22T01:26:28.571 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-22T01:26:28.571 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-22T01:26:28.571 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-22T01:26:28.571 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.env: 0x558c561b2c60 2024-03-22T01:26:28.571 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.fs: PosixFileSystem 2024-03-22T01:26:28.571 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.info_log: 0x558c58211ae0 2024-03-22T01:26:28.572 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.max_file_opening_threads: 16 2024-03-22T01:26:28.572 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.statistics: (nil) 2024-03-22T01:26:28.572 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.use_fsync: 0 2024-03-22T01:26:28.572 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.max_log_file_size: 0 2024-03-22T01:26:28.572 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-22T01:26:28.572 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-22T01:26:28.572 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.keep_log_file_num: 1000 2024-03-22T01:26:28.572 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.recycle_log_file_num: 0 2024-03-22T01:26:28.572 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.allow_fallocate: 1 2024-03-22T01:26:28.572 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.allow_mmap_reads: 0 2024-03-22T01:26:28.572 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.allow_mmap_writes: 0 2024-03-22T01:26:28.572 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.use_direct_reads: 0 2024-03-22T01:26:28.572 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-22T01:26:28.573 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.create_missing_column_families: 0 2024-03-22T01:26:28.573 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.db_log_dir: 2024-03-22T01:26:28.573 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.wal_dir: 2024-03-22T01:26:28.573 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-22T01:26:28.573 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-22T01:26:28.573 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-22T01:26:28.573 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-22T01:26:28.573 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-22T01:26:28.573 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-22T01:26:28.573 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.advise_random_on_open: 1 2024-03-22T01:26:28.573 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.db_write_buffer_size: 0 2024-03-22T01:26:28.573 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.write_buffer_manager: 0x558c581e21e0 2024-03-22T01:26:28.573 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-22T01:26:28.573 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-22T01:26:28.573 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-22T01:26:28.574 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.rate_limiter: (nil) 2024-03-22T01:26:28.574 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-22T01:26:28.575 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.wal_recovery_mode: 2 2024-03-22T01:26:28.575 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.enable_thread_tracking: 0 2024-03-22T01:26:28.576 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.enable_pipelined_write: 0 2024-03-22T01:26:28.576 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.unordered_write: 0 2024-03-22T01:26:28.576 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-22T01:26:28.576 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-22T01:26:28.576 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-22T01:26:28.576 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-22T01:26:28.576 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.row_cache: None 2024-03-22T01:26:28.576 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.wal_filter: None 2024-03-22T01:26:28.576 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-22T01:26:28.576 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.allow_ingest_behind: 0 2024-03-22T01:26:28.576 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.two_write_queues: 0 2024-03-22T01:26:28.576 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.manual_wal_flush: 0 2024-03-22T01:26:28.576 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.wal_compression: 0 2024-03-22T01:26:28.577 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.atomic_flush: 0 2024-03-22T01:26:28.577 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-22T01:26:28.577 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-22T01:26:28.577 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-22T01:26:28.577 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.log_readahead_size: 0 2024-03-22T01:26:28.577 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-22T01:26:28.577 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.best_efforts_recovery: 0 2024-03-22T01:26:28.577 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-22T01:26:28.577 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-22T01:26:28.577 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.allow_data_in_errors: 0 2024-03-22T01:26:28.577 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.db_host_id: __hostname__ 2024-03-22T01:26:28.577 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-22T01:26:28.577 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.max_background_jobs: 2 2024-03-22T01:26:28.577 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.max_background_compactions: -1 2024-03-22T01:26:28.577 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.max_subcompactions: 1 2024-03-22T01:26:28.578 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-22T01:26:28.578 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-22T01:26:28.578 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-22T01:26:28.578 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.max_total_wal_size: 0 2024-03-22T01:26:28.578 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-22T01:26:28.578 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-22T01:26:28.578 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-22T01:26:28.578 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-22T01:26:28.578 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.max_open_files: -1 2024-03-22T01:26:28.578 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.bytes_per_sync: 0 2024-03-22T01:26:28.578 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-22T01:26:28.578 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-22T01:26:28.578 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.compaction_readahead_size: 0 2024-03-22T01:26:28.578 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.max_background_flushes: -1 2024-03-22T01:26:28.579 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Compression algorithms supported: 2024-03-22T01:26:28.579 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-22T01:26:28.579 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: kZSTD supported: 0 2024-03-22T01:26:28.579 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: kXpressCompression supported: 0 2024-03-22T01:26:28.579 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: kLZ4HCCompression supported: 1 2024-03-22T01:26:28.579 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: kZlibCompression supported: 1 2024-03-22T01:26:28.579 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: kSnappyCompression supported: 1 2024-03-22T01:26:28.579 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: kLZ4Compression supported: 1 2024-03-22T01:26:28.579 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: kBZip2Compression supported: 0 2024-03-22T01:26:28.579 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-22T01:26:28.579 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-22T01:26:28.579 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-22T01:26:28.579 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-22T01:26:28.579 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-22T01:26:28.580 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.merge_operator: 2024-03-22T01:26:28.580 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.compaction_filter: None 2024-03-22T01:26:28.580 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.compaction_filter_factory: None 2024-03-22T01:26:28.580 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.sst_partitioner_factory: None 2024-03-22T01:26:28.580 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-22T01:26:28.580 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-22T01:26:28.580 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x558c58211be0) 2024-03-22T01:26:28.580 INFO:journalctl@ceph.mon.a.smithi072.stdout: cache_index_and_filter_blocks: 1 2024-03-22T01:26:28.580 INFO:journalctl@ceph.mon.a.smithi072.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-22T01:26:28.580 INFO:journalctl@ceph.mon.a.smithi072.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-22T01:26:28.580 INFO:journalctl@ceph.mon.a.smithi072.stdout: pin_top_level_index_and_filter: 1 2024-03-22T01:26:28.580 INFO:journalctl@ceph.mon.a.smithi072.stdout: index_type: 0 2024-03-22T01:26:28.580 INFO:journalctl@ceph.mon.a.smithi072.stdout: data_block_index_type: 0 2024-03-22T01:26:28.580 INFO:journalctl@ceph.mon.a.smithi072.stdout: index_shortening: 1 2024-03-22T01:26:28.580 INFO:journalctl@ceph.mon.a.smithi072.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-22T01:26:28.581 INFO:journalctl@ceph.mon.a.smithi072.stdout: checksum: 4 2024-03-22T01:26:28.581 INFO:journalctl@ceph.mon.a.smithi072.stdout: no_block_cache: 0 2024-03-22T01:26:28.581 INFO:journalctl@ceph.mon.a.smithi072.stdout: block_cache: 0x558c58215090 2024-03-22T01:26:28.581 INFO:journalctl@ceph.mon.a.smithi072.stdout: block_cache_name: BinnedLRUCache 2024-03-22T01:26:28.581 INFO:journalctl@ceph.mon.a.smithi072.stdout: block_cache_options: 2024-03-22T01:26:28.581 INFO:journalctl@ceph.mon.a.smithi072.stdout: capacity : 536870912 2024-03-22T01:26:28.581 INFO:journalctl@ceph.mon.a.smithi072.stdout: num_shard_bits : 4 2024-03-22T01:26:28.581 INFO:journalctl@ceph.mon.a.smithi072.stdout: strict_capacity_limit : 0 2024-03-22T01:26:28.581 INFO:journalctl@ceph.mon.a.smithi072.stdout: high_pri_pool_ratio: 0.000 2024-03-22T01:26:28.581 INFO:journalctl@ceph.mon.a.smithi072.stdout: block_cache_compressed: (nil) 2024-03-22T01:26:28.581 INFO:journalctl@ceph.mon.a.smithi072.stdout: persistent_cache: (nil) 2024-03-22T01:26:28.581 INFO:journalctl@ceph.mon.a.smithi072.stdout: block_size: 4096 2024-03-22T01:26:28.581 INFO:journalctl@ceph.mon.a.smithi072.stdout: block_size_deviation: 10 2024-03-22T01:26:28.581 INFO:journalctl@ceph.mon.a.smithi072.stdout: block_restart_interval: 16 2024-03-22T01:26:28.582 INFO:journalctl@ceph.mon.a.smithi072.stdout: index_block_restart_interval: 1 2024-03-22T01:26:28.582 INFO:journalctl@ceph.mon.a.smithi072.stdout: metadata_block_size: 4096 2024-03-22T01:26:28.582 INFO:journalctl@ceph.mon.a.smithi072.stdout: partition_filters: 0 2024-03-22T01:26:28.582 INFO:journalctl@ceph.mon.a.smithi072.stdout: use_delta_encoding: 1 2024-03-22T01:26:28.582 INFO:journalctl@ceph.mon.a.smithi072.stdout: filter_policy: bloomfilter 2024-03-22T01:26:28.582 INFO:journalctl@ceph.mon.a.smithi072.stdout: whole_key_filtering: 1 2024-03-22T01:26:28.582 INFO:journalctl@ceph.mon.a.smithi072.stdout: verify_compression: 0 2024-03-22T01:26:28.582 INFO:journalctl@ceph.mon.a.smithi072.stdout: read_amp_bytes_per_bit: 0 2024-03-22T01:26:28.582 INFO:journalctl@ceph.mon.a.smithi072.stdout: format_version: 5 2024-03-22T01:26:28.582 INFO:journalctl@ceph.mon.a.smithi072.stdout: enable_index_compression: 1 2024-03-22T01:26:28.582 INFO:journalctl@ceph.mon.a.smithi072.stdout: block_align: 0 2024-03-22T01:26:28.582 INFO:journalctl@ceph.mon.a.smithi072.stdout: max_auto_readahead_size: 262144 2024-03-22T01:26:28.582 INFO:journalctl@ceph.mon.a.smithi072.stdout: prepopulate_block_cache: 0 2024-03-22T01:26:28.582 INFO:journalctl@ceph.mon.a.smithi072.stdout: initial_auto_readahead_size: 8192 2024-03-22T01:26:28.583 INFO:journalctl@ceph.mon.a.smithi072.stdout: num_file_reads_for_auto_readahead: 2 2024-03-22T01:26:28.583 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.write_buffer_size: 33554432 2024-03-22T01:26:28.583 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.max_write_buffer_number: 2 2024-03-22T01:26:28.583 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.compression: NoCompression 2024-03-22T01:26:28.583 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.bottommost_compression: Disabled 2024-03-22T01:26:28.583 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.prefix_extractor: nullptr 2024-03-22T01:26:28.583 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-22T01:26:28.583 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.num_levels: 7 2024-03-22T01:26:28.583 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-22T01:26:28.583 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-22T01:26:28.583 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-22T01:26:28.583 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-22T01:26:28.583 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-22T01:26:28.583 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-22T01:26:28.583 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-22T01:26:28.584 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-22T01:26:28.584 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-22T01:26:28.584 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-22T01:26:28.584 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-22T01:26:28.584 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-22T01:26:28.584 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-22T01:26:28.584 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.compression_opts.level: 32767 2024-03-22T01:26:28.584 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.compression_opts.strategy: 0 2024-03-22T01:26:28.584 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-22T01:26:28.584 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-22T01:26:28.584 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-22T01:26:28.584 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-22T01:26:28.584 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.compression_opts.enabled: false 2024-03-22T01:26:28.584 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-22T01:26:28.585 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-22T01:26:28.585 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-22T01:26:28.585 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-22T01:26:28.585 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.target_file_size_base: 67108864 2024-03-22T01:26:28.585 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-22T01:26:28.585 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-22T01:26:28.585 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-22T01:26:28.585 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-22T01:26:28.585 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-22T01:26:28.585 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-22T01:26:28.585 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-22T01:26:28.585 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-22T01:26:28.585 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-22T01:26:28.585 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-22T01:26:28.585 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-22T01:26:28.586 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-22T01:26:28.586 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-22T01:26:28.586 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-22T01:26:28.586 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.arena_block_size: 1048576 2024-03-22T01:26:28.586 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-22T01:26:28.586 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-22T01:26:28.586 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.disable_auto_compactions: 0 2024-03-22T01:26:28.586 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-22T01:26:28.587 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-22T01:26:28.587 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-22T01:26:28.587 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-22T01:26:28.587 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-22T01:26:28.587 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-22T01:26:28.587 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-22T01:26:28.587 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-22T01:26:28.587 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-22T01:26:28.587 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-22T01:26:28.587 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.table_properties_collectors: 2024-03-22T01:26:28.587 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.inplace_update_support: 0 2024-03-22T01:26:28.587 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-22T01:26:28.587 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-22T01:26:28.587 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-22T01:26:28.587 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-22T01:26:28.588 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.bloom_locality: 0 2024-03-22T01:26:28.588 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.max_successive_merges: 0 2024-03-22T01:26:28.588 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-22T01:26:28.588 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.paranoid_file_checks: 0 2024-03-22T01:26:28.588 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.force_consistency_checks: 1 2024-03-22T01:26:28.588 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.report_bg_io_stats: 0 2024-03-22T01:26:28.588 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.ttl: 2592000 2024-03-22T01:26:28.588 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-22T01:26:28.588 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-22T01:26:28.588 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-22T01:26:28.588 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.enable_blob_files: false 2024-03-22T01:26:28.589 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.min_blob_size: 0 2024-03-22T01:26:28.589 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.blob_file_size: 268435456 2024-03-22T01:26:28.589 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-22T01:26:28.589 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-22T01:26:28.589 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-22T01:26:28.589 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-22T01:26:28.589 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-22T01:26:28.589 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.blob_file_starting_level: 0 2024-03-22T01:26:28.589 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-22T01:26:28.589 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-22T01:26:28.589 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-22T01:26:28.589 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 33ab0156-a7c1-49fc-b16d-fe3fc0e7b464 2024-03-22T01:26:28.589 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711070788497666, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-22T01:26:28.590 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-22T01:26:28.590 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711070788498657, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 100083, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 326, "table_properties": {"data_size": 97991, "index_size": 310, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14151, "raw_average_key_size": 49, "raw_value_size": 90292, "raw_average_value_size": 316, "num_data_blocks": 12, "num_entries": 285, "num_filter_entries": 285, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711070788, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "33ab0156-a7c1-49fc-b16d-fe3fc0e7b464", "db_session_id": "ST3VH6AW8BH4GJF0AC27", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-22T01:26:28.590 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711070788498739, "job": 1, "event": "recovery_finished"} 2024-03-22T01:26:28.590 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-22T01:26:28.590 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-22T01:26:28.590 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x558c58304000 2024-03-22T01:26:28.590 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: DB pointer 0x558c582ee000 2024-03-22T01:26:28.590 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-22T01:26:28.590 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-22T01:26:28.590 INFO:journalctl@ceph.mon.a.smithi072.stdout: ** DB Stats ** 2024-03-22T01:26:28.590 INFO:journalctl@ceph.mon.a.smithi072.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-22T01:26:28.590 INFO:journalctl@ceph.mon.a.smithi072.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-22T01:26:28.590 INFO:journalctl@ceph.mon.a.smithi072.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-22T01:26:28.591 INFO:journalctl@ceph.mon.a.smithi072.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-22T01:26:28.591 INFO:journalctl@ceph.mon.a.smithi072.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-22T01:26:28.591 INFO:journalctl@ceph.mon.a.smithi072.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-22T01:26:28.591 INFO:journalctl@ceph.mon.a.smithi072.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-22T01:26:28.591 INFO:journalctl@ceph.mon.a.smithi072.stdout: 2024-03-22T01:26:28.591 INFO:journalctl@ceph.mon.a.smithi072.stdout: ** Compaction Stats [default] ** 2024-03-22T01:26:28.591 INFO:journalctl@ceph.mon.a.smithi072.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-22T01:26:28.591 INFO:journalctl@ceph.mon.a.smithi072.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-22T01:26:28.591 INFO:journalctl@ceph.mon.a.smithi072.stdout: L0 2/0 99.57 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 150.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-22T01:26:28.591 INFO:journalctl@ceph.mon.a.smithi072.stdout: Sum 2/0 99.57 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 150.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-22T01:26:28.591 INFO:journalctl@ceph.mon.a.smithi072.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 150.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-22T01:26:28.591 INFO:journalctl@ceph.mon.a.smithi072.stdout: 2024-03-22T01:26:28.591 INFO:journalctl@ceph.mon.a.smithi072.stdout: ** Compaction Stats [default] ** 2024-03-22T01:26:28.591 INFO:journalctl@ceph.mon.a.smithi072.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-22T01:26:28.591 INFO:journalctl@ceph.mon.a.smithi072.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-22T01:26:28.592 INFO:journalctl@ceph.mon.a.smithi072.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 150.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-22T01:26:28.592 INFO:journalctl@ceph.mon.a.smithi072.stdout: 2024-03-22T01:26:28.592 INFO:journalctl@ceph.mon.a.smithi072.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-22T01:26:28.592 INFO:journalctl@ceph.mon.a.smithi072.stdout: 2024-03-22T01:26:28.592 INFO:journalctl@ceph.mon.a.smithi072.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-22T01:26:28.592 INFO:journalctl@ceph.mon.a.smithi072.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-22T01:26:28.592 INFO:journalctl@ceph.mon.a.smithi072.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-22T01:26:28.592 INFO:journalctl@ceph.mon.a.smithi072.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-22T01:26:28.592 INFO:journalctl@ceph.mon.a.smithi072.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-22T01:26:28.592 INFO:journalctl@ceph.mon.a.smithi072.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-22T01:26:28.592 INFO:journalctl@ceph.mon.a.smithi072.stdout: Cumulative compaction: 0.00 GB write, 32.37 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-22T01:26:28.592 INFO:journalctl@ceph.mon.a.smithi072.stdout: Interval compaction: 0.00 GB write, 32.37 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-22T01:26:28.592 INFO:journalctl@ceph.mon.a.smithi072.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-22T01:26:28.592 INFO:journalctl@ceph.mon.a.smithi072.stdout: Block cache BinnedLRUCache@0x558c58215090#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-22T01:26:28.593 INFO:journalctl@ceph.mon.a.smithi072.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-03-22T01:26:28.593 INFO:journalctl@ceph.mon.a.smithi072.stdout: 2024-03-22T01:26:28.593 INFO:journalctl@ceph.mon.a.smithi072.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-22T01:26:28.593 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: starting mon.a rank 0 at public addrs [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] at bind addrs [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 2024-03-22T01:26:28.593 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: mon.a@-1(???) e1 preinit fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 2024-03-22T01:26:28.593 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: mon.a@-1(???).mds e1 new map 2024-03-22T01:26:28.593 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: mon.a@-1(???).mds e1 print_map 2024-03-22T01:26:28.593 INFO:journalctl@ceph.mon.a.smithi072.stdout: e1 2024-03-22T01:26:28.593 INFO:journalctl@ceph.mon.a.smithi072.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-22T01:26:28.593 INFO:journalctl@ceph.mon.a.smithi072.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-22T01:26:28.593 INFO:journalctl@ceph.mon.a.smithi072.stdout: legacy client fscid: -1 2024-03-22T01:26:28.593 INFO:journalctl@ceph.mon.a.smithi072.stdout: 2024-03-22T01:26:28.593 INFO:journalctl@ceph.mon.a.smithi072.stdout: No filesystems configured 2024-03-22T01:26:28.593 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-22T01:26:28.593 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-22T01:26:28.594 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-22T01:26:28.594 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-22T01:26:28.594 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-22T01:26:28.594 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 bash[32395]: c91280b7adc3cb78e5ecaa1c015a5fce371dddbe8c964b3c10de41107ae3c499 2024-03-22T01:26:28.594 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 systemd[1]: Started Ceph mon.a for 022a99bc-e7eb-11ee-95cd-87774f69a715. 2024-03-22T01:26:28.594 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-22T01:26:28.594 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: monmap epoch 1 2024-03-22T01:26:28.594 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 2024-03-22T01:26:28.594 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: last_changed 2024-03-22T01:26:20.854185+0000 2024-03-22T01:26:28.594 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: created 2024-03-22T01:26:20.854185+0000 2024-03-22T01:26:28.594 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: min_mon_release 19 (squid) 2024-03-22T01:26:28.594 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: election_strategy: 1 2024-03-22T01:26:28.594 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: 0: [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] mon.a 2024-03-22T01:26:28.594 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: fsmap 2024-03-22T01:26:28.595 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: osdmap e1: 0 total, 0 up, 0 in 2024-03-22T01:26:28.595 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:28 smithi072 ceph-mon[32414]: mgrmap e1: no daemons active 2024-03-22T01:26:29.621 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:28.991+0000 7f521b062700 1 Processor -- start 2024-03-22T01:26:29.621 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:28.991+0000 7f521b062700 1 -- start start 2024-03-22T01:26:29.621 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:28.991+0000 7f521b062700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5214107020 0x7f5214107440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:26:29.621 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:28.991+0000 7f521b062700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f5214107a10 con 0x7f5214107020 2024-03-22T01:26:29.621 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:28.992+0000 7f5218dfe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5214107020 0x7f5214107440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:26:29.621 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:28.992+0000 7f5218dfe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5214107020 0x7f5214107440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:40794/0 (socket says 172.21.15.72:40794) 2024-03-22T01:26:29.622 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:28.992+0000 7f5218dfe700 1 -- 172.21.15.72:0/861130125 learned_addr learned my addr 172.21.15.72:0/861130125 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:26:29.622 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:28.992+0000 7f5218dfe700 1 -- 172.21.15.72:0/861130125 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5214108220 con 0x7f5214107020 2024-03-22T01:26:29.622 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:28.992+0000 7f5218dfe700 1 --2- 172.21.15.72:0/861130125 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5214107020 0x7f5214107440 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f52040098a0 tx=0x7f5204009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4c4845af08c2da1 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:26:29.622 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:28.992+0000 7f52137fe700 1 -- 172.21.15.72:0/861130125 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5204004160 con 0x7f5214107020 2024-03-22T01:26:29.622 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:28.992+0000 7f52137fe700 1 -- 172.21.15.72:0/861130125 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f5204004300 con 0x7f5214107020 2024-03-22T01:26:29.622 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:28.992+0000 7f52137fe700 1 -- 172.21.15.72:0/861130125 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5204011b50 con 0x7f5214107020 2024-03-22T01:26:29.622 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:28.993+0000 7f521b062700 1 -- 172.21.15.72:0/861130125 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5214107020 msgr2=0x7f5214107440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:26:29.622 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:28.993+0000 7f521b062700 1 --2- 172.21.15.72:0/861130125 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5214107020 0x7f5214107440 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f52040098a0 tx=0x7f5204009bb0 comp rx=0 tx=0).stop 2024-03-22T01:26:29.622 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:28.993+0000 7f521b062700 1 -- 172.21.15.72:0/861130125 shutdown_connections 2024-03-22T01:26:29.622 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:28.993+0000 7f521b062700 1 --2- 172.21.15.72:0/861130125 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5214107020 0x7f5214107440 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:26:29.622 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:28.993+0000 7f521b062700 1 -- 172.21.15.72:0/861130125 >> 172.21.15.72:0/861130125 conn(0x7f5214102600 msgr2=0x7f5214104a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:26:29.622 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:28.993+0000 7f521b062700 1 -- 172.21.15.72:0/861130125 shutdown_connections 2024-03-22T01:26:29.622 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:28.993+0000 7f521b062700 1 -- 172.21.15.72:0/861130125 wait complete. 2024-03-22T01:26:29.623 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:28.994+0000 7f521b062700 1 Processor -- start 2024-03-22T01:26:29.623 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:28.994+0000 7f521b062700 1 -- start start 2024-03-22T01:26:29.623 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:28.994+0000 7f521b062700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5214107020 0x7f521407ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:26:29.623 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:28.994+0000 7f521b062700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f521410b5d0 con 0x7f5214107020 2024-03-22T01:26:29.623 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:28.994+0000 7f5218dfe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5214107020 0x7f521407ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:26:29.623 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:28.994+0000 7f5218dfe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5214107020 0x7f521407ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:40804/0 (socket says 172.21.15.72:40804) 2024-03-22T01:26:29.623 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:28.994+0000 7f5218dfe700 1 -- 172.21.15.72:0/2091648495 learned_addr learned my addr 172.21.15.72:0/2091648495 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:26:29.623 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:28.994+0000 7f5218dfe700 1 -- 172.21.15.72:0/2091648495 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f521419ed70 con 0x7f5214107020 2024-03-22T01:26:29.623 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:28.994+0000 7f5218dfe700 1 --2- 172.21.15.72:0/2091648495 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5214107020 0x7f521407ba90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f5204009fb0 tx=0x7f5204011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:26:29.623 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:28.995+0000 7f5211ffb700 1 -- 172.21.15.72:0/2091648495 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f52040038b0 con 0x7f5214107020 2024-03-22T01:26:29.624 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:28.995+0000 7f5211ffb700 1 -- 172.21.15.72:0/2091648495 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f5204011e50 con 0x7f5214107020 2024-03-22T01:26:29.624 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:28.995+0000 7f521b062700 1 -- 172.21.15.72:0/2091648495 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f521419ef40 con 0x7f5214107020 2024-03-22T01:26:29.624 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:28.995+0000 7f5211ffb700 1 -- 172.21.15.72:0/2091648495 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5204019550 con 0x7f5214107020 2024-03-22T01:26:29.624 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:28.995+0000 7f521b062700 1 -- 172.21.15.72:0/2091648495 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f521419f360 con 0x7f5214107020 2024-03-22T01:26:29.624 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:28.995+0000 7f5211ffb700 1 -- 172.21.15.72:0/2091648495 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f5204019be0 con 0x7f5214107020 2024-03-22T01:26:29.624 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:28.995+0000 7f5211ffb700 1 -- 172.21.15.72:0/2091648495 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f5204024730 con 0x7f5214107020 2024-03-22T01:26:29.624 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:28.996+0000 7f521b062700 1 -- 172.21.15.72:0/2091648495 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5214063cb0 con 0x7f5214107020 2024-03-22T01:26:29.624 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:28.998+0000 7f5211ffb700 1 -- 172.21.15.72:0/2091648495 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f5204003a50 con 0x7f5214107020 2024-03-22T01:26:29.624 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:29.063+0000 7f521b062700 1 -- 172.21.15.72:0/2091648495 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f521419f680 con 0x7f5214107020 2024-03-22T01:26:29.624 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:29.065+0000 7f5211ffb700 1 -- 172.21.15.72:0/2091648495 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7f52040249d0 con 0x7f5214107020 2024-03-22T01:26:29.624 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:29.066+0000 7f521b062700 1 -- 172.21.15.72:0/2091648495 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5214107020 msgr2=0x7f521407ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:26:29.624 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:29.066+0000 7f521b062700 1 --2- 172.21.15.72:0/2091648495 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5214107020 0x7f521407ba90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f5204009fb0 tx=0x7f5204011bf0 comp rx=0 tx=0).stop 2024-03-22T01:26:29.624 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:29.066+0000 7f521b062700 1 -- 172.21.15.72:0/2091648495 shutdown_connections 2024-03-22T01:26:29.625 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:29.066+0000 7f521b062700 1 --2- 172.21.15.72:0/2091648495 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5214107020 0x7f521407ba90 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:26:29.625 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:29.066+0000 7f521b062700 1 -- 172.21.15.72:0/2091648495 >> 172.21.15.72:0/2091648495 conn(0x7f5214102600 msgr2=0x7f52141030a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:26:29.625 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:29.066+0000 7f521b062700 1 -- 172.21.15.72:0/2091648495 shutdown_connections 2024-03-22T01:26:29.625 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:29.066+0000 7f521b062700 1 -- 172.21.15.72:0/2091648495 wait complete. 2024-03-22T01:26:29.625 INFO:teuthology.orchestra.run.smithi072.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-22T01:26:29.639 INFO:teuthology.orchestra.run.smithi072.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-22T01:26:29.639 INFO:teuthology.orchestra.run.smithi072.stdout:Creating mgr... 2024-03-22T01:26:29.639 INFO:teuthology.orchestra.run.smithi072.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-22T01:26:29.640 INFO:teuthology.orchestra.run.smithi072.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-22T01:26:29.873 INFO:teuthology.orchestra.run.smithi072.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mgr.x 2024-03-22T01:26:29.873 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stderr Failed to reset failed state of unit ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mgr.x.service: Unit ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mgr.x.service not loaded. 2024-03-22T01:26:30.018 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-022a99bc-e7eb-11ee-95cd-87774f69a715.target.wants/ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mgr.x.service → /etc/systemd/system/ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@.service. 2024-03-22T01:26:30.235 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:30 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/2091648495' entity='client.admin' 2024-03-22T01:26:30.500 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:30 smithi072 podman[32629]: 2024-03-22 01:26:30.428182265 +0000 UTC m=+0.289635031 container start 8322249439be5b27b6af56646022c3966777c1eb0f4f46a617dc035fcd8237fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, ceph=True) 2024-03-22T01:26:30.500 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:30 smithi072 bash[32629]: 8322249439be5b27b6af56646022c3966777c1eb0f4f46a617dc035fcd8237fe 2024-03-22T01:26:30.512 INFO:teuthology.orchestra.run.smithi072.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-22T01:26:30.513 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stdout disabled 2024-03-22T01:26:30.520 INFO:teuthology.orchestra.run.smithi072.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-22T01:26:30.520 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stdout inactive 2024-03-22T01:26:30.520 INFO:teuthology.orchestra.run.smithi072.stdout:firewalld.service is not enabled 2024-03-22T01:26:30.520 INFO:teuthology.orchestra.run.smithi072.stdout:Not possible to enable service . firewalld.service is not available 2024-03-22T01:26:30.528 INFO:teuthology.orchestra.run.smithi072.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-22T01:26:30.528 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stdout disabled 2024-03-22T01:26:30.535 INFO:teuthology.orchestra.run.smithi072.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-22T01:26:30.535 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stdout inactive 2024-03-22T01:26:30.535 INFO:teuthology.orchestra.run.smithi072.stdout:firewalld.service is not enabled 2024-03-22T01:26:30.535 INFO:teuthology.orchestra.run.smithi072.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-22T01:26:30.535 INFO:teuthology.orchestra.run.smithi072.stdout:Waiting for mgr to start... 2024-03-22T01:26:30.535 INFO:teuthology.orchestra.run.smithi072.stdout:Waiting for mgr... 2024-03-22T01:26:30.752 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:30 smithi072 systemd[1]: Started Ceph mgr.x for 022a99bc-e7eb-11ee-95cd-87774f69a715. 2024-03-22T01:26:31.038 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:30 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:26:30.984+0000 7fd31f069200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-22T01:26:31.336 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:31 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/3860855014' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-22T01:26:31.337 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:31 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:26:31.083+0000 7fd31f069200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-22T01:26:31.619 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout 2024-03-22T01:26:31.619 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout { 2024-03-22T01:26:31.619 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "fsid": "022a99bc-e7eb-11ee-95cd-87774f69a715", 2024-03-22T01:26:31.620 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "health": { 2024-03-22T01:26:31.620 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-22T01:26:31.620 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-22T01:26:31.620 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-22T01:26:31.620 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-03-22T01:26:31.620 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-22T01:26:31.620 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-22T01:26:31.620 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout 0 2024-03-22T01:26:31.620 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout ], 2024-03-22T01:26:31.620 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-22T01:26:31.620 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "a" 2024-03-22T01:26:31.620 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout ], 2024-03-22T01:26:31.620 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-22T01:26:31.620 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-22T01:26:31.620 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-22T01:26:31.621 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-22T01:26:31.621 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-22T01:26:31.621 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-03-22T01:26:31.621 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-22T01:26:31.621 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-22T01:26:31.621 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-22T01:26:31.621 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-22T01:26:31.621 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-22T01:26:31.629 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-22T01:26:31.629 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-22T01:26:31.629 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-22T01:26:31.630 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-03-22T01:26:31.630 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-22T01:26:31.630 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-22T01:26:31.630 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-22T01:26:31.630 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-22T01:26:31.630 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-22T01:26:31.630 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-22T01:26:31.630 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-22T01:26:31.630 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-22T01:26:31.630 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-22T01:26:31.630 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-03-22T01:26:31.630 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-22T01:26:31.631 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-22T01:26:31.631 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-22T01:26:31.631 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-22T01:26:31.631 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-03-22T01:26:31.631 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-22T01:26:31.631 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-22T01:26:31.631 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-22T01:26:31.631 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-22T01:26:31.631 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-22T01:26:31.631 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-22T01:26:31.631 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "restful" 2024-03-22T01:26:31.631 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout ], 2024-03-22T01:26:31.631 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-22T01:26:31.631 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-03-22T01:26:31.631 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-22T01:26:31.632 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-22T01:26:31.632 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "modified": "2024-03-22T01:26:23.894300+0000", 2024-03-22T01:26:31.632 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-22T01:26:31.632 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-03-22T01:26:31.632 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-22T01:26:31.632 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout } 2024-03-22T01:26:31.632 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:30.958+0000 7f4b87fd0700 1 Processor -- start 2024-03-22T01:26:31.632 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:30.958+0000 7f4b87fd0700 1 -- start start 2024-03-22T01:26:31.632 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:30.958+0000 7f4b87fd0700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4b80108be0 0x7f4b8010afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:26:31.632 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:30.958+0000 7f4b87fd0700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f4b8007be30 con 0x7f4b80108be0 2024-03-22T01:26:31.632 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:30.958+0000 7f4b85d6c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4b80108be0 0x7f4b8010afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:26:31.632 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:30.958+0000 7f4b85d6c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4b80108be0 0x7f4b8010afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56090/0 (socket says 172.21.15.72:56090) 2024-03-22T01:26:31.632 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:30.958+0000 7f4b85d6c700 1 -- 172.21.15.72:0/3580999489 learned_addr learned my addr 172.21.15.72:0/3580999489 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:26:31.632 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:30.959+0000 7f4b85d6c700 1 -- 172.21.15.72:0/3580999489 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b8010b510 con 0x7f4b80108be0 2024-03-22T01:26:31.633 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:30.959+0000 7f4b85d6c700 1 --2- 172.21.15.72:0/3580999489 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4b80108be0 0x7f4b8010afd0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c0098a0 tx=0x7f4b7c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34df53fc1088f460 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:26:31.633 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:30.959+0000 7f4b84d6a700 1 -- 172.21.15.72:0/3580999489 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4b7c004160 con 0x7f4b80108be0 2024-03-22T01:26:31.633 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:30.959+0000 7f4b84d6a700 1 -- 172.21.15.72:0/3580999489 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4b7c004300 con 0x7f4b80108be0 2024-03-22T01:26:31.633 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:30.959+0000 7f4b84d6a700 1 -- 172.21.15.72:0/3580999489 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4b7c011b50 con 0x7f4b80108be0 2024-03-22T01:26:31.633 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:30.960+0000 7f4b87fd0700 1 -- 172.21.15.72:0/3580999489 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4b80108be0 msgr2=0x7f4b8010afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:26:31.633 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:30.960+0000 7f4b87fd0700 1 --2- 172.21.15.72:0/3580999489 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4b80108be0 0x7f4b8010afd0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c0098a0 tx=0x7f4b7c009bb0 comp rx=0 tx=0).stop 2024-03-22T01:26:31.633 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:30.960+0000 7f4b87fd0700 1 -- 172.21.15.72:0/3580999489 shutdown_connections 2024-03-22T01:26:31.633 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:30.960+0000 7f4b87fd0700 1 --2- 172.21.15.72:0/3580999489 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4b80108be0 0x7f4b8010afd0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:26:31.633 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:30.960+0000 7f4b87fd0700 1 -- 172.21.15.72:0/3580999489 >> 172.21.15.72:0/3580999489 conn(0x7f4b80102630 msgr2=0x7f4b80104a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:26:31.633 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:30.960+0000 7f4b87fd0700 1 -- 172.21.15.72:0/3580999489 shutdown_connections 2024-03-22T01:26:31.633 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:30.960+0000 7f4b87fd0700 1 -- 172.21.15.72:0/3580999489 wait complete. 2024-03-22T01:26:31.633 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:30.961+0000 7f4b87fd0700 1 Processor -- start 2024-03-22T01:26:31.633 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:30.961+0000 7f4b87fd0700 1 -- start start 2024-03-22T01:26:31.633 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:30.961+0000 7f4b87fd0700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4b80108be0 0x7f4b8019c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:26:31.633 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:30.961+0000 7f4b87fd0700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f4b8010c4e0 con 0x7f4b80108be0 2024-03-22T01:26:31.634 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:30.961+0000 7f4b85d6c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4b80108be0 0x7f4b8019c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:26:31.634 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:30.961+0000 7f4b85d6c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4b80108be0 0x7f4b8019c050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56096/0 (socket says 172.21.15.72:56096) 2024-03-22T01:26:31.634 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:30.961+0000 7f4b85d6c700 1 -- 172.21.15.72:0/3860855014 learned_addr learned my addr 172.21.15.72:0/3860855014 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:26:31.634 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:30.961+0000 7f4b85d6c700 1 -- 172.21.15.72:0/3860855014 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b8019c590 con 0x7f4b80108be0 2024-03-22T01:26:31.634 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:30.962+0000 7f4b85d6c700 1 --2- 172.21.15.72:0/3860855014 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4b80108be0 0x7f4b8019c050 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c011b60 tx=0x7f4b7c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:26:31.634 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:30.962+0000 7f4b76ffd700 1 -- 172.21.15.72:0/3860855014 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4b7c003970 con 0x7f4b80108be0 2024-03-22T01:26:31.634 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:30.962+0000 7f4b76ffd700 1 -- 172.21.15.72:0/3860855014 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4b7c003b10 con 0x7f4b80108be0 2024-03-22T01:26:31.634 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:30.962+0000 7f4b87fd0700 1 -- 172.21.15.72:0/3860855014 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b8019c820 con 0x7f4b80108be0 2024-03-22T01:26:31.634 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:30.962+0000 7f4b76ffd700 1 -- 172.21.15.72:0/3860855014 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4b7c024610 con 0x7f4b80108be0 2024-03-22T01:26:31.634 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:30.962+0000 7f4b87fd0700 1 -- 172.21.15.72:0/3860855014 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b8019f510 con 0x7f4b80108be0 2024-03-22T01:26:31.634 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:30.962+0000 7f4b76ffd700 1 -- 172.21.15.72:0/3860855014 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4b7c024c50 con 0x7f4b80108be0 2024-03-22T01:26:31.634 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:30.962+0000 7f4b76ffd700 1 -- 172.21.15.72:0/3860855014 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f4b7c011e00 con 0x7f4b80108be0 2024-03-22T01:26:31.634 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:30.963+0000 7f4b87fd0700 1 -- 172.21.15.72:0/3860855014 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b80063cb0 con 0x7f4b80108be0 2024-03-22T01:26:31.635 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:30.965+0000 7f4b76ffd700 1 -- 172.21.15.72:0/3860855014 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f4b7c01a9f0 con 0x7f4b80108be0 2024-03-22T01:26:31.635 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:31.037+0000 7f4b87fd0700 1 -- 172.21.15.72:0/3860855014 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f4b8019f6e0 con 0x7f4b80108be0 2024-03-22T01:26:31.635 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:31.037+0000 7f4b76ffd700 1 -- 172.21.15.72:0/3860855014 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f4b7c01a430 con 0x7f4b80108be0 2024-03-22T01:26:31.635 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:31.038+0000 7f4b87fd0700 1 -- 172.21.15.72:0/3860855014 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4b80108be0 msgr2=0x7f4b8019c050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:26:31.635 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:31.038+0000 7f4b87fd0700 1 --2- 172.21.15.72:0/3860855014 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4b80108be0 0x7f4b8019c050 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c011b60 tx=0x7f4b7c011c40 comp rx=0 tx=0).stop 2024-03-22T01:26:31.635 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:31.038+0000 7f4b87fd0700 1 -- 172.21.15.72:0/3860855014 shutdown_connections 2024-03-22T01:26:31.635 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:31.038+0000 7f4b87fd0700 1 --2- 172.21.15.72:0/3860855014 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4b80108be0 0x7f4b8019c050 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:26:31.635 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:31.038+0000 7f4b87fd0700 1 -- 172.21.15.72:0/3860855014 >> 172.21.15.72:0/3860855014 conn(0x7f4b80102630 msgr2=0x7f4b80103120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:26:31.635 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:31.039+0000 7f4b87fd0700 1 -- 172.21.15.72:0/3860855014 shutdown_connections 2024-03-22T01:26:31.635 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:31.039+0000 7f4b87fd0700 1 -- 172.21.15.72:0/3860855014 wait complete. 2024-03-22T01:26:31.635 INFO:teuthology.orchestra.run.smithi072.stdout:mgr not available, waiting (1/15)... 2024-03-22T01:26:31.872 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:31 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:26:31.727+0000 7fd31f069200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-22T01:26:31.872 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:31 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:26:31.817+0000 7fd31f069200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-22T01:26:32.372 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:31 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:26:31.958+0000 7fd31f069200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-22T01:26:32.372 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:32 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:26:32.051+0000 7fd31f069200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-22T01:26:32.372 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:32 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:26:32.133+0000 7fd31f069200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-22T01:26:32.679 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:32 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:26:32.393+0000 7fd31f069200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-22T01:26:32.680 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:32 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:26:32.502+0000 7fd31f069200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-22T01:26:32.680 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:32 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:26:32.586+0000 7fd31f069200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-22T01:26:32.680 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:32 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:26:32.680+0000 7fd31f069200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-22T01:26:33.122 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:32 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:26:32.861+0000 7fd31f069200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-22T01:26:34.035 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:33 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:26:33.829+0000 7fd31f069200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-22T01:26:34.035 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:33 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:26:33.960+0000 7fd31f069200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-22T01:26:34.299 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:34 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/2902603118' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-22T01:26:34.585 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:34 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:26:34.300+0000 7fd31f069200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-22T01:26:34.585 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:34 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:26:34.391+0000 7fd31f069200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-22T01:26:34.585 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:34 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:26:34.586+0000 7fd31f069200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-22T01:26:34.668 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout 2024-03-22T01:26:34.668 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout { 2024-03-22T01:26:34.668 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "fsid": "022a99bc-e7eb-11ee-95cd-87774f69a715", 2024-03-22T01:26:34.668 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "health": { 2024-03-22T01:26:34.668 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-22T01:26:34.668 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-22T01:26:34.668 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-22T01:26:34.668 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-03-22T01:26:34.669 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-22T01:26:34.669 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-22T01:26:34.669 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout 0 2024-03-22T01:26:34.669 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout ], 2024-03-22T01:26:34.669 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-22T01:26:34.669 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "a" 2024-03-22T01:26:34.669 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout ], 2024-03-22T01:26:34.669 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-03-22T01:26:34.669 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-22T01:26:34.669 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-22T01:26:34.669 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-22T01:26:34.669 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-22T01:26:34.669 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-03-22T01:26:34.669 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-22T01:26:34.669 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-22T01:26:34.670 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-22T01:26:34.670 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-22T01:26:34.670 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-22T01:26:34.678 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-22T01:26:34.678 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-22T01:26:34.678 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-22T01:26:34.678 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-03-22T01:26:34.678 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-22T01:26:34.678 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-22T01:26:34.678 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-22T01:26:34.678 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-22T01:26:34.678 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-22T01:26:34.678 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-22T01:26:34.679 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-22T01:26:34.679 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-22T01:26:34.679 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-22T01:26:34.679 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-03-22T01:26:34.679 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-22T01:26:34.679 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-22T01:26:34.679 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-22T01:26:34.679 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-22T01:26:34.679 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-03-22T01:26:34.679 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-22T01:26:34.679 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-22T01:26:34.679 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-22T01:26:34.679 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-22T01:26:34.679 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-22T01:26:34.680 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-22T01:26:34.680 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "restful" 2024-03-22T01:26:34.680 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout ], 2024-03-22T01:26:34.680 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-22T01:26:34.680 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-03-22T01:26:34.680 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-22T01:26:34.680 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-22T01:26:34.680 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "modified": "2024-03-22T01:26:23.894300+0000", 2024-03-22T01:26:34.680 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-22T01:26:34.680 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-03-22T01:26:34.680 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-22T01:26:34.680 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout } 2024-03-22T01:26:34.680 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:34.036+0000 7f3b3cacd700 1 Processor -- start 2024-03-22T01:26:34.680 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:34.036+0000 7f3b3cacd700 1 -- start start 2024-03-22T01:26:34.680 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:34.036+0000 7f3b3cacd700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3b38106440 0x7f3b38106860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:26:34.681 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:34.036+0000 7f3b3cacd700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f3b38106e30 con 0x7f3b38106440 2024-03-22T01:26:34.681 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:34.037+0000 7f3b3659c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3b38106440 0x7f3b38106860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:26:34.681 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:34.037+0000 7f3b3659c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3b38106440 0x7f3b38106860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56110/0 (socket says 172.21.15.72:56110) 2024-03-22T01:26:34.681 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:34.037+0000 7f3b3659c700 1 -- 172.21.15.72:0/1270832052 learned_addr learned my addr 172.21.15.72:0/1270832052 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:26:34.681 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:34.037+0000 7f3b3659c700 1 -- 172.21.15.72:0/1270832052 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b381093e0 con 0x7f3b38106440 2024-03-22T01:26:34.681 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:34.037+0000 7f3b3659c700 1 --2- 172.21.15.72:0/1270832052 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3b38106440 0x7f3b38106860 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3b280098a0 tx=0x7f3b28009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3a76e009bb3d9b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:26:34.681 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:34.037+0000 7f3b3559a700 1 -- 172.21.15.72:0/1270832052 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3b28004160 con 0x7f3b38106440 2024-03-22T01:26:34.681 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:34.037+0000 7f3b3559a700 1 -- 172.21.15.72:0/1270832052 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f3b28004300 con 0x7f3b38106440 2024-03-22T01:26:34.681 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:34.037+0000 7f3b3559a700 1 -- 172.21.15.72:0/1270832052 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3b28011b50 con 0x7f3b38106440 2024-03-22T01:26:34.681 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:34.038+0000 7f3b3cacd700 1 -- 172.21.15.72:0/1270832052 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3b38106440 msgr2=0x7f3b38106860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:26:34.681 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:34.038+0000 7f3b3cacd700 1 --2- 172.21.15.72:0/1270832052 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3b38106440 0x7f3b38106860 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3b280098a0 tx=0x7f3b28009bb0 comp rx=0 tx=0).stop 2024-03-22T01:26:34.681 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:34.038+0000 7f3b3cacd700 1 -- 172.21.15.72:0/1270832052 shutdown_connections 2024-03-22T01:26:34.681 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:34.038+0000 7f3b3cacd700 1 --2- 172.21.15.72:0/1270832052 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3b38106440 0x7f3b38106860 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:26:34.682 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:34.038+0000 7f3b3cacd700 1 -- 172.21.15.72:0/1270832052 >> 172.21.15.72:0/1270832052 conn(0x7f3b38101e20 msgr2=0x7f3b38104240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:26:34.682 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:34.038+0000 7f3b3cacd700 1 -- 172.21.15.72:0/1270832052 shutdown_connections 2024-03-22T01:26:34.682 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:34.038+0000 7f3b3cacd700 1 -- 172.21.15.72:0/1270832052 wait complete. 2024-03-22T01:26:34.682 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:34.039+0000 7f3b3cacd700 1 Processor -- start 2024-03-22T01:26:34.682 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:34.039+0000 7f3b3cacd700 1 -- start start 2024-03-22T01:26:34.682 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:34.039+0000 7f3b3cacd700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3b38106440 0x7f3b3819bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:26:34.682 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:34.039+0000 7f3b3cacd700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f3b3810c790 con 0x7f3b38106440 2024-03-22T01:26:34.682 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:34.040+0000 7f3b3659c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3b38106440 0x7f3b3819bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:26:34.682 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:34.040+0000 7f3b3659c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3b38106440 0x7f3b3819bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56116/0 (socket says 172.21.15.72:56116) 2024-03-22T01:26:34.682 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:34.040+0000 7f3b3659c700 1 -- 172.21.15.72:0/2902603118 learned_addr learned my addr 172.21.15.72:0/2902603118 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:26:34.682 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:34.040+0000 7f3b3659c700 1 -- 172.21.15.72:0/2902603118 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b3819c500 con 0x7f3b38106440 2024-03-22T01:26:34.682 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:34.040+0000 7f3b3659c700 1 --2- 172.21.15.72:0/2902603118 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3b38106440 0x7f3b3819bfc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3b28011670 tx=0x7f3b28011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:26:34.682 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:34.040+0000 7f3b277fe700 1 -- 172.21.15.72:0/2902603118 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3b280039c0 con 0x7f3b38106440 2024-03-22T01:26:34.682 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:34.040+0000 7f3b277fe700 1 -- 172.21.15.72:0/2902603118 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f3b28011e50 con 0x7f3b38106440 2024-03-22T01:26:34.683 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:34.040+0000 7f3b277fe700 1 -- 172.21.15.72:0/2902603118 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3b2801a5c0 con 0x7f3b38106440 2024-03-22T01:26:34.683 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:34.041+0000 7f3b3cacd700 1 -- 172.21.15.72:0/2902603118 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b3819c790 con 0x7f3b38106440 2024-03-22T01:26:34.683 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:34.041+0000 7f3b3cacd700 1 -- 172.21.15.72:0/2902603118 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b3819f480 con 0x7f3b38106440 2024-03-22T01:26:34.683 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:34.042+0000 7f3b277fe700 1 -- 172.21.15.72:0/2902603118 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f3b2801abc0 con 0x7f3b38106440 2024-03-22T01:26:34.683 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:34.042+0000 7f3b277fe700 1 -- 172.21.15.72:0/2902603118 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f3b2802d720 con 0x7f3b38106440 2024-03-22T01:26:34.683 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:34.042+0000 7f3b3cacd700 1 -- 172.21.15.72:0/2902603118 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b38063cb0 con 0x7f3b38106440 2024-03-22T01:26:34.683 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:34.044+0000 7f3b277fe700 1 -- 172.21.15.72:0/2902603118 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f3b28029060 con 0x7f3b38106440 2024-03-22T01:26:34.683 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:34.117+0000 7f3b3cacd700 1 -- 172.21.15.72:0/2902603118 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f3b3819f9c0 con 0x7f3b38106440 2024-03-22T01:26:34.683 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:34.117+0000 7f3b277fe700 1 -- 172.21.15.72:0/2902603118 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f3b28003b60 con 0x7f3b38106440 2024-03-22T01:26:34.683 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:34.118+0000 7f3b3cacd700 1 -- 172.21.15.72:0/2902603118 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3b38106440 msgr2=0x7f3b3819bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:26:34.683 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:34.118+0000 7f3b3cacd700 1 --2- 172.21.15.72:0/2902603118 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3b38106440 0x7f3b3819bfc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3b28011670 tx=0x7f3b28011b60 comp rx=0 tx=0).stop 2024-03-22T01:26:34.683 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:34.118+0000 7f3b3cacd700 1 -- 172.21.15.72:0/2902603118 shutdown_connections 2024-03-22T01:26:34.683 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:34.118+0000 7f3b3cacd700 1 --2- 172.21.15.72:0/2902603118 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3b38106440 0x7f3b3819bfc0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:26:34.683 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:34.118+0000 7f3b3cacd700 1 -- 172.21.15.72:0/2902603118 >> 172.21.15.72:0/2902603118 conn(0x7f3b38101e20 msgr2=0x7f3b381028a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:26:34.684 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:34.118+0000 7f3b3cacd700 1 -- 172.21.15.72:0/2902603118 shutdown_connections 2024-03-22T01:26:34.684 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:34.118+0000 7f3b3cacd700 1 -- 172.21.15.72:0/2902603118 wait complete. 2024-03-22T01:26:34.684 INFO:teuthology.orchestra.run.smithi072.stdout:mgr not available, waiting (2/15)... 2024-03-22T01:26:34.871 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:34 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:26:34.771+0000 7fd31f069200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-22T01:26:35.304 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:35 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:26:35.053+0000 7fd31f069200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-22T01:26:35.304 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:35 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:26:35.305+0000 7fd31f069200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-22T01:26:35.621 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:35 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:26:35.540+0000 7fd31f069200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-22T01:26:36.622 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:36 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:26:36.328+0000 7fd31f069200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-22T01:26:37.035 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:36 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:26:36.894+0000 7fd31f069200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-22T01:26:37.035 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:36 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:26:36.991+0000 7fd31f069200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-22T01:26:37.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:37 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/3072610119' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-22T01:26:37.372 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:37 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:26:37.216+0000 7fd31f069200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-22T01:26:37.675 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout 2024-03-22T01:26:37.675 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout { 2024-03-22T01:26:37.675 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "fsid": "022a99bc-e7eb-11ee-95cd-87774f69a715", 2024-03-22T01:26:37.675 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "health": { 2024-03-22T01:26:37.675 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-22T01:26:37.675 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-22T01:26:37.675 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-22T01:26:37.675 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-03-22T01:26:37.675 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-22T01:26:37.676 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-22T01:26:37.676 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout 0 2024-03-22T01:26:37.676 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout ], 2024-03-22T01:26:37.676 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-22T01:26:37.676 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "a" 2024-03-22T01:26:37.676 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout ], 2024-03-22T01:26:37.676 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-03-22T01:26:37.676 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-22T01:26:37.676 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-22T01:26:37.676 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-22T01:26:37.676 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-22T01:26:37.676 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-03-22T01:26:37.676 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-22T01:26:37.676 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-22T01:26:37.676 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-22T01:26:37.677 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-22T01:26:37.677 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-22T01:26:37.677 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-22T01:26:37.677 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-22T01:26:37.677 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-22T01:26:37.677 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-03-22T01:26:37.677 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-22T01:26:37.677 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-22T01:26:37.677 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-22T01:26:37.677 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-22T01:26:37.677 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-22T01:26:37.677 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-22T01:26:37.677 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-22T01:26:37.677 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-22T01:26:37.678 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-22T01:26:37.678 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-03-22T01:26:37.678 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-22T01:26:37.678 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-22T01:26:37.678 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-22T01:26:37.678 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-22T01:26:37.678 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-03-22T01:26:37.678 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-22T01:26:37.678 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-22T01:26:37.678 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-22T01:26:37.678 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-22T01:26:37.678 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-22T01:26:37.678 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-22T01:26:37.678 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "restful" 2024-03-22T01:26:37.679 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout ], 2024-03-22T01:26:37.679 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-22T01:26:37.679 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-03-22T01:26:37.679 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-22T01:26:37.679 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-22T01:26:37.679 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "modified": "2024-03-22T01:26:23.894300+0000", 2024-03-22T01:26:37.679 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-22T01:26:37.679 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-03-22T01:26:37.679 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-22T01:26:37.679 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout } 2024-03-22T01:26:37.679 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:37.036+0000 7f54ac1e0700 1 Processor -- start 2024-03-22T01:26:37.679 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:37.036+0000 7f54ac1e0700 1 -- start start 2024-03-22T01:26:37.679 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:37.036+0000 7f54ac1e0700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54a4106ff0 0x7f54a4107410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:26:37.679 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:37.036+0000 7f54ac1e0700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f54a41079e0 con 0x7f54a4106ff0 2024-03-22T01:26:37.679 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:37.036+0000 7f54a9f7c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54a4106ff0 0x7f54a4107410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:26:37.680 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:37.036+0000 7f54a9f7c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54a4106ff0 0x7f54a4107410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56124/0 (socket says 172.21.15.72:56124) 2024-03-22T01:26:37.680 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:37.036+0000 7f54a9f7c700 1 -- 172.21.15.72:0/2546019832 learned_addr learned my addr 172.21.15.72:0/2546019832 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:26:37.680 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:37.037+0000 7f54a9f7c700 1 -- 172.21.15.72:0/2546019832 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54a41081f0 con 0x7f54a4106ff0 2024-03-22T01:26:37.680 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:37.037+0000 7f54a9f7c700 1 --2- 172.21.15.72:0/2546019832 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54a4106ff0 0x7f54a4107410 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f54a000bb40 tx=0x7f54a000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68e0284b056bba80 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:26:37.680 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:37.037+0000 7f54a8f7a700 1 -- 172.21.15.72:0/2546019832 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f54a000dbe0 con 0x7f54a4106ff0 2024-03-22T01:26:37.680 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:37.037+0000 7f54a8f7a700 1 -- 172.21.15.72:0/2546019832 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f54a00044f0 con 0x7f54a4106ff0 2024-03-22T01:26:37.680 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:37.037+0000 7f54a8f7a700 1 -- 172.21.15.72:0/2546019832 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f54a0010470 con 0x7f54a4106ff0 2024-03-22T01:26:37.680 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:37.038+0000 7f54ac1e0700 1 -- 172.21.15.72:0/2546019832 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54a4106ff0 msgr2=0x7f54a4107410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:26:37.680 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:37.038+0000 7f54ac1e0700 1 --2- 172.21.15.72:0/2546019832 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54a4106ff0 0x7f54a4107410 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f54a000bb40 tx=0x7f54a000be50 comp rx=0 tx=0).stop 2024-03-22T01:26:37.680 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:37.038+0000 7f54ac1e0700 1 -- 172.21.15.72:0/2546019832 shutdown_connections 2024-03-22T01:26:37.680 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:37.038+0000 7f54ac1e0700 1 --2- 172.21.15.72:0/2546019832 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54a4106ff0 0x7f54a4107410 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:26:37.680 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:37.038+0000 7f54ac1e0700 1 -- 172.21.15.72:0/2546019832 >> 172.21.15.72:0/2546019832 conn(0x7f54a41025b0 msgr2=0x7f54a41049d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:26:37.680 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:37.038+0000 7f54ac1e0700 1 -- 172.21.15.72:0/2546019832 shutdown_connections 2024-03-22T01:26:37.680 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:37.038+0000 7f54ac1e0700 1 -- 172.21.15.72:0/2546019832 wait complete. 2024-03-22T01:26:37.681 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:37.038+0000 7f54ac1e0700 1 Processor -- start 2024-03-22T01:26:37.681 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:37.038+0000 7f54ac1e0700 1 -- start start 2024-03-22T01:26:37.681 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:37.039+0000 7f54ac1e0700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54a4106ff0 0x7f54a4199e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:26:37.681 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:37.039+0000 7f54ac1e0700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f54a407afb0 con 0x7f54a4106ff0 2024-03-22T01:26:37.681 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:37.039+0000 7f54a9f7c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54a4106ff0 0x7f54a4199e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:26:37.681 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:37.039+0000 7f54a9f7c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54a4106ff0 0x7f54a4199e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56126/0 (socket says 172.21.15.72:56126) 2024-03-22T01:26:37.681 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:37.039+0000 7f54a9f7c700 1 -- 172.21.15.72:0/3072610119 learned_addr learned my addr 172.21.15.72:0/3072610119 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:26:37.681 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:37.039+0000 7f54a9f7c700 1 -- 172.21.15.72:0/3072610119 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54a419a390 con 0x7f54a4106ff0 2024-03-22T01:26:37.681 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:37.039+0000 7f54a9f7c700 1 --2- 172.21.15.72:0/3072610119 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54a4106ff0 0x7f54a4199e50 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f54a000b340 tx=0x7f54a000dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:26:37.681 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:37.039+0000 7f549affd700 1 -- 172.21.15.72:0/3072610119 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f54a0027070 con 0x7f54a4106ff0 2024-03-22T01:26:37.681 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:37.039+0000 7f54ac1e0700 1 -- 172.21.15.72:0/3072610119 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54a419a620 con 0x7f54a4106ff0 2024-03-22T01:26:37.681 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:37.039+0000 7f549affd700 1 -- 172.21.15.72:0/3072610119 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f54a0003dd0 con 0x7f54a4106ff0 2024-03-22T01:26:37.681 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:37.039+0000 7f549affd700 1 -- 172.21.15.72:0/3072610119 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f54a0022a90 con 0x7f54a4106ff0 2024-03-22T01:26:37.681 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:37.039+0000 7f54ac1e0700 1 -- 172.21.15.72:0/3072610119 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54a419d310 con 0x7f54a4106ff0 2024-03-22T01:26:37.682 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:37.040+0000 7f549affd700 1 -- 172.21.15.72:0/3072610119 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f54a002a070 con 0x7f54a4106ff0 2024-03-22T01:26:37.682 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:37.040+0000 7f549affd700 1 -- 172.21.15.72:0/3072610119 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f54a001f070 con 0x7f54a4106ff0 2024-03-22T01:26:37.682 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:37.040+0000 7f54ac1e0700 1 -- 172.21.15.72:0/3072610119 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54a4063cb0 con 0x7f54a4106ff0 2024-03-22T01:26:37.682 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:37.042+0000 7f549affd700 1 -- 172.21.15.72:0/3072610119 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f54a0031030 con 0x7f54a4106ff0 2024-03-22T01:26:37.682 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:37.114+0000 7f54ac1e0700 1 -- 172.21.15.72:0/3072610119 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f54a419d4e0 con 0x7f54a4106ff0 2024-03-22T01:26:37.682 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:37.115+0000 7f549affd700 1 -- 172.21.15.72:0/3072610119 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f54a0015070 con 0x7f54a4106ff0 2024-03-22T01:26:37.682 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:37.115+0000 7f54ac1e0700 1 -- 172.21.15.72:0/3072610119 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54a4106ff0 msgr2=0x7f54a4199e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:26:37.682 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:37.115+0000 7f54ac1e0700 1 --2- 172.21.15.72:0/3072610119 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54a4106ff0 0x7f54a4199e50 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f54a000b340 tx=0x7f54a000dd90 comp rx=0 tx=0).stop 2024-03-22T01:26:37.682 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:37.116+0000 7f54ac1e0700 1 -- 172.21.15.72:0/3072610119 shutdown_connections 2024-03-22T01:26:37.682 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:37.116+0000 7f54ac1e0700 1 --2- 172.21.15.72:0/3072610119 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54a4106ff0 0x7f54a4199e50 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:26:37.682 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:37.116+0000 7f54ac1e0700 1 -- 172.21.15.72:0/3072610119 >> 172.21.15.72:0/3072610119 conn(0x7f54a41025b0 msgr2=0x7f54a4190470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:26:37.682 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:37.116+0000 7f54ac1e0700 1 -- 172.21.15.72:0/3072610119 shutdown_connections 2024-03-22T01:26:37.682 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:37.116+0000 7f54ac1e0700 1 -- 172.21.15.72:0/3072610119 wait complete. 2024-03-22T01:26:37.683 INFO:teuthology.orchestra.run.smithi072.stdout:mgr not available, waiting (3/15)... 2024-03-22T01:26:38.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:38 smithi072 ceph-mon[32414]: Activating manager daemon x 2024-03-22T01:26:38.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:38 smithi072 ceph-mon[32414]: mgrmap e2: x(active, starting, since 0.00333309s) 2024-03-22T01:26:38.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:38 smithi072 ceph-mon[32414]: from='mgr.14100 172.21.15.72:0/4132377058' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-22T01:26:38.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:38 smithi072 ceph-mon[32414]: from='mgr.14100 172.21.15.72:0/4132377058' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-22T01:26:38.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:38 smithi072 ceph-mon[32414]: from='mgr.14100 172.21.15.72:0/4132377058' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-22T01:26:38.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:38 smithi072 ceph-mon[32414]: from='mgr.14100 172.21.15.72:0/4132377058' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T01:26:38.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:38 smithi072 ceph-mon[32414]: from='mgr.14100 172.21.15.72:0/4132377058' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-22T01:26:38.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:38 smithi072 ceph-mon[32414]: Manager daemon x is now available 2024-03-22T01:26:38.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:38 smithi072 ceph-mon[32414]: from='mgr.14100 172.21.15.72:0/4132377058' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-22T01:26:38.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:38 smithi072 ceph-mon[32414]: from='mgr.14100 172.21.15.72:0/4132377058' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-22T01:26:38.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:38 smithi072 ceph-mon[32414]: from='mgr.14100 172.21.15.72:0/4132377058' entity='mgr.x' 2024-03-22T01:26:38.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:38 smithi072 ceph-mon[32414]: from='mgr.14100 172.21.15.72:0/4132377058' entity='mgr.x' 2024-03-22T01:26:38.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:38 smithi072 ceph-mon[32414]: from='mgr.14100 172.21.15.72:0/4132377058' entity='mgr.x' 2024-03-22T01:26:39.396 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:39 smithi072 ceph-mon[32414]: mgrmap e3: x(active, since 1.0061s) 2024-03-22T01:26:40.955 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout 2024-03-22T01:26:40.955 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout { 2024-03-22T01:26:40.956 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "fsid": "022a99bc-e7eb-11ee-95cd-87774f69a715", 2024-03-22T01:26:40.956 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "health": { 2024-03-22T01:26:40.956 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-22T01:26:40.956 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-22T01:26:40.956 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-22T01:26:40.956 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-03-22T01:26:40.956 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-22T01:26:40.956 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-22T01:26:40.956 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout 0 2024-03-22T01:26:40.956 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout ], 2024-03-22T01:26:40.956 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-22T01:26:40.956 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "a" 2024-03-22T01:26:40.956 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout ], 2024-03-22T01:26:40.957 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "quorum_age": 11, 2024-03-22T01:26:40.957 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-22T01:26:40.957 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-22T01:26:40.957 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-22T01:26:40.957 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-22T01:26:40.957 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-03-22T01:26:40.957 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-22T01:26:40.957 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-22T01:26:40.958 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-22T01:26:40.958 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-22T01:26:40.958 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-22T01:26:40.958 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-22T01:26:40.958 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-22T01:26:40.958 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-22T01:26:40.958 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-03-22T01:26:40.958 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-22T01:26:40.958 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-22T01:26:40.958 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-22T01:26:40.958 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-22T01:26:40.958 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-22T01:26:40.958 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-22T01:26:40.959 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-22T01:26:40.959 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-22T01:26:40.959 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-22T01:26:40.959 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-03-22T01:26:40.959 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-22T01:26:40.959 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-22T01:26:40.959 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-22T01:26:40.959 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-22T01:26:40.959 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-03-22T01:26:40.959 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-22T01:26:40.959 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-22T01:26:40.959 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-22T01:26:40.959 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-22T01:26:40.959 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-22T01:26:40.959 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-22T01:26:40.959 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "restful" 2024-03-22T01:26:40.960 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout ], 2024-03-22T01:26:40.960 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-22T01:26:40.960 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-03-22T01:26:40.960 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-22T01:26:40.960 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-22T01:26:40.960 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "modified": "2024-03-22T01:26:23.894300+0000", 2024-03-22T01:26:40.960 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-22T01:26:40.960 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-03-22T01:26:40.960 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-22T01:26:40.960 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout } 2024-03-22T01:26:40.960 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:40.066+0000 7fa6edfde700 1 Processor -- start 2024-03-22T01:26:40.960 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:40.066+0000 7fa6edfde700 1 -- start start 2024-03-22T01:26:40.960 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:40.066+0000 7fa6edfde700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa6e8107020 0x7fa6e8107440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:26:40.961 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:40.066+0000 7fa6edfde700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fa6e8107a10 con 0x7fa6e8107020 2024-03-22T01:26:40.961 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:40.067+0000 7fa6e77fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa6e8107020 0x7fa6e8107440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:26:40.961 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:40.067+0000 7fa6e77fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa6e8107020 0x7fa6e8107440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56194/0 (socket says 172.21.15.72:56194) 2024-03-22T01:26:40.961 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:40.067+0000 7fa6e77fe700 1 -- 172.21.15.72:0/4185260295 learned_addr learned my addr 172.21.15.72:0/4185260295 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:26:40.961 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:40.067+0000 7fa6e77fe700 1 -- 172.21.15.72:0/4185260295 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6e8108220 con 0x7fa6e8107020 2024-03-22T01:26:40.961 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:40.067+0000 7fa6e77fe700 1 --2- 172.21.15.72:0/4185260295 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa6e8107020 0x7fa6e8107440 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa6d80098a0 tx=0x7fa6d8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31f5cea439cc9f99 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:26:40.961 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:40.067+0000 7fa6e6ffd700 1 -- 172.21.15.72:0/4185260295 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa6d8004160 con 0x7fa6e8107020 2024-03-22T01:26:40.962 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:40.067+0000 7fa6e6ffd700 1 -- 172.21.15.72:0/4185260295 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa6d8004300 con 0x7fa6e8107020 2024-03-22T01:26:40.962 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:40.067+0000 7fa6e6ffd700 1 -- 172.21.15.72:0/4185260295 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa6d8011b50 con 0x7fa6e8107020 2024-03-22T01:26:40.962 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:40.068+0000 7fa6edfde700 1 -- 172.21.15.72:0/4185260295 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa6e8107020 msgr2=0x7fa6e8107440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:26:40.962 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:40.068+0000 7fa6edfde700 1 --2- 172.21.15.72:0/4185260295 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa6e8107020 0x7fa6e8107440 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa6d80098a0 tx=0x7fa6d8009bb0 comp rx=0 tx=0).stop 2024-03-22T01:26:40.962 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:40.068+0000 7fa6edfde700 1 -- 172.21.15.72:0/4185260295 shutdown_connections 2024-03-22T01:26:40.962 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:40.068+0000 7fa6edfde700 1 --2- 172.21.15.72:0/4185260295 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa6e8107020 0x7fa6e8107440 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:26:40.962 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:40.068+0000 7fa6edfde700 1 -- 172.21.15.72:0/4185260295 >> 172.21.15.72:0/4185260295 conn(0x7fa6e81025c0 msgr2=0x7fa6e8104a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:26:40.962 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:40.068+0000 7fa6edfde700 1 -- 172.21.15.72:0/4185260295 shutdown_connections 2024-03-22T01:26:40.962 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:40.068+0000 7fa6edfde700 1 -- 172.21.15.72:0/4185260295 wait complete. 2024-03-22T01:26:40.962 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:40.069+0000 7fa6edfde700 1 Processor -- start 2024-03-22T01:26:40.962 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:40.069+0000 7fa6edfde700 1 -- start start 2024-03-22T01:26:40.962 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:40.069+0000 7fa6edfde700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa6e8107020 0x7fa6e819c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:26:40.963 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:40.069+0000 7fa6edfde700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fa6e810b5d0 con 0x7fa6e8107020 2024-03-22T01:26:40.963 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:40.069+0000 7fa6e77fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa6e8107020 0x7fa6e819c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:26:40.963 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:40.069+0000 7fa6e77fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa6e8107020 0x7fa6e819c070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56202/0 (socket says 172.21.15.72:56202) 2024-03-22T01:26:40.963 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:40.069+0000 7fa6e77fe700 1 -- 172.21.15.72:0/873465022 learned_addr learned my addr 172.21.15.72:0/873465022 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:26:40.963 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:40.070+0000 7fa6e77fe700 1 -- 172.21.15.72:0/873465022 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6e819c5b0 con 0x7fa6e8107020 2024-03-22T01:26:40.963 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:40.070+0000 7fa6e77fe700 1 --2- 172.21.15.72:0/873465022 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa6e8107020 0x7fa6e819c070 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa6d8011670 tx=0x7fa6d8011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:26:40.963 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:40.070+0000 7fa6e57fa700 1 -- 172.21.15.72:0/873465022 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa6d80039c0 con 0x7fa6e8107020 2024-03-22T01:26:40.963 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:40.070+0000 7fa6e57fa700 1 -- 172.21.15.72:0/873465022 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa6d8011e50 con 0x7fa6e8107020 2024-03-22T01:26:40.963 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:40.070+0000 7fa6edfde700 1 -- 172.21.15.72:0/873465022 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6e819c840 con 0x7fa6e8107020 2024-03-22T01:26:40.963 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:40.070+0000 7fa6e57fa700 1 -- 172.21.15.72:0/873465022 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa6d801a5c0 con 0x7fa6e8107020 2024-03-22T01:26:40.963 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:40.070+0000 7fa6edfde700 1 -- 172.21.15.72:0/873465022 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6e819f530 con 0x7fa6e8107020 2024-03-22T01:26:40.963 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:40.071+0000 7fa6e57fa700 1 -- 172.21.15.72:0/873465022 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 3) ==== 46177+0+0 (secure 0 0 0) 0x7fa6d801a870 con 0x7fa6e8107020 2024-03-22T01:26:40.963 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:40.071+0000 7fa6edfde700 1 -- 172.21.15.72:0/873465022 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6e8063cb0 con 0x7fa6e8107020 2024-03-22T01:26:40.964 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:40.071+0000 7fa6e57fa700 1 --2- 172.21.15.72:0/873465022 >> [v2:172.21.15.72:6800/3518012916,v1:172.21.15.72:6801/3518012916] conn(0x7fa6d40390e0 0x7fa6d403b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:26:40.964 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:40.071+0000 7fa6e57fa700 1 -- 172.21.15.72:0/873465022 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fa6d804d250 con 0x7fa6e8107020 2024-03-22T01:26:40.964 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:40.071+0000 7fa6deffd700 1 --2- 172.21.15.72:0/873465022 >> [v2:172.21.15.72:6800/3518012916,v1:172.21.15.72:6801/3518012916] conn(0x7fa6d40390e0 0x7fa6d403b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:26:40.964 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:40.072+0000 7fa6deffd700 1 --2- 172.21.15.72:0/873465022 >> [v2:172.21.15.72:6800/3518012916,v1:172.21.15.72:6801/3518012916] conn(0x7fa6d40390e0 0x7fa6d403b5a0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa6d0006fd0 tx=0x7fa6d0006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:26:40.964 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:40.076+0000 7fa6e57fa700 1 -- 172.21.15.72:0/873465022 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa6d8019c70 con 0x7fa6e8107020 2024-03-22T01:26:40.964 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:40.234+0000 7fa6e57fa700 1 -- 172.21.15.72:0/873465022 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mgrmap(e 4) ==== 46283+0+0 (secure 0 0 0) 0x7fa6d802d3f0 con 0x7fa6e8107020 2024-03-22T01:26:40.964 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:40.362+0000 7fa6edfde700 1 -- 172.21.15.72:0/873465022 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fa6e819f700 con 0x7fa6e8107020 2024-03-22T01:26:40.964 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:40.363+0000 7fa6e57fa700 1 -- 172.21.15.72:0/873465022 <== mon.0 v2:172.21.15.72:3300/0 8 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fa6e819f700 con 0x7fa6e8107020 2024-03-22T01:26:40.964 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:40.365+0000 7fa6edfde700 1 -- 172.21.15.72:0/873465022 >> [v2:172.21.15.72:6800/3518012916,v1:172.21.15.72:6801/3518012916] conn(0x7fa6d40390e0 msgr2=0x7fa6d403b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:26:40.964 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:40.365+0000 7fa6edfde700 1 --2- 172.21.15.72:0/873465022 >> [v2:172.21.15.72:6800/3518012916,v1:172.21.15.72:6801/3518012916] conn(0x7fa6d40390e0 0x7fa6d403b5a0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa6d0006fd0 tx=0x7fa6d0006e40 comp rx=0 tx=0).stop 2024-03-22T01:26:40.964 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:40.365+0000 7fa6edfde700 1 -- 172.21.15.72:0/873465022 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa6e8107020 msgr2=0x7fa6e819c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:26:40.964 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:40.365+0000 7fa6edfde700 1 --2- 172.21.15.72:0/873465022 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa6e8107020 0x7fa6e819c070 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa6d8011670 tx=0x7fa6d8011b60 comp rx=0 tx=0).stop 2024-03-22T01:26:40.964 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:40.365+0000 7fa6edfde700 1 -- 172.21.15.72:0/873465022 shutdown_connections 2024-03-22T01:26:40.964 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:40.365+0000 7fa6edfde700 1 --2- 172.21.15.72:0/873465022 >> [v2:172.21.15.72:6800/3518012916,v1:172.21.15.72:6801/3518012916] conn(0x7fa6d40390e0 0x7fa6d403b5a0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:26:40.964 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:40.365+0000 7fa6edfde700 1 --2- 172.21.15.72:0/873465022 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa6e8107020 0x7fa6e819c070 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:26:40.965 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:40.365+0000 7fa6edfde700 1 -- 172.21.15.72:0/873465022 >> 172.21.15.72:0/873465022 conn(0x7fa6e81025c0 msgr2=0x7fa6e8103090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:26:40.965 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:40.365+0000 7fa6edfde700 1 -- 172.21.15.72:0/873465022 shutdown_connections 2024-03-22T01:26:40.965 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:40.365+0000 7fa6edfde700 1 -- 172.21.15.72:0/873465022 wait complete. 2024-03-22T01:26:40.965 INFO:teuthology.orchestra.run.smithi072.stdout:mgr is available 2024-03-22T01:26:41.334 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:41 smithi072 ceph-mon[32414]: mgrmap e4: x(active, since 3s) 2024-03-22T01:26:41.334 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:41 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/873465022' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-22T01:26:42.227 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout 2024-03-22T01:26:42.228 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout [global] 2024-03-22T01:26:42.228 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout fsid = 022a99bc-e7eb-11ee-95cd-87774f69a715 2024-03-22T01:26:42.228 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-22T01:26:42.228 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-22T01:26:42.228 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-22T01:26:42.228 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-22T01:26:42.228 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout 2024-03-22T01:26:42.228 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-22T01:26:42.228 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-22T01:26:42.228 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout 2024-03-22T01:26:42.228 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout [osd] 2024-03-22T01:26:42.228 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout filestore_flush_min = 0 2024-03-22T01:26:42.229 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-22T01:26:42.229 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-22T01:26:42.229 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-22T01:26:42.229 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-22T01:26:42.229 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-22T01:26:42.229 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:41.339+0000 7f2e0a4ea700 1 Processor -- start 2024-03-22T01:26:42.229 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:41.339+0000 7f2e0a4ea700 1 -- start start 2024-03-22T01:26:42.229 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:41.339+0000 7f2e0a4ea700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2e0407de00 0x7f2e0407c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:26:42.229 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:41.339+0000 7f2e0a4ea700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f2e0407c840 con 0x7f2e0407de00 2024-03-22T01:26:42.229 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:41.339+0000 7f2e03fff700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2e0407de00 0x7f2e0407c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:26:42.229 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:41.339+0000 7f2e03fff700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2e0407de00 0x7f2e0407c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43182/0 (socket says 172.21.15.72:43182) 2024-03-22T01:26:42.229 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:41.339+0000 7f2e03fff700 1 -- 172.21.15.72:0/2444795953 learned_addr learned my addr 172.21.15.72:0/2444795953 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:26:42.229 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:41.339+0000 7f2e03fff700 1 -- 172.21.15.72:0/2444795953 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e0407c9c0 con 0x7f2e0407de00 2024-03-22T01:26:42.229 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:41.340+0000 7f2e03fff700 1 --2- 172.21.15.72:0/2444795953 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2e0407de00 0x7f2e0407c300 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2df40098a0 tx=0x7f2df4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81e4b3e2942b37c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:26:42.230 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:41.340+0000 7f2e02ffd700 1 -- 172.21.15.72:0/2444795953 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2df4004160 con 0x7f2e0407de00 2024-03-22T01:26:42.230 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:41.340+0000 7f2e02ffd700 1 -- 172.21.15.72:0/2444795953 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2df4004300 con 0x7f2e0407de00 2024-03-22T01:26:42.230 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:41.340+0000 7f2e02ffd700 1 -- 172.21.15.72:0/2444795953 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2df4011b50 con 0x7f2e0407de00 2024-03-22T01:26:42.230 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:41.340+0000 7f2e0a4ea700 1 -- 172.21.15.72:0/2444795953 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2e0407de00 msgr2=0x7f2e0407c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:26:42.230 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:41.340+0000 7f2e0a4ea700 1 --2- 172.21.15.72:0/2444795953 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2e0407de00 0x7f2e0407c300 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2df40098a0 tx=0x7f2df4009bb0 comp rx=0 tx=0).stop 2024-03-22T01:26:42.230 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:41.340+0000 7f2e0a4ea700 1 -- 172.21.15.72:0/2444795953 shutdown_connections 2024-03-22T01:26:42.230 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:41.340+0000 7f2e0a4ea700 1 --2- 172.21.15.72:0/2444795953 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2e0407de00 0x7f2e0407c300 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:26:42.230 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:41.340+0000 7f2e0a4ea700 1 -- 172.21.15.72:0/2444795953 >> 172.21.15.72:0/2444795953 conn(0x7f2e041036f0 msgr2=0x7f2e04105b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:26:42.230 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:41.340+0000 7f2e0a4ea700 1 -- 172.21.15.72:0/2444795953 shutdown_connections 2024-03-22T01:26:42.230 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:41.340+0000 7f2e0a4ea700 1 -- 172.21.15.72:0/2444795953 wait complete. 2024-03-22T01:26:42.230 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:41.341+0000 7f2e0a4ea700 1 Processor -- start 2024-03-22T01:26:42.230 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:41.341+0000 7f2e0a4ea700 1 -- start start 2024-03-22T01:26:42.230 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:41.341+0000 7f2e0a4ea700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2e0407de00 0x7f2e0419e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:26:42.231 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:41.341+0000 7f2e0a4ea700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f2e0410e6c0 con 0x7f2e0407de00 2024-03-22T01:26:42.231 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:41.341+0000 7f2e03fff700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2e0407de00 0x7f2e0419e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:26:42.231 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:41.342+0000 7f2e03fff700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2e0407de00 0x7f2e0419e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43186/0 (socket says 172.21.15.72:43186) 2024-03-22T01:26:42.231 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:41.342+0000 7f2e03fff700 1 -- 172.21.15.72:0/3078629302 learned_addr learned my addr 172.21.15.72:0/3078629302 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:26:42.231 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:41.342+0000 7f2e03fff700 1 -- 172.21.15.72:0/3078629302 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e0419e680 con 0x7f2e0407de00 2024-03-22T01:26:42.231 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:41.342+0000 7f2e03fff700 1 --2- 172.21.15.72:0/3078629302 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2e0407de00 0x7f2e0419e140 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f2df4011b60 tx=0x7f2df4011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:26:42.231 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:41.342+0000 7f2e017fa700 1 -- 172.21.15.72:0/3078629302 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2df4003b20 con 0x7f2e0407de00 2024-03-22T01:26:42.231 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:41.342+0000 7f2e017fa700 1 -- 172.21.15.72:0/3078629302 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2df4003cc0 con 0x7f2e0407de00 2024-03-22T01:26:42.231 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:41.342+0000 7f2e017fa700 1 -- 172.21.15.72:0/3078629302 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2df4024840 con 0x7f2e0407de00 2024-03-22T01:26:42.231 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:41.342+0000 7f2e0a4ea700 1 -- 172.21.15.72:0/3078629302 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e0419e910 con 0x7f2e0407de00 2024-03-22T01:26:42.231 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:41.342+0000 7f2e0a4ea700 1 -- 172.21.15.72:0/3078629302 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e041a1600 con 0x7f2e0407de00 2024-03-22T01:26:42.231 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:41.343+0000 7f2e017fa700 1 -- 172.21.15.72:0/3078629302 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 4) ==== 46283+0+0 (secure 0 0 0) 0x7f2df4022020 con 0x7f2e0407de00 2024-03-22T01:26:42.231 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:41.343+0000 7f2e017fa700 1 --2- 172.21.15.72:0/3078629302 >> [v2:172.21.15.72:6800/3518012916,v1:172.21.15.72:6801/3518012916] conn(0x7f2dec0391f0 0x7f2dec03b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:26:42.232 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:41.343+0000 7f2e017fa700 1 -- 172.21.15.72:0/3078629302 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f2df4011e00 con 0x7f2e0407de00 2024-03-22T01:26:42.232 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:41.343+0000 7f2e0a4ea700 1 -- 172.21.15.72:0/3078629302 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e04063cb0 con 0x7f2e0407de00 2024-03-22T01:26:42.232 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:41.343+0000 7f2e037fe700 1 --2- 172.21.15.72:0/3078629302 >> [v2:172.21.15.72:6800/3518012916,v1:172.21.15.72:6801/3518012916] conn(0x7f2dec0391f0 0x7f2dec03b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:26:42.232 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:41.344+0000 7f2e037fe700 1 --2- 172.21.15.72:0/3078629302 >> [v2:172.21.15.72:6800/3518012916,v1:172.21.15.72:6801/3518012916] conn(0x7f2dec0391f0 0x7f2dec03b6b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2df8006fd0 tx=0x7f2df8006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:26:42.232 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:41.348+0000 7f2e017fa700 1 -- 172.21.15.72:0/3078629302 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2df404cc60 con 0x7f2e0407de00 2024-03-22T01:26:42.232 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:41.539+0000 7f2e0a4ea700 1 -- 172.21.15.72:0/3078629302 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f2e041a17d0 con 0x7f2e0407de00 2024-03-22T01:26:42.232 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:41.540+0000 7f2e017fa700 1 -- 172.21.15.72:0/3078629302 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+398 (secure 0 0 0) 0x7f2df400bc70 con 0x7f2e0407de00 2024-03-22T01:26:42.232 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:41.542+0000 7f2e0a4ea700 1 -- 172.21.15.72:0/3078629302 >> [v2:172.21.15.72:6800/3518012916,v1:172.21.15.72:6801/3518012916] conn(0x7f2dec0391f0 msgr2=0x7f2dec03b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:26:42.232 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:41.542+0000 7f2e0a4ea700 1 --2- 172.21.15.72:0/3078629302 >> [v2:172.21.15.72:6800/3518012916,v1:172.21.15.72:6801/3518012916] conn(0x7f2dec0391f0 0x7f2dec03b6b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2df8006fd0 tx=0x7f2df8006e40 comp rx=0 tx=0).stop 2024-03-22T01:26:42.232 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:41.542+0000 7f2e0a4ea700 1 -- 172.21.15.72:0/3078629302 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2e0407de00 msgr2=0x7f2e0419e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:26:42.232 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:41.542+0000 7f2e0a4ea700 1 --2- 172.21.15.72:0/3078629302 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2e0407de00 0x7f2e0419e140 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f2df4011b60 tx=0x7f2df4011c40 comp rx=0 tx=0).stop 2024-03-22T01:26:42.232 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:41.542+0000 7f2e0a4ea700 1 -- 172.21.15.72:0/3078629302 shutdown_connections 2024-03-22T01:26:42.232 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:41.542+0000 7f2e0a4ea700 1 --2- 172.21.15.72:0/3078629302 >> [v2:172.21.15.72:6800/3518012916,v1:172.21.15.72:6801/3518012916] conn(0x7f2dec0391f0 0x7f2dec03b6b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:26:42.233 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:41.542+0000 7f2e0a4ea700 1 --2- 172.21.15.72:0/3078629302 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2e0407de00 0x7f2e0419e140 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:26:42.233 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:41.542+0000 7f2e0a4ea700 1 -- 172.21.15.72:0/3078629302 >> 172.21.15.72:0/3078629302 conn(0x7f2e041036f0 msgr2=0x7f2e04104170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:26:42.233 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:41.542+0000 7f2e0a4ea700 1 -- 172.21.15.72:0/3078629302 shutdown_connections 2024-03-22T01:26:42.233 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:41.542+0000 7f2e0a4ea700 1 -- 172.21.15.72:0/3078629302 wait complete. 2024-03-22T01:26:42.233 INFO:teuthology.orchestra.run.smithi072.stdout:Enabling cephadm module... 2024-03-22T01:26:42.344 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:42 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/3078629302' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-22T01:26:43.495 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:43 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/3825036538' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-22T01:26:43.496 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:43 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: ignoring --setuser ceph since I am not root 2024-03-22T01:26:43.496 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:43 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: ignoring --setgroup ceph since I am not root 2024-03-22T01:26:43.747 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:42.615+0000 7fe61453e700 1 Processor -- start 2024-03-22T01:26:43.747 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:42.615+0000 7fe61453e700 1 -- start start 2024-03-22T01:26:43.748 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:42.615+0000 7fe61453e700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe60c108b90 0x7fe60c10af80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:26:43.748 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:42.615+0000 7fe61453e700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fe60c07be70 con 0x7fe60c108b90 2024-03-22T01:26:43.748 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:42.615+0000 7fe6122da700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe60c108b90 0x7fe60c10af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:26:43.748 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:42.615+0000 7fe6122da700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe60c108b90 0x7fe60c10af80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43194/0 (socket says 172.21.15.72:43194) 2024-03-22T01:26:43.748 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:42.615+0000 7fe6122da700 1 -- 172.21.15.72:0/1072300723 learned_addr learned my addr 172.21.15.72:0/1072300723 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:26:43.748 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:42.615+0000 7fe6122da700 1 -- 172.21.15.72:0/1072300723 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe60c10b4c0 con 0x7fe60c108b90 2024-03-22T01:26:43.748 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:42.616+0000 7fe6122da700 1 --2- 172.21.15.72:0/1072300723 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe60c108b90 0x7fe60c10af80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe6080098a0 tx=0x7fe608009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56529c6039e7f48a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:26:43.748 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:42.616+0000 7fe6112d8700 1 -- 172.21.15.72:0/1072300723 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe608004160 con 0x7fe60c108b90 2024-03-22T01:26:43.748 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:42.616+0000 7fe6112d8700 1 -- 172.21.15.72:0/1072300723 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe608004300 con 0x7fe60c108b90 2024-03-22T01:26:43.748 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:42.616+0000 7fe6112d8700 1 -- 172.21.15.72:0/1072300723 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe608011b50 con 0x7fe60c108b90 2024-03-22T01:26:43.748 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:42.616+0000 7fe61453e700 1 -- 172.21.15.72:0/1072300723 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe60c108b90 msgr2=0x7fe60c10af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:26:43.748 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:42.616+0000 7fe61453e700 1 --2- 172.21.15.72:0/1072300723 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe60c108b90 0x7fe60c10af80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe6080098a0 tx=0x7fe608009bb0 comp rx=0 tx=0).stop 2024-03-22T01:26:43.748 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:42.616+0000 7fe61453e700 1 -- 172.21.15.72:0/1072300723 shutdown_connections 2024-03-22T01:26:43.749 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:42.616+0000 7fe61453e700 1 --2- 172.21.15.72:0/1072300723 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe60c108b90 0x7fe60c10af80 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:26:43.749 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:42.616+0000 7fe61453e700 1 -- 172.21.15.72:0/1072300723 >> 172.21.15.72:0/1072300723 conn(0x7fe60c1025a0 msgr2=0x7fe60c1049e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:26:43.749 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:42.616+0000 7fe61453e700 1 -- 172.21.15.72:0/1072300723 shutdown_connections 2024-03-22T01:26:43.749 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:42.617+0000 7fe61453e700 1 -- 172.21.15.72:0/1072300723 wait complete. 2024-03-22T01:26:43.749 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:42.617+0000 7fe61453e700 1 Processor -- start 2024-03-22T01:26:43.749 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:42.617+0000 7fe61453e700 1 -- start start 2024-03-22T01:26:43.749 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:42.617+0000 7fe61453e700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe60c108b90 0x7fe60c19bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:26:43.749 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:42.617+0000 7fe61453e700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fe60c10c470 con 0x7fe60c108b90 2024-03-22T01:26:43.749 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:42.617+0000 7fe6122da700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe60c108b90 0x7fe60c19bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:26:43.749 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:42.617+0000 7fe6122da700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe60c108b90 0x7fe60c19bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43200/0 (socket says 172.21.15.72:43200) 2024-03-22T01:26:43.749 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:42.617+0000 7fe6122da700 1 -- 172.21.15.72:0/3825036538 learned_addr learned my addr 172.21.15.72:0/3825036538 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:26:43.749 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:42.618+0000 7fe6122da700 1 -- 172.21.15.72:0/3825036538 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe60c19c490 con 0x7fe60c108b90 2024-03-22T01:26:43.749 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:42.618+0000 7fe6122da700 1 --2- 172.21.15.72:0/3825036538 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe60c108b90 0x7fe60c19bf50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe608011b60 tx=0x7fe608011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:26:43.749 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:42.618+0000 7fe6037fe700 1 -- 172.21.15.72:0/3825036538 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe608003b20 con 0x7fe60c108b90 2024-03-22T01:26:43.750 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:42.618+0000 7fe61453e700 1 -- 172.21.15.72:0/3825036538 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe60c19c780 con 0x7fe60c108b90 2024-03-22T01:26:43.750 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:42.618+0000 7fe6037fe700 1 -- 172.21.15.72:0/3825036538 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe608003cc0 con 0x7fe60c108b90 2024-03-22T01:26:43.750 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:42.618+0000 7fe6037fe700 1 -- 172.21.15.72:0/3825036538 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe608024840 con 0x7fe60c108b90 2024-03-22T01:26:43.750 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:42.618+0000 7fe61453e700 1 -- 172.21.15.72:0/3825036538 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe60c19f410 con 0x7fe60c108b90 2024-03-22T01:26:43.750 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:42.619+0000 7fe6037fe700 1 -- 172.21.15.72:0/3825036538 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 4) ==== 46283+0+0 (secure 0 0 0) 0x7fe608022020 con 0x7fe60c108b90 2024-03-22T01:26:43.750 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:42.619+0000 7fe61453e700 1 -- 172.21.15.72:0/3825036538 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe60c063cb0 con 0x7fe60c108b90 2024-03-22T01:26:43.750 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:42.619+0000 7fe6037fe700 1 --2- 172.21.15.72:0/3825036538 >> [v2:172.21.15.72:6800/3518012916,v1:172.21.15.72:6801/3518012916] conn(0x7fe5f80391f0 0x7fe5f803b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:26:43.750 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:42.619+0000 7fe6037fe700 1 -- 172.21.15.72:0/3825036538 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fe608011e00 con 0x7fe60c108b90 2024-03-22T01:26:43.750 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:42.619+0000 7fe611ad9700 1 --2- 172.21.15.72:0/3825036538 >> [v2:172.21.15.72:6800/3518012916,v1:172.21.15.72:6801/3518012916] conn(0x7fe5f80391f0 0x7fe5f803b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:26:43.750 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:42.619+0000 7fe611ad9700 1 --2- 172.21.15.72:0/3825036538 >> [v2:172.21.15.72:6800/3518012916,v1:172.21.15.72:6801/3518012916] conn(0x7fe5f80391f0 0x7fe5f803b6b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc006fd0 tx=0x7fe5fc006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:26:43.750 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:42.624+0000 7fe6037fe700 1 -- 172.21.15.72:0/3825036538 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe60800bc70 con 0x7fe60c108b90 2024-03-22T01:26:43.750 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:42.870+0000 7fe61453e700 1 -- 172.21.15.72:0/3825036538 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7fe60c19fbc0 con 0x7fe60c108b90 2024-03-22T01:26:43.750 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:43.238+0000 7fe6037fe700 1 -- 172.21.15.72:0/3825036538 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7fe60801a960 con 0x7fe60c108b90 2024-03-22T01:26:43.750 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:43.239+0000 7fe6037fe700 1 -- 172.21.15.72:0/3825036538 <== mon.0 v2:172.21.15.72:3300/0 8 ==== mgrmap(e 5) ==== 46294+0+0 (secure 0 0 0) 0x7fe608052020 con 0x7fe60c108b90 2024-03-22T01:26:43.751 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:43.241+0000 7fe61453e700 1 -- 172.21.15.72:0/3825036538 >> [v2:172.21.15.72:6800/3518012916,v1:172.21.15.72:6801/3518012916] conn(0x7fe5f80391f0 msgr2=0x7fe5f803b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:26:43.751 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:43.241+0000 7fe61453e700 1 --2- 172.21.15.72:0/3825036538 >> [v2:172.21.15.72:6800/3518012916,v1:172.21.15.72:6801/3518012916] conn(0x7fe5f80391f0 0x7fe5f803b6b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc006fd0 tx=0x7fe5fc006e40 comp rx=0 tx=0).stop 2024-03-22T01:26:43.751 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:43.241+0000 7fe61453e700 1 -- 172.21.15.72:0/3825036538 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe60c108b90 msgr2=0x7fe60c19bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:26:43.751 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:43.241+0000 7fe61453e700 1 --2- 172.21.15.72:0/3825036538 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe60c108b90 0x7fe60c19bf50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe608011b60 tx=0x7fe608011c40 comp rx=0 tx=0).stop 2024-03-22T01:26:43.751 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:43.241+0000 7fe61453e700 1 -- 172.21.15.72:0/3825036538 shutdown_connections 2024-03-22T01:26:43.751 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:43.241+0000 7fe61453e700 1 --2- 172.21.15.72:0/3825036538 >> [v2:172.21.15.72:6800/3518012916,v1:172.21.15.72:6801/3518012916] conn(0x7fe5f80391f0 0x7fe5f803b6b0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:26:43.751 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:43.241+0000 7fe61453e700 1 --2- 172.21.15.72:0/3825036538 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe60c108b90 0x7fe60c19bf50 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:26:43.751 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:43.241+0000 7fe61453e700 1 -- 172.21.15.72:0/3825036538 >> 172.21.15.72:0/3825036538 conn(0x7fe60c1025a0 msgr2=0x7fe60c102fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:26:43.751 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:43.241+0000 7fe61453e700 1 -- 172.21.15.72:0/3825036538 shutdown_connections 2024-03-22T01:26:43.751 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:43.241+0000 7fe61453e700 1 -- 172.21.15.72:0/3825036538 wait complete. 2024-03-22T01:26:44.063 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:43 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:26:43.803+0000 7f29c64ad200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-22T01:26:44.063 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:43 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:26:43.898+0000 7f29c64ad200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-22T01:26:44.366 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:44 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/3825036538' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-22T01:26:44.367 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:44 smithi072 ceph-mon[32414]: mgrmap e5: x(active, since 6s) 2024-03-22T01:26:44.622 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:44 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:26:44.560+0000 7f29c64ad200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-22T01:26:44.974 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:44 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:26:44.654+0000 7f29c64ad200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-22T01:26:44.974 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:44 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:26:44.796+0000 7f29c64ad200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-22T01:26:44.974 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:44 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:26:44.891+0000 7f29c64ad200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-22T01:26:44.974 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:44 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:26:44.975+0000 7f29c64ad200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-22T01:26:44.989 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout { 2024-03-22T01:26:44.989 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-22T01:26:44.989 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-22T01:26:44.989 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-22T01:26:44.989 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-22T01:26:44.989 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout } 2024-03-22T01:26:44.989 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:44.163+0000 7fb52355d700 1 Processor -- start 2024-03-22T01:26:44.990 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:44.163+0000 7fb52355d700 1 -- start start 2024-03-22T01:26:44.990 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:44.163+0000 7fb52355d700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb51c107020 0x7fb51c107440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:26:44.990 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:44.163+0000 7fb52355d700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fb51c107a10 con 0x7fb51c107020 2024-03-22T01:26:44.990 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:44.163+0000 7fb5212f9700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb51c107020 0x7fb51c107440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:26:44.990 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:44.163+0000 7fb5212f9700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb51c107020 0x7fb51c107440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43226/0 (socket says 172.21.15.72:43226) 2024-03-22T01:26:44.990 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:44.163+0000 7fb5212f9700 1 -- 172.21.15.72:0/1882666703 learned_addr learned my addr 172.21.15.72:0/1882666703 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:26:44.990 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:44.164+0000 7fb5212f9700 1 -- 172.21.15.72:0/1882666703 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb51c108220 con 0x7fb51c107020 2024-03-22T01:26:44.990 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:44.164+0000 7fb5212f9700 1 --2- 172.21.15.72:0/1882666703 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb51c107020 0x7fb51c107440 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb5180098a0 tx=0x7fb518009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ee378aaf48115d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:26:44.990 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:44.164+0000 7fb513fff700 1 -- 172.21.15.72:0/1882666703 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb518004160 con 0x7fb51c107020 2024-03-22T01:26:44.990 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:44.164+0000 7fb513fff700 1 -- 172.21.15.72:0/1882666703 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb518004300 con 0x7fb51c107020 2024-03-22T01:26:44.990 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:44.164+0000 7fb513fff700 1 -- 172.21.15.72:0/1882666703 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb518011b50 con 0x7fb51c107020 2024-03-22T01:26:44.990 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:44.164+0000 7fb52355d700 1 -- 172.21.15.72:0/1882666703 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb51c107020 msgr2=0x7fb51c107440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:26:44.990 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:44.164+0000 7fb52355d700 1 --2- 172.21.15.72:0/1882666703 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb51c107020 0x7fb51c107440 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb5180098a0 tx=0x7fb518009bb0 comp rx=0 tx=0).stop 2024-03-22T01:26:44.991 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:44.165+0000 7fb52355d700 1 -- 172.21.15.72:0/1882666703 shutdown_connections 2024-03-22T01:26:44.991 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:44.165+0000 7fb52355d700 1 --2- 172.21.15.72:0/1882666703 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb51c107020 0x7fb51c107440 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:26:44.991 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:44.165+0000 7fb52355d700 1 -- 172.21.15.72:0/1882666703 >> 172.21.15.72:0/1882666703 conn(0x7fb51c1025c0 msgr2=0x7fb51c104a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:26:44.991 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:44.165+0000 7fb52355d700 1 -- 172.21.15.72:0/1882666703 shutdown_connections 2024-03-22T01:26:44.991 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:44.165+0000 7fb52355d700 1 -- 172.21.15.72:0/1882666703 wait complete. 2024-03-22T01:26:44.991 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:44.165+0000 7fb52355d700 1 Processor -- start 2024-03-22T01:26:44.991 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:44.165+0000 7fb52355d700 1 -- start start 2024-03-22T01:26:44.991 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:44.165+0000 7fb52355d700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb51c107020 0x7fb51c19e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:26:44.991 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:44.165+0000 7fb52355d700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fb51c10b5d0 con 0x7fb51c107020 2024-03-22T01:26:44.991 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:44.166+0000 7fb5212f9700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb51c107020 0x7fb51c19e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:26:44.991 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:44.166+0000 7fb5212f9700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb51c107020 0x7fb51c19e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43240/0 (socket says 172.21.15.72:43240) 2024-03-22T01:26:44.991 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:44.166+0000 7fb5212f9700 1 -- 172.21.15.72:0/847219634 learned_addr learned my addr 172.21.15.72:0/847219634 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:26:44.991 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:44.166+0000 7fb5212f9700 1 -- 172.21.15.72:0/847219634 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb51c19e7c0 con 0x7fb51c107020 2024-03-22T01:26:44.992 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:44.166+0000 7fb5212f9700 1 --2- 172.21.15.72:0/847219634 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb51c107020 0x7fb51c19e280 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb518011670 tx=0x7fb518011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:26:44.992 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:44.166+0000 7fb5127fc700 1 -- 172.21.15.72:0/847219634 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb5180039c0 con 0x7fb51c107020 2024-03-22T01:26:44.992 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:44.166+0000 7fb5127fc700 1 -- 172.21.15.72:0/847219634 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb518011e50 con 0x7fb51c107020 2024-03-22T01:26:44.992 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:44.166+0000 7fb52355d700 1 -- 172.21.15.72:0/847219634 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb51c19ea50 con 0x7fb51c107020 2024-03-22T01:26:44.992 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:44.166+0000 7fb5127fc700 1 -- 172.21.15.72:0/847219634 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb51801a5c0 con 0x7fb51c107020 2024-03-22T01:26:44.992 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:44.166+0000 7fb52355d700 1 -- 172.21.15.72:0/847219634 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb51c1a1740 con 0x7fb51c107020 2024-03-22T01:26:44.992 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:44.167+0000 7fb5127fc700 1 -- 172.21.15.72:0/847219634 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 46294+0+0 (secure 0 0 0) 0x7fb51801a870 con 0x7fb51c107020 2024-03-22T01:26:44.992 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:44.167+0000 7fb5127fc700 1 --2- 172.21.15.72:0/847219634 >> [v2:172.21.15.72:6800/3518012916,v1:172.21.15.72:6801/3518012916] conn(0x7fb508039240 0x7fb50803b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:26:44.992 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:44.167+0000 7fb5127fc700 1 -- 172.21.15.72:0/847219634 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fb51804e930 con 0x7fb51c107020 2024-03-22T01:26:44.992 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:44.167+0000 7fb520af8700 1 -- 172.21.15.72:0/847219634 >> [v2:172.21.15.72:6800/3518012916,v1:172.21.15.72:6801/3518012916] conn(0x7fb508039240 msgr2=0x7fb50803b700 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.72:6800/3518012916 2024-03-22T01:26:44.992 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:44.167+0000 7fb520af8700 1 --2- 172.21.15.72:0/847219634 >> [v2:172.21.15.72:6800/3518012916,v1:172.21.15.72:6801/3518012916] conn(0x7fb508039240 0x7fb50803b700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-22T01:26:44.992 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:44.167+0000 7fb52355d700 1 -- 172.21.15.72:0/847219634 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb51c063cb0 con 0x7fb51c107020 2024-03-22T01:26:44.992 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:44.172+0000 7fb5127fc700 1 -- 172.21.15.72:0/847219634 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb51800b3e0 con 0x7fb51c107020 2024-03-22T01:26:44.992 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:44.368+0000 7fb520af8700 1 -- 172.21.15.72:0/847219634 >> [v2:172.21.15.72:6800/3518012916,v1:172.21.15.72:6801/3518012916] conn(0x7fb508039240 msgr2=0x7fb50803b700 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.72:6800/3518012916 2024-03-22T01:26:44.993 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:44.368+0000 7fb520af8700 1 --2- 172.21.15.72:0/847219634 >> [v2:172.21.15.72:6800/3518012916,v1:172.21.15.72:6801/3518012916] conn(0x7fb508039240 0x7fb50803b700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-22T01:26:44.993 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:44.433+0000 7fb52355d700 1 -- 172.21.15.72:0/847219634 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fb51c02df00 con 0x7fb51c107020 2024-03-22T01:26:44.993 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:44.434+0000 7fb5127fc700 1 -- 172.21.15.72:0/847219634 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7fb518020240 con 0x7fb51c107020 2024-03-22T01:26:44.993 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:44.436+0000 7fb52355d700 1 -- 172.21.15.72:0/847219634 >> [v2:172.21.15.72:6800/3518012916,v1:172.21.15.72:6801/3518012916] conn(0x7fb508039240 msgr2=0x7fb50803b700 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-22T01:26:44.993 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:44.436+0000 7fb52355d700 1 --2- 172.21.15.72:0/847219634 >> [v2:172.21.15.72:6800/3518012916,v1:172.21.15.72:6801/3518012916] conn(0x7fb508039240 0x7fb50803b700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:26:44.993 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:44.436+0000 7fb52355d700 1 -- 172.21.15.72:0/847219634 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb51c107020 msgr2=0x7fb51c19e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:26:44.993 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:44.436+0000 7fb52355d700 1 --2- 172.21.15.72:0/847219634 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb51c107020 0x7fb51c19e280 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb518011670 tx=0x7fb518011b60 comp rx=0 tx=0).stop 2024-03-22T01:26:44.993 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:44.436+0000 7fb52355d700 1 -- 172.21.15.72:0/847219634 shutdown_connections 2024-03-22T01:26:44.993 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:44.436+0000 7fb52355d700 1 --2- 172.21.15.72:0/847219634 >> [v2:172.21.15.72:6800/3518012916,v1:172.21.15.72:6801/3518012916] conn(0x7fb508039240 0x7fb50803b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:26:44.993 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:44.436+0000 7fb52355d700 1 --2- 172.21.15.72:0/847219634 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb51c107020 0x7fb51c19e280 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:26:44.993 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:44.436+0000 7fb52355d700 1 -- 172.21.15.72:0/847219634 >> 172.21.15.72:0/847219634 conn(0x7fb51c1025c0 msgr2=0x7fb51c1033e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:26:44.993 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:44.436+0000 7fb52355d700 1 -- 172.21.15.72:0/847219634 shutdown_connections 2024-03-22T01:26:44.993 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:44.436+0000 7fb52355d700 1 -- 172.21.15.72:0/847219634 wait complete. 2024-03-22T01:26:44.993 INFO:teuthology.orchestra.run.smithi072.stdout:Waiting for the mgr to restart... 2024-03-22T01:26:44.994 INFO:teuthology.orchestra.run.smithi072.stdout:Waiting for mgr epoch 5... 2024-03-22T01:26:45.550 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:45 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/847219634' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-22T01:26:45.550 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:45 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:26:45.255+0000 7f29c64ad200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-22T01:26:45.550 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:45 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:26:45.368+0000 7f29c64ad200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-22T01:26:45.550 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:45 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:26:45.457+0000 7f29c64ad200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-22T01:26:45.550 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:45 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:26:45.551+0000 7f29c64ad200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-22T01:26:45.871 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:45 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:26:45.725+0000 7f29c64ad200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-22T01:26:47.122 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:46 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:26:46.668+0000 7f29c64ad200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-22T01:26:47.122 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:46 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:26:46.795+0000 7f29c64ad200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-22T01:26:47.413 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:47 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:26:47.133+0000 7f29c64ad200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-22T01:26:47.413 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:47 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:26:47.221+0000 7f29c64ad200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-22T01:26:47.414 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:47 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:26:47.414+0000 7f29c64ad200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-22T01:26:47.871 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:47 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:26:47.597+0000 7f29c64ad200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-22T01:26:48.149 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:47 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:26:47.879+0000 7f29c64ad200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-22T01:26:48.150 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:48 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:26:48.150+0000 7f29c64ad200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-22T01:26:48.622 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:48 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:26:48.386+0000 7f29c64ad200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-22T01:26:49.622 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:49 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:26:49.183+0000 7f29c64ad200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-22T01:26:50.060 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:49 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:26:49.741+0000 7f29c64ad200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-22T01:26:50.060 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:49 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:26:49.841+0000 7f29c64ad200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-22T01:26:50.060 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:50 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:26:50.060+0000 7f29c64ad200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-22T01:26:50.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:50 smithi072 ceph-mon[32414]: Active manager daemon x restarted 2024-03-22T01:26:50.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:50 smithi072 ceph-mon[32414]: Activating manager daemon x 2024-03-22T01:26:50.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:50 smithi072 ceph-mon[32414]: osdmap e2: 0 total, 0 up, 0 in 2024-03-22T01:26:50.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:50 smithi072 ceph-mon[32414]: mgrmap e6: x(active, starting, since 0.00360856s) 2024-03-22T01:26:50.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:50 smithi072 ceph-mon[32414]: from='mgr.14120 172.21.15.72:0/4135194081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T01:26:50.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:50 smithi072 ceph-mon[32414]: from='mgr.14120 172.21.15.72:0/4135194081' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-22T01:26:50.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:50 smithi072 ceph-mon[32414]: from='mgr.14120 172.21.15.72:0/4135194081' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-22T01:26:50.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:50 smithi072 ceph-mon[32414]: from='mgr.14120 172.21.15.72:0/4135194081' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-22T01:26:50.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:50 smithi072 ceph-mon[32414]: from='mgr.14120 172.21.15.72:0/4135194081' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-22T01:26:50.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:50 smithi072 ceph-mon[32414]: Manager daemon x is now available 2024-03-22T01:26:50.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:50 smithi072 ceph-mon[32414]: from='mgr.14120 172.21.15.72:0/4135194081' entity='mgr.x' 2024-03-22T01:26:50.373 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:50 smithi072 ceph-mon[32414]: from='mgr.14120 172.21.15.72:0/4135194081' entity='mgr.x' 2024-03-22T01:26:50.373 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:50 smithi072 ceph-mon[32414]: from='mgr.14120 172.21.15.72:0/4135194081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:26:50.373 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:50 smithi072 ceph-mon[32414]: from='mgr.14120 172.21.15.72:0/4135194081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:26:51.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:51 smithi072 ceph-mon[32414]: Found migration_current of "None". Setting to last migration. 2024-03-22T01:26:51.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:51 smithi072 ceph-mon[32414]: from='mgr.14120 172.21.15.72:0/4135194081' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-22T01:26:51.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:51 smithi072 ceph-mon[32414]: from='mgr.14120 172.21.15.72:0/4135194081' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-22T01:26:51.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:51 smithi072 ceph-mon[32414]: from='mgr.14120 172.21.15.72:0/4135194081' entity='mgr.x' 2024-03-22T01:26:51.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:51 smithi072 ceph-mon[32414]: from='mgr.14120 172.21.15.72:0/4135194081' entity='mgr.x' 2024-03-22T01:26:51.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:51 smithi072 ceph-mon[32414]: from='mgr.14120 172.21.15.72:0/4135194081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:26:51.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:51 smithi072 ceph-mon[32414]: mgrmap e7: x(active, since 1.00565s) 2024-03-22T01:26:51.634 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout { 2024-03-22T01:26:51.634 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-22T01:26:51.634 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-22T01:26:51.634 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout } 2024-03-22T01:26:51.634 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:45.379+0000 7f3556c3f700 1 Processor -- start 2024-03-22T01:26:51.634 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:45.379+0000 7f3556c3f700 1 -- start start 2024-03-22T01:26:51.634 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:45.379+0000 7f3556c3f700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3550108bf0 0x7f355010afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:26:51.634 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:45.379+0000 7f3556c3f700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f3550075db0 con 0x7f3550108bf0 2024-03-22T01:26:51.634 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:45.379+0000 7f35549db700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3550108bf0 0x7f355010afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:26:51.635 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:45.379+0000 7f35549db700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3550108bf0 0x7f355010afe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43242/0 (socket says 172.21.15.72:43242) 2024-03-22T01:26:51.635 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:45.379+0000 7f35549db700 1 -- 172.21.15.72:0/3076053204 learned_addr learned my addr 172.21.15.72:0/3076053204 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:26:51.636 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:45.380+0000 7f35549db700 1 -- 172.21.15.72:0/3076053204 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f355010b520 con 0x7f3550108bf0 2024-03-22T01:26:51.636 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:45.380+0000 7f35549db700 1 --2- 172.21.15.72:0/3076053204 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3550108bf0 0x7f355010afe0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f35440098a0 tx=0x7f3544009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fd6ce3cbfa1475b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:26:51.636 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:45.380+0000 7f354f7fe700 1 -- 172.21.15.72:0/3076053204 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3544004160 con 0x7f3550108bf0 2024-03-22T01:26:51.636 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:45.380+0000 7f354f7fe700 1 -- 172.21.15.72:0/3076053204 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f3544004300 con 0x7f3550108bf0 2024-03-22T01:26:51.636 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:45.380+0000 7f354f7fe700 1 -- 172.21.15.72:0/3076053204 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3544011b50 con 0x7f3550108bf0 2024-03-22T01:26:51.636 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:45.381+0000 7f3556c3f700 1 -- 172.21.15.72:0/3076053204 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3550108bf0 msgr2=0x7f355010afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:26:51.636 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:45.381+0000 7f3556c3f700 1 --2- 172.21.15.72:0/3076053204 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3550108bf0 0x7f355010afe0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f35440098a0 tx=0x7f3544009bb0 comp rx=0 tx=0).stop 2024-03-22T01:26:51.636 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:45.381+0000 7f3556c3f700 1 -- 172.21.15.72:0/3076053204 shutdown_connections 2024-03-22T01:26:51.636 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:45.381+0000 7f3556c3f700 1 --2- 172.21.15.72:0/3076053204 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3550108bf0 0x7f355010afe0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:26:51.636 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:45.381+0000 7f3556c3f700 1 -- 172.21.15.72:0/3076053204 >> 172.21.15.72:0/3076053204 conn(0x7f35501025c0 msgr2=0x7f3550104a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:26:51.637 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:45.381+0000 7f3556c3f700 1 -- 172.21.15.72:0/3076053204 shutdown_connections 2024-03-22T01:26:51.637 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:45.381+0000 7f3556c3f700 1 -- 172.21.15.72:0/3076053204 wait complete. 2024-03-22T01:26:51.637 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:45.381+0000 7f3556c3f700 1 Processor -- start 2024-03-22T01:26:51.637 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:45.381+0000 7f3556c3f700 1 -- start start 2024-03-22T01:26:51.637 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:45.382+0000 7f3556c3f700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3550108bf0 0x7f355019c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:26:51.637 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:45.382+0000 7f3556c3f700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f355010c4f0 con 0x7f3550108bf0 2024-03-22T01:26:51.637 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:45.382+0000 7f35549db700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3550108bf0 0x7f355019c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:26:51.637 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:45.382+0000 7f35549db700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3550108bf0 0x7f355019c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43256/0 (socket says 172.21.15.72:43256) 2024-03-22T01:26:51.637 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:45.382+0000 7f35549db700 1 -- 172.21.15.72:0/1592211061 learned_addr learned my addr 172.21.15.72:0/1592211061 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:26:51.637 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:45.382+0000 7f35549db700 1 -- 172.21.15.72:0/1592211061 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f355019c540 con 0x7f3550108bf0 2024-03-22T01:26:51.637 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:45.382+0000 7f35549db700 1 --2- 172.21.15.72:0/1592211061 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3550108bf0 0x7f355019c000 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3544011b60 tx=0x7f3544011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:26:51.637 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:45.382+0000 7f354dffb700 1 -- 172.21.15.72:0/1592211061 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3544003b20 con 0x7f3550108bf0 2024-03-22T01:26:51.637 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:45.382+0000 7f354dffb700 1 -- 172.21.15.72:0/1592211061 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f3544003cc0 con 0x7f3550108bf0 2024-03-22T01:26:51.637 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:45.382+0000 7f3556c3f700 1 -- 172.21.15.72:0/1592211061 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f355019c7d0 con 0x7f3550108bf0 2024-03-22T01:26:51.638 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:45.382+0000 7f354dffb700 1 -- 172.21.15.72:0/1592211061 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3544024840 con 0x7f3550108bf0 2024-03-22T01:26:51.638 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:45.383+0000 7f3556c3f700 1 -- 172.21.15.72:0/1592211061 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f355019f4c0 con 0x7f3550108bf0 2024-03-22T01:26:51.638 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:45.384+0000 7f354dffb700 1 -- 172.21.15.72:0/1592211061 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 46294+0+0 (secure 0 0 0) 0x7f3544022020 con 0x7f3550108bf0 2024-03-22T01:26:51.638 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:45.384+0000 7f354dffb700 1 --2- 172.21.15.72:0/1592211061 >> [v2:172.21.15.72:6800/3518012916,v1:172.21.15.72:6801/3518012916] conn(0x7f3540039290 0x7f354003b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:26:51.638 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:45.384+0000 7f354dffb700 1 -- 172.21.15.72:0/1592211061 --> [v2:172.21.15.72:6800/3518012916,v1:172.21.15.72:6801/3518012916] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f354003be60 con 0x7f3540039290 2024-03-22T01:26:51.638 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:45.384+0000 7f354dffb700 1 -- 172.21.15.72:0/1592211061 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f3544011e00 con 0x7f3550108bf0 2024-03-22T01:26:51.638 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:45.384+0000 7f354ffff700 1 -- 172.21.15.72:0/1592211061 >> [v2:172.21.15.72:6800/3518012916,v1:172.21.15.72:6801/3518012916] conn(0x7f3540039290 msgr2=0x7f354003b750 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.72:6800/3518012916 2024-03-22T01:26:51.638 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:45.384+0000 7f354ffff700 1 --2- 172.21.15.72:0/1592211061 >> [v2:172.21.15.72:6800/3518012916,v1:172.21.15.72:6801/3518012916] conn(0x7f3540039290 0x7f354003b750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-22T01:26:51.638 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:45.585+0000 7f354ffff700 1 -- 172.21.15.72:0/1592211061 >> [v2:172.21.15.72:6800/3518012916,v1:172.21.15.72:6801/3518012916] conn(0x7f3540039290 msgr2=0x7f354003b750 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.72:6800/3518012916 2024-03-22T01:26:51.638 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:45.585+0000 7f354ffff700 1 --2- 172.21.15.72:0/1592211061 >> [v2:172.21.15.72:6800/3518012916,v1:172.21.15.72:6801/3518012916] conn(0x7f3540039290 0x7f354003b750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-22T01:26:51.638 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:45.986+0000 7f354ffff700 1 -- 172.21.15.72:0/1592211061 >> [v2:172.21.15.72:6800/3518012916,v1:172.21.15.72:6801/3518012916] conn(0x7f3540039290 msgr2=0x7f354003b750 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.72:6800/3518012916 2024-03-22T01:26:51.638 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:45.986+0000 7f354ffff700 1 --2- 172.21.15.72:0/1592211061 >> [v2:172.21.15.72:6800/3518012916,v1:172.21.15.72:6801/3518012916] conn(0x7f3540039290 0x7f354003b750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-22T01:26:51.639 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:46.787+0000 7f354ffff700 1 -- 172.21.15.72:0/1592211061 >> [v2:172.21.15.72:6800/3518012916,v1:172.21.15.72:6801/3518012916] conn(0x7f3540039290 msgr2=0x7f354003b750 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.72:6800/3518012916 2024-03-22T01:26:51.639 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:46.787+0000 7f354ffff700 1 --2- 172.21.15.72:0/1592211061 >> [v2:172.21.15.72:6800/3518012916,v1:172.21.15.72:6801/3518012916] conn(0x7f3540039290 0x7f354003b750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-22T01:26:51.639 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:48.388+0000 7f354ffff700 1 -- 172.21.15.72:0/1592211061 >> [v2:172.21.15.72:6800/3518012916,v1:172.21.15.72:6801/3518012916] conn(0x7f3540039290 msgr2=0x7f354003b750 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.72:6800/3518012916 2024-03-22T01:26:51.639 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:48.388+0000 7f354ffff700 1 --2- 172.21.15.72:0/1592211061 >> [v2:172.21.15.72:6800/3518012916,v1:172.21.15.72:6801/3518012916] conn(0x7f3540039290 0x7f354003b750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-22T01:26:51.639 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:50.076+0000 7f354dffb700 1 -- 172.21.15.72:0/1592211061 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mgrmap(e 6) ==== 46061+0+0 (secure 0 0 0) 0x7f3544024a20 con 0x7f3550108bf0 2024-03-22T01:26:51.639 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:50.076+0000 7f354dffb700 1 -- 172.21.15.72:0/1592211061 >> [v2:172.21.15.72:6800/3518012916,v1:172.21.15.72:6801/3518012916] conn(0x7f3540039290 msgr2=0x7f354003b750 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-22T01:26:51.639 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:50.076+0000 7f354dffb700 1 --2- 172.21.15.72:0/1592211061 >> [v2:172.21.15.72:6800/3518012916,v1:172.21.15.72:6801/3518012916] conn(0x7f3540039290 0x7f354003b750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:26:51.639 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:51.077+0000 7f354dffb700 1 -- 172.21.15.72:0/1592211061 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mgrmap(e 7) ==== 46188+0+0 (secure 0 0 0) 0x7f354404ed70 con 0x7f3550108bf0 2024-03-22T01:26:51.639 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:51.077+0000 7f354dffb700 1 --2- 172.21.15.72:0/1592211061 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7f354003cb80 0x7f354003ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:26:51.639 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:51.077+0000 7f354dffb700 1 -- 172.21.15.72:0/1592211061 --> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f3544022e80 con 0x7f354003cb80 2024-03-22T01:26:51.639 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:51.078+0000 7f354ffff700 1 --2- 172.21.15.72:0/1592211061 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7f354003cb80 0x7f354003ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:26:51.639 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:51.078+0000 7f354ffff700 1 --2- 172.21.15.72:0/1592211061 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7f354003cb80 0x7f354003ef70 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f353c003a10 tx=0x7f353c0092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:26:51.640 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:51.079+0000 7f354dffb700 1 -- 172.21.15.72:0/1592211061 <== mgr.14120 v2:172.21.15.72:6800/1513232913 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f3544022e80 con 0x7f354003cb80 2024-03-22T01:26:51.640 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:51.086+0000 7f3556c3f700 1 -- 172.21.15.72:0/1592211061 --> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f3550063cb0 con 0x7f354003cb80 2024-03-22T01:26:51.640 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:51.086+0000 7f354dffb700 1 -- 172.21.15.72:0/1592211061 <== mgr.14120 v2:172.21.15.72:6800/1513232913 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f3550063cb0 con 0x7f354003cb80 2024-03-22T01:26:51.640 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:51.086+0000 7f3556c3f700 1 -- 172.21.15.72:0/1592211061 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7f354003cb80 msgr2=0x7f354003ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:26:51.640 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:51.086+0000 7f3556c3f700 1 --2- 172.21.15.72:0/1592211061 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7f354003cb80 0x7f354003ef70 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f353c003a10 tx=0x7f353c0092b0 comp rx=0 tx=0).stop 2024-03-22T01:26:51.640 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:51.086+0000 7f3556c3f700 1 -- 172.21.15.72:0/1592211061 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3550108bf0 msgr2=0x7f355019c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:26:51.640 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:51.086+0000 7f3556c3f700 1 --2- 172.21.15.72:0/1592211061 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3550108bf0 0x7f355019c000 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3544011b60 tx=0x7f3544011c40 comp rx=0 tx=0).stop 2024-03-22T01:26:51.640 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:51.086+0000 7f3556c3f700 1 -- 172.21.15.72:0/1592211061 shutdown_connections 2024-03-22T01:26:51.640 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:51.086+0000 7f3556c3f700 1 --2- 172.21.15.72:0/1592211061 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7f354003cb80 0x7f354003ef70 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:26:51.640 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:51.086+0000 7f3556c3f700 1 --2- 172.21.15.72:0/1592211061 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3550108bf0 0x7f355019c000 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:26:51.640 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:51.086+0000 7f3556c3f700 1 --2- 172.21.15.72:0/1592211061 >> [v2:172.21.15.72:6800/3518012916,v1:172.21.15.72:6801/3518012916] conn(0x7f3540039290 0x7f354003b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:26:51.640 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:51.086+0000 7f3556c3f700 1 -- 172.21.15.72:0/1592211061 >> 172.21.15.72:0/1592211061 conn(0x7f35501025c0 msgr2=0x7f3550103060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:26:51.640 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:51.086+0000 7f3556c3f700 1 -- 172.21.15.72:0/1592211061 shutdown_connections 2024-03-22T01:26:51.640 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:51.086+0000 7f3556c3f700 1 -- 172.21.15.72:0/1592211061 wait complete. 2024-03-22T01:26:51.641 INFO:teuthology.orchestra.run.smithi072.stdout:mgr epoch 5 is available 2024-03-22T01:26:51.641 INFO:teuthology.orchestra.run.smithi072.stdout:Setting orchestrator backend to cephadm... 2024-03-22T01:26:52.310 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:52 smithi072 ceph-mon[32414]: [22/Mar/2024:01:26:50] ENGINE Bus STARTING 2024-03-22T01:26:52.310 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:52 smithi072 ceph-mon[32414]: [22/Mar/2024:01:26:50] ENGINE Serving on http://172.21.15.72:8765 2024-03-22T01:26:52.310 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:52 smithi072 ceph-mon[32414]: [22/Mar/2024:01:26:51] ENGINE Serving on https://172.21.15.72:7150 2024-03-22T01:26:52.310 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:52 smithi072 ceph-mon[32414]: [22/Mar/2024:01:26:51] ENGINE Bus STARTED 2024-03-22T01:26:52.311 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:52 smithi072 ceph-mon[32414]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-22T01:26:52.311 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:52 smithi072 ceph-mon[32414]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-22T01:26:52.862 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:52.088+0000 7f29844fe700 1 Processor -- start 2024-03-22T01:26:52.862 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:52.088+0000 7f29844fe700 1 -- start start 2024-03-22T01:26:52.862 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:52.089+0000 7f29844fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f297c1085d0 0x7f297c1089f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:26:52.862 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:52.089+0000 7f29844fe700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f297c108fc0 con 0x7f297c1085d0 2024-03-22T01:26:52.862 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:52.089+0000 7f298229a700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f297c1085d0 0x7f297c1089f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:26:52.862 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:52.089+0000 7f298229a700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f297c1085d0 0x7f297c1089f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44190/0 (socket says 172.21.15.72:44190) 2024-03-22T01:26:52.862 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:52.089+0000 7f298229a700 1 -- 172.21.15.72:0/2130876275 learned_addr learned my addr 172.21.15.72:0/2130876275 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:26:52.863 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:52.089+0000 7f298229a700 1 -- 172.21.15.72:0/2130876275 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f297c107280 con 0x7f297c1085d0 2024-03-22T01:26:52.863 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:52.089+0000 7f298229a700 1 --2- 172.21.15.72:0/2130876275 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f297c1085d0 0x7f297c1089f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f29780098a0 tx=0x7f2978009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a45d28372e8d32c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:26:52.863 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:52.090+0000 7f2981298700 1 -- 172.21.15.72:0/2130876275 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2978004160 con 0x7f297c1085d0 2024-03-22T01:26:52.863 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:52.090+0000 7f2981298700 1 -- 172.21.15.72:0/2130876275 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2978004300 con 0x7f297c1085d0 2024-03-22T01:26:52.863 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:52.090+0000 7f2981298700 1 -- 172.21.15.72:0/2130876275 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2978011b50 con 0x7f297c1085d0 2024-03-22T01:26:52.863 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:52.090+0000 7f29844fe700 1 -- 172.21.15.72:0/2130876275 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f297c1085d0 msgr2=0x7f297c1089f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:26:52.863 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:52.090+0000 7f29844fe700 1 --2- 172.21.15.72:0/2130876275 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f297c1085d0 0x7f297c1089f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f29780098a0 tx=0x7f2978009bb0 comp rx=0 tx=0).stop 2024-03-22T01:26:52.863 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:52.090+0000 7f29844fe700 1 -- 172.21.15.72:0/2130876275 shutdown_connections 2024-03-22T01:26:52.863 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:52.090+0000 7f29844fe700 1 --2- 172.21.15.72:0/2130876275 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f297c1085d0 0x7f297c1089f0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:26:52.863 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:52.090+0000 7f29844fe700 1 -- 172.21.15.72:0/2130876275 >> 172.21.15.72:0/2130876275 conn(0x7f297c101d30 msgr2=0x7f297c104150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:26:52.863 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:52.090+0000 7f29844fe700 1 -- 172.21.15.72:0/2130876275 shutdown_connections 2024-03-22T01:26:52.864 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:52.090+0000 7f29844fe700 1 -- 172.21.15.72:0/2130876275 wait complete. 2024-03-22T01:26:52.864 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:52.091+0000 7f29844fe700 1 Processor -- start 2024-03-22T01:26:52.864 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:52.091+0000 7f29844fe700 1 -- start start 2024-03-22T01:26:52.864 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:52.091+0000 7f29844fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f297c1085d0 0x7f297c19af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:26:52.864 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:52.091+0000 7f29844fe700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f297c10b640 con 0x7f297c1085d0 2024-03-22T01:26:52.864 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:52.091+0000 7f298229a700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f297c1085d0 0x7f297c19af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:26:52.864 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:52.091+0000 7f298229a700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f297c1085d0 0x7f297c19af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44206/0 (socket says 172.21.15.72:44206) 2024-03-22T01:26:52.864 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:52.091+0000 7f298229a700 1 -- 172.21.15.72:0/2136848008 learned_addr learned my addr 172.21.15.72:0/2136848008 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:26:52.864 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:52.091+0000 7f298229a700 1 -- 172.21.15.72:0/2136848008 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f297c19b450 con 0x7f297c1085d0 2024-03-22T01:26:52.864 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:52.091+0000 7f298229a700 1 --2- 172.21.15.72:0/2136848008 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f297c1085d0 0x7f297c19af10 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2978011670 tx=0x7f2978011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:26:52.864 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:52.092+0000 7f29737fe700 1 -- 172.21.15.72:0/2136848008 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f29780039c0 con 0x7f297c1085d0 2024-03-22T01:26:52.864 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:52.092+0000 7f29737fe700 1 -- 172.21.15.72:0/2136848008 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2978011e50 con 0x7f297c1085d0 2024-03-22T01:26:52.864 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:52.092+0000 7f29844fe700 1 -- 172.21.15.72:0/2136848008 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f297c19b6e0 con 0x7f297c1085d0 2024-03-22T01:26:52.865 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:52.092+0000 7f29737fe700 1 -- 172.21.15.72:0/2136848008 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f297801a5c0 con 0x7f297c1085d0 2024-03-22T01:26:52.865 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:52.092+0000 7f29844fe700 1 -- 172.21.15.72:0/2136848008 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f297c19e3d0 con 0x7f297c1085d0 2024-03-22T01:26:52.865 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:52.092+0000 7f29737fe700 1 -- 172.21.15.72:0/2136848008 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 7) ==== 46188+0+0 (secure 0 0 0) 0x7f2978024430 con 0x7f297c1085d0 2024-03-22T01:26:52.865 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:52.093+0000 7f29737fe700 1 --2- 172.21.15.72:0/2136848008 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7f2968039130 0x7f296803b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:26:52.865 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:52.093+0000 7f29844fe700 1 -- 172.21.15.72:0/2136848008 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f297c063cb0 con 0x7f297c1085d0 2024-03-22T01:26:52.865 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:52.093+0000 7f29737fe700 1 -- 172.21.15.72:0/2136848008 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f297804d3a0 con 0x7f297c1085d0 2024-03-22T01:26:52.865 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:52.093+0000 7f2981a99700 1 --2- 172.21.15.72:0/2136848008 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7f2968039130 0x7f296803b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:26:52.865 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:52.093+0000 7f2981a99700 1 --2- 172.21.15.72:0/2136848008 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7f2968039130 0x7f296803b5f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f296c006fd0 tx=0x7f296c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:26:52.865 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:52.098+0000 7f29737fe700 1 -- 172.21.15.72:0/2136848008 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f297800f2e0 con 0x7f297c1085d0 2024-03-22T01:26:52.865 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:52.132+0000 7f29737fe700 1 -- 172.21.15.72:0/2136848008 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mgrmap(e 8) ==== 46294+0+0 (secure 0 0 0) 0x7f297802d3f0 con 0x7f297c1085d0 2024-03-22T01:26:52.865 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:52.310+0000 7f29844fe700 1 -- 172.21.15.72:0/2136848008 --> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f297c102f10 con 0x7f2968039130 2024-03-22T01:26:52.865 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:52.317+0000 7f29737fe700 1 -- 172.21.15.72:0/2136848008 <== mgr.14120 v2:172.21.15.72:6800/1513232913 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f297c102f10 con 0x7f2968039130 2024-03-22T01:26:52.865 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:52.319+0000 7f29844fe700 1 -- 172.21.15.72:0/2136848008 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7f2968039130 msgr2=0x7f296803b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:26:52.866 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:52.319+0000 7f29844fe700 1 --2- 172.21.15.72:0/2136848008 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7f2968039130 0x7f296803b5f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f296c006fd0 tx=0x7f296c006e40 comp rx=0 tx=0).stop 2024-03-22T01:26:52.866 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:52.319+0000 7f29844fe700 1 -- 172.21.15.72:0/2136848008 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f297c1085d0 msgr2=0x7f297c19af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:26:52.866 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:52.319+0000 7f29844fe700 1 --2- 172.21.15.72:0/2136848008 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f297c1085d0 0x7f297c19af10 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2978011670 tx=0x7f2978011b60 comp rx=0 tx=0).stop 2024-03-22T01:26:52.866 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:52.319+0000 7f29844fe700 1 -- 172.21.15.72:0/2136848008 shutdown_connections 2024-03-22T01:26:52.866 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:52.319+0000 7f29844fe700 1 --2- 172.21.15.72:0/2136848008 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7f2968039130 0x7f296803b5f0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:26:52.866 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:52.319+0000 7f29844fe700 1 --2- 172.21.15.72:0/2136848008 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f297c1085d0 0x7f297c19af10 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:26:52.866 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:52.319+0000 7f29844fe700 1 -- 172.21.15.72:0/2136848008 >> 172.21.15.72:0/2136848008 conn(0x7f297c101d30 msgr2=0x7f297c102800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:26:52.866 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:52.319+0000 7f29844fe700 1 -- 172.21.15.72:0/2136848008 shutdown_connections 2024-03-22T01:26:52.866 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:52.319+0000 7f29844fe700 1 -- 172.21.15.72:0/2136848008 wait complete. 2024-03-22T01:26:53.249 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:53 smithi072 ceph-mon[32414]: mgrmap e8: x(active, since 2s) 2024-03-22T01:26:53.249 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:53 smithi072 ceph-mon[32414]: from='mgr.14120 172.21.15.72:0/4135194081' entity='mgr.x' 2024-03-22T01:26:53.249 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:53 smithi072 ceph-mon[32414]: from='mgr.14120 172.21.15.72:0/4135194081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:26:54.011 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-22T01:26:54.011 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:53.250+0000 7fab3b1fd700 1 Processor -- start 2024-03-22T01:26:54.011 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:53.250+0000 7fab3b1fd700 1 -- start start 2024-03-22T01:26:54.012 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:53.250+0000 7fab3b1fd700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fab34108170 0x7fab34108590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:26:54.012 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:53.250+0000 7fab3b1fd700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fab34108b60 con 0x7fab34108170 2024-03-22T01:26:54.012 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:53.251+0000 7fab38f99700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fab34108170 0x7fab34108590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:26:54.012 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:53.251+0000 7fab38f99700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fab34108170 0x7fab34108590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44208/0 (socket says 172.21.15.72:44208) 2024-03-22T01:26:54.012 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:53.251+0000 7fab38f99700 1 -- 172.21.15.72:0/1023266023 learned_addr learned my addr 172.21.15.72:0/1023266023 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:26:54.012 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:53.251+0000 7fab38f99700 1 -- 172.21.15.72:0/1023266023 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab341092d0 con 0x7fab34108170 2024-03-22T01:26:54.012 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:53.251+0000 7fab38f99700 1 --2- 172.21.15.72:0/1023266023 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fab34108170 0x7fab34108590 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fab30009e80 tx=0x7fab3000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e35c1cf3db74b43c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:26:54.012 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:53.251+0000 7fab2b7fe700 1 -- 172.21.15.72:0/1023266023 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fab30017070 con 0x7fab34108170 2024-03-22T01:26:54.012 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:53.251+0000 7fab2b7fe700 1 -- 172.21.15.72:0/1023266023 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fab3000b8c0 con 0x7fab34108170 2024-03-22T01:26:54.012 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:53.251+0000 7fab2b7fe700 1 -- 172.21.15.72:0/1023266023 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fab300045c0 con 0x7fab34108170 2024-03-22T01:26:54.012 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:53.252+0000 7fab3b1fd700 1 -- 172.21.15.72:0/1023266023 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fab34108170 msgr2=0x7fab34108590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:26:54.012 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:53.252+0000 7fab3b1fd700 1 --2- 172.21.15.72:0/1023266023 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fab34108170 0x7fab34108590 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fab30009e80 tx=0x7fab3000b330 comp rx=0 tx=0).stop 2024-03-22T01:26:54.013 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:53.252+0000 7fab3b1fd700 1 -- 172.21.15.72:0/1023266023 shutdown_connections 2024-03-22T01:26:54.013 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:53.252+0000 7fab3b1fd700 1 --2- 172.21.15.72:0/1023266023 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fab34108170 0x7fab34108590 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:26:54.013 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:53.252+0000 7fab3b1fd700 1 -- 172.21.15.72:0/1023266023 >> 172.21.15.72:0/1023266023 conn(0x7fab34103750 msgr2=0x7fab34105b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:26:54.013 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:53.252+0000 7fab3b1fd700 1 -- 172.21.15.72:0/1023266023 shutdown_connections 2024-03-22T01:26:54.013 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:53.252+0000 7fab3b1fd700 1 -- 172.21.15.72:0/1023266023 wait complete. 2024-03-22T01:26:54.013 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:53.253+0000 7fab3b1fd700 1 Processor -- start 2024-03-22T01:26:54.013 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:53.253+0000 7fab3b1fd700 1 -- start start 2024-03-22T01:26:54.013 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:53.253+0000 7fab3b1fd700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fab34108170 0x7fab3407db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:26:54.013 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:53.253+0000 7fab3b1fd700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fab3410c680 con 0x7fab34108170 2024-03-22T01:26:54.013 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:53.253+0000 7fab38f99700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fab34108170 0x7fab3407db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:26:54.013 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:53.253+0000 7fab38f99700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fab34108170 0x7fab3407db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44214/0 (socket says 172.21.15.72:44214) 2024-03-22T01:26:54.013 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:53.253+0000 7fab38f99700 1 -- 172.21.15.72:0/1001829791 learned_addr learned my addr 172.21.15.72:0/1001829791 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:26:54.013 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:53.253+0000 7fab38f99700 1 -- 172.21.15.72:0/1001829791 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab3407e090 con 0x7fab34108170 2024-03-22T01:26:54.014 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:53.253+0000 7fab38f99700 1 --2- 172.21.15.72:0/1001829791 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fab34108170 0x7fab3407db50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fab30009e50 tx=0x7fab3000bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:26:54.014 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:53.253+0000 7fab29ffb700 1 -- 172.21.15.72:0/1001829791 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fab30020020 con 0x7fab34108170 2024-03-22T01:26:54.014 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:53.254+0000 7fab29ffb700 1 -- 172.21.15.72:0/1001829791 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fab30003b50 con 0x7fab34108170 2024-03-22T01:26:54.014 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:53.254+0000 7fab29ffb700 1 -- 172.21.15.72:0/1001829791 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fab30017040 con 0x7fab34108170 2024-03-22T01:26:54.014 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:53.254+0000 7fab3b1fd700 1 -- 172.21.15.72:0/1001829791 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab3407c280 con 0x7fab34108170 2024-03-22T01:26:54.014 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:53.254+0000 7fab3b1fd700 1 -- 172.21.15.72:0/1001829791 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab3407c7c0 con 0x7fab34108170 2024-03-22T01:26:54.014 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:53.255+0000 7fab29ffb700 1 -- 172.21.15.72:0/1001829791 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 8) ==== 46294+0+0 (secure 0 0 0) 0x7fab30023020 con 0x7fab34108170 2024-03-22T01:26:54.014 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:53.255+0000 7fab3b1fd700 1 -- 172.21.15.72:0/1001829791 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab34063cb0 con 0x7fab34108170 2024-03-22T01:26:54.014 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:53.255+0000 7fab29ffb700 1 --2- 172.21.15.72:0/1001829791 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7fab1c0391f0 0x7fab1c03b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:26:54.014 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:53.255+0000 7fab29ffb700 1 -- 172.21.15.72:0/1001829791 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fab3000f050 con 0x7fab34108170 2024-03-22T01:26:54.014 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:53.255+0000 7fab2bfff700 1 --2- 172.21.15.72:0/1001829791 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7fab1c0391f0 0x7fab1c03b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:26:54.014 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:53.256+0000 7fab2bfff700 1 --2- 172.21.15.72:0/1001829791 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7fab1c0391f0 0x7fab1c03b6b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fab24006fd0 tx=0x7fab24006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:26:54.015 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:53.260+0000 7fab29ffb700 1 -- 172.21.15.72:0/1001829791 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fab3002a020 con 0x7fab34108170 2024-03-22T01:26:54.015 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:53.456+0000 7fab3b1fd700 1 -- 172.21.15.72:0/1001829791 --> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7fab341048b0 con 0x7fab1c0391f0 2024-03-22T01:26:54.015 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:53.457+0000 7fab29ffb700 1 -- 172.21.15.72:0/1001829791 <== mgr.14120 v2:172.21.15.72:6800/1513232913 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7fab341048b0 con 0x7fab1c0391f0 2024-03-22T01:26:54.015 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:53.459+0000 7fab3b1fd700 1 -- 172.21.15.72:0/1001829791 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7fab1c0391f0 msgr2=0x7fab1c03b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:26:54.015 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:53.459+0000 7fab3b1fd700 1 --2- 172.21.15.72:0/1001829791 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7fab1c0391f0 0x7fab1c03b6b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fab24006fd0 tx=0x7fab24006e40 comp rx=0 tx=0).stop 2024-03-22T01:26:54.015 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:53.459+0000 7fab3b1fd700 1 -- 172.21.15.72:0/1001829791 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fab34108170 msgr2=0x7fab3407db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:26:54.015 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:53.459+0000 7fab3b1fd700 1 --2- 172.21.15.72:0/1001829791 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fab34108170 0x7fab3407db50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fab30009e50 tx=0x7fab3000bd30 comp rx=0 tx=0).stop 2024-03-22T01:26:54.015 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:53.459+0000 7fab3b1fd700 1 -- 172.21.15.72:0/1001829791 shutdown_connections 2024-03-22T01:26:54.015 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:53.459+0000 7fab3b1fd700 1 --2- 172.21.15.72:0/1001829791 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7fab1c0391f0 0x7fab1c03b6b0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:26:54.015 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:53.459+0000 7fab3b1fd700 1 --2- 172.21.15.72:0/1001829791 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fab34108170 0x7fab3407db50 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:26:54.015 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:53.459+0000 7fab3b1fd700 1 -- 172.21.15.72:0/1001829791 >> 172.21.15.72:0/1001829791 conn(0x7fab34103750 msgr2=0x7fab341041a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:26:54.015 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:53.459+0000 7fab3b1fd700 1 -- 172.21.15.72:0/1001829791 shutdown_connections 2024-03-22T01:26:54.015 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:53.459+0000 7fab3b1fd700 1 -- 172.21.15.72:0/1001829791 wait complete. 2024-03-22T01:26:54.015 INFO:teuthology.orchestra.run.smithi072.stdout:Generating ssh key... 2024-03-22T01:26:54.350 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:54 smithi072 ceph-mon[32414]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:26:55.274 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:55 smithi072 ceph-mon[32414]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:26:55.274 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:55 smithi072 ceph-mon[32414]: from='mgr.14120 172.21.15.72:0/4135194081' entity='mgr.x' 2024-03-22T01:26:55.274 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:55 smithi072 ceph-mon[32414]: from='mgr.14120 172.21.15.72:0/4135194081' entity='mgr.x' 2024-03-22T01:26:55.274 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:54 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: Generating public/private rsa key pair. 2024-03-22T01:26:55.274 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:54 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: Your identification has been saved in /tmp/tmp4z4u5gp2/key. 2024-03-22T01:26:55.274 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:54 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: Your public key has been saved in /tmp/tmp4z4u5gp2/key.pub. 2024-03-22T01:26:55.274 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:54 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: The key fingerprint is: 2024-03-22T01:26:55.275 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:54 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: SHA256:3kYz8QAnOjBoEiklYe8IoZhj9Shizsu7EZlOUWCtGyU ceph-022a99bc-e7eb-11ee-95cd-87774f69a715 2024-03-22T01:26:55.275 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:54 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: The key's randomart image is: 2024-03-22T01:26:55.275 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:54 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: +---[RSA 3072]----+ 2024-03-22T01:26:55.275 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:54 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: |=O++o o . | 2024-03-22T01:26:55.275 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:54 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: |OE*ooo . + | 2024-03-22T01:26:55.275 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:54 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: |OB+o .o o | 2024-03-22T01:26:55.275 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:54 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: |B+B . + | 2024-03-22T01:26:55.275 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:54 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: | Oo. S + . | 2024-03-22T01:26:55.275 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:54 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: |+.o . o o | 2024-03-22T01:26:55.275 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:54 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: | = . o | 2024-03-22T01:26:55.275 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:54 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: | o . | 2024-03-22T01:26:55.275 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:54 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: | o. | 2024-03-22T01:26:55.276 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:26:54 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: +----[SHA256]-----+ 2024-03-22T01:26:55.535 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:54.452+0000 7fec31946700 1 Processor -- start 2024-03-22T01:26:55.535 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:54.452+0000 7fec31946700 1 -- start start 2024-03-22T01:26:55.535 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:54.452+0000 7fec31946700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec2c107010 0x7fec2c107430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:26:55.535 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:54.452+0000 7fec31946700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fec2c107a00 con 0x7fec2c107010 2024-03-22T01:26:55.535 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:54.452+0000 7fec2affd700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec2c107010 0x7fec2c107430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:26:55.535 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:54.452+0000 7fec2affd700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec2c107010 0x7fec2c107430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44226/0 (socket says 172.21.15.72:44226) 2024-03-22T01:26:55.535 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:54.452+0000 7fec2affd700 1 -- 172.21.15.72:0/1024542201 learned_addr learned my addr 172.21.15.72:0/1024542201 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:26:55.535 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:54.453+0000 7fec2affd700 1 -- 172.21.15.72:0/1024542201 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec2c108210 con 0x7fec2c107010 2024-03-22T01:26:55.535 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:54.453+0000 7fec2affd700 1 --2- 172.21.15.72:0/1024542201 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec2c107010 0x7fec2c107430 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fec1c0098a0 tx=0x7fec1c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd0f28cb6845c309 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:26:55.535 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:54.453+0000 7fec2a7fc700 1 -- 172.21.15.72:0/1024542201 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fec1c004160 con 0x7fec2c107010 2024-03-22T01:26:55.535 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:54.453+0000 7fec2a7fc700 1 -- 172.21.15.72:0/1024542201 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fec1c004300 con 0x7fec2c107010 2024-03-22T01:26:55.536 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:54.453+0000 7fec2a7fc700 1 -- 172.21.15.72:0/1024542201 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fec1c011b50 con 0x7fec2c107010 2024-03-22T01:26:55.536 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:54.454+0000 7fec31946700 1 -- 172.21.15.72:0/1024542201 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec2c107010 msgr2=0x7fec2c107430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:26:55.536 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:54.454+0000 7fec31946700 1 --2- 172.21.15.72:0/1024542201 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec2c107010 0x7fec2c107430 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fec1c0098a0 tx=0x7fec1c009bb0 comp rx=0 tx=0).stop 2024-03-22T01:26:55.536 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:54.454+0000 7fec31946700 1 -- 172.21.15.72:0/1024542201 shutdown_connections 2024-03-22T01:26:55.536 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:54.454+0000 7fec31946700 1 --2- 172.21.15.72:0/1024542201 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec2c107010 0x7fec2c107430 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:26:55.536 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:54.454+0000 7fec31946700 1 -- 172.21.15.72:0/1024542201 >> 172.21.15.72:0/1024542201 conn(0x7fec2c1025d0 msgr2=0x7fec2c1049f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:26:55.536 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:54.454+0000 7fec31946700 1 -- 172.21.15.72:0/1024542201 shutdown_connections 2024-03-22T01:26:55.536 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:54.454+0000 7fec31946700 1 -- 172.21.15.72:0/1024542201 wait complete. 2024-03-22T01:26:55.536 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:54.454+0000 7fec31946700 1 Processor -- start 2024-03-22T01:26:55.536 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:54.454+0000 7fec31946700 1 -- start start 2024-03-22T01:26:55.536 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:54.455+0000 7fec31946700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec2c107010 0x7fec2c19c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:26:55.536 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:54.455+0000 7fec31946700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fec2c10b5c0 con 0x7fec2c107010 2024-03-22T01:26:55.536 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:54.455+0000 7fec2affd700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec2c107010 0x7fec2c19c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:26:55.537 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:54.455+0000 7fec2affd700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec2c107010 0x7fec2c19c070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44228/0 (socket says 172.21.15.72:44228) 2024-03-22T01:26:55.537 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:54.455+0000 7fec2affd700 1 -- 172.21.15.72:0/715026741 learned_addr learned my addr 172.21.15.72:0/715026741 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:26:55.537 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:54.455+0000 7fec2affd700 1 -- 172.21.15.72:0/715026741 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec2c19c5b0 con 0x7fec2c107010 2024-03-22T01:26:55.537 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:54.455+0000 7fec2affd700 1 --2- 172.21.15.72:0/715026741 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec2c107010 0x7fec2c19c070 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fec1c009fb0 tx=0x7fec1c011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:26:55.537 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:54.455+0000 7fec30944700 1 -- 172.21.15.72:0/715026741 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fec1c0038b0 con 0x7fec2c107010 2024-03-22T01:26:55.537 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:54.455+0000 7fec30944700 1 -- 172.21.15.72:0/715026741 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fec1c011e50 con 0x7fec2c107010 2024-03-22T01:26:55.537 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:54.455+0000 7fec30944700 1 -- 172.21.15.72:0/715026741 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fec1c019550 con 0x7fec2c107010 2024-03-22T01:26:55.537 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:54.455+0000 7fec31946700 1 -- 172.21.15.72:0/715026741 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec2c19c840 con 0x7fec2c107010 2024-03-22T01:26:55.537 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:54.455+0000 7fec31946700 1 -- 172.21.15.72:0/715026741 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec2c19f530 con 0x7fec2c107010 2024-03-22T01:26:55.537 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:54.456+0000 7fec30944700 1 -- 172.21.15.72:0/715026741 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 8) ==== 46294+0+0 (secure 0 0 0) 0x7fec1c019850 con 0x7fec2c107010 2024-03-22T01:26:55.537 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:54.456+0000 7fec31946700 1 -- 172.21.15.72:0/715026741 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec2c063cb0 con 0x7fec2c107010 2024-03-22T01:26:55.537 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:54.456+0000 7fec30944700 1 --2- 172.21.15.72:0/715026741 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7fec180391f0 0x7fec1803b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:26:55.537 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:54.456+0000 7fec30944700 1 -- 172.21.15.72:0/715026741 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fec1c04d410 con 0x7fec2c107010 2024-03-22T01:26:55.537 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:54.456+0000 7fec227fc700 1 --2- 172.21.15.72:0/715026741 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7fec180391f0 0x7fec1803b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:26:55.538 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:54.457+0000 7fec227fc700 1 --2- 172.21.15.72:0/715026741 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7fec180391f0 0x7fec1803b6b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fec14006fd0 tx=0x7fec14006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:26:55.538 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:54.461+0000 7fec30944700 1 -- 172.21.15.72:0/715026741 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fec1c00f2e0 con 0x7fec2c107010 2024-03-22T01:26:55.538 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:54.659+0000 7fec31946700 1 -- 172.21.15.72:0/715026741 --> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7fec2c0008d0 con 0x7fec180391f0 2024-03-22T01:26:55.538 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:54.971+0000 7fec30944700 1 -- 172.21.15.72:0/715026741 <== mgr.14120 v2:172.21.15.72:6800/1513232913 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fec2c0008d0 con 0x7fec180391f0 2024-03-22T01:26:55.538 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:54.973+0000 7fec31946700 1 -- 172.21.15.72:0/715026741 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7fec180391f0 msgr2=0x7fec1803b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:26:55.538 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:54.973+0000 7fec31946700 1 --2- 172.21.15.72:0/715026741 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7fec180391f0 0x7fec1803b6b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fec14006fd0 tx=0x7fec14006e40 comp rx=0 tx=0).stop 2024-03-22T01:26:55.538 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:54.973+0000 7fec31946700 1 -- 172.21.15.72:0/715026741 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec2c107010 msgr2=0x7fec2c19c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:26:55.538 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:54.973+0000 7fec31946700 1 --2- 172.21.15.72:0/715026741 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec2c107010 0x7fec2c19c070 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fec1c009fb0 tx=0x7fec1c011bf0 comp rx=0 tx=0).stop 2024-03-22T01:26:55.538 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:54.973+0000 7fec31946700 1 -- 172.21.15.72:0/715026741 shutdown_connections 2024-03-22T01:26:55.538 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:54.973+0000 7fec31946700 1 --2- 172.21.15.72:0/715026741 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7fec180391f0 0x7fec1803b6b0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:26:55.538 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:54.973+0000 7fec31946700 1 --2- 172.21.15.72:0/715026741 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec2c107010 0x7fec2c19c070 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:26:55.538 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:54.974+0000 7fec31946700 1 -- 172.21.15.72:0/715026741 >> 172.21.15.72:0/715026741 conn(0x7fec2c1025d0 msgr2=0x7fec2c1033e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:26:55.539 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:54.974+0000 7fec31946700 1 -- 172.21.15.72:0/715026741 shutdown_connections 2024-03-22T01:26:55.539 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:54.974+0000 7fec31946700 1 -- 172.21.15.72:0/715026741 wait complete. 2024-03-22T01:26:56.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:56 smithi072 ceph-mon[32414]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:26:56.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:56 smithi072 ceph-mon[32414]: Generating ssh key... 2024-03-22T01:26:56.641 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCqbdTGzUNA4dKEUFqNy/KZI9cTM28jUFtRQ5/XYhqEZL3s9G8rf/nzrptmYNympWhRnYmwO3o/vrIGHTrMUMBNHDw3n/K+ZbaCM8nPEgbCXSrvMMFTTAu9xGsP1DI1bJTDnZmWOXDCBawkSLBt+xS1ThnAmtHtTQZjaONP5VHTDrsRnVZXvnyZCyVz/ewpgXrXHmFkjwamnDREOKsrMgj2O+i4KS6lD2lDRAjw9IfNhJVHv9zrkPtsphMEjgJ2hJY+LxEY1FFflMEOhTZTxe5TocU+rNfp7FumGu7MwsGbvL9uSkfvLad/1YcvwB2SDeZN69tGKwBZXsD1ayeRrUnflkxoIiQ+UKhObJQtwSX8w3rq01puWJkiXVS398jgf1pxaiFQW50KTmBubt28cXYQ90Al8a89xMHrAFGjIET38coyVhYZfRRe1W8T3WglAS48hK0KrDaDWTH/jQjhPUpV7zAG2FepmcmJtb1aqNY+agHkJiPamzhpCfz6xxysod8= ceph-022a99bc-e7eb-11ee-95cd-87774f69a715 2024-03-22T01:26:56.641 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:55.905+0000 7fe0f955e700 1 Processor -- start 2024-03-22T01:26:56.641 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:55.905+0000 7fe0f955e700 1 -- start start 2024-03-22T01:26:56.641 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:55.905+0000 7fe0f955e700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe0f4108170 0x7fe0f4108590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:26:56.641 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:55.905+0000 7fe0f955e700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fe0f4108b60 con 0x7fe0f4108170 2024-03-22T01:26:56.641 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:55.906+0000 7fe0f2ffd700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe0f4108170 0x7fe0f4108590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:26:56.641 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:55.906+0000 7fe0f2ffd700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe0f4108170 0x7fe0f4108590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44236/0 (socket says 172.21.15.72:44236) 2024-03-22T01:26:56.641 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:55.906+0000 7fe0f2ffd700 1 -- 172.21.15.72:0/3032301173 learned_addr learned my addr 172.21.15.72:0/3032301173 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:26:56.642 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:55.906+0000 7fe0f2ffd700 1 -- 172.21.15.72:0/3032301173 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0f41092d0 con 0x7fe0f4108170 2024-03-22T01:26:56.642 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:55.906+0000 7fe0f2ffd700 1 --2- 172.21.15.72:0/3032301173 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe0f4108170 0x7fe0f4108590 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec0098a0 tx=0x7fe0ec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e263e3b98e6b7a7a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:26:56.642 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:55.906+0000 7fe0f1ffb700 1 -- 172.21.15.72:0/3032301173 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe0ec004160 con 0x7fe0f4108170 2024-03-22T01:26:56.642 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:55.906+0000 7fe0f1ffb700 1 -- 172.21.15.72:0/3032301173 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe0ec004300 con 0x7fe0f4108170 2024-03-22T01:26:56.642 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:55.906+0000 7fe0f1ffb700 1 -- 172.21.15.72:0/3032301173 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe0ec011b50 con 0x7fe0f4108170 2024-03-22T01:26:56.642 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:55.907+0000 7fe0f955e700 1 -- 172.21.15.72:0/3032301173 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe0f4108170 msgr2=0x7fe0f4108590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:26:56.642 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:55.907+0000 7fe0f955e700 1 --2- 172.21.15.72:0/3032301173 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe0f4108170 0x7fe0f4108590 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec0098a0 tx=0x7fe0ec009bb0 comp rx=0 tx=0).stop 2024-03-22T01:26:56.642 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:55.907+0000 7fe0f955e700 1 -- 172.21.15.72:0/3032301173 shutdown_connections 2024-03-22T01:26:56.642 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:55.907+0000 7fe0f955e700 1 --2- 172.21.15.72:0/3032301173 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe0f4108170 0x7fe0f4108590 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:26:56.642 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:55.907+0000 7fe0f955e700 1 -- 172.21.15.72:0/3032301173 >> 172.21.15.72:0/3032301173 conn(0x7fe0f4103750 msgr2=0x7fe0f4105b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:26:56.642 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:55.907+0000 7fe0f955e700 1 -- 172.21.15.72:0/3032301173 shutdown_connections 2024-03-22T01:26:56.642 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:55.907+0000 7fe0f955e700 1 -- 172.21.15.72:0/3032301173 wait complete. 2024-03-22T01:26:56.642 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:55.907+0000 7fe0f955e700 1 Processor -- start 2024-03-22T01:26:56.643 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:55.908+0000 7fe0f955e700 1 -- start start 2024-03-22T01:26:56.643 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:55.908+0000 7fe0f955e700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe0f4108170 0x7fe0f419c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:26:56.643 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:55.908+0000 7fe0f955e700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fe0f410c680 con 0x7fe0f4108170 2024-03-22T01:26:56.643 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:55.908+0000 7fe0f2ffd700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe0f4108170 0x7fe0f419c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:26:56.643 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:55.908+0000 7fe0f2ffd700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe0f4108170 0x7fe0f419c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44244/0 (socket says 172.21.15.72:44244) 2024-03-22T01:26:56.643 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:55.908+0000 7fe0f2ffd700 1 -- 172.21.15.72:0/608618905 learned_addr learned my addr 172.21.15.72:0/608618905 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:26:56.643 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:55.908+0000 7fe0f2ffd700 1 -- 172.21.15.72:0/608618905 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0f419c540 con 0x7fe0f4108170 2024-03-22T01:26:56.643 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:55.908+0000 7fe0f2ffd700 1 --2- 172.21.15.72:0/608618905 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe0f4108170 0x7fe0f419c000 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec009fb0 tx=0x7fe0ec011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:26:56.643 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:55.908+0000 7fe0dbfff700 1 -- 172.21.15.72:0/608618905 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe0ec0038b0 con 0x7fe0f4108170 2024-03-22T01:26:56.643 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:55.908+0000 7fe0dbfff700 1 -- 172.21.15.72:0/608618905 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe0ec011e50 con 0x7fe0f4108170 2024-03-22T01:26:56.643 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:55.909+0000 7fe0f955e700 1 -- 172.21.15.72:0/608618905 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0f419c7d0 con 0x7fe0f4108170 2024-03-22T01:26:56.643 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:55.909+0000 7fe0dbfff700 1 -- 172.21.15.72:0/608618905 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe0ec019550 con 0x7fe0f4108170 2024-03-22T01:26:56.643 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:55.909+0000 7fe0f955e700 1 -- 172.21.15.72:0/608618905 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0f419f4c0 con 0x7fe0f4108170 2024-03-22T01:26:56.643 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:55.909+0000 7fe0dbfff700 1 -- 172.21.15.72:0/608618905 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 8) ==== 46294+0+0 (secure 0 0 0) 0x7fe0ec019800 con 0x7fe0f4108170 2024-03-22T01:26:56.644 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:55.910+0000 7fe0f955e700 1 -- 172.21.15.72:0/608618905 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0f4063cb0 con 0x7fe0f4108170 2024-03-22T01:26:56.644 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:55.910+0000 7fe0dbfff700 1 --2- 172.21.15.72:0/608618905 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7fe0dc039240 0x7fe0dc03b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:26:56.644 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:55.910+0000 7fe0dbfff700 1 -- 172.21.15.72:0/608618905 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fe0ec04d3d0 con 0x7fe0f4108170 2024-03-22T01:26:56.644 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:55.910+0000 7fe0f27fc700 1 --2- 172.21.15.72:0/608618905 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7fe0dc039240 0x7fe0dc03b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:26:56.644 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:55.910+0000 7fe0f27fc700 1 --2- 172.21.15.72:0/608618905 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7fe0dc039240 0x7fe0dc03b700 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe0e800ad30 tx=0x7fe0e80093f0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:26:56.644 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:55.915+0000 7fe0dbfff700 1 -- 172.21.15.72:0/608618905 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe0ec020240 con 0x7fe0f4108170 2024-03-22T01:26:56.644 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:56.111+0000 7fe0f955e700 1 -- 172.21.15.72:0/608618905 --> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7fe0f41048e0 con 0x7fe0dc039240 2024-03-22T01:26:56.644 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:56.112+0000 7fe0dbfff700 1 -- 172.21.15.72:0/608618905 <== mgr.14120 v2:172.21.15.72:6800/1513232913 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7fe0f41048e0 con 0x7fe0dc039240 2024-03-22T01:26:56.644 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:56.114+0000 7fe0f955e700 1 -- 172.21.15.72:0/608618905 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7fe0dc039240 msgr2=0x7fe0dc03b700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:26:56.644 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:56.114+0000 7fe0f955e700 1 --2- 172.21.15.72:0/608618905 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7fe0dc039240 0x7fe0dc03b700 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe0e800ad30 tx=0x7fe0e80093f0 comp rx=0 tx=0).stop 2024-03-22T01:26:56.644 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:56.114+0000 7fe0f955e700 1 -- 172.21.15.72:0/608618905 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe0f4108170 msgr2=0x7fe0f419c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:26:56.644 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:56.114+0000 7fe0f955e700 1 --2- 172.21.15.72:0/608618905 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe0f4108170 0x7fe0f419c000 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec009fb0 tx=0x7fe0ec011bf0 comp rx=0 tx=0).stop 2024-03-22T01:26:56.644 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:56.114+0000 7fe0f955e700 1 -- 172.21.15.72:0/608618905 shutdown_connections 2024-03-22T01:26:56.644 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:56.114+0000 7fe0f955e700 1 --2- 172.21.15.72:0/608618905 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7fe0dc039240 0x7fe0dc03b700 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:26:56.645 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:56.114+0000 7fe0f955e700 1 --2- 172.21.15.72:0/608618905 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe0f4108170 0x7fe0f419c000 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:26:56.645 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:56.114+0000 7fe0f955e700 1 -- 172.21.15.72:0/608618905 >> 172.21.15.72:0/608618905 conn(0x7fe0f4103750 msgr2=0x7fe0f41041d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:26:56.645 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:56.114+0000 7fe0f955e700 1 -- 172.21.15.72:0/608618905 shutdown_connections 2024-03-22T01:26:56.645 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:56.114+0000 7fe0f955e700 1 -- 172.21.15.72:0/608618905 wait complete. 2024-03-22T01:26:56.645 INFO:teuthology.orchestra.run.smithi072.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-22T01:26:56.645 INFO:teuthology.orchestra.run.smithi072.stdout:Adding key to root@localhost authorized_keys... 2024-03-22T01:26:56.645 INFO:teuthology.orchestra.run.smithi072.stdout:Adding host smithi072... 2024-03-22T01:26:57.226 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:57 smithi072 ceph-mon[32414]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:26:58.206 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:58 smithi072 ceph-mon[32414]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi072", "addr": "172.21.15.72", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:26:59.621 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:26:59 smithi072 ceph-mon[32414]: Deploying cephadm binary to smithi072 2024-03-22T01:27:01.968 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout Added host 'smithi072' with addr '172.21.15.72' 2024-03-22T01:27:01.968 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:57.015+0000 7f786cf27700 1 Processor -- start 2024-03-22T01:27:01.968 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:57.015+0000 7f786cf27700 1 -- start start 2024-03-22T01:27:01.968 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:57.016+0000 7f786cf27700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7868108190 0x7f78681085b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:27:01.968 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:57.016+0000 7f786cf27700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f7868108b80 con 0x7f7868108190 2024-03-22T01:27:01.968 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:57.016+0000 7f786659c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7868108190 0x7f78681085b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:27:01.968 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:57.016+0000 7f786659c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7868108190 0x7f78681085b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44256/0 (socket says 172.21.15.72:44256) 2024-03-22T01:27:01.968 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:57.016+0000 7f786659c700 1 -- 172.21.15.72:0/3445434487 learned_addr learned my addr 172.21.15.72:0/3445434487 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:27:01.968 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:57.016+0000 7f786659c700 1 -- 172.21.15.72:0/3445434487 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78681092f0 con 0x7f7868108190 2024-03-22T01:27:01.968 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:57.016+0000 7f786659c700 1 --2- 172.21.15.72:0/3445434487 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7868108190 0x7f78681085b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f78580098a0 tx=0x7f7858009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=200c157ff27f1986 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:27:01.969 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:57.016+0000 7f786559a700 1 -- 172.21.15.72:0/3445434487 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7858004160 con 0x7f7868108190 2024-03-22T01:27:01.969 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:57.016+0000 7f786559a700 1 -- 172.21.15.72:0/3445434487 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7858004300 con 0x7f7868108190 2024-03-22T01:27:01.969 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:57.016+0000 7f786559a700 1 -- 172.21.15.72:0/3445434487 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7858011b50 con 0x7f7868108190 2024-03-22T01:27:01.969 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:57.017+0000 7f786cf27700 1 -- 172.21.15.72:0/3445434487 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7868108190 msgr2=0x7f78681085b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:01.969 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:57.017+0000 7f786cf27700 1 --2- 172.21.15.72:0/3445434487 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7868108190 0x7f78681085b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f78580098a0 tx=0x7f7858009bb0 comp rx=0 tx=0).stop 2024-03-22T01:27:01.969 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:57.017+0000 7f786cf27700 1 -- 172.21.15.72:0/3445434487 shutdown_connections 2024-03-22T01:27:01.969 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:57.017+0000 7f786cf27700 1 --2- 172.21.15.72:0/3445434487 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7868108190 0x7f78681085b0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:01.969 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:57.017+0000 7f786cf27700 1 -- 172.21.15.72:0/3445434487 >> 172.21.15.72:0/3445434487 conn(0x7f7868103790 msgr2=0x7f7868105bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:27:01.969 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:57.017+0000 7f786cf27700 1 -- 172.21.15.72:0/3445434487 shutdown_connections 2024-03-22T01:27:01.969 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:57.017+0000 7f786cf27700 1 -- 172.21.15.72:0/3445434487 wait complete. 2024-03-22T01:27:01.969 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:57.018+0000 7f786cf27700 1 Processor -- start 2024-03-22T01:27:01.969 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:57.018+0000 7f786cf27700 1 -- start start 2024-03-22T01:27:01.969 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:57.018+0000 7f786cf27700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7868108190 0x7f786819c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:27:01.969 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:57.018+0000 7f786cf27700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f786810c6a0 con 0x7f7868108190 2024-03-22T01:27:01.970 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:57.018+0000 7f786659c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7868108190 0x7f786819c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:27:01.970 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:57.018+0000 7f786659c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7868108190 0x7f786819c0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44262/0 (socket says 172.21.15.72:44262) 2024-03-22T01:27:01.970 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:57.018+0000 7f786659c700 1 -- 172.21.15.72:0/1254667660 learned_addr learned my addr 172.21.15.72:0/1254667660 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:27:01.970 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:57.018+0000 7f786659c700 1 -- 172.21.15.72:0/1254667660 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f786819c5e0 con 0x7f7868108190 2024-03-22T01:27:01.970 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:57.018+0000 7f786659c700 1 --2- 172.21.15.72:0/1254667660 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7868108190 0x7f786819c0a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7858009fb0 tx=0x7f7858011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:27:01.970 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:57.019+0000 7f78577fe700 1 -- 172.21.15.72:0/1254667660 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f78580038b0 con 0x7f7868108190 2024-03-22T01:27:01.970 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:57.019+0000 7f78577fe700 1 -- 172.21.15.72:0/1254667660 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7858011e50 con 0x7f7868108190 2024-03-22T01:27:01.970 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:57.019+0000 7f786cf27700 1 -- 172.21.15.72:0/1254667660 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f786819c870 con 0x7f7868108190 2024-03-22T01:27:01.970 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:57.019+0000 7f78577fe700 1 -- 172.21.15.72:0/1254667660 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7858019550 con 0x7f7868108190 2024-03-22T01:27:01.970 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:57.019+0000 7f786cf27700 1 -- 172.21.15.72:0/1254667660 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f786819f560 con 0x7f7868108190 2024-03-22T01:27:01.970 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:57.019+0000 7f78577fe700 1 -- 172.21.15.72:0/1254667660 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 8) ==== 46294+0+0 (secure 0 0 0) 0x7f7858019800 con 0x7f7868108190 2024-03-22T01:27:01.970 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:57.019+0000 7f786cf27700 1 -- 172.21.15.72:0/1254667660 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7868063cb0 con 0x7f7868108190 2024-03-22T01:27:01.970 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:57.020+0000 7f78577fe700 1 --2- 172.21.15.72:0/1254667660 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7f7850039240 0x7f785003b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:27:01.970 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:57.020+0000 7f78577fe700 1 -- 172.21.15.72:0/1254667660 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f785804d530 con 0x7f7868108190 2024-03-22T01:27:01.971 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:57.020+0000 7f7865d9b700 1 --2- 172.21.15.72:0/1254667660 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7f7850039240 0x7f785003b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:27:01.971 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:57.020+0000 7f7865d9b700 1 --2- 172.21.15.72:0/1254667660 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7f7850039240 0x7f785003b700 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f785c006fd0 tx=0x7f785c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:27:01.971 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:57.025+0000 7f78577fe700 1 -- 172.21.15.72:0/1254667660 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7858003ac0 con 0x7f7868108190 2024-03-22T01:27:01.971 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:26:57.228+0000 7f786cf27700 1 -- 172.21.15.72:0/1254667660 --> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi072", "addr": "172.21.15.72", "target": ["mon-mgr", ""]}) -- 0x7f7868104990 con 0x7f7850039240 2024-03-22T01:27:01.971 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:01.401+0000 7f78577fe700 1 -- 172.21.15.72:0/1254667660 <== mgr.14120 v2:172.21.15.72:6800/1513232913 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f7868104990 con 0x7f7850039240 2024-03-22T01:27:01.971 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:01.404+0000 7f786cf27700 1 -- 172.21.15.72:0/1254667660 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7f7850039240 msgr2=0x7f785003b700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:01.971 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:01.404+0000 7f786cf27700 1 --2- 172.21.15.72:0/1254667660 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7f7850039240 0x7f785003b700 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f785c006fd0 tx=0x7f785c006e40 comp rx=0 tx=0).stop 2024-03-22T01:27:01.971 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:01.404+0000 7f786cf27700 1 -- 172.21.15.72:0/1254667660 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7868108190 msgr2=0x7f786819c0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:01.971 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:01.404+0000 7f786cf27700 1 --2- 172.21.15.72:0/1254667660 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7868108190 0x7f786819c0a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7858009fb0 tx=0x7f7858011bf0 comp rx=0 tx=0).stop 2024-03-22T01:27:01.971 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:01.404+0000 7f786cf27700 1 -- 172.21.15.72:0/1254667660 shutdown_connections 2024-03-22T01:27:01.971 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:01.404+0000 7f786cf27700 1 --2- 172.21.15.72:0/1254667660 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7f7850039240 0x7f785003b700 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:01.971 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:01.404+0000 7f786cf27700 1 --2- 172.21.15.72:0/1254667660 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7868108190 0x7f786819c0a0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:01.971 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:01.404+0000 7f786cf27700 1 -- 172.21.15.72:0/1254667660 >> 172.21.15.72:0/1254667660 conn(0x7f7868103790 msgr2=0x7f7868104280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:27:01.972 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:01.404+0000 7f786cf27700 1 -- 172.21.15.72:0/1254667660 shutdown_connections 2024-03-22T01:27:01.972 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:01.404+0000 7f786cf27700 1 -- 172.21.15.72:0/1254667660 wait complete. 2024-03-22T01:27:01.972 INFO:teuthology.orchestra.run.smithi072.stdout:Deploying unmanaged mon service... 2024-03-22T01:27:02.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:02 smithi072 ceph-mon[32414]: from='mgr.14120 172.21.15.72:0/4135194081' entity='mgr.x' 2024-03-22T01:27:02.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:02 smithi072 ceph-mon[32414]: Added host smithi072 2024-03-22T01:27:02.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:02 smithi072 ceph-mon[32414]: from='mgr.14120 172.21.15.72:0/4135194081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:27:03.224 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-22T01:27:03.224 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:02.399+0000 7f7f5f31e700 1 Processor -- start 2024-03-22T01:27:03.224 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:02.400+0000 7f7f5f31e700 1 -- start start 2024-03-22T01:27:03.224 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:02.400+0000 7f7f5f31e700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7f58108c00 0x7f7f5810aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:27:03.225 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:02.400+0000 7f7f5f31e700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f7f5807be10 con 0x7f7f58108c00 2024-03-22T01:27:03.225 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:02.400+0000 7f7f5d0ba700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7f58108c00 0x7f7f5810aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:27:03.225 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:02.400+0000 7f7f5d0ba700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7f58108c00 0x7f7f5810aff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:41354/0 (socket says 172.21.15.72:41354) 2024-03-22T01:27:03.225 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:02.400+0000 7f7f5d0ba700 1 -- 172.21.15.72:0/1479195820 learned_addr learned my addr 172.21.15.72:0/1479195820 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:27:03.225 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:02.400+0000 7f7f5d0ba700 1 -- 172.21.15.72:0/1479195820 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f5810b530 con 0x7f7f58108c00 2024-03-22T01:27:03.225 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:02.401+0000 7f7f5d0ba700 1 --2- 172.21.15.72:0/1479195820 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7f58108c00 0x7f7f5810aff0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7f480098a0 tx=0x7f7f48009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5cce2cf454c3418 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:27:03.225 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:02.402+0000 7f7f4ffff700 1 -- 172.21.15.72:0/1479195820 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7f48004160 con 0x7f7f58108c00 2024-03-22T01:27:03.225 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:02.403+0000 7f7f4ffff700 1 -- 172.21.15.72:0/1479195820 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7f48004300 con 0x7f7f58108c00 2024-03-22T01:27:03.225 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:02.403+0000 7f7f4ffff700 1 -- 172.21.15.72:0/1479195820 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7f48011b50 con 0x7f7f58108c00 2024-03-22T01:27:03.225 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:02.403+0000 7f7f5f31e700 1 -- 172.21.15.72:0/1479195820 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7f58108c00 msgr2=0x7f7f5810aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:03.225 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:02.403+0000 7f7f5f31e700 1 --2- 172.21.15.72:0/1479195820 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7f58108c00 0x7f7f5810aff0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7f480098a0 tx=0x7f7f48009bb0 comp rx=0 tx=0).stop 2024-03-22T01:27:03.225 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:02.403+0000 7f7f5f31e700 1 -- 172.21.15.72:0/1479195820 shutdown_connections 2024-03-22T01:27:03.226 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:02.403+0000 7f7f5f31e700 1 --2- 172.21.15.72:0/1479195820 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7f58108c00 0x7f7f5810aff0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:03.226 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:02.403+0000 7f7f5f31e700 1 -- 172.21.15.72:0/1479195820 >> 172.21.15.72:0/1479195820 conn(0x7f7f581025d0 msgr2=0x7f7f581049f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:27:03.226 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:02.403+0000 7f7f5f31e700 1 -- 172.21.15.72:0/1479195820 shutdown_connections 2024-03-22T01:27:03.226 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:02.403+0000 7f7f5f31e700 1 -- 172.21.15.72:0/1479195820 wait complete. 2024-03-22T01:27:03.226 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:02.404+0000 7f7f5f31e700 1 Processor -- start 2024-03-22T01:27:03.226 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:02.404+0000 7f7f5f31e700 1 -- start start 2024-03-22T01:27:03.226 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:02.404+0000 7f7f5f31e700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7f58108c00 0x7f7f58108480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:27:03.226 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:02.404+0000 7f7f5f31e700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f7f5810c4f0 con 0x7f7f58108c00 2024-03-22T01:27:03.226 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:02.404+0000 7f7f5d0ba700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7f58108c00 0x7f7f58108480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:27:03.226 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:02.404+0000 7f7f5d0ba700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7f58108c00 0x7f7f58108480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:41360/0 (socket says 172.21.15.72:41360) 2024-03-22T01:27:03.226 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:02.404+0000 7f7f5d0ba700 1 -- 172.21.15.72:0/3397688215 learned_addr learned my addr 172.21.15.72:0/3397688215 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:27:03.226 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:02.404+0000 7f7f5d0ba700 1 -- 172.21.15.72:0/3397688215 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f581089c0 con 0x7f7f58108c00 2024-03-22T01:27:03.226 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:02.405+0000 7f7f5d0ba700 1 --2- 172.21.15.72:0/3397688215 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7f58108c00 0x7f7f58108480 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7f48011b60 tx=0x7f7f48011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:27:03.227 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:02.405+0000 7f7f4e7fc700 1 -- 172.21.15.72:0/3397688215 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7f48003970 con 0x7f7f58108c00 2024-03-22T01:27:03.227 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:02.405+0000 7f7f4e7fc700 1 -- 172.21.15.72:0/3397688215 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7f48003b10 con 0x7f7f58108c00 2024-03-22T01:27:03.227 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:02.405+0000 7f7f4e7fc700 1 -- 172.21.15.72:0/3397688215 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7f48024610 con 0x7f7f58108c00 2024-03-22T01:27:03.227 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:02.405+0000 7f7f5f31e700 1 -- 172.21.15.72:0/3397688215 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f58106bb0 con 0x7f7f58108c00 2024-03-22T01:27:03.227 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:02.405+0000 7f7f5f31e700 1 -- 172.21.15.72:0/3397688215 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f581070f0 con 0x7f7f58108c00 2024-03-22T01:27:03.227 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:02.406+0000 7f7f4e7fc700 1 -- 172.21.15.72:0/3397688215 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 8) ==== 46294+0+0 (secure 0 0 0) 0x7f7f480248c0 con 0x7f7f58108c00 2024-03-22T01:27:03.227 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:02.406+0000 7f7f5f31e700 1 -- 172.21.15.72:0/3397688215 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f58063cb0 con 0x7f7f58108c00 2024-03-22T01:27:03.227 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:02.406+0000 7f7f4e7fc700 1 --2- 172.21.15.72:0/3397688215 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7f7f44039290 0x7f7f4403b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:27:03.227 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:02.406+0000 7f7f4e7fc700 1 -- 172.21.15.72:0/3397688215 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f7f48011e00 con 0x7f7f58108c00 2024-03-22T01:27:03.227 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:02.406+0000 7f7f5c8b9700 1 --2- 172.21.15.72:0/3397688215 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7f7f44039290 0x7f7f4403b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:27:03.227 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:02.406+0000 7f7f5c8b9700 1 --2- 172.21.15.72:0/3397688215 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7f7f44039290 0x7f7f4403b750 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7f54006fd0 tx=0x7f7f54006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:27:03.227 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:02.411+0000 7f7f4e7fc700 1 -- 172.21.15.72:0/3397688215 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7f4800ba00 con 0x7f7f58108c00 2024-03-22T01:27:03.227 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:02.644+0000 7f7f5f31e700 1 -- 172.21.15.72:0/3397688215 --> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f7f58103750 con 0x7f7f44039290 2024-03-22T01:27:03.227 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:02.652+0000 7f7f4e7fc700 1 -- 172.21.15.72:0/3397688215 <== mgr.14120 v2:172.21.15.72:6800/1513232913 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f7f58103750 con 0x7f7f44039290 2024-03-22T01:27:03.228 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:02.654+0000 7f7f5f31e700 1 -- 172.21.15.72:0/3397688215 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7f7f44039290 msgr2=0x7f7f4403b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:03.228 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:02.654+0000 7f7f5f31e700 1 --2- 172.21.15.72:0/3397688215 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7f7f44039290 0x7f7f4403b750 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7f54006fd0 tx=0x7f7f54006e40 comp rx=0 tx=0).stop 2024-03-22T01:27:03.228 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:02.654+0000 7f7f5f31e700 1 -- 172.21.15.72:0/3397688215 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7f58108c00 msgr2=0x7f7f58108480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:03.228 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:02.654+0000 7f7f5f31e700 1 --2- 172.21.15.72:0/3397688215 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7f58108c00 0x7f7f58108480 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7f48011b60 tx=0x7f7f48011c40 comp rx=0 tx=0).stop 2024-03-22T01:27:03.228 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:02.654+0000 7f7f5f31e700 1 -- 172.21.15.72:0/3397688215 shutdown_connections 2024-03-22T01:27:03.228 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:02.654+0000 7f7f5f31e700 1 --2- 172.21.15.72:0/3397688215 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7f7f44039290 0x7f7f4403b750 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:03.228 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:02.654+0000 7f7f5f31e700 1 --2- 172.21.15.72:0/3397688215 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7f58108c00 0x7f7f58108480 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:03.228 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:02.654+0000 7f7f5f31e700 1 -- 172.21.15.72:0/3397688215 >> 172.21.15.72:0/3397688215 conn(0x7f7f581025d0 msgr2=0x7f7f58103040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:27:03.228 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:02.654+0000 7f7f5f31e700 1 -- 172.21.15.72:0/3397688215 shutdown_connections 2024-03-22T01:27:03.228 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:02.654+0000 7f7f5f31e700 1 -- 172.21.15.72:0/3397688215 wait complete. 2024-03-22T01:27:03.228 INFO:teuthology.orchestra.run.smithi072.stdout:Deploying unmanaged mgr service... 2024-03-22T01:27:03.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:03 smithi072 ceph-mon[32414]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:27:03.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:03 smithi072 ceph-mon[32414]: Saving service mon spec with placement count:5 2024-03-22T01:27:03.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:03 smithi072 ceph-mon[32414]: from='mgr.14120 172.21.15.72:0/4135194081' entity='mgr.x' 2024-03-22T01:27:04.541 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-22T01:27:04.542 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:03.794+0000 7f3e5ce25700 1 Processor -- start 2024-03-22T01:27:04.542 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:03.794+0000 7f3e5ce25700 1 -- start start 2024-03-22T01:27:04.542 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:03.795+0000 7f3e5ce25700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e58108c70 0x7f3e5810b060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:27:04.542 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:03.795+0000 7f3e5ce25700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f3e5807bdb0 con 0x7f3e58108c70 2024-03-22T01:27:04.542 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:03.795+0000 7f3e5659c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e58108c70 0x7f3e5810b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:27:04.542 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:03.795+0000 7f3e5659c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e58108c70 0x7f3e5810b060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:41368/0 (socket says 172.21.15.72:41368) 2024-03-22T01:27:04.542 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:03.795+0000 7f3e5659c700 1 -- 172.21.15.72:0/3697681619 learned_addr learned my addr 172.21.15.72:0/3697681619 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:27:04.542 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:03.795+0000 7f3e5659c700 1 -- 172.21.15.72:0/3697681619 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e5810b5a0 con 0x7f3e58108c70 2024-03-22T01:27:04.542 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:03.795+0000 7f3e5659c700 1 --2- 172.21.15.72:0/3697681619 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e58108c70 0x7f3e5810b060 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3e50009e80 tx=0x7f3e5000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d25201fe42bd374b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:27:04.543 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:03.795+0000 7f3e5559a700 1 -- 172.21.15.72:0/3697681619 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3e50017070 con 0x7f3e58108c70 2024-03-22T01:27:04.543 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:03.795+0000 7f3e5559a700 1 -- 172.21.15.72:0/3697681619 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f3e5000b8c0 con 0x7f3e58108c70 2024-03-22T01:27:04.543 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:03.795+0000 7f3e5559a700 1 -- 172.21.15.72:0/3697681619 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3e500045c0 con 0x7f3e58108c70 2024-03-22T01:27:04.543 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:03.796+0000 7f3e5ce25700 1 -- 172.21.15.72:0/3697681619 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e58108c70 msgr2=0x7f3e5810b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:04.543 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:03.796+0000 7f3e5ce25700 1 --2- 172.21.15.72:0/3697681619 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e58108c70 0x7f3e5810b060 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3e50009e80 tx=0x7f3e5000b330 comp rx=0 tx=0).stop 2024-03-22T01:27:04.543 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:03.796+0000 7f3e5ce25700 1 -- 172.21.15.72:0/3697681619 shutdown_connections 2024-03-22T01:27:04.543 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:03.796+0000 7f3e5ce25700 1 --2- 172.21.15.72:0/3697681619 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e58108c70 0x7f3e5810b060 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:04.543 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:03.796+0000 7f3e5ce25700 1 -- 172.21.15.72:0/3697681619 >> 172.21.15.72:0/3697681619 conn(0x7f3e58102600 msgr2=0x7f3e58104a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:27:04.543 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:03.796+0000 7f3e5ce25700 1 -- 172.21.15.72:0/3697681619 shutdown_connections 2024-03-22T01:27:04.543 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:03.796+0000 7f3e5ce25700 1 -- 172.21.15.72:0/3697681619 wait complete. 2024-03-22T01:27:04.543 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:03.797+0000 7f3e5ce25700 1 Processor -- start 2024-03-22T01:27:04.543 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:03.797+0000 7f3e5ce25700 1 -- start start 2024-03-22T01:27:04.543 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:03.797+0000 7f3e5ce25700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e58108c70 0x7f3e581084f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:27:04.543 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:03.797+0000 7f3e5ce25700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f3e5810c500 con 0x7f3e58108c70 2024-03-22T01:27:04.544 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:03.797+0000 7f3e5659c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e58108c70 0x7f3e581084f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:27:04.544 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:03.797+0000 7f3e5659c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e58108c70 0x7f3e581084f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:41378/0 (socket says 172.21.15.72:41378) 2024-03-22T01:27:04.544 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:03.797+0000 7f3e5659c700 1 -- 172.21.15.72:0/3886751799 learned_addr learned my addr 172.21.15.72:0/3886751799 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:27:04.544 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:03.797+0000 7f3e5659c700 1 -- 172.21.15.72:0/3886751799 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e58108a30 con 0x7f3e58108c70 2024-03-22T01:27:04.544 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:03.797+0000 7f3e5659c700 1 --2- 172.21.15.72:0/3886751799 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e58108c70 0x7f3e581084f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3e50009e50 tx=0x7f3e5000bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:27:04.544 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:03.798+0000 7f3e477fe700 1 -- 172.21.15.72:0/3886751799 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3e50020020 con 0x7f3e58108c70 2024-03-22T01:27:04.544 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:03.798+0000 7f3e477fe700 1 -- 172.21.15.72:0/3886751799 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f3e50003b50 con 0x7f3e58108c70 2024-03-22T01:27:04.544 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:03.798+0000 7f3e477fe700 1 -- 172.21.15.72:0/3886751799 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3e50017040 con 0x7f3e58108c70 2024-03-22T01:27:04.544 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:03.798+0000 7f3e5ce25700 1 -- 172.21.15.72:0/3886751799 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e58106c20 con 0x7f3e58108c70 2024-03-22T01:27:04.544 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:03.798+0000 7f3e5ce25700 1 -- 172.21.15.72:0/3886751799 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e58107160 con 0x7f3e58108c70 2024-03-22T01:27:04.544 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:03.799+0000 7f3e477fe700 1 -- 172.21.15.72:0/3886751799 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 8) ==== 46294+0+0 (secure 0 0 0) 0x7f3e50023020 con 0x7f3e58108c70 2024-03-22T01:27:04.544 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:03.799+0000 7f3e5ce25700 1 -- 172.21.15.72:0/3886751799 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e58063cb0 con 0x7f3e58108c70 2024-03-22T01:27:04.544 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:03.799+0000 7f3e477fe700 1 --2- 172.21.15.72:0/3886751799 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7f3e400391f0 0x7f3e4003b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:27:04.544 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:03.799+0000 7f3e477fe700 1 -- 172.21.15.72:0/3886751799 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f3e5000f050 con 0x7f3e58108c70 2024-03-22T01:27:04.545 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:03.799+0000 7f3e55d9b700 1 --2- 172.21.15.72:0/3886751799 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7f3e400391f0 0x7f3e4003b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:27:04.545 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:03.799+0000 7f3e55d9b700 1 --2- 172.21.15.72:0/3886751799 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7f3e400391f0 0x7f3e4003b6b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c006fd0 tx=0x7f3e4c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:27:04.545 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:03.804+0000 7f3e477fe700 1 -- 172.21.15.72:0/3886751799 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3e5002a020 con 0x7f3e58108c70 2024-03-22T01:27:04.545 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:04.023+0000 7f3e5ce25700 1 -- 172.21.15.72:0/3886751799 --> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f3e58103760 con 0x7f3e400391f0 2024-03-22T01:27:04.545 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:04.030+0000 7f3e477fe700 1 -- 172.21.15.72:0/3886751799 <== mgr.14120 v2:172.21.15.72:6800/1513232913 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f3e58103760 con 0x7f3e400391f0 2024-03-22T01:27:04.545 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:04.033+0000 7f3e5ce25700 1 -- 172.21.15.72:0/3886751799 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7f3e400391f0 msgr2=0x7f3e4003b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:04.545 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:04.033+0000 7f3e5ce25700 1 --2- 172.21.15.72:0/3886751799 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7f3e400391f0 0x7f3e4003b6b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c006fd0 tx=0x7f3e4c006e40 comp rx=0 tx=0).stop 2024-03-22T01:27:04.545 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:04.033+0000 7f3e5ce25700 1 -- 172.21.15.72:0/3886751799 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e58108c70 msgr2=0x7f3e581084f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:04.545 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:04.033+0000 7f3e5ce25700 1 --2- 172.21.15.72:0/3886751799 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e58108c70 0x7f3e581084f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3e50009e50 tx=0x7f3e5000bd30 comp rx=0 tx=0).stop 2024-03-22T01:27:04.545 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:04.033+0000 7f3e5ce25700 1 -- 172.21.15.72:0/3886751799 shutdown_connections 2024-03-22T01:27:04.545 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:04.033+0000 7f3e5ce25700 1 --2- 172.21.15.72:0/3886751799 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7f3e400391f0 0x7f3e4003b6b0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:04.545 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:04.033+0000 7f3e5ce25700 1 --2- 172.21.15.72:0/3886751799 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e58108c70 0x7f3e581084f0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:04.545 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:04.033+0000 7f3e5ce25700 1 -- 172.21.15.72:0/3886751799 >> 172.21.15.72:0/3886751799 conn(0x7f3e58102600 msgr2=0x7f3e58103050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:27:04.545 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:04.033+0000 7f3e5ce25700 1 -- 172.21.15.72:0/3886751799 shutdown_connections 2024-03-22T01:27:04.546 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:04.033+0000 7f3e5ce25700 1 -- 172.21.15.72:0/3886751799 wait complete. 2024-03-22T01:27:05.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:05 smithi072 ceph-mon[32414]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:27:05.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:05 smithi072 ceph-mon[32414]: Saving service mgr spec with placement count:2 2024-03-22T01:27:05.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:05 smithi072 ceph-mon[32414]: from='mgr.14120 172.21.15.72:0/4135194081' entity='mgr.x' 2024-03-22T01:27:06.175 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:06 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/3006052737' entity='client.admin' 2024-03-22T01:27:06.175 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:06 smithi072 ceph-mon[32414]: from='mgr.14120 172.21.15.72:0/4135194081' entity='mgr.x' 2024-03-22T01:27:06.181 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:04.935+0000 7f53dd3cf700 1 Processor -- start 2024-03-22T01:27:06.182 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:04.935+0000 7f53dd3cf700 1 -- start start 2024-03-22T01:27:06.182 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:04.935+0000 7f53dd3cf700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53d8108bd0 0x7f53d810afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:27:06.182 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:04.935+0000 7f53dd3cf700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f53d8075dd0 con 0x7f53d8108bd0 2024-03-22T01:27:06.182 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:04.935+0000 7f53d6ffd700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53d8108bd0 0x7f53d810afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:27:06.182 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:04.936+0000 7f53d6ffd700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53d8108bd0 0x7f53d810afc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:41382/0 (socket says 172.21.15.72:41382) 2024-03-22T01:27:06.182 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:04.936+0000 7f53d6ffd700 1 -- 172.21.15.72:0/4155797321 learned_addr learned my addr 172.21.15.72:0/4155797321 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:27:06.182 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:04.936+0000 7f53d6ffd700 1 -- 172.21.15.72:0/4155797321 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53d810b500 con 0x7f53d8108bd0 2024-03-22T01:27:06.182 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:04.936+0000 7f53d6ffd700 1 --2- 172.21.15.72:0/4155797321 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53d8108bd0 0x7f53d810afc0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f53c8009e80 tx=0x7f53c800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f57e3dda10283cf7 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:27:06.182 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:04.936+0000 7f53d67fc700 1 -- 172.21.15.72:0/4155797321 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f53c8017070 con 0x7f53d8108bd0 2024-03-22T01:27:06.182 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:04.936+0000 7f53d67fc700 1 -- 172.21.15.72:0/4155797321 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f53c800b8c0 con 0x7f53d8108bd0 2024-03-22T01:27:06.182 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:04.936+0000 7f53d67fc700 1 -- 172.21.15.72:0/4155797321 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f53c80045c0 con 0x7f53d8108bd0 2024-03-22T01:27:06.183 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:04.937+0000 7f53dd3cf700 1 -- 172.21.15.72:0/4155797321 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53d8108bd0 msgr2=0x7f53d810afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:06.183 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:04.937+0000 7f53dd3cf700 1 --2- 172.21.15.72:0/4155797321 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53d8108bd0 0x7f53d810afc0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f53c8009e80 tx=0x7f53c800b330 comp rx=0 tx=0).stop 2024-03-22T01:27:06.183 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:04.937+0000 7f53dd3cf700 1 -- 172.21.15.72:0/4155797321 shutdown_connections 2024-03-22T01:27:06.183 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:04.937+0000 7f53dd3cf700 1 --2- 172.21.15.72:0/4155797321 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53d8108bd0 0x7f53d810afc0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:06.183 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:04.937+0000 7f53dd3cf700 1 -- 172.21.15.72:0/4155797321 >> 172.21.15.72:0/4155797321 conn(0x7f53d81025c0 msgr2=0x7f53d8104a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:27:06.183 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:04.937+0000 7f53dd3cf700 1 -- 172.21.15.72:0/4155797321 shutdown_connections 2024-03-22T01:27:06.183 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:04.937+0000 7f53dd3cf700 1 -- 172.21.15.72:0/4155797321 wait complete. 2024-03-22T01:27:06.183 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:04.937+0000 7f53dd3cf700 1 Processor -- start 2024-03-22T01:27:06.183 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:04.938+0000 7f53dd3cf700 1 -- start start 2024-03-22T01:27:06.183 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:04.938+0000 7f53dd3cf700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53d8108bd0 0x7f53d819bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:27:06.183 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:04.938+0000 7f53dd3cf700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f53d810c520 con 0x7f53d8108bd0 2024-03-22T01:27:06.183 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:04.938+0000 7f53d6ffd700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53d8108bd0 0x7f53d819bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:27:06.183 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:04.938+0000 7f53d6ffd700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53d8108bd0 0x7f53d819bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:41388/0 (socket says 172.21.15.72:41388) 2024-03-22T01:27:06.183 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:04.938+0000 7f53d6ffd700 1 -- 172.21.15.72:0/3006052737 learned_addr learned my addr 172.21.15.72:0/3006052737 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:27:06.183 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:04.938+0000 7f53d6ffd700 1 -- 172.21.15.72:0/3006052737 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53d819c4c0 con 0x7f53d8108bd0 2024-03-22T01:27:06.184 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:04.938+0000 7f53d6ffd700 1 --2- 172.21.15.72:0/3006052737 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53d8108bd0 0x7f53d819bf80 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f53c80072a0 tx=0x7f53c800bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:27:06.184 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:04.938+0000 7f53d4ff9700 1 -- 172.21.15.72:0/3006052737 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f53c8020020 con 0x7f53d8108bd0 2024-03-22T01:27:06.184 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:04.938+0000 7f53d4ff9700 1 -- 172.21.15.72:0/3006052737 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f53c8003b50 con 0x7f53d8108bd0 2024-03-22T01:27:06.184 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:04.938+0000 7f53dd3cf700 1 -- 172.21.15.72:0/3006052737 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53d819c750 con 0x7f53d8108bd0 2024-03-22T01:27:06.184 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:04.938+0000 7f53d4ff9700 1 -- 172.21.15.72:0/3006052737 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f53c8017040 con 0x7f53d8108bd0 2024-03-22T01:27:06.184 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:04.938+0000 7f53dd3cf700 1 -- 172.21.15.72:0/3006052737 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53d819f440 con 0x7f53d8108bd0 2024-03-22T01:27:06.184 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:04.939+0000 7f53d4ff9700 1 -- 172.21.15.72:0/3006052737 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 8) ==== 46294+0+0 (secure 0 0 0) 0x7f53c8023020 con 0x7f53d8108bd0 2024-03-22T01:27:06.184 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:04.939+0000 7f53d4ff9700 1 --2- 172.21.15.72:0/3006052737 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7f53c40391f0 0x7f53c403b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:27:06.184 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:04.939+0000 7f53d4ff9700 1 -- 172.21.15.72:0/3006052737 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f53c800f050 con 0x7f53d8108bd0 2024-03-22T01:27:06.184 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:04.939+0000 7f53ce7fc700 1 --2- 172.21.15.72:0/3006052737 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7f53c40391f0 0x7f53c403b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:27:06.184 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:04.939+0000 7f53dd3cf700 1 -- 172.21.15.72:0/3006052737 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53d8063cb0 con 0x7f53d8108bd0 2024-03-22T01:27:06.184 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:04.940+0000 7f53ce7fc700 1 --2- 172.21.15.72:0/3006052737 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7f53c40391f0 0x7f53c403b6b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f53c0006fd0 tx=0x7f53c0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:27:06.184 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:04.944+0000 7f53d4ff9700 1 -- 172.21.15.72:0/3006052737 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f53c802a020 con 0x7f53d8108bd0 2024-03-22T01:27:06.184 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:05.141+0000 7f53dd3cf700 1 -- 172.21.15.72:0/3006052737 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f53d819f9a0 con 0x7f53d8108bd0 2024-03-22T01:27:06.185 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:05.146+0000 7f53d4ff9700 1 -- 172.21.15.72:0/3006052737 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v6)=0 v6) ==== 142+0+0 (secure 0 0 0) 0x7f53c8018240 con 0x7f53d8108bd0 2024-03-22T01:27:06.185 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:05.148+0000 7f53dd3cf700 1 -- 172.21.15.72:0/3006052737 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7f53c40391f0 msgr2=0x7f53c403b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:06.185 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:05.148+0000 7f53dd3cf700 1 --2- 172.21.15.72:0/3006052737 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7f53c40391f0 0x7f53c403b6b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f53c0006fd0 tx=0x7f53c0006e40 comp rx=0 tx=0).stop 2024-03-22T01:27:06.185 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:05.148+0000 7f53dd3cf700 1 -- 172.21.15.72:0/3006052737 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53d8108bd0 msgr2=0x7f53d819bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:06.185 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:05.148+0000 7f53dd3cf700 1 --2- 172.21.15.72:0/3006052737 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53d8108bd0 0x7f53d819bf80 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f53c80072a0 tx=0x7f53c800bd50 comp rx=0 tx=0).stop 2024-03-22T01:27:06.185 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:05.149+0000 7f53dd3cf700 1 -- 172.21.15.72:0/3006052737 shutdown_connections 2024-03-22T01:27:06.185 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:05.149+0000 7f53dd3cf700 1 --2- 172.21.15.72:0/3006052737 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7f53c40391f0 0x7f53c403b6b0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:06.185 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:05.149+0000 7f53dd3cf700 1 --2- 172.21.15.72:0/3006052737 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53d8108bd0 0x7f53d819bf80 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:06.185 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:05.149+0000 7f53dd3cf700 1 -- 172.21.15.72:0/3006052737 >> 172.21.15.72:0/3006052737 conn(0x7f53d81025c0 msgr2=0x7f53d8102fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:27:06.185 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:05.149+0000 7f53dd3cf700 1 -- 172.21.15.72:0/3006052737 shutdown_connections 2024-03-22T01:27:06.185 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:05.149+0000 7f53dd3cf700 1 -- 172.21.15.72:0/3006052737 wait complete. 2024-03-22T01:27:07.399 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:06.604+0000 7f81095ff700 1 Processor -- start 2024-03-22T01:27:07.399 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:06.604+0000 7f81095ff700 1 -- start start 2024-03-22T01:27:07.399 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:06.605+0000 7f81095ff700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8104106780 0x7f8104106ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:27:07.399 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:06.605+0000 7f81095ff700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f8104107170 con 0x7f8104106780 2024-03-22T01:27:07.399 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:06.605+0000 7f8102ffd700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8104106780 0x7f8104106ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:27:07.399 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:06.605+0000 7f8102ffd700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8104106780 0x7f8104106ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:41398/0 (socket says 172.21.15.72:41398) 2024-03-22T01:27:07.399 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:06.605+0000 7f8102ffd700 1 -- 172.21.15.72:0/2173553972 learned_addr learned my addr 172.21.15.72:0/2173553972 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:27:07.399 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:06.605+0000 7f8102ffd700 1 -- 172.21.15.72:0/2173553972 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8104107980 con 0x7f8104106780 2024-03-22T01:27:07.400 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:06.605+0000 7f8102ffd700 1 --2- 172.21.15.72:0/2173553972 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8104106780 0x7f8104106ba0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f80f400bb40 tx=0x7f80f400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b437472fe16ae9ec server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:27:07.400 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:06.605+0000 7f8101ffb700 1 -- 172.21.15.72:0/2173553972 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f80f400dad0 con 0x7f8104106780 2024-03-22T01:27:07.400 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:06.606+0000 7f8101ffb700 1 -- 172.21.15.72:0/2173553972 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80f4004510 con 0x7f8104106780 2024-03-22T01:27:07.400 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:06.606+0000 7f8101ffb700 1 -- 172.21.15.72:0/2173553972 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f80f4003e60 con 0x7f8104106780 2024-03-22T01:27:07.400 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:06.606+0000 7f81095ff700 1 -- 172.21.15.72:0/2173553972 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8104106780 msgr2=0x7f8104106ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:07.400 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:06.606+0000 7f81095ff700 1 --2- 172.21.15.72:0/2173553972 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8104106780 0x7f8104106ba0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f80f400bb40 tx=0x7f80f400be50 comp rx=0 tx=0).stop 2024-03-22T01:27:07.400 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:06.606+0000 7f81095ff700 1 -- 172.21.15.72:0/2173553972 shutdown_connections 2024-03-22T01:27:07.400 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:06.606+0000 7f81095ff700 1 --2- 172.21.15.72:0/2173553972 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8104106780 0x7f8104106ba0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:07.400 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:06.607+0000 7f81095ff700 1 -- 172.21.15.72:0/2173553972 >> 172.21.15.72:0/2173553972 conn(0x7f8104101de0 msgr2=0x7f8104104200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:27:07.400 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:06.607+0000 7f81095ff700 1 -- 172.21.15.72:0/2173553972 shutdown_connections 2024-03-22T01:27:07.400 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:06.607+0000 7f81095ff700 1 -- 172.21.15.72:0/2173553972 wait complete. 2024-03-22T01:27:07.400 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:06.607+0000 7f81095ff700 1 Processor -- start 2024-03-22T01:27:07.400 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:06.607+0000 7f81095ff700 1 -- start start 2024-03-22T01:27:07.401 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:06.607+0000 7f81095ff700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8104106780 0x7f810419e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:27:07.401 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:06.607+0000 7f81095ff700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f8104108150 con 0x7f8104106780 2024-03-22T01:27:07.401 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:06.608+0000 7f8102ffd700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8104106780 0x7f810419e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:27:07.401 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:06.608+0000 7f8102ffd700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8104106780 0x7f810419e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:41406/0 (socket says 172.21.15.72:41406) 2024-03-22T01:27:07.401 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:06.608+0000 7f8102ffd700 1 -- 172.21.15.72:0/68970687 learned_addr learned my addr 172.21.15.72:0/68970687 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:27:07.401 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:06.608+0000 7f8102ffd700 1 -- 172.21.15.72:0/68970687 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f810419e7b0 con 0x7f8104106780 2024-03-22T01:27:07.401 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:06.608+0000 7f8102ffd700 1 --2- 172.21.15.72:0/68970687 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8104106780 0x7f810419e270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f80f400d750 tx=0x7f80f4003fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:27:07.401 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:06.608+0000 7f80ebfff700 1 -- 172.21.15.72:0/68970687 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f80f4017580 con 0x7f8104106780 2024-03-22T01:27:07.401 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:06.608+0000 7f80ebfff700 1 -- 172.21.15.72:0/68970687 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80f4024070 con 0x7f8104106780 2024-03-22T01:27:07.401 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:06.608+0000 7f81095ff700 1 -- 172.21.15.72:0/68970687 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f810419ea40 con 0x7f8104106780 2024-03-22T01:27:07.401 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:06.608+0000 7f80ebfff700 1 -- 172.21.15.72:0/68970687 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f80f401f890 con 0x7f8104106780 2024-03-22T01:27:07.401 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:06.608+0000 7f81095ff700 1 -- 172.21.15.72:0/68970687 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81041a1730 con 0x7f8104106780 2024-03-22T01:27:07.401 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:06.609+0000 7f80ebfff700 1 -- 172.21.15.72:0/68970687 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 8) ==== 46294+0+0 (secure 0 0 0) 0x7f80f401fb90 con 0x7f8104106780 2024-03-22T01:27:07.401 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:06.609+0000 7f81095ff700 1 -- 172.21.15.72:0/68970687 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8104063cb0 con 0x7f8104106780 2024-03-22T01:27:07.402 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:06.609+0000 7f80ebfff700 1 --2- 172.21.15.72:0/68970687 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7f80ec039240 0x7f80ec03b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:27:07.402 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:06.609+0000 7f80ebfff700 1 -- 172.21.15.72:0/68970687 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f80f404d760 con 0x7f8104106780 2024-03-22T01:27:07.402 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:06.610+0000 7f81027fc700 1 --2- 172.21.15.72:0/68970687 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7f80ec039240 0x7f80ec03b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:27:07.402 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:06.610+0000 7f81027fc700 1 --2- 172.21.15.72:0/68970687 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7f80ec039240 0x7f80ec03b700 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f80f8006fd0 tx=0x7f80f8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:27:07.402 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:06.614+0000 7f80ebfff700 1 -- 172.21.15.72:0/68970687 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f80f4012070 con 0x7f8104106780 2024-03-22T01:27:07.402 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:06.816+0000 7f81095ff700 1 -- 172.21.15.72:0/68970687 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f810402e030 con 0x7f8104106780 2024-03-22T01:27:07.402 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:06.819+0000 7f80ebfff700 1 -- 172.21.15.72:0/68970687 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f80f4024db0 con 0x7f8104106780 2024-03-22T01:27:07.402 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:06.821+0000 7f81095ff700 1 -- 172.21.15.72:0/68970687 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7f80ec039240 msgr2=0x7f80ec03b700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:07.402 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:06.821+0000 7f81095ff700 1 --2- 172.21.15.72:0/68970687 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7f80ec039240 0x7f80ec03b700 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f80f8006fd0 tx=0x7f80f8006e40 comp rx=0 tx=0).stop 2024-03-22T01:27:07.402 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:06.821+0000 7f81095ff700 1 -- 172.21.15.72:0/68970687 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8104106780 msgr2=0x7f810419e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:07.402 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:06.821+0000 7f81095ff700 1 --2- 172.21.15.72:0/68970687 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8104106780 0x7f810419e270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f80f400d750 tx=0x7f80f4003fb0 comp rx=0 tx=0).stop 2024-03-22T01:27:07.402 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:06.821+0000 7f81095ff700 1 -- 172.21.15.72:0/68970687 shutdown_connections 2024-03-22T01:27:07.403 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:06.821+0000 7f81095ff700 1 --2- 172.21.15.72:0/68970687 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7f80ec039240 0x7f80ec03b700 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:07.403 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:06.821+0000 7f81095ff700 1 --2- 172.21.15.72:0/68970687 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8104106780 0x7f810419e270 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:07.403 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:06.821+0000 7f81095ff700 1 -- 172.21.15.72:0/68970687 >> 172.21.15.72:0/68970687 conn(0x7f8104101de0 msgr2=0x7f8104190a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:27:07.403 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:06.822+0000 7f81095ff700 1 -- 172.21.15.72:0/68970687 shutdown_connections 2024-03-22T01:27:07.403 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:06.822+0000 7f81095ff700 1 -- 172.21.15.72:0/68970687 wait complete. 2024-03-22T01:27:07.403 INFO:teuthology.orchestra.run.smithi072.stdout:Enabling the dashboard module... 2024-03-22T01:27:07.697 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:07 smithi072 ceph-mon[32414]: from='mgr.14120 172.21.15.72:0/4135194081' entity='mgr.x' 2024-03-22T01:27:07.697 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:07 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/68970687' entity='client.admin' 2024-03-22T01:27:08.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:08 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/4192901734' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-22T01:27:08.622 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:27:08 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: ignoring --setuser ceph since I am not root 2024-03-22T01:27:08.622 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:27:08 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: ignoring --setgroup ceph since I am not root 2024-03-22T01:27:09.122 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:27:09 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:27:09.054+0000 7f6218cab200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-22T01:27:09.137 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:07.798+0000 7f622e1a6700 1 Processor -- start 2024-03-22T01:27:09.137 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:07.798+0000 7f622e1a6700 1 -- start start 2024-03-22T01:27:09.137 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:07.799+0000 7f622e1a6700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6228108be0 0x7f622810afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:27:09.138 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:07.799+0000 7f622e1a6700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f622807be30 con 0x7f6228108be0 2024-03-22T01:27:09.138 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:07.799+0000 7f62277fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6228108be0 0x7f622810afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:27:09.138 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:07.799+0000 7f62277fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6228108be0 0x7f622810afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:41410/0 (socket says 172.21.15.72:41410) 2024-03-22T01:27:09.138 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:07.799+0000 7f62277fe700 1 -- 172.21.15.72:0/1265966257 learned_addr learned my addr 172.21.15.72:0/1265966257 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:27:09.138 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:07.799+0000 7f62277fe700 1 -- 172.21.15.72:0/1265966257 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f622810b510 con 0x7f6228108be0 2024-03-22T01:27:09.138 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:07.799+0000 7f62277fe700 1 --2- 172.21.15.72:0/1265966257 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6228108be0 0x7f622810afd0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f62180098a0 tx=0x7f6218009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55dd0ed3f07e23f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:27:09.138 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:07.799+0000 7f6226ffd700 1 -- 172.21.15.72:0/1265966257 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6218004160 con 0x7f6228108be0 2024-03-22T01:27:09.138 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:07.799+0000 7f6226ffd700 1 -- 172.21.15.72:0/1265966257 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6218004300 con 0x7f6228108be0 2024-03-22T01:27:09.138 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:07.799+0000 7f6226ffd700 1 -- 172.21.15.72:0/1265966257 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6218011b20 con 0x7f6228108be0 2024-03-22T01:27:09.138 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:07.800+0000 7f622e1a6700 1 -- 172.21.15.72:0/1265966257 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6228108be0 msgr2=0x7f622810afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:09.138 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:07.800+0000 7f622e1a6700 1 --2- 172.21.15.72:0/1265966257 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6228108be0 0x7f622810afd0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f62180098a0 tx=0x7f6218009bb0 comp rx=0 tx=0).stop 2024-03-22T01:27:09.138 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:07.800+0000 7f622e1a6700 1 -- 172.21.15.72:0/1265966257 shutdown_connections 2024-03-22T01:27:09.138 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:07.800+0000 7f622e1a6700 1 --2- 172.21.15.72:0/1265966257 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6228108be0 0x7f622810afd0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:09.139 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:07.800+0000 7f622e1a6700 1 -- 172.21.15.72:0/1265966257 >> 172.21.15.72:0/1265966257 conn(0x7f6228102630 msgr2=0x7f6228104a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:27:09.139 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:07.800+0000 7f622e1a6700 1 -- 172.21.15.72:0/1265966257 shutdown_connections 2024-03-22T01:27:09.139 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:07.800+0000 7f622e1a6700 1 -- 172.21.15.72:0/1265966257 wait complete. 2024-03-22T01:27:09.139 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:07.801+0000 7f622e1a6700 1 Processor -- start 2024-03-22T01:27:09.139 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:07.801+0000 7f622e1a6700 1 -- start start 2024-03-22T01:27:09.139 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:07.801+0000 7f622e1a6700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6228108be0 0x7f622819bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:27:09.139 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:07.801+0000 7f622e1a6700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f622810c520 con 0x7f6228108be0 2024-03-22T01:27:09.139 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:07.801+0000 7f62277fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6228108be0 0x7f622819bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:27:09.139 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:07.801+0000 7f62277fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6228108be0 0x7f622819bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:41416/0 (socket says 172.21.15.72:41416) 2024-03-22T01:27:09.139 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:07.801+0000 7f62277fe700 1 -- 172.21.15.72:0/4192901734 learned_addr learned my addr 172.21.15.72:0/4192901734 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:27:09.139 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:07.801+0000 7f62277fe700 1 -- 172.21.15.72:0/4192901734 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f622819c520 con 0x7f6228108be0 2024-03-22T01:27:09.139 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:07.801+0000 7f62277fe700 1 --2- 172.21.15.72:0/4192901734 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6228108be0 0x7f622819bfe0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6218011b60 tx=0x7f6218011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:27:09.139 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:07.801+0000 7f62257fa700 1 -- 172.21.15.72:0/4192901734 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6218003b60 con 0x7f6228108be0 2024-03-22T01:27:09.139 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:07.802+0000 7f622e1a6700 1 -- 172.21.15.72:0/4192901734 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f622819c7b0 con 0x7f6228108be0 2024-03-22T01:27:09.140 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:07.802+0000 7f62257fa700 1 -- 172.21.15.72:0/4192901734 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f621801a910 con 0x7f6228108be0 2024-03-22T01:27:09.140 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:07.802+0000 7f62257fa700 1 -- 172.21.15.72:0/4192901734 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6218024930 con 0x7f6228108be0 2024-03-22T01:27:09.140 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:07.802+0000 7f622e1a6700 1 -- 172.21.15.72:0/4192901734 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f622819cc10 con 0x7f6228108be0 2024-03-22T01:27:09.140 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:07.802+0000 7f62257fa700 1 -- 172.21.15.72:0/4192901734 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 8) ==== 46294+0+0 (secure 0 0 0) 0x7f6218022020 con 0x7f6228108be0 2024-03-22T01:27:09.140 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:07.802+0000 7f622e1a6700 1 -- 172.21.15.72:0/4192901734 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6228063cb0 con 0x7f6228108be0 2024-03-22T01:27:09.140 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:07.803+0000 7f62257fa700 1 --2- 172.21.15.72:0/4192901734 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7f6214039240 0x7f621403b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:27:09.140 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:07.803+0000 7f62257fa700 1 -- 172.21.15.72:0/4192901734 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f621804d5b0 con 0x7f6228108be0 2024-03-22T01:27:09.140 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:07.803+0000 7f621effd700 1 --2- 172.21.15.72:0/4192901734 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7f6214039240 0x7f621403b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:27:09.140 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:07.803+0000 7f621effd700 1 --2- 172.21.15.72:0/4192901734 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7f6214039240 0x7f621403b700 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f6210006fd0 tx=0x7f6210006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:27:09.140 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:07.807+0000 7f62257fa700 1 -- 172.21.15.72:0/4192901734 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6218019c70 con 0x7f6228108be0 2024-03-22T01:27:09.140 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:08.076+0000 7f622e1a6700 1 -- 172.21.15.72:0/4192901734 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f622819f6a0 con 0x7f6228108be0 2024-03-22T01:27:09.140 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:08.462+0000 7f62257fa700 1 -- 172.21.15.72:0/4192901734 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f621800bc00 con 0x7f6228108be0 2024-03-22T01:27:09.140 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:08.463+0000 7f62257fa700 1 -- 172.21.15.72:0/4192901734 <== mon.0 v2:172.21.15.72:3300/0 8 ==== mgrmap(e 9) ==== 46307+0+0 (secure 0 0 0) 0x7f621801ab30 con 0x7f6228108be0 2024-03-22T01:27:09.140 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:08.464+0000 7f622e1a6700 1 -- 172.21.15.72:0/4192901734 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7f6214039240 msgr2=0x7f621403b700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:09.141 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:08.464+0000 7f622e1a6700 1 --2- 172.21.15.72:0/4192901734 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7f6214039240 0x7f621403b700 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f6210006fd0 tx=0x7f6210006e40 comp rx=0 tx=0).stop 2024-03-22T01:27:09.141 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:08.465+0000 7f622e1a6700 1 -- 172.21.15.72:0/4192901734 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6228108be0 msgr2=0x7f622819bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:09.141 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:08.465+0000 7f622e1a6700 1 --2- 172.21.15.72:0/4192901734 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6228108be0 0x7f622819bfe0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6218011b60 tx=0x7f6218011c40 comp rx=0 tx=0).stop 2024-03-22T01:27:09.141 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:08.465+0000 7f622e1a6700 1 -- 172.21.15.72:0/4192901734 shutdown_connections 2024-03-22T01:27:09.141 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:08.465+0000 7f622e1a6700 1 --2- 172.21.15.72:0/4192901734 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7f6214039240 0x7f621403b700 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:09.141 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:08.465+0000 7f622e1a6700 1 --2- 172.21.15.72:0/4192901734 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6228108be0 0x7f622819bfe0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:09.141 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:08.465+0000 7f622e1a6700 1 -- 172.21.15.72:0/4192901734 >> 172.21.15.72:0/4192901734 conn(0x7f6228102630 msgr2=0x7f62281030d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:27:09.141 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:08.465+0000 7f622e1a6700 1 -- 172.21.15.72:0/4192901734 shutdown_connections 2024-03-22T01:27:09.141 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:08.465+0000 7f622e1a6700 1 -- 172.21.15.72:0/4192901734 wait complete. 2024-03-22T01:27:09.420 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:27:09 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:27:09.152+0000 7f6218cab200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-22T01:27:09.726 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:09 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/4192901734' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-22T01:27:09.727 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:09 smithi072 ceph-mon[32414]: mgrmap e9: x(active, since 18s) 2024-03-22T01:27:10.064 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:27:09 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:27:09.825+0000 7f6218cab200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-22T01:27:10.064 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:27:09 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:27:09.919+0000 7f6218cab200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-22T01:27:10.064 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:27:10 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:27:10.065+0000 7f6218cab200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-22T01:27:10.329 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:27:10 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:27:10.161+0000 7f6218cab200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-22T01:27:10.329 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:27:10 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:27:10.244+0000 7f6218cab200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-22T01:27:10.345 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout { 2024-03-22T01:27:10.345 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-22T01:27:10.345 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-22T01:27:10.345 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-22T01:27:10.345 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-22T01:27:10.346 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout } 2024-03-22T01:27:10.346 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:09.522+0000 7f60ba22a700 1 Processor -- start 2024-03-22T01:27:10.346 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:09.522+0000 7f60ba22a700 1 -- start start 2024-03-22T01:27:10.346 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:09.522+0000 7f60ba22a700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60b4106a00 0x7f60b4106e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:27:10.346 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:09.522+0000 7f60ba22a700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f60b4075d90 con 0x7f60b4106a00 2024-03-22T01:27:10.347 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:09.523+0000 7f60b37fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60b4106a00 0x7f60b4106e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:27:10.347 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:09.523+0000 7f60b37fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60b4106a00 0x7f60b4106e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:41438/0 (socket says 172.21.15.72:41438) 2024-03-22T01:27:10.347 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:09.523+0000 7f60b37fe700 1 -- 172.21.15.72:0/2877132901 learned_addr learned my addr 172.21.15.72:0/2877132901 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:27:10.347 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:09.523+0000 7f60b37fe700 1 -- 172.21.15.72:0/2877132901 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60b4107360 con 0x7f60b4106a00 2024-03-22T01:27:10.347 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:09.523+0000 7f60b37fe700 1 --2- 172.21.15.72:0/2877132901 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60b4106a00 0x7f60b4106e20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f60a80098a0 tx=0x7f60a8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=485e473e29b6a2c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:27:10.347 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:09.523+0000 7f60b27fc700 1 -- 172.21.15.72:0/2877132901 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f60a8004160 con 0x7f60b4106a00 2024-03-22T01:27:10.348 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:09.523+0000 7f60b27fc700 1 -- 172.21.15.72:0/2877132901 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60a8004300 con 0x7f60b4106a00 2024-03-22T01:27:10.348 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:09.523+0000 7f60b27fc700 1 -- 172.21.15.72:0/2877132901 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f60a8011b20 con 0x7f60b4106a00 2024-03-22T01:27:10.348 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:09.524+0000 7f60ba22a700 1 -- 172.21.15.72:0/2877132901 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60b4106a00 msgr2=0x7f60b4106e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:10.348 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:09.524+0000 7f60ba22a700 1 --2- 172.21.15.72:0/2877132901 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60b4106a00 0x7f60b4106e20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f60a80098a0 tx=0x7f60a8009bb0 comp rx=0 tx=0).stop 2024-03-22T01:27:10.348 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:09.524+0000 7f60ba22a700 1 -- 172.21.15.72:0/2877132901 shutdown_connections 2024-03-22T01:27:10.348 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:09.524+0000 7f60ba22a700 1 --2- 172.21.15.72:0/2877132901 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60b4106a00 0x7f60b4106e20 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:10.348 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:09.524+0000 7f60ba22a700 1 -- 172.21.15.72:0/2877132901 >> 172.21.15.72:0/2877132901 conn(0x7f60b41025c0 msgr2=0x7f60b4104a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:27:10.349 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:09.524+0000 7f60ba22a700 1 -- 172.21.15.72:0/2877132901 shutdown_connections 2024-03-22T01:27:10.349 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:09.524+0000 7f60ba22a700 1 -- 172.21.15.72:0/2877132901 wait complete. 2024-03-22T01:27:10.349 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:09.524+0000 7f60ba22a700 1 Processor -- start 2024-03-22T01:27:10.349 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:09.525+0000 7f60ba22a700 1 -- start start 2024-03-22T01:27:10.349 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:09.525+0000 7f60ba22a700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60b4106a00 0x7f60b419e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:27:10.349 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:09.525+0000 7f60ba22a700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f60b4108430 con 0x7f60b4106a00 2024-03-22T01:27:10.349 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:09.525+0000 7f60b37fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60b4106a00 0x7f60b419e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:27:10.349 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:09.525+0000 7f60b37fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60b4106a00 0x7f60b419e190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:41446/0 (socket says 172.21.15.72:41446) 2024-03-22T01:27:10.350 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:09.525+0000 7f60b37fe700 1 -- 172.21.15.72:0/840137854 learned_addr learned my addr 172.21.15.72:0/840137854 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:27:10.350 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:09.525+0000 7f60b37fe700 1 -- 172.21.15.72:0/840137854 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60b419e6d0 con 0x7f60b4106a00 2024-03-22T01:27:10.350 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:09.525+0000 7f60b37fe700 1 --2- 172.21.15.72:0/840137854 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60b4106a00 0x7f60b419e190 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f60a80117e0 tx=0x7f60a8011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:27:10.350 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:09.526+0000 7f60b0ff9700 1 -- 172.21.15.72:0/840137854 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f60a8003ba0 con 0x7f60b4106a00 2024-03-22T01:27:10.350 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:09.526+0000 7f60ba22a700 1 -- 172.21.15.72:0/840137854 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60b419e960 con 0x7f60b4106a00 2024-03-22T01:27:10.350 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:09.526+0000 7f60b0ff9700 1 -- 172.21.15.72:0/840137854 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60a801a910 con 0x7f60b4106a00 2024-03-22T01:27:10.350 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:09.526+0000 7f60b0ff9700 1 -- 172.21.15.72:0/840137854 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f60a80248f0 con 0x7f60b4106a00 2024-03-22T01:27:10.350 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:09.526+0000 7f60ba22a700 1 -- 172.21.15.72:0/840137854 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60b419edc0 con 0x7f60b4106a00 2024-03-22T01:27:10.351 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:09.527+0000 7f60ba22a700 1 -- 172.21.15.72:0/840137854 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60b4063cb0 con 0x7f60b4106a00 2024-03-22T01:27:10.351 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:09.527+0000 7f60b0ff9700 1 -- 172.21.15.72:0/840137854 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 9) ==== 46307+0+0 (secure 0 0 0) 0x7f60a8022020 con 0x7f60b4106a00 2024-03-22T01:27:10.351 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:09.527+0000 7f60b0ff9700 1 --2- 172.21.15.72:0/840137854 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7f609c039290 0x7f609c03b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:27:10.351 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:09.527+0000 7f60b0ff9700 1 -- 172.21.15.72:0/840137854 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f60a804da30 con 0x7f60b4106a00 2024-03-22T01:27:10.351 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:09.527+0000 7f60b2ffd700 1 -- 172.21.15.72:0/840137854 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7f609c039290 msgr2=0x7f609c03b750 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.72:6800/1513232913 2024-03-22T01:27:10.351 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:09.527+0000 7f60b2ffd700 1 --2- 172.21.15.72:0/840137854 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7f609c039290 0x7f609c03b750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-22T01:27:10.351 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:09.532+0000 7f60b0ff9700 1 -- 172.21.15.72:0/840137854 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f60a800b3e0 con 0x7f60b4106a00 2024-03-22T01:27:10.352 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:09.728+0000 7f60b2ffd700 1 -- 172.21.15.72:0/840137854 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7f609c039290 msgr2=0x7f609c03b750 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.72:6800/1513232913 2024-03-22T01:27:10.352 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:09.728+0000 7f60b2ffd700 1 --2- 172.21.15.72:0/840137854 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7f609c039290 0x7f609c03b750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-22T01:27:10.352 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:09.797+0000 7f60ba22a700 1 -- 172.21.15.72:0/840137854 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f60b4063290 con 0x7f60b4106a00 2024-03-22T01:27:10.352 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:09.797+0000 7f60b0ff9700 1 -- 172.21.15.72:0/840137854 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f60a80172b0 con 0x7f60b4106a00 2024-03-22T01:27:10.352 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:09.799+0000 7f60ba22a700 1 -- 172.21.15.72:0/840137854 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7f609c039290 msgr2=0x7f609c03b750 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-22T01:27:10.352 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:09.799+0000 7f60ba22a700 1 --2- 172.21.15.72:0/840137854 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7f609c039290 0x7f609c03b750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:10.353 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:09.799+0000 7f60ba22a700 1 -- 172.21.15.72:0/840137854 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60b4106a00 msgr2=0x7f60b419e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:10.353 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:09.799+0000 7f60ba22a700 1 --2- 172.21.15.72:0/840137854 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60b4106a00 0x7f60b419e190 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f60a80117e0 tx=0x7f60a8011ba0 comp rx=0 tx=0).stop 2024-03-22T01:27:10.353 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:09.799+0000 7f60ba22a700 1 -- 172.21.15.72:0/840137854 shutdown_connections 2024-03-22T01:27:10.353 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:09.799+0000 7f60ba22a700 1 --2- 172.21.15.72:0/840137854 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7f609c039290 0x7f609c03b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:10.353 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:09.799+0000 7f60ba22a700 1 --2- 172.21.15.72:0/840137854 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60b4106a00 0x7f60b419e190 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:10.353 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:09.799+0000 7f60ba22a700 1 -- 172.21.15.72:0/840137854 >> 172.21.15.72:0/840137854 conn(0x7f60b41025c0 msgr2=0x7f60b4194e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:27:10.353 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:09.800+0000 7f60ba22a700 1 -- 172.21.15.72:0/840137854 shutdown_connections 2024-03-22T01:27:10.353 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:09.800+0000 7f60ba22a700 1 -- 172.21.15.72:0/840137854 wait complete. 2024-03-22T01:27:10.353 INFO:teuthology.orchestra.run.smithi072.stdout:Waiting for the mgr to restart... 2024-03-22T01:27:10.353 INFO:teuthology.orchestra.run.smithi072.stdout:Waiting for mgr epoch 9... 2024-03-22T01:27:10.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:10 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/840137854' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-22T01:27:10.622 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:27:10 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:27:10.515+0000 7f6218cab200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-22T01:27:10.957 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:27:10 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:27:10.627+0000 7f6218cab200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-22T01:27:10.957 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:27:10 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:27:10.716+0000 7f6218cab200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-22T01:27:10.957 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:27:10 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:27:10.815+0000 7f6218cab200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-22T01:27:11.358 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:27:10 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:27:10.991+0000 7f6218cab200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-22T01:27:12.372 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:27:11 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:27:11.938+0000 7f6218cab200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-22T01:27:12.372 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:27:12 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:27:12.068+0000 7f6218cab200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-22T01:27:12.684 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:27:12 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:27:12.399+0000 7f6218cab200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-22T01:27:12.684 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:27:12 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:27:12.490+0000 7f6218cab200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-22T01:27:12.684 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:27:12 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:27:12.685+0000 7f6218cab200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-22T01:27:13.122 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:27:12 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:27:12.870+0000 7f6218cab200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-22T01:27:13.408 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:27:13 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:27:13.154+0000 7f6218cab200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-22T01:27:13.409 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:27:13 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:27:13.409+0000 7f6218cab200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-22T01:27:13.760 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:27:13 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:27:13.647+0000 7f6218cab200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-22T01:27:14.871 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:27:14 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:27:14.447+0000 7f6218cab200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-22T01:27:15.326 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:27:15 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:27:15.009+0000 7f6218cab200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-22T01:27:15.326 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:27:15 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:27:15.108+0000 7f6218cab200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-22T01:27:15.326 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:27:15 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:27:15.326+0000 7f6218cab200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-22T01:27:15.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:15 smithi072 ceph-mon[32414]: Active manager daemon x restarted 2024-03-22T01:27:15.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:15 smithi072 ceph-mon[32414]: Activating manager daemon x 2024-03-22T01:27:15.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:15 smithi072 ceph-mon[32414]: osdmap e3: 0 total, 0 up, 0 in 2024-03-22T01:27:15.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:15 smithi072 ceph-mon[32414]: mgrmap e10: x(active, starting, since 0.00323481s) 2024-03-22T01:27:15.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:15 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T01:27:15.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:15 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-22T01:27:15.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:15 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-22T01:27:15.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:15 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-22T01:27:15.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:15 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-22T01:27:15.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:15 smithi072 ceph-mon[32414]: Manager daemon x is now available 2024-03-22T01:27:15.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:15 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:27:16.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:16 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-22T01:27:16.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:16 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-22T01:27:16.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:16 smithi072 ceph-mon[32414]: mgrmap e11: x(active, since 1.00577s) 2024-03-22T01:27:17.161 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout { 2024-03-22T01:27:17.161 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-22T01:27:17.161 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-22T01:27:17.161 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout } 2024-03-22T01:27:17.161 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:10.753+0000 7fe46a99d700 1 Processor -- start 2024-03-22T01:27:17.161 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:10.753+0000 7fe46a99d700 1 -- start start 2024-03-22T01:27:17.161 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:10.753+0000 7fe46a99d700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe4641096e0 0x7fe464109b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:27:17.162 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:10.753+0000 7fe46a99d700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fe46410a0d0 con 0x7fe4641096e0 2024-03-22T01:27:17.162 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:10.753+0000 7fe463fff700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe4641096e0 0x7fe464109b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:27:17.162 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:10.753+0000 7fe463fff700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe4641096e0 0x7fe464109b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58378/0 (socket says 172.21.15.72:58378) 2024-03-22T01:27:17.162 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:10.753+0000 7fe463fff700 1 -- 172.21.15.72:0/351880200 learned_addr learned my addr 172.21.15.72:0/351880200 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:27:17.162 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:10.754+0000 7fe463fff700 1 -- 172.21.15.72:0/351880200 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe46410a250 con 0x7fe4641096e0 2024-03-22T01:27:17.162 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:10.754+0000 7fe463fff700 1 --2- 172.21.15.72:0/351880200 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe4641096e0 0x7fe464109b00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe4540098a0 tx=0x7fe454009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3c5d22d11be7d30 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:27:17.162 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:10.754+0000 7fe462ffd700 1 -- 172.21.15.72:0/351880200 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe454004160 con 0x7fe4641096e0 2024-03-22T01:27:17.162 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:10.754+0000 7fe462ffd700 1 -- 172.21.15.72:0/351880200 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe454004300 con 0x7fe4641096e0 2024-03-22T01:27:17.162 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:10.754+0000 7fe462ffd700 1 -- 172.21.15.72:0/351880200 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe454011b20 con 0x7fe4641096e0 2024-03-22T01:27:17.162 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:10.754+0000 7fe46a99d700 1 -- 172.21.15.72:0/351880200 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe4641096e0 msgr2=0x7fe464109b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:17.162 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:10.754+0000 7fe46a99d700 1 --2- 172.21.15.72:0/351880200 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe4641096e0 0x7fe464109b00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe4540098a0 tx=0x7fe454009bb0 comp rx=0 tx=0).stop 2024-03-22T01:27:17.162 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:10.755+0000 7fe46a99d700 1 -- 172.21.15.72:0/351880200 shutdown_connections 2024-03-22T01:27:17.162 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:10.755+0000 7fe46a99d700 1 --2- 172.21.15.72:0/351880200 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe4641096e0 0x7fe464109b00 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:17.163 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:10.755+0000 7fe46a99d700 1 -- 172.21.15.72:0/351880200 >> 172.21.15.72:0/351880200 conn(0x7fe464101e20 msgr2=0x7fe464104240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:27:17.163 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:10.755+0000 7fe46a99d700 1 -- 172.21.15.72:0/351880200 shutdown_connections 2024-03-22T01:27:17.163 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:10.755+0000 7fe46a99d700 1 -- 172.21.15.72:0/351880200 wait complete. 2024-03-22T01:27:17.163 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:10.755+0000 7fe46a99d700 1 Processor -- start 2024-03-22T01:27:17.163 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:10.755+0000 7fe46a99d700 1 -- start start 2024-03-22T01:27:17.163 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:10.755+0000 7fe46a99d700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe4641096e0 0x7fe46419e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:27:17.163 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:10.755+0000 7fe46a99d700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fe464106ff0 con 0x7fe4641096e0 2024-03-22T01:27:17.163 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:10.756+0000 7fe463fff700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe4641096e0 0x7fe46419e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:27:17.163 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:10.756+0000 7fe463fff700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe4641096e0 0x7fe46419e0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58382/0 (socket says 172.21.15.72:58382) 2024-03-22T01:27:17.163 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:10.756+0000 7fe463fff700 1 -- 172.21.15.72:0/2350645986 learned_addr learned my addr 172.21.15.72:0/2350645986 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:27:17.163 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:10.756+0000 7fe463fff700 1 -- 172.21.15.72:0/2350645986 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe46419e630 con 0x7fe4641096e0 2024-03-22T01:27:17.163 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:10.756+0000 7fe463fff700 1 --2- 172.21.15.72:0/2350645986 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe4641096e0 0x7fe46419e0f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe4540117e0 tx=0x7fe454011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:27:17.163 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:10.756+0000 7fe4617fa700 1 -- 172.21.15.72:0/2350645986 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe454003bf0 con 0x7fe4641096e0 2024-03-22T01:27:17.163 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:10.756+0000 7fe4617fa700 1 -- 172.21.15.72:0/2350645986 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe45401a910 con 0x7fe4641096e0 2024-03-22T01:27:17.164 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:10.756+0000 7fe4617fa700 1 -- 172.21.15.72:0/2350645986 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe4540249a0 con 0x7fe4641096e0 2024-03-22T01:27:17.164 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:10.756+0000 7fe46a99d700 1 -- 172.21.15.72:0/2350645986 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe46419e920 con 0x7fe4641096e0 2024-03-22T01:27:17.164 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:10.756+0000 7fe46a99d700 1 -- 172.21.15.72:0/2350645986 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4641a15b0 con 0x7fe4641096e0 2024-03-22T01:27:17.164 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:10.757+0000 7fe4617fa700 1 -- 172.21.15.72:0/2350645986 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 9) ==== 46307+0+0 (secure 0 0 0) 0x7fe454022020 con 0x7fe4641096e0 2024-03-22T01:27:17.164 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:10.757+0000 7fe4617fa700 1 --2- 172.21.15.72:0/2350645986 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7fe44c039240 0x7fe44c03b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:27:17.164 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:10.757+0000 7fe46a99d700 1 -- 172.21.15.72:0/2350645986 --> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fe464063cb0 con 0x7fe44c039240 2024-03-22T01:27:17.164 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:10.757+0000 7fe4617fa700 1 -- 172.21.15.72:0/2350645986 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fe45404d4f0 con 0x7fe4641096e0 2024-03-22T01:27:17.164 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:10.757+0000 7fe4637fe700 1 -- 172.21.15.72:0/2350645986 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7fe44c039240 msgr2=0x7fe44c03b700 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.72:6800/1513232913 2024-03-22T01:27:17.164 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:10.757+0000 7fe4637fe700 1 --2- 172.21.15.72:0/2350645986 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7fe44c039240 0x7fe44c03b700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-22T01:27:17.164 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:10.958+0000 7fe4637fe700 1 -- 172.21.15.72:0/2350645986 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7fe44c039240 msgr2=0x7fe44c03b700 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.72:6800/1513232913 2024-03-22T01:27:17.164 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:10.958+0000 7fe4637fe700 1 --2- 172.21.15.72:0/2350645986 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7fe44c039240 0x7fe44c03b700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-22T01:27:17.164 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:11.359+0000 7fe4637fe700 1 -- 172.21.15.72:0/2350645986 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7fe44c039240 msgr2=0x7fe44c03b700 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.72:6800/1513232913 2024-03-22T01:27:17.164 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:11.359+0000 7fe4637fe700 1 --2- 172.21.15.72:0/2350645986 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7fe44c039240 0x7fe44c03b700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-22T01:27:17.164 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:12.160+0000 7fe4637fe700 1 -- 172.21.15.72:0/2350645986 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7fe44c039240 msgr2=0x7fe44c03b700 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.72:6800/1513232913 2024-03-22T01:27:17.165 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:12.160+0000 7fe4637fe700 1 --2- 172.21.15.72:0/2350645986 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7fe44c039240 0x7fe44c03b700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-22T01:27:17.165 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:13.761+0000 7fe4637fe700 1 -- 172.21.15.72:0/2350645986 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7fe44c039240 msgr2=0x7fe44c03b700 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.72:6800/1513232913 2024-03-22T01:27:17.165 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:13.761+0000 7fe4637fe700 1 --2- 172.21.15.72:0/2350645986 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7fe44c039240 0x7fe44c03b700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-22T01:27:17.165 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:15.341+0000 7fe4617fa700 1 -- 172.21.15.72:0/2350645986 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mgrmap(e 10) ==== 46074+0+0 (secure 0 0 0) 0x7fe45402d910 con 0x7fe4641096e0 2024-03-22T01:27:17.165 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:15.341+0000 7fe4617fa700 1 -- 172.21.15.72:0/2350645986 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7fe44c039240 msgr2=0x7fe44c03b700 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-22T01:27:17.165 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:15.341+0000 7fe4617fa700 1 --2- 172.21.15.72:0/2350645986 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7fe44c039240 0x7fe44c03b700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:17.165 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:16.342+0000 7fe4617fa700 1 -- 172.21.15.72:0/2350645986 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mgrmap(e 11) ==== 46201+0+0 (secure 0 0 0) 0x7fe45402d3f0 con 0x7fe4641096e0 2024-03-22T01:27:17.165 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:16.342+0000 7fe4617fa700 1 --2- 172.21.15.72:0/2350645986 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe44c03cad0 0x7fe44c03eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:27:17.165 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:16.342+0000 7fe4617fa700 1 -- 172.21.15.72:0/2350645986 --> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fe45402ae70 con 0x7fe44c03cad0 2024-03-22T01:27:17.165 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:16.343+0000 7fe4637fe700 1 --2- 172.21.15.72:0/2350645986 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe44c03cad0 0x7fe44c03eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:27:17.165 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:16.343+0000 7fe4637fe700 1 --2- 172.21.15.72:0/2350645986 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe44c03cad0 0x7fe44c03eec0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe458003a10 tx=0x7fe4580092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:27:17.166 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:16.344+0000 7fe4617fa700 1 -- 172.21.15.72:0/2350645986 <== mgr.14152 v2:172.21.15.72:6800/401403200 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7fe45402ae70 con 0x7fe44c03cad0 2024-03-22T01:27:17.166 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:16.356+0000 7fe46a99d700 1 -- 172.21.15.72:0/2350645986 --> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fe464102f30 con 0x7fe44c03cad0 2024-03-22T01:27:17.166 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:16.356+0000 7fe4617fa700 1 -- 172.21.15.72:0/2350645986 <== mgr.14152 v2:172.21.15.72:6800/401403200 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7fe464102f30 con 0x7fe44c03cad0 2024-03-22T01:27:17.166 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:16.357+0000 7fe46a99d700 1 -- 172.21.15.72:0/2350645986 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe44c03cad0 msgr2=0x7fe44c03eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:17.166 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:16.357+0000 7fe46a99d700 1 --2- 172.21.15.72:0/2350645986 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe44c03cad0 0x7fe44c03eec0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe458003a10 tx=0x7fe4580092b0 comp rx=0 tx=0).stop 2024-03-22T01:27:17.166 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:16.357+0000 7fe46a99d700 1 -- 172.21.15.72:0/2350645986 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe4641096e0 msgr2=0x7fe46419e0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:17.166 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:16.357+0000 7fe46a99d700 1 --2- 172.21.15.72:0/2350645986 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe4641096e0 0x7fe46419e0f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe4540117e0 tx=0x7fe454011c10 comp rx=0 tx=0).stop 2024-03-22T01:27:17.166 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:16.357+0000 7fe46a99d700 1 -- 172.21.15.72:0/2350645986 shutdown_connections 2024-03-22T01:27:17.166 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:16.357+0000 7fe46a99d700 1 --2- 172.21.15.72:0/2350645986 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe44c03cad0 0x7fe44c03eec0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:17.166 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:16.357+0000 7fe46a99d700 1 --2- 172.21.15.72:0/2350645986 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe4641096e0 0x7fe46419e0f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:17.166 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:16.357+0000 7fe46a99d700 1 --2- 172.21.15.72:0/2350645986 >> [v2:172.21.15.72:6800/1513232913,v1:172.21.15.72:6801/1513232913] conn(0x7fe44c039240 0x7fe44c03b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:17.166 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:16.357+0000 7fe46a99d700 1 -- 172.21.15.72:0/2350645986 >> 172.21.15.72:0/2350645986 conn(0x7fe464101e20 msgr2=0x7fe464102820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:27:17.166 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:16.357+0000 7fe46a99d700 1 -- 172.21.15.72:0/2350645986 shutdown_connections 2024-03-22T01:27:17.166 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:16.357+0000 7fe46a99d700 1 -- 172.21.15.72:0/2350645986 wait complete. 2024-03-22T01:27:17.167 INFO:teuthology.orchestra.run.smithi072.stdout:mgr epoch 9 is available 2024-03-22T01:27:17.167 INFO:teuthology.orchestra.run.smithi072.stdout:Generating a dashboard self-signed certificate... 2024-03-22T01:27:17.583 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:17 smithi072 ceph-mon[32414]: [22/Mar/2024:01:27:16] ENGINE Bus STARTING 2024-03-22T01:27:17.584 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:17 smithi072 ceph-mon[32414]: [22/Mar/2024:01:27:16] ENGINE Serving on http://172.21.15.72:8765 2024-03-22T01:27:17.584 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:17 smithi072 ceph-mon[32414]: [22/Mar/2024:01:27:16] ENGINE Serving on https://172.21.15.72:7150 2024-03-22T01:27:17.584 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:17 smithi072 ceph-mon[32414]: [22/Mar/2024:01:27:16] ENGINE Bus STARTED 2024-03-22T01:27:17.584 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:17 smithi072 ceph-mon[32414]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-22T01:27:17.584 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:17 smithi072 ceph-mon[32414]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-22T01:27:17.584 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:17 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:27:18.443 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-22T01:27:18.444 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:17.619+0000 7f20e4a37700 1 Processor -- start 2024-03-22T01:27:18.444 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:17.619+0000 7f20e4a37700 1 -- start start 2024-03-22T01:27:18.444 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:17.620+0000 7f20e4a37700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20e0106ff0 0x7f20e0107410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:27:18.444 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:17.620+0000 7f20e4a37700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f20e01079e0 con 0x7f20e0106ff0 2024-03-22T01:27:18.444 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:17.620+0000 7f20de59c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20e0106ff0 0x7f20e0107410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:27:18.444 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:17.620+0000 7f20de59c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20e0106ff0 0x7f20e0107410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58480/0 (socket says 172.21.15.72:58480) 2024-03-22T01:27:18.444 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:17.620+0000 7f20de59c700 1 -- 172.21.15.72:0/2644223784 learned_addr learned my addr 172.21.15.72:0/2644223784 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:27:18.444 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:17.620+0000 7f20de59c700 1 -- 172.21.15.72:0/2644223784 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20e01081f0 con 0x7f20e0106ff0 2024-03-22T01:27:18.444 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:17.620+0000 7f20de59c700 1 --2- 172.21.15.72:0/2644223784 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20e0106ff0 0x7f20e0107410 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f20d00098a0 tx=0x7f20d0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=652ae11daf98db0b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:27:18.444 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:17.621+0000 7f20dd59a700 1 -- 172.21.15.72:0/2644223784 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f20d0004160 con 0x7f20e0106ff0 2024-03-22T01:27:18.444 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:17.621+0000 7f20dd59a700 1 -- 172.21.15.72:0/2644223784 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20d0004300 con 0x7f20e0106ff0 2024-03-22T01:27:18.445 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:17.621+0000 7f20dd59a700 1 -- 172.21.15.72:0/2644223784 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f20d0011b20 con 0x7f20e0106ff0 2024-03-22T01:27:18.445 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:17.621+0000 7f20e4a37700 1 -- 172.21.15.72:0/2644223784 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20e0106ff0 msgr2=0x7f20e0107410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:18.445 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:17.621+0000 7f20e4a37700 1 --2- 172.21.15.72:0/2644223784 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20e0106ff0 0x7f20e0107410 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f20d00098a0 tx=0x7f20d0009bb0 comp rx=0 tx=0).stop 2024-03-22T01:27:18.445 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:17.621+0000 7f20e4a37700 1 -- 172.21.15.72:0/2644223784 shutdown_connections 2024-03-22T01:27:18.445 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:17.621+0000 7f20e4a37700 1 --2- 172.21.15.72:0/2644223784 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20e0106ff0 0x7f20e0107410 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:18.445 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:17.621+0000 7f20e4a37700 1 -- 172.21.15.72:0/2644223784 >> 172.21.15.72:0/2644223784 conn(0x7f20e01025b0 msgr2=0x7f20e01049d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:27:18.445 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:17.622+0000 7f20e4a37700 1 -- 172.21.15.72:0/2644223784 shutdown_connections 2024-03-22T01:27:18.445 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:17.622+0000 7f20e4a37700 1 -- 172.21.15.72:0/2644223784 wait complete. 2024-03-22T01:27:18.445 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:17.622+0000 7f20e4a37700 1 Processor -- start 2024-03-22T01:27:18.445 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:17.622+0000 7f20e4a37700 1 -- start start 2024-03-22T01:27:18.445 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:17.623+0000 7f20e4a37700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20e0106ff0 0x7f20e019c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:27:18.445 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:17.623+0000 7f20e4a37700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f20e010b5a0 con 0x7f20e0106ff0 2024-03-22T01:27:18.445 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:17.623+0000 7f20de59c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20e0106ff0 0x7f20e019c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:27:18.445 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:17.623+0000 7f20de59c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20e0106ff0 0x7f20e019c030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58490/0 (socket says 172.21.15.72:58490) 2024-03-22T01:27:18.446 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:17.623+0000 7f20de59c700 1 -- 172.21.15.72:0/3670128421 learned_addr learned my addr 172.21.15.72:0/3670128421 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:27:18.446 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:17.623+0000 7f20de59c700 1 -- 172.21.15.72:0/3670128421 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20e019c570 con 0x7f20e0106ff0 2024-03-22T01:27:18.446 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:17.623+0000 7f20de59c700 1 --2- 172.21.15.72:0/3670128421 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20e0106ff0 0x7f20e019c030 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f20d00116a0 tx=0x7f20d0011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:27:18.446 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:17.623+0000 7f20cf7fe700 1 -- 172.21.15.72:0/3670128421 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f20d0003a40 con 0x7f20e0106ff0 2024-03-22T01:27:18.446 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:17.623+0000 7f20cf7fe700 1 -- 172.21.15.72:0/3670128421 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20d0003be0 con 0x7f20e0106ff0 2024-03-22T01:27:18.446 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:17.623+0000 7f20e4a37700 1 -- 172.21.15.72:0/3670128421 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20e019c800 con 0x7f20e0106ff0 2024-03-22T01:27:18.446 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:17.623+0000 7f20cf7fe700 1 -- 172.21.15.72:0/3670128421 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f20d001a720 con 0x7f20e0106ff0 2024-03-22T01:27:18.446 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:17.623+0000 7f20e4a37700 1 -- 172.21.15.72:0/3670128421 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20e019cc60 con 0x7f20e0106ff0 2024-03-22T01:27:18.446 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:17.624+0000 7f20cf7fe700 1 -- 172.21.15.72:0/3670128421 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 11) ==== 46201+0+0 (secure 0 0 0) 0x7f20d001aa80 con 0x7f20e0106ff0 2024-03-22T01:27:18.446 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:17.624+0000 7f20cf7fe700 1 --2- 172.21.15.72:0/3670128421 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f20c8039130 0x7f20c803b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:27:18.446 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:17.624+0000 7f20cf7fe700 1 -- 172.21.15.72:0/3670128421 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f20d004d700 con 0x7f20e0106ff0 2024-03-22T01:27:18.446 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:17.624+0000 7f20ddd9b700 1 --2- 172.21.15.72:0/3670128421 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f20c8039130 0x7f20c803b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:27:18.447 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:17.624+0000 7f20e4a37700 1 -- 172.21.15.72:0/3670128421 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20e0063cb0 con 0x7f20e0106ff0 2024-03-22T01:27:18.447 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:17.625+0000 7f20ddd9b700 1 --2- 172.21.15.72:0/3670128421 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f20c8039130 0x7f20c803b5f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f20d4006fd0 tx=0x7f20d4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:27:18.447 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:17.630+0000 7f20cf7fe700 1 -- 172.21.15.72:0/3670128421 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f20d0011e20 con 0x7f20e0106ff0 2024-03-22T01:27:18.447 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:17.840+0000 7f20e4a37700 1 -- 172.21.15.72:0/3670128421 --> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f20e0062ac0 con 0x7f20c8039130 2024-03-22T01:27:18.447 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:17.858+0000 7f20cf7fe700 1 -- 172.21.15.72:0/3670128421 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mgrmap(e 12) ==== 46307+0+0 (secure 0 0 0) 0x7f20d002d910 con 0x7f20e0106ff0 2024-03-22T01:27:18.447 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:17.886+0000 7f20cf7fe700 1 -- 172.21.15.72:0/3670128421 <== mgr.14152 v2:172.21.15.72:6800/401403200 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f20e0062ac0 con 0x7f20c8039130 2024-03-22T01:27:18.447 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:17.888+0000 7f20e4a37700 1 -- 172.21.15.72:0/3670128421 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f20c8039130 msgr2=0x7f20c803b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:18.447 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:17.888+0000 7f20e4a37700 1 --2- 172.21.15.72:0/3670128421 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f20c8039130 0x7f20c803b5f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f20d4006fd0 tx=0x7f20d4006e40 comp rx=0 tx=0).stop 2024-03-22T01:27:18.447 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:17.888+0000 7f20e4a37700 1 -- 172.21.15.72:0/3670128421 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20e0106ff0 msgr2=0x7f20e019c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:18.447 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:17.888+0000 7f20e4a37700 1 --2- 172.21.15.72:0/3670128421 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20e0106ff0 0x7f20e019c030 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f20d00116a0 tx=0x7f20d0011ba0 comp rx=0 tx=0).stop 2024-03-22T01:27:18.447 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:17.888+0000 7f20e4a37700 1 -- 172.21.15.72:0/3670128421 shutdown_connections 2024-03-22T01:27:18.447 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:17.888+0000 7f20e4a37700 1 --2- 172.21.15.72:0/3670128421 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f20c8039130 0x7f20c803b5f0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:18.447 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:17.888+0000 7f20e4a37700 1 --2- 172.21.15.72:0/3670128421 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20e0106ff0 0x7f20e019c030 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:18.447 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:17.888+0000 7f20e4a37700 1 -- 172.21.15.72:0/3670128421 >> 172.21.15.72:0/3670128421 conn(0x7f20e01025b0 msgr2=0x7f20e01033a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:27:18.448 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:17.888+0000 7f20e4a37700 1 -- 172.21.15.72:0/3670128421 shutdown_connections 2024-03-22T01:27:18.448 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:17.888+0000 7f20e4a37700 1 -- 172.21.15.72:0/3670128421 wait complete. 2024-03-22T01:27:18.448 INFO:teuthology.orchestra.run.smithi072.stdout:Creating initial admin user... 2024-03-22T01:27:19.074 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:18 smithi072 ceph-mon[32414]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:27:19.074 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:18 smithi072 ceph-mon[32414]: mgrmap e12: x(active, since 2s) 2024-03-22T01:27:19.074 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:18 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:27:19.074 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:18 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:27:19.074 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:18 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:27:19.966 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$oxznY3P53NdYNWXxgwOMQuIBGxcwsMFmvMkXA6IpC1J/eYjXa3vgq", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711070839, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-22T01:27:19.966 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:18.839+0000 7f678d383700 1 Processor -- start 2024-03-22T01:27:19.966 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:18.839+0000 7f678d383700 1 -- start start 2024-03-22T01:27:19.966 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:18.839+0000 7f678d383700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6788107010 0x7f6788107430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:27:19.967 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:18.839+0000 7f678d383700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f6788107a00 con 0x7f6788107010 2024-03-22T01:27:19.967 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:18.839+0000 7f6786ffd700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6788107010 0x7f6788107430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:27:19.967 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:18.839+0000 7f6786ffd700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6788107010 0x7f6788107430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58506/0 (socket says 172.21.15.72:58506) 2024-03-22T01:27:19.967 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:18.839+0000 7f6786ffd700 1 -- 172.21.15.72:0/3850960337 learned_addr learned my addr 172.21.15.72:0/3850960337 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:27:19.967 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:18.839+0000 7f6786ffd700 1 -- 172.21.15.72:0/3850960337 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6788108210 con 0x7f6788107010 2024-03-22T01:27:19.967 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:18.840+0000 7f6786ffd700 1 --2- 172.21.15.72:0/3850960337 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6788107010 0x7f6788107430 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f677800bb40 tx=0x7f677800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c63664e92176534 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:27:19.967 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:18.840+0000 7f6785ffb700 1 -- 172.21.15.72:0/3850960337 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f677800dbe0 con 0x7f6788107010 2024-03-22T01:27:19.967 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:18.840+0000 7f6785ffb700 1 -- 172.21.15.72:0/3850960337 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6778004510 con 0x7f6788107010 2024-03-22T01:27:19.967 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:18.840+0000 7f6785ffb700 1 -- 172.21.15.72:0/3850960337 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6778010470 con 0x7f6788107010 2024-03-22T01:27:19.967 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:18.840+0000 7f678d383700 1 -- 172.21.15.72:0/3850960337 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6788107010 msgr2=0x7f6788107430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:19.967 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:18.840+0000 7f678d383700 1 --2- 172.21.15.72:0/3850960337 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6788107010 0x7f6788107430 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f677800bb40 tx=0x7f677800be50 comp rx=0 tx=0).stop 2024-03-22T01:27:19.967 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:18.840+0000 7f678d383700 1 -- 172.21.15.72:0/3850960337 shutdown_connections 2024-03-22T01:27:19.967 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:18.840+0000 7f678d383700 1 --2- 172.21.15.72:0/3850960337 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6788107010 0x7f6788107430 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:19.968 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:18.840+0000 7f678d383700 1 -- 172.21.15.72:0/3850960337 >> 172.21.15.72:0/3850960337 conn(0x7f67881025d0 msgr2=0x7f67881049f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:27:19.968 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:18.840+0000 7f678d383700 1 -- 172.21.15.72:0/3850960337 shutdown_connections 2024-03-22T01:27:19.968 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:18.840+0000 7f678d383700 1 -- 172.21.15.72:0/3850960337 wait complete. 2024-03-22T01:27:19.968 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:18.841+0000 7f678d383700 1 Processor -- start 2024-03-22T01:27:19.968 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:18.841+0000 7f678d383700 1 -- start start 2024-03-22T01:27:19.968 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:18.841+0000 7f678d383700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6788107010 0x7f678819e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:27:19.968 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:18.841+0000 7f678d383700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f678807afb0 con 0x7f6788107010 2024-03-22T01:27:19.968 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:18.842+0000 7f6786ffd700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6788107010 0x7f678819e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:27:19.968 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:18.842+0000 7f6786ffd700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6788107010 0x7f678819e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58510/0 (socket says 172.21.15.72:58510) 2024-03-22T01:27:19.968 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:18.842+0000 7f6786ffd700 1 -- 172.21.15.72:0/235135681 learned_addr learned my addr 172.21.15.72:0/235135681 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:27:19.968 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:18.842+0000 7f6786ffd700 1 -- 172.21.15.72:0/235135681 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f678819e810 con 0x7f6788107010 2024-03-22T01:27:19.968 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:18.842+0000 7f6786ffd700 1 --2- 172.21.15.72:0/235135681 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6788107010 0x7f678819e2d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6778003bb0 tx=0x7f67780040b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:27:19.968 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:18.842+0000 7f676ffff700 1 -- 172.21.15.72:0/235135681 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6778010770 con 0x7f6788107010 2024-03-22T01:27:19.969 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:18.842+0000 7f678d383700 1 -- 172.21.15.72:0/235135681 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f678819eaa0 con 0x7f6788107010 2024-03-22T01:27:19.969 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:18.842+0000 7f676ffff700 1 -- 172.21.15.72:0/235135681 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f677801d070 con 0x7f6788107010 2024-03-22T01:27:19.969 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:18.842+0000 7f676ffff700 1 -- 172.21.15.72:0/235135681 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6778019b60 con 0x7f6788107010 2024-03-22T01:27:19.969 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:18.842+0000 7f678d383700 1 -- 172.21.15.72:0/235135681 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67881a1790 con 0x7f6788107010 2024-03-22T01:27:19.969 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:18.843+0000 7f676ffff700 1 -- 172.21.15.72:0/235135681 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 12) ==== 46307+0+0 (secure 0 0 0) 0x7f6778028060 con 0x7f6788107010 2024-03-22T01:27:19.969 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:18.843+0000 7f678d383700 1 -- 172.21.15.72:0/235135681 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6788063cb0 con 0x7f6788107010 2024-03-22T01:27:19.969 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:18.843+0000 7f676ffff700 1 --2- 172.21.15.72:0/235135681 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f6770039240 0x7f677003b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:27:19.969 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:18.843+0000 7f676ffff700 1 -- 172.21.15.72:0/235135681 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f677804d7a0 con 0x7f6788107010 2024-03-22T01:27:19.969 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:18.843+0000 7f67867fc700 1 --2- 172.21.15.72:0/235135681 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f6770039240 0x7f677003b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:27:19.969 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:18.844+0000 7f67867fc700 1 --2- 172.21.15.72:0/235135681 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f6770039240 0x7f677003b700 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f677c006fd0 tx=0x7f677c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:27:19.969 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:18.848+0000 7f676ffff700 1 -- 172.21.15.72:0/235135681 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6778015070 con 0x7f6788107010 2024-03-22T01:27:19.969 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:19.075+0000 7f678d383700 1 -- 172.21.15.72:0/235135681 --> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f678802e070 con 0x7f6770039240 2024-03-22T01:27:19.969 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:19.299+0000 7f676ffff700 1 -- 172.21.15.72:0/235135681 <== mgr.14152 v2:172.21.15.72:6800/401403200 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f678802e070 con 0x7f6770039240 2024-03-22T01:27:19.970 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:19.302+0000 7f678d383700 1 -- 172.21.15.72:0/235135681 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f6770039240 msgr2=0x7f677003b700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:19.970 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:19.302+0000 7f678d383700 1 --2- 172.21.15.72:0/235135681 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f6770039240 0x7f677003b700 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f677c006fd0 tx=0x7f677c006e40 comp rx=0 tx=0).stop 2024-03-22T01:27:19.970 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:19.302+0000 7f678d383700 1 -- 172.21.15.72:0/235135681 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6788107010 msgr2=0x7f678819e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:19.970 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:19.302+0000 7f678d383700 1 --2- 172.21.15.72:0/235135681 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6788107010 0x7f678819e2d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6778003bb0 tx=0x7f67780040b0 comp rx=0 tx=0).stop 2024-03-22T01:27:19.970 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:19.302+0000 7f678d383700 1 -- 172.21.15.72:0/235135681 shutdown_connections 2024-03-22T01:27:19.970 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:19.302+0000 7f678d383700 1 --2- 172.21.15.72:0/235135681 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f6770039240 0x7f677003b700 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:19.970 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:19.302+0000 7f678d383700 1 --2- 172.21.15.72:0/235135681 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6788107010 0x7f678819e2d0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:19.970 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:19.302+0000 7f678d383700 1 -- 172.21.15.72:0/235135681 >> 172.21.15.72:0/235135681 conn(0x7f67881025d0 msgr2=0x7f67881909e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:27:19.970 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:19.302+0000 7f678d383700 1 -- 172.21.15.72:0/235135681 shutdown_connections 2024-03-22T01:27:19.970 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:19.302+0000 7f678d383700 1 -- 172.21.15.72:0/235135681 wait complete. 2024-03-22T01:27:19.970 INFO:teuthology.orchestra.run.smithi072.stdout:Fetching dashboard port number... 2024-03-22T01:27:20.323 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:20 smithi072 ceph-mon[32414]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:27:20.323 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:20 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:27:21.201 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout 8443 2024-03-22T01:27:21.202 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:20.353+0000 7f7646641700 1 Processor -- start 2024-03-22T01:27:21.202 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:20.353+0000 7f7646641700 1 -- start start 2024-03-22T01:27:21.202 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:20.353+0000 7f7646641700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7640107020 0x7f7640107440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:27:21.202 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:20.353+0000 7f7646641700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f7640107a10 con 0x7f7640107020 2024-03-22T01:27:21.202 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:20.353+0000 7f763ffff700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7640107020 0x7f7640107440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:27:21.202 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:20.353+0000 7f763ffff700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7640107020 0x7f7640107440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58514/0 (socket says 172.21.15.72:58514) 2024-03-22T01:27:21.202 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:20.353+0000 7f763ffff700 1 -- 172.21.15.72:0/3902115102 learned_addr learned my addr 172.21.15.72:0/3902115102 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:27:21.202 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:20.353+0000 7f763ffff700 1 -- 172.21.15.72:0/3902115102 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7640108220 con 0x7f7640107020 2024-03-22T01:27:21.202 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:20.353+0000 7f763ffff700 1 --2- 172.21.15.72:0/3902115102 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7640107020 0x7f7640107440 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f76300098a0 tx=0x7f7630009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=533b5b0feec2de99 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:27:21.202 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:20.353+0000 7f763effd700 1 -- 172.21.15.72:0/3902115102 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7630004160 con 0x7f7640107020 2024-03-22T01:27:21.202 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:20.354+0000 7f763effd700 1 -- 172.21.15.72:0/3902115102 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7630004300 con 0x7f7640107020 2024-03-22T01:27:21.202 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:20.354+0000 7f763effd700 1 -- 172.21.15.72:0/3902115102 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7630011b20 con 0x7f7640107020 2024-03-22T01:27:21.203 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:20.354+0000 7f7646641700 1 -- 172.21.15.72:0/3902115102 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7640107020 msgr2=0x7f7640107440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:21.203 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:20.354+0000 7f7646641700 1 --2- 172.21.15.72:0/3902115102 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7640107020 0x7f7640107440 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f76300098a0 tx=0x7f7630009bb0 comp rx=0 tx=0).stop 2024-03-22T01:27:21.203 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:20.354+0000 7f7646641700 1 -- 172.21.15.72:0/3902115102 shutdown_connections 2024-03-22T01:27:21.203 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:20.354+0000 7f7646641700 1 --2- 172.21.15.72:0/3902115102 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7640107020 0x7f7640107440 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:21.203 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:20.354+0000 7f7646641700 1 -- 172.21.15.72:0/3902115102 >> 172.21.15.72:0/3902115102 conn(0x7f76401025c0 msgr2=0x7f7640104a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:27:21.203 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:20.354+0000 7f7646641700 1 -- 172.21.15.72:0/3902115102 shutdown_connections 2024-03-22T01:27:21.203 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:20.354+0000 7f7646641700 1 -- 172.21.15.72:0/3902115102 wait complete. 2024-03-22T01:27:21.203 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:20.355+0000 7f7646641700 1 Processor -- start 2024-03-22T01:27:21.203 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:20.355+0000 7f7646641700 1 -- start start 2024-03-22T01:27:21.203 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:20.355+0000 7f7646641700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7640107020 0x7f764019bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:27:21.203 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:20.355+0000 7f7646641700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f764010b5d0 con 0x7f7640107020 2024-03-22T01:27:21.203 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:20.355+0000 7f763ffff700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7640107020 0x7f764019bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:27:21.203 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:20.355+0000 7f763ffff700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7640107020 0x7f764019bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58520/0 (socket says 172.21.15.72:58520) 2024-03-22T01:27:21.203 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:20.355+0000 7f763ffff700 1 -- 172.21.15.72:0/2910233364 learned_addr learned my addr 172.21.15.72:0/2910233364 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:27:21.204 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:20.355+0000 7f763ffff700 1 -- 172.21.15.72:0/2910233364 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f764019c530 con 0x7f7640107020 2024-03-22T01:27:21.204 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:20.356+0000 7f763ffff700 1 --2- 172.21.15.72:0/2910233364 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7640107020 0x7f764019bff0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f76300117e0 tx=0x7f7630011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:27:21.204 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:20.356+0000 7f763d7fa700 1 -- 172.21.15.72:0/2910233364 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7630003bf0 con 0x7f7640107020 2024-03-22T01:27:21.204 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:20.356+0000 7f763d7fa700 1 -- 172.21.15.72:0/2910233364 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f763001a910 con 0x7f7640107020 2024-03-22T01:27:21.204 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:20.356+0000 7f7646641700 1 -- 172.21.15.72:0/2910233364 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f764019c7c0 con 0x7f7640107020 2024-03-22T01:27:21.204 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:20.356+0000 7f763d7fa700 1 -- 172.21.15.72:0/2910233364 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f76300249a0 con 0x7f7640107020 2024-03-22T01:27:21.204 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:20.356+0000 7f7646641700 1 -- 172.21.15.72:0/2910233364 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f764019cc20 con 0x7f7640107020 2024-03-22T01:27:21.204 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:20.357+0000 7f763d7fa700 1 -- 172.21.15.72:0/2910233364 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 12) ==== 46307+0+0 (secure 0 0 0) 0x7f7630022020 con 0x7f7640107020 2024-03-22T01:27:21.204 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:20.357+0000 7f763d7fa700 1 --2- 172.21.15.72:0/2910233364 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f7628039240 0x7f762803b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:27:21.204 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:20.357+0000 7f763d7fa700 1 -- 172.21.15.72:0/2910233364 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f763004d5e0 con 0x7f7640107020 2024-03-22T01:27:21.204 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:20.357+0000 7f7646641700 1 -- 172.21.15.72:0/2910233364 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7640063cb0 con 0x7f7640107020 2024-03-22T01:27:21.204 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:20.357+0000 7f763f7fe700 1 --2- 172.21.15.72:0/2910233364 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f7628039240 0x7f762803b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:27:21.204 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:20.357+0000 7f763f7fe700 1 --2- 172.21.15.72:0/2910233364 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f7628039240 0x7f762803b700 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7634006fd0 tx=0x7f7634006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:27:21.205 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:20.362+0000 7f763d7fa700 1 -- 172.21.15.72:0/2910233364 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7630011e20 con 0x7f7640107020 2024-03-22T01:27:21.205 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:20.568+0000 7f7646641700 1 -- 172.21.15.72:0/2910233364 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f764019f690 con 0x7f7640107020 2024-03-22T01:27:21.205 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:20.569+0000 7f763d7fa700 1 -- 172.21.15.72:0/2910233364 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f763000b9d0 con 0x7f7640107020 2024-03-22T01:27:21.205 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:20.571+0000 7f7646641700 1 -- 172.21.15.72:0/2910233364 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f7628039240 msgr2=0x7f762803b700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:21.205 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:20.571+0000 7f7646641700 1 --2- 172.21.15.72:0/2910233364 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f7628039240 0x7f762803b700 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7634006fd0 tx=0x7f7634006e40 comp rx=0 tx=0).stop 2024-03-22T01:27:21.205 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:20.571+0000 7f7646641700 1 -- 172.21.15.72:0/2910233364 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7640107020 msgr2=0x7f764019bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:21.205 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:20.571+0000 7f7646641700 1 --2- 172.21.15.72:0/2910233364 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7640107020 0x7f764019bff0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f76300117e0 tx=0x7f7630011c10 comp rx=0 tx=0).stop 2024-03-22T01:27:21.205 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:20.571+0000 7f7646641700 1 -- 172.21.15.72:0/2910233364 shutdown_connections 2024-03-22T01:27:21.205 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:20.571+0000 7f7646641700 1 --2- 172.21.15.72:0/2910233364 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f7628039240 0x7f762803b700 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:21.205 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:20.571+0000 7f7646641700 1 --2- 172.21.15.72:0/2910233364 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7640107020 0x7f764019bff0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:21.205 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:20.571+0000 7f7646641700 1 -- 172.21.15.72:0/2910233364 >> 172.21.15.72:0/2910233364 conn(0x7f76401025c0 msgr2=0x7f7640102ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:27:21.205 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:20.571+0000 7f7646641700 1 -- 172.21.15.72:0/2910233364 shutdown_connections 2024-03-22T01:27:21.205 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:20.571+0000 7f7646641700 1 -- 172.21.15.72:0/2910233364 wait complete. 2024-03-22T01:27:21.210 INFO:teuthology.orchestra.run.smithi072.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-22T01:27:21.210 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stdout disabled 2024-03-22T01:27:21.217 INFO:teuthology.orchestra.run.smithi072.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-22T01:27:21.217 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stdout inactive 2024-03-22T01:27:21.217 INFO:teuthology.orchestra.run.smithi072.stdout:firewalld.service is not enabled 2024-03-22T01:27:21.217 INFO:teuthology.orchestra.run.smithi072.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-22T01:27:21.219 INFO:teuthology.orchestra.run.smithi072.stdout:Ceph Dashboard is now available at: 2024-03-22T01:27:21.219 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:27:21.219 INFO:teuthology.orchestra.run.smithi072.stdout: URL: https://smithi072.front.sepia.ceph.com:8443/ 2024-03-22T01:27:21.219 INFO:teuthology.orchestra.run.smithi072.stdout: User: admin 2024-03-22T01:27:21.219 INFO:teuthology.orchestra.run.smithi072.stdout: Password: avirh8h2n9 2024-03-22T01:27:21.219 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:27:21.220 INFO:teuthology.orchestra.run.smithi072.stdout:Saving cluster configuration to /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/config directory 2024-03-22T01:27:21.220 INFO:teuthology.orchestra.run.smithi072.stdout:Enabling autotune for osd_memory_target 2024-03-22T01:27:21.601 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:21 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/2910233364' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-22T01:27:22.392 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:21.630+0000 7f47d9241700 1 Processor -- start 2024-03-22T01:27:22.393 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:21.630+0000 7f47d9241700 1 -- start start 2024-03-22T01:27:22.393 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:21.630+0000 7f47d9241700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f47d4108170 0x7f47d4108590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:27:22.393 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:21.630+0000 7f47d9241700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f47d4108b60 con 0x7f47d4108170 2024-03-22T01:27:22.393 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:21.630+0000 7f47d2d9d700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f47d4108170 0x7f47d4108590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:27:22.393 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:21.630+0000 7f47d2d9d700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f47d4108170 0x7f47d4108590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36990/0 (socket says 172.21.15.72:36990) 2024-03-22T01:27:22.393 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:21.630+0000 7f47d2d9d700 1 -- 172.21.15.72:0/2819648446 learned_addr learned my addr 172.21.15.72:0/2819648446 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:27:22.393 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:21.630+0000 7f47d2d9d700 1 -- 172.21.15.72:0/2819648446 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47d41092d0 con 0x7f47d4108170 2024-03-22T01:27:22.393 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:21.630+0000 7f47d2d9d700 1 --2- 172.21.15.72:0/2819648446 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f47d4108170 0x7f47d4108590 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f47c40098a0 tx=0x7f47c4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=319702399bcac749 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:27:22.393 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:21.631+0000 7f47d1d9b700 1 -- 172.21.15.72:0/2819648446 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f47c4004160 con 0x7f47d4108170 2024-03-22T01:27:22.393 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:21.631+0000 7f47d1d9b700 1 -- 172.21.15.72:0/2819648446 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47c4004300 con 0x7f47d4108170 2024-03-22T01:27:22.393 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:21.631+0000 7f47d1d9b700 1 -- 172.21.15.72:0/2819648446 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f47c4011b20 con 0x7f47d4108170 2024-03-22T01:27:22.393 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:21.631+0000 7f47d9241700 1 -- 172.21.15.72:0/2819648446 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f47d4108170 msgr2=0x7f47d4108590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:22.393 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:21.631+0000 7f47d9241700 1 --2- 172.21.15.72:0/2819648446 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f47d4108170 0x7f47d4108590 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f47c40098a0 tx=0x7f47c4009bb0 comp rx=0 tx=0).stop 2024-03-22T01:27:22.394 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:21.631+0000 7f47d9241700 1 -- 172.21.15.72:0/2819648446 shutdown_connections 2024-03-22T01:27:22.394 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:21.631+0000 7f47d9241700 1 --2- 172.21.15.72:0/2819648446 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f47d4108170 0x7f47d4108590 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:22.394 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:21.631+0000 7f47d9241700 1 -- 172.21.15.72:0/2819648446 >> 172.21.15.72:0/2819648446 conn(0x7f47d4103750 msgr2=0x7f47d4105b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:27:22.394 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:21.631+0000 7f47d9241700 1 -- 172.21.15.72:0/2819648446 shutdown_connections 2024-03-22T01:27:22.394 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:21.631+0000 7f47d9241700 1 -- 172.21.15.72:0/2819648446 wait complete. 2024-03-22T01:27:22.394 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:21.632+0000 7f47d9241700 1 Processor -- start 2024-03-22T01:27:22.394 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:21.632+0000 7f47d9241700 1 -- start start 2024-03-22T01:27:22.394 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:21.632+0000 7f47d9241700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f47d4108170 0x7f47d419bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:27:22.394 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:21.632+0000 7f47d9241700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f47d410c680 con 0x7f47d4108170 2024-03-22T01:27:22.394 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:21.632+0000 7f47d2d9d700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f47d4108170 0x7f47d419bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:27:22.394 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:21.632+0000 7f47d2d9d700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f47d4108170 0x7f47d419bfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36994/0 (socket says 172.21.15.72:36994) 2024-03-22T01:27:22.394 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:21.632+0000 7f47d2d9d700 1 -- 172.21.15.72:0/2683793607 learned_addr learned my addr 172.21.15.72:0/2683793607 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:27:22.394 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:21.633+0000 7f47d2d9d700 1 -- 172.21.15.72:0/2683793607 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47d419c4e0 con 0x7f47d4108170 2024-03-22T01:27:22.394 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:21.633+0000 7f47d2d9d700 1 --2- 172.21.15.72:0/2683793607 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f47d4108170 0x7f47d419bfa0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f47c40116a0 tx=0x7f47c4011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:27:22.395 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:21.633+0000 7f47bbfff700 1 -- 172.21.15.72:0/2683793607 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f47c4003a40 con 0x7f47d4108170 2024-03-22T01:27:22.395 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:21.633+0000 7f47bbfff700 1 -- 172.21.15.72:0/2683793607 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47c4003be0 con 0x7f47d4108170 2024-03-22T01:27:22.395 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:21.633+0000 7f47d9241700 1 -- 172.21.15.72:0/2683793607 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47d419c770 con 0x7f47d4108170 2024-03-22T01:27:22.395 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:21.633+0000 7f47bbfff700 1 -- 172.21.15.72:0/2683793607 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f47c401a720 con 0x7f47d4108170 2024-03-22T01:27:22.395 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:21.633+0000 7f47d9241700 1 -- 172.21.15.72:0/2683793607 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47d419cbd0 con 0x7f47d4108170 2024-03-22T01:27:22.395 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:21.634+0000 7f47bbfff700 1 -- 172.21.15.72:0/2683793607 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 12) ==== 46307+0+0 (secure 0 0 0) 0x7f47c401aa20 con 0x7f47d4108170 2024-03-22T01:27:22.395 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:21.634+0000 7f47d9241700 1 -- 172.21.15.72:0/2683793607 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47d4063cb0 con 0x7f47d4108170 2024-03-22T01:27:22.395 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:21.634+0000 7f47bbfff700 1 --2- 172.21.15.72:0/2683793607 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f47bc039240 0x7f47bc03b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:27:22.395 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:21.634+0000 7f47bbfff700 1 -- 172.21.15.72:0/2683793607 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f47c404d960 con 0x7f47d4108170 2024-03-22T01:27:22.395 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:21.634+0000 7f47d259c700 1 --2- 172.21.15.72:0/2683793607 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f47bc039240 0x7f47bc03b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:27:22.395 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:21.635+0000 7f47d259c700 1 --2- 172.21.15.72:0/2683793607 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f47bc039240 0x7f47bc03b700 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f47c8006fd0 tx=0x7f47c8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:27:22.395 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:21.639+0000 7f47bbfff700 1 -- 172.21.15.72:0/2683793607 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f47c4011e20 con 0x7f47d4108170 2024-03-22T01:27:22.395 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:21.836+0000 7f47d9241700 1 -- 172.21.15.72:0/2683793607 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f47d419f5a0 con 0x7f47d4108170 2024-03-22T01:27:22.395 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:21.837+0000 7f47bbfff700 1 -- 172.21.15.72:0/2683793607 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f47c400ba00 con 0x7f47d4108170 2024-03-22T01:27:22.396 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:21.839+0000 7f47d9241700 1 -- 172.21.15.72:0/2683793607 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f47bc039240 msgr2=0x7f47bc03b700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:22.396 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:21.839+0000 7f47d9241700 1 --2- 172.21.15.72:0/2683793607 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f47bc039240 0x7f47bc03b700 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f47c8006fd0 tx=0x7f47c8006e40 comp rx=0 tx=0).stop 2024-03-22T01:27:22.396 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:21.839+0000 7f47d9241700 1 -- 172.21.15.72:0/2683793607 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f47d4108170 msgr2=0x7f47d419bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:22.396 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:21.839+0000 7f47d9241700 1 --2- 172.21.15.72:0/2683793607 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f47d4108170 0x7f47d419bfa0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f47c40116a0 tx=0x7f47c4011ba0 comp rx=0 tx=0).stop 2024-03-22T01:27:22.396 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:21.839+0000 7f47d9241700 1 -- 172.21.15.72:0/2683793607 shutdown_connections 2024-03-22T01:27:22.396 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:21.839+0000 7f47d9241700 1 --2- 172.21.15.72:0/2683793607 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f47bc039240 0x7f47bc03b700 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:22.396 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:21.839+0000 7f47d9241700 1 --2- 172.21.15.72:0/2683793607 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f47d4108170 0x7f47d419bfa0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:22.396 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:21.839+0000 7f47d9241700 1 -- 172.21.15.72:0/2683793607 >> 172.21.15.72:0/2683793607 conn(0x7f47d4103750 msgr2=0x7f47d41041a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:27:22.396 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:21.839+0000 7f47d9241700 1 -- 172.21.15.72:0/2683793607 shutdown_connections 2024-03-22T01:27:22.396 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:21.839+0000 7f47d9241700 1 -- 172.21.15.72:0/2683793607 wait complete. 2024-03-22T01:27:23.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:23 smithi072 ceph-mon[32414]: mgrmap e13: x(active, since 6s) 2024-03-22T01:27:23.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:23 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/3144289022' entity='client.admin' 2024-03-22T01:27:23.681 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:22.800+0000 7f5d54264700 1 Processor -- start 2024-03-22T01:27:23.681 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:22.800+0000 7f5d54264700 1 -- start start 2024-03-22T01:27:23.681 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:22.800+0000 7f5d54264700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5d4c07dde0 0x7f5d4c07c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:27:23.681 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:22.800+0000 7f5d54264700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f5d4c07c820 con 0x7f5d4c07dde0 2024-03-22T01:27:23.682 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:22.800+0000 7f5d52000700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5d4c07dde0 0x7f5d4c07c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:27:23.682 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:22.800+0000 7f5d52000700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5d4c07dde0 0x7f5d4c07c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37002/0 (socket says 172.21.15.72:37002) 2024-03-22T01:27:23.682 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:22.800+0000 7f5d52000700 1 -- 172.21.15.72:0/3254860296 learned_addr learned my addr 172.21.15.72:0/3254860296 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:27:23.682 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:22.800+0000 7f5d52000700 1 -- 172.21.15.72:0/3254860296 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d4c07c9a0 con 0x7f5d4c07dde0 2024-03-22T01:27:23.682 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:22.801+0000 7f5d52000700 1 --2- 172.21.15.72:0/3254860296 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5d4c07dde0 0x7f5d4c07c2e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5d480098a0 tx=0x7f5d48009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8b08a2d39c10ae2 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:27:23.682 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:22.801+0000 7f5d50ffe700 1 -- 172.21.15.72:0/3254860296 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5d48004160 con 0x7f5d4c07dde0 2024-03-22T01:27:23.682 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:22.801+0000 7f5d50ffe700 1 -- 172.21.15.72:0/3254860296 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d48004300 con 0x7f5d4c07dde0 2024-03-22T01:27:23.682 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:22.801+0000 7f5d50ffe700 1 -- 172.21.15.72:0/3254860296 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5d48011b20 con 0x7f5d4c07dde0 2024-03-22T01:27:23.682 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:22.801+0000 7f5d54264700 1 -- 172.21.15.72:0/3254860296 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5d4c07dde0 msgr2=0x7f5d4c07c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:23.682 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:22.801+0000 7f5d54264700 1 --2- 172.21.15.72:0/3254860296 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5d4c07dde0 0x7f5d4c07c2e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5d480098a0 tx=0x7f5d48009bb0 comp rx=0 tx=0).stop 2024-03-22T01:27:23.682 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:22.802+0000 7f5d54264700 1 -- 172.21.15.72:0/3254860296 shutdown_connections 2024-03-22T01:27:23.682 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:22.802+0000 7f5d54264700 1 --2- 172.21.15.72:0/3254860296 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5d4c07dde0 0x7f5d4c07c2e0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:23.682 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:22.802+0000 7f5d54264700 1 -- 172.21.15.72:0/3254860296 >> 172.21.15.72:0/3254860296 conn(0x7f5d4c1036c0 msgr2=0x7f5d4c105b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:27:23.683 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:22.802+0000 7f5d54264700 1 -- 172.21.15.72:0/3254860296 shutdown_connections 2024-03-22T01:27:23.683 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:22.802+0000 7f5d54264700 1 -- 172.21.15.72:0/3254860296 wait complete. 2024-03-22T01:27:23.683 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:22.802+0000 7f5d54264700 1 Processor -- start 2024-03-22T01:27:23.683 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:22.802+0000 7f5d54264700 1 -- start start 2024-03-22T01:27:23.683 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:22.802+0000 7f5d54264700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5d4c07dde0 0x7f5d4c19e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:27:23.683 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:22.802+0000 7f5d54264700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f5d4c10e6e0 con 0x7f5d4c07dde0 2024-03-22T01:27:23.683 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:22.802+0000 7f5d52000700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5d4c07dde0 0x7f5d4c19e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:27:23.683 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:22.802+0000 7f5d52000700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5d4c07dde0 0x7f5d4c19e160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37016/0 (socket says 172.21.15.72:37016) 2024-03-22T01:27:23.683 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:22.802+0000 7f5d52000700 1 -- 172.21.15.72:0/3144289022 learned_addr learned my addr 172.21.15.72:0/3144289022 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:27:23.683 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:22.803+0000 7f5d52000700 1 -- 172.21.15.72:0/3144289022 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d4c19e6a0 con 0x7f5d4c07dde0 2024-03-22T01:27:23.683 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:22.803+0000 7f5d52000700 1 --2- 172.21.15.72:0/3144289022 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5d4c07dde0 0x7f5d4c19e160 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5d48011b60 tx=0x7f5d48011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:27:23.683 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:22.803+0000 7f5d42ffd700 1 -- 172.21.15.72:0/3144289022 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5d48003b60 con 0x7f5d4c07dde0 2024-03-22T01:27:23.683 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:22.803+0000 7f5d42ffd700 1 -- 172.21.15.72:0/3144289022 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d4801a910 con 0x7f5d4c07dde0 2024-03-22T01:27:23.684 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:22.803+0000 7f5d54264700 1 -- 172.21.15.72:0/3144289022 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d4c19e930 con 0x7f5d4c07dde0 2024-03-22T01:27:23.684 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:22.803+0000 7f5d42ffd700 1 -- 172.21.15.72:0/3144289022 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5d48024930 con 0x7f5d4c07dde0 2024-03-22T01:27:23.684 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:22.803+0000 7f5d54264700 1 -- 172.21.15.72:0/3144289022 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d4c19ed90 con 0x7f5d4c07dde0 2024-03-22T01:27:23.684 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:22.804+0000 7f5d42ffd700 1 -- 172.21.15.72:0/3144289022 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 13) ==== 46350+0+0 (secure 0 0 0) 0x7f5d48022020 con 0x7f5d4c07dde0 2024-03-22T01:27:23.684 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:22.804+0000 7f5d54264700 1 -- 172.21.15.72:0/3144289022 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d4c063cb0 con 0x7f5d4c07dde0 2024-03-22T01:27:23.684 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:22.804+0000 7f5d42ffd700 1 --2- 172.21.15.72:0/3144289022 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f5d38039390 0x7f5d3803b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:27:23.684 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:22.804+0000 7f5d42ffd700 1 -- 172.21.15.72:0/3144289022 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f5d4804d5e0 con 0x7f5d4c07dde0 2024-03-22T01:27:23.684 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:22.804+0000 7f5d517ff700 1 --2- 172.21.15.72:0/3144289022 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f5d38039390 0x7f5d3803b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:27:23.684 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:22.805+0000 7f5d517ff700 1 --2- 172.21.15.72:0/3144289022 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f5d38039390 0x7f5d3803b850 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c006fd0 tx=0x7f5d3c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:27:23.684 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:22.809+0000 7f5d42ffd700 1 -- 172.21.15.72:0/3144289022 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5d48016df0 con 0x7f5d4c07dde0 2024-03-22T01:27:23.684 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:23.114+0000 7f5d54264700 1 -- 172.21.15.72:0/3144289022 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f5d4c1a1760 con 0x7f5d4c07dde0 2024-03-22T01:27:23.684 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:23.117+0000 7f5d42ffd700 1 -- 172.21.15.72:0/3144289022 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v26)=0 set mgr/dashboard/cluster/status v26) ==== 153+0+0 (secure 0 0 0) 0x7f5d4800bc70 con 0x7f5d4c07dde0 2024-03-22T01:27:23.684 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:23.118+0000 7f5d54264700 1 -- 172.21.15.72:0/3144289022 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f5d38039390 msgr2=0x7f5d3803b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:23.684 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:23.119+0000 7f5d54264700 1 --2- 172.21.15.72:0/3144289022 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f5d38039390 0x7f5d3803b850 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c006fd0 tx=0x7f5d3c006e40 comp rx=0 tx=0).stop 2024-03-22T01:27:23.685 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:23.119+0000 7f5d54264700 1 -- 172.21.15.72:0/3144289022 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5d4c07dde0 msgr2=0x7f5d4c19e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:23.685 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:23.119+0000 7f5d54264700 1 --2- 172.21.15.72:0/3144289022 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5d4c07dde0 0x7f5d4c19e160 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5d48011b60 tx=0x7f5d48011c40 comp rx=0 tx=0).stop 2024-03-22T01:27:23.685 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:23.119+0000 7f5d54264700 1 -- 172.21.15.72:0/3144289022 shutdown_connections 2024-03-22T01:27:23.685 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:23.119+0000 7f5d54264700 1 --2- 172.21.15.72:0/3144289022 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f5d38039390 0x7f5d3803b850 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:23.685 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:23.119+0000 7f5d54264700 1 --2- 172.21.15.72:0/3144289022 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5d4c07dde0 0x7f5d4c19e160 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:23.685 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:23.119+0000 7f5d54264700 1 -- 172.21.15.72:0/3144289022 >> 172.21.15.72:0/3144289022 conn(0x7f5d4c1036c0 msgr2=0x7f5d4c104120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:27:23.685 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:23.119+0000 7f5d54264700 1 -- 172.21.15.72:0/3144289022 shutdown_connections 2024-03-22T01:27:23.685 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-22T01:27:23.119+0000 7f5d54264700 1 -- 172.21.15.72:0/3144289022 wait complete. 2024-03-22T01:27:23.685 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-22T01:27:23.685 INFO:teuthology.orchestra.run.smithi072.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-22T01:27:23.685 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:27:23.685 INFO:teuthology.orchestra.run.smithi072.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-22T01:27:23.685 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:27:23.685 INFO:teuthology.orchestra.run.smithi072.stdout:Or, if you are only running a single cluster on this host: 2024-03-22T01:27:23.686 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:27:23.686 INFO:teuthology.orchestra.run.smithi072.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-22T01:27:23.686 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:27:23.686 INFO:teuthology.orchestra.run.smithi072.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-22T01:27:23.686 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:27:23.686 INFO:teuthology.orchestra.run.smithi072.stdout: ceph telemetry on 2024-03-22T01:27:23.686 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:27:23.686 INFO:teuthology.orchestra.run.smithi072.stdout:For more information see: 2024-03-22T01:27:23.686 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:27:23.686 INFO:teuthology.orchestra.run.smithi072.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-22T01:27:23.686 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:27:23.686 INFO:teuthology.orchestra.run.smithi072.stdout:Bootstrap complete. 2024-03-22T01:27:23.719 INFO:tasks.cephadm:Fetching config... 2024-03-22T01:27:23.719 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-22T01:27:23.720 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-22T01:27:23.735 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-22T01:27:23.735 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-22T01:27:23.735 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-22T01:27:23.790 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-22T01:27:23.790 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-22T01:27:23.790 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-22T01:27:23.856 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-22T01:27:23.857 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-22T01:27:23.857 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-22T01:27:23.911 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-22T01:27:23.912 DEBUG:teuthology.orchestra.run.smithi072:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCqbdTGzUNA4dKEUFqNy/KZI9cTM28jUFtRQ5/XYhqEZL3s9G8rf/nzrptmYNympWhRnYmwO3o/vrIGHTrMUMBNHDw3n/K+ZbaCM8nPEgbCXSrvMMFTTAu9xGsP1DI1bJTDnZmWOXDCBawkSLBt+xS1ThnAmtHtTQZjaONP5VHTDrsRnVZXvnyZCyVz/ewpgXrXHmFkjwamnDREOKsrMgj2O+i4KS6lD2lDRAjw9IfNhJVHv9zrkPtsphMEjgJ2hJY+LxEY1FFflMEOhTZTxe5TocU+rNfp7FumGu7MwsGbvL9uSkfvLad/1YcvwB2SDeZN69tGKwBZXsD1ayeRrUnflkxoIiQ+UKhObJQtwSX8w3rq01puWJkiXVS398jgf1pxaiFQW50KTmBubt28cXYQ90Al8a89xMHrAFGjIET38coyVhYZfRRe1W8T3WglAS48hK0KrDaDWTH/jQjhPUpV7zAG2FepmcmJtb1aqNY+agHkJiPamzhpCfz6xxysod8= ceph-022a99bc-e7eb-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-22T01:27:23.992 INFO:teuthology.orchestra.run.smithi072.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCqbdTGzUNA4dKEUFqNy/KZI9cTM28jUFtRQ5/XYhqEZL3s9G8rf/nzrptmYNympWhRnYmwO3o/vrIGHTrMUMBNHDw3n/K+ZbaCM8nPEgbCXSrvMMFTTAu9xGsP1DI1bJTDnZmWOXDCBawkSLBt+xS1ThnAmtHtTQZjaONP5VHTDrsRnVZXvnyZCyVz/ewpgXrXHmFkjwamnDREOKsrMgj2O+i4KS6lD2lDRAjw9IfNhJVHv9zrkPtsphMEjgJ2hJY+LxEY1FFflMEOhTZTxe5TocU+rNfp7FumGu7MwsGbvL9uSkfvLad/1YcvwB2SDeZN69tGKwBZXsD1ayeRrUnflkxoIiQ+UKhObJQtwSX8w3rq01puWJkiXVS398jgf1pxaiFQW50KTmBubt28cXYQ90Al8a89xMHrAFGjIET38coyVhYZfRRe1W8T3WglAS48hK0KrDaDWTH/jQjhPUpV7zAG2FepmcmJtb1aqNY+agHkJiPamzhpCfz6xxysod8= ceph-022a99bc-e7eb-11ee-95cd-87774f69a715 2024-03-22T01:27:24.006 DEBUG:teuthology.orchestra.run.smithi089:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCqbdTGzUNA4dKEUFqNy/KZI9cTM28jUFtRQ5/XYhqEZL3s9G8rf/nzrptmYNympWhRnYmwO3o/vrIGHTrMUMBNHDw3n/K+ZbaCM8nPEgbCXSrvMMFTTAu9xGsP1DI1bJTDnZmWOXDCBawkSLBt+xS1ThnAmtHtTQZjaONP5VHTDrsRnVZXvnyZCyVz/ewpgXrXHmFkjwamnDREOKsrMgj2O+i4KS6lD2lDRAjw9IfNhJVHv9zrkPtsphMEjgJ2hJY+LxEY1FFflMEOhTZTxe5TocU+rNfp7FumGu7MwsGbvL9uSkfvLad/1YcvwB2SDeZN69tGKwBZXsD1ayeRrUnflkxoIiQ+UKhObJQtwSX8w3rq01puWJkiXVS398jgf1pxaiFQW50KTmBubt28cXYQ90Al8a89xMHrAFGjIET38coyVhYZfRRe1W8T3WglAS48hK0KrDaDWTH/jQjhPUpV7zAG2FepmcmJtb1aqNY+agHkJiPamzhpCfz6xxysod8= ceph-022a99bc-e7eb-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-22T01:27:24.050 INFO:teuthology.orchestra.run.smithi089.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCqbdTGzUNA4dKEUFqNy/KZI9cTM28jUFtRQ5/XYhqEZL3s9G8rf/nzrptmYNympWhRnYmwO3o/vrIGHTrMUMBNHDw3n/K+ZbaCM8nPEgbCXSrvMMFTTAu9xGsP1DI1bJTDnZmWOXDCBawkSLBt+xS1ThnAmtHtTQZjaONP5VHTDrsRnVZXvnyZCyVz/ewpgXrXHmFkjwamnDREOKsrMgj2O+i4KS6lD2lDRAjw9IfNhJVHv9zrkPtsphMEjgJ2hJY+LxEY1FFflMEOhTZTxe5TocU+rNfp7FumGu7MwsGbvL9uSkfvLad/1YcvwB2SDeZN69tGKwBZXsD1ayeRrUnflkxoIiQ+UKhObJQtwSX8w3rq01puWJkiXVS398jgf1pxaiFQW50KTmBubt28cXYQ90Al8a89xMHrAFGjIET38coyVhYZfRRe1W8T3WglAS48hK0KrDaDWTH/jQjhPUpV7zAG2FepmcmJtb1aqNY+agHkJiPamzhpCfz6xxysod8= ceph-022a99bc-e7eb-11ee-95cd-87774f69a715 2024-03-22T01:27:24.065 DEBUG:teuthology.orchestra.run.smithi098:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCqbdTGzUNA4dKEUFqNy/KZI9cTM28jUFtRQ5/XYhqEZL3s9G8rf/nzrptmYNympWhRnYmwO3o/vrIGHTrMUMBNHDw3n/K+ZbaCM8nPEgbCXSrvMMFTTAu9xGsP1DI1bJTDnZmWOXDCBawkSLBt+xS1ThnAmtHtTQZjaONP5VHTDrsRnVZXvnyZCyVz/ewpgXrXHmFkjwamnDREOKsrMgj2O+i4KS6lD2lDRAjw9IfNhJVHv9zrkPtsphMEjgJ2hJY+LxEY1FFflMEOhTZTxe5TocU+rNfp7FumGu7MwsGbvL9uSkfvLad/1YcvwB2SDeZN69tGKwBZXsD1ayeRrUnflkxoIiQ+UKhObJQtwSX8w3rq01puWJkiXVS398jgf1pxaiFQW50KTmBubt28cXYQ90Al8a89xMHrAFGjIET38coyVhYZfRRe1W8T3WglAS48hK0KrDaDWTH/jQjhPUpV7zAG2FepmcmJtb1aqNY+agHkJiPamzhpCfz6xxysod8= ceph-022a99bc-e7eb-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-22T01:27:24.115 INFO:teuthology.orchestra.run.smithi098.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCqbdTGzUNA4dKEUFqNy/KZI9cTM28jUFtRQ5/XYhqEZL3s9G8rf/nzrptmYNympWhRnYmwO3o/vrIGHTrMUMBNHDw3n/K+ZbaCM8nPEgbCXSrvMMFTTAu9xGsP1DI1bJTDnZmWOXDCBawkSLBt+xS1ThnAmtHtTQZjaONP5VHTDrsRnVZXvnyZCyVz/ewpgXrXHmFkjwamnDREOKsrMgj2O+i4KS6lD2lDRAjw9IfNhJVHv9zrkPtsphMEjgJ2hJY+LxEY1FFflMEOhTZTxe5TocU+rNfp7FumGu7MwsGbvL9uSkfvLad/1YcvwB2SDeZN69tGKwBZXsD1ayeRrUnflkxoIiQ+UKhObJQtwSX8w3rq01puWJkiXVS398jgf1pxaiFQW50KTmBubt28cXYQ90Al8a89xMHrAFGjIET38coyVhYZfRRe1W8T3WglAS48hK0KrDaDWTH/jQjhPUpV7zAG2FepmcmJtb1aqNY+agHkJiPamzhpCfz6xxysod8= ceph-022a99bc-e7eb-11ee-95cd-87774f69a715 2024-03-22T01:27:24.131 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-22T01:27:24.370 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:27:26.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:26 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:27:26.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:26 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:27:26.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:26 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi072", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:27:26.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:26 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:27:26.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:26 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:27:26.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:26 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:27:26.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:26.661+0000 7f6735a1c700 1 -- 172.21.15.72:0/4193200074 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f67301025a0 msgr2=0x7f67301029c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:26.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:26.661+0000 7f6735a1c700 1 --2- 172.21.15.72:0/4193200074 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f67301025a0 0x7f67301029c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6720009910 tx=0x7f6720009c20 comp rx=0 tx=0).stop 2024-03-22T01:27:26.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:26.661+0000 7f6735a1c700 1 -- 172.21.15.72:0/4193200074 shutdown_connections 2024-03-22T01:27:26.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:26.661+0000 7f6735a1c700 1 --2- 172.21.15.72:0/4193200074 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f67301025a0 0x7f67301029c0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:26.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:26.661+0000 7f6735a1c700 1 -- 172.21.15.72:0/4193200074 >> 172.21.15.72:0/4193200074 conn(0x7f67300fbce0 msgr2=0x7f67300fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:27:26.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:26.661+0000 7f6735a1c700 1 -- 172.21.15.72:0/4193200074 shutdown_connections 2024-03-22T01:27:26.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:26.661+0000 7f6735a1c700 1 -- 172.21.15.72:0/4193200074 wait complete. 2024-03-22T01:27:26.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:26.662+0000 7f6735a1c700 1 Processor -- start 2024-03-22T01:27:26.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:26.662+0000 7f6735a1c700 1 -- start start 2024-03-22T01:27:26.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:26.662+0000 7f6735a1c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f67301025a0 0x7f67300fdba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:27:26.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:26.662+0000 7f6735a1c700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f6730107050 con 0x7f67301025a0 2024-03-22T01:27:26.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:26.662+0000 7f672effd700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f67301025a0 0x7f67300fdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:27:26.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:26.662+0000 7f672effd700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f67301025a0 0x7f67300fdba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37042/0 (socket says 172.21.15.72:37042) 2024-03-22T01:27:26.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:26.662+0000 7f672effd700 1 -- 172.21.15.72:0/4076277438 learned_addr learned my addr 172.21.15.72:0/4076277438 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:27:26.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:26.662+0000 7f672effd700 1 -- 172.21.15.72:0/4076277438 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67300fe0e0 con 0x7f67301025a0 2024-03-22T01:27:26.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:26.662+0000 7f672effd700 1 --2- 172.21.15.72:0/4076277438 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f67301025a0 0x7f67300fdba0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6720006010 tx=0x7f6720005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:27:26.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:26.663+0000 7f6734a1a700 1 -- 172.21.15.72:0/4076277438 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6720027070 con 0x7f67301025a0 2024-03-22T01:27:26.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:26.663+0000 7f6734a1a700 1 -- 172.21.15.72:0/4076277438 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f672000be40 con 0x7f67301025a0 2024-03-22T01:27:26.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:26.663+0000 7f6735a1c700 1 -- 172.21.15.72:0/4076277438 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67300fc2d0 con 0x7f67301025a0 2024-03-22T01:27:26.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:26.663+0000 7f6734a1a700 1 -- 172.21.15.72:0/4076277438 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f672000f460 con 0x7f67301025a0 2024-03-22T01:27:26.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:26.663+0000 7f6735a1c700 1 -- 172.21.15.72:0/4076277438 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67300fc810 con 0x7f67301025a0 2024-03-22T01:27:26.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:26.664+0000 7f6734a1a700 1 -- 172.21.15.72:0/4076277438 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 13) ==== 46350+0+0 (secure 0 0 0) 0x7f672002a070 con 0x7f67301025a0 2024-03-22T01:27:26.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:26.664+0000 7f6735a1c700 1 -- 172.21.15.72:0/4076277438 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67301077d0 con 0x7f67301025a0 2024-03-22T01:27:26.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:26.664+0000 7f6734a1a700 1 --2- 172.21.15.72:0/4076277438 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f6718039220 0x7f671803b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:27:26.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:26.664+0000 7f6734a1a700 1 -- 172.21.15.72:0/4076277438 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f672001e030 con 0x7f67301025a0 2024-03-22T01:27:26.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:26.664+0000 7f672e7fc700 1 --2- 172.21.15.72:0/4076277438 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f6718039220 0x7f671803b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:27:26.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:26.664+0000 7f672e7fc700 1 --2- 172.21.15.72:0/4076277438 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f6718039220 0x7f671803b6e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6724006fd0 tx=0x7f6724006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:27:26.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:26.673+0000 7f6734a1a700 1 -- 172.21.15.72:0/4076277438 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6720013070 con 0x7f67301025a0 2024-03-22T01:27:26.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:26.866+0000 7f6735a1c700 1 -- 172.21.15.72:0/4076277438 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f6730190d40 con 0x7f67301025a0 2024-03-22T01:27:26.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:26.869+0000 7f6734a1a700 1 -- 172.21.15.72:0/4076277438 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f672002a360 con 0x7f67301025a0 2024-03-22T01:27:26.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:26.871+0000 7f6735a1c700 1 -- 172.21.15.72:0/4076277438 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f6718039220 msgr2=0x7f671803b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:26.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:26.871+0000 7f6735a1c700 1 --2- 172.21.15.72:0/4076277438 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f6718039220 0x7f671803b6e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6724006fd0 tx=0x7f6724006e40 comp rx=0 tx=0).stop 2024-03-22T01:27:26.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:26.871+0000 7f6735a1c700 1 -- 172.21.15.72:0/4076277438 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f67301025a0 msgr2=0x7f67300fdba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:26.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:26.871+0000 7f6735a1c700 1 --2- 172.21.15.72:0/4076277438 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f67301025a0 0x7f67300fdba0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6720006010 tx=0x7f6720005050 comp rx=0 tx=0).stop 2024-03-22T01:27:26.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:26.872+0000 7f6735a1c700 1 -- 172.21.15.72:0/4076277438 shutdown_connections 2024-03-22T01:27:26.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:26.872+0000 7f6735a1c700 1 --2- 172.21.15.72:0/4076277438 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f6718039220 0x7f671803b6e0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:26.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:26.872+0000 7f6735a1c700 1 --2- 172.21.15.72:0/4076277438 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f67301025a0 0x7f67300fdba0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:26.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:26.872+0000 7f6735a1c700 1 -- 172.21.15.72:0/4076277438 >> 172.21.15.72:0/4076277438 conn(0x7f67300fbce0 msgr2=0x7f6730100520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:27:26.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:26.872+0000 7f6735a1c700 1 -- 172.21.15.72:0/4076277438 shutdown_connections 2024-03-22T01:27:26.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:26.872+0000 7f6735a1c700 1 -- 172.21.15.72:0/4076277438 wait complete. 2024-03-22T01:27:27.414 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-22T01:27:27.414 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-22T01:27:27.647 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:27:28.061 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:27 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/4076277438' entity='client.admin' 2024-03-22T01:27:28.062 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:27 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:27:28.062 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:27 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:27:29.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:29.114+0000 7f76dd39b700 1 -- 172.21.15.72:0/4131984683 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f76d81020e0 msgr2=0x7f76d8102500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:29.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:29.114+0000 7f76dd39b700 1 --2- 172.21.15.72:0/4131984683 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f76d81020e0 0x7f76d8102500 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f76c8009910 tx=0x7f76c8009c20 comp rx=0 tx=0).stop 2024-03-22T01:27:29.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:29.115+0000 7f76dd39b700 1 -- 172.21.15.72:0/4131984683 shutdown_connections 2024-03-22T01:27:29.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:29.115+0000 7f76dd39b700 1 --2- 172.21.15.72:0/4131984683 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f76d81020e0 0x7f76d8102500 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:29.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:29.115+0000 7f76dd39b700 1 -- 172.21.15.72:0/4131984683 >> 172.21.15.72:0/4131984683 conn(0x7f76d80fd6a0 msgr2=0x7f76d80ffac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:27:29.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:29.115+0000 7f76dd39b700 1 -- 172.21.15.72:0/4131984683 shutdown_connections 2024-03-22T01:27:29.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:29.115+0000 7f76dd39b700 1 -- 172.21.15.72:0/4131984683 wait complete. 2024-03-22T01:27:29.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:29.115+0000 7f76dd39b700 1 Processor -- start 2024-03-22T01:27:29.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:29.115+0000 7f76dd39b700 1 -- start start 2024-03-22T01:27:29.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:29.116+0000 7f76dd39b700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f76d81020e0 0x7f76d81977b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:27:29.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:29.116+0000 7f76dd39b700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f76d8108170 con 0x7f76d81020e0 2024-03-22T01:27:29.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:29.116+0000 7f76d6ffd700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f76d81020e0 0x7f76d81977b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:27:29.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:29.116+0000 7f76d6ffd700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f76d81020e0 0x7f76d81977b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37060/0 (socket says 172.21.15.72:37060) 2024-03-22T01:27:29.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:29.116+0000 7f76d6ffd700 1 -- 172.21.15.72:0/3084519398 learned_addr learned my addr 172.21.15.72:0/3084519398 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:27:29.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:29.116+0000 7f76d6ffd700 1 -- 172.21.15.72:0/3084519398 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76d8197cf0 con 0x7f76d81020e0 2024-03-22T01:27:29.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:29.116+0000 7f76d6ffd700 1 --2- 172.21.15.72:0/3084519398 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f76d81020e0 0x7f76d81977b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f76c8005850 tx=0x7f76c8005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:27:29.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:29.116+0000 7f76cffff700 1 -- 172.21.15.72:0/3084519398 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f76c8026070 con 0x7f76d81020e0 2024-03-22T01:27:29.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:29.116+0000 7f76dd39b700 1 -- 172.21.15.72:0/3084519398 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76d8197f80 con 0x7f76d81020e0 2024-03-22T01:27:29.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:29.116+0000 7f76cffff700 1 -- 172.21.15.72:0/3084519398 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76c80044a0 con 0x7f76d81020e0 2024-03-22T01:27:29.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:29.116+0000 7f76cffff700 1 -- 172.21.15.72:0/3084519398 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f76c8003ab0 con 0x7f76d81020e0 2024-03-22T01:27:29.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:29.116+0000 7f76dd39b700 1 -- 172.21.15.72:0/3084519398 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76d819ac70 con 0x7f76d81020e0 2024-03-22T01:27:29.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:29.117+0000 7f76cffff700 1 -- 172.21.15.72:0/3084519398 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 13) ==== 46350+0+0 (secure 0 0 0) 0x7f76c8029070 con 0x7f76d81020e0 2024-03-22T01:27:29.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:29.117+0000 7f76dd39b700 1 -- 172.21.15.72:0/3084519398 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76d81088f0 con 0x7f76d81020e0 2024-03-22T01:27:29.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:29.117+0000 7f76cffff700 1 --2- 172.21.15.72:0/3084519398 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f76c4039220 0x7f76c403b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:27:29.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:29.117+0000 7f76cffff700 1 -- 172.21.15.72:0/3084519398 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f76c801e070 con 0x7f76d81020e0 2024-03-22T01:27:29.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:29.117+0000 7f76d67fc700 1 --2- 172.21.15.72:0/3084519398 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f76c4039220 0x7f76c403b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:27:29.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:29.118+0000 7f76d67fc700 1 --2- 172.21.15.72:0/3084519398 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f76c4039220 0x7f76c403b6e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f76c0006fd0 tx=0x7f76c0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:27:29.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:29.122+0000 7f76cffff700 1 -- 172.21.15.72:0/3084519398 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f76c8013070 con 0x7f76d81020e0 2024-03-22T01:27:29.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:29.320+0000 7f76dd39b700 1 -- 172.21.15.72:0/3084519398 --> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f76d802de80 con 0x7f76c4039220 2024-03-22T01:27:29.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:29.324+0000 7f76cffff700 1 -- 172.21.15.72:0/3084519398 <== mgr.14152 v2:172.21.15.72:6800/401403200 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f76d802de80 con 0x7f76c4039220 2024-03-22T01:27:29.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:29.326+0000 7f76dd39b700 1 -- 172.21.15.72:0/3084519398 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f76c4039220 msgr2=0x7f76c403b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:29.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:29.326+0000 7f76dd39b700 1 --2- 172.21.15.72:0/3084519398 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f76c4039220 0x7f76c403b6e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f76c0006fd0 tx=0x7f76c0006e40 comp rx=0 tx=0).stop 2024-03-22T01:27:29.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:29.326+0000 7f76dd39b700 1 -- 172.21.15.72:0/3084519398 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f76d81020e0 msgr2=0x7f76d81977b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:29.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:29.326+0000 7f76dd39b700 1 --2- 172.21.15.72:0/3084519398 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f76d81020e0 0x7f76d81977b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f76c8005850 tx=0x7f76c8005030 comp rx=0 tx=0).stop 2024-03-22T01:27:29.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:29.326+0000 7f76dd39b700 1 -- 172.21.15.72:0/3084519398 shutdown_connections 2024-03-22T01:27:29.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:29.326+0000 7f76dd39b700 1 --2- 172.21.15.72:0/3084519398 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f76c4039220 0x7f76c403b6e0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:29.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:29.326+0000 7f76dd39b700 1 --2- 172.21.15.72:0/3084519398 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f76d81020e0 0x7f76d81977b0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:29.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:29.326+0000 7f76dd39b700 1 -- 172.21.15.72:0/3084519398 >> 172.21.15.72:0/3084519398 conn(0x7f76d80fd6a0 msgr2=0x7f76d80fe0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:27:29.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:29.326+0000 7f76dd39b700 1 -- 172.21.15.72:0/3084519398 shutdown_connections 2024-03-22T01:27:29.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:29.326+0000 7f76dd39b700 1 -- 172.21.15.72:0/3084519398 wait complete. 2024-03-22T01:27:30.066 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi089 2024-03-22T01:27:30.066 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-22T01:27:30.066 DEBUG:teuthology.orchestra.run.smithi089:> dd of=/etc/ceph/ceph.conf 2024-03-22T01:27:30.085 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-22T01:27:30.086 DEBUG:teuthology.orchestra.run.smithi089:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-22T01:27:30.143 INFO:tasks.cephadm:Adding host smithi089 to orchestrator... 2024-03-22T01:27:30.143 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph orch host add smithi089 2024-03-22T01:27:30.384 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:27:30.589 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:30 smithi072 ceph-mon[32414]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:27:30.589 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:30 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:27:30.589 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:30 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:27:30.589 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:30 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:27:30.589 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:30 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:27:30.589 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:30 smithi072 ceph-mon[32414]: Updating smithi072:/etc/ceph/ceph.conf 2024-03-22T01:27:30.589 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:30 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:27:30.589 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:30 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:27:30.589 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:30 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:27:31.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:31 smithi072 ceph-mon[32414]: Updating smithi072:/var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T01:27:31.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:31 smithi072 ceph-mon[32414]: Updating smithi072:/etc/ceph/ceph.client.admin.keyring 2024-03-22T01:27:31.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:31 smithi072 ceph-mon[32414]: Updating smithi072:/var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T01:27:31.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:31.834+0000 7fa834a14700 1 -- 172.21.15.72:0/2538376727 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa8301066e0 msgr2=0x7fa830106ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:31.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:31.834+0000 7fa834a14700 1 --2- 172.21.15.72:0/2538376727 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa8301066e0 0x7fa830106ac0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa820009910 tx=0x7fa820009c20 comp rx=0 tx=0).stop 2024-03-22T01:27:31.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:31.834+0000 7fa834a14700 1 -- 172.21.15.72:0/2538376727 shutdown_connections 2024-03-22T01:27:31.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:31.834+0000 7fa834a14700 1 --2- 172.21.15.72:0/2538376727 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa8301066e0 0x7fa830106ac0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:31.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:31.834+0000 7fa834a14700 1 -- 172.21.15.72:0/2538376727 >> 172.21.15.72:0/2538376727 conn(0x7fa8300fc1d0 msgr2=0x7fa8300fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:27:31.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:31.834+0000 7fa834a14700 1 -- 172.21.15.72:0/2538376727 shutdown_connections 2024-03-22T01:27:31.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:31.834+0000 7fa834a14700 1 -- 172.21.15.72:0/2538376727 wait complete. 2024-03-22T01:27:31.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:31.835+0000 7fa834a14700 1 Processor -- start 2024-03-22T01:27:31.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:31.835+0000 7fa834a14700 1 -- start start 2024-03-22T01:27:31.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:31.835+0000 7fa834a14700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa8301066e0 0x7fa83019c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:27:31.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:31.835+0000 7fa834a14700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fa83010ad20 con 0x7fa8301066e0 2024-03-22T01:27:31.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:31.835+0000 7fa82e59c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa8301066e0 0x7fa83019c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:27:31.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:31.835+0000 7fa82e59c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa8301066e0 0x7fa83019c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45232/0 (socket says 172.21.15.72:45232) 2024-03-22T01:27:31.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:31.835+0000 7fa82e59c700 1 -- 172.21.15.72:0/313528869 learned_addr learned my addr 172.21.15.72:0/313528869 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:27:31.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:31.835+0000 7fa82e59c700 1 -- 172.21.15.72:0/313528869 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa83019cc10 con 0x7fa8301066e0 2024-03-22T01:27:31.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:31.835+0000 7fa82e59c700 1 --2- 172.21.15.72:0/313528869 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa8301066e0 0x7fa83019c6d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa820005850 tx=0x7fa820005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:27:31.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:31.836+0000 7fa81f7fe700 1 -- 172.21.15.72:0/313528869 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa820026070 con 0x7fa8301066e0 2024-03-22T01:27:31.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:31.836+0000 7fa81f7fe700 1 -- 172.21.15.72:0/313528869 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa82000bbd0 con 0x7fa8301066e0 2024-03-22T01:27:31.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:31.836+0000 7fa834a14700 1 -- 172.21.15.72:0/313528869 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa83019cf00 con 0x7fa8301066e0 2024-03-22T01:27:31.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:31.836+0000 7fa81f7fe700 1 -- 172.21.15.72:0/313528869 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa820003970 con 0x7fa8301066e0 2024-03-22T01:27:31.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:31.836+0000 7fa834a14700 1 -- 172.21.15.72:0/313528869 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa830196890 con 0x7fa8301066e0 2024-03-22T01:27:31.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:31.837+0000 7fa81f7fe700 1 -- 172.21.15.72:0/313528869 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 13) ==== 46350+0+0 (secure 0 0 0) 0x7fa820029070 con 0x7fa8301066e0 2024-03-22T01:27:31.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:31.837+0000 7fa834a14700 1 -- 172.21.15.72:0/313528869 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa830101e20 con 0x7fa8301066e0 2024-03-22T01:27:31.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:31.837+0000 7fa81f7fe700 1 --2- 172.21.15.72:0/313528869 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fa818039220 0x7fa81803b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:27:31.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:31.837+0000 7fa81f7fe700 1 -- 172.21.15.72:0/313528869 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fa82001e070 con 0x7fa8301066e0 2024-03-22T01:27:31.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:31.837+0000 7fa82dd9b700 1 --2- 172.21.15.72:0/313528869 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fa818039220 0x7fa81803b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:27:31.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:31.837+0000 7fa82dd9b700 1 --2- 172.21.15.72:0/313528869 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fa818039220 0x7fa81803b6e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa824006fd0 tx=0x7fa824006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:27:31.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:31.842+0000 7fa81f7fe700 1 -- 172.21.15.72:0/313528869 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa820013070 con 0x7fa8301066e0 2024-03-22T01:27:32.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:32.043+0000 7fa834a14700 1 -- 172.21.15.72:0/313528869 --> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi089", "target": ["mon-mgr", ""]}) -- 0x7fa830197e30 con 0x7fa818039220 2024-03-22T01:27:32.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:32 smithi072 ceph-mon[32414]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi089", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:27:33.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:33 smithi072 ceph-mon[32414]: Deploying cephadm binary to smithi089 2024-03-22T01:27:36.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:36.283+0000 7fa81f7fe700 1 -- 172.21.15.72:0/313528869 <== mgr.14152 v2:172.21.15.72:6800/401403200 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7fa830197e30 con 0x7fa818039220 2024-03-22T01:27:36.283 INFO:teuthology.orchestra.run.smithi072.stdout:Added host 'smithi089' with addr '172.21.15.89' 2024-03-22T01:27:36.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:36.286+0000 7fa834a14700 1 -- 172.21.15.72:0/313528869 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fa818039220 msgr2=0x7fa81803b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:36.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:36.286+0000 7fa834a14700 1 --2- 172.21.15.72:0/313528869 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fa818039220 0x7fa81803b6e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa824006fd0 tx=0x7fa824006e40 comp rx=0 tx=0).stop 2024-03-22T01:27:36.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:36.286+0000 7fa834a14700 1 -- 172.21.15.72:0/313528869 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa8301066e0 msgr2=0x7fa83019c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:36.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:36.286+0000 7fa834a14700 1 --2- 172.21.15.72:0/313528869 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa8301066e0 0x7fa83019c6d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa820005850 tx=0x7fa820005030 comp rx=0 tx=0).stop 2024-03-22T01:27:36.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:36.286+0000 7fa834a14700 1 -- 172.21.15.72:0/313528869 shutdown_connections 2024-03-22T01:27:36.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:36.286+0000 7fa834a14700 1 --2- 172.21.15.72:0/313528869 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fa818039220 0x7fa81803b6e0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:36.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:36.286+0000 7fa834a14700 1 --2- 172.21.15.72:0/313528869 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa8301066e0 0x7fa83019c6d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:36.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:36.286+0000 7fa834a14700 1 -- 172.21.15.72:0/313528869 >> 172.21.15.72:0/313528869 conn(0x7fa8300fc1d0 msgr2=0x7fa8300fdbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:27:36.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:36.286+0000 7fa834a14700 1 -- 172.21.15.72:0/313528869 shutdown_connections 2024-03-22T01:27:36.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:36.286+0000 7fa834a14700 1 -- 172.21.15.72:0/313528869 wait complete. 2024-03-22T01:27:36.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:36 smithi072 ceph-mon[32414]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:27:36.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:36 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:27:36.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:36 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:27:36.857 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph orch host ls --format=json 2024-03-22T01:27:37.087 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:27:37.487 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:37 smithi072 ceph-mon[32414]: Added host smithi089 2024-03-22T01:27:37.487 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:37 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:27:37.488 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:37 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:27:38.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:38.555+0000 7f6b72a8f700 1 -- 172.21.15.72:0/1396364761 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b6c1011e0 msgr2=0x7f6b6c1015c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:38.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:38.555+0000 7f6b72a8f700 1 --2- 172.21.15.72:0/1396364761 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b6c1011e0 0x7f6b6c1015c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6b60009910 tx=0x7f6b60009c20 comp rx=0 tx=0).stop 2024-03-22T01:27:38.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:38.555+0000 7f6b72a8f700 1 -- 172.21.15.72:0/1396364761 shutdown_connections 2024-03-22T01:27:38.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:38.555+0000 7f6b72a8f700 1 --2- 172.21.15.72:0/1396364761 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b6c1011e0 0x7f6b6c1015c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:38.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:38.555+0000 7f6b72a8f700 1 -- 172.21.15.72:0/1396364761 >> 172.21.15.72:0/1396364761 conn(0x7f6b6c0fcad0 msgr2=0x7f6b6c0feef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:27:38.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:38.556+0000 7f6b72a8f700 1 -- 172.21.15.72:0/1396364761 shutdown_connections 2024-03-22T01:27:38.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:38.556+0000 7f6b72a8f700 1 -- 172.21.15.72:0/1396364761 wait complete. 2024-03-22T01:27:38.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:38.556+0000 7f6b72a8f700 1 Processor -- start 2024-03-22T01:27:38.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:38.556+0000 7f6b72a8f700 1 -- start start 2024-03-22T01:27:38.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:38.556+0000 7f6b72a8f700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b6c1011e0 0x7f6b6c1989e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:27:38.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:38.556+0000 7f6b72a8f700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f6b6c10b5b0 con 0x7f6b6c1011e0 2024-03-22T01:27:38.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:38.556+0000 7f6b7082b700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b6c1011e0 0x7f6b6c1989e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:27:38.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:38.556+0000 7f6b7082b700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b6c1011e0 0x7f6b6c1989e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45252/0 (socket says 172.21.15.72:45252) 2024-03-22T01:27:38.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:38.556+0000 7f6b7082b700 1 -- 172.21.15.72:0/3872161569 learned_addr learned my addr 172.21.15.72:0/3872161569 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:27:38.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:38.557+0000 7f6b7082b700 1 -- 172.21.15.72:0/3872161569 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b6c198f20 con 0x7f6b6c1011e0 2024-03-22T01:27:38.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:38.557+0000 7f6b7082b700 1 --2- 172.21.15.72:0/3872161569 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b6c1011e0 0x7f6b6c1989e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6b60005850 tx=0x7f6b60005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:27:38.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:38.557+0000 7f6b69ffb700 1 -- 172.21.15.72:0/3872161569 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6b60026070 con 0x7f6b6c1011e0 2024-03-22T01:27:38.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:38.557+0000 7f6b69ffb700 1 -- 172.21.15.72:0/3872161569 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b6000bbd0 con 0x7f6b6c1011e0 2024-03-22T01:27:38.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:38.557+0000 7f6b69ffb700 1 -- 172.21.15.72:0/3872161569 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6b60003970 con 0x7f6b6c1011e0 2024-03-22T01:27:38.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:38.557+0000 7f6b72a8f700 1 -- 172.21.15.72:0/3872161569 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b6c199210 con 0x7f6b6c1011e0 2024-03-22T01:27:38.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:38.557+0000 7f6b72a8f700 1 -- 172.21.15.72:0/3872161569 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b6c1996d0 con 0x7f6b6c1011e0 2024-03-22T01:27:38.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:38.558+0000 7f6b69ffb700 1 -- 172.21.15.72:0/3872161569 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 13) ==== 46350+0+0 (secure 0 0 0) 0x7f6b60029070 con 0x7f6b6c1011e0 2024-03-22T01:27:38.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:38.558+0000 7f6b72a8f700 1 -- 172.21.15.72:0/3872161569 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b6c1026c0 con 0x7f6b6c1011e0 2024-03-22T01:27:38.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:38.558+0000 7f6b69ffb700 1 --2- 172.21.15.72:0/3872161569 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f6b5c039280 0x7f6b5c03b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:27:38.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:38.558+0000 7f6b69ffb700 1 -- 172.21.15.72:0/3872161569 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f6b6001e070 con 0x7f6b6c1011e0 2024-03-22T01:27:38.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:38.558+0000 7f6b6bfff700 1 --2- 172.21.15.72:0/3872161569 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f6b5c039280 0x7f6b5c03b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:27:38.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:38.559+0000 7f6b6bfff700 1 --2- 172.21.15.72:0/3872161569 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f6b5c039280 0x7f6b5c03b740 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f6b58006fd0 tx=0x7f6b58006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:27:38.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:38.563+0000 7f6b69ffb700 1 -- 172.21.15.72:0/3872161569 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6b60013070 con 0x7f6b6c1011e0 2024-03-22T01:27:38.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:38 smithi072 ceph-mon[32414]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:27:38.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:38.765+0000 7f6b72a8f700 1 -- 172.21.15.72:0/3872161569 --> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6b6c06aad0 con 0x7f6b5c039280 2024-03-22T01:27:38.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:38.766+0000 7f6b69ffb700 1 -- 172.21.15.72:0/3872161569 <== mgr.14152 v2:172.21.15.72:6800/401403200 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+159 (secure 0 0 0) 0x7f6b6c06aad0 con 0x7f6b5c039280 2024-03-22T01:27:38.765 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:27:38.765 INFO:teuthology.orchestra.run.smithi072.stdout:[{"addr": "172.21.15.72", "hostname": "smithi072", "labels": [], "status": ""}, {"addr": "172.21.15.89", "hostname": "smithi089", "labels": [], "status": ""}] 2024-03-22T01:27:38.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:38.769+0000 7f6b72a8f700 1 -- 172.21.15.72:0/3872161569 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f6b5c039280 msgr2=0x7f6b5c03b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:38.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:38.769+0000 7f6b72a8f700 1 --2- 172.21.15.72:0/3872161569 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f6b5c039280 0x7f6b5c03b740 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f6b58006fd0 tx=0x7f6b58006e40 comp rx=0 tx=0).stop 2024-03-22T01:27:38.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:38.769+0000 7f6b72a8f700 1 -- 172.21.15.72:0/3872161569 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b6c1011e0 msgr2=0x7f6b6c1989e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:38.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:38.769+0000 7f6b72a8f700 1 --2- 172.21.15.72:0/3872161569 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b6c1011e0 0x7f6b6c1989e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6b60005850 tx=0x7f6b60005030 comp rx=0 tx=0).stop 2024-03-22T01:27:38.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:38.769+0000 7f6b72a8f700 1 -- 172.21.15.72:0/3872161569 shutdown_connections 2024-03-22T01:27:38.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:38.769+0000 7f6b72a8f700 1 --2- 172.21.15.72:0/3872161569 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f6b5c039280 0x7f6b5c03b740 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:38.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:38.769+0000 7f6b72a8f700 1 --2- 172.21.15.72:0/3872161569 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b6c1011e0 0x7f6b6c1989e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:38.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:38.769+0000 7f6b72a8f700 1 -- 172.21.15.72:0/3872161569 >> 172.21.15.72:0/3872161569 conn(0x7f6b6c0fcad0 msgr2=0x7f6b6c102bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:27:38.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:38.769+0000 7f6b72a8f700 1 -- 172.21.15.72:0/3872161569 shutdown_connections 2024-03-22T01:27:38.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:38.769+0000 7f6b72a8f700 1 -- 172.21.15.72:0/3872161569 wait complete. 2024-03-22T01:27:39.335 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi098 2024-03-22T01:27:39.335 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-22T01:27:39.335 DEBUG:teuthology.orchestra.run.smithi098:> dd of=/etc/ceph/ceph.conf 2024-03-22T01:27:39.355 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-22T01:27:39.355 DEBUG:teuthology.orchestra.run.smithi098:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-22T01:27:39.413 INFO:tasks.cephadm:Adding host smithi098 to orchestrator... 2024-03-22T01:27:39.413 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph orch host add smithi098 2024-03-22T01:27:39.657 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:27:40.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:39 smithi072 ceph-mon[32414]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-22T01:27:40.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:39 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:27:40.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:39 smithi072 ceph-mon[32414]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:27:40.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:41.000+0000 7f4c41ad2700 1 -- 172.21.15.72:0/1943802380 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c3c1032c0 msgr2=0x7f4c3c1036a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:40.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:41.000+0000 7f4c41ad2700 1 --2- 172.21.15.72:0/1943802380 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c3c1032c0 0x7f4c3c1036a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c009910 tx=0x7f4c2c009c20 comp rx=0 tx=0).stop 2024-03-22T01:27:40.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:41.000+0000 7f4c41ad2700 1 -- 172.21.15.72:0/1943802380 shutdown_connections 2024-03-22T01:27:41.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:41.000+0000 7f4c41ad2700 1 --2- 172.21.15.72:0/1943802380 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c3c1032c0 0x7f4c3c1036a0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:41.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:41.000+0000 7f4c41ad2700 1 -- 172.21.15.72:0/1943802380 >> 172.21.15.72:0/1943802380 conn(0x7f4c3c0fca70 msgr2=0x7f4c3c0fee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:27:41.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:41.000+0000 7f4c41ad2700 1 -- 172.21.15.72:0/1943802380 shutdown_connections 2024-03-22T01:27:41.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:41.000+0000 7f4c41ad2700 1 -- 172.21.15.72:0/1943802380 wait complete. 2024-03-22T01:27:41.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:41.000+0000 7f4c41ad2700 1 Processor -- start 2024-03-22T01:27:41.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:41.001+0000 7f4c41ad2700 1 -- start start 2024-03-22T01:27:41.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:41.001+0000 7f4c41ad2700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c3c1032c0 0x7f4c3c19d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:27:41.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:41.001+0000 7f4c41ad2700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f4c3c10c710 con 0x7f4c3c1032c0 2024-03-22T01:27:41.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:41.001+0000 7f4c3b7fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c3c1032c0 0x7f4c3c19d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:27:41.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:41.001+0000 7f4c3b7fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c3c1032c0 0x7f4c3c19d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38200/0 (socket says 172.21.15.72:38200) 2024-03-22T01:27:41.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:41.001+0000 7f4c3b7fe700 1 -- 172.21.15.72:0/144566864 learned_addr learned my addr 172.21.15.72:0/144566864 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:27:41.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:41.001+0000 7f4c3b7fe700 1 -- 172.21.15.72:0/144566864 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c3c19ddf0 con 0x7f4c3c1032c0 2024-03-22T01:27:41.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:41.001+0000 7f4c3b7fe700 1 --2- 172.21.15.72:0/144566864 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c3c1032c0 0x7f4c3c19d8b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c006010 tx=0x7f4c2c005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:27:41.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:41.002+0000 7f4c38ff9700 1 -- 172.21.15.72:0/144566864 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4c2c026070 con 0x7f4c3c1032c0 2024-03-22T01:27:41.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:41.002+0000 7f4c38ff9700 1 -- 172.21.15.72:0/144566864 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c2c00bc30 con 0x7f4c3c1032c0 2024-03-22T01:27:41.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:41.002+0000 7f4c38ff9700 1 -- 172.21.15.72:0/144566864 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4c2c003ab0 con 0x7f4c3c1032c0 2024-03-22T01:27:41.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:41.002+0000 7f4c41ad2700 1 -- 172.21.15.72:0/144566864 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c3c19e080 con 0x7f4c3c1032c0 2024-03-22T01:27:41.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:41.002+0000 7f4c41ad2700 1 -- 172.21.15.72:0/144566864 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c3c197a70 con 0x7f4c3c1032c0 2024-03-22T01:27:41.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:41.002+0000 7f4c38ff9700 1 -- 172.21.15.72:0/144566864 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 13) ==== 46350+0+0 (secure 0 0 0) 0x7f4c2c029070 con 0x7f4c3c1032c0 2024-03-22T01:27:41.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:41.003+0000 7f4c38ff9700 1 --2- 172.21.15.72:0/144566864 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f4c24039280 0x7f4c2403b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:27:41.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:41.003+0000 7f4c38ff9700 1 -- 172.21.15.72:0/144566864 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f4c2c01e070 con 0x7f4c3c1032c0 2024-03-22T01:27:41.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:41.003+0000 7f4c3affd700 1 --2- 172.21.15.72:0/144566864 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f4c24039280 0x7f4c2403b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:27:41.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:41.003+0000 7f4c41ad2700 1 -- 172.21.15.72:0/144566864 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c3c06ad20 con 0x7f4c3c1032c0 2024-03-22T01:27:41.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:41.003+0000 7f4c3affd700 1 --2- 172.21.15.72:0/144566864 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f4c24039280 0x7f4c2403b740 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4c30006fd0 tx=0x7f4c30006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:27:41.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:41.008+0000 7f4c38ff9700 1 -- 172.21.15.72:0/144566864 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4c2c030020 con 0x7f4c3c1032c0 2024-03-22T01:27:41.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:41.209+0000 7f4c41ad2700 1 -- 172.21.15.72:0/144566864 --> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi098", "target": ["mon-mgr", ""]}) -- 0x7f4c3c063cb0 con 0x7f4c24039280 2024-03-22T01:27:42.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:42 smithi072 ceph-mon[32414]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi098", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:27:42.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:42 smithi072 ceph-mon[32414]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:27:42.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:42 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:27:42.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:42 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:27:42.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:42 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:27:42.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:42 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:27:42.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:42 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi089", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:27:42.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:42 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:27:42.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:42 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:27:43.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:43 smithi072 ceph-mon[32414]: Deploying cephadm binary to smithi098 2024-03-22T01:27:43.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:43 smithi072 ceph-mon[32414]: Updating smithi089:/etc/ceph/ceph.conf 2024-03-22T01:27:43.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:43 smithi072 ceph-mon[32414]: Updating smithi089:/var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T01:27:44.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:44 smithi072 ceph-mon[32414]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:27:45.307 INFO:teuthology.orchestra.run.smithi072.stdout:Added host 'smithi098' with addr '172.21.15.98' 2024-03-22T01:27:45.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:45.308+0000 7f4c38ff9700 1 -- 172.21.15.72:0/144566864 <== mgr.14152 v2:172.21.15.72:6800/401403200 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f4c3c063cb0 con 0x7f4c24039280 2024-03-22T01:27:45.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:45.310+0000 7f4c41ad2700 1 -- 172.21.15.72:0/144566864 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f4c24039280 msgr2=0x7f4c2403b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:45.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:45.310+0000 7f4c41ad2700 1 --2- 172.21.15.72:0/144566864 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f4c24039280 0x7f4c2403b740 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4c30006fd0 tx=0x7f4c30006e40 comp rx=0 tx=0).stop 2024-03-22T01:27:45.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:45.310+0000 7f4c41ad2700 1 -- 172.21.15.72:0/144566864 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c3c1032c0 msgr2=0x7f4c3c19d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:45.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:45.310+0000 7f4c41ad2700 1 --2- 172.21.15.72:0/144566864 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c3c1032c0 0x7f4c3c19d8b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c006010 tx=0x7f4c2c005050 comp rx=0 tx=0).stop 2024-03-22T01:27:45.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:45.310+0000 7f4c41ad2700 1 -- 172.21.15.72:0/144566864 shutdown_connections 2024-03-22T01:27:45.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:45.310+0000 7f4c41ad2700 1 --2- 172.21.15.72:0/144566864 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f4c24039280 0x7f4c2403b740 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:45.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:45.310+0000 7f4c41ad2700 1 --2- 172.21.15.72:0/144566864 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c3c1032c0 0x7f4c3c19d8b0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:45.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:45.310+0000 7f4c41ad2700 1 -- 172.21.15.72:0/144566864 >> 172.21.15.72:0/144566864 conn(0x7f4c3c0fca70 msgr2=0x7f4c3c104070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:27:45.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:45.310+0000 7f4c41ad2700 1 -- 172.21.15.72:0/144566864 shutdown_connections 2024-03-22T01:27:45.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:45.310+0000 7f4c41ad2700 1 -- 172.21.15.72:0/144566864 wait complete. 2024-03-22T01:27:46.040 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph orch host ls --format=json 2024-03-22T01:27:46.276 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:27:46.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:46 smithi072 ceph-mon[32414]: Updating smithi089:/etc/ceph/ceph.client.admin.keyring 2024-03-22T01:27:46.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:46 smithi072 ceph-mon[32414]: Updating smithi089:/var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T01:27:46.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:46 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:27:46.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:46 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:27:46.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:46 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:27:46.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:46 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:27:46.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:46 smithi072 ceph-mon[32414]: Added host smithi098 2024-03-22T01:27:46.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:46 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:27:46.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:46 smithi072 ceph-mon[32414]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:27:46.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:46 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:27:47.619 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:47 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:27:47.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:47.645+0000 7f51d8a9d700 1 -- 172.21.15.72:0/3109547481 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51d41011b0 msgr2=0x7f51d4101590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:47.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:47.645+0000 7f51d8a9d700 1 --2- 172.21.15.72:0/3109547481 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51d41011b0 0x7f51d4101590 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f51cc00b0a0 tx=0x7f51cc00b3b0 comp rx=0 tx=0).stop 2024-03-22T01:27:47.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:47.645+0000 7f51d8a9d700 1 -- 172.21.15.72:0/3109547481 shutdown_connections 2024-03-22T01:27:47.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:47.645+0000 7f51d8a9d700 1 --2- 172.21.15.72:0/3109547481 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51d41011b0 0x7f51d4101590 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:47.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:47.645+0000 7f51d8a9d700 1 -- 172.21.15.72:0/3109547481 >> 172.21.15.72:0/3109547481 conn(0x7f51d4077bc0 msgr2=0x7f51d40fee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:27:47.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:47.645+0000 7f51d8a9d700 1 -- 172.21.15.72:0/3109547481 shutdown_connections 2024-03-22T01:27:47.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:47.645+0000 7f51d8a9d700 1 -- 172.21.15.72:0/3109547481 wait complete. 2024-03-22T01:27:47.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:47.646+0000 7f51d8a9d700 1 Processor -- start 2024-03-22T01:27:47.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:47.646+0000 7f51d8a9d700 1 -- start start 2024-03-22T01:27:47.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:47.646+0000 7f51d8a9d700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51d41011b0 0x7f51d419b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:27:47.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:47.646+0000 7f51d8a9d700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f51d410a470 con 0x7f51d41011b0 2024-03-22T01:27:47.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:47.646+0000 7f51d259c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51d41011b0 0x7f51d419b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:27:47.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:47.646+0000 7f51d259c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51d41011b0 0x7f51d419b640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38212/0 (socket says 172.21.15.72:38212) 2024-03-22T01:27:47.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:47.646+0000 7f51d259c700 1 -- 172.21.15.72:0/465824392 learned_addr learned my addr 172.21.15.72:0/465824392 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:27:47.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:47.646+0000 7f51d259c700 1 -- 172.21.15.72:0/465824392 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51d419bb80 con 0x7f51d41011b0 2024-03-22T01:27:47.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:47.646+0000 7f51d259c700 1 --2- 172.21.15.72:0/465824392 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51d41011b0 0x7f51d419b640 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f51cc015040 tx=0x7f51cc003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:27:47.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:47.647+0000 7f51c37fe700 1 -- 172.21.15.72:0/465824392 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f51cc017040 con 0x7f51d41011b0 2024-03-22T01:27:47.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:47.647+0000 7f51c37fe700 1 -- 172.21.15.72:0/465824392 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51cc022070 con 0x7f51d41011b0 2024-03-22T01:27:47.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:47.647+0000 7f51c37fe700 1 -- 172.21.15.72:0/465824392 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f51cc01d5d0 con 0x7f51d41011b0 2024-03-22T01:27:47.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:47.647+0000 7f51d8a9d700 1 -- 172.21.15.72:0/465824392 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51d419be10 con 0x7f51d41011b0 2024-03-22T01:27:47.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:47.647+0000 7f51d8a9d700 1 -- 172.21.15.72:0/465824392 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51d4195710 con 0x7f51d41011b0 2024-03-22T01:27:47.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:47.647+0000 7f51c37fe700 1 -- 172.21.15.72:0/465824392 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 13) ==== 46350+0+0 (secure 0 0 0) 0x7f51cc01d8d0 con 0x7f51d41011b0 2024-03-22T01:27:47.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:47.648+0000 7f51d8a9d700 1 -- 172.21.15.72:0/465824392 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51d4102690 con 0x7f51d41011b0 2024-03-22T01:27:47.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:47.648+0000 7f51c37fe700 1 --2- 172.21.15.72:0/465824392 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f51bc03b440 0x7f51bc03d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:27:47.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:47.648+0000 7f51c37fe700 1 -- 172.21.15.72:0/465824392 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f51cc00f050 con 0x7f51d41011b0 2024-03-22T01:27:47.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:47.648+0000 7f51d1d9b700 1 --2- 172.21.15.72:0/465824392 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f51bc03b440 0x7f51bc03d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:27:47.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:47.648+0000 7f51d1d9b700 1 --2- 172.21.15.72:0/465824392 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f51bc03b440 0x7f51bc03d900 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f51c8006fd0 tx=0x7f51c8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:27:47.652 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:47.653+0000 7f51c37fe700 1 -- 172.21.15.72:0/465824392 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f51cc007e20 con 0x7f51d41011b0 2024-03-22T01:27:47.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:47.851+0000 7f51d8a9d700 1 -- 172.21.15.72:0/465824392 --> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f51d4196c40 con 0x7f51bc03b440 2024-03-22T01:27:47.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:47.852+0000 7f51c37fe700 1 -- 172.21.15.72:0/465824392 <== mgr.14152 v2:172.21.15.72:6800/401403200 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+238 (secure 0 0 0) 0x7f51d4196c40 con 0x7f51bc03b440 2024-03-22T01:27:47.852 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:27:47.852 INFO:teuthology.orchestra.run.smithi072.stdout:[{"addr": "172.21.15.72", "hostname": "smithi072", "labels": [], "status": ""}, {"addr": "172.21.15.89", "hostname": "smithi089", "labels": [], "status": ""}, {"addr": "172.21.15.98", "hostname": "smithi098", "labels": [], "status": ""}] 2024-03-22T01:27:47.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:47.854+0000 7f51d8a9d700 1 -- 172.21.15.72:0/465824392 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f51bc03b440 msgr2=0x7f51bc03d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:47.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:47.854+0000 7f51d8a9d700 1 --2- 172.21.15.72:0/465824392 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f51bc03b440 0x7f51bc03d900 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f51c8006fd0 tx=0x7f51c8006e40 comp rx=0 tx=0).stop 2024-03-22T01:27:47.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:47.854+0000 7f51d8a9d700 1 -- 172.21.15.72:0/465824392 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51d41011b0 msgr2=0x7f51d419b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:47.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:47.854+0000 7f51d8a9d700 1 --2- 172.21.15.72:0/465824392 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51d41011b0 0x7f51d419b640 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f51cc015040 tx=0x7f51cc003d50 comp rx=0 tx=0).stop 2024-03-22T01:27:47.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:47.854+0000 7f51d8a9d700 1 -- 172.21.15.72:0/465824392 shutdown_connections 2024-03-22T01:27:47.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:47.854+0000 7f51d8a9d700 1 --2- 172.21.15.72:0/465824392 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f51bc03b440 0x7f51bc03d900 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:47.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:47.854+0000 7f51d8a9d700 1 --2- 172.21.15.72:0/465824392 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51d41011b0 0x7f51d419b640 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:47.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:47.854+0000 7f51d8a9d700 1 -- 172.21.15.72:0/465824392 >> 172.21.15.72:0/465824392 conn(0x7f51d4077bc0 msgr2=0x7f51d4102b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:27:47.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:47.854+0000 7f51d8a9d700 1 -- 172.21.15.72:0/465824392 shutdown_connections 2024-03-22T01:27:47.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:47.854+0000 7f51d8a9d700 1 -- 172.21.15.72:0/465824392 wait complete. 2024-03-22T01:27:48.444 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:48 smithi072 ceph-mon[32414]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:27:48.444 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:48 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:27:48.472 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-22T01:27:48.472 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph osd crush tunables default 2024-03-22T01:27:48.705 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:27:49.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:49 smithi072 ceph-mon[32414]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-22T01:27:50.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:50.134+0000 7fbc4b6b9700 1 -- 172.21.15.72:0/1811028376 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc441022c0 msgr2=0x7fbc441026a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:50.140 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:50.134+0000 7fbc4b6b9700 1 --2- 172.21.15.72:0/1811028376 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc441022c0 0x7fbc441026a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fbc38009910 tx=0x7fbc38009c20 comp rx=0 tx=0).stop 2024-03-22T01:27:50.140 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:50.134+0000 7fbc4b6b9700 1 -- 172.21.15.72:0/1811028376 shutdown_connections 2024-03-22T01:27:50.140 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:50.134+0000 7fbc4b6b9700 1 --2- 172.21.15.72:0/1811028376 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc441022c0 0x7fbc441026a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:50.140 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:50.134+0000 7fbc4b6b9700 1 -- 172.21.15.72:0/1811028376 >> 172.21.15.72:0/1811028376 conn(0x7fbc440fdb90 msgr2=0x7fbc440fffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:27:50.140 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:50.135+0000 7fbc4b6b9700 1 -- 172.21.15.72:0/1811028376 shutdown_connections 2024-03-22T01:27:50.140 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:50.135+0000 7fbc4b6b9700 1 -- 172.21.15.72:0/1811028376 wait complete. 2024-03-22T01:27:50.140 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:50.135+0000 7fbc4b6b9700 1 Processor -- start 2024-03-22T01:27:50.140 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:50.135+0000 7fbc4b6b9700 1 -- start start 2024-03-22T01:27:50.140 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:50.135+0000 7fbc4b6b9700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc441022c0 0x7fbc4419d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:27:50.140 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:50.135+0000 7fbc4b6b9700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fbc4410c690 con 0x7fbc441022c0 2024-03-22T01:27:50.140 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:50.135+0000 7fbc49455700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc441022c0 0x7fbc4419d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:27:50.140 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:50.135+0000 7fbc49455700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc441022c0 0x7fbc4419d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38224/0 (socket says 172.21.15.72:38224) 2024-03-22T01:27:50.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:50.135+0000 7fbc49455700 1 -- 172.21.15.72:0/2186558934 learned_addr learned my addr 172.21.15.72:0/2186558934 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:27:50.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:50.136+0000 7fbc49455700 1 -- 172.21.15.72:0/2186558934 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc4419dde0 con 0x7fbc441022c0 2024-03-22T01:27:50.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:50.136+0000 7fbc49455700 1 --2- 172.21.15.72:0/2186558934 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc441022c0 0x7fbc4419d8a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fbc38005850 tx=0x7fbc38005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:27:50.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:50.136+0000 7fbc367fc700 1 -- 172.21.15.72:0/2186558934 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbc38026070 con 0x7fbc441022c0 2024-03-22T01:27:50.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:50.136+0000 7fbc367fc700 1 -- 172.21.15.72:0/2186558934 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc3800bbd0 con 0x7fbc441022c0 2024-03-22T01:27:50.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:50.136+0000 7fbc4b6b9700 1 -- 172.21.15.72:0/2186558934 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc4419e0d0 con 0x7fbc441022c0 2024-03-22T01:27:50.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:50.136+0000 7fbc367fc700 1 -- 172.21.15.72:0/2186558934 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbc38003970 con 0x7fbc441022c0 2024-03-22T01:27:50.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:50.136+0000 7fbc4b6b9700 1 -- 172.21.15.72:0/2186558934 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc44197a60 con 0x7fbc441022c0 2024-03-22T01:27:50.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:50.137+0000 7fbc367fc700 1 -- 172.21.15.72:0/2186558934 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 13) ==== 46350+0+0 (secure 0 0 0) 0x7fbc38029070 con 0x7fbc441022c0 2024-03-22T01:27:50.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:50.137+0000 7fbc4b6b9700 1 -- 172.21.15.72:0/2186558934 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc441037a0 con 0x7fbc441022c0 2024-03-22T01:27:50.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:50.137+0000 7fbc367fc700 1 --2- 172.21.15.72:0/2186558934 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fbc30039280 0x7fbc3003b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:27:50.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:50.137+0000 7fbc367fc700 1 -- 172.21.15.72:0/2186558934 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fbc3801e070 con 0x7fbc441022c0 2024-03-22T01:27:50.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:50.137+0000 7fbc48c54700 1 --2- 172.21.15.72:0/2186558934 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fbc30039280 0x7fbc3003b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:27:50.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:50.137+0000 7fbc48c54700 1 --2- 172.21.15.72:0/2186558934 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fbc30039280 0x7fbc3003b740 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fbc40006fd0 tx=0x7fbc40006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:27:50.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:50.142+0000 7fbc367fc700 1 -- 172.21.15.72:0/2186558934 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbc38013070 con 0x7fbc441022c0 2024-03-22T01:27:50.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:50.342+0000 7fbc4b6b9700 1 -- 172.21.15.72:0/2186558934 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fbc44198f90 con 0x7fbc441022c0 2024-03-22T01:27:50.620 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:50 smithi072 ceph-mon[32414]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:27:51.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:51.332+0000 7fbc367fc700 1 -- 172.21.15.72:0/2186558934 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7fbc38030090 con 0x7fbc441022c0 2024-03-22T01:27:51.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:51.335+0000 7fbc4b6b9700 1 -- 172.21.15.72:0/2186558934 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fbc30039280 msgr2=0x7fbc3003b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:51.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:51.335+0000 7fbc4b6b9700 1 --2- 172.21.15.72:0/2186558934 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fbc30039280 0x7fbc3003b740 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fbc40006fd0 tx=0x7fbc40006e40 comp rx=0 tx=0).stop 2024-03-22T01:27:51.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:51.335+0000 7fbc4b6b9700 1 -- 172.21.15.72:0/2186558934 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc441022c0 msgr2=0x7fbc4419d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:51.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:51.335+0000 7fbc4b6b9700 1 --2- 172.21.15.72:0/2186558934 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc441022c0 0x7fbc4419d8a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fbc38005850 tx=0x7fbc38005030 comp rx=0 tx=0).stop 2024-03-22T01:27:51.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:51.335+0000 7fbc4b6b9700 1 -- 172.21.15.72:0/2186558934 shutdown_connections 2024-03-22T01:27:51.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:51.335+0000 7fbc4b6b9700 1 --2- 172.21.15.72:0/2186558934 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fbc30039280 0x7fbc3003b740 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:51.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:51.335+0000 7fbc4b6b9700 1 --2- 172.21.15.72:0/2186558934 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc441022c0 0x7fbc4419d8a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:51.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:51.335+0000 7fbc4b6b9700 1 -- 172.21.15.72:0/2186558934 >> 172.21.15.72:0/2186558934 conn(0x7fbc440fdb90 msgr2=0x7fbc441050f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:27:51.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:51.335+0000 7fbc4b6b9700 1 -- 172.21.15.72:0/2186558934 shutdown_connections 2024-03-22T01:27:51.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:27:51.335+0000 7fbc4b6b9700 1 -- 172.21.15.72:0/2186558934 wait complete. 2024-03-22T01:27:51.336 INFO:teuthology.orchestra.run.smithi072.stderr:adjusted tunables profile to default 2024-03-22T01:27:51.621 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:51 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/2186558934' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-22T01:27:51.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:51 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:27:51.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:51 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:27:51.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:51 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:27:51.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:51 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:27:51.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:51 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi098", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:27:51.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:51 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:27:51.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:51 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:27:51.961 INFO:tasks.cephadm:Adding mon.a on smithi072 2024-03-22T01:27:51.961 INFO:tasks.cephadm:Adding mon.b on smithi089 2024-03-22T01:27:51.961 INFO:tasks.cephadm:Adding mon.c on smithi098 2024-03-22T01:27:51.961 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph orch apply mon '3;smithi072:172.21.15.72=a;smithi089:172.21.15.89=b;smithi098:172.21.15.98=c' 2024-03-22T01:27:52.192 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T01:27:52.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:52 smithi072 ceph-mon[32414]: Updating smithi098:/etc/ceph/ceph.conf 2024-03-22T01:27:52.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:52 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/2186558934' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-22T01:27:52.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:52 smithi072 ceph-mon[32414]: osdmap e4: 0 total, 0 up, 0 in 2024-03-22T01:27:52.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:52 smithi072 ceph-mon[32414]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:27:52.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:52 smithi072 ceph-mon[32414]: Updating smithi098:/var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T01:27:52.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:52 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:27:52.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:52 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:27:52.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:52 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:27:53.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:53 smithi072 ceph-mon[32414]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-03-22T01:27:53.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:53 smithi072 ceph-mon[32414]: Updating smithi098:/var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T01:27:53.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:53.630+0000 7fb6473a0700 1 -- 172.21.15.98:0/712800851 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb6401022b0 msgr2=0x7fb640102690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:53.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:53.630+0000 7fb6473a0700 1 --2- 172.21.15.98:0/712800851 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb6401022b0 0x7fb640102690 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb630009910 tx=0x7fb630009c20 comp rx=0 tx=0).stop 2024-03-22T01:27:53.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:53.630+0000 7fb6473a0700 1 -- 172.21.15.98:0/712800851 shutdown_connections 2024-03-22T01:27:53.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:53.630+0000 7fb6473a0700 1 --2- 172.21.15.98:0/712800851 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb6401022b0 0x7fb640102690 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:53.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:53.630+0000 7fb6473a0700 1 -- 172.21.15.98:0/712800851 >> 172.21.15.98:0/712800851 conn(0x7fb6400fdbc0 msgr2=0x7fb6400fffe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:27:53.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:53.630+0000 7fb6473a0700 1 -- 172.21.15.98:0/712800851 shutdown_connections 2024-03-22T01:27:53.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:53.630+0000 7fb6473a0700 1 -- 172.21.15.98:0/712800851 wait complete. 2024-03-22T01:27:53.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:53.630+0000 7fb6473a0700 1 Processor -- start 2024-03-22T01:27:53.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:53.630+0000 7fb6473a0700 1 -- start start 2024-03-22T01:27:53.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:53.631+0000 7fb6473a0700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb6401022b0 0x7fb64019d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:27:53.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:53.631+0000 7fb6473a0700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fb64010c680 con 0x7fb6401022b0 2024-03-22T01:27:53.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:53.631+0000 7fb64513c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb6401022b0 0x7fb64019d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:27:53.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:53.631+0000 7fb64513c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb6401022b0 0x7fb64019d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.98:42902/0 (socket says 172.21.15.98:42902) 2024-03-22T01:27:53.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:53.631+0000 7fb64513c700 1 -- 172.21.15.98:0/2511564234 learned_addr learned my addr 172.21.15.98:0/2511564234 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-22T01:27:53.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:53.631+0000 7fb64513c700 1 -- 172.21.15.98:0/2511564234 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb64019dd30 con 0x7fb6401022b0 2024-03-22T01:27:53.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:53.631+0000 7fb64513c700 1 --2- 172.21.15.98:0/2511564234 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb6401022b0 0x7fb64019d7f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb630005850 tx=0x7fb630005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:27:53.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:53.632+0000 7fb6367fc700 1 -- 172.21.15.98:0/2511564234 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb630026070 con 0x7fb6401022b0 2024-03-22T01:27:53.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:53.632+0000 7fb6473a0700 1 -- 172.21.15.98:0/2511564234 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb64019e020 con 0x7fb6401022b0 2024-03-22T01:27:53.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:53.632+0000 7fb6367fc700 1 -- 172.21.15.98:0/2511564234 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb63000bbd0 con 0x7fb6401022b0 2024-03-22T01:27:53.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:53.632+0000 7fb6367fc700 1 -- 172.21.15.98:0/2511564234 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb630003970 con 0x7fb6401022b0 2024-03-22T01:27:53.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:53.632+0000 7fb6473a0700 1 -- 172.21.15.98:0/2511564234 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6401979b0 con 0x7fb6401022b0 2024-03-22T01:27:53.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:53.632+0000 7fb6367fc700 1 -- 172.21.15.98:0/2511564234 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 13) ==== 46350+0+0 (secure 0 0 0) 0x7fb630029070 con 0x7fb6401022b0 2024-03-22T01:27:53.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:53.632+0000 7fb6473a0700 1 -- 172.21.15.98:0/2511564234 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb640103790 con 0x7fb6401022b0 2024-03-22T01:27:53.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:53.633+0000 7fb6367fc700 1 --2- 172.21.15.98:0/2511564234 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fb62c039220 0x7fb62c03b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:27:53.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:53.633+0000 7fb6367fc700 1 -- 172.21.15.98:0/2511564234 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fb63001e070 con 0x7fb6401022b0 2024-03-22T01:27:53.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:53.633+0000 7fb64493b700 1 --2- 172.21.15.98:0/2511564234 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fb62c039220 0x7fb62c03b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:27:53.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:53.633+0000 7fb64493b700 1 --2- 172.21.15.98:0/2511564234 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fb62c039220 0x7fb62c03b6e0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb63c006fd0 tx=0x7fb63c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:27:53.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:53.638+0000 7fb6367fc700 1 -- 172.21.15.98:0/2511564234 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb630013070 con 0x7fb6401022b0 2024-03-22T01:27:53.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:53.850+0000 7fb6473a0700 1 -- 172.21.15.98:0/2511564234 --> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi072:172.21.15.72=a;smithi089:172.21.15.89=b;smithi098:172.21.15.98=c", "target": ["mon-mgr", ""]}) -- 0x7fb640198f50 con 0x7fb62c039220 2024-03-22T01:27:53.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:53.857+0000 7fb6367fc700 1 -- 172.21.15.98:0/2511564234 <== mgr.14152 v2:172.21.15.72:6800/401403200 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fb640198f50 con 0x7fb62c039220 2024-03-22T01:27:53.856 INFO:teuthology.orchestra.run.smithi098.stdout:Scheduled mon update... 2024-03-22T01:27:53.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:53.859+0000 7fb6473a0700 1 -- 172.21.15.98:0/2511564234 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fb62c039220 msgr2=0x7fb62c03b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:53.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:53.859+0000 7fb6473a0700 1 --2- 172.21.15.98:0/2511564234 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fb62c039220 0x7fb62c03b6e0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb63c006fd0 tx=0x7fb63c006e40 comp rx=0 tx=0).stop 2024-03-22T01:27:53.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:53.859+0000 7fb6473a0700 1 -- 172.21.15.98:0/2511564234 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb6401022b0 msgr2=0x7fb64019d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:53.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:53.859+0000 7fb6473a0700 1 --2- 172.21.15.98:0/2511564234 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb6401022b0 0x7fb64019d7f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb630005850 tx=0x7fb630005030 comp rx=0 tx=0).stop 2024-03-22T01:27:53.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:53.859+0000 7fb6473a0700 1 -- 172.21.15.98:0/2511564234 shutdown_connections 2024-03-22T01:27:53.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:53.859+0000 7fb6473a0700 1 --2- 172.21.15.98:0/2511564234 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fb62c039220 0x7fb62c03b6e0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:53.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:53.859+0000 7fb6473a0700 1 --2- 172.21.15.98:0/2511564234 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb6401022b0 0x7fb64019d7f0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:53.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:53.859+0000 7fb6473a0700 1 -- 172.21.15.98:0/2511564234 >> 172.21.15.98:0/2511564234 conn(0x7fb6400fdbc0 msgr2=0x7fb640103c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:27:53.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:53.859+0000 7fb6473a0700 1 -- 172.21.15.98:0/2511564234 shutdown_connections 2024-03-22T01:27:53.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:53.859+0000 7fb6473a0700 1 -- 172.21.15.98:0/2511564234 wait complete. 2024-03-22T01:27:54.442 DEBUG:teuthology.orchestra.run.smithi089:mon.b> sudo journalctl -f -n 0 -u ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mon.b.service 2024-03-22T01:27:54.444 DEBUG:teuthology.orchestra.run.smithi098:mon.c> sudo journalctl -f -n 0 -u ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mon.c.service 2024-03-22T01:27:54.446 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-22T01:27:54.447 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-22T01:27:54.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:54 smithi072 ceph-mon[32414]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:27:54.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:54 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:27:54.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:54 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:27:54.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:54 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:27:54.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:54 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:27:54.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:54 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:27:54.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:54 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T01:27:54.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:54 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:27:55.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:55 smithi072 ceph-mon[32414]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi072:172.21.15.72=a;smithi089:172.21.15.89=b;smithi098:172.21.15.98=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:27:55.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:55 smithi072 ceph-mon[32414]: Saving service mon spec with placement smithi072:172.21.15.72=a;smithi089:172.21.15.89=b;smithi098:172.21.15.98=c;count:3 2024-03-22T01:27:55.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:55 smithi072 ceph-mon[32414]: Deploying daemon mon.c on smithi098 2024-03-22T01:27:56.478 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.c/config 2024-03-22T01:27:56.621 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:56 smithi072 ceph-mon[32414]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:27:58.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:58.192+0000 7fae825fb700 1 -- 172.21.15.98:0/3805470252 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fae7c1012b0 msgr2=0x7fae7c101690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:58.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:58.192+0000 7fae825fb700 1 --2- 172.21.15.98:0/3805470252 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fae7c1012b0 0x7fae7c101690 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fae6c009910 tx=0x7fae6c009c20 comp rx=0 tx=0).stop 2024-03-22T01:27:58.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:58.192+0000 7fae825fb700 1 -- 172.21.15.98:0/3805470252 shutdown_connections 2024-03-22T01:27:58.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:58.192+0000 7fae825fb700 1 --2- 172.21.15.98:0/3805470252 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fae7c1012b0 0x7fae7c101690 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:58.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:58.192+0000 7fae825fb700 1 -- 172.21.15.98:0/3805470252 >> 172.21.15.98:0/3805470252 conn(0x7fae7c0fcb20 msgr2=0x7fae7c0fef40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:27:58.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:58.192+0000 7fae825fb700 1 -- 172.21.15.98:0/3805470252 shutdown_connections 2024-03-22T01:27:58.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:58.192+0000 7fae825fb700 1 -- 172.21.15.98:0/3805470252 wait complete. 2024-03-22T01:27:58.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:58.193+0000 7fae825fb700 1 Processor -- start 2024-03-22T01:27:58.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:58.193+0000 7fae825fb700 1 -- start start 2024-03-22T01:27:58.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:58.193+0000 7fae825fb700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fae7c1012b0 0x7fae7c102d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:27:58.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:58.193+0000 7fae825fb700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fae7c10b680 con 0x7fae7c1012b0 2024-03-22T01:27:58.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:58.193+0000 7fae7bfff700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fae7c1012b0 0x7fae7c102d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:27:58.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:58.193+0000 7fae7bfff700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fae7c1012b0 0x7fae7c102d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.98:42806/0 (socket says 172.21.15.98:42806) 2024-03-22T01:27:58.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:58.193+0000 7fae7bfff700 1 -- 172.21.15.98:0/3765591339 learned_addr learned my addr 172.21.15.98:0/3765591339 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-22T01:27:58.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:58.194+0000 7fae7bfff700 1 -- 172.21.15.98:0/3765591339 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae7c104c40 con 0x7fae7c1012b0 2024-03-22T01:27:58.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:58.194+0000 7fae7bfff700 1 --2- 172.21.15.98:0/3765591339 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fae7c1012b0 0x7fae7c102d50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fae6c005850 tx=0x7fae6c005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:27:58.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:58.194+0000 7fae797fa700 1 -- 172.21.15.98:0/3765591339 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fae6c026070 con 0x7fae7c1012b0 2024-03-22T01:27:58.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:58.194+0000 7fae797fa700 1 -- 172.21.15.98:0/3765591339 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae6c00bc30 con 0x7fae7c1012b0 2024-03-22T01:27:58.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:58.194+0000 7fae825fb700 1 -- 172.21.15.98:0/3765591339 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae7c1032f0 con 0x7fae7c1012b0 2024-03-22T01:27:58.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:58.194+0000 7fae797fa700 1 -- 172.21.15.98:0/3765591339 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fae6c003a20 con 0x7fae7c1012b0 2024-03-22T01:27:58.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:58.194+0000 7fae825fb700 1 -- 172.21.15.98:0/3765591339 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae7c1037b0 con 0x7fae7c1012b0 2024-03-22T01:27:58.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:58.195+0000 7fae797fa700 1 -- 172.21.15.98:0/3765591339 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 13) ==== 46350+0+0 (secure 0 0 0) 0x7fae6c029070 con 0x7fae7c1012b0 2024-03-22T01:27:58.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:58.195+0000 7fae825fb700 1 -- 172.21.15.98:0/3765591339 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae7c102790 con 0x7fae7c1012b0 2024-03-22T01:27:58.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:58.195+0000 7fae797fa700 1 --2- 172.21.15.98:0/3765591339 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fae64039220 0x7fae6403b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:27:58.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:58.195+0000 7fae797fa700 1 -- 172.21.15.98:0/3765591339 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fae6c01e070 con 0x7fae7c1012b0 2024-03-22T01:27:58.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:58.196+0000 7fae7b7fe700 1 --2- 172.21.15.98:0/3765591339 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fae64039220 0x7fae6403b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:27:58.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:58.196+0000 7fae7b7fe700 1 --2- 172.21.15.98:0/3765591339 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fae64039220 0x7fae6403b6e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fae70006fd0 tx=0x7fae70006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:27:58.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:58.201+0000 7fae797fa700 1 -- 172.21.15.98:0/3765591339 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fae6c013070 con 0x7fae7c1012b0 2024-03-22T01:27:58.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:58.487+0000 7fae825fb700 1 -- 172.21.15.98:0/3765591339 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fae7c104320 con 0x7fae7c1012b0 2024-03-22T01:27:58.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:58.488+0000 7fae797fa700 1 -- 172.21.15.98:0/3765591339 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7fae6c030070 con 0x7fae7c1012b0 2024-03-22T01:27:58.486 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-22T01:27:58.486 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":1,"fsid":"022a99bc-e7eb-11ee-95cd-87774f69a715","modified":"2024-03-22T01:26:20.854185Z","created":"2024-03-22T01:26:20.854185Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:3300","nonce":0},{"type":"v1","addr":"172.21.15.72:6789","nonce":0}]},"addr":"172.21.15.72:6789/0","public_addr":"172.21.15.72:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-22T01:27:58.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:58.490+0000 7fae825fb700 1 -- 172.21.15.98:0/3765591339 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fae64039220 msgr2=0x7fae6403b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:58.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:58.490+0000 7fae825fb700 1 --2- 172.21.15.98:0/3765591339 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fae64039220 0x7fae6403b6e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fae70006fd0 tx=0x7fae70006e40 comp rx=0 tx=0).stop 2024-03-22T01:27:58.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:58.490+0000 7fae825fb700 1 -- 172.21.15.98:0/3765591339 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fae7c1012b0 msgr2=0x7fae7c102d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:27:58.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:58.490+0000 7fae825fb700 1 --2- 172.21.15.98:0/3765591339 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fae7c1012b0 0x7fae7c102d50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fae6c005850 tx=0x7fae6c005030 comp rx=0 tx=0).stop 2024-03-22T01:27:58.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:58.490+0000 7fae825fb700 1 -- 172.21.15.98:0/3765591339 shutdown_connections 2024-03-22T01:27:58.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:58.490+0000 7fae825fb700 1 --2- 172.21.15.98:0/3765591339 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fae64039220 0x7fae6403b6e0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:58.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:58.490+0000 7fae825fb700 1 --2- 172.21.15.98:0/3765591339 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fae7c1012b0 0x7fae7c102d50 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:27:58.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:58.490+0000 7fae825fb700 1 -- 172.21.15.98:0/3765591339 >> 172.21.15.98:0/3765591339 conn(0x7fae7c0fcb20 msgr2=0x7fae7c06a860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:27:58.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:58.490+0000 7fae825fb700 1 -- 172.21.15.98:0/3765591339 shutdown_connections 2024-03-22T01:27:58.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:27:58.490+0000 7fae825fb700 1 -- 172.21.15.98:0/3765591339 wait complete. 2024-03-22T01:27:58.489 INFO:teuthology.orchestra.run.smithi098.stderr:dumped monmap epoch 1 2024-03-22T01:27:58.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:27:58 smithi072 ceph-mon[32414]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:00.350 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-22T01:28:00.351 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-22T01:28:00.592 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.c/config 2024-03-22T01:28:04.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:03 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T01:28:04.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:03 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T01:28:04.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:03 smithi072 ceph-mon[32414]: mon.a calling monitor election 2024-03-22T01:28:04.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:03 smithi072 ceph-mon[32414]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:04.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:03 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T01:28:04.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:03 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T01:28:04.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:03 smithi072 ceph-mon[32414]: mon.c calling monitor election 2024-03-22T01:28:04.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:03 smithi072 ceph-mon[32414]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:04.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:03 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T01:28:04.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:03 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T01:28:04.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:03 smithi072 ceph-mon[32414]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:04.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:03 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T01:28:04.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:03 smithi072 ceph-mon[32414]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-22T01:28:04.124 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:03 smithi072 ceph-mon[32414]: monmap epoch 2 2024-03-22T01:28:04.124 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:03 smithi072 ceph-mon[32414]: fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 2024-03-22T01:28:04.124 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:03 smithi072 ceph-mon[32414]: last_changed 2024-03-22T01:27:58.619382+0000 2024-03-22T01:28:04.124 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:03 smithi072 ceph-mon[32414]: created 2024-03-22T01:26:20.854185+0000 2024-03-22T01:28:04.124 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:03 smithi072 ceph-mon[32414]: min_mon_release 19 (squid) 2024-03-22T01:28:04.124 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:03 smithi072 ceph-mon[32414]: election_strategy: 1 2024-03-22T01:28:04.124 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:03 smithi072 ceph-mon[32414]: 0: [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] mon.a 2024-03-22T01:28:04.124 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:03 smithi072 ceph-mon[32414]: 1: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.c 2024-03-22T01:28:04.124 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:03 smithi072 ceph-mon[32414]: fsmap 2024-03-22T01:28:04.124 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:03 smithi072 ceph-mon[32414]: osdmap e4: 0 total, 0 up, 0 in 2024-03-22T01:28:04.124 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:03 smithi072 ceph-mon[32414]: mgrmap e13: x(active, since 48s) 2024-03-22T01:28:04.124 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:03 smithi072 ceph-mon[32414]: overall HEALTH_OK 2024-03-22T01:28:04.124 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:03 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:04.124 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:03 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:04.124 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:03 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:04.125 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:03 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T01:28:04.125 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:03 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:28:04.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:03 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:28:05.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:04 smithi072 ceph-mon[32414]: Deploying daemon mon.b on smithi089 2024-03-22T01:28:05.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:04 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T01:28:05.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:04 smithi098 ceph-mon[36468]: Deploying daemon mon.b on smithi089 2024-03-22T01:28:05.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:04 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T01:28:05.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:05.174+0000 7f807f4de700 1 -- 172.21.15.98:0/854975372 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f80781012a0 msgr2=0x7f8064005610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:28:05.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:05.174+0000 7f807f4de700 1 --2- 172.21.15.98:0/854975372 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f80781012a0 0x7f8064005610 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8074005fa0 tx=0x7f807400ffc0 comp rx=0 tx=0).stop 2024-03-22T01:28:05.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:05.174+0000 7f807f4de700 1 -- 172.21.15.98:0/854975372 shutdown_connections 2024-03-22T01:28:05.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:05.174+0000 7f807f4de700 1 --2- 172.21.15.98:0/854975372 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f80781012a0 0x7f8064005610 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:28:05.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:05.174+0000 7f807f4de700 1 -- 172.21.15.98:0/854975372 >> 172.21.15.98:0/854975372 conn(0x7f80780fcb30 msgr2=0x7f80780fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:28:05.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:05.174+0000 7f807f4de700 1 -- 172.21.15.98:0/854975372 shutdown_connections 2024-03-22T01:28:05.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:05.174+0000 7f807f4de700 1 -- 172.21.15.98:0/854975372 wait complete. 2024-03-22T01:28:05.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:05.175+0000 7f807f4de700 1 Processor -- start 2024-03-22T01:28:05.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:05.175+0000 7f807f4de700 1 -- start start 2024-03-22T01:28:05.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:05.175+0000 7f807f4de700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f80781012a0 0x7f8078198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:28:05.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:05.175+0000 7f807f4de700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8078199010 0x7f8078199490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:28:05.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:05.175+0000 7f807f4de700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f80781090d0 con 0x7f80781012a0 2024-03-22T01:28:05.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:05.175+0000 7f807f4de700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8078108f50 con 0x7f8078199010 2024-03-22T01:28:05.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:05.175+0000 7f807ca79700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8078199010 0x7f8078199490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:28:05.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:05.175+0000 7f807d27a700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f80781012a0 0x7f8078198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:28:05.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:05.175+0000 7f807d27a700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f80781012a0 0x7f8078198ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.98:42874/0 (socket says 172.21.15.98:42874) 2024-03-22T01:28:05.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:05.175+0000 7f807ca79700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8078199010 0x7f8078199490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55144/0 (socket says 172.21.15.98:55144) 2024-03-22T01:28:05.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:05.175+0000 7f807d27a700 1 -- 172.21.15.98:0/1997905460 learned_addr learned my addr 172.21.15.98:0/1997905460 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-22T01:28:05.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:05.175+0000 7f807ca79700 1 -- 172.21.15.98:0/1997905460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8078199010 msgr2=0x7f8078199490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-22T01:28:05.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:05.175+0000 7f807ca79700 1 -- 172.21.15.98:0/1997905460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8078199010 msgr2=0x7f8078199490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-22T01:28:05.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:05.175+0000 7f807ca79700 1 --2- 172.21.15.98:0/1997905460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8078199010 0x7f8078199490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-22T01:28:05.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:05.175+0000 7f807ca79700 1 --2- 172.21.15.98:0/1997905460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8078199010 0x7f8078199490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-22T01:28:05.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:05.176+0000 7f807d27a700 1 -- 172.21.15.98:0/1997905460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8078199010 msgr2=0x7f8078199490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-22T01:28:05.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:05.176+0000 7f807d27a700 1 --2- 172.21.15.98:0/1997905460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8078199010 0x7f8078199490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:28:05.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:05.176+0000 7f807d27a700 1 -- 172.21.15.98:0/1997905460 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80781a1b40 con 0x7f80781012a0 2024-03-22T01:28:05.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:05.176+0000 7f807d27a700 1 --2- 172.21.15.98:0/1997905460 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f80781012a0 0x7f8078198ad0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f807400f510 tx=0x7f8074004d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:28:05.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:05.176+0000 7f806e7fc700 1 -- 172.21.15.98:0/1997905460 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f80740053b0 con 0x7f80781012a0 2024-03-22T01:28:05.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:05.176+0000 7f806e7fc700 1 -- 172.21.15.98:0/1997905460 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8074005550 con 0x7f80781012a0 2024-03-22T01:28:05.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:05.176+0000 7f807f4de700 1 -- 172.21.15.98:0/1997905460 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80781a1d10 con 0x7f80781012a0 2024-03-22T01:28:05.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:05.176+0000 7f806e7fc700 1 -- 172.21.15.98:0/1997905460 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f8074011810 con 0x7f80781012a0 2024-03-22T01:28:05.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:05.176+0000 7f807f4de700 1 -- 172.21.15.98:0/1997905460 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80781a21f0 con 0x7f80781012a0 2024-03-22T01:28:05.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:05.177+0000 7f806e7fc700 1 -- 172.21.15.98:0/1997905460 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 13) ==== 46350+0+0 (secure 0 0 0) 0x7f8074011ac0 con 0x7f80781012a0 2024-03-22T01:28:05.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:05.177+0000 7f806e7fc700 1 --2- 172.21.15.98:0/1997905460 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f805c038fe0 0x7f805c03b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:28:05.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:05.177+0000 7f807f4de700 1 -- 172.21.15.98:0/1997905460 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8078102780 con 0x7f80781012a0 2024-03-22T01:28:05.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:05.177+0000 7f806e7fc700 1 -- 172.21.15.98:0/1997905460 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f807404eb00 con 0x7f80781012a0 2024-03-22T01:28:05.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:05.177+0000 7f807ca79700 1 --2- 172.21.15.98:0/1997905460 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f805c038fe0 0x7f805c03b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:28:05.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:05.178+0000 7f807ca79700 1 --2- 172.21.15.98:0/1997905460 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f805c038fe0 0x7f805c03b4a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8068009850 tx=0x7f8068008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:28:05.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:05.183+0000 7f806e7fc700 1 -- 172.21.15.98:0/1997905460 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8074022070 con 0x7f80781012a0 2024-03-22T01:28:05.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:05.459+0000 7f807f4de700 1 -- 172.21.15.98:0/1997905460 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f807802df30 con 0x7f80781012a0 2024-03-22T01:28:05.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:05.460+0000 7f806e7fc700 1 -- 172.21.15.98:0/1997905460 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1011 (secure 0 0 0) 0x7f807402d360 con 0x7f80781012a0 2024-03-22T01:28:05.459 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-22T01:28:05.459 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":2,"fsid":"022a99bc-e7eb-11ee-95cd-87774f69a715","modified":"2024-03-22T01:27:58.619382Z","created":"2024-03-22T01:26:20.854185Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:3300","nonce":0},{"type":"v1","addr":"172.21.15.72:6789","nonce":0}]},"addr":"172.21.15.72:6789/0","public_addr":"172.21.15.72:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-22T01:28:05.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:05.462+0000 7f807f4de700 1 -- 172.21.15.98:0/1997905460 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f805c038fe0 msgr2=0x7f805c03b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:28:05.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:05.462+0000 7f807f4de700 1 --2- 172.21.15.98:0/1997905460 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f805c038fe0 0x7f805c03b4a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8068009850 tx=0x7f8068008040 comp rx=0 tx=0).stop 2024-03-22T01:28:05.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:05.462+0000 7f807f4de700 1 -- 172.21.15.98:0/1997905460 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f80781012a0 msgr2=0x7f8078198ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:28:05.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:05.462+0000 7f807f4de700 1 --2- 172.21.15.98:0/1997905460 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f80781012a0 0x7f8078198ad0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f807400f510 tx=0x7f8074004d60 comp rx=0 tx=0).stop 2024-03-22T01:28:05.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:05.462+0000 7f807f4de700 1 -- 172.21.15.98:0/1997905460 shutdown_connections 2024-03-22T01:28:05.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:05.462+0000 7f807f4de700 1 --2- 172.21.15.98:0/1997905460 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f805c038fe0 0x7f805c03b4a0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:28:05.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:05.463+0000 7f807f4de700 1 --2- 172.21.15.98:0/1997905460 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f80781012a0 0x7f8078198ad0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:28:05.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:05.463+0000 7f807f4de700 1 --2- 172.21.15.98:0/1997905460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8078199010 0x7f8078199490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:28:05.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:05.463+0000 7f807f4de700 1 -- 172.21.15.98:0/1997905460 >> 172.21.15.98:0/1997905460 conn(0x7f80780fcb30 msgr2=0x7f8078100000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:28:05.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:05.463+0000 7f807f4de700 1 -- 172.21.15.98:0/1997905460 shutdown_connections 2024-03-22T01:28:05.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:05.463+0000 7f807f4de700 1 -- 172.21.15.98:0/1997905460 wait complete. 2024-03-22T01:28:05.463 INFO:teuthology.orchestra.run.smithi098.stderr:dumped monmap epoch 2 2024-03-22T01:28:06.051 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:05 smithi098 ceph-mon[36468]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:06.052 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:05 smithi098 ceph-mon[36468]: from='client.? 172.21.15.98:0/1997905460' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-22T01:28:06.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:05 smithi072 ceph-mon[32414]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:06.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:05 smithi072 ceph-mon[32414]: from='client.? 172.21.15.98:0/1997905460' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-22T01:28:07.078 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-22T01:28:07.079 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-22T01:28:07.319 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.c/config 2024-03-22T01:28:07.478 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:07 smithi089 ceph-mon[35770]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-22T01:28:07.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:07 smithi089 bash[35751]: b9ecbcab627969172d76b51739bb6f24eb5e3fabd3eb23fe9481281ccef7053a 2024-03-22T01:28:07.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:07 smithi089 systemd[1]: Started Ceph mon.b for 022a99bc-e7eb-11ee-95cd-87774f69a715. 2024-03-22T01:28:12.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:12 smithi072 ceph-mon[32414]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:12.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:12 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T01:28:12.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:12 smithi072 ceph-mon[32414]: mon.c calling monitor election 2024-03-22T01:28:12.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:12 smithi072 ceph-mon[32414]: mon.a calling monitor election 2024-03-22T01:28:12.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:12 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T01:28:12.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:12 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T01:28:12.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:12 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T01:28:12.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:12 smithi072 ceph-mon[32414]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:12.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:12 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T01:28:12.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:12 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T01:28:12.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:12 smithi072 ceph-mon[32414]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:12.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:12 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T01:28:12.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:12 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T01:28:12.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:12 smithi072 ceph-mon[32414]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-22T01:28:12.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:12 smithi072 ceph-mon[32414]: monmap epoch 3 2024-03-22T01:28:12.874 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:12 smithi072 ceph-mon[32414]: fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 2024-03-22T01:28:12.874 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:12 smithi072 ceph-mon[32414]: last_changed 2024-03-22T01:28:07.459548+0000 2024-03-22T01:28:12.874 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:12 smithi072 ceph-mon[32414]: created 2024-03-22T01:26:20.854185+0000 2024-03-22T01:28:12.874 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:12 smithi072 ceph-mon[32414]: min_mon_release 19 (squid) 2024-03-22T01:28:12.874 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:12 smithi072 ceph-mon[32414]: election_strategy: 1 2024-03-22T01:28:12.874 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:12 smithi072 ceph-mon[32414]: 0: [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] mon.a 2024-03-22T01:28:12.874 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:12 smithi072 ceph-mon[32414]: 1: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.c 2024-03-22T01:28:12.874 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:12 smithi072 ceph-mon[32414]: 2: [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] mon.b 2024-03-22T01:28:12.874 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:12 smithi072 ceph-mon[32414]: fsmap 2024-03-22T01:28:12.874 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:12 smithi072 ceph-mon[32414]: osdmap e4: 0 total, 0 up, 0 in 2024-03-22T01:28:12.874 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:12 smithi072 ceph-mon[32414]: mgrmap e13: x(active, since 57s) 2024-03-22T01:28:12.874 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:12 smithi072 ceph-mon[32414]: overall HEALTH_OK 2024-03-22T01:28:12.874 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:12 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:12.874 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:12 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:12.875 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:12 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:12.875 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:12 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:12.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:12 smithi098 ceph-mon[36468]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:12.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:12 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T01:28:12.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:12 smithi098 ceph-mon[36468]: mon.c calling monitor election 2024-03-22T01:28:12.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:12 smithi098 ceph-mon[36468]: mon.a calling monitor election 2024-03-22T01:28:12.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:12 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T01:28:12.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:12 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T01:28:12.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:12 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T01:28:12.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:12 smithi098 ceph-mon[36468]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:12.923 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:12 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T01:28:12.923 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:12 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T01:28:12.923 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:12 smithi098 ceph-mon[36468]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:12.923 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:12 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T01:28:12.923 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:12 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T01:28:12.923 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:12 smithi098 ceph-mon[36468]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-22T01:28:12.923 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:12 smithi098 ceph-mon[36468]: monmap epoch 3 2024-03-22T01:28:12.923 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:12 smithi098 ceph-mon[36468]: fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 2024-03-22T01:28:12.923 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:12 smithi098 ceph-mon[36468]: last_changed 2024-03-22T01:28:07.459548+0000 2024-03-22T01:28:12.923 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:12 smithi098 ceph-mon[36468]: created 2024-03-22T01:26:20.854185+0000 2024-03-22T01:28:12.923 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:12 smithi098 ceph-mon[36468]: min_mon_release 19 (squid) 2024-03-22T01:28:12.923 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:12 smithi098 ceph-mon[36468]: election_strategy: 1 2024-03-22T01:28:12.923 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:12 smithi098 ceph-mon[36468]: 0: [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] mon.a 2024-03-22T01:28:12.923 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:12 smithi098 ceph-mon[36468]: 1: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.c 2024-03-22T01:28:12.924 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:12 smithi098 ceph-mon[36468]: 2: [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] mon.b 2024-03-22T01:28:12.924 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:12 smithi098 ceph-mon[36468]: fsmap 2024-03-22T01:28:12.924 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:12 smithi098 ceph-mon[36468]: osdmap e4: 0 total, 0 up, 0 in 2024-03-22T01:28:12.924 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:12 smithi098 ceph-mon[36468]: mgrmap e13: x(active, since 57s) 2024-03-22T01:28:12.924 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:12 smithi098 ceph-mon[36468]: overall HEALTH_OK 2024-03-22T01:28:12.924 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:12 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:12.924 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:12 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:12.924 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:12 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:12.924 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:12 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:13.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:13.080+0000 7f51b133a700 1 -- 172.21.15.98:0/893847689 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51ac1012b0 msgr2=0x7f5194005610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:28:13.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:13.080+0000 7f51b133a700 1 --2- 172.21.15.98:0/893847689 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51ac1012b0 0x7f5194005610 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f51a0005fa0 tx=0x7f51a000ff70 comp rx=0 tx=0).stop 2024-03-22T01:28:13.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:13.080+0000 7f51b133a700 1 -- 172.21.15.98:0/893847689 shutdown_connections 2024-03-22T01:28:13.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:13.080+0000 7f51b133a700 1 --2- 172.21.15.98:0/893847689 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51ac1012b0 0x7f5194005610 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:28:13.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:13.081+0000 7f51b133a700 1 -- 172.21.15.98:0/893847689 >> 172.21.15.98:0/893847689 conn(0x7f51ac0fcb20 msgr2=0x7f51ac0fef40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:28:13.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:13.081+0000 7f51b133a700 1 -- 172.21.15.98:0/893847689 shutdown_connections 2024-03-22T01:28:13.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:13.081+0000 7f51b133a700 1 -- 172.21.15.98:0/893847689 wait complete. 2024-03-22T01:28:13.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:13.081+0000 7f51b133a700 1 Processor -- start 2024-03-22T01:28:13.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:13.082+0000 7f51b133a700 1 -- start start 2024-03-22T01:28:13.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:13.082+0000 7f51b133a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51ac1012b0 0x7f51ac0fd3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:28:13.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:13.082+0000 7f51b133a700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51ac0fd910 0x7f51ac0fdd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:28:13.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:13.082+0000 7f51b133a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51ac1a3f60 0x7f51ac1a6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:28:13.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:13.082+0000 7f51b133a700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f51ac109090 con 0x7f51ac0fd910 2024-03-22T01:28:13.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:13.082+0000 7f51b133a700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f51ac108f10 con 0x7f51ac1012b0 2024-03-22T01:28:13.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:13.082+0000 7f51b133a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f51ac109210 con 0x7f51ac1a3f60 2024-03-22T01:28:13.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:13.082+0000 7f51ab7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51ac1a3f60 0x7f51ac1a6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:28:13.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:13.082+0000 7f51ab7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51ac1a3f60 0x7f51ac1a6350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47148/0 (socket says 172.21.15.98:47148) 2024-03-22T01:28:13.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:13.082+0000 7f51ab7fe700 1 -- 172.21.15.98:0/2731482415 learned_addr learned my addr 172.21.15.98:0/2731482415 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-22T01:28:13.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:13.082+0000 7f51ab7fe700 1 -- 172.21.15.98:0/2731482415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51ac1a3f60 msgr2=0x7f51ac1a6350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-22T01:28:13.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:13.082+0000 7f51ab7fe700 1 -- 172.21.15.98:0/2731482415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51ac1a3f60 msgr2=0x7f51ac1a6350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-22T01:28:13.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:13.082+0000 7f51ab7fe700 1 --2- 172.21.15.98:0/2731482415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51ac1a3f60 0x7f51ac1a6350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-22T01:28:13.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:13.082+0000 7f51aa7fc700 1 --2- 172.21.15.98:0/2731482415 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51ac0fd910 0x7f51ac0fdd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:28:13.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:13.082+0000 7f51aaffd700 1 --2- 172.21.15.98:0/2731482415 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51ac1012b0 0x7f51ac0fd3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:28:13.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:13.082+0000 7f51ab7fe700 1 --2- 172.21.15.98:0/2731482415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51ac1a3f60 0x7f51ac1a6350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-22T01:28:13.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:13.082+0000 7f51aaffd700 1 -- 172.21.15.98:0/2731482415 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51ac1012b0 msgr2=0x7f51ac0fd3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-22T01:28:13.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:13.082+0000 7f51aaffd700 1 -- 172.21.15.98:0/2731482415 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51ac1012b0 msgr2=0x7f51ac0fd3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-22T01:28:13.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:13.082+0000 7f51aaffd700 1 --2- 172.21.15.98:0/2731482415 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51ac1012b0 0x7f51ac0fd3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-22T01:28:13.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:13.082+0000 7f51aaffd700 1 --2- 172.21.15.98:0/2731482415 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51ac1012b0 0x7f51ac0fd3d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-22T01:28:13.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:13.082+0000 7f51aa7fc700 1 -- 172.21.15.98:0/2731482415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51ac1a3f60 msgr2=0x7f51ac1a6350 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-22T01:28:13.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:13.082+0000 7f51aa7fc700 1 --2- 172.21.15.98:0/2731482415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51ac1a3f60 0x7f51ac1a6350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:28:13.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:13.082+0000 7f51aa7fc700 1 -- 172.21.15.98:0/2731482415 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51ac1012b0 msgr2=0x7f51ac0fd3d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-22T01:28:13.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:13.082+0000 7f51aa7fc700 1 --2- 172.21.15.98:0/2731482415 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51ac1012b0 0x7f51ac0fd3d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:28:13.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:13.082+0000 7f51aa7fc700 1 -- 172.21.15.98:0/2731482415 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51ac1a69f0 con 0x7f51ac0fd910 2024-03-22T01:28:13.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:13.083+0000 7f51aa7fc700 1 --2- 172.21.15.98:0/2731482415 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51ac0fd910 0x7f51ac0fdd90 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f519c00b550 tx=0x7f519c00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:28:13.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:13.083+0000 7f5193fff700 1 -- 172.21.15.98:0/2731482415 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f519c004230 con 0x7f51ac0fd910 2024-03-22T01:28:13.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:13.083+0000 7f5193fff700 1 -- 172.21.15.98:0/2731482415 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f519c0043d0 con 0x7f51ac0fd910 2024-03-22T01:28:13.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:13.083+0000 7f51b133a700 1 -- 172.21.15.98:0/2731482415 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51ac1a6ce0 con 0x7f51ac0fd910 2024-03-22T01:28:13.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:13.083+0000 7f5193fff700 1 -- 172.21.15.98:0/2731482415 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f519c011c20 con 0x7f51ac0fd910 2024-03-22T01:28:13.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:13.083+0000 7f51b133a700 1 -- 172.21.15.98:0/2731482415 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51ac1a7270 con 0x7f51ac0fd910 2024-03-22T01:28:13.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:13.084+0000 7f5193fff700 1 -- 172.21.15.98:0/2731482415 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 13) ==== 46350+0+0 (secure 0 0 0) 0x7f519c01a400 con 0x7f51ac0fd910 2024-03-22T01:28:13.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:13.084+0000 7f51b133a700 1 -- 172.21.15.98:0/2731482415 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51ac063cb0 con 0x7f51ac0fd910 2024-03-22T01:28:13.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:13.084+0000 7f5193fff700 1 --2- 172.21.15.98:0/2731482415 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f51980396b0 0x7f519803bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:28:13.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:13.084+0000 7f5193fff700 1 -- 172.21.15.98:0/2731482415 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f519c04cef0 con 0x7f51ac0fd910 2024-03-22T01:28:13.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:13.084+0000 7f51aaffd700 1 --2- 172.21.15.98:0/2731482415 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f51980396b0 0x7f519803bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:28:13.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:13.085+0000 7f51aaffd700 1 --2- 172.21.15.98:0/2731482415 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f51980396b0 0x7f519803bb70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f51a00046c0 tx=0x7f51a000f3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:28:13.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:13.089+0000 7f5193fff700 1 -- 172.21.15.98:0/2731482415 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f519c0151c0 con 0x7f51ac0fd910 2024-03-22T01:28:13.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:13.388+0000 7f51b133a700 1 -- 172.21.15.98:0/2731482415 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f51ac02dd70 con 0x7f51ac0fd910 2024-03-22T01:28:13.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:13.388+0000 7f5193fff700 1 -- 172.21.15.98:0/2731482415 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1275 (secure 0 0 0) 0x7f519c011dc0 con 0x7f51ac0fd910 2024-03-22T01:28:13.387 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-22T01:28:13.388 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":3,"fsid":"022a99bc-e7eb-11ee-95cd-87774f69a715","modified":"2024-03-22T01:28:07.459548Z","created":"2024-03-22T01:26:20.854185Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:3300","nonce":0},{"type":"v1","addr":"172.21.15.72:6789","nonce":0}]},"addr":"172.21.15.72:6789/0","public_addr":"172.21.15.72:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-22T01:28:13.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:13.391+0000 7f51b133a700 1 -- 172.21.15.98:0/2731482415 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f51980396b0 msgr2=0x7f519803bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:28:13.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:13.391+0000 7f51b133a700 1 --2- 172.21.15.98:0/2731482415 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f51980396b0 0x7f519803bb70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f51a00046c0 tx=0x7f51a000f3c0 comp rx=0 tx=0).stop 2024-03-22T01:28:13.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:13.391+0000 7f51b133a700 1 -- 172.21.15.98:0/2731482415 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51ac0fd910 msgr2=0x7f51ac0fdd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:28:13.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:13.391+0000 7f51b133a700 1 --2- 172.21.15.98:0/2731482415 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51ac0fd910 0x7f51ac0fdd90 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f519c00b550 tx=0x7f519c00b910 comp rx=0 tx=0).stop 2024-03-22T01:28:13.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:13.391+0000 7f51b133a700 1 -- 172.21.15.98:0/2731482415 shutdown_connections 2024-03-22T01:28:13.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:13.391+0000 7f51b133a700 1 --2- 172.21.15.98:0/2731482415 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f51980396b0 0x7f519803bb70 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:28:13.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:13.391+0000 7f51b133a700 1 --2- 172.21.15.98:0/2731482415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51ac1a3f60 0x7f51ac1a6350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:28:13.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:13.391+0000 7f51b133a700 1 --2- 172.21.15.98:0/2731482415 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51ac1012b0 0x7f51ac0fd3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:28:13.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:13.391+0000 7f51b133a700 1 --2- 172.21.15.98:0/2731482415 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51ac0fd910 0x7f51ac0fdd90 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:28:13.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:13.391+0000 7f51b133a700 1 -- 172.21.15.98:0/2731482415 >> 172.21.15.98:0/2731482415 conn(0x7f51ac0fcb20 msgr2=0x7f51ac107710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:28:13.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:13.391+0000 7f51b133a700 1 -- 172.21.15.98:0/2731482415 shutdown_connections 2024-03-22T01:28:13.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:13.391+0000 7f51b133a700 1 -- 172.21.15.98:0/2731482415 wait complete. 2024-03-22T01:28:13.391 INFO:teuthology.orchestra.run.smithi098.stderr:dumped monmap epoch 3 2024-03-22T01:28:13.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:13 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:28:13.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:13 smithi098 ceph-mon[36468]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:13.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:13 smithi098 ceph-mon[36468]: from='client.? 172.21.15.98:0/2731482415' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-22T01:28:13.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:13 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T01:28:13.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:13 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:28:13.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:13 smithi072 ceph-mon[32414]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:13.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:13 smithi072 ceph-mon[32414]: from='client.? 172.21.15.98:0/2731482415' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-22T01:28:13.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:13 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T01:28:14.024 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-22T01:28:14.024 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph config generate-minimal-conf 2024-03-22T01:28:14.270 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:28:14.830 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:14 smithi089 ceph-mon[35770]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:14.830 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:14 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T01:28:14.830 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:14 smithi089 ceph-mon[35770]: mon.c calling monitor election 2024-03-22T01:28:14.830 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:14 smithi089 ceph-mon[35770]: mon.a calling monitor election 2024-03-22T01:28:14.830 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:14 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T01:28:14.830 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:14 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T01:28:14.830 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:14 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T01:28:14.830 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:14 smithi089 ceph-mon[35770]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:14.830 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:14 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T01:28:14.830 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:14 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T01:28:14.831 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:14 smithi089 ceph-mon[35770]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:14.831 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:14 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T01:28:14.831 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:14 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T01:28:14.831 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:14 smithi089 ceph-mon[35770]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-22T01:28:14.831 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:14 smithi089 ceph-mon[35770]: monmap epoch 3 2024-03-22T01:28:14.831 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:14 smithi089 ceph-mon[35770]: fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 2024-03-22T01:28:14.831 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:14 smithi089 ceph-mon[35770]: last_changed 2024-03-22T01:28:07.459548+0000 2024-03-22T01:28:14.831 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:14 smithi089 ceph-mon[35770]: created 2024-03-22T01:26:20.854185+0000 2024-03-22T01:28:14.831 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:14 smithi089 ceph-mon[35770]: min_mon_release 19 (squid) 2024-03-22T01:28:14.831 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:14 smithi089 ceph-mon[35770]: election_strategy: 1 2024-03-22T01:28:14.831 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:14 smithi089 ceph-mon[35770]: 0: [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] mon.a 2024-03-22T01:28:14.831 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:14 smithi089 ceph-mon[35770]: 1: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.c 2024-03-22T01:28:14.831 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:14 smithi089 ceph-mon[35770]: 2: [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] mon.b 2024-03-22T01:28:14.831 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:14 smithi089 ceph-mon[35770]: fsmap 2024-03-22T01:28:14.831 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:14 smithi089 ceph-mon[35770]: osdmap e4: 0 total, 0 up, 0 in 2024-03-22T01:28:14.832 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:14 smithi089 ceph-mon[35770]: mgrmap e13: x(active, since 57s) 2024-03-22T01:28:14.832 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:14 smithi089 ceph-mon[35770]: overall HEALTH_OK 2024-03-22T01:28:14.832 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:14 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:14.832 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:14 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:14.832 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:14 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:14.832 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:14 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:14.832 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:14 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:28:14.832 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:14 smithi089 ceph-mon[35770]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:14.832 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:14 smithi089 ceph-mon[35770]: from='client.? 172.21.15.98:0/2731482415' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-22T01:28:14.832 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:14 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T01:28:14.832 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:14 smithi089 ceph-mon[35770]: mon.b calling monitor election 2024-03-22T01:28:14.832 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:14 smithi089 ceph-mon[35770]: mon.b calling monitor election 2024-03-22T01:28:14.833 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:14 smithi089 ceph-mon[35770]: mon.c calling monitor election 2024-03-22T01:28:14.833 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:14 smithi089 ceph-mon[35770]: mon.a calling monitor election 2024-03-22T01:28:14.833 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:14 smithi089 ceph-mon[35770]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-22T01:28:14.833 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:14 smithi089 ceph-mon[35770]: monmap epoch 3 2024-03-22T01:28:14.833 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:14 smithi089 ceph-mon[35770]: fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 2024-03-22T01:28:14.833 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:14 smithi089 ceph-mon[35770]: last_changed 2024-03-22T01:28:07.459548+0000 2024-03-22T01:28:14.833 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:14 smithi089 ceph-mon[35770]: created 2024-03-22T01:26:20.854185+0000 2024-03-22T01:28:14.833 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:14 smithi089 ceph-mon[35770]: min_mon_release 19 (squid) 2024-03-22T01:28:14.833 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:14 smithi089 ceph-mon[35770]: election_strategy: 1 2024-03-22T01:28:14.833 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:14 smithi089 ceph-mon[35770]: 0: [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] mon.a 2024-03-22T01:28:14.833 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:14 smithi089 ceph-mon[35770]: 1: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.c 2024-03-22T01:28:14.833 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:14 smithi089 ceph-mon[35770]: 2: [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] mon.b 2024-03-22T01:28:14.833 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:14 smithi089 ceph-mon[35770]: fsmap 2024-03-22T01:28:14.833 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:14 smithi089 ceph-mon[35770]: osdmap e4: 0 total, 0 up, 0 in 2024-03-22T01:28:14.833 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:14 smithi089 ceph-mon[35770]: mgrmap e13: x(active, since 59s) 2024-03-22T01:28:14.834 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:14 smithi089 ceph-mon[35770]: overall HEALTH_OK 2024-03-22T01:28:14.834 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:14 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:14.834 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:14 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:14.834 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:14 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:28:14.834 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:14 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:28:15.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:14 smithi072 ceph-mon[32414]: mon.b calling monitor election 2024-03-22T01:28:15.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:14 smithi072 ceph-mon[32414]: mon.b calling monitor election 2024-03-22T01:28:15.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:14 smithi072 ceph-mon[32414]: mon.c calling monitor election 2024-03-22T01:28:15.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:14 smithi072 ceph-mon[32414]: mon.a calling monitor election 2024-03-22T01:28:15.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:14 smithi072 ceph-mon[32414]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-22T01:28:15.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:14 smithi072 ceph-mon[32414]: monmap epoch 3 2024-03-22T01:28:15.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:14 smithi072 ceph-mon[32414]: fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 2024-03-22T01:28:15.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:14 smithi072 ceph-mon[32414]: last_changed 2024-03-22T01:28:07.459548+0000 2024-03-22T01:28:15.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:14 smithi072 ceph-mon[32414]: created 2024-03-22T01:26:20.854185+0000 2024-03-22T01:28:15.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:14 smithi072 ceph-mon[32414]: min_mon_release 19 (squid) 2024-03-22T01:28:15.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:14 smithi072 ceph-mon[32414]: election_strategy: 1 2024-03-22T01:28:15.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:14 smithi072 ceph-mon[32414]: 0: [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] mon.a 2024-03-22T01:28:15.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:14 smithi072 ceph-mon[32414]: 1: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.c 2024-03-22T01:28:15.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:14 smithi072 ceph-mon[32414]: 2: [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] mon.b 2024-03-22T01:28:15.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:14 smithi072 ceph-mon[32414]: fsmap 2024-03-22T01:28:15.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:14 smithi072 ceph-mon[32414]: osdmap e4: 0 total, 0 up, 0 in 2024-03-22T01:28:15.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:14 smithi072 ceph-mon[32414]: mgrmap e13: x(active, since 59s) 2024-03-22T01:28:15.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:14 smithi072 ceph-mon[32414]: overall HEALTH_OK 2024-03-22T01:28:15.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:14 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:15.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:14 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:15.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:14 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:28:15.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:14 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:28:15.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:14 smithi098 ceph-mon[36468]: mon.b calling monitor election 2024-03-22T01:28:15.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:14 smithi098 ceph-mon[36468]: mon.b calling monitor election 2024-03-22T01:28:15.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:14 smithi098 ceph-mon[36468]: mon.c calling monitor election 2024-03-22T01:28:15.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:14 smithi098 ceph-mon[36468]: mon.a calling monitor election 2024-03-22T01:28:15.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:14 smithi098 ceph-mon[36468]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-22T01:28:15.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:14 smithi098 ceph-mon[36468]: monmap epoch 3 2024-03-22T01:28:15.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:14 smithi098 ceph-mon[36468]: fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 2024-03-22T01:28:15.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:14 smithi098 ceph-mon[36468]: last_changed 2024-03-22T01:28:07.459548+0000 2024-03-22T01:28:15.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:14 smithi098 ceph-mon[36468]: created 2024-03-22T01:26:20.854185+0000 2024-03-22T01:28:15.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:14 smithi098 ceph-mon[36468]: min_mon_release 19 (squid) 2024-03-22T01:28:15.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:14 smithi098 ceph-mon[36468]: election_strategy: 1 2024-03-22T01:28:15.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:14 smithi098 ceph-mon[36468]: 0: [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] mon.a 2024-03-22T01:28:15.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:14 smithi098 ceph-mon[36468]: 1: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.c 2024-03-22T01:28:15.173 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:14 smithi098 ceph-mon[36468]: 2: [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] mon.b 2024-03-22T01:28:15.173 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:14 smithi098 ceph-mon[36468]: fsmap 2024-03-22T01:28:15.173 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:14 smithi098 ceph-mon[36468]: osdmap e4: 0 total, 0 up, 0 in 2024-03-22T01:28:15.173 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:14 smithi098 ceph-mon[36468]: mgrmap e13: x(active, since 59s) 2024-03-22T01:28:15.173 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:14 smithi098 ceph-mon[36468]: overall HEALTH_OK 2024-03-22T01:28:15.173 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:14 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:15.173 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:14 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:15.173 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:14 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:28:15.173 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:14 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:28:15.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:15.828+0000 7f8fbaa85700 1 -- 172.21.15.72:0/2098252199 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8fb4103220 msgr2=0x7f8fb4103600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:28:15.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:15.828+0000 7f8fbaa85700 1 --2- 172.21.15.72:0/2098252199 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8fb4103220 0x7f8fb4103600 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8fa8009910 tx=0x7f8fa8009c20 comp rx=0 tx=0).stop 2024-03-22T01:28:15.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:15.828+0000 7f8fbaa85700 1 -- 172.21.15.72:0/2098252199 shutdown_connections 2024-03-22T01:28:15.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:15.828+0000 7f8fbaa85700 1 --2- 172.21.15.72:0/2098252199 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8fb4103220 0x7f8fb4103600 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:28:15.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:15.828+0000 7f8fbaa85700 1 -- 172.21.15.72:0/2098252199 >> 172.21.15.72:0/2098252199 conn(0x7f8fb40fca70 msgr2=0x7f8fb40fee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:28:15.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:15.828+0000 7f8fbaa85700 1 -- 172.21.15.72:0/2098252199 shutdown_connections 2024-03-22T01:28:15.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:15.828+0000 7f8fbaa85700 1 -- 172.21.15.72:0/2098252199 wait complete. 2024-03-22T01:28:15.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:15.829+0000 7f8fbaa85700 1 Processor -- start 2024-03-22T01:28:15.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:15.829+0000 7f8fbaa85700 1 -- start start 2024-03-22T01:28:15.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:15.829+0000 7f8fbaa85700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8fb4103220 0x7f8fb4100e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:28:15.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:15.829+0000 7f8fbaa85700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fb4102ac0 0x7f8fb41013c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:28:15.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:15.829+0000 7f8fbaa85700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8fb4101900 0x7f8fb41aa860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:28:15.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:15.829+0000 7f8fbaa85700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f8fb410c6c0 con 0x7f8fb4103220 2024-03-22T01:28:15.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:15.829+0000 7f8fbaa85700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8fb410c540 con 0x7f8fb4102ac0 2024-03-22T01:28:15.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:15.829+0000 7f8fbaa85700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8fb410c840 con 0x7f8fb4101900 2024-03-22T01:28:15.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:15.829+0000 7f8fb8821700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8fb4103220 0x7f8fb4100e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:28:15.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:15.829+0000 7f8fb8821700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8fb4103220 0x7f8fb4100e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37214/0 (socket says 172.21.15.72:37214) 2024-03-22T01:28:15.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:15.829+0000 7f8fb8821700 1 -- 172.21.15.72:0/1154813940 learned_addr learned my addr 172.21.15.72:0/1154813940 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:28:15.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:15.829+0000 7f8fb8821700 1 -- 172.21.15.72:0/1154813940 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8fb4101900 msgr2=0x7f8fb41aa860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:28:15.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:15.829+0000 7f8fb8821700 1 --2- 172.21.15.72:0/1154813940 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8fb4101900 0x7f8fb41aa860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:28:15.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:15.829+0000 7f8fb8821700 1 -- 172.21.15.72:0/1154813940 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fb4102ac0 msgr2=0x7f8fb41013c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:28:15.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:15.829+0000 7f8fb8821700 1 --2- 172.21.15.72:0/1154813940 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fb4102ac0 0x7f8fb41013c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:28:15.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:15.829+0000 7f8fb8821700 1 -- 172.21.15.72:0/1154813940 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fb41aafb0 con 0x7f8fb4103220 2024-03-22T01:28:15.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:15.830+0000 7f8fb8821700 1 --2- 172.21.15.72:0/1154813940 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8fb4103220 0x7f8fb4100e80 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8fa8009fd0 tx=0x7f8fa800be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:28:15.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:15.830+0000 7f8fb1ffb700 1 -- 172.21.15.72:0/1154813940 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa8027070 con 0x7f8fb4103220 2024-03-22T01:28:15.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:15.830+0000 7f8fb1ffb700 1 -- 172.21.15.72:0/1154813940 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8fa80050d0 con 0x7f8fb4103220 2024-03-22T01:28:15.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:15.830+0000 7f8fbaa85700 1 -- 172.21.15.72:0/1154813940 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fb41ab240 con 0x7f8fb4103220 2024-03-22T01:28:15.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:15.830+0000 7f8fb1ffb700 1 -- 172.21.15.72:0/1154813940 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa800f460 con 0x7f8fb4103220 2024-03-22T01:28:15.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:15.830+0000 7f8fbaa85700 1 -- 172.21.15.72:0/1154813940 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fb41ab720 con 0x7f8fb4103220 2024-03-22T01:28:15.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:15.831+0000 7f8fb1ffb700 1 -- 172.21.15.72:0/1154813940 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 13) ==== 46350+0+0 (secure 0 0 0) 0x7f8fa802a070 con 0x7f8fb4103220 2024-03-22T01:28:15.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:15.831+0000 7f8fb1ffb700 1 --2- 172.21.15.72:0/1154813940 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f8fa40396b0 0x7f8fa403bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:28:15.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:15.831+0000 7f8fb1ffb700 1 -- 172.21.15.72:0/1154813940 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f8fa804d4c0 con 0x7f8fb4103220 2024-03-22T01:28:15.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:15.831+0000 7f8fbaa85700 1 -- 172.21.15.72:0/1154813940 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fb406ae40 con 0x7f8fb4103220 2024-03-22T01:28:15.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:15.831+0000 7f8fb3fff700 1 --2- 172.21.15.72:0/1154813940 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f8fa40396b0 0x7f8fa403bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:28:15.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:15.831+0000 7f8fb3fff700 1 --2- 172.21.15.72:0/1154813940 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f8fa40396b0 0x7f8fa403bb70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8fb419bff0 tx=0x7f8fa0006dd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:28:15.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:15.836+0000 7f8fb1ffb700 1 -- 172.21.15.72:0/1154813940 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8fa8013070 con 0x7f8fb4103220 2024-03-22T01:28:16.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:16.040+0000 7f8fbaa85700 1 -- 172.21.15.72:0/1154813940 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f8fb41ab980 con 0x7f8fb4103220 2024-03-22T01:28:16.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:16.040+0000 7f8fb1ffb700 1 -- 172.21.15.72:0/1154813940 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+271 (secure 0 0 0) 0x7f8fa8031020 con 0x7f8fb4103220 2024-03-22T01:28:16.039 INFO:teuthology.orchestra.run.smithi072.stdout:# minimal ceph.conf for 022a99bc-e7eb-11ee-95cd-87774f69a715 2024-03-22T01:28:16.039 INFO:teuthology.orchestra.run.smithi072.stdout:[global] 2024-03-22T01:28:16.039 INFO:teuthology.orchestra.run.smithi072.stdout: fsid = 022a99bc-e7eb-11ee-95cd-87774f69a715 2024-03-22T01:28:16.039 INFO:teuthology.orchestra.run.smithi072.stdout: mon_host = [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-03-22T01:28:16.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:16.042+0000 7f8fbaa85700 1 -- 172.21.15.72:0/1154813940 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f8fa40396b0 msgr2=0x7f8fa403bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:28:16.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:16.042+0000 7f8fbaa85700 1 --2- 172.21.15.72:0/1154813940 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f8fa40396b0 0x7f8fa403bb70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8fb419bff0 tx=0x7f8fa0006dd0 comp rx=0 tx=0).stop 2024-03-22T01:28:16.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:16.042+0000 7f8fbaa85700 1 -- 172.21.15.72:0/1154813940 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8fb4103220 msgr2=0x7f8fb4100e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:28:16.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:16.042+0000 7f8fbaa85700 1 --2- 172.21.15.72:0/1154813940 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8fb4103220 0x7f8fb4100e80 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8fa8009fd0 tx=0x7f8fa800be90 comp rx=0 tx=0).stop 2024-03-22T01:28:16.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:16.042+0000 7f8fbaa85700 1 -- 172.21.15.72:0/1154813940 shutdown_connections 2024-03-22T01:28:16.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:16.043+0000 7f8fbaa85700 1 --2- 172.21.15.72:0/1154813940 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f8fa40396b0 0x7f8fa403bb70 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:28:16.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:16.043+0000 7f8fbaa85700 1 --2- 172.21.15.72:0/1154813940 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8fb4101900 0x7f8fb41aa860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:28:16.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:16.043+0000 7f8fbaa85700 1 --2- 172.21.15.72:0/1154813940 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8fb4103220 0x7f8fb4100e80 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:28:16.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:16.043+0000 7f8fbaa85700 1 --2- 172.21.15.72:0/1154813940 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fb4102ac0 0x7f8fb41013c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:28:16.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:16.043+0000 7f8fbaa85700 1 -- 172.21.15.72:0/1154813940 >> 172.21.15.72:0/1154813940 conn(0x7f8fb40fca70 msgr2=0x7f8fb4105fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:28:16.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:16.043+0000 7f8fbaa85700 1 -- 172.21.15.72:0/1154813940 shutdown_connections 2024-03-22T01:28:16.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:16.043+0000 7f8fbaa85700 1 -- 172.21.15.72:0/1154813940 wait complete. 2024-03-22T01:28:16.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:16 smithi089 ceph-mon[35770]: Updating smithi072:/etc/ceph/ceph.conf 2024-03-22T01:28:16.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:16 smithi089 ceph-mon[35770]: Updating smithi089:/etc/ceph/ceph.conf 2024-03-22T01:28:16.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:16 smithi089 ceph-mon[35770]: Updating smithi098:/etc/ceph/ceph.conf 2024-03-22T01:28:16.359 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:16 smithi089 ceph-mon[35770]: Updating smithi089:/var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T01:28:16.359 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:16 smithi089 ceph-mon[35770]: Updating smithi072:/var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T01:28:16.359 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:16 smithi089 ceph-mon[35770]: Updating smithi098:/var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T01:28:16.359 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:16 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:16.359 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:16 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:16.359 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:16 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:16.359 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:16 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:16.359 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:16 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:16.359 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:16 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:16.359 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:16 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:16.359 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:16 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:16.359 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:16 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:16.359 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:16 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:16.359 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:16 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:16.359 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:16 smithi089 ceph-mon[35770]: Reconfiguring mon.a (unknown last config time)... 2024-03-22T01:28:16.360 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:16 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T01:28:16.360 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:16 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-22T01:28:16.360 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:16 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:28:16.360 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:16 smithi089 ceph-mon[35770]: Reconfiguring daemon mon.a on smithi072 2024-03-22T01:28:16.360 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:16 smithi089 ceph-mon[35770]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:16.360 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:16 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:16.360 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:16 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T01:28:16.360 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:16 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/1154813940' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:28:16.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:16 smithi072 ceph-mon[32414]: Updating smithi072:/etc/ceph/ceph.conf 2024-03-22T01:28:16.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:16 smithi072 ceph-mon[32414]: Updating smithi089:/etc/ceph/ceph.conf 2024-03-22T01:28:16.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:16 smithi072 ceph-mon[32414]: Updating smithi098:/etc/ceph/ceph.conf 2024-03-22T01:28:16.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:16 smithi072 ceph-mon[32414]: Updating smithi089:/var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T01:28:16.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:16 smithi072 ceph-mon[32414]: Updating smithi072:/var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T01:28:16.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:16 smithi072 ceph-mon[32414]: Updating smithi098:/var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T01:28:16.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:16 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:16.373 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:16 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:16.373 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:16 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:16.373 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:16 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:16.373 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:16 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:16.373 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:16 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:16.373 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:16 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:16.373 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:16 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:16.373 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:16 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:16.373 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:16 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:16.373 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:16 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:16.373 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:16 smithi072 ceph-mon[32414]: Reconfiguring mon.a (unknown last config time)... 2024-03-22T01:28:16.373 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:16 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T01:28:16.373 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:16 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-22T01:28:16.373 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:16 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:28:16.373 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:16 smithi072 ceph-mon[32414]: Reconfiguring daemon mon.a on smithi072 2024-03-22T01:28:16.374 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:16 smithi072 ceph-mon[32414]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:16.374 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:16 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:16.374 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:16 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T01:28:16.374 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:16 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/1154813940' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:28:16.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:16 smithi098 ceph-mon[36468]: Updating smithi072:/etc/ceph/ceph.conf 2024-03-22T01:28:16.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:16 smithi098 ceph-mon[36468]: Updating smithi089:/etc/ceph/ceph.conf 2024-03-22T01:28:16.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:16 smithi098 ceph-mon[36468]: Updating smithi098:/etc/ceph/ceph.conf 2024-03-22T01:28:16.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:16 smithi098 ceph-mon[36468]: Updating smithi089:/var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T01:28:16.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:16 smithi098 ceph-mon[36468]: Updating smithi072:/var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T01:28:16.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:16 smithi098 ceph-mon[36468]: Updating smithi098:/var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T01:28:16.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:16 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:16.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:16 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:16.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:16 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:16.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:16 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:16.423 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:16 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:16.423 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:16 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:16.423 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:16 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:16.423 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:16 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:16.423 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:16 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:16.423 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:16 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:16.423 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:16 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:16.423 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:16 smithi098 ceph-mon[36468]: Reconfiguring mon.a (unknown last config time)... 2024-03-22T01:28:16.423 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:16 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T01:28:16.423 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:16 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-22T01:28:16.423 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:16 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:28:16.423 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:16 smithi098 ceph-mon[36468]: Reconfiguring daemon mon.a on smithi072 2024-03-22T01:28:16.423 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:16 smithi098 ceph-mon[36468]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:16.423 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:16 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:16.423 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:16 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T01:28:16.423 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:16 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/1154813940' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:28:17.353 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-22T01:28:17.353 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-22T01:28:17.353 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/etc/ceph/ceph.conf 2024-03-22T01:28:17.383 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-22T01:28:17.383 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-22T01:28:17.451 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-22T01:28:17.451 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/etc/ceph/ceph.conf 2024-03-22T01:28:17.480 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-22T01:28:17.481 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-22T01:28:17.548 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-22T01:28:17.548 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/ceph.conf 2024-03-22T01:28:17.582 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-22T01:28:17.583 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-22T01:28:17.652 INFO:tasks.cephadm:Adding mgr.x on smithi072 2024-03-22T01:28:17.653 INFO:tasks.cephadm:Adding mgr.y on smithi089 2024-03-22T01:28:17.653 INFO:tasks.cephadm:Adding mgr.z on smithi098 2024-03-22T01:28:17.653 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph orch apply mgr '3;smithi072=x;smithi089=y;smithi098=z' 2024-03-22T01:28:17.923 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.c/config 2024-03-22T01:28:18.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:17 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:18.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:17 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:18.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:17 smithi089 ceph-mon[35770]: Reconfiguring mon.b (monmap changed)... 2024-03-22T01:28:18.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:17 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T01:28:18.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:17 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-22T01:28:18.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:17 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:28:18.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:17 smithi089 ceph-mon[35770]: Reconfiguring daemon mon.b on smithi089 2024-03-22T01:28:18.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:17 smithi089 ceph-mon[35770]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:18.321 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:17 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:18.321 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:17 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:18.321 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:17 smithi098 ceph-mon[36468]: Reconfiguring mon.b (monmap changed)... 2024-03-22T01:28:18.322 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:17 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T01:28:18.322 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:17 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-22T01:28:18.322 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:17 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:28:18.322 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:17 smithi098 ceph-mon[36468]: Reconfiguring daemon mon.b on smithi089 2024-03-22T01:28:18.322 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:17 smithi098 ceph-mon[36468]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:18.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:17 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:18.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:17 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:18.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:17 smithi072 ceph-mon[32414]: Reconfiguring mon.b (monmap changed)... 2024-03-22T01:28:18.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:17 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T01:28:18.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:17 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-22T01:28:18.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:17 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:28:18.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:17 smithi072 ceph-mon[32414]: Reconfiguring daemon mon.b on smithi089 2024-03-22T01:28:18.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:17 smithi072 ceph-mon[32414]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:19.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:19.431+0000 7fd49bcf0700 1 -- 172.21.15.98:0/3165719834 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd494102cf0 msgr2=0x7fd4941050e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:28:19.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:19.431+0000 7fd49bcf0700 1 --2- 172.21.15.98:0/3165719834 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd494102cf0 0x7fd4941050e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd49000b0a0 tx=0x7fd49000b3b0 comp rx=0 tx=0).stop 2024-03-22T01:28:19.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:19.432+0000 7fd49bcf0700 1 -- 172.21.15.98:0/3165719834 shutdown_connections 2024-03-22T01:28:19.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:19.432+0000 7fd49bcf0700 1 --2- 172.21.15.98:0/3165719834 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd494102cf0 0x7fd4941050e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:28:19.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:19.432+0000 7fd49bcf0700 1 -- 172.21.15.98:0/3165719834 >> 172.21.15.98:0/3165719834 conn(0x7fd4940fc660 msgr2=0x7fd4940feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:28:19.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:19.432+0000 7fd49bcf0700 1 -- 172.21.15.98:0/3165719834 shutdown_connections 2024-03-22T01:28:19.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:19.432+0000 7fd49bcf0700 1 -- 172.21.15.98:0/3165719834 wait complete. 2024-03-22T01:28:19.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:19.432+0000 7fd49bcf0700 1 Processor -- start 2024-03-22T01:28:19.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:19.432+0000 7fd49bcf0700 1 -- start start 2024-03-22T01:28:19.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:19.432+0000 7fd49bcf0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd494102cf0 0x7fd4941979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:28:19.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:19.432+0000 7fd49bcf0700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd494197ef0 0x7fd49419cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:28:19.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:19.432+0000 7fd49bcf0700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd494198370 0x7fd49419f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:28:19.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:19.432+0000 7fd49bcf0700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fd494108220 con 0x7fd494198370 2024-03-22T01:28:19.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:19.432+0000 7fd49bcf0700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd4941080a0 con 0x7fd494197ef0 2024-03-22T01:28:19.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:19.432+0000 7fd49bcf0700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd4941083a0 con 0x7fd494102cf0 2024-03-22T01:28:19.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:19.433+0000 7fd499a8c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd494102cf0 0x7fd4941979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:28:19.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:19.433+0000 7fd499a8c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd494102cf0 0x7fd4941979b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57254/0 (socket says 172.21.15.98:57254) 2024-03-22T01:28:19.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:19.433+0000 7fd499a8c700 1 -- 172.21.15.98:0/3516647199 learned_addr learned my addr 172.21.15.98:0/3516647199 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-22T01:28:19.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:19.433+0000 7fd49a28d700 1 --2- 172.21.15.98:0/3516647199 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd494198370 0x7fd49419f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:28:19.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:19.433+0000 7fd49928b700 1 --2- 172.21.15.98:0/3516647199 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd494197ef0 0x7fd49419cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:28:19.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:19.433+0000 7fd499a8c700 1 -- 172.21.15.98:0/3516647199 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd494197ef0 msgr2=0x7fd49419cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:28:19.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:19.433+0000 7fd499a8c700 1 --2- 172.21.15.98:0/3516647199 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd494197ef0 0x7fd49419cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:28:19.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:19.433+0000 7fd499a8c700 1 -- 172.21.15.98:0/3516647199 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd494198370 msgr2=0x7fd49419f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:28:19.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:19.433+0000 7fd499a8c700 1 --2- 172.21.15.98:0/3516647199 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd494198370 0x7fd49419f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:28:19.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:19.433+0000 7fd499a8c700 1 -- 172.21.15.98:0/3516647199 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd49419fdc0 con 0x7fd494102cf0 2024-03-22T01:28:19.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:19.433+0000 7fd49928b700 1 --2- 172.21.15.98:0/3516647199 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd494197ef0 0x7fd49419cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T01:28:19.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:19.433+0000 7fd499a8c700 1 --2- 172.21.15.98:0/3516647199 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd494102cf0 0x7fd4941979b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd490015040 tx=0x7fd490009740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:28:19.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:19.433+0000 7fd48affd700 1 -- 172.21.15.98:0/3516647199 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd490017040 con 0x7fd494102cf0 2024-03-22T01:28:19.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:19.433+0000 7fd48affd700 1 -- 172.21.15.98:0/3516647199 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd49000be10 con 0x7fd494102cf0 2024-03-22T01:28:19.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:19.433+0000 7fd48affd700 1 -- 172.21.15.98:0/3516647199 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd490012a20 con 0x7fd494102cf0 2024-03-22T01:28:19.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:19.433+0000 7fd49bcf0700 1 -- 172.21.15.98:0/3516647199 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4941a0050 con 0x7fd494102cf0 2024-03-22T01:28:19.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:19.434+0000 7fd49bcf0700 1 -- 172.21.15.98:0/3516647199 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4941a05c0 con 0x7fd494102cf0 2024-03-22T01:28:19.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:19.434+0000 7fd48affd700 1 -- 172.21.15.98:0/3516647199 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46350+0+0 (secure 0 0 0) 0x7fd490012bc0 con 0x7fd494102cf0 2024-03-22T01:28:19.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:19.434+0000 7fd48affd700 1 --2- 172.21.15.98:0/3516647199 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fd4800396b0 0x7fd48003bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:28:19.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:19.434+0000 7fd48affd700 1 -- 172.21.15.98:0/3516647199 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fd49000f020 con 0x7fd494102cf0 2024-03-22T01:28:19.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:19.434+0000 7fd49bcf0700 1 -- 172.21.15.98:0/3516647199 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd494063cb0 con 0x7fd494102cf0 2024-03-22T01:28:19.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:19.435+0000 7fd49928b700 1 --2- 172.21.15.98:0/3516647199 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fd4800396b0 0x7fd48003bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:28:19.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:19.435+0000 7fd49928b700 1 --2- 172.21.15.98:0/3516647199 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fd4800396b0 0x7fd48003bb70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd4840079b0 tx=0x7fd484008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:28:19.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:19.440+0000 7fd48affd700 1 -- 172.21.15.98:0/3516647199 <== mon.1 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd490029020 con 0x7fd494102cf0 2024-03-22T01:28:19.579 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:19 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:19.579 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:19 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:19.579 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:19 smithi098 ceph-mon[36468]: Reconfiguring mon.c (monmap changed)... 2024-03-22T01:28:19.579 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:19 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T01:28:19.580 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:19 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-22T01:28:19.580 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:19 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:28:19.580 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:19 smithi098 ceph-mon[36468]: Reconfiguring daemon mon.c on smithi098 2024-03-22T01:28:19.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:19 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:19.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:19 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:19.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:19 smithi089 ceph-mon[35770]: Reconfiguring mon.c (monmap changed)... 2024-03-22T01:28:19.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:19 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T01:28:19.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:19 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-22T01:28:19.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:19 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:28:19.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:19 smithi089 ceph-mon[35770]: Reconfiguring daemon mon.c on smithi098 2024-03-22T01:28:19.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:19 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:19.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:19 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:19.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:19 smithi072 ceph-mon[32414]: Reconfiguring mon.c (monmap changed)... 2024-03-22T01:28:19.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:19 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T01:28:19.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:19 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-22T01:28:19.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:19 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:28:19.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:19 smithi072 ceph-mon[32414]: Reconfiguring daemon mon.c on smithi098 2024-03-22T01:28:19.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:19.654+0000 7fd49bcf0700 1 -- 172.21.15.98:0/3516647199 --> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi072=x;smithi089=y;smithi098=z", "target": ["mon-mgr", ""]}) -- 0x7fd494062ac0 con 0x7fd4800396b0 2024-03-22T01:28:19.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:19.660+0000 7fd48affd700 1 -- 172.21.15.98:0/3516647199 <== mgr.14152 v2:172.21.15.72:6800/401403200 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fd494062ac0 con 0x7fd4800396b0 2024-03-22T01:28:19.659 INFO:teuthology.orchestra.run.smithi098.stdout:Scheduled mgr update... 2024-03-22T01:28:19.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:19.662+0000 7fd49bcf0700 1 -- 172.21.15.98:0/3516647199 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fd4800396b0 msgr2=0x7fd48003bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:28:19.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:19.662+0000 7fd49bcf0700 1 --2- 172.21.15.98:0/3516647199 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fd4800396b0 0x7fd48003bb70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd4840079b0 tx=0x7fd484008040 comp rx=0 tx=0).stop 2024-03-22T01:28:19.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:19.662+0000 7fd49bcf0700 1 -- 172.21.15.98:0/3516647199 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd494102cf0 msgr2=0x7fd4941979b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:28:19.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:19.662+0000 7fd49bcf0700 1 --2- 172.21.15.98:0/3516647199 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd494102cf0 0x7fd4941979b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd490015040 tx=0x7fd490009740 comp rx=0 tx=0).stop 2024-03-22T01:28:19.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:19.662+0000 7fd49bcf0700 1 -- 172.21.15.98:0/3516647199 shutdown_connections 2024-03-22T01:28:19.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:19.662+0000 7fd49bcf0700 1 --2- 172.21.15.98:0/3516647199 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fd4800396b0 0x7fd48003bb70 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:28:19.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:19.662+0000 7fd49bcf0700 1 --2- 172.21.15.98:0/3516647199 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd494198370 0x7fd49419f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:28:19.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:19.662+0000 7fd49bcf0700 1 --2- 172.21.15.98:0/3516647199 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd494102cf0 0x7fd4941979b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:28:19.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:19.662+0000 7fd49bcf0700 1 --2- 172.21.15.98:0/3516647199 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd494197ef0 0x7fd49419cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:28:19.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:19.662+0000 7fd49bcf0700 1 -- 172.21.15.98:0/3516647199 >> 172.21.15.98:0/3516647199 conn(0x7fd4940fc660 msgr2=0x7fd4940fd2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:28:19.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:19.662+0000 7fd49bcf0700 1 -- 172.21.15.98:0/3516647199 shutdown_connections 2024-03-22T01:28:19.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:28:19.662+0000 7fd49bcf0700 1 -- 172.21.15.98:0/3516647199 wait complete. 2024-03-22T01:28:20.270 DEBUG:teuthology.orchestra.run.smithi089:mgr.y> sudo journalctl -f -n 0 -u ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mgr.y.service 2024-03-22T01:28:20.273 DEBUG:teuthology.orchestra.run.smithi098:mgr.z> sudo journalctl -f -n 0 -u ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mgr.z.service 2024-03-22T01:28:20.275 INFO:tasks.cephadm:Deploying OSDs... 2024-03-22T01:28:20.275 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-22T01:28:20.275 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/scratch_devs of=/dev/stdout 2024-03-22T01:28:20.294 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-22T01:28:20.294 DEBUG:teuthology.orchestra.run.smithi072:> stat /dev/vg_nvme/lv_1 2024-03-22T01:28:20.353 INFO:teuthology.orchestra.run.smithi072.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-22T01:28:20.353 INFO:teuthology.orchestra.run.smithi072.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-22T01:28:20.353 INFO:teuthology.orchestra.run.smithi072.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-22T01:28:20.353 INFO:teuthology.orchestra.run.smithi072.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-22T01:28:20.353 INFO:teuthology.orchestra.run.smithi072.stdout:Context: system_u:object_r:device_t:s0 2024-03-22T01:28:20.354 INFO:teuthology.orchestra.run.smithi072.stdout:Access: 2024-03-22 01:27:24.606462130 +0000 2024-03-22T01:28:20.354 INFO:teuthology.orchestra.run.smithi072.stdout:Modify: 2024-03-22 01:22:02.397015221 +0000 2024-03-22T01:28:20.354 INFO:teuthology.orchestra.run.smithi072.stdout:Change: 2024-03-22 01:22:02.397015221 +0000 2024-03-22T01:28:20.354 INFO:teuthology.orchestra.run.smithi072.stdout: Birth: 2024-03-22 01:22:02.397015221 +0000 2024-03-22T01:28:20.354 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-22T01:28:20.424 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records in 2024-03-22T01:28:20.424 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records out 2024-03-22T01:28:20.424 INFO:teuthology.orchestra.run.smithi072.stderr:512 bytes copied, 0.000281312 s, 1.8 MB/s 2024-03-22T01:28:20.425 DEBUG:teuthology.orchestra.run.smithi072:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-22T01:28:20.487 DEBUG:teuthology.orchestra.run.smithi072:> stat /dev/vg_nvme/lv_2 2024-03-22T01:28:20.544 INFO:teuthology.orchestra.run.smithi072.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-22T01:28:20.544 INFO:teuthology.orchestra.run.smithi072.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-22T01:28:20.544 INFO:teuthology.orchestra.run.smithi072.stdout:Device: 5h/5d Inode: 682 Links: 1 2024-03-22T01:28:20.544 INFO:teuthology.orchestra.run.smithi072.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-22T01:28:20.544 INFO:teuthology.orchestra.run.smithi072.stdout:Context: system_u:object_r:device_t:s0 2024-03-22T01:28:20.544 INFO:teuthology.orchestra.run.smithi072.stdout:Access: 2024-03-22 01:27:24.639461422 +0000 2024-03-22T01:28:20.544 INFO:teuthology.orchestra.run.smithi072.stdout:Modify: 2024-03-22 01:22:02.737007942 +0000 2024-03-22T01:28:20.544 INFO:teuthology.orchestra.run.smithi072.stdout:Change: 2024-03-22 01:22:02.737007942 +0000 2024-03-22T01:28:20.545 INFO:teuthology.orchestra.run.smithi072.stdout: Birth: 2024-03-22 01:22:02.737007942 +0000 2024-03-22T01:28:20.545 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-22T01:28:20.607 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:20 smithi089 ceph-mon[35770]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:20.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:20 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:20.610 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:20 smithi072 ceph-mon[32414]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:20.610 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:20 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:20.612 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records in 2024-03-22T01:28:20.612 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records out 2024-03-22T01:28:20.612 INFO:teuthology.orchestra.run.smithi072.stderr:512 bytes copied, 0.000269322 s, 1.9 MB/s 2024-03-22T01:28:20.614 DEBUG:teuthology.orchestra.run.smithi072:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-22T01:28:20.670 DEBUG:teuthology.orchestra.run.smithi072:> stat /dev/vg_nvme/lv_3 2024-03-22T01:28:20.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:20 smithi098 ceph-mon[36468]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:20.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:20 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:20.727 INFO:teuthology.orchestra.run.smithi072.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-22T01:28:20.727 INFO:teuthology.orchestra.run.smithi072.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-22T01:28:20.727 INFO:teuthology.orchestra.run.smithi072.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-03-22T01:28:20.727 INFO:teuthology.orchestra.run.smithi072.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-22T01:28:20.727 INFO:teuthology.orchestra.run.smithi072.stdout:Context: system_u:object_r:device_t:s0 2024-03-22T01:28:20.727 INFO:teuthology.orchestra.run.smithi072.stdout:Access: 2024-03-22 01:27:24.671460735 +0000 2024-03-22T01:28:20.728 INFO:teuthology.orchestra.run.smithi072.stdout:Modify: 2024-03-22 01:22:03.064000941 +0000 2024-03-22T01:28:20.728 INFO:teuthology.orchestra.run.smithi072.stdout:Change: 2024-03-22 01:22:03.064000941 +0000 2024-03-22T01:28:20.728 INFO:teuthology.orchestra.run.smithi072.stdout: Birth: 2024-03-22 01:22:03.064000941 +0000 2024-03-22T01:28:20.728 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-22T01:28:20.794 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records in 2024-03-22T01:28:20.794 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records out 2024-03-22T01:28:20.794 INFO:teuthology.orchestra.run.smithi072.stderr:512 bytes copied, 0.000220118 s, 2.3 MB/s 2024-03-22T01:28:20.796 DEBUG:teuthology.orchestra.run.smithi072:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-22T01:28:20.854 DEBUG:teuthology.orchestra.run.smithi072:> stat /dev/vg_nvme/lv_4 2024-03-22T01:28:20.910 INFO:teuthology.orchestra.run.smithi072.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-22T01:28:20.911 INFO:teuthology.orchestra.run.smithi072.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-22T01:28:20.911 INFO:teuthology.orchestra.run.smithi072.stdout:Device: 5h/5d Inode: 720 Links: 1 2024-03-22T01:28:20.911 INFO:teuthology.orchestra.run.smithi072.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-22T01:28:20.911 INFO:teuthology.orchestra.run.smithi072.stdout:Context: system_u:object_r:device_t:s0 2024-03-22T01:28:20.911 INFO:teuthology.orchestra.run.smithi072.stdout:Access: 2024-03-22 01:27:24.702460070 +0000 2024-03-22T01:28:20.911 INFO:teuthology.orchestra.run.smithi072.stdout:Modify: 2024-03-22 01:22:03.395993833 +0000 2024-03-22T01:28:20.911 INFO:teuthology.orchestra.run.smithi072.stdout:Change: 2024-03-22 01:22:03.395993833 +0000 2024-03-22T01:28:20.911 INFO:teuthology.orchestra.run.smithi072.stdout: Birth: 2024-03-22 01:22:03.395993833 +0000 2024-03-22T01:28:20.912 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-22T01:28:20.976 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records in 2024-03-22T01:28:20.977 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records out 2024-03-22T01:28:20.977 INFO:teuthology.orchestra.run.smithi072.stderr:512 bytes copied, 0.000215431 s, 2.4 MB/s 2024-03-22T01:28:20.978 DEBUG:teuthology.orchestra.run.smithi072:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-22T01:28:21.035 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-22T01:28:21.035 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/scratch_devs of=/dev/stdout 2024-03-22T01:28:21.054 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-22T01:28:21.054 DEBUG:teuthology.orchestra.run.smithi089:> stat /dev/vg_nvme/lv_1 2024-03-22T01:28:21.111 INFO:teuthology.orchestra.run.smithi089.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-22T01:28:21.111 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-22T01:28:21.112 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 5h/5d Inode: 661 Links: 1 2024-03-22T01:28:21.112 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-22T01:28:21.112 INFO:teuthology.orchestra.run.smithi089.stdout:Context: system_u:object_r:device_t:s0 2024-03-22T01:28:21.112 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 2024-03-22 01:28:06.920700246 +0000 2024-03-22T01:28:21.112 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2024-03-22 01:22:30.090674218 +0000 2024-03-22T01:28:21.112 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2024-03-22 01:22:30.090674218 +0000 2024-03-22T01:28:21.112 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: 2024-03-22 01:22:30.090674218 +0000 2024-03-22T01:28:21.113 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-22T01:28:21.181 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records in 2024-03-22T01:28:21.182 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records out 2024-03-22T01:28:21.182 INFO:teuthology.orchestra.run.smithi089.stderr:512 bytes copied, 0.000352469 s, 1.5 MB/s 2024-03-22T01:28:21.183 DEBUG:teuthology.orchestra.run.smithi089:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-22T01:28:21.240 DEBUG:teuthology.orchestra.run.smithi089:> stat /dev/vg_nvme/lv_2 2024-03-22T01:28:21.296 INFO:teuthology.orchestra.run.smithi089.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-22T01:28:21.296 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-22T01:28:21.296 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 5h/5d Inode: 680 Links: 1 2024-03-22T01:28:21.296 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-22T01:28:21.296 INFO:teuthology.orchestra.run.smithi089.stdout:Context: system_u:object_r:device_t:s0 2024-03-22T01:28:21.296 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 2024-03-22 01:28:06.920700246 +0000 2024-03-22T01:28:21.296 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2024-03-22 01:22:30.426666334 +0000 2024-03-22T01:28:21.296 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2024-03-22 01:22:30.426666334 +0000 2024-03-22T01:28:21.296 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: 2024-03-22 01:22:30.426666334 +0000 2024-03-22T01:28:21.297 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-22T01:28:21.363 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records in 2024-03-22T01:28:21.364 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records out 2024-03-22T01:28:21.364 INFO:teuthology.orchestra.run.smithi089.stderr:512 bytes copied, 0.00020483 s, 2.5 MB/s 2024-03-22T01:28:21.365 DEBUG:teuthology.orchestra.run.smithi089:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-22T01:28:21.421 DEBUG:teuthology.orchestra.run.smithi089:> stat /dev/vg_nvme/lv_3 2024-03-22T01:28:21.477 INFO:teuthology.orchestra.run.smithi089.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-22T01:28:21.477 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-22T01:28:21.477 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 5h/5d Inode: 699 Links: 1 2024-03-22T01:28:21.477 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-22T01:28:21.477 INFO:teuthology.orchestra.run.smithi089.stdout:Context: system_u:object_r:device_t:s0 2024-03-22T01:28:21.477 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 2024-03-22 01:28:06.920700246 +0000 2024-03-22T01:28:21.478 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2024-03-22 01:22:30.770658261 +0000 2024-03-22T01:28:21.478 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2024-03-22 01:22:30.770658261 +0000 2024-03-22T01:28:21.478 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: 2024-03-22 01:22:30.770658261 +0000 2024-03-22T01:28:21.478 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-22T01:28:21.542 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:21 smithi089 ceph-mon[35770]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi072=x;smithi089=y;smithi098=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:28:21.542 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:21 smithi089 ceph-mon[35770]: Saving service mgr spec with placement smithi072=x;smithi089=y;smithi098=z;count:3 2024-03-22T01:28:21.542 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:21 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:21.542 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:21 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:21.542 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:21 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:28:21.544 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records in 2024-03-22T01:28:21.544 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records out 2024-03-22T01:28:21.544 INFO:teuthology.orchestra.run.smithi089.stderr:512 bytes copied, 0.000236371 s, 2.2 MB/s 2024-03-22T01:28:21.545 DEBUG:teuthology.orchestra.run.smithi089:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-22T01:28:21.603 DEBUG:teuthology.orchestra.run.smithi089:> stat /dev/vg_nvme/lv_4 2024-03-22T01:28:21.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:21 smithi072 ceph-mon[32414]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi072=x;smithi089=y;smithi098=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:28:21.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:21 smithi072 ceph-mon[32414]: Saving service mgr spec with placement smithi072=x;smithi089=y;smithi098=z;count:3 2024-03-22T01:28:21.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:21 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:21.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:21 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:21.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:21 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:28:21.659 INFO:teuthology.orchestra.run.smithi089.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-22T01:28:21.659 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-22T01:28:21.659 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 5h/5d Inode: 719 Links: 1 2024-03-22T01:28:21.659 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-22T01:28:21.659 INFO:teuthology.orchestra.run.smithi089.stdout:Context: system_u:object_r:device_t:s0 2024-03-22T01:28:21.659 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 2024-03-22 01:28:06.921700222 +0000 2024-03-22T01:28:21.659 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2024-03-22 01:22:31.116650142 +0000 2024-03-22T01:28:21.659 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2024-03-22 01:22:31.116650142 +0000 2024-03-22T01:28:21.659 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: 2024-03-22 01:22:31.116650142 +0000 2024-03-22T01:28:21.659 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-22T01:28:21.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:21 smithi098 ceph-mon[36468]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi072=x;smithi089=y;smithi098=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:28:21.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:21 smithi098 ceph-mon[36468]: Saving service mgr spec with placement smithi072=x;smithi089=y;smithi098=z;count:3 2024-03-22T01:28:21.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:21 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:21.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:21 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:21.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:21 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:28:21.724 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records in 2024-03-22T01:28:21.725 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records out 2024-03-22T01:28:21.725 INFO:teuthology.orchestra.run.smithi089.stderr:512 bytes copied, 0.000325022 s, 1.6 MB/s 2024-03-22T01:28:21.726 DEBUG:teuthology.orchestra.run.smithi089:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-22T01:28:21.781 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-22T01:28:21.782 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/scratch_devs of=/dev/stdout 2024-03-22T01:28:21.797 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-22T01:28:21.797 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_1 2024-03-22T01:28:21.851 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-22T01:28:21.851 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-22T01:28:21.851 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-03-22T01:28:21.852 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-22T01:28:21.852 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-03-22T01:28:21.852 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-03-22 01:27:57.422034758 +0000 2024-03-22T01:28:21.852 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-03-22 01:22:10.139087159 +0000 2024-03-22T01:28:21.852 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-03-22 01:22:10.139087159 +0000 2024-03-22T01:28:21.852 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-03-22 01:22:10.139087159 +0000 2024-03-22T01:28:21.852 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-22T01:28:21.918 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-03-22T01:28:21.918 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-03-22T01:28:21.918 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.00019732 s, 2.6 MB/s 2024-03-22T01:28:21.923 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-22T01:28:21.978 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_2 2024-03-22T01:28:22.032 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-22T01:28:22.032 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-22T01:28:22.032 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 681 Links: 1 2024-03-22T01:28:22.032 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-22T01:28:22.032 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-03-22T01:28:22.032 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-03-22 01:27:57.423034743 +0000 2024-03-22T01:28:22.032 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-03-22 01:22:10.504081689 +0000 2024-03-22T01:28:22.032 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-03-22 01:22:10.504081689 +0000 2024-03-22T01:28:22.032 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-03-22 01:22:10.504081689 +0000 2024-03-22T01:28:22.032 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-22T01:28:22.098 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-03-22T01:28:22.099 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-03-22T01:28:22.099 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000195419 s, 2.6 MB/s 2024-03-22T01:28:22.101 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-22T01:28:22.156 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_3 2024-03-22T01:28:22.213 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-22T01:28:22.213 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-22T01:28:22.213 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 700 Links: 1 2024-03-22T01:28:22.213 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-22T01:28:22.213 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-03-22T01:28:22.213 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-03-22 01:27:57.423034743 +0000 2024-03-22T01:28:22.213 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-03-22 01:22:10.833076759 +0000 2024-03-22T01:28:22.213 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-03-22 01:22:10.833076759 +0000 2024-03-22T01:28:22.213 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-03-22 01:22:10.833076759 +0000 2024-03-22T01:28:22.214 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-22T01:28:22.278 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-03-22T01:28:22.278 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-03-22T01:28:22.278 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000197382 s, 2.6 MB/s 2024-03-22T01:28:22.279 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-22T01:28:22.335 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_4 2024-03-22T01:28:22.390 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-22T01:28:22.391 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-22T01:28:22.391 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 719 Links: 1 2024-03-22T01:28:22.391 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-22T01:28:22.391 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-03-22T01:28:22.391 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-03-22 01:27:57.423034743 +0000 2024-03-22T01:28:22.391 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-03-22 01:22:11.179071574 +0000 2024-03-22T01:28:22.391 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-03-22 01:22:11.179071574 +0000 2024-03-22T01:28:22.391 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-03-22 01:22:11.179071574 +0000 2024-03-22T01:28:22.391 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-22T01:28:22.456 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-03-22T01:28:22.457 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-03-22T01:28:22.457 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000210242 s, 2.4 MB/s 2024-03-22T01:28:22.458 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-22T01:28:22.510 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:22 smithi098 ceph-mon[36468]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:22.510 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:22 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:28:22.510 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:22 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:28:22.510 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:22 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:22.510 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:22 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T01:28:22.510 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:22 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-22T01:28:22.510 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:22 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-22T01:28:22.511 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:22 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:28:22.513 INFO:tasks.cephadm:Deploying osd.0 on smithi072 with /dev/vg_nvme/lv_4... 2024-03-22T01:28:22.513 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-22T01:28:22.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:22 smithi072 ceph-mon[32414]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:22.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:22 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:28:22.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:22 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:28:22.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:22 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:22.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:22 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T01:28:22.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:22 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-22T01:28:22.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:22 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-22T01:28:22.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:22 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:28:22.757 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:28:22.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:22 smithi089 ceph-mon[35770]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:22.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:22 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:28:22.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:22 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:28:22.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:22 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:22.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:22 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T01:28:22.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:22 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-22T01:28:22.859 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:22 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-22T01:28:22.859 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:22 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:28:23.493 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:23 smithi098 ceph-mon[36468]: Deploying daemon mgr.z on smithi098 2024-03-22T01:28:23.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:23 smithi072 ceph-mon[32414]: Deploying daemon mgr.z on smithi098 2024-03-22T01:28:23.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:23 smithi089 ceph-mon[35770]: Deploying daemon mgr.z on smithi098 2024-03-22T01:28:23.878 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:28:23 smithi098 systemd[1]: Starting Ceph mgr.z for 022a99bc-e7eb-11ee-95cd-87774f69a715... 2024-03-22T01:28:24.173 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:28:23 smithi098 podman[38304]: 2024-03-22 01:28:23.782493079 +0000 UTC m=+0.017630393 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T01:28:24.174 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:28:23 smithi098 podman[38304]: 2024-03-22 01:28:23.883182813 +0000 UTC m=+0.118320117 container create 63922caaa5007c1d8aae8b8b237d96785f7ac051bb1cb0e4be13f40c31bc6e58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-z, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-22T01:28:24.174 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:28:24 smithi098 podman[38304]: 2024-03-22 01:28:24.094900842 +0000 UTC m=+0.330038147 container init 63922caaa5007c1d8aae8b8b237d96785f7ac051bb1cb0e4be13f40c31bc6e58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-z, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-22T01:28:24.174 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:28:24 smithi098 podman[38304]: 2024-03-22 01:28:24.09738535 +0000 UTC m=+0.332522651 container start 63922caaa5007c1d8aae8b8b237d96785f7ac051bb1cb0e4be13f40c31bc6e58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-z, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-22T01:28:24.174 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:28:24 smithi098 ceph-mgr[38323]: -- 172.21.15.98:0/822785564 <== mon.1 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5620e97205a0 con 0x5620e96f5000 2024-03-22T01:28:24.447 INFO:teuthology.orchestra.run.smithi072.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-22T01:28:24.447 INFO:teuthology.orchestra.run.smithi072.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-22T01:28:24.447 INFO:teuthology.orchestra.run.smithi072.stderr: stderr: 10+0 records in 2024-03-22T01:28:24.447 INFO:teuthology.orchestra.run.smithi072.stderr:10+0 records out 2024-03-22T01:28:24.447 INFO:teuthology.orchestra.run.smithi072.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114067 s, 919 MB/s 2024-03-22T01:28:24.447 INFO:teuthology.orchestra.run.smithi072.stderr:--> Zapping successful for: 2024-03-22T01:28:24.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:24 smithi072 ceph-mon[32414]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:24.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:24 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:24.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:24 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:24.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:24 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:24.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:24 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T01:28:24.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:24 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-22T01:28:24.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:24 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-22T01:28:24.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:24 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:28:24.640 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:28:24 smithi098 bash[38304]: 63922caaa5007c1d8aae8b8b237d96785f7ac051bb1cb0e4be13f40c31bc6e58 2024-03-22T01:28:24.640 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:28:24 smithi098 systemd[1]: Started Ceph mgr.z for 022a99bc-e7eb-11ee-95cd-87774f69a715. 2024-03-22T01:28:24.640 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:24 smithi098 ceph-mon[36468]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:24.640 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:24 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:24.640 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:24 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:24.640 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:24 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:24.640 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:24 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T01:28:24.640 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:24 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-22T01:28:24.641 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:24 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-22T01:28:24.641 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:24 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:28:24.758 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:24 smithi089 ceph-mon[35770]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:24.758 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:24 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:24.759 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:24 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:24.759 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:24 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:24.759 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:24 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T01:28:24.759 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:24 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-22T01:28:24.759 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:24 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-22T01:28:24.759 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:24 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:28:24.921 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:28:24 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-z[38319]: 2024-03-22T01:28:24.640+0000 7fa26c0aa200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-22T01:28:24.921 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:28:24 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-z[38319]: 2024-03-22T01:28:24.732+0000 7fa26c0aa200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-22T01:28:25.000 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi072:vg_nvme/lv_4 2024-03-22T01:28:25.233 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:28:25.515 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:25 smithi098 ceph-mon[36468]: Deploying daemon mgr.y on smithi089 2024-03-22T01:28:25.515 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:28:25 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-z[38319]: 2024-03-22T01:28:25.375+0000 7fa26c0aa200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-22T01:28:25.516 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:28:25 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-z[38319]: 2024-03-22T01:28:25.470+0000 7fa26c0aa200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-22T01:28:25.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:25 smithi072 ceph-mon[32414]: Deploying daemon mgr.y on smithi089 2024-03-22T01:28:25.751 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:25 smithi089 ceph-mon[35770]: Deploying daemon mgr.y on smithi089 2024-03-22T01:28:25.800 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:28:25 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-z[38319]: 2024-03-22T01:28:25.617+0000 7fa26c0aa200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-22T01:28:25.800 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:28:25 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-z[38319]: 2024-03-22T01:28:25.716+0000 7fa26c0aa200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-22T01:28:25.800 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:28:25 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-z[38319]: 2024-03-22T01:28:25.801+0000 7fa26c0aa200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-22T01:28:26.358 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:28:26 smithi089 systemd[1]: Starting Ceph mgr.y for 022a99bc-e7eb-11ee-95cd-87774f69a715... 2024-03-22T01:28:26.363 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:28:26 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-z[38319]: 2024-03-22T01:28:26.070+0000 7fa26c0aa200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-22T01:28:26.363 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:28:26 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-z[38319]: 2024-03-22T01:28:26.181+0000 7fa26c0aa200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-22T01:28:26.363 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:28:26 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-z[38319]: 2024-03-22T01:28:26.268+0000 7fa26c0aa200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-22T01:28:26.363 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:28:26 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-z[38319]: 2024-03-22T01:28:26.364+0000 7fa26c0aa200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-22T01:28:26.552 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:26 smithi072 ceph-mon[32414]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:26.647 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:26 smithi089 ceph-mon[35770]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:26.647 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:28:26 smithi089 podman[37149]: 2024-03-22 01:28:26.320781827 +0000 UTC m=+0.018100464 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T01:28:26.648 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:28:26 smithi089 podman[37149]: 2024-03-22 01:28:26.438574391 +0000 UTC m=+0.135893069 container create 87cbf62f31f74cc9ac89d59c8eb9d251dee549f75efb912369ab06c22519f1d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-y, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0) 2024-03-22T01:28:26.648 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:28:26 smithi089 podman[37149]: 2024-03-22 01:28:26.649403171 +0000 UTC m=+0.346721799 container init 87cbf62f31f74cc9ac89d59c8eb9d251dee549f75efb912369ab06c22519f1d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0) 2024-03-22T01:28:26.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:26 smithi098 ceph-mon[36468]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:26.671 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:28:26 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-z[38319]: 2024-03-22T01:28:26.543+0000 7fa26c0aa200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-22T01:28:26.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:26.765+0000 7f8a3ac3b700 1 -- 172.21.15.72:0/430529561 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8a341026a0 msgr2=0x7f8a34102ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:28:26.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:26.765+0000 7f8a3ac3b700 1 --2- 172.21.15.72:0/430529561 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8a341026a0 0x7f8a34102ac0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8a300099c0 tx=0x7f8a30009cd0 comp rx=0 tx=0).stop 2024-03-22T01:28:26.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:26.765+0000 7f8a3ac3b700 1 -- 172.21.15.72:0/430529561 shutdown_connections 2024-03-22T01:28:26.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:26.765+0000 7f8a3ac3b700 1 --2- 172.21.15.72:0/430529561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a34104240 0x7f8a3410aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:28:26.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:26.765+0000 7f8a3ac3b700 1 --2- 172.21.15.72:0/430529561 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8a341026a0 0x7f8a34102ac0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:28:26.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:26.765+0000 7f8a3ac3b700 1 --2- 172.21.15.72:0/430529561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a341038a0 0x7f8a34103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:28:26.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:26.765+0000 7f8a3ac3b700 1 -- 172.21.15.72:0/430529561 >> 172.21.15.72:0/430529561 conn(0x7f8a340fdc60 msgr2=0x7f8a34100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:28:26.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:26.765+0000 7f8a3ac3b700 1 -- 172.21.15.72:0/430529561 shutdown_connections 2024-03-22T01:28:26.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:26.765+0000 7f8a3ac3b700 1 -- 172.21.15.72:0/430529561 wait complete. 2024-03-22T01:28:26.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:26.765+0000 7f8a3ac3b700 1 Processor -- start 2024-03-22T01:28:26.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:26.765+0000 7f8a3ac3b700 1 -- start start 2024-03-22T01:28:26.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:26.766+0000 7f8a3ac3b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a341026a0 0x7f8a341988b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:28:26.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:26.766+0000 7f8a3ac3b700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8a341038a0 0x7f8a34198df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:28:26.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:26.766+0000 7f8a3ac3b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a34104240 0x7f8a3419fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:28:26.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:26.766+0000 7f8a3ac3b700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f8a3410e290 con 0x7f8a341038a0 2024-03-22T01:28:26.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:26.766+0000 7f8a3ac3b700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8a3410e110 con 0x7f8a341026a0 2024-03-22T01:28:26.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:26.766+0000 7f8a3ac3b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8a3410e410 con 0x7f8a34104240 2024-03-22T01:28:26.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:26.766+0000 7f8a2bfff700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8a341038a0 0x7f8a34198df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:28:26.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:26.766+0000 7f8a2bfff700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8a341038a0 0x7f8a34198df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54000/0 (socket says 172.21.15.72:54000) 2024-03-22T01:28:26.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:26.766+0000 7f8a2bfff700 1 -- 172.21.15.72:0/3980239210 learned_addr learned my addr 172.21.15.72:0/3980239210 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:28:26.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:26.766+0000 7f8a389d7700 1 --2- 172.21.15.72:0/3980239210 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a341026a0 0x7f8a341988b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:28:26.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:26.766+0000 7f8a391d8700 1 --2- 172.21.15.72:0/3980239210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a34104240 0x7f8a3419fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:28:26.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:26.766+0000 7f8a2bfff700 1 -- 172.21.15.72:0/3980239210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a34104240 msgr2=0x7f8a3419fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:28:26.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:26.766+0000 7f8a2bfff700 1 --2- 172.21.15.72:0/3980239210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a34104240 0x7f8a3419fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:28:26.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:26.766+0000 7f8a2bfff700 1 -- 172.21.15.72:0/3980239210 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a341026a0 msgr2=0x7f8a341988b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:28:26.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:26.766+0000 7f8a2bfff700 1 --2- 172.21.15.72:0/3980239210 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a341026a0 0x7f8a341988b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:28:26.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:26.766+0000 7f8a2bfff700 1 -- 172.21.15.72:0/3980239210 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a341a0510 con 0x7f8a341038a0 2024-03-22T01:28:26.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:26.766+0000 7f8a2bfff700 1 --2- 172.21.15.72:0/3980239210 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8a341038a0 0x7f8a34198df0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8a2400d9b0 tx=0x7f8a2400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:28:26.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:26.766+0000 7f8a29ffb700 1 -- 172.21.15.72:0/3980239210 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a24017070 con 0x7f8a341038a0 2024-03-22T01:28:26.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:26.766+0000 7f8a29ffb700 1 -- 172.21.15.72:0/3980239210 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a24005020 con 0x7f8a341038a0 2024-03-22T01:28:26.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:26.766+0000 7f8a3ac3b700 1 -- 172.21.15.72:0/3980239210 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a341a0800 con 0x7f8a341038a0 2024-03-22T01:28:26.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:26.766+0000 7f8a29ffb700 1 -- 172.21.15.72:0/3980239210 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a24009a90 con 0x7f8a341038a0 2024-03-22T01:28:26.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:26.767+0000 7f8a3ac3b700 1 -- 172.21.15.72:0/3980239210 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a341a0d40 con 0x7f8a341038a0 2024-03-22T01:28:26.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:26.767+0000 7f8a29ffb700 1 -- 172.21.15.72:0/3980239210 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 13) ==== 46350+0+0 (secure 0 0 0) 0x7f8a24009c30 con 0x7f8a341038a0 2024-03-22T01:28:26.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:26.767+0000 7f8a29ffb700 1 --2- 172.21.15.72:0/3980239210 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f8a1c039460 0x7f8a1c03b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:28:26.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:26.767+0000 7f8a29ffb700 1 -- 172.21.15.72:0/3980239210 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f8a2404c280 con 0x7f8a341038a0 2024-03-22T01:28:26.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:26.768+0000 7f8a389d7700 1 --2- 172.21.15.72:0/3980239210 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f8a1c039460 0x7f8a1c03b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:28:26.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:26.768+0000 7f8a3ac3b700 1 -- 172.21.15.72:0/3980239210 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a34069340 con 0x7f8a341038a0 2024-03-22T01:28:26.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:26.768+0000 7f8a389d7700 1 --2- 172.21.15.72:0/3980239210 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f8a1c039460 0x7f8a1c03b920 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8a3000b580 tx=0x7f8a30005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:28:26.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:26.773+0000 7f8a29ffb700 1 -- 172.21.15.72:0/3980239210 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8a24020020 con 0x7f8a341038a0 2024-03-22T01:28:27.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:27.004+0000 7f8a3ac3b700 1 -- 172.21.15.72:0/3980239210 --> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi072:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f8a34062ac0 con 0x7f8a1c039460 2024-03-22T01:28:27.108 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:28:26 smithi089 podman[37149]: 2024-03-22 01:28:26.652019217 +0000 UTC m=+0.349337855 container start 87cbf62f31f74cc9ac89d59c8eb9d251dee549f75efb912369ab06c22519f1d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-y, ceph=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-22T01:28:27.108 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:28:26 smithi089 ceph-mgr[37168]: -- 172.21.15.89:0/497207002 <== mon.2 v2:172.21.15.89:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a5f74fc5a0 con 0x55a5f74d1000 2024-03-22T01:28:27.108 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:28:26 smithi089 bash[37149]: 87cbf62f31f74cc9ac89d59c8eb9d251dee549f75efb912369ab06c22519f1d0 2024-03-22T01:28:27.108 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:28:26 smithi089 systemd[1]: Started Ceph mgr.y for 022a99bc-e7eb-11ee-95cd-87774f69a715. 2024-03-22T01:28:27.607 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:28:27 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-y[37164]: 2024-03-22T01:28:27.219+0000 7fd1d3d7f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-22T01:28:27.608 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:28:27 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-y[37164]: 2024-03-22T01:28:27.316+0000 7fd1d3d7f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-22T01:28:27.849 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:28:27 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-z[38319]: 2024-03-22T01:28:27.556+0000 7fa26c0aa200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-22T01:28:27.849 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:28:27 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-z[38319]: 2024-03-22T01:28:27.697+0000 7fa26c0aa200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-22T01:28:27.850 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:27 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:27.850 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:27 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:27.850 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:27 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:27.850 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:27 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:27.850 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:27 smithi098 ceph-mon[36468]: from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi072:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:28:27.850 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:27 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T01:28:27.850 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:27 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T01:28:27.850 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:27 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:28:27.850 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:27 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:28:27.850 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:27 smithi098 ceph-mon[36468]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:27.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:27 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:27.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:27 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:27.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:27 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:27.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:27 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:27.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:27 smithi072 ceph-mon[32414]: from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi072:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:28:27.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:27 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T01:28:27.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:27 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T01:28:27.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:27 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:28:27.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:27 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:28:27.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:27 smithi072 ceph-mon[32414]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:28.107 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:28:28 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-y[37164]: 2024-03-22T01:28:28.015+0000 7fd1d3d7f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-22T01:28:28.107 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:28:28 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-y[37164]: 2024-03-22T01:28:28.107+0000 7fd1d3d7f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-22T01:28:28.107 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:27 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:28.107 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:27 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:28.107 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:27 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:28.107 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:27 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:28.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:27 smithi089 ceph-mon[35770]: from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi072:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:28:28.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:27 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T01:28:28.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:27 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T01:28:28.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:27 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:28:28.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:27 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:28:28.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:27 smithi089 ceph-mon[35770]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:28.130 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:28:28 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-z[38319]: 2024-03-22T01:28:28.041+0000 7fa26c0aa200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-22T01:28:28.130 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:28:28 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-z[38319]: 2024-03-22T01:28:28.130+0000 7fa26c0aa200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-22T01:28:28.358 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:28:28 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-y[37164]: 2024-03-22T01:28:28.252+0000 7fd1d3d7f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-22T01:28:28.358 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:28:28 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-y[37164]: 2024-03-22T01:28:28.352+0000 7fd1d3d7f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-22T01:28:28.421 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:28:28 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-z[38319]: 2024-03-22T01:28:28.328+0000 7fa26c0aa200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-22T01:28:28.710 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:28:28 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-y[37164]: 2024-03-22T01:28:28.437+0000 7fd1d3d7f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-22T01:28:28.710 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:28:28 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-y[37164]: 2024-03-22T01:28:28.710+0000 7fd1d3d7f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-22T01:28:28.800 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:28:28 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-z[38319]: 2024-03-22T01:28:28.520+0000 7fa26c0aa200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-22T01:28:28.996 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:28:28 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-y[37164]: 2024-03-22T01:28:28.819+0000 7fd1d3d7f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-22T01:28:28.996 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:28:28 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-y[37164]: 2024-03-22T01:28:28.903+0000 7fd1d3d7f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-22T01:28:28.996 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:28:28 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-y[37164]: 2024-03-22T01:28:28.996+0000 7fd1d3d7f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-22T01:28:29.057 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:28:28 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-z[38319]: 2024-03-22T01:28:28.801+0000 7fa26c0aa200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-22T01:28:29.312 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:28:29 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-z[38319]: 2024-03-22T01:28:29.058+0000 7fa26c0aa200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-22T01:28:29.312 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:28:29 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-z[38319]: 2024-03-22T01:28:29.293+0000 7fa26c0aa200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-22T01:28:29.312 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:28:29 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-y[37164]: 2024-03-22T01:28:29.171+0000 7fd1d3d7f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-22T01:28:29.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:29 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:29.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:29 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:29.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:29 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:29.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:29 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:29.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:29 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:28:29.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:29 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:28:29.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:29 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:29.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:29 smithi089 ceph-mon[35770]: Reconfiguring mgr.x (unknown last config time)... 2024-03-22T01:28:29.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:29 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T01:28:29.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:29 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-22T01:28:29.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:29 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:28:29.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:29 smithi089 ceph-mon[35770]: Reconfiguring daemon mgr.x on smithi072 2024-03-22T01:28:29.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:29 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:29.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:29 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:29.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:29 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:29.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:29 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:29.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:29 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:28:29.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:29 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:28:29.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:29 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:29.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:29 smithi072 ceph-mon[32414]: Reconfiguring mgr.x (unknown last config time)... 2024-03-22T01:28:29.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:29 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T01:28:29.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:29 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-22T01:28:29.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:29 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:28:29.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:29 smithi072 ceph-mon[32414]: Reconfiguring daemon mgr.x on smithi072 2024-03-22T01:28:29.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:29 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:29.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:29 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:29.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:29 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:29.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:29 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:29.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:29 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:28:29.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:29 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:28:29.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:29 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:29.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:29 smithi098 ceph-mon[36468]: Reconfiguring mgr.x (unknown last config time)... 2024-03-22T01:28:29.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:29 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T01:28:29.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:29 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-22T01:28:29.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:29 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:28:29.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:29 smithi098 ceph-mon[36468]: Reconfiguring daemon mgr.x on smithi072 2024-03-22T01:28:30.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:30 smithi098 ceph-mon[36468]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:30.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:30 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/2402272695' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "16c15bb0-db92-4191-9063-38b4890e81e3"}]: dispatch 2024-03-22T01:28:30.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:30 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/2402272695' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "16c15bb0-db92-4191-9063-38b4890e81e3"}]': finished 2024-03-22T01:28:30.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:30 smithi098 ceph-mon[36468]: osdmap e5: 1 total, 0 up, 1 in 2024-03-22T01:28:30.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:30 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T01:28:30.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:30 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/2357353834' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T01:28:30.422 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:28:30 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-z[38319]: 2024-03-22T01:28:30.087+0000 7fa26c0aa200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-22T01:28:30.593 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:30 smithi089 ceph-mon[35770]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:30.593 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:30 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/2402272695' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "16c15bb0-db92-4191-9063-38b4890e81e3"}]: dispatch 2024-03-22T01:28:30.593 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:30 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/2402272695' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "16c15bb0-db92-4191-9063-38b4890e81e3"}]': finished 2024-03-22T01:28:30.593 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:30 smithi089 ceph-mon[35770]: osdmap e5: 1 total, 0 up, 1 in 2024-03-22T01:28:30.593 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:30 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T01:28:30.593 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:30 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/2357353834' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T01:28:30.594 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:28:30 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-y[37164]: 2024-03-22T01:28:30.132+0000 7fd1d3d7f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-22T01:28:30.594 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:28:30 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-y[37164]: 2024-03-22T01:28:30.261+0000 7fd1d3d7f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-22T01:28:30.594 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:28:30 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-y[37164]: 2024-03-22T01:28:30.594+0000 7fd1d3d7f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-22T01:28:30.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:30 smithi072 ceph-mon[32414]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:30.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:30 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/2402272695' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "16c15bb0-db92-4191-9063-38b4890e81e3"}]: dispatch 2024-03-22T01:28:30.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:30 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/2402272695' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "16c15bb0-db92-4191-9063-38b4890e81e3"}]': finished 2024-03-22T01:28:30.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:30 smithi072 ceph-mon[32414]: osdmap e5: 1 total, 0 up, 1 in 2024-03-22T01:28:30.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:30 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T01:28:30.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:30 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/2357353834' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T01:28:30.858 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:28:30 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-y[37164]: 2024-03-22T01:28:30.680+0000 7fd1d3d7f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-22T01:28:30.921 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:28:30 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-z[38319]: 2024-03-22T01:28:30.650+0000 7fa26c0aa200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-22T01:28:30.921 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:28:30 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-z[38319]: 2024-03-22T01:28:30.746+0000 7fa26c0aa200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-22T01:28:31.333 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:28:30 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-y[37164]: 2024-03-22T01:28:30.874+0000 7fd1d3d7f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-22T01:28:31.333 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:28:31 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-y[37164]: 2024-03-22T01:28:31.055+0000 7fd1d3d7f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-22T01:28:31.421 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:28:30 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-z[38319]: 2024-03-22T01:28:30.957+0000 7fa26c0aa200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-22T01:28:31.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:31.424+0000 7f8a29ffb700 1 -- 172.21.15.72:0/3980239210 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mgrmap(e 14) ==== 92074+0+0 (secure 0 0 0) 0x7f8a2401b400 con 0x7f8a341038a0 2024-03-22T01:28:31.592 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:28:31 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-y[37164]: 2024-03-22T01:28:31.334+0000 7fd1d3d7f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-22T01:28:31.593 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:28:31 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-y[37164]: 2024-03-22T01:28:31.593+0000 7fd1d3d7f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-22T01:28:31.593 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:31 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:31.593 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:31 smithi089 ceph-mon[35770]: Standby manager daemon z started 2024-03-22T01:28:31.593 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:31 smithi089 ceph-mon[35770]: from='mgr.? 172.21.15.98:0/1395886277' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-22T01:28:31.593 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:31 smithi089 ceph-mon[35770]: from='mgr.? 172.21.15.98:0/1395886277' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-22T01:28:31.593 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:31 smithi089 ceph-mon[35770]: from='mgr.? 172.21.15.98:0/1395886277' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-22T01:28:31.593 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:31 smithi089 ceph-mon[35770]: from='mgr.? 172.21.15.98:0/1395886277' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-22T01:28:31.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:31 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:31.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:31 smithi098 ceph-mon[36468]: Standby manager daemon z started 2024-03-22T01:28:31.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:31 smithi098 ceph-mon[36468]: from='mgr.? 172.21.15.98:0/1395886277' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-22T01:28:31.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:31 smithi098 ceph-mon[36468]: from='mgr.? 172.21.15.98:0/1395886277' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-22T01:28:31.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:31 smithi098 ceph-mon[36468]: from='mgr.? 172.21.15.98:0/1395886277' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-22T01:28:31.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:31 smithi098 ceph-mon[36468]: from='mgr.? 172.21.15.98:0/1395886277' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-22T01:28:31.857 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:28:31 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-y[37164]: 2024-03-22T01:28:31.825+0000 7fd1d3d7f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-22T01:28:31.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:31 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:31.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:31 smithi072 ceph-mon[32414]: Standby manager daemon z started 2024-03-22T01:28:31.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:31 smithi072 ceph-mon[32414]: from='mgr.? 172.21.15.98:0/1395886277' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-22T01:28:31.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:31 smithi072 ceph-mon[32414]: from='mgr.? 172.21.15.98:0/1395886277' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-22T01:28:31.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:31 smithi072 ceph-mon[32414]: from='mgr.? 172.21.15.98:0/1395886277' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-22T01:28:31.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:31 smithi072 ceph-mon[32414]: from='mgr.? 172.21.15.98:0/1395886277' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-22T01:28:32.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:32 smithi089 ceph-mon[35770]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:32.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:32 smithi089 ceph-mon[35770]: mgrmap e14: x(active, since 76s), standbys: z 2024-03-22T01:28:32.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:32 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-22T01:28:32.858 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:28:32 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-y[37164]: 2024-03-22T01:28:32.616+0000 7fd1d3d7f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-22T01:28:32.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:32 smithi072 ceph-mon[32414]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:32.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:32 smithi072 ceph-mon[32414]: mgrmap e14: x(active, since 76s), standbys: z 2024-03-22T01:28:32.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:32 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-22T01:28:32.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:32 smithi098 ceph-mon[36468]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:32.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:32 smithi098 ceph-mon[36468]: mgrmap e14: x(active, since 76s), standbys: z 2024-03-22T01:28:32.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:32 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-22T01:28:33.481 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:28:33 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-y[37164]: 2024-03-22T01:28:33.173+0000 7fd1d3d7f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-22T01:28:33.481 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:28:33 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-y[37164]: 2024-03-22T01:28:33.268+0000 7fd1d3d7f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-22T01:28:33.481 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:28:33 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-y[37164]: 2024-03-22T01:28:33.481+0000 7fd1d3d7f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-22T01:28:34.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:34.434+0000 7f8a29ffb700 1 -- 172.21.15.72:0/3980239210 <== mon.0 v2:172.21.15.72:3300/0 8 ==== mgrmap(e 15) ==== 137798+0+0 (secure 0 0 0) 0x7f8a24020ad0 con 0x7f8a341038a0 2024-03-22T01:28:34.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:34 smithi089 ceph-mon[35770]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:34.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:34 smithi089 ceph-mon[35770]: Standby manager daemon y started 2024-03-22T01:28:34.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:34 smithi089 ceph-mon[35770]: from='mgr.? 172.21.15.89:0/647959360' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-22T01:28:34.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:34 smithi089 ceph-mon[35770]: from='mgr.? 172.21.15.89:0/647959360' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-22T01:28:34.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:34 smithi089 ceph-mon[35770]: from='mgr.? 172.21.15.89:0/647959360' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-22T01:28:34.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:34 smithi089 ceph-mon[35770]: from='mgr.? 172.21.15.89:0/647959360' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-22T01:28:34.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:34 smithi072 ceph-mon[32414]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:34.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:34 smithi072 ceph-mon[32414]: Standby manager daemon y started 2024-03-22T01:28:34.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:34 smithi072 ceph-mon[32414]: from='mgr.? 172.21.15.89:0/647959360' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-22T01:28:34.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:34 smithi072 ceph-mon[32414]: from='mgr.? 172.21.15.89:0/647959360' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-22T01:28:34.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:34 smithi072 ceph-mon[32414]: from='mgr.? 172.21.15.89:0/647959360' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-22T01:28:34.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:34 smithi072 ceph-mon[32414]: from='mgr.? 172.21.15.89:0/647959360' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-22T01:28:34.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:34 smithi098 ceph-mon[36468]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:34.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:34 smithi098 ceph-mon[36468]: Standby manager daemon y started 2024-03-22T01:28:34.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:34 smithi098 ceph-mon[36468]: from='mgr.? 172.21.15.89:0/647959360' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-22T01:28:34.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:34 smithi098 ceph-mon[36468]: from='mgr.? 172.21.15.89:0/647959360' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-22T01:28:34.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:34 smithi098 ceph-mon[36468]: from='mgr.? 172.21.15.89:0/647959360' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-22T01:28:34.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:34 smithi098 ceph-mon[36468]: from='mgr.? 172.21.15.89:0/647959360' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-22T01:28:35.781 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:35 smithi072 ceph-mon[32414]: mgrmap e15: x(active, since 79s), standbys: z, y 2024-03-22T01:28:35.781 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:35 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-22T01:28:35.781 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:35 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:35.781 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:35 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:35.782 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:35 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:28:35.782 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:35 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:28:35.782 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:35 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:28:35.782 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:35 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:35.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:35 smithi089 ceph-mon[35770]: mgrmap e15: x(active, since 79s), standbys: z, y 2024-03-22T01:28:35.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:35 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-22T01:28:35.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:35 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:35.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:35 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:35.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:35 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:28:35.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:35 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:28:35.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:35 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:28:35.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:35 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:35.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:35 smithi098 ceph-mon[36468]: mgrmap e15: x(active, since 79s), standbys: z, y 2024-03-22T01:28:35.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:35 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-22T01:28:35.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:35 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:35.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:35 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:35.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:35 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:28:35.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:35 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:28:35.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:35 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:28:35.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:35 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:36.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:36 smithi072 ceph-mon[32414]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:36.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:36 smithi089 ceph-mon[35770]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:36.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:36 smithi098 ceph-mon[36468]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:37.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:37 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-22T01:28:37.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:37 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:28:37.871 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:37 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-22T01:28:37.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:37 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:28:37.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:37 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-22T01:28:37.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:37 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:28:38.748 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:38 smithi072 ceph-mon[32414]: Deploying daemon osd.0 on smithi072 2024-03-22T01:28:38.748 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:38 smithi072 ceph-mon[32414]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:38.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:38 smithi089 ceph-mon[35770]: Deploying daemon osd.0 on smithi072 2024-03-22T01:28:38.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:38 smithi089 ceph-mon[35770]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:38.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:38 smithi098 ceph-mon[36468]: Deploying daemon osd.0 on smithi072 2024-03-22T01:28:38.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:38 smithi098 ceph-mon[36468]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:40.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:40 smithi072 ceph-mon[32414]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:40.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:40 smithi089 ceph-mon[35770]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:40.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:40 smithi098 ceph-mon[36468]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:42.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:42 smithi072 ceph-mon[32414]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:42.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:42 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:42.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:42 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:42.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:42 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:28:42.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:42 smithi089 ceph-mon[35770]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:42.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:42 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:42.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:42 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:42.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:42 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:28:42.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:42 smithi098 ceph-mon[36468]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:42.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:42 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:42.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:42 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:42.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:42 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:28:44.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:44 smithi072 ceph-mon[32414]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:44.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:44 smithi089 ceph-mon[35770]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:44.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:44 smithi098 ceph-mon[36468]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:45.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:45.657+0000 7f8a29ffb700 1 -- 172.21.15.72:0/3980239210 <== mgr.14152 v2:172.21.15.72:6800/401403200 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f8a34062ac0 con 0x7f8a1c039460 2024-03-22T01:28:45.657 INFO:teuthology.orchestra.run.smithi072.stdout:Created osd(s) 0 on host 'smithi072' 2024-03-22T01:28:45.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:45.659+0000 7f8a3ac3b700 1 -- 172.21.15.72:0/3980239210 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f8a1c039460 msgr2=0x7f8a1c03b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:28:45.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:45.659+0000 7f8a3ac3b700 1 --2- 172.21.15.72:0/3980239210 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f8a1c039460 0x7f8a1c03b920 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8a3000b580 tx=0x7f8a30005fb0 comp rx=0 tx=0).stop 2024-03-22T01:28:45.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:45.659+0000 7f8a3ac3b700 1 -- 172.21.15.72:0/3980239210 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8a341038a0 msgr2=0x7f8a34198df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:28:45.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:45.659+0000 7f8a3ac3b700 1 --2- 172.21.15.72:0/3980239210 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8a341038a0 0x7f8a34198df0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8a2400d9b0 tx=0x7f8a2400dcc0 comp rx=0 tx=0).stop 2024-03-22T01:28:45.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:45.659+0000 7f8a3ac3b700 1 -- 172.21.15.72:0/3980239210 shutdown_connections 2024-03-22T01:28:45.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:45.659+0000 7f8a3ac3b700 1 --2- 172.21.15.72:0/3980239210 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f8a1c039460 0x7f8a1c03b920 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:28:45.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:45.659+0000 7f8a3ac3b700 1 --2- 172.21.15.72:0/3980239210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a34104240 0x7f8a3419fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:28:45.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:45.659+0000 7f8a3ac3b700 1 --2- 172.21.15.72:0/3980239210 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a341026a0 0x7f8a341988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:28:45.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:45.659+0000 7f8a3ac3b700 1 --2- 172.21.15.72:0/3980239210 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8a341038a0 0x7f8a34198df0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:28:45.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:45.659+0000 7f8a3ac3b700 1 -- 172.21.15.72:0/3980239210 >> 172.21.15.72:0/3980239210 conn(0x7f8a340fdc60 msgr2=0x7f8a34106c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:28:45.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:45.660+0000 7f8a3ac3b700 1 -- 172.21.15.72:0/3980239210 shutdown_connections 2024-03-22T01:28:45.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:28:45.660+0000 7f8a3ac3b700 1 -- 172.21.15.72:0/3980239210 wait complete. 2024-03-22T01:28:46.209 DEBUG:teuthology.orchestra.run.smithi072:osd.0> sudo journalctl -f -n 0 -u ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@osd.0.service 2024-03-22T01:28:46.211 INFO:tasks.cephadm:Deploying osd.1 on smithi089 with /dev/vg_nvme/lv_4... 2024-03-22T01:28:46.212 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-22T01:28:46.453 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.b/config 2024-03-22T01:28:46.575 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:46 smithi089 ceph-mon[35770]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:46.576 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:46 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:46.576 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:46 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:46.576 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:46 smithi089 ceph-mon[35770]: from='osd.0 [v2:172.21.15.72:6802/661906791,v1:172.21.15.72:6803/661906791]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-22T01:28:46.576 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:46 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:46.576 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:46 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:46.622 INFO:journalctl@ceph.osd.0.smithi072.stdout:Mar 22 01:28:46 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-0[44792]: 2024-03-22T01:28:46.306+0000 7ff05ae5e7c0 -1 osd.0 0 log_to_monitors true 2024-03-22T01:28:46.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:46 smithi072 ceph-mon[32414]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:46.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:46 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:46.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:46 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:46.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:46 smithi072 ceph-mon[32414]: from='osd.0 [v2:172.21.15.72:6802/661906791,v1:172.21.15.72:6803/661906791]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-22T01:28:46.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:46 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:46.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:46 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:46.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:46 smithi098 ceph-mon[36468]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:46.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:46 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:46.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:46 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:46.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:46 smithi098 ceph-mon[36468]: from='osd.0 [v2:172.21.15.72:6802/661906791,v1:172.21.15.72:6803/661906791]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-22T01:28:46.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:46 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:46.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:46 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:47.809 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:47 smithi089 ceph-mon[35770]: from='osd.0 [v2:172.21.15.72:6802/661906791,v1:172.21.15.72:6803/661906791]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-22T01:28:47.809 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:47 smithi089 ceph-mon[35770]: osdmap e6: 1 total, 0 up, 1 in 2024-03-22T01:28:47.809 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:47 smithi089 ceph-mon[35770]: from='osd.0 [v2:172.21.15.72:6802/661906791,v1:172.21.15.72:6803/661906791]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]: dispatch 2024-03-22T01:28:47.809 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:47 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T01:28:47.809 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:47 smithi089 ceph-mon[35770]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:47.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:47 smithi098 ceph-mon[36468]: from='osd.0 [v2:172.21.15.72:6802/661906791,v1:172.21.15.72:6803/661906791]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-22T01:28:47.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:47 smithi098 ceph-mon[36468]: osdmap e6: 1 total, 0 up, 1 in 2024-03-22T01:28:47.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:47 smithi098 ceph-mon[36468]: from='osd.0 [v2:172.21.15.72:6802/661906791,v1:172.21.15.72:6803/661906791]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]: dispatch 2024-03-22T01:28:47.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:47 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T01:28:47.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:47 smithi098 ceph-mon[36468]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:47.923 INFO:journalctl@ceph.osd.0.smithi072.stdout:Mar 22 01:28:47 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-0[44792]: 2024-03-22T01:28:47.674+0000 7ff0504c3700 -1 osd.0 0 waiting for initial osdmap 2024-03-22T01:28:47.923 INFO:journalctl@ceph.osd.0.smithi072.stdout:Mar 22 01:28:47 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-0[44792]: 2024-03-22T01:28:47.677+0000 7ff04c2b6700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-22T01:28:47.923 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:47 smithi072 ceph-mon[32414]: from='osd.0 [v2:172.21.15.72:6802/661906791,v1:172.21.15.72:6803/661906791]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-22T01:28:47.923 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:47 smithi072 ceph-mon[32414]: osdmap e6: 1 total, 0 up, 1 in 2024-03-22T01:28:47.923 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:47 smithi072 ceph-mon[32414]: from='osd.0 [v2:172.21.15.72:6802/661906791,v1:172.21.15.72:6803/661906791]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]: dispatch 2024-03-22T01:28:47.924 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:47 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T01:28:47.924 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:47 smithi072 ceph-mon[32414]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T01:28:48.386 INFO:teuthology.orchestra.run.smithi089.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-22T01:28:48.387 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-22T01:28:48.387 INFO:teuthology.orchestra.run.smithi089.stderr: stderr: 10+0 records in 2024-03-22T01:28:48.387 INFO:teuthology.orchestra.run.smithi089.stderr:10+0 records out 2024-03-22T01:28:48.387 INFO:teuthology.orchestra.run.smithi089.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113042 s, 928 MB/s 2024-03-22T01:28:48.387 INFO:teuthology.orchestra.run.smithi089.stderr:--> Zapping successful for: 2024-03-22T01:28:48.930 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:48 smithi072 ceph-mon[32414]: from='osd.0 [v2:172.21.15.72:6802/661906791,v1:172.21.15.72:6803/661906791]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]': finished 2024-03-22T01:28:48.931 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:48 smithi072 ceph-mon[32414]: osdmap e7: 1 total, 0 up, 1 in 2024-03-22T01:28:48.931 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:48 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T01:28:48.931 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:48 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T01:28:48.978 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:48 smithi089 ceph-mon[35770]: from='osd.0 [v2:172.21.15.72:6802/661906791,v1:172.21.15.72:6803/661906791]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]': finished 2024-03-22T01:28:48.978 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:48 smithi089 ceph-mon[35770]: osdmap e7: 1 total, 0 up, 1 in 2024-03-22T01:28:48.978 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:48 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T01:28:48.978 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:48 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T01:28:49.005 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi089:vg_nvme/lv_4 2024-03-22T01:28:49.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:48 smithi098 ceph-mon[36468]: from='osd.0 [v2:172.21.15.72:6802/661906791,v1:172.21.15.72:6803/661906791]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]': finished 2024-03-22T01:28:49.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:48 smithi098 ceph-mon[36468]: osdmap e7: 1 total, 0 up, 1 in 2024-03-22T01:28:49.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:48 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T01:28:49.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:48 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T01:28:49.243 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.b/config 2024-03-22T01:28:49.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:49 smithi072 ceph-mon[32414]: purged_snaps scrub starts 2024-03-22T01:28:49.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:49 smithi072 ceph-mon[32414]: purged_snaps scrub ok 2024-03-22T01:28:49.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:49 smithi072 ceph-mon[32414]: osd.0 [v2:172.21.15.72:6802/661906791,v1:172.21.15.72:6803/661906791] boot 2024-03-22T01:28:49.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:49 smithi072 ceph-mon[32414]: osdmap e8: 1 total, 1 up, 1 in 2024-03-22T01:28:49.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:49 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T01:28:49.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:49 smithi072 ceph-mon[32414]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-22T01:28:49.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:49 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:49.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:49 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:49.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:49 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:28:49.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:49 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:49.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:49 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:28:49.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:49 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:28:49.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:49 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:50.018 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:49 smithi089 ceph-mon[35770]: purged_snaps scrub starts 2024-03-22T01:28:50.018 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:49 smithi089 ceph-mon[35770]: purged_snaps scrub ok 2024-03-22T01:28:50.018 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:49 smithi089 ceph-mon[35770]: osd.0 [v2:172.21.15.72:6802/661906791,v1:172.21.15.72:6803/661906791] boot 2024-03-22T01:28:50.018 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:49 smithi089 ceph-mon[35770]: osdmap e8: 1 total, 1 up, 1 in 2024-03-22T01:28:50.018 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:49 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T01:28:50.018 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:49 smithi089 ceph-mon[35770]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-22T01:28:50.018 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:49 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:50.018 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:49 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:50.018 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:49 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:28:50.018 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:49 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:50.019 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:49 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:28:50.019 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:49 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:28:50.019 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:49 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:50.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:49 smithi098 ceph-mon[36468]: purged_snaps scrub starts 2024-03-22T01:28:50.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:49 smithi098 ceph-mon[36468]: purged_snaps scrub ok 2024-03-22T01:28:50.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:49 smithi098 ceph-mon[36468]: osd.0 [v2:172.21.15.72:6802/661906791,v1:172.21.15.72:6803/661906791] boot 2024-03-22T01:28:50.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:49 smithi098 ceph-mon[36468]: osdmap e8: 1 total, 1 up, 1 in 2024-03-22T01:28:50.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:49 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T01:28:50.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:49 smithi098 ceph-mon[36468]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-22T01:28:50.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:49 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:50.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:49 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:50.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:49 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:28:50.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:49 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:50.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:49 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:28:50.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:49 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:28:50.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:49 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:50.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:28:50.807+0000 7fee2b315700 1 -- 172.21.15.89:0/2145795774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fee24104250 msgr2=0x7fee2410ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:28:50.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:28:50.807+0000 7fee2b315700 1 --2- 172.21.15.89:0/2145795774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fee24104250 0x7fee2410ab00 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fee1c00b0d0 tx=0x7fee1c00b3e0 comp rx=0 tx=0).stop 2024-03-22T01:28:50.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:28:50.807+0000 7fee2b315700 1 -- 172.21.15.89:0/2145795774 shutdown_connections 2024-03-22T01:28:50.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:28:50.807+0000 7fee2b315700 1 --2- 172.21.15.89:0/2145795774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fee24104250 0x7fee2410ab00 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:28:50.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:28:50.807+0000 7fee2b315700 1 --2- 172.21.15.89:0/2145795774 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fee241026b0 0x7fee24102ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:28:50.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:28:50.807+0000 7fee2b315700 1 --2- 172.21.15.89:0/2145795774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee241038b0 0x7fee24103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:28:50.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:28:50.807+0000 7fee2b315700 1 -- 172.21.15.89:0/2145795774 >> 172.21.15.89:0/2145795774 conn(0x7fee240fdc90 msgr2=0x7fee241000b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:28:50.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:28:50.807+0000 7fee2b315700 1 -- 172.21.15.89:0/2145795774 shutdown_connections 2024-03-22T01:28:50.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:28:50.807+0000 7fee2b315700 1 -- 172.21.15.89:0/2145795774 wait complete. 2024-03-22T01:28:50.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:28:50.808+0000 7fee2b315700 1 Processor -- start 2024-03-22T01:28:50.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:28:50.808+0000 7fee2b315700 1 -- start start 2024-03-22T01:28:50.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:28:50.808+0000 7fee2b315700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fee241026b0 0x7fee24076420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:28:50.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:28:50.808+0000 7fee2b315700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee241038b0 0x7fee24074a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:28:50.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:28:50.808+0000 7fee2b315700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fee24104250 0x7fee24074fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:28:50.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:28:50.808+0000 7fee2b315700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fee2410e2a0 con 0x7fee24104250 2024-03-22T01:28:50.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:28:50.808+0000 7fee2b315700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fee2410e120 con 0x7fee241026b0 2024-03-22T01:28:50.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:28:50.808+0000 7fee2b315700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fee2410e420 con 0x7fee241038b0 2024-03-22T01:28:50.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:28:50.808+0000 7fee290b1700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fee241026b0 0x7fee24076420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:28:50.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:28:50.808+0000 7fee290b1700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fee241026b0 0x7fee24076420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60818/0 (socket says 172.21.15.89:60818) 2024-03-22T01:28:50.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:28:50.808+0000 7fee290b1700 1 -- 172.21.15.89:0/1860477096 learned_addr learned my addr 172.21.15.89:0/1860477096 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-22T01:28:50.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:28:50.808+0000 7fee298b2700 1 --2- 172.21.15.89:0/1860477096 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fee24104250 0x7fee24074fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:28:50.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:28:50.808+0000 7fee288b0700 1 --2- 172.21.15.89:0/1860477096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee241038b0 0x7fee24074a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:28:50.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:28:50.808+0000 7fee290b1700 1 -- 172.21.15.89:0/1860477096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee241038b0 msgr2=0x7fee24074a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:28:50.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:28:50.808+0000 7fee290b1700 1 --2- 172.21.15.89:0/1860477096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee241038b0 0x7fee24074a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:28:50.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:28:50.808+0000 7fee290b1700 1 -- 172.21.15.89:0/1860477096 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fee24104250 msgr2=0x7fee24074fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:28:50.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:28:50.808+0000 7fee290b1700 1 --2- 172.21.15.89:0/1860477096 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fee24104250 0x7fee24074fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:28:50.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:28:50.808+0000 7fee290b1700 1 -- 172.21.15.89:0/1860477096 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee24075860 con 0x7fee241026b0 2024-03-22T01:28:50.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:28:50.808+0000 7fee288b0700 1 --2- 172.21.15.89:0/1860477096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee241038b0 0x7fee24074a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:28:50.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:28:50.808+0000 7fee290b1700 1 --2- 172.21.15.89:0/1860477096 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fee241026b0 0x7fee24076420 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fee2000c990 tx=0x7fee2000cca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:28:50.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:28:50.809+0000 7fee1a7fc700 1 -- 172.21.15.89:0/1860477096 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee20007b00 con 0x7fee241026b0 2024-03-22T01:28:50.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:28:50.809+0000 7fee1a7fc700 1 -- 172.21.15.89:0/1860477096 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee20007ca0 con 0x7fee241026b0 2024-03-22T01:28:50.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:28:50.809+0000 7fee2b315700 1 -- 172.21.15.89:0/1860477096 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee241a2950 con 0x7fee241026b0 2024-03-22T01:28:50.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:28:50.809+0000 7fee1a7fc700 1 -- 172.21.15.89:0/1860477096 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee20018680 con 0x7fee241026b0 2024-03-22T01:28:50.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:28:50.809+0000 7fee2b315700 1 -- 172.21.15.89:0/1860477096 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee241a2e60 con 0x7fee241026b0 2024-03-22T01:28:50.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:28:50.809+0000 7fee2b315700 1 -- 172.21.15.89:0/1860477096 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee24069340 con 0x7fee241026b0 2024-03-22T01:28:50.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:28:50.811+0000 7fee1a7fc700 1 -- 172.21.15.89:0/1860477096 <== mon.2 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 15) ==== 137798+0+0 (secure 0 0 0) 0x7fee2000f450 con 0x7fee241026b0 2024-03-22T01:28:50.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:28:50.811+0000 7fee1a7fc700 1 --2- 172.21.15.89:0/1860477096 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fee100a2f20 0x7fee100a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:28:50.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:28:50.811+0000 7fee1a7fc700 1 -- 172.21.15.89:0/1860477096 <== mon.2 v2:172.21.15.89:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7fee200cd910 con 0x7fee241026b0 2024-03-22T01:28:50.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:28:50.811+0000 7fee288b0700 1 --2- 172.21.15.89:0/1860477096 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fee100a2f20 0x7fee100a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:28:50.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:28:50.812+0000 7fee288b0700 1 --2- 172.21.15.89:0/1860477096 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fee100a2f20 0x7fee100a53e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fee1400bcf0 tx=0x7fee1400b2c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:28:50.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:28:50.816+0000 7fee1a7fc700 1 -- 172.21.15.89:0/1860477096 <== mon.2 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fee20014030 con 0x7fee241026b0 2024-03-22T01:28:50.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:50 smithi089 ceph-mon[35770]: Detected new or changed devices on smithi072 2024-03-22T01:28:50.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:50 smithi089 ceph-mon[35770]: Adjusting osd_memory_target on smithi072 to 17088M 2024-03-22T01:28:50.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:50 smithi089 ceph-mon[35770]: osdmap e9: 1 total, 1 up, 1 in 2024-03-22T01:28:50.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:50 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:28:50.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:50 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:28:50.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:50 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:28:50.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:50 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:51.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:28:51.017+0000 7fee2b315700 1 -- 172.21.15.89:0/1860477096 --> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fee24062ac0 con 0x7fee100a2f20 2024-03-22T01:28:51.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:50 smithi072 ceph-mon[32414]: Detected new or changed devices on smithi072 2024-03-22T01:28:51.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:50 smithi072 ceph-mon[32414]: Adjusting osd_memory_target on smithi072 to 17088M 2024-03-22T01:28:51.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:50 smithi072 ceph-mon[32414]: osdmap e9: 1 total, 1 up, 1 in 2024-03-22T01:28:51.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:50 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:28:51.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:50 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:28:51.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:50 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:28:51.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:50 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:51.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:50 smithi098 ceph-mon[36468]: Detected new or changed devices on smithi072 2024-03-22T01:28:51.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:50 smithi098 ceph-mon[36468]: Adjusting osd_memory_target on smithi072 to 17088M 2024-03-22T01:28:51.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:50 smithi098 ceph-mon[36468]: osdmap e9: 1 total, 1 up, 1 in 2024-03-22T01:28:51.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:50 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:28:51.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:50 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:28:51.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:50 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:28:51.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:50 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:28:51.899 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:51 smithi089 ceph-mon[35770]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:28:51.899 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:51 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T01:28:51.899 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:51 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T01:28:51.899 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:51 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:28:51.899 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:51 smithi089 ceph-mon[35770]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-22T01:28:52.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:51 smithi072 ceph-mon[32414]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:28:52.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:51 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T01:28:52.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:51 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T01:28:52.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:51 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:28:52.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:51 smithi072 ceph-mon[32414]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-22T01:28:52.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:51 smithi098 ceph-mon[36468]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:28:52.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:51 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T01:28:52.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:51 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T01:28:52.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:51 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:28:52.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:51 smithi098 ceph-mon[36468]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-22T01:28:54.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:54 smithi098 ceph-mon[36468]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-22T01:28:54.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:54 smithi098 ceph-mon[36468]: from='client.? 172.21.15.89:0/237148552' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "981290ba-97dc-496b-aa22-d51704184b10"}]: dispatch 2024-03-22T01:28:54.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:54 smithi098 ceph-mon[36468]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "981290ba-97dc-496b-aa22-d51704184b10"}]: dispatch 2024-03-22T01:28:54.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:54 smithi098 ceph-mon[36468]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "981290ba-97dc-496b-aa22-d51704184b10"}]': finished 2024-03-22T01:28:54.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:54 smithi098 ceph-mon[36468]: osdmap e10: 2 total, 1 up, 2 in 2024-03-22T01:28:54.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:54 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T01:28:54.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:54 smithi098 ceph-mon[36468]: from='client.? 172.21.15.89:0/3917850241' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T01:28:54.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:54 smithi089 ceph-mon[35770]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-22T01:28:54.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:54 smithi089 ceph-mon[35770]: from='client.? 172.21.15.89:0/237148552' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "981290ba-97dc-496b-aa22-d51704184b10"}]: dispatch 2024-03-22T01:28:54.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:54 smithi089 ceph-mon[35770]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "981290ba-97dc-496b-aa22-d51704184b10"}]: dispatch 2024-03-22T01:28:54.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:54 smithi089 ceph-mon[35770]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "981290ba-97dc-496b-aa22-d51704184b10"}]': finished 2024-03-22T01:28:54.859 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:54 smithi089 ceph-mon[35770]: osdmap e10: 2 total, 1 up, 2 in 2024-03-22T01:28:54.859 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:54 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T01:28:54.859 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:54 smithi089 ceph-mon[35770]: from='client.? 172.21.15.89:0/3917850241' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T01:28:54.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:54 smithi072 ceph-mon[32414]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-22T01:28:54.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:54 smithi072 ceph-mon[32414]: from='client.? 172.21.15.89:0/237148552' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "981290ba-97dc-496b-aa22-d51704184b10"}]: dispatch 2024-03-22T01:28:54.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:54 smithi072 ceph-mon[32414]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "981290ba-97dc-496b-aa22-d51704184b10"}]: dispatch 2024-03-22T01:28:54.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:54 smithi072 ceph-mon[32414]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "981290ba-97dc-496b-aa22-d51704184b10"}]': finished 2024-03-22T01:28:54.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:54 smithi072 ceph-mon[32414]: osdmap e10: 2 total, 1 up, 2 in 2024-03-22T01:28:54.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:54 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T01:28:54.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:54 smithi072 ceph-mon[32414]: from='client.? 172.21.15.89:0/3917850241' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T01:28:56.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:56 smithi098 ceph-mon[36468]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-22T01:28:56.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:56 smithi089 ceph-mon[35770]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-22T01:28:56.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:56 smithi072 ceph-mon[32414]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-22T01:28:58.670 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:28:58 smithi089 ceph-mon[35770]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-22T01:28:58.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:28:58 smithi098 ceph-mon[36468]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-22T01:28:58.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:28:58 smithi072 ceph-mon[32414]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-22T01:29:00.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:00 smithi098 ceph-mon[36468]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-22T01:29:00.672 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:00 smithi089 ceph-mon[35770]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-22T01:29:00.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:00 smithi072 ceph-mon[32414]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-22T01:29:01.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:01 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-22T01:29:01.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:01 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:29:01.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:01 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-22T01:29:01.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:01 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:29:01.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:01 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-22T01:29:01.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:01 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:29:02.665 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:02 smithi089 ceph-mon[35770]: Deploying daemon osd.1 on smithi089 2024-03-22T01:29:02.665 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:02 smithi089 ceph-mon[35770]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-22T01:29:02.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:02 smithi072 ceph-mon[32414]: Deploying daemon osd.1 on smithi089 2024-03-22T01:29:02.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:02 smithi072 ceph-mon[32414]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-22T01:29:02.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:02 smithi098 ceph-mon[36468]: Deploying daemon osd.1 on smithi089 2024-03-22T01:29:02.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:02 smithi098 ceph-mon[36468]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-22T01:29:04.498 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:04 smithi089 ceph-mon[35770]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-22T01:29:04.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:04 smithi072 ceph-mon[32414]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-22T01:29:04.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:04 smithi098 ceph-mon[36468]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-22T01:29:06.787 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:06 smithi089 ceph-mon[35770]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-22T01:29:06.788 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:06 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:06.788 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:06 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:06.788 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:06 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:29:06.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:06 smithi072 ceph-mon[32414]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-22T01:29:06.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:06 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:06.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:06 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:06.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:06 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:29:06.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:06 smithi098 ceph-mon[36468]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-22T01:29:06.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:06 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:06.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:06 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:06.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:06 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:29:08.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:08 smithi089 ceph-mon[35770]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-22T01:29:08.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:08 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:08.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:08 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:08.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:08 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:29:08.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:08 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:29:08.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:08 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:08.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:08 smithi072 ceph-mon[32414]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-22T01:29:08.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:08 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:08.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:08 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:08.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:08 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:29:08.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:08 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:29:08.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:08 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:08.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:08 smithi098 ceph-mon[36468]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-22T01:29:08.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:08 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:08.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:08 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:08.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:08 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:29:08.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:08 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:29:08.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:08 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:09.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:29:09.781+0000 7fee1a7fc700 1 -- 172.21.15.89:0/1860477096 <== mgr.14152 v2:172.21.15.72:6800/401403200 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fee24062ac0 con 0x7fee100a2f20 2024-03-22T01:29:09.781 INFO:teuthology.orchestra.run.smithi089.stdout:Created osd(s) 1 on host 'smithi089' 2024-03-22T01:29:09.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:29:09.784+0000 7fee2b315700 1 -- 172.21.15.89:0/1860477096 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fee100a2f20 msgr2=0x7fee100a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:29:09.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:29:09.784+0000 7fee2b315700 1 --2- 172.21.15.89:0/1860477096 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fee100a2f20 0x7fee100a53e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fee1400bcf0 tx=0x7fee1400b2c0 comp rx=0 tx=0).stop 2024-03-22T01:29:09.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:29:09.784+0000 7fee2b315700 1 -- 172.21.15.89:0/1860477096 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fee241026b0 msgr2=0x7fee24076420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:29:09.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:29:09.784+0000 7fee2b315700 1 --2- 172.21.15.89:0/1860477096 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fee241026b0 0x7fee24076420 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fee2000c990 tx=0x7fee2000cca0 comp rx=0 tx=0).stop 2024-03-22T01:29:09.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:29:09.784+0000 7fee2b315700 1 -- 172.21.15.89:0/1860477096 shutdown_connections 2024-03-22T01:29:09.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:29:09.784+0000 7fee2b315700 1 --2- 172.21.15.89:0/1860477096 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fee100a2f20 0x7fee100a53e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:29:09.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:29:09.784+0000 7fee2b315700 1 --2- 172.21.15.89:0/1860477096 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fee24104250 0x7fee24074fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:29:09.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:29:09.784+0000 7fee2b315700 1 --2- 172.21.15.89:0/1860477096 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fee241026b0 0x7fee24076420 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:29:09.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:29:09.784+0000 7fee2b315700 1 --2- 172.21.15.89:0/1860477096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee241038b0 0x7fee24074a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:29:09.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:29:09.784+0000 7fee2b315700 1 -- 172.21.15.89:0/1860477096 >> 172.21.15.89:0/1860477096 conn(0x7fee240fdc90 msgr2=0x7fee24106b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:29:09.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:29:09.784+0000 7fee2b315700 1 -- 172.21.15.89:0/1860477096 shutdown_connections 2024-03-22T01:29:09.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:29:09.784+0000 7fee2b315700 1 -- 172.21.15.89:0/1860477096 wait complete. 2024-03-22T01:29:10.365 DEBUG:teuthology.orchestra.run.smithi089:osd.1> sudo journalctl -f -n 0 -u ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@osd.1.service 2024-03-22T01:29:10.368 INFO:tasks.cephadm:Deploying osd.2 on smithi098 with /dev/vg_nvme/lv_4... 2024-03-22T01:29:10.368 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-22T01:29:10.607 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:10 smithi089 ceph-mon[35770]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-22T01:29:10.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:10 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:10.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:10 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:10.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:10 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:29:10.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:10 smithi089 ceph-mon[35770]: from='osd.1 [v2:172.21.15.89:6800/2870590550,v1:172.21.15.89:6801/2870590550]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-22T01:29:10.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:10 smithi089 ceph-mon[35770]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-22T01:29:10.615 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.c/config 2024-03-22T01:29:10.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:10 smithi098 ceph-mon[36468]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-22T01:29:10.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:10 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:10.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:10 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:10.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:10 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:29:10.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:10 smithi098 ceph-mon[36468]: from='osd.1 [v2:172.21.15.89:6800/2870590550,v1:172.21.15.89:6801/2870590550]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-22T01:29:10.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:10 smithi098 ceph-mon[36468]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-22T01:29:10.874 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:10 smithi072 ceph-mon[32414]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-22T01:29:10.874 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:10 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:10.874 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:10 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:10.874 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:10 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:29:10.874 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:10 smithi072 ceph-mon[32414]: from='osd.1 [v2:172.21.15.89:6800/2870590550,v1:172.21.15.89:6801/2870590550]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-22T01:29:10.875 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:10 smithi072 ceph-mon[32414]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-22T01:29:11.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:11 smithi098 ceph-mon[36468]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-22T01:29:11.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:11 smithi098 ceph-mon[36468]: from='osd.1 [v2:172.21.15.89:6800/2870590550,v1:172.21.15.89:6801/2870590550]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-03-22T01:29:11.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:11 smithi098 ceph-mon[36468]: osdmap e11: 2 total, 1 up, 2 in 2024-03-22T01:29:11.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:11 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T01:29:11.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:11 smithi098 ceph-mon[36468]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-03-22T01:29:11.927 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:11 smithi098 ceph-mon[36468]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-22T01:29:11.965 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:11 smithi089 ceph-mon[35770]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-22T01:29:11.965 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:11 smithi089 ceph-mon[35770]: from='osd.1 [v2:172.21.15.89:6800/2870590550,v1:172.21.15.89:6801/2870590550]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-03-22T01:29:11.965 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:11 smithi089 ceph-mon[35770]: osdmap e11: 2 total, 1 up, 2 in 2024-03-22T01:29:11.966 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:11 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T01:29:11.966 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:11 smithi089 ceph-mon[35770]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-03-22T01:29:11.966 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:11 smithi089 ceph-mon[35770]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-22T01:29:11.966 INFO:journalctl@ceph.osd.1.smithi089.stdout:Mar 22 01:29:11 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-1[40372]: 2024-03-22T01:29:11.742+0000 7fc2b65eb700 -1 osd.1 0 waiting for initial osdmap 2024-03-22T01:29:11.966 INFO:journalctl@ceph.osd.1.smithi089.stdout:Mar 22 01:29:11 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-1[40372]: 2024-03-22T01:29:11.745+0000 7fc2b23de700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-22T01:29:12.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:11 smithi072 ceph-mon[32414]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-22T01:29:12.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:11 smithi072 ceph-mon[32414]: from='osd.1 [v2:172.21.15.89:6800/2870590550,v1:172.21.15.89:6801/2870590550]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-03-22T01:29:12.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:11 smithi072 ceph-mon[32414]: osdmap e11: 2 total, 1 up, 2 in 2024-03-22T01:29:12.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:11 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T01:29:12.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:11 smithi072 ceph-mon[32414]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-03-22T01:29:12.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:11 smithi072 ceph-mon[32414]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-22T01:29:12.445 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-22T01:29:12.445 INFO:teuthology.orchestra.run.smithi098.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-22T01:29:12.445 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10+0 records in 2024-03-22T01:29:12.445 INFO:teuthology.orchestra.run.smithi098.stderr:10+0 records out 2024-03-22T01:29:12.446 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114653 s, 915 MB/s 2024-03-22T01:29:12.446 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping successful for: 2024-03-22T01:29:13.023 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:12 smithi072 ceph-mon[32414]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2024-03-22T01:29:13.023 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:12 smithi072 ceph-mon[32414]: osdmap e12: 2 total, 1 up, 2 in 2024-03-22T01:29:13.023 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:12 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T01:29:13.023 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:12 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T01:29:13.057 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:12 smithi098 ceph-mon[36468]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2024-03-22T01:29:13.058 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:12 smithi098 ceph-mon[36468]: osdmap e12: 2 total, 1 up, 2 in 2024-03-22T01:29:13.058 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:12 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T01:29:13.058 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:12 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T01:29:13.084 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi098:vg_nvme/lv_4 2024-03-22T01:29:13.096 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:12 smithi089 ceph-mon[35770]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2024-03-22T01:29:13.096 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:12 smithi089 ceph-mon[35770]: osdmap e12: 2 total, 1 up, 2 in 2024-03-22T01:29:13.096 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:12 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T01:29:13.096 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:12 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T01:29:13.320 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.c/config 2024-03-22T01:29:14.055 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:13 smithi098 ceph-mon[36468]: purged_snaps scrub starts 2024-03-22T01:29:14.055 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:13 smithi098 ceph-mon[36468]: purged_snaps scrub ok 2024-03-22T01:29:14.055 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:13 smithi098 ceph-mon[36468]: osd.1 [v2:172.21.15.89:6800/2870590550,v1:172.21.15.89:6801/2870590550] boot 2024-03-22T01:29:14.055 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:13 smithi098 ceph-mon[36468]: osdmap e13: 2 total, 2 up, 2 in 2024-03-22T01:29:14.055 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:13 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T01:29:14.056 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:13 smithi098 ceph-mon[36468]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-22T01:29:14.056 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:13 smithi098 ceph-mon[36468]: Detected new or changed devices on smithi089 2024-03-22T01:29:14.056 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:13 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:14.056 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:13 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:14.056 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:13 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:29:14.056 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:13 smithi098 ceph-mon[36468]: Adjusting osd_memory_target on smithi089 to 17080M 2024-03-22T01:29:14.056 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:13 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:14.056 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:13 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:29:14.056 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:13 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:29:14.056 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:13 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:14.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:13 smithi089 ceph-mon[35770]: purged_snaps scrub starts 2024-03-22T01:29:14.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:13 smithi089 ceph-mon[35770]: purged_snaps scrub ok 2024-03-22T01:29:14.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:13 smithi089 ceph-mon[35770]: osd.1 [v2:172.21.15.89:6800/2870590550,v1:172.21.15.89:6801/2870590550] boot 2024-03-22T01:29:14.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:13 smithi089 ceph-mon[35770]: osdmap e13: 2 total, 2 up, 2 in 2024-03-22T01:29:14.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:13 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T01:29:14.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:13 smithi089 ceph-mon[35770]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-22T01:29:14.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:13 smithi089 ceph-mon[35770]: Detected new or changed devices on smithi089 2024-03-22T01:29:14.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:13 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:14.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:13 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:14.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:13 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:29:14.109 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:13 smithi089 ceph-mon[35770]: Adjusting osd_memory_target on smithi089 to 17080M 2024-03-22T01:29:14.109 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:13 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:14.109 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:13 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:29:14.109 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:13 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:29:14.109 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:13 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:14.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:13 smithi072 ceph-mon[32414]: purged_snaps scrub starts 2024-03-22T01:29:14.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:13 smithi072 ceph-mon[32414]: purged_snaps scrub ok 2024-03-22T01:29:14.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:13 smithi072 ceph-mon[32414]: osd.1 [v2:172.21.15.89:6800/2870590550,v1:172.21.15.89:6801/2870590550] boot 2024-03-22T01:29:14.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:13 smithi072 ceph-mon[32414]: osdmap e13: 2 total, 2 up, 2 in 2024-03-22T01:29:14.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:13 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T01:29:14.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:13 smithi072 ceph-mon[32414]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-22T01:29:14.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:13 smithi072 ceph-mon[32414]: Detected new or changed devices on smithi089 2024-03-22T01:29:14.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:13 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:14.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:13 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:14.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:13 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:29:14.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:13 smithi072 ceph-mon[32414]: Adjusting osd_memory_target on smithi089 to 17080M 2024-03-22T01:29:14.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:13 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:14.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:13 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:29:14.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:13 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:29:14.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:13 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:14.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:29:14.729+0000 7f27bde1c700 1 -- 172.21.15.98:0/3584059001 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27b8100750 msgr2=0x7f27b8100bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:29:14.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:29:14.729+0000 7f27bde1c700 1 --2- 172.21.15.98:0/3584059001 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27b8100750 0x7f27b8100bd0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f27a0009990 tx=0x7f27a0009ca0 comp rx=0 tx=0).stop 2024-03-22T01:29:14.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:29:14.729+0000 7f27bde1c700 1 -- 172.21.15.98:0/3584059001 shutdown_connections 2024-03-22T01:29:14.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:29:14.729+0000 7f27bde1c700 1 --2- 172.21.15.98:0/3584059001 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27b8101110 0x7f27b810acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:29:14.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:29:14.729+0000 7f27bde1c700 1 --2- 172.21.15.98:0/3584059001 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27b8103cd0 0x7f27b81040f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:29:14.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:29:14.729+0000 7f27bde1c700 1 --2- 172.21.15.98:0/3584059001 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27b8100750 0x7f27b8100bd0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:29:14.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:29:14.729+0000 7f27bde1c700 1 -- 172.21.15.98:0/3584059001 >> 172.21.15.98:0/3584059001 conn(0x7f27b80fc2d0 msgr2=0x7f27b80fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:29:14.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:29:14.729+0000 7f27bde1c700 1 -- 172.21.15.98:0/3584059001 shutdown_connections 2024-03-22T01:29:14.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:29:14.729+0000 7f27bde1c700 1 -- 172.21.15.98:0/3584059001 wait complete. 2024-03-22T01:29:14.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:29:14.730+0000 7f27bde1c700 1 Processor -- start 2024-03-22T01:29:14.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:29:14.730+0000 7f27bde1c700 1 -- start start 2024-03-22T01:29:14.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:29:14.730+0000 7f27bde1c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27b8100750 0x7f27b81033d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:29:14.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:29:14.730+0000 7f27bde1c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27b8101110 0x7f27b8101a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:29:14.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:29:14.730+0000 7f27bde1c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27b8103cd0 0x7f27b8101f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:29:14.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:29:14.730+0000 7f27bde1c700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f27b810e330 con 0x7f27b8101110 2024-03-22T01:29:14.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:29:14.730+0000 7f27bde1c700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f27b810e1b0 con 0x7f27b8103cd0 2024-03-22T01:29:14.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:29:14.730+0000 7f27bde1c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f27b810e4b0 con 0x7f27b8100750 2024-03-22T01:29:14.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:29:14.730+0000 7f27b77fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27b8100750 0x7f27b81033d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:29:14.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:29:14.730+0000 7f27b77fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27b8100750 0x7f27b81033d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43876/0 (socket says 172.21.15.98:43876) 2024-03-22T01:29:14.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:29:14.730+0000 7f27b77fe700 1 -- 172.21.15.98:0/417025496 learned_addr learned my addr 172.21.15.98:0/417025496 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-22T01:29:14.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:29:14.730+0000 7f27aeffd700 1 --2- 172.21.15.98:0/417025496 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27b8101110 0x7f27b8101a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:29:14.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:29:14.731+0000 7f27b7fff700 1 --2- 172.21.15.98:0/417025496 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27b8103cd0 0x7f27b8101f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:29:14.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:29:14.731+0000 7f27b77fe700 1 -- 172.21.15.98:0/417025496 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27b8103cd0 msgr2=0x7f27b8101f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:29:14.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:29:14.731+0000 7f27b77fe700 1 --2- 172.21.15.98:0/417025496 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27b8103cd0 0x7f27b8101f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:29:14.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:29:14.731+0000 7f27b77fe700 1 -- 172.21.15.98:0/417025496 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27b8101110 msgr2=0x7f27b8101a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:29:14.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:29:14.731+0000 7f27b77fe700 1 --2- 172.21.15.98:0/417025496 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27b8101110 0x7f27b8101a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:29:14.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:29:14.731+0000 7f27b77fe700 1 -- 172.21.15.98:0/417025496 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27b8102810 con 0x7f27b8100750 2024-03-22T01:29:14.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:29:14.731+0000 7f27b77fe700 1 --2- 172.21.15.98:0/417025496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27b8100750 0x7f27b81033d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f27a800d980 tx=0x7f27a800dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:29:14.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:29:14.731+0000 7f27b57fa700 1 -- 172.21.15.98:0/417025496 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27a8017030 con 0x7f27b8100750 2024-03-22T01:29:14.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:29:14.731+0000 7f27b57fa700 1 -- 172.21.15.98:0/417025496 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27a8004500 con 0x7f27b8100750 2024-03-22T01:29:14.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:29:14.731+0000 7f27b57fa700 1 -- 172.21.15.98:0/417025496 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27a8003aa0 con 0x7f27b8100750 2024-03-22T01:29:14.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:29:14.731+0000 7f27bde1c700 1 -- 172.21.15.98:0/417025496 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27b81a2ad0 con 0x7f27b8100750 2024-03-22T01:29:14.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:29:14.731+0000 7f27bde1c700 1 -- 172.21.15.98:0/417025496 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27b81a3010 con 0x7f27b8100750 2024-03-22T01:29:14.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:29:14.732+0000 7f27bde1c700 1 -- 172.21.15.98:0/417025496 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27b8069340 con 0x7f27b8100750 2024-03-22T01:29:14.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:29:14.733+0000 7f27b57fa700 1 -- 172.21.15.98:0/417025496 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) ==== 137798+0+0 (secure 0 0 0) 0x7f27a801a070 con 0x7f27b8100750 2024-03-22T01:29:14.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:29:14.734+0000 7f27b57fa700 1 --2- 172.21.15.98:0/417025496 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f27a40a2fa0 0x7f27a40a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:29:14.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:29:14.734+0000 7f27b57fa700 1 -- 172.21.15.98:0/417025496 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7f27a80cd420 con 0x7f27b8100750 2024-03-22T01:29:14.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:29:14.734+0000 7f27aeffd700 1 --2- 172.21.15.98:0/417025496 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f27a40a2fa0 0x7f27a40a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:29:14.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:29:14.734+0000 7f27aeffd700 1 --2- 172.21.15.98:0/417025496 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f27a40a2fa0 0x7f27a40a5460 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f27a0003890 tx=0x7f27a0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:29:14.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:29:14.739+0000 7f27b57fa700 1 -- 172.21.15.98:0/417025496 <== mon.1 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f27a80232f0 con 0x7f27b8100750 2024-03-22T01:29:14.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:29:14.943+0000 7f27bde1c700 1 -- 172.21.15.98:0/417025496 --> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f27b8062ac0 con 0x7f27a40a2fa0 2024-03-22T01:29:15.107 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:14 smithi089 ceph-mon[35770]: osdmap e14: 2 total, 2 up, 2 in 2024-03-22T01:29:15.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:14 smithi072 ceph-mon[32414]: osdmap e14: 2 total, 2 up, 2 in 2024-03-22T01:29:15.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:14 smithi098 ceph-mon[36468]: osdmap e14: 2 total, 2 up, 2 in 2024-03-22T01:29:15.796 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:15 smithi098 ceph-mon[36468]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:29:15.796 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:15 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T01:29:15.796 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:15 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T01:29:15.796 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:15 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:29:15.796 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:15 smithi098 ceph-mon[36468]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-22T01:29:16.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:15 smithi089 ceph-mon[35770]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:29:16.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:15 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T01:29:16.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:15 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T01:29:16.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:15 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:29:16.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:15 smithi089 ceph-mon[35770]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-22T01:29:16.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:15 smithi072 ceph-mon[32414]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:29:16.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:15 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T01:29:16.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:15 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T01:29:16.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:15 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:29:16.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:15 smithi072 ceph-mon[32414]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-22T01:29:18.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:18 smithi098 ceph-mon[36468]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-22T01:29:18.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:18 smithi098 ceph-mon[36468]: from='client.? 172.21.15.98:0/2009213917' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f07ed658-f545-46a1-8edd-8b1d5d49885c"}]: dispatch 2024-03-22T01:29:18.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:18 smithi098 ceph-mon[36468]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f07ed658-f545-46a1-8edd-8b1d5d49885c"}]: dispatch 2024-03-22T01:29:18.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:18 smithi098 ceph-mon[36468]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f07ed658-f545-46a1-8edd-8b1d5d49885c"}]': finished 2024-03-22T01:29:18.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:18 smithi098 ceph-mon[36468]: osdmap e15: 3 total, 2 up, 3 in 2024-03-22T01:29:18.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:18 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T01:29:18.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:18 smithi098 ceph-mon[36468]: from='client.? 172.21.15.98:0/190702328' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T01:29:18.862 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:18 smithi089 ceph-mon[35770]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-22T01:29:18.863 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:18 smithi089 ceph-mon[35770]: from='client.? 172.21.15.98:0/2009213917' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f07ed658-f545-46a1-8edd-8b1d5d49885c"}]: dispatch 2024-03-22T01:29:18.863 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:18 smithi089 ceph-mon[35770]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f07ed658-f545-46a1-8edd-8b1d5d49885c"}]: dispatch 2024-03-22T01:29:18.863 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:18 smithi089 ceph-mon[35770]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f07ed658-f545-46a1-8edd-8b1d5d49885c"}]': finished 2024-03-22T01:29:18.863 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:18 smithi089 ceph-mon[35770]: osdmap e15: 3 total, 2 up, 3 in 2024-03-22T01:29:18.863 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:18 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T01:29:18.863 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:18 smithi089 ceph-mon[35770]: from='client.? 172.21.15.98:0/190702328' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T01:29:18.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:18 smithi072 ceph-mon[32414]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-22T01:29:18.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:18 smithi072 ceph-mon[32414]: from='client.? 172.21.15.98:0/2009213917' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f07ed658-f545-46a1-8edd-8b1d5d49885c"}]: dispatch 2024-03-22T01:29:18.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:18 smithi072 ceph-mon[32414]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f07ed658-f545-46a1-8edd-8b1d5d49885c"}]: dispatch 2024-03-22T01:29:18.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:18 smithi072 ceph-mon[32414]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f07ed658-f545-46a1-8edd-8b1d5d49885c"}]': finished 2024-03-22T01:29:18.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:18 smithi072 ceph-mon[32414]: osdmap e15: 3 total, 2 up, 3 in 2024-03-22T01:29:18.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:18 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T01:29:18.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:18 smithi072 ceph-mon[32414]: from='client.? 172.21.15.98:0/190702328' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T01:29:20.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:20 smithi098 ceph-mon[36468]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-22T01:29:20.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:20 smithi089 ceph-mon[35770]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-22T01:29:20.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:20 smithi072 ceph-mon[32414]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-22T01:29:22.660 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:22 smithi098 ceph-mon[36468]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-22T01:29:22.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:22 smithi089 ceph-mon[35770]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-22T01:29:22.871 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:22 smithi072 ceph-mon[32414]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-22T01:29:24.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:24 smithi072 ceph-mon[32414]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-22T01:29:24.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:24 smithi098 ceph-mon[36468]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-22T01:29:24.677 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:24 smithi089 ceph-mon[35770]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-22T01:29:25.565 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:25 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-22T01:29:25.565 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:25 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:29:25.565 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:25 smithi098 ceph-mon[36468]: Deploying daemon osd.2 on smithi098 2024-03-22T01:29:25.855 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:25 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-22T01:29:25.855 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:25 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:29:25.856 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:25 smithi089 ceph-mon[35770]: Deploying daemon osd.2 on smithi098 2024-03-22T01:29:25.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:25 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-22T01:29:25.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:25 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:29:25.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:25 smithi072 ceph-mon[32414]: Deploying daemon osd.2 on smithi098 2024-03-22T01:29:26.527 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:26 smithi098 ceph-mon[36468]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-22T01:29:26.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:26 smithi089 ceph-mon[35770]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-22T01:29:26.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:26 smithi072 ceph-mon[32414]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-22T01:29:28.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:28 smithi089 ceph-mon[35770]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-22T01:29:28.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:28 smithi072 ceph-mon[32414]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-22T01:29:28.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:28 smithi098 ceph-mon[36468]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-22T01:29:30.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:30 smithi098 ceph-mon[36468]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-22T01:29:30.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:30 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:30.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:30 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:30.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:30 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:29:30.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:30 smithi089 ceph-mon[35770]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-22T01:29:30.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:30 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:30.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:30 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:30.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:30 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:29:30.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:30 smithi072 ceph-mon[32414]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-22T01:29:30.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:30 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:30.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:30 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:30.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:30 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:29:32.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:32 smithi098 ceph-mon[36468]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-22T01:29:32.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:32 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:32.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:32 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:32.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:32 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:29:32.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:32 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:29:32.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:32 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:32.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:32 smithi089 ceph-mon[35770]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-22T01:29:32.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:32 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:32.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:32 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:32.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:32 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:29:32.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:32 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:29:32.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:32 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:32.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:32 smithi072 ceph-mon[32414]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-22T01:29:32.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:32 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:32.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:32 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:32.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:32 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:29:32.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:32 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:29:32.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:32 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:33.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:29:33.312+0000 7f27b57fa700 1 -- 172.21.15.98:0/417025496 <== mgr.14152 v2:172.21.15.72:6800/401403200 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f27b8062ac0 con 0x7f27a40a2fa0 2024-03-22T01:29:33.312 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 2 on host 'smithi098' 2024-03-22T01:29:33.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:29:33.315+0000 7f27bde1c700 1 -- 172.21.15.98:0/417025496 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f27a40a2fa0 msgr2=0x7f27a40a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:29:33.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:29:33.315+0000 7f27bde1c700 1 --2- 172.21.15.98:0/417025496 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f27a40a2fa0 0x7f27a40a5460 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f27a0003890 tx=0x7f27a0005fd0 comp rx=0 tx=0).stop 2024-03-22T01:29:33.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:29:33.315+0000 7f27bde1c700 1 -- 172.21.15.98:0/417025496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27b8100750 msgr2=0x7f27b81033d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:29:33.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:29:33.315+0000 7f27bde1c700 1 --2- 172.21.15.98:0/417025496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27b8100750 0x7f27b81033d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f27a800d980 tx=0x7f27a800dc90 comp rx=0 tx=0).stop 2024-03-22T01:29:33.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:29:33.315+0000 7f27bde1c700 1 -- 172.21.15.98:0/417025496 shutdown_connections 2024-03-22T01:29:33.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:29:33.315+0000 7f27bde1c700 1 --2- 172.21.15.98:0/417025496 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f27a40a2fa0 0x7f27a40a5460 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:29:33.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:29:33.315+0000 7f27bde1c700 1 --2- 172.21.15.98:0/417025496 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27b8103cd0 0x7f27b8101f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:29:33.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:29:33.315+0000 7f27bde1c700 1 --2- 172.21.15.98:0/417025496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27b8100750 0x7f27b81033d0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:29:33.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:29:33.315+0000 7f27bde1c700 1 --2- 172.21.15.98:0/417025496 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27b8101110 0x7f27b8101a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:29:33.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:29:33.315+0000 7f27bde1c700 1 -- 172.21.15.98:0/417025496 >> 172.21.15.98:0/417025496 conn(0x7f27b80fc2d0 msgr2=0x7f27b8106d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:29:33.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:29:33.315+0000 7f27bde1c700 1 -- 172.21.15.98:0/417025496 shutdown_connections 2024-03-22T01:29:33.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:29:33.315+0000 7f27bde1c700 1 -- 172.21.15.98:0/417025496 wait complete. 2024-03-22T01:29:33.951 DEBUG:teuthology.orchestra.run.smithi098:osd.2> sudo journalctl -f -n 0 -u ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@osd.2.service 2024-03-22T01:29:33.953 INFO:tasks.cephadm:Deploying osd.3 on smithi072 with /dev/vg_nvme/lv_3... 2024-03-22T01:29:33.954 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-22T01:29:34.201 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:29:34.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:34 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:34.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:34 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:34.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:34 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:29:34.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:34 smithi089 ceph-mon[35770]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-22T01:29:34.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:34 smithi089 ceph-mon[35770]: from='osd.2 [v2:172.21.15.98:6800/3865778118,v1:172.21.15.98:6801/3865778118]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-22T01:29:34.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:34 smithi089 ceph-mon[35770]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-22T01:29:34.613 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:34 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:34.613 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:34 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:34.613 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:34 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:29:34.613 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:34 smithi072 ceph-mon[32414]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-22T01:29:34.613 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:34 smithi072 ceph-mon[32414]: from='osd.2 [v2:172.21.15.98:6800/3865778118,v1:172.21.15.98:6801/3865778118]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-22T01:29:34.613 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:34 smithi072 ceph-mon[32414]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-22T01:29:34.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:34 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:34.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:34 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:34.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:34 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:29:34.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:34 smithi098 ceph-mon[36468]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-22T01:29:34.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:34 smithi098 ceph-mon[36468]: from='osd.2 [v2:172.21.15.98:6800/3865778118,v1:172.21.15.98:6801/3865778118]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-22T01:29:34.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:34 smithi098 ceph-mon[36468]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-22T01:29:35.413 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:35 smithi098 ceph-mon[36468]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-22T01:29:35.413 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:35 smithi098 ceph-mon[36468]: osdmap e16: 3 total, 2 up, 3 in 2024-03-22T01:29:35.413 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:35 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T01:29:35.413 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:35 smithi098 ceph-mon[36468]: from='osd.2 [v2:172.21.15.98:6800/3865778118,v1:172.21.15.98:6801/3865778118]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-22T01:29:35.413 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:35 smithi098 ceph-mon[36468]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-22T01:29:35.413 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 22 01:29:35 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-2[41603]: 2024-03-22T01:29:35.305+0000 7f7a5fc06700 -1 osd.2 0 waiting for initial osdmap 2024-03-22T01:29:35.413 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 22 01:29:35 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-2[41603]: 2024-03-22T01:29:35.309+0000 7f7a5b9f9700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-22T01:29:35.501 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:35 smithi072 ceph-mon[32414]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-22T01:29:35.501 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:35 smithi072 ceph-mon[32414]: osdmap e16: 3 total, 2 up, 3 in 2024-03-22T01:29:35.501 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:35 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T01:29:35.501 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:35 smithi072 ceph-mon[32414]: from='osd.2 [v2:172.21.15.98:6800/3865778118,v1:172.21.15.98:6801/3865778118]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-22T01:29:35.501 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:35 smithi072 ceph-mon[32414]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-22T01:29:35.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:35 smithi089 ceph-mon[35770]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-22T01:29:35.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:35 smithi089 ceph-mon[35770]: osdmap e16: 3 total, 2 up, 3 in 2024-03-22T01:29:35.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:35 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T01:29:35.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:35 smithi089 ceph-mon[35770]: from='osd.2 [v2:172.21.15.98:6800/3865778118,v1:172.21.15.98:6801/3865778118]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-22T01:29:35.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:35 smithi089 ceph-mon[35770]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-22T01:29:36.081 INFO:teuthology.orchestra.run.smithi072.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-22T01:29:36.081 INFO:teuthology.orchestra.run.smithi072.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-22T01:29:36.081 INFO:teuthology.orchestra.run.smithi072.stderr: stderr: 10+0 records in 2024-03-22T01:29:36.081 INFO:teuthology.orchestra.run.smithi072.stderr:10+0 records out 2024-03-22T01:29:36.081 INFO:teuthology.orchestra.run.smithi072.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011309 s, 927 MB/s 2024-03-22T01:29:36.081 INFO:teuthology.orchestra.run.smithi072.stderr:--> Zapping successful for: 2024-03-22T01:29:36.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:36 smithi098 ceph-mon[36468]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-22T01:29:36.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:36 smithi098 ceph-mon[36468]: osdmap e17: 3 total, 2 up, 3 in 2024-03-22T01:29:36.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:36 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T01:29:36.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:36 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T01:29:36.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:36 smithi098 ceph-mon[36468]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-22T01:29:36.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:36 smithi089 ceph-mon[35770]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-22T01:29:36.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:36 smithi089 ceph-mon[35770]: osdmap e17: 3 total, 2 up, 3 in 2024-03-22T01:29:36.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:36 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T01:29:36.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:36 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T01:29:36.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:36 smithi089 ceph-mon[35770]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-22T01:29:36.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:36 smithi072 ceph-mon[32414]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-22T01:29:36.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:36 smithi072 ceph-mon[32414]: osdmap e17: 3 total, 2 up, 3 in 2024-03-22T01:29:36.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:36 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T01:29:36.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:36 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T01:29:36.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:36 smithi072 ceph-mon[32414]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-22T01:29:36.689 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi072:vg_nvme/lv_3 2024-03-22T01:29:36.924 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:29:37.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:37 smithi089 ceph-mon[35770]: purged_snaps scrub starts 2024-03-22T01:29:37.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:37 smithi089 ceph-mon[35770]: purged_snaps scrub ok 2024-03-22T01:29:37.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:37 smithi089 ceph-mon[35770]: osd.2 [v2:172.21.15.98:6800/3865778118,v1:172.21.15.98:6801/3865778118] boot 2024-03-22T01:29:37.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:37 smithi089 ceph-mon[35770]: osdmap e18: 3 total, 3 up, 3 in 2024-03-22T01:29:37.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:37 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T01:29:37.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:37 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:37.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:37 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:37.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:37 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:29:37.609 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:37 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:37.609 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:37 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:29:37.609 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:37 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:29:37.609 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:37 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:37.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:37 smithi072 ceph-mon[32414]: purged_snaps scrub starts 2024-03-22T01:29:37.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:37 smithi072 ceph-mon[32414]: purged_snaps scrub ok 2024-03-22T01:29:37.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:37 smithi072 ceph-mon[32414]: osd.2 [v2:172.21.15.98:6800/3865778118,v1:172.21.15.98:6801/3865778118] boot 2024-03-22T01:29:37.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:37 smithi072 ceph-mon[32414]: osdmap e18: 3 total, 3 up, 3 in 2024-03-22T01:29:37.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:37 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T01:29:37.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:37 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:37.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:37 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:37.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:37 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:29:37.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:37 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:37.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:37 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:29:37.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:37 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:29:37.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:37 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:37.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:37 smithi098 ceph-mon[36468]: purged_snaps scrub starts 2024-03-22T01:29:37.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:37 smithi098 ceph-mon[36468]: purged_snaps scrub ok 2024-03-22T01:29:37.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:37 smithi098 ceph-mon[36468]: osd.2 [v2:172.21.15.98:6800/3865778118,v1:172.21.15.98:6801/3865778118] boot 2024-03-22T01:29:37.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:37 smithi098 ceph-mon[36468]: osdmap e18: 3 total, 3 up, 3 in 2024-03-22T01:29:37.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:37 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T01:29:37.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:37 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:37.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:37 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:37.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:37 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:29:37.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:37 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:37.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:37 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:29:37.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:37 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:29:37.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:37 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:38.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:38 smithi072 ceph-mon[32414]: Detected new or changed devices on smithi098 2024-03-22T01:29:38.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:38 smithi072 ceph-mon[32414]: Adjusting osd_memory_target on smithi098 to 17080M 2024-03-22T01:29:38.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:38 smithi072 ceph-mon[32414]: osdmap e19: 3 total, 3 up, 3 in 2024-03-22T01:29:38.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:38 smithi072 ceph-mon[32414]: pgmap v80: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-22T01:29:38.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:38 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-22T01:29:38.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:29:38.401+0000 7fb1ab8dc700 1 -- 172.21.15.72:0/919719425 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1a41026b0 msgr2=0x7fb1a4102ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:29:38.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:29:38.401+0000 7fb1ab8dc700 1 --2- 172.21.15.72:0/919719425 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1a41026b0 0x7fb1a4102ad0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb1a00099c0 tx=0x7fb1a0009cd0 comp rx=0 tx=0).stop 2024-03-22T01:29:38.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:29:38.401+0000 7fb1ab8dc700 1 -- 172.21.15.72:0/919719425 shutdown_connections 2024-03-22T01:29:38.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:29:38.401+0000 7fb1ab8dc700 1 --2- 172.21.15.72:0/919719425 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1a4104250 0x7fb1a410ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:29:38.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:29:38.401+0000 7fb1ab8dc700 1 --2- 172.21.15.72:0/919719425 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1a41026b0 0x7fb1a4102ad0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:29:38.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:29:38.401+0000 7fb1ab8dc700 1 --2- 172.21.15.72:0/919719425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1a41038b0 0x7fb1a4103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:29:38.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:29:38.401+0000 7fb1ab8dc700 1 -- 172.21.15.72:0/919719425 >> 172.21.15.72:0/919719425 conn(0x7fb1a40fdc90 msgr2=0x7fb1a41000b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:29:38.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:29:38.401+0000 7fb1ab8dc700 1 -- 172.21.15.72:0/919719425 shutdown_connections 2024-03-22T01:29:38.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:29:38.401+0000 7fb1ab8dc700 1 -- 172.21.15.72:0/919719425 wait complete. 2024-03-22T01:29:38.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:29:38.402+0000 7fb1ab8dc700 1 Processor -- start 2024-03-22T01:29:38.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:29:38.402+0000 7fb1ab8dc700 1 -- start start 2024-03-22T01:29:38.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:29:38.402+0000 7fb1ab8dc700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1a41026b0 0x7fb1a41988e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:29:38.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:29:38.402+0000 7fb1ab8dc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1a41038b0 0x7fb1a4198e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:29:38.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:29:38.402+0000 7fb1ab8dc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1a4104250 0x7fb1a419fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:29:38.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:29:38.402+0000 7fb1ab8dc700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a410e2a0 con 0x7fb1a41026b0 2024-03-22T01:29:38.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:29:38.402+0000 7fb1ab8dc700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a410e120 con 0x7fb1a4104250 2024-03-22T01:29:38.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:29:38.402+0000 7fb1ab8dc700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a410e420 con 0x7fb1a41038b0 2024-03-22T01:29:38.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:29:38.402+0000 7fb1a9678700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1a41026b0 0x7fb1a41988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:29:38.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:29:38.402+0000 7fb1a9678700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1a41026b0 0x7fb1a41988e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:35338/0 (socket says 172.21.15.72:35338) 2024-03-22T01:29:38.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:29:38.402+0000 7fb1a9678700 1 -- 172.21.15.72:0/3507456079 learned_addr learned my addr 172.21.15.72:0/3507456079 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:29:38.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:29:38.402+0000 7fb1a8e77700 1 --2- 172.21.15.72:0/3507456079 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1a41038b0 0x7fb1a4198e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:29:38.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:29:38.402+0000 7fb1a9e79700 1 --2- 172.21.15.72:0/3507456079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1a4104250 0x7fb1a419fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:29:38.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:29:38.402+0000 7fb1a9678700 1 -- 172.21.15.72:0/3507456079 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1a41038b0 msgr2=0x7fb1a4198e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:29:38.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:29:38.402+0000 7fb1a9678700 1 --2- 172.21.15.72:0/3507456079 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1a41038b0 0x7fb1a4198e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:29:38.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:29:38.402+0000 7fb1a9678700 1 -- 172.21.15.72:0/3507456079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1a4104250 msgr2=0x7fb1a419fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:29:38.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:29:38.402+0000 7fb1a9678700 1 --2- 172.21.15.72:0/3507456079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1a4104250 0x7fb1a419fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:29:38.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:29:38.402+0000 7fb1a9678700 1 -- 172.21.15.72:0/3507456079 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1a41a0540 con 0x7fb1a41026b0 2024-03-22T01:29:38.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:29:38.402+0000 7fb1a9678700 1 --2- 172.21.15.72:0/3507456079 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1a41026b0 0x7fb1a41988e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb1a0009510 tx=0x7fb1a0009540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:29:38.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:29:38.403+0000 7fb19a7fc700 1 -- 172.21.15.72:0/3507456079 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a0026070 con 0x7fb1a41026b0 2024-03-22T01:29:38.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:29:38.403+0000 7fb19a7fc700 1 -- 172.21.15.72:0/3507456079 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1a0005940 con 0x7fb1a41026b0 2024-03-22T01:29:38.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:29:38.403+0000 7fb19a7fc700 1 -- 172.21.15.72:0/3507456079 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a000fdc0 con 0x7fb1a41026b0 2024-03-22T01:29:38.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:29:38.403+0000 7fb1ab8dc700 1 -- 172.21.15.72:0/3507456079 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1a41a07d0 con 0x7fb1a41026b0 2024-03-22T01:29:38.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:29:38.403+0000 7fb1ab8dc700 1 -- 172.21.15.72:0/3507456079 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1a41a0d00 con 0x7fb1a41026b0 2024-03-22T01:29:38.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:29:38.404+0000 7fb1ab8dc700 1 -- 172.21.15.72:0/3507456079 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1a4069340 con 0x7fb1a41026b0 2024-03-22T01:29:38.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:29:38.405+0000 7fb19a7fc700 1 -- 172.21.15.72:0/3507456079 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 15) ==== 137798+0+0 (secure 0 0 0) 0x7fb1a0029070 con 0x7fb1a41026b0 2024-03-22T01:29:38.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:29:38.405+0000 7fb19a7fc700 1 --2- 172.21.15.72:0/3507456079 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fb1900a2fa0 0x7fb1900a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:29:38.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:29:38.406+0000 7fb1a8e77700 1 --2- 172.21.15.72:0/3507456079 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fb1900a2fa0 0x7fb1900a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:29:38.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:29:38.406+0000 7fb19a7fc700 1 -- 172.21.15.72:0/3507456079 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3226+0+0 (secure 0 0 0) 0x7fb1a00cebf0 con 0x7fb1a41026b0 2024-03-22T01:29:38.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:29:38.406+0000 7fb1a8e77700 1 --2- 172.21.15.72:0/3507456079 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fb1900a2fa0 0x7fb1900a5460 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb194004630 tx=0x7fb19400b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:29:38.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:29:38.409+0000 7fb19a7fc700 1 -- 172.21.15.72:0/3507456079 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb1a0030080 con 0x7fb1a41026b0 2024-03-22T01:29:38.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:38 smithi089 ceph-mon[35770]: Detected new or changed devices on smithi098 2024-03-22T01:29:38.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:38 smithi089 ceph-mon[35770]: Adjusting osd_memory_target on smithi098 to 17080M 2024-03-22T01:29:38.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:38 smithi089 ceph-mon[35770]: osdmap e19: 3 total, 3 up, 3 in 2024-03-22T01:29:38.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:38 smithi089 ceph-mon[35770]: pgmap v80: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-22T01:29:38.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:38 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-22T01:29:38.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:29:38.615+0000 7fb1ab8dc700 1 -- 172.21.15.72:0/3507456079 --> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi072:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fb1a4062ac0 con 0x7fb1900a2fa0 2024-03-22T01:29:38.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:38 smithi098 ceph-mon[36468]: Detected new or changed devices on smithi098 2024-03-22T01:29:38.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:38 smithi098 ceph-mon[36468]: Adjusting osd_memory_target on smithi098 to 17080M 2024-03-22T01:29:38.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:38 smithi098 ceph-mon[36468]: osdmap e19: 3 total, 3 up, 3 in 2024-03-22T01:29:38.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:38 smithi098 ceph-mon[36468]: pgmap v80: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-22T01:29:38.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:38 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-22T01:29:39.536 INFO:journalctl@ceph.osd.0.smithi072.stdout:Mar 22 01:29:39 smithi072 sudo[49612]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-22T01:29:39.537 INFO:journalctl@ceph.osd.0.smithi072.stdout:Mar 22 01:29:39 smithi072 sudo[49612]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-22T01:29:39.537 INFO:journalctl@ceph.osd.0.smithi072.stdout:Mar 22 01:29:39 smithi072 sudo[49612]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-22T01:29:39.537 INFO:journalctl@ceph.osd.0.smithi072.stdout:Mar 22 01:29:39 smithi072 sudo[49612]: pam_unix(sudo:session): session closed for user root 2024-03-22T01:29:39.537 INFO:journalctl@ceph.osd.0.smithi072.stdout:Mar 22 01:29:39 smithi072 sudo[49617]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-22T01:29:39.537 INFO:journalctl@ceph.osd.0.smithi072.stdout:Mar 22 01:29:39 smithi072 sudo[49617]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-22T01:29:39.537 INFO:journalctl@ceph.osd.0.smithi072.stdout:Mar 22 01:29:39 smithi072 sudo[49617]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-22T01:29:39.537 INFO:journalctl@ceph.osd.0.smithi072.stdout:Mar 22 01:29:39 smithi072 sudo[49617]: pam_unix(sudo:session): session closed for user root 2024-03-22T01:29:39.537 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:39 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-22T01:29:39.537 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:39 smithi072 ceph-mon[32414]: osdmap e20: 3 total, 3 up, 3 in 2024-03-22T01:29:39.537 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:39 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-22T01:29:39.538 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:39 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T01:29:39.538 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:39 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T01:29:39.538 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:39 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:29:39.576 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:39 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-22T01:29:39.577 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:39 smithi098 ceph-mon[36468]: osdmap e20: 3 total, 3 up, 3 in 2024-03-22T01:29:39.577 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:39 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-22T01:29:39.577 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:39 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T01:29:39.577 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:39 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T01:29:39.577 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:39 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:29:39.577 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 22 01:29:39 smithi098 sudo[45691]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-22T01:29:39.577 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 22 01:29:39 smithi098 sudo[45691]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-22T01:29:39.577 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 22 01:29:39 smithi098 sudo[45691]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-22T01:29:39.577 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 22 01:29:39 smithi098 sudo[45691]: pam_unix(sudo:session): session closed for user root 2024-03-22T01:29:39.577 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 22 01:29:39 smithi098 sudo[45694]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-22T01:29:39.578 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 22 01:29:39 smithi098 sudo[45694]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-22T01:29:39.578 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 22 01:29:39 smithi098 sudo[45694]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-22T01:29:39.578 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 22 01:29:39 smithi098 sudo[45694]: pam_unix(sudo:session): session closed for user root 2024-03-22T01:29:39.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:39 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-22T01:29:39.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:39 smithi089 ceph-mon[35770]: osdmap e20: 3 total, 3 up, 3 in 2024-03-22T01:29:39.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:39 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-22T01:29:39.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:39 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T01:29:39.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:39 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T01:29:39.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:39 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:29:39.608 INFO:journalctl@ceph.osd.1.smithi089.stdout:Mar 22 01:29:39 smithi089 sudo[44527]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-22T01:29:39.608 INFO:journalctl@ceph.osd.1.smithi089.stdout:Mar 22 01:29:39 smithi089 sudo[44527]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-22T01:29:39.608 INFO:journalctl@ceph.osd.1.smithi089.stdout:Mar 22 01:29:39 smithi089 sudo[44527]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-22T01:29:39.609 INFO:journalctl@ceph.osd.1.smithi089.stdout:Mar 22 01:29:39 smithi089 sudo[44527]: pam_unix(sudo:session): session closed for user root 2024-03-22T01:29:39.609 INFO:journalctl@ceph.osd.1.smithi089.stdout:Mar 22 01:29:39 smithi089 sudo[44530]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-22T01:29:39.609 INFO:journalctl@ceph.osd.1.smithi089.stdout:Mar 22 01:29:39 smithi089 sudo[44530]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-22T01:29:39.609 INFO:journalctl@ceph.osd.1.smithi089.stdout:Mar 22 01:29:39 smithi089 sudo[44530]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-22T01:29:39.609 INFO:journalctl@ceph.osd.1.smithi089.stdout:Mar 22 01:29:39 smithi089 sudo[44530]: pam_unix(sudo:session): session closed for user root 2024-03-22T01:29:39.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:39 smithi072 sudo[49631]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-22T01:29:39.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:39 smithi072 sudo[49631]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-22T01:29:39.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:39 smithi072 sudo[49631]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-22T01:29:39.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:39 smithi072 sudo[49631]: pam_unix(sudo:session): session closed for user root 2024-03-22T01:29:39.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:39 smithi072 sudo[49634]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-22T01:29:39.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:39 smithi072 sudo[49634]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-22T01:29:39.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:39 smithi072 sudo[49634]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-22T01:29:39.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:39 smithi072 sudo[49634]: pam_unix(sudo:session): session closed for user root 2024-03-22T01:29:39.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:39 smithi098 sudo[45697]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-22T01:29:39.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:39 smithi098 sudo[45697]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-22T01:29:39.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:39 smithi098 sudo[45697]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-22T01:29:39.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:39 smithi098 sudo[45697]: pam_unix(sudo:session): session closed for user root 2024-03-22T01:29:39.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:39 smithi098 sudo[45700]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-22T01:29:39.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:39 smithi098 sudo[45700]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-22T01:29:39.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:39 smithi098 sudo[45700]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-22T01:29:39.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:39 smithi098 sudo[45700]: pam_unix(sudo:session): session closed for user root 2024-03-22T01:29:40.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:39 smithi089 sudo[44533]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-22T01:29:40.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:39 smithi089 sudo[44533]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-22T01:29:40.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:39 smithi089 sudo[44533]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-22T01:29:40.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:39 smithi089 sudo[44533]: pam_unix(sudo:session): session closed for user root 2024-03-22T01:29:40.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:39 smithi089 sudo[44536]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-22T01:29:40.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:39 smithi089 sudo[44536]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-22T01:29:40.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:39 smithi089 sudo[44536]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-22T01:29:40.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:39 smithi089 sudo[44536]: pam_unix(sudo:session): session closed for user root 2024-03-22T01:29:40.452 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:40 smithi072 ceph-mon[32414]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi072:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:29:40.453 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:40 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-22T01:29:40.453 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:40 smithi072 ceph-mon[32414]: osdmap e21: 3 total, 3 up, 3 in 2024-03-22T01:29:40.453 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:40 smithi072 ceph-mon[32414]: pgmap v83: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-22T01:29:40.453 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:40 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:40.453 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:40 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:40.453 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:40 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:40.453 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:40 smithi072 ceph-mon[32414]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-22T01:29:40.453 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:40 smithi072 ceph-mon[32414]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-22T01:29:40.453 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:40 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T01:29:40.453 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:40 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T01:29:40.453 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:40 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T01:29:40.453 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:40 smithi072 ceph-mon[32414]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-22T01:29:40.453 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:40 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T01:29:40.454 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:40 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T01:29:40.454 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:40 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T01:29:40.454 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:40 smithi072 ceph-mon[32414]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-22T01:29:40.454 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:40 smithi072 ceph-mon[32414]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-22T01:29:40.454 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:40 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T01:29:40.454 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:40 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T01:29:40.454 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:40 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T01:29:40.454 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:40 smithi072 ceph-mon[32414]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-22T01:29:40.454 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:40 smithi072 ceph-mon[32414]: osdmap e22: 3 total, 3 up, 3 in 2024-03-22T01:29:40.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:29:40.609+0000 7fb19a7fc700 1 -- 172.21.15.72:0/3507456079 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7fb1a0099000 con 0x7fb1a41026b0 2024-03-22T01:29:40.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:40 smithi089 ceph-mon[35770]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi072:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:29:40.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:40 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-22T01:29:40.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:40 smithi089 ceph-mon[35770]: osdmap e21: 3 total, 3 up, 3 in 2024-03-22T01:29:40.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:40 smithi089 ceph-mon[35770]: pgmap v83: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-22T01:29:40.609 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:40 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:40.609 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:40 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:40.609 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:40 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:40.609 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:40 smithi089 ceph-mon[35770]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-22T01:29:40.609 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:40 smithi089 ceph-mon[35770]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-22T01:29:40.609 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:40 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T01:29:40.609 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:40 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T01:29:40.609 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:40 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T01:29:40.609 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:40 smithi089 ceph-mon[35770]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-22T01:29:40.609 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:40 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T01:29:40.609 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:40 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T01:29:40.609 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:40 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T01:29:40.609 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:40 smithi089 ceph-mon[35770]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-22T01:29:40.609 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:40 smithi089 ceph-mon[35770]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-22T01:29:40.610 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:40 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T01:29:40.610 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:40 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T01:29:40.610 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:40 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T01:29:40.610 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:40 smithi089 ceph-mon[35770]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-22T01:29:40.610 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:40 smithi089 ceph-mon[35770]: osdmap e22: 3 total, 3 up, 3 in 2024-03-22T01:29:40.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:40 smithi098 ceph-mon[36468]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi072:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:29:40.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:40 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-22T01:29:40.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:40 smithi098 ceph-mon[36468]: osdmap e21: 3 total, 3 up, 3 in 2024-03-22T01:29:40.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:40 smithi098 ceph-mon[36468]: pgmap v83: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-22T01:29:40.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:40 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:40.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:40 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:40.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:40 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:40.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:40 smithi098 ceph-mon[36468]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-22T01:29:40.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:40 smithi098 ceph-mon[36468]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-22T01:29:40.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:40 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T01:29:40.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:40 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T01:29:40.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:40 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T01:29:40.673 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:40 smithi098 ceph-mon[36468]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-22T01:29:40.673 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:40 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T01:29:40.673 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:40 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T01:29:40.673 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:40 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T01:29:40.673 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:40 smithi098 ceph-mon[36468]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-22T01:29:40.673 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:40 smithi098 ceph-mon[36468]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-22T01:29:40.673 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:40 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T01:29:40.673 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:40 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T01:29:40.673 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:40 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T01:29:40.673 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:40 smithi098 ceph-mon[36468]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-22T01:29:40.673 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:40 smithi098 ceph-mon[36468]: osdmap e22: 3 total, 3 up, 3 in 2024-03-22T01:29:41.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:41 smithi089 ceph-mon[35770]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-22T01:29:41.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:41 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/3167868843' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ccc4bc40-2c48-46d2-8548-0c05b44f9f1a"}]: dispatch 2024-03-22T01:29:41.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:41 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/3167868843' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ccc4bc40-2c48-46d2-8548-0c05b44f9f1a"}]': finished 2024-03-22T01:29:41.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:41 smithi089 ceph-mon[35770]: osdmap e23: 4 total, 3 up, 4 in 2024-03-22T01:29:41.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:41 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T01:29:41.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:41 smithi089 ceph-mon[35770]: pgmap v86: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-22T01:29:41.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:41 smithi072 ceph-mon[32414]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-22T01:29:41.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:41 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/3167868843' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ccc4bc40-2c48-46d2-8548-0c05b44f9f1a"}]: dispatch 2024-03-22T01:29:41.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:41 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/3167868843' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ccc4bc40-2c48-46d2-8548-0c05b44f9f1a"}]': finished 2024-03-22T01:29:41.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:41 smithi072 ceph-mon[32414]: osdmap e23: 4 total, 3 up, 4 in 2024-03-22T01:29:41.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:41 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T01:29:41.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:41 smithi072 ceph-mon[32414]: pgmap v86: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-22T01:29:41.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:41 smithi098 ceph-mon[36468]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-22T01:29:41.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:41 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/3167868843' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ccc4bc40-2c48-46d2-8548-0c05b44f9f1a"}]: dispatch 2024-03-22T01:29:41.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:41 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/3167868843' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ccc4bc40-2c48-46d2-8548-0c05b44f9f1a"}]': finished 2024-03-22T01:29:41.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:41 smithi098 ceph-mon[36468]: osdmap e23: 4 total, 3 up, 4 in 2024-03-22T01:29:41.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:41 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T01:29:41.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:41 smithi098 ceph-mon[36468]: pgmap v86: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-22T01:29:42.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:42 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/2862699244' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T01:29:42.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:42 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/2862699244' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T01:29:42.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:42 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/2862699244' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T01:29:43.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:43 smithi072 ceph-mon[32414]: pgmap v87: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-22T01:29:43.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:43 smithi098 ceph-mon[36468]: pgmap v87: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-22T01:29:44.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:43 smithi089 ceph-mon[35770]: pgmap v87: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-22T01:29:46.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:46 smithi098 ceph-mon[36468]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-22T01:29:46.759 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:46 smithi072 ceph-mon[32414]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-22T01:29:46.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:46 smithi089 ceph-mon[35770]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-22T01:29:48.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:48 smithi098 ceph-mon[36468]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-22T01:29:48.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:48 smithi089 ceph-mon[35770]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-22T01:29:48.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:48 smithi072 ceph-mon[32414]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-22T01:29:49.616 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:49 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-22T01:29:49.616 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:49 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:29:49.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:49 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-22T01:29:49.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:49 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:29:49.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:49 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-22T01:29:49.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:49 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:29:50.594 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:50 smithi072 ceph-mon[32414]: Deploying daemon osd.3 on smithi072 2024-03-22T01:29:50.594 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:50 smithi072 ceph-mon[32414]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-22T01:29:50.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:50 smithi089 ceph-mon[35770]: Deploying daemon osd.3 on smithi072 2024-03-22T01:29:50.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:50 smithi089 ceph-mon[35770]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-22T01:29:50.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:50 smithi098 ceph-mon[36468]: Deploying daemon osd.3 on smithi072 2024-03-22T01:29:50.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:50 smithi098 ceph-mon[36468]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-22T01:29:52.768 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:52 smithi072 ceph-mon[32414]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-22T01:29:52.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:52 smithi089 ceph-mon[35770]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-22T01:29:52.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:52 smithi098 ceph-mon[36468]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-22T01:29:54.723 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:54 smithi098 ceph-mon[36468]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-22T01:29:54.724 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:54 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:54.724 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:54 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:54.815 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:54 smithi072 ceph-mon[32414]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-22T01:29:54.815 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:54 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:54.816 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:54 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:54.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:54 smithi089 ceph-mon[35770]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-22T01:29:54.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:54 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:54.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:54 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:55.774 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:55 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:29:55.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:55 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:29:55.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:55 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:29:56.584 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:56 smithi072 ceph-mon[32414]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-22T01:29:56.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:56 smithi089 ceph-mon[35770]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-22T01:29:56.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:56 smithi098 ceph-mon[36468]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-22T01:29:58.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:29:58.020+0000 7fb19a7fc700 1 -- 172.21.15.72:0/3507456079 <== mgr.14152 v2:172.21.15.72:6800/401403200 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fb1a4062ac0 con 0x7fb1900a2fa0 2024-03-22T01:29:58.019 INFO:teuthology.orchestra.run.smithi072.stdout:Created osd(s) 3 on host 'smithi072' 2024-03-22T01:29:58.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:29:58.022+0000 7fb1ab8dc700 1 -- 172.21.15.72:0/3507456079 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fb1900a2fa0 msgr2=0x7fb1900a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:29:58.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:29:58.022+0000 7fb1ab8dc700 1 --2- 172.21.15.72:0/3507456079 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fb1900a2fa0 0x7fb1900a5460 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb194004630 tx=0x7fb19400b3c0 comp rx=0 tx=0).stop 2024-03-22T01:29:58.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:29:58.022+0000 7fb1ab8dc700 1 -- 172.21.15.72:0/3507456079 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1a41026b0 msgr2=0x7fb1a41988e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:29:58.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:29:58.022+0000 7fb1ab8dc700 1 --2- 172.21.15.72:0/3507456079 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1a41026b0 0x7fb1a41988e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb1a0009510 tx=0x7fb1a0009540 comp rx=0 tx=0).stop 2024-03-22T01:29:58.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:29:58.022+0000 7fb1ab8dc700 1 -- 172.21.15.72:0/3507456079 shutdown_connections 2024-03-22T01:29:58.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:29:58.022+0000 7fb1ab8dc700 1 --2- 172.21.15.72:0/3507456079 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fb1900a2fa0 0x7fb1900a5460 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:29:58.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:29:58.022+0000 7fb1ab8dc700 1 --2- 172.21.15.72:0/3507456079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1a4104250 0x7fb1a419fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:29:58.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:29:58.022+0000 7fb1ab8dc700 1 --2- 172.21.15.72:0/3507456079 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1a41026b0 0x7fb1a41988e0 secure :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb1a0009510 tx=0x7fb1a0009540 comp rx=0 tx=0).stop 2024-03-22T01:29:58.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:29:58.022+0000 7fb1ab8dc700 1 --2- 172.21.15.72:0/3507456079 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1a41038b0 0x7fb1a4198e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:29:58.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:29:58.022+0000 7fb1ab8dc700 1 -- 172.21.15.72:0/3507456079 >> 172.21.15.72:0/3507456079 conn(0x7fb1a40fdc90 msgr2=0x7fb1a4106c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:29:58.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:29:58.023+0000 7fb1ab8dc700 1 -- 172.21.15.72:0/3507456079 shutdown_connections 2024-03-22T01:29:58.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:29:58.023+0000 7fb1ab8dc700 1 -- 172.21.15.72:0/3507456079 wait complete. 2024-03-22T01:29:58.318 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:58 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:58.318 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:58 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:58.318 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:58 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:29:58.318 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:58 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:29:58.318 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:58 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:58.318 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:58 smithi072 ceph-mon[32414]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-22T01:29:58.318 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:58 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:58.319 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:58 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:58.319 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:58 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:29:58.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:58 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:58.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:58 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:58.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:58 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:29:58.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:58 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:29:58.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:58 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:58.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:58 smithi089 ceph-mon[35770]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-22T01:29:58.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:58 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:58.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:58 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:58.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:58 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:29:58.645 DEBUG:teuthology.orchestra.run.smithi072:osd.3> sudo journalctl -f -n 0 -u ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@osd.3.service 2024-03-22T01:29:58.648 INFO:tasks.cephadm:Deploying osd.4 on smithi089 with /dev/vg_nvme/lv_3... 2024-03-22T01:29:58.648 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-22T01:29:58.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:58 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:58.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:58 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:58.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:58 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:29:58.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:58 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:29:58.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:58 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:58.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:58 smithi098 ceph-mon[36468]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-22T01:29:58.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:58 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:58.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:58 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:29:58.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:58 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:29:58.891 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.b/config 2024-03-22T01:29:59.313 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:29:59 smithi089 ceph-mon[35770]: from='osd.3 [v2:172.21.15.72:6810/1865951395,v1:172.21.15.72:6811/1865951395]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-22T01:29:59.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:29:59 smithi072 ceph-mon[32414]: from='osd.3 [v2:172.21.15.72:6810/1865951395,v1:172.21.15.72:6811/1865951395]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-22T01:29:59.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:29:59 smithi098 ceph-mon[36468]: from='osd.3 [v2:172.21.15.72:6810/1865951395,v1:172.21.15.72:6811/1865951395]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-22T01:30:00.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:00 smithi089 ceph-mon[35770]: from='osd.3 [v2:172.21.15.72:6810/1865951395,v1:172.21.15.72:6811/1865951395]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-22T01:30:00.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:00 smithi089 ceph-mon[35770]: osdmap e24: 4 total, 3 up, 4 in 2024-03-22T01:30:00.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:00 smithi089 ceph-mon[35770]: from='osd.3 [v2:172.21.15.72:6810/1865951395,v1:172.21.15.72:6811/1865951395]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]: dispatch 2024-03-22T01:30:00.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:00 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T01:30:00.609 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:00 smithi089 ceph-mon[35770]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-22T01:30:00.609 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:00 smithi089 ceph-mon[35770]: overall HEALTH_OK 2024-03-22T01:30:00.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:00 smithi072 ceph-mon[32414]: from='osd.3 [v2:172.21.15.72:6810/1865951395,v1:172.21.15.72:6811/1865951395]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-22T01:30:00.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:00 smithi072 ceph-mon[32414]: osdmap e24: 4 total, 3 up, 4 in 2024-03-22T01:30:00.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:00 smithi072 ceph-mon[32414]: from='osd.3 [v2:172.21.15.72:6810/1865951395,v1:172.21.15.72:6811/1865951395]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]: dispatch 2024-03-22T01:30:00.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:00 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T01:30:00.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:00 smithi072 ceph-mon[32414]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-22T01:30:00.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:00 smithi072 ceph-mon[32414]: overall HEALTH_OK 2024-03-22T01:30:00.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:00 smithi098 ceph-mon[36468]: from='osd.3 [v2:172.21.15.72:6810/1865951395,v1:172.21.15.72:6811/1865951395]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-22T01:30:00.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:00 smithi098 ceph-mon[36468]: osdmap e24: 4 total, 3 up, 4 in 2024-03-22T01:30:00.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:00 smithi098 ceph-mon[36468]: from='osd.3 [v2:172.21.15.72:6810/1865951395,v1:172.21.15.72:6811/1865951395]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]: dispatch 2024-03-22T01:30:00.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:00 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T01:30:00.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:00 smithi098 ceph-mon[36468]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-22T01:30:00.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:00 smithi098 ceph-mon[36468]: overall HEALTH_OK 2024-03-22T01:30:00.777 INFO:teuthology.orchestra.run.smithi089.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-22T01:30:00.778 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-22T01:30:00.778 INFO:teuthology.orchestra.run.smithi089.stderr: stderr: 10+0 records in 2024-03-22T01:30:00.778 INFO:teuthology.orchestra.run.smithi089.stderr:10+0 records out 2024-03-22T01:30:00.778 INFO:teuthology.orchestra.run.smithi089.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0112863 s, 929 MB/s 2024-03-22T01:30:00.778 INFO:teuthology.orchestra.run.smithi089.stderr:--> Zapping successful for: 2024-03-22T01:30:01.240 INFO:journalctl@ceph.osd.3.smithi072.stdout:Mar 22 01:30:00 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-3[51749]: 2024-03-22T01:30:00.951+0000 7f461bcd9700 -1 osd.3 0 waiting for initial osdmap 2024-03-22T01:30:01.240 INFO:journalctl@ceph.osd.3.smithi072.stdout:Mar 22 01:30:00 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-3[51749]: 2024-03-22T01:30:00.957+0000 7f4613ac4700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-22T01:30:01.424 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:01 smithi089 ceph-mon[35770]: from='osd.3 [v2:172.21.15.72:6810/1865951395,v1:172.21.15.72:6811/1865951395]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]': finished 2024-03-22T01:30:01.424 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:01 smithi089 ceph-mon[35770]: osdmap e25: 4 total, 3 up, 4 in 2024-03-22T01:30:01.424 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:01 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T01:30:01.424 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:01 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T01:30:01.424 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:01 smithi089 ceph-mon[35770]: from='osd.3 [v2:172.21.15.72:6810/1865951395,v1:172.21.15.72:6811/1865951395]' entity='osd.3' 2024-03-22T01:30:01.424 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:01 smithi089 ceph-mon[35770]: osd.3 [v2:172.21.15.72:6810/1865951395,v1:172.21.15.72:6811/1865951395] boot 2024-03-22T01:30:01.424 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:01 smithi089 ceph-mon[35770]: osdmap e26: 4 total, 4 up, 4 in 2024-03-22T01:30:01.424 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:01 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T01:30:01.450 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi089:vg_nvme/lv_3 2024-03-22T01:30:01.552 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:01 smithi072 ceph-mon[32414]: from='osd.3 [v2:172.21.15.72:6810/1865951395,v1:172.21.15.72:6811/1865951395]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]': finished 2024-03-22T01:30:01.552 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:01 smithi072 ceph-mon[32414]: osdmap e25: 4 total, 3 up, 4 in 2024-03-22T01:30:01.552 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:01 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T01:30:01.552 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:01 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T01:30:01.552 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:01 smithi072 ceph-mon[32414]: from='osd.3 [v2:172.21.15.72:6810/1865951395,v1:172.21.15.72:6811/1865951395]' entity='osd.3' 2024-03-22T01:30:01.553 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:01 smithi072 ceph-mon[32414]: osd.3 [v2:172.21.15.72:6810/1865951395,v1:172.21.15.72:6811/1865951395] boot 2024-03-22T01:30:01.553 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:01 smithi072 ceph-mon[32414]: osdmap e26: 4 total, 4 up, 4 in 2024-03-22T01:30:01.553 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:01 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T01:30:01.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:01 smithi098 ceph-mon[36468]: from='osd.3 [v2:172.21.15.72:6810/1865951395,v1:172.21.15.72:6811/1865951395]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]': finished 2024-03-22T01:30:01.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:01 smithi098 ceph-mon[36468]: osdmap e25: 4 total, 3 up, 4 in 2024-03-22T01:30:01.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:01 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T01:30:01.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:01 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T01:30:01.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:01 smithi098 ceph-mon[36468]: from='osd.3 [v2:172.21.15.72:6810/1865951395,v1:172.21.15.72:6811/1865951395]' entity='osd.3' 2024-03-22T01:30:01.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:01 smithi098 ceph-mon[36468]: osd.3 [v2:172.21.15.72:6810/1865951395,v1:172.21.15.72:6811/1865951395] boot 2024-03-22T01:30:01.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:01 smithi098 ceph-mon[36468]: osdmap e26: 4 total, 4 up, 4 in 2024-03-22T01:30:01.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:01 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T01:30:01.685 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.b/config 2024-03-22T01:30:02.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:02 smithi089 ceph-mon[35770]: purged_snaps scrub starts 2024-03-22T01:30:02.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:02 smithi089 ceph-mon[35770]: purged_snaps scrub ok 2024-03-22T01:30:02.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:02 smithi089 ceph-mon[35770]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 357 GiB / 358 GiB avail 2024-03-22T01:30:02.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:02 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:02.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:02 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:02.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:02 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:30:02.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:02 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:30:02.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:02 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:02.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:02 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:30:02.359 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:02 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:30:02.359 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:02 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:02.359 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:02 smithi089 ceph-mon[35770]: osdmap e27: 4 total, 4 up, 4 in 2024-03-22T01:30:02.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:02 smithi072 ceph-mon[32414]: purged_snaps scrub starts 2024-03-22T01:30:02.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:02 smithi072 ceph-mon[32414]: purged_snaps scrub ok 2024-03-22T01:30:02.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:02 smithi072 ceph-mon[32414]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 357 GiB / 358 GiB avail 2024-03-22T01:30:02.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:02 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:02.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:02 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:02.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:02 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:30:02.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:02 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:30:02.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:02 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:02.624 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:02 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:30:02.624 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:02 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:30:02.624 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:02 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:02.624 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:02 smithi072 ceph-mon[32414]: osdmap e27: 4 total, 4 up, 4 in 2024-03-22T01:30:02.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:02 smithi098 ceph-mon[36468]: purged_snaps scrub starts 2024-03-22T01:30:02.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:02 smithi098 ceph-mon[36468]: purged_snaps scrub ok 2024-03-22T01:30:02.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:02 smithi098 ceph-mon[36468]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 357 GiB / 358 GiB avail 2024-03-22T01:30:02.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:02 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:02.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:02 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:02.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:02 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:30:02.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:02 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:30:02.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:02 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:02.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:02 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:30:02.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:02 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:30:02.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:02 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:02.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:02 smithi098 ceph-mon[36468]: osdmap e27: 4 total, 4 up, 4 in 2024-03-22T01:30:03.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:30:03.280+0000 7f6b42473700 1 -- 172.21.15.89:0/1291078543 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b3c103140 msgr2=0x7f6b3c1099f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:30:03.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:30:03.280+0000 7f6b42473700 1 --2- 172.21.15.89:0/1291078543 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b3c103140 0x7f6b3c1099f0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6b3400b0d0 tx=0x7f6b3400b3e0 comp rx=0 tx=0).stop 2024-03-22T01:30:03.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:30:03.280+0000 7f6b42473700 1 -- 172.21.15.89:0/1291078543 shutdown_connections 2024-03-22T01:30:03.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:30:03.280+0000 7f6b42473700 1 --2- 172.21.15.89:0/1291078543 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b3c103140 0x7f6b3c1099f0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:30:03.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:30:03.280+0000 7f6b42473700 1 --2- 172.21.15.89:0/1291078543 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b3c1015a0 0x7f6b3c1019c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:30:03.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:30:03.280+0000 7f6b42473700 1 --2- 172.21.15.89:0/1291078543 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b3c1027a0 0x7f6b3c102c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:30:03.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:30:03.280+0000 7f6b42473700 1 -- 172.21.15.89:0/1291078543 >> 172.21.15.89:0/1291078543 conn(0x7f6b3c0fcb40 msgr2=0x7f6b3c0fef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:30:03.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:30:03.280+0000 7f6b42473700 1 -- 172.21.15.89:0/1291078543 shutdown_connections 2024-03-22T01:30:03.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:30:03.280+0000 7f6b42473700 1 -- 172.21.15.89:0/1291078543 wait complete. 2024-03-22T01:30:03.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:30:03.281+0000 7f6b42473700 1 Processor -- start 2024-03-22T01:30:03.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:30:03.281+0000 7f6b42473700 1 -- start start 2024-03-22T01:30:03.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:30:03.281+0000 7f6b42473700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b3c1015a0 0x7f6b3c197690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:30:03.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:30:03.281+0000 7f6b42473700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b3c1027a0 0x7f6b3c197bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:30:03.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:30:03.281+0000 7f6b42473700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b3c103140 0x7f6b3c19ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:30:03.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:30:03.281+0000 7f6b42473700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f6b3c10d170 con 0x7f6b3c103140 2024-03-22T01:30:03.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:30:03.281+0000 7f6b42473700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f6b3c10cff0 con 0x7f6b3c1027a0 2024-03-22T01:30:03.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:30:03.281+0000 7f6b42473700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6b3c10d2f0 con 0x7f6b3c1015a0 2024-03-22T01:30:03.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:30:03.281+0000 7f6b3b7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b3c1027a0 0x7f6b3c197bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:30:03.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:30:03.281+0000 7f6b3b7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b3c1027a0 0x7f6b3c197bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:32886/0 (socket says 172.21.15.89:32886) 2024-03-22T01:30:03.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:30:03.281+0000 7f6b3b7fe700 1 -- 172.21.15.89:0/1946813845 learned_addr learned my addr 172.21.15.89:0/1946813845 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-22T01:30:03.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:30:03.281+0000 7f6b40a10700 1 --2- 172.21.15.89:0/1946813845 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b3c103140 0x7f6b3c19ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:30:03.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:30:03.281+0000 7f6b3bfff700 1 --2- 172.21.15.89:0/1946813845 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b3c1015a0 0x7f6b3c197690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:30:03.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:30:03.281+0000 7f6b3b7fe700 1 -- 172.21.15.89:0/1946813845 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b3c1015a0 msgr2=0x7f6b3c197690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:30:03.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:30:03.281+0000 7f6b3b7fe700 1 --2- 172.21.15.89:0/1946813845 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b3c1015a0 0x7f6b3c197690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:30:03.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:30:03.281+0000 7f6b3b7fe700 1 -- 172.21.15.89:0/1946813845 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b3c103140 msgr2=0x7f6b3c19ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:30:03.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:30:03.281+0000 7f6b3b7fe700 1 --2- 172.21.15.89:0/1946813845 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b3c103140 0x7f6b3c19ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:30:03.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:30:03.281+0000 7f6b3b7fe700 1 -- 172.21.15.89:0/1946813845 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b3c19f2f0 con 0x7f6b3c1027a0 2024-03-22T01:30:03.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:30:03.281+0000 7f6b40a10700 1 --2- 172.21.15.89:0/1946813845 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b3c103140 0x7f6b3c19ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:30:03.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:30:03.282+0000 7f6b3b7fe700 1 --2- 172.21.15.89:0/1946813845 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b3c1027a0 0x7f6b3c197bd0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c00d950 tx=0x7f6b2c00dd10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:30:03.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:30:03.282+0000 7f6b397fa700 1 -- 172.21.15.89:0/1946813845 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b2c016070 con 0x7f6b3c1027a0 2024-03-22T01:30:03.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:30:03.282+0000 7f6b42473700 1 -- 172.21.15.89:0/1946813845 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b3c06ac20 con 0x7f6b3c1027a0 2024-03-22T01:30:03.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:30:03.282+0000 7f6b397fa700 1 -- 172.21.15.89:0/1946813845 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b2c009b40 con 0x7f6b3c1027a0 2024-03-22T01:30:03.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:30:03.282+0000 7f6b397fa700 1 -- 172.21.15.89:0/1946813845 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b2c00f620 con 0x7f6b3c1027a0 2024-03-22T01:30:03.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:30:03.282+0000 7f6b42473700 1 -- 172.21.15.89:0/1946813845 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b3c06b160 con 0x7f6b3c1027a0 2024-03-22T01:30:03.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:30:03.283+0000 7f6b42473700 1 -- 172.21.15.89:0/1946813845 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b3c069340 con 0x7f6b3c1027a0 2024-03-22T01:30:03.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:30:03.284+0000 7f6b397fa700 1 -- 172.21.15.89:0/1946813845 <== mon.2 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f6b2c019070 con 0x7f6b3c1027a0 2024-03-22T01:30:03.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:30:03.284+0000 7f6b397fa700 1 --2- 172.21.15.89:0/1946813845 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f6b240a51c0 0x7f6b240a7680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:30:03.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:30:03.284+0000 7f6b397fa700 1 -- 172.21.15.89:0/1946813845 <== mon.2 v2:172.21.15.89:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 3698+0+0 (secure 0 0 0) 0x7f6b2c0ce200 con 0x7f6b3c1027a0 2024-03-22T01:30:03.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:30:03.285+0000 7f6b3bfff700 1 --2- 172.21.15.89:0/1946813845 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f6b240a51c0 0x7f6b240a7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:30:03.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:30:03.285+0000 7f6b3bfff700 1 --2- 172.21.15.89:0/1946813845 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f6b240a51c0 0x7f6b240a7680 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6b30005fd0 tx=0x7f6b30005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:30:03.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:30:03.289+0000 7f6b397fa700 1 -- 172.21.15.89:0/1946813845 <== mon.2 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6b2c09c230 con 0x7f6b3c1027a0 2024-03-22T01:30:03.357 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:03 smithi089 ceph-mon[35770]: Detected new or changed devices on smithi072 2024-03-22T01:30:03.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:03 smithi089 ceph-mon[35770]: Adjusting osd_memory_target on smithi072 to 8544M 2024-03-22T01:30:03.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:03 smithi089 ceph-mon[35770]: osdmap e28: 4 total, 4 up, 4 in 2024-03-22T01:30:03.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:30:03.493+0000 7f6b42473700 1 -- 172.21.15.89:0/1946813845 --> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f6b3c06b3c0 con 0x7f6b240a51c0 2024-03-22T01:30:03.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:03 smithi072 ceph-mon[32414]: Detected new or changed devices on smithi072 2024-03-22T01:30:03.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:03 smithi072 ceph-mon[32414]: Adjusting osd_memory_target on smithi072 to 8544M 2024-03-22T01:30:03.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:03 smithi072 ceph-mon[32414]: osdmap e28: 4 total, 4 up, 4 in 2024-03-22T01:30:03.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:03 smithi098 ceph-mon[36468]: Detected new or changed devices on smithi072 2024-03-22T01:30:03.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:03 smithi098 ceph-mon[36468]: Adjusting osd_memory_target on smithi072 to 8544M 2024-03-22T01:30:03.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:03 smithi098 ceph-mon[36468]: osdmap e28: 4 total, 4 up, 4 in 2024-03-22T01:30:04.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:04 smithi089 ceph-mon[35770]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 357 GiB / 358 GiB avail 2024-03-22T01:30:04.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:04 smithi089 ceph-mon[35770]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:30:04.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:04 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T01:30:04.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:04 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T01:30:04.359 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:04 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:30:04.359 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:04 smithi089 ceph-mon[35770]: osdmap e29: 4 total, 4 up, 4 in 2024-03-22T01:30:04.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:04 smithi072 ceph-mon[32414]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 357 GiB / 358 GiB avail 2024-03-22T01:30:04.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:04 smithi072 ceph-mon[32414]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:30:04.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:04 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T01:30:04.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:04 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T01:30:04.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:04 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:30:04.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:04 smithi072 ceph-mon[32414]: osdmap e29: 4 total, 4 up, 4 in 2024-03-22T01:30:04.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:04 smithi098 ceph-mon[36468]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 357 GiB / 358 GiB avail 2024-03-22T01:30:04.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:04 smithi098 ceph-mon[36468]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:30:04.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:04 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T01:30:04.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:04 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T01:30:04.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:04 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:30:04.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:04 smithi098 ceph-mon[36468]: osdmap e29: 4 total, 4 up, 4 in 2024-03-22T01:30:05.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:05 smithi089 ceph-mon[35770]: osdmap e30: 4 total, 4 up, 4 in 2024-03-22T01:30:05.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:05 smithi072 ceph-mon[32414]: osdmap e30: 4 total, 4 up, 4 in 2024-03-22T01:30:05.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:05 smithi098 ceph-mon[36468]: osdmap e30: 4 total, 4 up, 4 in 2024-03-22T01:30:06.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:06 smithi089 ceph-mon[35770]: pgmap v105: 1 pgs: 1 peering; 577 KiB data, 236 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-22T01:30:06.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:06 smithi089 ceph-mon[35770]: from='client.? 172.21.15.89:0/1917363203' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1812439f-810a-4ed2-9395-c00c9a085a13"}]: dispatch 2024-03-22T01:30:06.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:06 smithi089 ceph-mon[35770]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1812439f-810a-4ed2-9395-c00c9a085a13"}]: dispatch 2024-03-22T01:30:06.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:06 smithi089 ceph-mon[35770]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1812439f-810a-4ed2-9395-c00c9a085a13"}]': finished 2024-03-22T01:30:06.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:06 smithi089 ceph-mon[35770]: osdmap e31: 5 total, 4 up, 5 in 2024-03-22T01:30:06.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:06 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T01:30:06.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:06 smithi072 ceph-mon[32414]: pgmap v105: 1 pgs: 1 peering; 577 KiB data, 236 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-22T01:30:06.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:06 smithi072 ceph-mon[32414]: from='client.? 172.21.15.89:0/1917363203' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1812439f-810a-4ed2-9395-c00c9a085a13"}]: dispatch 2024-03-22T01:30:06.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:06 smithi072 ceph-mon[32414]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1812439f-810a-4ed2-9395-c00c9a085a13"}]: dispatch 2024-03-22T01:30:06.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:06 smithi072 ceph-mon[32414]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1812439f-810a-4ed2-9395-c00c9a085a13"}]': finished 2024-03-22T01:30:06.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:06 smithi072 ceph-mon[32414]: osdmap e31: 5 total, 4 up, 5 in 2024-03-22T01:30:06.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:06 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T01:30:06.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:06 smithi098 ceph-mon[36468]: pgmap v105: 1 pgs: 1 peering; 577 KiB data, 236 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-22T01:30:06.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:06 smithi098 ceph-mon[36468]: from='client.? 172.21.15.89:0/1917363203' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1812439f-810a-4ed2-9395-c00c9a085a13"}]: dispatch 2024-03-22T01:30:06.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:06 smithi098 ceph-mon[36468]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1812439f-810a-4ed2-9395-c00c9a085a13"}]: dispatch 2024-03-22T01:30:06.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:06 smithi098 ceph-mon[36468]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1812439f-810a-4ed2-9395-c00c9a085a13"}]': finished 2024-03-22T01:30:06.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:06 smithi098 ceph-mon[36468]: osdmap e31: 5 total, 4 up, 5 in 2024-03-22T01:30:06.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:06 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T01:30:07.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:07 smithi089 ceph-mon[35770]: from='client.? 172.21.15.89:0/3017007700' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T01:30:07.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:07 smithi072 ceph-mon[32414]: from='client.? 172.21.15.89:0/3017007700' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T01:30:07.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:07 smithi098 ceph-mon[36468]: from='client.? 172.21.15.89:0/3017007700' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T01:30:08.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:08 smithi089 ceph-mon[35770]: pgmap v107: 1 pgs: 1 peering; 577 KiB data, 113 MiB used, 357 GiB / 358 GiB avail; 138 KiB/s, 0 objects/s recovering 2024-03-22T01:30:08.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:08 smithi072 ceph-mon[32414]: pgmap v107: 1 pgs: 1 peering; 577 KiB data, 113 MiB used, 357 GiB / 358 GiB avail; 138 KiB/s, 0 objects/s recovering 2024-03-22T01:30:08.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:08 smithi098 ceph-mon[36468]: pgmap v107: 1 pgs: 1 peering; 577 KiB data, 113 MiB used, 357 GiB / 358 GiB avail; 138 KiB/s, 0 objects/s recovering 2024-03-22T01:30:09.827 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:09 smithi089 ceph-mon[35770]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-22T01:30:09.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:09 smithi072 ceph-mon[32414]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-22T01:30:09.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:09 smithi098 ceph-mon[36468]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-22T01:30:12.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:12 smithi089 ceph-mon[35770]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-03-22T01:30:12.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:12 smithi072 ceph-mon[32414]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-03-22T01:30:12.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:12 smithi098 ceph-mon[36468]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-03-22T01:30:13.773 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:13 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-22T01:30:13.773 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:13 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:30:13.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:13 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-22T01:30:13.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:13 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:30:13.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:13 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-22T01:30:13.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:13 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:30:14.587 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:14 smithi089 ceph-mon[35770]: Deploying daemon osd.4 on smithi089 2024-03-22T01:30:14.587 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:14 smithi089 ceph-mon[35770]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-22T01:30:14.788 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:14 smithi072 ceph-mon[32414]: Deploying daemon osd.4 on smithi089 2024-03-22T01:30:14.788 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:14 smithi072 ceph-mon[32414]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-22T01:30:14.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:14 smithi098 ceph-mon[36468]: Deploying daemon osd.4 on smithi089 2024-03-22T01:30:14.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:14 smithi098 ceph-mon[36468]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-22T01:30:16.725 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:16 smithi089 ceph-mon[35770]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-22T01:30:16.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:16 smithi072 ceph-mon[32414]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-22T01:30:16.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:16 smithi098 ceph-mon[36468]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-22T01:30:18.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:18 smithi089 ceph-mon[35770]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-22T01:30:18.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:18 smithi072 ceph-mon[32414]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-22T01:30:18.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:18 smithi098 ceph-mon[36468]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-22T01:30:20.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:19 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:20.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:19 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:20.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:19 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:30:20.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:19 smithi098 ceph-mon[36468]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-22T01:30:20.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:19 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:20.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:19 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:20.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:19 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:30:20.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:19 smithi089 ceph-mon[35770]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-22T01:30:20.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:19 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:20.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:19 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:20.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:19 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:30:20.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:19 smithi072 ceph-mon[32414]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-22T01:30:22.741 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:22 smithi089 ceph-mon[35770]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-22T01:30:22.741 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:22 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:22.741 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:22 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:22.742 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:22 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:30:22.742 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:22 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:30:22.742 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:22 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:22.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:30:22.808+0000 7f6b397fa700 1 -- 172.21.15.89:0/1946813845 <== mgr.14152 v2:172.21.15.72:6800/401403200 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f6b3c06b3c0 con 0x7f6b240a51c0 2024-03-22T01:30:22.808 INFO:teuthology.orchestra.run.smithi089.stdout:Created osd(s) 4 on host 'smithi089' 2024-03-22T01:30:22.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:30:22.810+0000 7f6b42473700 1 -- 172.21.15.89:0/1946813845 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f6b240a51c0 msgr2=0x7f6b240a7680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:30:22.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:30:22.810+0000 7f6b42473700 1 --2- 172.21.15.89:0/1946813845 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f6b240a51c0 0x7f6b240a7680 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6b30005fd0 tx=0x7f6b30005e50 comp rx=0 tx=0).stop 2024-03-22T01:30:22.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:30:22.810+0000 7f6b42473700 1 -- 172.21.15.89:0/1946813845 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b3c1027a0 msgr2=0x7f6b3c197bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:30:22.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:30:22.810+0000 7f6b42473700 1 --2- 172.21.15.89:0/1946813845 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b3c1027a0 0x7f6b3c197bd0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c00d950 tx=0x7f6b2c00dd10 comp rx=0 tx=0).stop 2024-03-22T01:30:22.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:30:22.811+0000 7f6b42473700 1 -- 172.21.15.89:0/1946813845 shutdown_connections 2024-03-22T01:30:22.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:30:22.811+0000 7f6b42473700 1 --2- 172.21.15.89:0/1946813845 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f6b240a51c0 0x7f6b240a7680 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:30:22.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:30:22.811+0000 7f6b42473700 1 --2- 172.21.15.89:0/1946813845 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b3c103140 0x7f6b3c19ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:30:22.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:30:22.811+0000 7f6b42473700 1 --2- 172.21.15.89:0/1946813845 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b3c1015a0 0x7f6b3c197690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:30:22.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:30:22.811+0000 7f6b42473700 1 --2- 172.21.15.89:0/1946813845 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b3c1027a0 0x7f6b3c197bd0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:30:22.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:30:22.811+0000 7f6b42473700 1 -- 172.21.15.89:0/1946813845 >> 172.21.15.89:0/1946813845 conn(0x7f6b3c0fcb40 msgr2=0x7f6b3c1059d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:30:22.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:30:22.811+0000 7f6b42473700 1 -- 172.21.15.89:0/1946813845 shutdown_connections 2024-03-22T01:30:22.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:30:22.811+0000 7f6b42473700 1 -- 172.21.15.89:0/1946813845 wait complete. 2024-03-22T01:30:22.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:22 smithi072 ceph-mon[32414]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-22T01:30:22.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:22 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:22.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:22 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:22.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:22 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:30:22.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:22 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:30:22.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:22 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:22.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:22 smithi098 ceph-mon[36468]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-22T01:30:22.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:22 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:22.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:22 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:22.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:22 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:30:22.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:22 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:30:22.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:22 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:23.521 DEBUG:teuthology.orchestra.run.smithi089:osd.4> sudo journalctl -f -n 0 -u ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@osd.4.service 2024-03-22T01:30:23.523 INFO:tasks.cephadm:Deploying osd.5 on smithi098 with /dev/vg_nvme/lv_3... 2024-03-22T01:30:23.523 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-22T01:30:23.770 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.c/config 2024-03-22T01:30:23.892 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:23 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:23.892 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:23 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:23.892 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:23 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:30:23.892 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:23 smithi098 ceph-mon[36468]: from='osd.4 [v2:172.21.15.89:6808/2683760795,v1:172.21.15.89:6809/2683760795]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-22T01:30:23.893 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:23 smithi098 ceph-mon[36468]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-22T01:30:23.893 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:23 smithi098 ceph-mon[36468]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-22T01:30:24.090 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:23 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:24.090 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:23 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:24.090 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:23 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:30:24.091 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:23 smithi089 ceph-mon[35770]: from='osd.4 [v2:172.21.15.89:6808/2683760795,v1:172.21.15.89:6809/2683760795]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-22T01:30:24.091 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:23 smithi089 ceph-mon[35770]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-22T01:30:24.091 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:23 smithi089 ceph-mon[35770]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-22T01:30:24.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:23 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:24.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:23 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:24.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:23 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:30:24.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:23 smithi072 ceph-mon[32414]: from='osd.4 [v2:172.21.15.89:6808/2683760795,v1:172.21.15.89:6809/2683760795]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-22T01:30:24.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:23 smithi072 ceph-mon[32414]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-22T01:30:24.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:23 smithi072 ceph-mon[32414]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-22T01:30:24.887 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:24 smithi098 ceph-mon[36468]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-22T01:30:24.887 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:24 smithi098 ceph-mon[36468]: from='osd.4 [v2:172.21.15.89:6808/2683760795,v1:172.21.15.89:6809/2683760795]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-03-22T01:30:24.888 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:24 smithi098 ceph-mon[36468]: osdmap e32: 5 total, 4 up, 5 in 2024-03-22T01:30:24.888 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:24 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T01:30:24.888 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:24 smithi098 ceph-mon[36468]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-03-22T01:30:25.074 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:24 smithi089 ceph-mon[35770]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-22T01:30:25.074 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:24 smithi089 ceph-mon[35770]: from='osd.4 [v2:172.21.15.89:6808/2683760795,v1:172.21.15.89:6809/2683760795]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-03-22T01:30:25.075 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:24 smithi089 ceph-mon[35770]: osdmap e32: 5 total, 4 up, 5 in 2024-03-22T01:30:25.075 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:24 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T01:30:25.075 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:24 smithi089 ceph-mon[35770]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-03-22T01:30:25.075 INFO:journalctl@ceph.osd.4.smithi089.stdout:Mar 22 01:30:24 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-4[47352]: 2024-03-22T01:30:24.801+0000 7fda48866700 -1 osd.4 0 waiting for initial osdmap 2024-03-22T01:30:25.075 INFO:journalctl@ceph.osd.4.smithi089.stdout:Mar 22 01:30:24 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-4[47352]: 2024-03-22T01:30:24.807+0000 7fda44659700 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-22T01:30:25.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:24 smithi072 ceph-mon[32414]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-22T01:30:25.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:24 smithi072 ceph-mon[32414]: from='osd.4 [v2:172.21.15.89:6808/2683760795,v1:172.21.15.89:6809/2683760795]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-03-22T01:30:25.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:24 smithi072 ceph-mon[32414]: osdmap e32: 5 total, 4 up, 5 in 2024-03-22T01:30:25.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:24 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T01:30:25.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:24 smithi072 ceph-mon[32414]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-03-22T01:30:25.773 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-22T01:30:25.773 INFO:teuthology.orchestra.run.smithi098.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-22T01:30:25.774 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10+0 records in 2024-03-22T01:30:25.774 INFO:teuthology.orchestra.run.smithi098.stderr:10+0 records out 2024-03-22T01:30:25.774 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112445 s, 933 MB/s 2024-03-22T01:30:25.774 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping successful for: 2024-03-22T01:30:26.082 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:25 smithi098 ceph-mon[36468]: purged_snaps scrub starts 2024-03-22T01:30:26.082 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:25 smithi098 ceph-mon[36468]: purged_snaps scrub ok 2024-03-22T01:30:26.083 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:25 smithi098 ceph-mon[36468]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2024-03-22T01:30:26.083 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:25 smithi098 ceph-mon[36468]: osdmap e33: 5 total, 4 up, 5 in 2024-03-22T01:30:26.083 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:25 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T01:30:26.083 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:25 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T01:30:26.083 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:25 smithi098 ceph-mon[36468]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-22T01:30:26.083 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:25 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T01:30:26.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:25 smithi089 ceph-mon[35770]: purged_snaps scrub starts 2024-03-22T01:30:26.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:25 smithi089 ceph-mon[35770]: purged_snaps scrub ok 2024-03-22T01:30:26.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:25 smithi089 ceph-mon[35770]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2024-03-22T01:30:26.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:25 smithi089 ceph-mon[35770]: osdmap e33: 5 total, 4 up, 5 in 2024-03-22T01:30:26.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:25 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T01:30:26.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:25 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T01:30:26.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:25 smithi089 ceph-mon[35770]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-22T01:30:26.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:25 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T01:30:26.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:25 smithi072 ceph-mon[32414]: purged_snaps scrub starts 2024-03-22T01:30:26.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:25 smithi072 ceph-mon[32414]: purged_snaps scrub ok 2024-03-22T01:30:26.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:25 smithi072 ceph-mon[32414]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2024-03-22T01:30:26.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:25 smithi072 ceph-mon[32414]: osdmap e33: 5 total, 4 up, 5 in 2024-03-22T01:30:26.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:25 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T01:30:26.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:25 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T01:30:26.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:25 smithi072 ceph-mon[32414]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-22T01:30:26.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:25 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T01:30:26.389 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi098:vg_nvme/lv_3 2024-03-22T01:30:26.641 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.c/config 2024-03-22T01:30:26.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:26 smithi098 ceph-mon[36468]: osd.4 [v2:172.21.15.89:6808/2683760795,v1:172.21.15.89:6809/2683760795] boot 2024-03-22T01:30:26.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:26 smithi098 ceph-mon[36468]: osdmap e34: 5 total, 5 up, 5 in 2024-03-22T01:30:26.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:26 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T01:30:26.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:26 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:26.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:26 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:26.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:26 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:30:26.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:26 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:30:26.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:26 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:26.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:26 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:30:26.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:26 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:30:26.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:26 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:27.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:26 smithi089 ceph-mon[35770]: osd.4 [v2:172.21.15.89:6808/2683760795,v1:172.21.15.89:6809/2683760795] boot 2024-03-22T01:30:27.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:26 smithi089 ceph-mon[35770]: osdmap e34: 5 total, 5 up, 5 in 2024-03-22T01:30:27.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:26 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T01:30:27.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:26 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:27.109 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:26 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:27.109 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:26 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:30:27.109 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:26 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:30:27.109 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:26 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:27.109 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:26 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:30:27.109 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:26 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:30:27.109 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:26 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:27.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:26 smithi072 ceph-mon[32414]: osd.4 [v2:172.21.15.89:6808/2683760795,v1:172.21.15.89:6809/2683760795] boot 2024-03-22T01:30:27.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:26 smithi072 ceph-mon[32414]: osdmap e34: 5 total, 5 up, 5 in 2024-03-22T01:30:27.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:26 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T01:30:27.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:26 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:27.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:26 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:27.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:26 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:30:27.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:26 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:30:27.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:26 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:27.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:26 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:30:27.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:26 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:30:27.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:26 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:28.105 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:27 smithi098 ceph-mon[36468]: Detected new or changed devices on smithi089 2024-03-22T01:30:28.105 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:27 smithi098 ceph-mon[36468]: Adjusting osd_memory_target on smithi089 to 8540M 2024-03-22T01:30:28.105 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:27 smithi098 ceph-mon[36468]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-22T01:30:28.105 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:27 smithi098 ceph-mon[36468]: osdmap e35: 5 total, 5 up, 5 in 2024-03-22T01:30:28.107 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:27 smithi089 ceph-mon[35770]: Detected new or changed devices on smithi089 2024-03-22T01:30:28.107 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:27 smithi089 ceph-mon[35770]: Adjusting osd_memory_target on smithi089 to 8540M 2024-03-22T01:30:28.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:27 smithi089 ceph-mon[35770]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-22T01:30:28.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:27 smithi089 ceph-mon[35770]: osdmap e35: 5 total, 5 up, 5 in 2024-03-22T01:30:28.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:27 smithi072 ceph-mon[32414]: Detected new or changed devices on smithi089 2024-03-22T01:30:28.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:27 smithi072 ceph-mon[32414]: Adjusting osd_memory_target on smithi089 to 8540M 2024-03-22T01:30:28.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:27 smithi072 ceph-mon[32414]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-22T01:30:28.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:27 smithi072 ceph-mon[32414]: osdmap e35: 5 total, 5 up, 5 in 2024-03-22T01:30:28.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:30:28.342+0000 7fc5b2a71700 1 -- 172.21.15.98:0/25434421 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5ac103cb0 msgr2=0x7fc5ac1040d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:30:28.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:30:28.342+0000 7fc5b2a71700 1 --2- 172.21.15.98:0/25434421 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5ac103cb0 0x7fc5ac1040d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc5a0009990 tx=0x7fc5a0009ca0 comp rx=0 tx=0).stop 2024-03-22T01:30:28.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:30:28.342+0000 7fc5b2a71700 1 -- 172.21.15.98:0/25434421 shutdown_connections 2024-03-22T01:30:28.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:30:28.342+0000 7fc5b2a71700 1 --2- 172.21.15.98:0/25434421 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc5ac1010d0 0x7fc5ac10aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:30:28.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:30:28.342+0000 7fc5b2a71700 1 --2- 172.21.15.98:0/25434421 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5ac103cb0 0x7fc5ac1040d0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:30:28.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:30:28.342+0000 7fc5b2a71700 1 --2- 172.21.15.98:0/25434421 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5ac100730 0x7fc5ac100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:30:28.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:30:28.342+0000 7fc5b2a71700 1 -- 172.21.15.98:0/25434421 >> 172.21.15.98:0/25434421 conn(0x7fc5ac0fc2d0 msgr2=0x7fc5ac0fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:30:28.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:30:28.342+0000 7fc5b2a71700 1 -- 172.21.15.98:0/25434421 shutdown_connections 2024-03-22T01:30:28.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:30:28.342+0000 7fc5b2a71700 1 -- 172.21.15.98:0/25434421 wait complete. 2024-03-22T01:30:28.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:30:28.342+0000 7fc5b2a71700 1 Processor -- start 2024-03-22T01:30:28.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:30:28.342+0000 7fc5b2a71700 1 -- start start 2024-03-22T01:30:28.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:30:28.343+0000 7fc5b2a71700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc5ac100730 0x7fc5ac1033b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:30:28.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:30:28.343+0000 7fc5b2a71700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5ac1010d0 0x7fc5ac101a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:30:28.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:30:28.343+0000 7fc5b2a71700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5ac103cb0 0x7fc5ac101f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:30:28.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:30:28.343+0000 7fc5b2a71700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fc5ac10e250 con 0x7fc5ac100730 2024-03-22T01:30:28.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:30:28.343+0000 7fc5b2a71700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc5ac10e0d0 con 0x7fc5ac1010d0 2024-03-22T01:30:28.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:30:28.343+0000 7fc5b2a71700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc5ac10e3d0 con 0x7fc5ac103cb0 2024-03-22T01:30:28.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:30:28.343+0000 7fc5b100e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5ac103cb0 0x7fc5ac101f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:30:28.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:30:28.343+0000 7fc5b100e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5ac103cb0 0x7fc5ac101f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35066/0 (socket says 172.21.15.98:35066) 2024-03-22T01:30:28.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:30:28.343+0000 7fc5b100e700 1 -- 172.21.15.98:0/2760319605 learned_addr learned my addr 172.21.15.98:0/2760319605 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-22T01:30:28.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:30:28.343+0000 7fc5b080d700 1 --2- 172.21.15.98:0/2760319605 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc5ac100730 0x7fc5ac1033b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:30:28.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:30:28.343+0000 7fc5abfff700 1 --2- 172.21.15.98:0/2760319605 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5ac1010d0 0x7fc5ac101a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:30:28.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:30:28.343+0000 7fc5b100e700 1 -- 172.21.15.98:0/2760319605 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5ac1010d0 msgr2=0x7fc5ac101a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:30:28.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:30:28.343+0000 7fc5b100e700 1 --2- 172.21.15.98:0/2760319605 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5ac1010d0 0x7fc5ac101a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:30:28.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:30:28.343+0000 7fc5b100e700 1 -- 172.21.15.98:0/2760319605 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc5ac100730 msgr2=0x7fc5ac1033b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:30:28.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:30:28.343+0000 7fc5b100e700 1 --2- 172.21.15.98:0/2760319605 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc5ac100730 0x7fc5ac1033b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:30:28.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:30:28.343+0000 7fc5b100e700 1 -- 172.21.15.98:0/2760319605 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5ac1027f0 con 0x7fc5ac103cb0 2024-03-22T01:30:28.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:30:28.343+0000 7fc5b100e700 1 --2- 172.21.15.98:0/2760319605 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5ac103cb0 0x7fc5ac101f40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc5a4007ee0 tx=0x7fc5a400d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:30:28.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:30:28.343+0000 7fc5a9ffb700 1 -- 172.21.15.98:0/2760319605 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5a400dbd0 con 0x7fc5ac103cb0 2024-03-22T01:30:28.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:30:28.343+0000 7fc5a9ffb700 1 -- 172.21.15.98:0/2760319605 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5a400dd70 con 0x7fc5ac103cb0 2024-03-22T01:30:28.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:30:28.343+0000 7fc5b2a71700 1 -- 172.21.15.98:0/2760319605 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5ac1a2a20 con 0x7fc5ac103cb0 2024-03-22T01:30:28.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:30:28.343+0000 7fc5a9ffb700 1 -- 172.21.15.98:0/2760319605 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5a4005a60 con 0x7fc5ac103cb0 2024-03-22T01:30:28.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:30:28.344+0000 7fc5b2a71700 1 -- 172.21.15.98:0/2760319605 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5ac1a2ee0 con 0x7fc5ac103cb0 2024-03-22T01:30:28.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:30:28.345+0000 7fc5b2a71700 1 -- 172.21.15.98:0/2760319605 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5ac069340 con 0x7fc5ac103cb0 2024-03-22T01:30:28.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:30:28.345+0000 7fc5a9ffb700 1 -- 172.21.15.98:0/2760319605 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7fc5a401a020 con 0x7fc5ac103cb0 2024-03-22T01:30:28.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:30:28.346+0000 7fc5a9ffb700 1 --2- 172.21.15.98:0/2760319605 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fc59c0a2fb0 0x7fc59c0a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:30:28.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:30:28.346+0000 7fc5a9ffb700 1 -- 172.21.15.98:0/2760319605 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 4101+0+0 (secure 0 0 0) 0x7fc5a40cdad0 con 0x7fc5ac103cb0 2024-03-22T01:30:28.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:30:28.346+0000 7fc5b080d700 1 --2- 172.21.15.98:0/2760319605 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fc59c0a2fb0 0x7fc59c0a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:30:28.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:30:28.347+0000 7fc5b080d700 1 --2- 172.21.15.98:0/2760319605 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fc59c0a2fb0 0x7fc59c0a5470 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc5a000b2e0 tx=0x7fc5a000b880 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:30:28.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:30:28.350+0000 7fc5a9ffb700 1 -- 172.21.15.98:0/2760319605 <== mon.1 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc5a4010070 con 0x7fc5ac103cb0 2024-03-22T01:30:28.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:30:28.556+0000 7fc5b2a71700 1 -- 172.21.15.98:0/2760319605 --> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fc5ac062b80 con 0x7fc59c0a2fb0 2024-03-22T01:30:29.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:28 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T01:30:29.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:28 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T01:30:29.109 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:28 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:30:29.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:28 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T01:30:29.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:28 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T01:30:29.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:28 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:30:29.148 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:28 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T01:30:29.148 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:28 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T01:30:29.148 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:28 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:30:30.079 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:29 smithi098 ceph-mon[36468]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:30:30.079 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:29 smithi098 ceph-mon[36468]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-22T01:30:30.107 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:29 smithi089 ceph-mon[35770]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:30:30.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:29 smithi089 ceph-mon[35770]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-22T01:30:30.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:29 smithi072 ceph-mon[32414]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:30:30.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:29 smithi072 ceph-mon[32414]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-22T01:30:31.611 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:31 smithi089 ceph-mon[35770]: from='client.? 172.21.15.98:0/358050781' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e78954d8-f4e3-4d58-8536-d2d59c843386"}]: dispatch 2024-03-22T01:30:31.611 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:31 smithi089 ceph-mon[35770]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e78954d8-f4e3-4d58-8536-d2d59c843386"}]: dispatch 2024-03-22T01:30:31.611 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:31 smithi089 ceph-mon[35770]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e78954d8-f4e3-4d58-8536-d2d59c843386"}]': finished 2024-03-22T01:30:31.611 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:31 smithi089 ceph-mon[35770]: osdmap e36: 6 total, 5 up, 6 in 2024-03-22T01:30:31.611 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:31 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T01:30:31.620 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:31 smithi098 ceph-mon[36468]: from='client.? 172.21.15.98:0/358050781' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e78954d8-f4e3-4d58-8536-d2d59c843386"}]: dispatch 2024-03-22T01:30:31.620 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:31 smithi098 ceph-mon[36468]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e78954d8-f4e3-4d58-8536-d2d59c843386"}]: dispatch 2024-03-22T01:30:31.620 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:31 smithi098 ceph-mon[36468]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e78954d8-f4e3-4d58-8536-d2d59c843386"}]': finished 2024-03-22T01:30:31.620 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:31 smithi098 ceph-mon[36468]: osdmap e36: 6 total, 5 up, 6 in 2024-03-22T01:30:31.620 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:31 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T01:30:31.621 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:31 smithi072 ceph-mon[32414]: from='client.? 172.21.15.98:0/358050781' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e78954d8-f4e3-4d58-8536-d2d59c843386"}]: dispatch 2024-03-22T01:30:31.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:31 smithi072 ceph-mon[32414]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e78954d8-f4e3-4d58-8536-d2d59c843386"}]: dispatch 2024-03-22T01:30:31.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:31 smithi072 ceph-mon[32414]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e78954d8-f4e3-4d58-8536-d2d59c843386"}]': finished 2024-03-22T01:30:31.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:31 smithi072 ceph-mon[32414]: osdmap e36: 6 total, 5 up, 6 in 2024-03-22T01:30:31.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:31 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T01:30:32.607 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:32 smithi089 ceph-mon[35770]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-22T01:30:32.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:32 smithi089 ceph-mon[35770]: from='client.? 172.21.15.98:0/159646467' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T01:30:32.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:32 smithi072 ceph-mon[32414]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-22T01:30:32.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:32 smithi072 ceph-mon[32414]: from='client.? 172.21.15.98:0/159646467' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T01:30:32.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:32 smithi098 ceph-mon[36468]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-22T01:30:32.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:32 smithi098 ceph-mon[36468]: from='client.? 172.21.15.98:0/159646467' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T01:30:34.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:34 smithi089 ceph-mon[35770]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-22T01:30:34.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:34 smithi072 ceph-mon[32414]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-22T01:30:34.889 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:34 smithi098 ceph-mon[36468]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-22T01:30:36.509 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:36 smithi098 ceph-mon[36468]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-22T01:30:36.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:36 smithi089 ceph-mon[35770]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-22T01:30:36.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:36 smithi072 ceph-mon[32414]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-22T01:30:38.553 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:38 smithi098 ceph-mon[36468]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-22T01:30:38.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:38 smithi089 ceph-mon[35770]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-22T01:30:38.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:38 smithi072 ceph-mon[32414]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-22T01:30:39.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:39 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-22T01:30:39.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:39 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:30:39.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:39 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-22T01:30:39.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:39 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:30:39.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:39 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-22T01:30:39.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:39 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:30:40.712 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:40 smithi098 ceph-mon[36468]: Deploying daemon osd.5 on smithi098 2024-03-22T01:30:40.712 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:40 smithi098 ceph-mon[36468]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-22T01:30:40.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:40 smithi089 ceph-mon[35770]: Deploying daemon osd.5 on smithi098 2024-03-22T01:30:40.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:40 smithi089 ceph-mon[35770]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-22T01:30:40.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:40 smithi072 ceph-mon[32414]: Deploying daemon osd.5 on smithi098 2024-03-22T01:30:40.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:40 smithi072 ceph-mon[32414]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-22T01:30:42.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:42 smithi098 ceph-mon[36468]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-22T01:30:42.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:42 smithi089 ceph-mon[35770]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-22T01:30:42.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:42 smithi072 ceph-mon[32414]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-22T01:30:44.697 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:44 smithi072 ceph-mon[32414]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-22T01:30:44.697 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:44 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:44.697 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:44 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:44.698 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:44 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:30:44.725 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:44 smithi098 ceph-mon[36468]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-22T01:30:44.726 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:44 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:44.726 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:44 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:44.726 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:44 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:30:44.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:44 smithi089 ceph-mon[35770]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-22T01:30:44.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:44 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:44.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:44 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:44.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:44 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:30:46.625 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:46 smithi098 ceph-mon[36468]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-22T01:30:46.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:46 smithi089 ceph-mon[35770]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-22T01:30:46.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:46 smithi072 ceph-mon[32414]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-22T01:30:47.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:30:47.574+0000 7fc5a9ffb700 1 -- 172.21.15.98:0/2760319605 <== mgr.14152 v2:172.21.15.72:6800/401403200 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fc5ac062b80 con 0x7fc59c0a2fb0 2024-03-22T01:30:47.574 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 5 on host 'smithi098' 2024-03-22T01:30:47.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:30:47.576+0000 7fc5b2a71700 1 -- 172.21.15.98:0/2760319605 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fc59c0a2fb0 msgr2=0x7fc59c0a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:30:47.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:30:47.576+0000 7fc5b2a71700 1 --2- 172.21.15.98:0/2760319605 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fc59c0a2fb0 0x7fc59c0a5470 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc5a000b2e0 tx=0x7fc5a000b880 comp rx=0 tx=0).stop 2024-03-22T01:30:47.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:30:47.576+0000 7fc5b2a71700 1 -- 172.21.15.98:0/2760319605 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5ac103cb0 msgr2=0x7fc5ac101f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:30:47.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:30:47.576+0000 7fc5b2a71700 1 --2- 172.21.15.98:0/2760319605 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5ac103cb0 0x7fc5ac101f40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc5a4007ee0 tx=0x7fc5a400d3b0 comp rx=0 tx=0).stop 2024-03-22T01:30:47.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:30:47.576+0000 7fc5b2a71700 1 -- 172.21.15.98:0/2760319605 shutdown_connections 2024-03-22T01:30:47.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:30:47.576+0000 7fc5b2a71700 1 --2- 172.21.15.98:0/2760319605 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fc59c0a2fb0 0x7fc59c0a5470 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:30:47.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:30:47.576+0000 7fc5b2a71700 1 --2- 172.21.15.98:0/2760319605 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5ac103cb0 0x7fc5ac101f40 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:30:47.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:30:47.576+0000 7fc5b2a71700 1 --2- 172.21.15.98:0/2760319605 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc5ac100730 0x7fc5ac1033b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:30:47.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:30:47.576+0000 7fc5b2a71700 1 --2- 172.21.15.98:0/2760319605 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5ac1010d0 0x7fc5ac101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:30:47.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:30:47.576+0000 7fc5b2a71700 1 -- 172.21.15.98:0/2760319605 >> 172.21.15.98:0/2760319605 conn(0x7fc5ac0fc2d0 msgr2=0x7fc5ac106d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:30:47.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:30:47.577+0000 7fc5b2a71700 1 -- 172.21.15.98:0/2760319605 shutdown_connections 2024-03-22T01:30:47.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:30:47.577+0000 7fc5b2a71700 1 -- 172.21.15.98:0/2760319605 wait complete. 2024-03-22T01:30:48.194 DEBUG:teuthology.orchestra.run.smithi098:osd.5> sudo journalctl -f -n 0 -u ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@osd.5.service 2024-03-22T01:30:48.197 INFO:tasks.cephadm:Deploying osd.6 on smithi072 with /dev/vg_nvme/lv_2... 2024-03-22T01:30:48.197 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-22T01:30:48.443 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:30:48.720 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:48 smithi098 ceph-mon[36468]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-22T01:30:48.720 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:48 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:48.720 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:48 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:48.720 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:48 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:48.721 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:48 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:48.721 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:48 smithi098 ceph-mon[36468]: from='osd.5 [v2:172.21.15.98:6808/2635838572,v1:172.21.15.98:6809/2635838572]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-22T01:30:48.721 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:48 smithi098 ceph-mon[36468]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-22T01:30:48.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:48 smithi072 ceph-mon[32414]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-22T01:30:48.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:48 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:48.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:48 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:48.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:48 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:48.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:48 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:48.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:48 smithi072 ceph-mon[32414]: from='osd.5 [v2:172.21.15.98:6808/2635838572,v1:172.21.15.98:6809/2635838572]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-22T01:30:48.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:48 smithi072 ceph-mon[32414]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-22T01:30:48.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:48 smithi089 ceph-mon[35770]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-22T01:30:48.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:48 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:48.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:48 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:48.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:48 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:48.859 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:48 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:48.859 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:48 smithi089 ceph-mon[35770]: from='osd.5 [v2:172.21.15.98:6808/2635838572,v1:172.21.15.98:6809/2635838572]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-22T01:30:48.859 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:48 smithi089 ceph-mon[35770]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-22T01:30:49.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:49 smithi098 ceph-mon[36468]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-22T01:30:49.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:49 smithi098 ceph-mon[36468]: osdmap e37: 6 total, 5 up, 6 in 2024-03-22T01:30:49.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:49 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T01:30:49.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:49 smithi098 ceph-mon[36468]: from='osd.5 [v2:172.21.15.98:6808/2635838572,v1:172.21.15.98:6809/2635838572]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-22T01:30:49.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:49 smithi098 ceph-mon[36468]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-22T01:30:49.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:49 smithi098 ceph-mon[36468]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-22T01:30:50.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:49 smithi072 ceph-mon[32414]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-22T01:30:50.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:49 smithi072 ceph-mon[32414]: osdmap e37: 6 total, 5 up, 6 in 2024-03-22T01:30:50.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:49 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T01:30:50.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:49 smithi072 ceph-mon[32414]: from='osd.5 [v2:172.21.15.98:6808/2635838572,v1:172.21.15.98:6809/2635838572]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-22T01:30:50.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:49 smithi072 ceph-mon[32414]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-22T01:30:50.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:49 smithi072 ceph-mon[32414]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-22T01:30:50.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:49 smithi089 ceph-mon[35770]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-22T01:30:50.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:49 smithi089 ceph-mon[35770]: osdmap e37: 6 total, 5 up, 6 in 2024-03-22T01:30:50.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:49 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T01:30:50.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:49 smithi089 ceph-mon[35770]: from='osd.5 [v2:172.21.15.98:6808/2635838572,v1:172.21.15.98:6809/2635838572]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-22T01:30:50.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:49 smithi089 ceph-mon[35770]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-22T01:30:50.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:49 smithi089 ceph-mon[35770]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-22T01:30:50.516 INFO:teuthology.orchestra.run.smithi072.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-22T01:30:50.516 INFO:teuthology.orchestra.run.smithi072.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-22T01:30:50.517 INFO:teuthology.orchestra.run.smithi072.stderr: stderr: 10+0 records in 2024-03-22T01:30:50.517 INFO:teuthology.orchestra.run.smithi072.stderr:10+0 records out 2024-03-22T01:30:50.517 INFO:teuthology.orchestra.run.smithi072.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112784 s, 930 MB/s 2024-03-22T01:30:50.517 INFO:teuthology.orchestra.run.smithi072.stderr:--> Zapping successful for: 2024-03-22T01:30:50.873 INFO:journalctl@ceph.osd.5.smithi098.stdout:Mar 22 01:30:50 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-5[48584]: 2024-03-22T01:30:50.567+0000 7f9330891700 -1 osd.5 0 waiting for initial osdmap 2024-03-22T01:30:50.873 INFO:journalctl@ceph.osd.5.smithi098.stdout:Mar 22 01:30:50 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-5[48584]: 2024-03-22T01:30:50.573+0000 7f932867c700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-22T01:30:51.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:50 smithi072 ceph-mon[32414]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-22T01:30:51.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:50 smithi072 ceph-mon[32414]: osdmap e38: 6 total, 5 up, 6 in 2024-03-22T01:30:51.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:50 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T01:30:51.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:50 smithi072 ceph-mon[32414]: from='osd.5 ' entity='osd.5' 2024-03-22T01:30:51.172 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi072:vg_nvme/lv_2 2024-03-22T01:30:51.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:50 smithi098 ceph-mon[36468]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-22T01:30:51.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:50 smithi098 ceph-mon[36468]: osdmap e38: 6 total, 5 up, 6 in 2024-03-22T01:30:51.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:50 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T01:30:51.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:50 smithi098 ceph-mon[36468]: from='osd.5 ' entity='osd.5' 2024-03-22T01:30:51.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:50 smithi089 ceph-mon[35770]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-22T01:30:51.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:50 smithi089 ceph-mon[35770]: osdmap e38: 6 total, 5 up, 6 in 2024-03-22T01:30:51.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:50 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T01:30:51.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:50 smithi089 ceph-mon[35770]: from='osd.5 ' entity='osd.5' 2024-03-22T01:30:51.410 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:30:52.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:51 smithi072 ceph-mon[32414]: purged_snaps scrub starts 2024-03-22T01:30:52.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:51 smithi072 ceph-mon[32414]: purged_snaps scrub ok 2024-03-22T01:30:52.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:51 smithi072 ceph-mon[32414]: osd.5 [v2:172.21.15.98:6808/2635838572,v1:172.21.15.98:6809/2635838572] boot 2024-03-22T01:30:52.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:51 smithi072 ceph-mon[32414]: osdmap e39: 6 total, 6 up, 6 in 2024-03-22T01:30:52.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:51 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T01:30:52.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:51 smithi072 ceph-mon[32414]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 536 GiB / 536 GiB avail 2024-03-22T01:30:52.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:51 smithi072 ceph-mon[32414]: osdmap e40: 6 total, 6 up, 6 in 2024-03-22T01:30:52.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:51 smithi098 ceph-mon[36468]: purged_snaps scrub starts 2024-03-22T01:30:52.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:51 smithi098 ceph-mon[36468]: purged_snaps scrub ok 2024-03-22T01:30:52.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:51 smithi098 ceph-mon[36468]: osd.5 [v2:172.21.15.98:6808/2635838572,v1:172.21.15.98:6809/2635838572] boot 2024-03-22T01:30:52.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:51 smithi098 ceph-mon[36468]: osdmap e39: 6 total, 6 up, 6 in 2024-03-22T01:30:52.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:51 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T01:30:52.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:51 smithi098 ceph-mon[36468]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 536 GiB / 536 GiB avail 2024-03-22T01:30:52.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:51 smithi098 ceph-mon[36468]: osdmap e40: 6 total, 6 up, 6 in 2024-03-22T01:30:52.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:51 smithi089 ceph-mon[35770]: purged_snaps scrub starts 2024-03-22T01:30:52.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:51 smithi089 ceph-mon[35770]: purged_snaps scrub ok 2024-03-22T01:30:52.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:51 smithi089 ceph-mon[35770]: osd.5 [v2:172.21.15.98:6808/2635838572,v1:172.21.15.98:6809/2635838572] boot 2024-03-22T01:30:52.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:51 smithi089 ceph-mon[35770]: osdmap e39: 6 total, 6 up, 6 in 2024-03-22T01:30:52.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:51 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T01:30:52.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:51 smithi089 ceph-mon[35770]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 536 GiB / 536 GiB avail 2024-03-22T01:30:52.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:51 smithi089 ceph-mon[35770]: osdmap e40: 6 total, 6 up, 6 in 2024-03-22T01:30:53.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:30:53.049+0000 7ff604cfb700 1 -- 172.21.15.72:0/3592712548 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff600103150 msgr2=0x7ff600109a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:30:53.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:30:53.049+0000 7ff604cfb700 1 --2- 172.21.15.72:0/3592712548 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff600103150 0x7ff600109a00 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff5f800b0d0 tx=0x7ff5f800b3e0 comp rx=0 tx=0).stop 2024-03-22T01:30:53.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:30:53.049+0000 7ff604cfb700 1 -- 172.21.15.72:0/3592712548 shutdown_connections 2024-03-22T01:30:53.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:30:53.049+0000 7ff604cfb700 1 --2- 172.21.15.72:0/3592712548 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff600103150 0x7ff600109a00 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:30:53.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:30:53.049+0000 7ff604cfb700 1 --2- 172.21.15.72:0/3592712548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6001015b0 0x7ff6001019d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:30:53.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:30:53.049+0000 7ff604cfb700 1 --2- 172.21.15.72:0/3592712548 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6001027b0 0x7ff600102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:30:53.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:30:53.049+0000 7ff604cfb700 1 -- 172.21.15.72:0/3592712548 >> 172.21.15.72:0/3592712548 conn(0x7ff6000fcb50 msgr2=0x7ff6000fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:30:53.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:30:53.049+0000 7ff604cfb700 1 -- 172.21.15.72:0/3592712548 shutdown_connections 2024-03-22T01:30:53.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:30:53.049+0000 7ff604cfb700 1 -- 172.21.15.72:0/3592712548 wait complete. 2024-03-22T01:30:53.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:30:53.050+0000 7ff604cfb700 1 Processor -- start 2024-03-22T01:30:53.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:30:53.050+0000 7ff604cfb700 1 -- start start 2024-03-22T01:30:53.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:30:53.050+0000 7ff604cfb700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6001015b0 0x7ff6001988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:30:53.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:30:53.050+0000 7ff604cfb700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6001027b0 0x7ff600198e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:30:53.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:30:53.050+0000 7ff604cfb700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff600103150 0x7ff60019fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:30:53.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:30:53.050+0000 7ff604cfb700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7ff60010e310 con 0x7ff600103150 2024-03-22T01:30:53.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:30:53.050+0000 7ff604cfb700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff60010e190 con 0x7ff6001015b0 2024-03-22T01:30:53.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:30:53.050+0000 7ff604cfb700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff60010e490 con 0x7ff6001027b0 2024-03-22T01:30:53.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:30:53.050+0000 7ff5fed9d700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff600103150 0x7ff60019fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:30:53.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:30:53.050+0000 7ff5fed9d700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff600103150 0x7ff60019fe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:49508/0 (socket says 172.21.15.72:49508) 2024-03-22T01:30:53.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:30:53.050+0000 7ff5fed9d700 1 -- 172.21.15.72:0/3977240330 learned_addr learned my addr 172.21.15.72:0/3977240330 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:30:53.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:30:53.051+0000 7ff5fe59c700 1 --2- 172.21.15.72:0/3977240330 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6001015b0 0x7ff6001988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:30:53.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:30:53.051+0000 7ff5fdd9b700 1 --2- 172.21.15.72:0/3977240330 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6001027b0 0x7ff600198e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:30:53.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:30:53.051+0000 7ff5fed9d700 1 -- 172.21.15.72:0/3977240330 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6001027b0 msgr2=0x7ff600198e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:30:53.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:30:53.051+0000 7ff5fed9d700 1 --2- 172.21.15.72:0/3977240330 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6001027b0 0x7ff600198e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:30:53.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:30:53.051+0000 7ff5fed9d700 1 -- 172.21.15.72:0/3977240330 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6001015b0 msgr2=0x7ff6001988c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:30:53.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:30:53.051+0000 7ff5fed9d700 1 --2- 172.21.15.72:0/3977240330 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6001015b0 0x7ff6001988c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:30:53.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:30:53.051+0000 7ff5fed9d700 1 -- 172.21.15.72:0/3977240330 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6001a0520 con 0x7ff600103150 2024-03-22T01:30:53.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:30:53.051+0000 7ff5fe59c700 1 --2- 172.21.15.72:0/3977240330 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6001015b0 0x7ff6001988c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:30:53.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:30:53.051+0000 7ff5fed9d700 1 --2- 172.21.15.72:0/3977240330 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff600103150 0x7ff60019fe80 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ff5f8007d90 tx=0x7ff5f8007e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:30:53.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:30:53.051+0000 7ff5ef7fe700 1 -- 172.21.15.72:0/3977240330 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5f801f070 con 0x7ff600103150 2024-03-22T01:30:53.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:30:53.051+0000 7ff604cfb700 1 -- 172.21.15.72:0/3977240330 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6001a07b0 con 0x7ff600103150 2024-03-22T01:30:53.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:30:53.051+0000 7ff5ef7fe700 1 -- 172.21.15.72:0/3977240330 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5f8005530 con 0x7ff600103150 2024-03-22T01:30:53.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:30:53.051+0000 7ff5ef7fe700 1 -- 172.21.15.72:0/3977240330 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5f801b7d0 con 0x7ff600103150 2024-03-22T01:30:53.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:30:53.051+0000 7ff604cfb700 1 -- 172.21.15.72:0/3977240330 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6001a0cf0 con 0x7ff600103150 2024-03-22T01:30:53.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:30:53.052+0000 7ff604cfb700 1 -- 172.21.15.72:0/3977240330 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff600069340 con 0x7ff600103150 2024-03-22T01:30:53.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:30:53.054+0000 7ff5ef7fe700 1 -- 172.21.15.72:0/3977240330 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7ff5f8022070 con 0x7ff600103150 2024-03-22T01:30:53.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:30:53.055+0000 7ff5ef7fe700 1 --2- 172.21.15.72:0/3977240330 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7ff5e80a2fb0 0x7ff5e80a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:30:53.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:30:53.055+0000 7ff5fe59c700 1 --2- 172.21.15.72:0/3977240330 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7ff5e80a2fb0 0x7ff5e80a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:30:53.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:30:53.055+0000 7ff5ef7fe700 1 -- 172.21.15.72:0/3977240330 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4562+0+0 (secure 0 0 0) 0x7ff5f80cf360 con 0x7ff600103150 2024-03-22T01:30:53.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:30:53.055+0000 7ff5fe59c700 1 --2- 172.21.15.72:0/3977240330 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7ff5e80a2fb0 0x7ff5e80a5470 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff5f0005950 tx=0x7ff5f00058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:30:53.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:30:53.058+0000 7ff5ef7fe700 1 -- 172.21.15.72:0/3977240330 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff5f809d020 con 0x7ff600103150 2024-03-22T01:30:53.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:52 smithi072 ceph-mon[32414]: Detected new or changed devices on smithi098 2024-03-22T01:30:53.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:52 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:53.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:52 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:53.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:52 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:30:53.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:52 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:30:53.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:52 smithi072 ceph-mon[32414]: Adjusting osd_memory_target on smithi098 to 8540M 2024-03-22T01:30:53.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:52 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:53.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:52 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:30:53.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:52 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:30:53.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:52 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:53.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:52 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:30:53.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:52 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:30:53.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:52 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:30:53.124 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:52 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:53.124 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:52 smithi072 ceph-mon[32414]: osdmap e41: 6 total, 6 up, 6 in 2024-03-22T01:30:53.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:30:53.263+0000 7ff604cfb700 1 -- 172.21.15.72:0/3977240330 --> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi072:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7ff600062ac0 con 0x7ff5e80a2fb0 2024-03-22T01:30:53.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:52 smithi089 ceph-mon[35770]: Detected new or changed devices on smithi098 2024-03-22T01:30:53.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:52 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:53.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:52 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:53.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:52 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:30:53.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:52 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:30:53.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:52 smithi089 ceph-mon[35770]: Adjusting osd_memory_target on smithi098 to 8540M 2024-03-22T01:30:53.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:52 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:53.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:52 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:30:53.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:52 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:30:53.359 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:52 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:53.359 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:52 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:30:53.359 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:52 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:30:53.359 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:52 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:30:53.359 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:52 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:53.359 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:52 smithi089 ceph-mon[35770]: osdmap e41: 6 total, 6 up, 6 in 2024-03-22T01:30:53.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:52 smithi098 ceph-mon[36468]: Detected new or changed devices on smithi098 2024-03-22T01:30:53.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:52 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:53.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:52 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:53.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:52 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:30:53.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:52 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:30:53.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:52 smithi098 ceph-mon[36468]: Adjusting osd_memory_target on smithi098 to 8540M 2024-03-22T01:30:53.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:52 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:53.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:52 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:30:53.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:52 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:30:53.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:52 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:53.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:52 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:30:53.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:52 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:30:53.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:52 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:30:53.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:52 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:30:53.423 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:52 smithi098 ceph-mon[36468]: osdmap e41: 6 total, 6 up, 6 in 2024-03-22T01:30:54.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:53 smithi072 ceph-mon[32414]: from='client.14355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi072:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:30:54.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:53 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T01:30:54.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:53 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T01:30:54.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:53 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:30:54.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:53 smithi072 ceph-mon[32414]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 536 GiB / 536 GiB avail 2024-03-22T01:30:54.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:53 smithi072 ceph-mon[32414]: osdmap e42: 6 total, 6 up, 6 in 2024-03-22T01:30:54.259 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:53 smithi089 ceph-mon[35770]: from='client.14355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi072:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:30:54.259 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:53 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T01:30:54.259 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:53 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T01:30:54.259 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:53 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:30:54.259 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:53 smithi089 ceph-mon[35770]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 536 GiB / 536 GiB avail 2024-03-22T01:30:54.259 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:53 smithi089 ceph-mon[35770]: osdmap e42: 6 total, 6 up, 6 in 2024-03-22T01:30:54.323 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:53 smithi098 ceph-mon[36468]: from='client.14355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi072:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:30:54.323 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:53 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T01:30:54.323 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:53 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T01:30:54.323 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:53 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:30:54.323 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:53 smithi098 ceph-mon[36468]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 536 GiB / 536 GiB avail 2024-03-22T01:30:54.323 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:53 smithi098 ceph-mon[36468]: osdmap e42: 6 total, 6 up, 6 in 2024-03-22T01:30:55.593 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:55 smithi098 ceph-mon[36468]: osdmap e43: 6 total, 6 up, 6 in 2024-03-22T01:30:55.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:55 smithi089 ceph-mon[35770]: osdmap e43: 6 total, 6 up, 6 in 2024-03-22T01:30:55.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:55 smithi072 ceph-mon[32414]: osdmap e43: 6 total, 6 up, 6 in 2024-03-22T01:30:56.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:56 smithi072 ceph-mon[32414]: pgmap v143: 1 pgs: 1 peering; 577 KiB data, 294 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-22T01:30:56.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:56 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/2494717481' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc6c8be2-b64c-4959-8015-7b0ea6d25599"}]: dispatch 2024-03-22T01:30:56.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:56 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/2494717481' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dc6c8be2-b64c-4959-8015-7b0ea6d25599"}]': finished 2024-03-22T01:30:56.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:56 smithi072 ceph-mon[32414]: osdmap e44: 7 total, 6 up, 7 in 2024-03-22T01:30:56.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:56 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T01:30:56.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:56 smithi089 ceph-mon[35770]: pgmap v143: 1 pgs: 1 peering; 577 KiB data, 294 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-22T01:30:56.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:56 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/2494717481' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc6c8be2-b64c-4959-8015-7b0ea6d25599"}]: dispatch 2024-03-22T01:30:56.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:56 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/2494717481' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dc6c8be2-b64c-4959-8015-7b0ea6d25599"}]': finished 2024-03-22T01:30:56.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:56 smithi089 ceph-mon[35770]: osdmap e44: 7 total, 6 up, 7 in 2024-03-22T01:30:56.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:56 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T01:30:56.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:56 smithi098 ceph-mon[36468]: pgmap v143: 1 pgs: 1 peering; 577 KiB data, 294 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-22T01:30:56.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:56 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/2494717481' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc6c8be2-b64c-4959-8015-7b0ea6d25599"}]: dispatch 2024-03-22T01:30:56.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:56 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/2494717481' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dc6c8be2-b64c-4959-8015-7b0ea6d25599"}]': finished 2024-03-22T01:30:56.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:56 smithi098 ceph-mon[36468]: osdmap e44: 7 total, 6 up, 7 in 2024-03-22T01:30:56.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:56 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T01:30:57.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:57 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/917817370' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T01:30:57.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:57 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/917817370' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T01:30:57.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:57 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/917817370' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T01:30:58.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:58 smithi089 ceph-mon[35770]: pgmap v145: 1 pgs: 1 peering; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 128 KiB/s, 0 objects/s recovering 2024-03-22T01:30:58.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:58 smithi072 ceph-mon[32414]: pgmap v145: 1 pgs: 1 peering; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 128 KiB/s, 0 objects/s recovering 2024-03-22T01:30:58.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:58 smithi098 ceph-mon[36468]: pgmap v145: 1 pgs: 1 peering; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 128 KiB/s, 0 objects/s recovering 2024-03-22T01:30:59.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:30:59 smithi089 ceph-mon[35770]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-22T01:30:59.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:30:59 smithi072 ceph-mon[32414]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-22T01:30:59.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:30:59 smithi098 ceph-mon[36468]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-22T01:31:02.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:02 smithi072 ceph-mon[32414]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-03-22T01:31:02.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:02 smithi089 ceph-mon[35770]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-03-22T01:31:02.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:02 smithi098 ceph-mon[36468]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-03-22T01:31:03.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:03 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-22T01:31:03.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:03 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:31:03.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:03 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-22T01:31:03.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:03 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:31:03.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:03 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-22T01:31:03.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:03 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:31:04.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:04 smithi072 ceph-mon[32414]: Deploying daemon osd.6 on smithi072 2024-03-22T01:31:04.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:04 smithi072 ceph-mon[32414]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-22T01:31:04.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:04 smithi089 ceph-mon[35770]: Deploying daemon osd.6 on smithi072 2024-03-22T01:31:04.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:04 smithi089 ceph-mon[35770]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-22T01:31:04.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:04 smithi098 ceph-mon[36468]: Deploying daemon osd.6 on smithi072 2024-03-22T01:31:04.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:04 smithi098 ceph-mon[36468]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-22T01:31:06.709 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:06 smithi072 ceph-mon[32414]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-22T01:31:06.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:06 smithi089 ceph-mon[35770]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-22T01:31:06.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:06 smithi098 ceph-mon[36468]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-22T01:31:08.667 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:08 smithi072 ceph-mon[32414]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-22T01:31:08.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:08 smithi089 ceph-mon[35770]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-22T01:31:08.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:08 smithi098 ceph-mon[36468]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-22T01:31:10.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:09 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:10.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:09 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:10.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:09 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:31:10.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:09 smithi089 ceph-mon[35770]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-22T01:31:10.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:09 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:10.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:09 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:10.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:09 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:31:10.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:09 smithi072 ceph-mon[32414]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-22T01:31:10.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:09 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:10.174 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:09 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:10.174 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:09 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:31:10.174 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:09 smithi098 ceph-mon[36468]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-22T01:31:12.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:31:12.430+0000 7ff5ef7fe700 1 -- 172.21.15.72:0/3977240330 <== mgr.14152 v2:172.21.15.72:6800/401403200 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ff600062ac0 con 0x7ff5e80a2fb0 2024-03-22T01:31:12.429 INFO:teuthology.orchestra.run.smithi072.stdout:Created osd(s) 6 on host 'smithi072' 2024-03-22T01:31:12.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:31:12.433+0000 7ff604cfb700 1 -- 172.21.15.72:0/3977240330 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7ff5e80a2fb0 msgr2=0x7ff5e80a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:31:12.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:31:12.433+0000 7ff604cfb700 1 --2- 172.21.15.72:0/3977240330 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7ff5e80a2fb0 0x7ff5e80a5470 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff5f0005950 tx=0x7ff5f00058e0 comp rx=0 tx=0).stop 2024-03-22T01:31:12.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:31:12.433+0000 7ff604cfb700 1 -- 172.21.15.72:0/3977240330 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff600103150 msgr2=0x7ff60019fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:31:12.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:31:12.433+0000 7ff604cfb700 1 --2- 172.21.15.72:0/3977240330 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff600103150 0x7ff60019fe80 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ff5f8007d90 tx=0x7ff5f8007e70 comp rx=0 tx=0).stop 2024-03-22T01:31:12.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:31:12.433+0000 7ff604cfb700 1 -- 172.21.15.72:0/3977240330 shutdown_connections 2024-03-22T01:31:12.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:31:12.433+0000 7ff604cfb700 1 --2- 172.21.15.72:0/3977240330 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7ff5e80a2fb0 0x7ff5e80a5470 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:31:12.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:31:12.433+0000 7ff604cfb700 1 --2- 172.21.15.72:0/3977240330 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff600103150 0x7ff60019fe80 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:31:12.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:31:12.433+0000 7ff604cfb700 1 --2- 172.21.15.72:0/3977240330 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6001015b0 0x7ff6001988c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:31:12.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:31:12.433+0000 7ff604cfb700 1 --2- 172.21.15.72:0/3977240330 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6001027b0 0x7ff600198e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:31:12.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:31:12.433+0000 7ff604cfb700 1 -- 172.21.15.72:0/3977240330 >> 172.21.15.72:0/3977240330 conn(0x7ff6000fcb50 msgr2=0x7ff600105ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:31:12.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:31:12.433+0000 7ff604cfb700 1 -- 172.21.15.72:0/3977240330 shutdown_connections 2024-03-22T01:31:12.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:31:12.433+0000 7ff604cfb700 1 -- 172.21.15.72:0/3977240330 wait complete. 2024-03-22T01:31:12.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:12 smithi072 ceph-mon[32414]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-22T01:31:12.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:12 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:12.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:12 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:12.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:12 smithi089 ceph-mon[35770]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-22T01:31:12.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:12 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:12.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:12 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:12.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:12 smithi098 ceph-mon[36468]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-22T01:31:12.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:12 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:12.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:12 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:13.136 DEBUG:teuthology.orchestra.run.smithi072:osd.6> sudo journalctl -f -n 0 -u ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@osd.6.service 2024-03-22T01:31:13.139 INFO:tasks.cephadm:Deploying osd.7 on smithi089 with /dev/vg_nvme/lv_2... 2024-03-22T01:31:13.139 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-22T01:31:13.382 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.b/config 2024-03-22T01:31:13.727 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:13 smithi089 ceph-mon[35770]: from='osd.6 [v2:172.21.15.72:6818/3158982661,v1:172.21.15.72:6819/3158982661]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-22T01:31:13.728 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:13 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:13.728 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:13 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:13.806 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:13 smithi072 ceph-mon[32414]: from='osd.6 [v2:172.21.15.72:6818/3158982661,v1:172.21.15.72:6819/3158982661]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-22T01:31:13.807 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:13 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:13.807 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:13 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:13.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:13 smithi098 ceph-mon[36468]: from='osd.6 [v2:172.21.15.72:6818/3158982661,v1:172.21.15.72:6819/3158982661]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-22T01:31:13.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:13 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:13.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:13 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:14.752 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:14 smithi072 ceph-mon[32414]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-22T01:31:14.752 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:14 smithi072 ceph-mon[32414]: from='osd.6 [v2:172.21.15.72:6818/3158982661,v1:172.21.15.72:6819/3158982661]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-22T01:31:14.753 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:14 smithi072 ceph-mon[32414]: osdmap e45: 7 total, 6 up, 7 in 2024-03-22T01:31:14.753 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:14 smithi072 ceph-mon[32414]: from='osd.6 [v2:172.21.15.72:6818/3158982661,v1:172.21.15.72:6819/3158982661]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]: dispatch 2024-03-22T01:31:14.753 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:14 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T01:31:14.753 INFO:journalctl@ceph.osd.6.smithi072.stdout:Mar 22 01:31:14 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-6[58711]: 2024-03-22T01:31:14.452+0000 7fa4225b8700 -1 osd.6 0 waiting for initial osdmap 2024-03-22T01:31:14.753 INFO:journalctl@ceph.osd.6.smithi072.stdout:Mar 22 01:31:14 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-6[58711]: 2024-03-22T01:31:14.463+0000 7fa41e3ab700 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-22T01:31:14.768 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:14 smithi089 ceph-mon[35770]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-22T01:31:14.768 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:14 smithi089 ceph-mon[35770]: from='osd.6 [v2:172.21.15.72:6818/3158982661,v1:172.21.15.72:6819/3158982661]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-22T01:31:14.768 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:14 smithi089 ceph-mon[35770]: osdmap e45: 7 total, 6 up, 7 in 2024-03-22T01:31:14.768 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:14 smithi089 ceph-mon[35770]: from='osd.6 [v2:172.21.15.72:6818/3158982661,v1:172.21.15.72:6819/3158982661]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]: dispatch 2024-03-22T01:31:14.768 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:14 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T01:31:14.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:14 smithi098 ceph-mon[36468]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-22T01:31:14.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:14 smithi098 ceph-mon[36468]: from='osd.6 [v2:172.21.15.72:6818/3158982661,v1:172.21.15.72:6819/3158982661]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-22T01:31:14.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:14 smithi098 ceph-mon[36468]: osdmap e45: 7 total, 6 up, 7 in 2024-03-22T01:31:14.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:14 smithi098 ceph-mon[36468]: from='osd.6 [v2:172.21.15.72:6818/3158982661,v1:172.21.15.72:6819/3158982661]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]: dispatch 2024-03-22T01:31:14.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:14 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T01:31:15.593 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:15 smithi089 ceph-mon[35770]: from='osd.6 [v2:172.21.15.72:6818/3158982661,v1:172.21.15.72:6819/3158982661]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]': finished 2024-03-22T01:31:15.594 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:15 smithi089 ceph-mon[35770]: osdmap e46: 7 total, 6 up, 7 in 2024-03-22T01:31:15.594 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:15 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T01:31:15.594 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:15 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T01:31:15.641 INFO:teuthology.orchestra.run.smithi089.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-22T01:31:15.642 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-22T01:31:15.642 INFO:teuthology.orchestra.run.smithi089.stderr: stderr: 10+0 records in 2024-03-22T01:31:15.642 INFO:teuthology.orchestra.run.smithi089.stderr:10+0 records out 2024-03-22T01:31:15.642 INFO:teuthology.orchestra.run.smithi089.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112659 s, 931 MB/s 2024-03-22T01:31:15.642 INFO:teuthology.orchestra.run.smithi089.stderr:--> Zapping successful for: 2024-03-22T01:31:15.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:15 smithi072 ceph-mon[32414]: from='osd.6 [v2:172.21.15.72:6818/3158982661,v1:172.21.15.72:6819/3158982661]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]': finished 2024-03-22T01:31:15.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:15 smithi072 ceph-mon[32414]: osdmap e46: 7 total, 6 up, 7 in 2024-03-22T01:31:15.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:15 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T01:31:15.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:15 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T01:31:15.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:15 smithi098 ceph-mon[36468]: from='osd.6 [v2:172.21.15.72:6818/3158982661,v1:172.21.15.72:6819/3158982661]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]': finished 2024-03-22T01:31:15.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:15 smithi098 ceph-mon[36468]: osdmap e46: 7 total, 6 up, 7 in 2024-03-22T01:31:15.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:15 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T01:31:15.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:15 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T01:31:16.359 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi089:vg_nvme/lv_2 2024-03-22T01:31:16.601 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.b/config 2024-03-22T01:31:16.607 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:16 smithi089 ceph-mon[35770]: purged_snaps scrub starts 2024-03-22T01:31:16.607 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:16 smithi089 ceph-mon[35770]: purged_snaps scrub ok 2024-03-22T01:31:16.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:16 smithi089 ceph-mon[35770]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-22T01:31:16.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:16 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T01:31:16.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:16 smithi089 ceph-mon[35770]: osd.6 [v2:172.21.15.72:6818/3158982661,v1:172.21.15.72:6819/3158982661] boot 2024-03-22T01:31:16.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:16 smithi089 ceph-mon[35770]: osdmap e47: 7 total, 7 up, 7 in 2024-03-22T01:31:16.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:16 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T01:31:16.755 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:16 smithi072 ceph-mon[32414]: purged_snaps scrub starts 2024-03-22T01:31:16.756 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:16 smithi072 ceph-mon[32414]: purged_snaps scrub ok 2024-03-22T01:31:16.756 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:16 smithi072 ceph-mon[32414]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-22T01:31:16.756 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:16 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T01:31:16.756 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:16 smithi072 ceph-mon[32414]: osd.6 [v2:172.21.15.72:6818/3158982661,v1:172.21.15.72:6819/3158982661] boot 2024-03-22T01:31:16.756 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:16 smithi072 ceph-mon[32414]: osdmap e47: 7 total, 7 up, 7 in 2024-03-22T01:31:16.756 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:16 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T01:31:16.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:16 smithi098 ceph-mon[36468]: purged_snaps scrub starts 2024-03-22T01:31:16.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:16 smithi098 ceph-mon[36468]: purged_snaps scrub ok 2024-03-22T01:31:16.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:16 smithi098 ceph-mon[36468]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-22T01:31:16.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:16 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T01:31:16.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:16 smithi098 ceph-mon[36468]: osd.6 [v2:172.21.15.72:6818/3158982661,v1:172.21.15.72:6819/3158982661] boot 2024-03-22T01:31:16.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:16 smithi098 ceph-mon[36468]: osdmap e47: 7 total, 7 up, 7 in 2024-03-22T01:31:16.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:16 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T01:31:17.681 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:17 smithi089 ceph-mon[35770]: osdmap e48: 7 total, 7 up, 7 in 2024-03-22T01:31:17.681 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:17 smithi089 ceph-mon[35770]: Detected new or changed devices on smithi072 2024-03-22T01:31:17.681 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:17 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:17.681 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:17 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:17.681 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:17 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:31:17.681 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:17 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:31:17.682 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:17 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:31:17.682 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:17 smithi089 ceph-mon[35770]: Adjusting osd_memory_target on smithi072 to 5696M 2024-03-22T01:31:17.682 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:17 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:17.682 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:17 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:31:17.682 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:17 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:31:17.682 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:17 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:17.682 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:17 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:31:17.682 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:17 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:31:17.682 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:17 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:31:17.682 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:17 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:17.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:17 smithi072 ceph-mon[32414]: osdmap e48: 7 total, 7 up, 7 in 2024-03-22T01:31:17.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:17 smithi072 ceph-mon[32414]: Detected new or changed devices on smithi072 2024-03-22T01:31:17.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:17 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:17.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:17 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:17.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:17 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:31:17.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:17 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:31:17.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:17 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:31:17.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:17 smithi072 ceph-mon[32414]: Adjusting osd_memory_target on smithi072 to 5696M 2024-03-22T01:31:17.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:17 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:17.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:17 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:31:17.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:17 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:31:17.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:17 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:17.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:17 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:31:17.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:17 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:31:17.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:17 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:31:17.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:17 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:17.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:17 smithi098 ceph-mon[36468]: osdmap e48: 7 total, 7 up, 7 in 2024-03-22T01:31:17.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:17 smithi098 ceph-mon[36468]: Detected new or changed devices on smithi072 2024-03-22T01:31:17.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:17 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:17.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:17 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:17.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:17 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:31:17.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:17 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:31:17.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:17 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:31:17.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:17 smithi098 ceph-mon[36468]: Adjusting osd_memory_target on smithi072 to 5696M 2024-03-22T01:31:17.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:17 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:17.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:17 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:31:17.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:17 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:31:17.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:17 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:17.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:17 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:31:17.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:17 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:31:17.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:17 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:31:17.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:17 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:18.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:31:18.244+0000 7f2e043bf700 1 -- 172.21.15.89:0/1526021708 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2dfc102850 msgr2=0x7f2dfc102cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:31:18.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:31:18.244+0000 7f2e043bf700 1 --2- 172.21.15.89:0/1526021708 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2dfc102850 0x7f2dfc102cb0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2df80099c0 tx=0x7f2df8009cd0 comp rx=0 tx=0).stop 2024-03-22T01:31:18.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:31:18.244+0000 7f2e043bf700 1 -- 172.21.15.89:0/1526021708 shutdown_connections 2024-03-22T01:31:18.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:31:18.244+0000 7f2e043bf700 1 --2- 172.21.15.89:0/1526021708 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2dfc1031f0 0x7f2dfc109aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:31:18.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:31:18.244+0000 7f2e043bf700 1 --2- 172.21.15.89:0/1526021708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2dfc101650 0x7f2dfc101a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:31:18.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:31:18.244+0000 7f2e043bf700 1 --2- 172.21.15.89:0/1526021708 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2dfc102850 0x7f2dfc102cb0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:31:18.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:31:18.244+0000 7f2e043bf700 1 -- 172.21.15.89:0/1526021708 >> 172.21.15.89:0/1526021708 conn(0x7f2dfc0fcc10 msgr2=0x7f2dfc0ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:31:18.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:31:18.245+0000 7f2e043bf700 1 -- 172.21.15.89:0/1526021708 shutdown_connections 2024-03-22T01:31:18.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:31:18.245+0000 7f2e043bf700 1 -- 172.21.15.89:0/1526021708 wait complete. 2024-03-22T01:31:18.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:31:18.245+0000 7f2e043bf700 1 Processor -- start 2024-03-22T01:31:18.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:31:18.245+0000 7f2e043bf700 1 -- start start 2024-03-22T01:31:18.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:31:18.245+0000 7f2e043bf700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2dfc101650 0x7f2dfc06b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:31:18.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:31:18.245+0000 7f2e043bf700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2dfc102850 0x7f2dfc19d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:31:18.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:31:18.245+0000 7f2e043bf700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2dfc1031f0 0x7f2dfc19fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:31:18.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:31:18.245+0000 7f2e043bf700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f2dfc10d1c0 con 0x7f2dfc101650 2024-03-22T01:31:18.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:31:18.245+0000 7f2e043bf700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2dfc10d040 con 0x7f2dfc1031f0 2024-03-22T01:31:18.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:31:18.245+0000 7f2e043bf700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2dfc10d340 con 0x7f2dfc102850 2024-03-22T01:31:18.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:31:18.245+0000 7f2e0295c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2dfc1031f0 0x7f2dfc19fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:31:18.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:31:18.245+0000 7f2e0295c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2dfc1031f0 0x7f2dfc19fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40000/0 (socket says 172.21.15.89:40000) 2024-03-22T01:31:18.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:31:18.245+0000 7f2e0295c700 1 -- 172.21.15.89:0/130390186 learned_addr learned my addr 172.21.15.89:0/130390186 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-22T01:31:18.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:31:18.245+0000 7f2e0215b700 1 --2- 172.21.15.89:0/130390186 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2dfc101650 0x7f2dfc06b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:31:18.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:31:18.246+0000 7f2e0195a700 1 --2- 172.21.15.89:0/130390186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2dfc102850 0x7f2dfc19d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:31:18.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:31:18.246+0000 7f2e0295c700 1 -- 172.21.15.89:0/130390186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2dfc102850 msgr2=0x7f2dfc19d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:31:18.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:31:18.246+0000 7f2e0295c700 1 --2- 172.21.15.89:0/130390186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2dfc102850 0x7f2dfc19d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:31:18.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:31:18.246+0000 7f2e0295c700 1 -- 172.21.15.89:0/130390186 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2dfc101650 msgr2=0x7f2dfc06b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:31:18.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:31:18.246+0000 7f2e0295c700 1 --2- 172.21.15.89:0/130390186 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2dfc101650 0x7f2dfc06b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:31:18.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:31:18.246+0000 7f2e0295c700 1 -- 172.21.15.89:0/130390186 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dfc1a0520 con 0x7f2dfc1031f0 2024-03-22T01:31:18.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:31:18.246+0000 7f2e0295c700 1 --2- 172.21.15.89:0/130390186 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2dfc1031f0 0x7f2dfc19fc50 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2df4007ee0 tx=0x7f2df400d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:31:18.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:31:18.246+0000 7f2df37fe700 1 -- 172.21.15.89:0/130390186 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2df400dbd0 con 0x7f2dfc1031f0 2024-03-22T01:31:18.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:31:18.246+0000 7f2e043bf700 1 -- 172.21.15.89:0/130390186 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2dfc1a0810 con 0x7f2dfc1031f0 2024-03-22T01:31:18.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:31:18.246+0000 7f2df37fe700 1 -- 172.21.15.89:0/130390186 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2df400dd70 con 0x7f2dfc1031f0 2024-03-22T01:31:18.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:31:18.246+0000 7f2df37fe700 1 -- 172.21.15.89:0/130390186 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2df4005a60 con 0x7f2dfc1031f0 2024-03-22T01:31:18.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:31:18.246+0000 7f2e043bf700 1 -- 172.21.15.89:0/130390186 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2dfc1a0d20 con 0x7f2dfc1031f0 2024-03-22T01:31:18.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:31:18.247+0000 7f2e043bf700 1 -- 172.21.15.89:0/130390186 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2dfc069340 con 0x7f2dfc1031f0 2024-03-22T01:31:18.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:31:18.248+0000 7f2df37fe700 1 -- 172.21.15.89:0/130390186 <== mon.2 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f2df401a020 con 0x7f2dfc1031f0 2024-03-22T01:31:18.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:31:18.249+0000 7f2df37fe700 1 --2- 172.21.15.89:0/130390186 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f2de80a2fb0 0x7f2de80a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:31:18.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:31:18.249+0000 7f2df37fe700 1 -- 172.21.15.89:0/130390186 <== mon.2 v2:172.21.15.89:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 4965+0+0 (secure 0 0 0) 0x7f2df40ce4a0 con 0x7f2dfc1031f0 2024-03-22T01:31:18.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:31:18.249+0000 7f2e0215b700 1 --2- 172.21.15.89:0/130390186 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f2de80a2fb0 0x7f2de80a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:31:18.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:31:18.250+0000 7f2e0215b700 1 --2- 172.21.15.89:0/130390186 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f2de80a2fb0 0x7f2de80a5470 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc1026b0 tx=0x7f2dec00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:31:18.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:31:18.252+0000 7f2df37fe700 1 -- 172.21.15.89:0/130390186 <== mon.2 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2df4010070 con 0x7f2dfc1031f0 2024-03-22T01:31:18.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:31:18.456+0000 7f2e043bf700 1 -- 172.21.15.89:0/130390186 --> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f2dfc062b80 con 0x7f2de80a2fb0 2024-03-22T01:31:18.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:18 smithi089 ceph-mon[35770]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-03-22T01:31:18.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:18 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T01:31:18.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:18 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T01:31:18.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:18 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:31:18.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:18 smithi072 ceph-mon[32414]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-03-22T01:31:18.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:18 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T01:31:18.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:18 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T01:31:18.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:18 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:31:18.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:18 smithi098 ceph-mon[36468]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-03-22T01:31:18.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:18 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T01:31:18.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:18 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T01:31:18.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:18 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:31:19.607 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:19 smithi089 ceph-mon[35770]: from='client.24275 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:31:19.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:19 smithi072 ceph-mon[32414]: from='client.24275 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:31:19.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:19 smithi098 ceph-mon[36468]: from='client.24275 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:31:20.728 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:20 smithi089 ceph-mon[35770]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-22T01:31:20.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:20 smithi072 ceph-mon[32414]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-22T01:31:20.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:20 smithi098 ceph-mon[36468]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-22T01:31:21.812 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:21 smithi089 ceph-mon[35770]: from='client.? 172.21.15.89:0/2364687900' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a73469f-cf3a-4306-ae93-ea46e21ebbbe"}]: dispatch 2024-03-22T01:31:21.812 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:21 smithi089 ceph-mon[35770]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a73469f-cf3a-4306-ae93-ea46e21ebbbe"}]: dispatch 2024-03-22T01:31:21.813 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:21 smithi089 ceph-mon[35770]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7a73469f-cf3a-4306-ae93-ea46e21ebbbe"}]': finished 2024-03-22T01:31:21.813 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:21 smithi089 ceph-mon[35770]: osdmap e49: 8 total, 7 up, 8 in 2024-03-22T01:31:21.813 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:21 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T01:31:21.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:21 smithi072 ceph-mon[32414]: from='client.? 172.21.15.89:0/2364687900' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a73469f-cf3a-4306-ae93-ea46e21ebbbe"}]: dispatch 2024-03-22T01:31:21.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:21 smithi072 ceph-mon[32414]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a73469f-cf3a-4306-ae93-ea46e21ebbbe"}]: dispatch 2024-03-22T01:31:21.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:21 smithi072 ceph-mon[32414]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7a73469f-cf3a-4306-ae93-ea46e21ebbbe"}]': finished 2024-03-22T01:31:21.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:21 smithi072 ceph-mon[32414]: osdmap e49: 8 total, 7 up, 8 in 2024-03-22T01:31:21.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:21 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T01:31:21.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:21 smithi098 ceph-mon[36468]: from='client.? 172.21.15.89:0/2364687900' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a73469f-cf3a-4306-ae93-ea46e21ebbbe"}]: dispatch 2024-03-22T01:31:21.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:21 smithi098 ceph-mon[36468]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a73469f-cf3a-4306-ae93-ea46e21ebbbe"}]: dispatch 2024-03-22T01:31:21.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:21 smithi098 ceph-mon[36468]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7a73469f-cf3a-4306-ae93-ea46e21ebbbe"}]': finished 2024-03-22T01:31:21.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:21 smithi098 ceph-mon[36468]: osdmap e49: 8 total, 7 up, 8 in 2024-03-22T01:31:21.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:21 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T01:31:22.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:22 smithi089 ceph-mon[35770]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-22T01:31:22.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:22 smithi089 ceph-mon[35770]: from='client.? 172.21.15.89:0/2802908235' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T01:31:22.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:22 smithi072 ceph-mon[32414]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-22T01:31:22.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:22 smithi072 ceph-mon[32414]: from='client.? 172.21.15.89:0/2802908235' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T01:31:22.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:22 smithi098 ceph-mon[36468]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-22T01:31:22.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:22 smithi098 ceph-mon[36468]: from='client.? 172.21.15.89:0/2802908235' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T01:31:24.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:24 smithi089 ceph-mon[35770]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-22T01:31:24.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:24 smithi072 ceph-mon[32414]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-22T01:31:24.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:24 smithi098 ceph-mon[36468]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-22T01:31:26.721 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:26 smithi089 ceph-mon[35770]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-22T01:31:26.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:26 smithi072 ceph-mon[32414]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-22T01:31:26.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:26 smithi098 ceph-mon[36468]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-22T01:31:28.590 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:28 smithi089 ceph-mon[35770]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-22T01:31:28.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:28 smithi072 ceph-mon[32414]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-22T01:31:28.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:28 smithi098 ceph-mon[36468]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-22T01:31:29.747 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:29 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-22T01:31:29.747 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:29 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:31:29.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:29 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-22T01:31:29.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:29 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:31:29.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:29 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-22T01:31:29.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:29 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:31:30.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:30 smithi089 ceph-mon[35770]: Deploying daemon osd.7 on smithi089 2024-03-22T01:31:30.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:30 smithi089 ceph-mon[35770]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-22T01:31:30.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:30 smithi072 ceph-mon[32414]: Deploying daemon osd.7 on smithi089 2024-03-22T01:31:30.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:30 smithi072 ceph-mon[32414]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-22T01:31:30.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:30 smithi098 ceph-mon[36468]: Deploying daemon osd.7 on smithi089 2024-03-22T01:31:30.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:30 smithi098 ceph-mon[36468]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-22T01:31:32.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:32 smithi089 ceph-mon[35770]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-22T01:31:32.874 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:32 smithi072 ceph-mon[32414]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-22T01:31:32.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:32 smithi098 ceph-mon[36468]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-22T01:31:34.628 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:34 smithi089 ceph-mon[35770]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-22T01:31:34.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:34 smithi072 ceph-mon[32414]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-22T01:31:34.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:34 smithi098 ceph-mon[36468]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-22T01:31:36.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:35 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:36.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:35 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:36.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:35 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:31:36.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:35 smithi089 ceph-mon[35770]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-22T01:31:36.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:35 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:36.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:35 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:36.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:35 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:31:36.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:35 smithi072 ceph-mon[32414]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-22T01:31:36.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:35 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:36.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:35 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:36.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:35 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:31:36.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:35 smithi098 ceph-mon[36468]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-22T01:31:38.323 INFO:teuthology.orchestra.run.smithi089.stdout:Created osd(s) 7 on host 'smithi089' 2024-03-22T01:31:38.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:31:38.323+0000 7f2df37fe700 1 -- 172.21.15.89:0/130390186 <== mgr.14152 v2:172.21.15.72:6800/401403200 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f2dfc062b80 con 0x7f2de80a2fb0 2024-03-22T01:31:38.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:31:38.326+0000 7f2e043bf700 1 -- 172.21.15.89:0/130390186 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f2de80a2fb0 msgr2=0x7f2de80a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:31:38.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:31:38.326+0000 7f2e043bf700 1 --2- 172.21.15.89:0/130390186 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f2de80a2fb0 0x7f2de80a5470 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc1026b0 tx=0x7f2dec00b410 comp rx=0 tx=0).stop 2024-03-22T01:31:38.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:31:38.326+0000 7f2e043bf700 1 -- 172.21.15.89:0/130390186 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2dfc1031f0 msgr2=0x7f2dfc19fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:31:38.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:31:38.326+0000 7f2e043bf700 1 --2- 172.21.15.89:0/130390186 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2dfc1031f0 0x7f2dfc19fc50 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2df4007ee0 tx=0x7f2df400d3b0 comp rx=0 tx=0).stop 2024-03-22T01:31:38.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:31:38.326+0000 7f2e043bf700 1 -- 172.21.15.89:0/130390186 shutdown_connections 2024-03-22T01:31:38.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:31:38.326+0000 7f2e043bf700 1 --2- 172.21.15.89:0/130390186 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f2de80a2fb0 0x7f2de80a5470 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:31:38.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:31:38.326+0000 7f2e043bf700 1 --2- 172.21.15.89:0/130390186 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2dfc1031f0 0x7f2dfc19fc50 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:31:38.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:31:38.326+0000 7f2e043bf700 1 --2- 172.21.15.89:0/130390186 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2dfc101650 0x7f2dfc06b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:31:38.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:31:38.326+0000 7f2e043bf700 1 --2- 172.21.15.89:0/130390186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2dfc102850 0x7f2dfc19d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:31:38.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:31:38.326+0000 7f2e043bf700 1 -- 172.21.15.89:0/130390186 >> 172.21.15.89:0/130390186 conn(0x7f2dfc0fcc10 msgr2=0x7f2dfc105bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:31:38.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:31:38.326+0000 7f2e043bf700 1 -- 172.21.15.89:0/130390186 shutdown_connections 2024-03-22T01:31:38.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:31:38.326+0000 7f2e043bf700 1 -- 172.21.15.89:0/130390186 wait complete. 2024-03-22T01:31:38.607 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:38 smithi089 ceph-mon[35770]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-22T01:31:38.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:38 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:38.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:38 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:38.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:38 smithi072 ceph-mon[32414]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-22T01:31:38.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:38 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:38.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:38 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:38.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:38 smithi098 ceph-mon[36468]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-22T01:31:38.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:38 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:38.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:38 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:39.051 DEBUG:teuthology.orchestra.run.smithi089:osd.7> sudo journalctl -f -n 0 -u ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@osd.7.service 2024-03-22T01:31:39.053 INFO:tasks.cephadm:Deploying osd.8 on smithi098 with /dev/vg_nvme/lv_2... 2024-03-22T01:31:39.054 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-22T01:31:39.300 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.c/config 2024-03-22T01:31:39.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:39 smithi098 ceph-mon[36468]: from='osd.7 [v2:172.21.15.89:6816/3075913234,v1:172.21.15.89:6817/3075913234]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-22T01:31:39.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:39 smithi098 ceph-mon[36468]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-22T01:31:39.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:39 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:39.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:39 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:39.783 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:39 smithi089 ceph-mon[35770]: from='osd.7 [v2:172.21.15.89:6816/3075913234,v1:172.21.15.89:6817/3075913234]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-22T01:31:39.783 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:39 smithi089 ceph-mon[35770]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-22T01:31:39.783 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:39 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:39.783 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:39 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:39.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:39 smithi072 ceph-mon[32414]: from='osd.7 [v2:172.21.15.89:6816/3075913234,v1:172.21.15.89:6817/3075913234]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-22T01:31:39.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:39 smithi072 ceph-mon[32414]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-22T01:31:39.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:39 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:39.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:39 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:40.736 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:40 smithi098 ceph-mon[36468]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-22T01:31:40.736 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:40 smithi098 ceph-mon[36468]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-22T01:31:40.736 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:40 smithi098 ceph-mon[36468]: osdmap e50: 8 total, 7 up, 8 in 2024-03-22T01:31:40.736 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:40 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T01:31:40.736 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:40 smithi098 ceph-mon[36468]: from='osd.7 [v2:172.21.15.89:6816/3075913234,v1:172.21.15.89:6817/3075913234]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-03-22T01:31:40.736 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:40 smithi098 ceph-mon[36468]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-03-22T01:31:40.848 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:40 smithi089 ceph-mon[35770]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-22T01:31:40.848 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:40 smithi089 ceph-mon[35770]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-22T01:31:40.848 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:40 smithi089 ceph-mon[35770]: osdmap e50: 8 total, 7 up, 8 in 2024-03-22T01:31:40.848 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:40 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T01:31:40.848 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:40 smithi089 ceph-mon[35770]: from='osd.7 [v2:172.21.15.89:6816/3075913234,v1:172.21.15.89:6817/3075913234]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-03-22T01:31:40.848 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:40 smithi089 ceph-mon[35770]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-03-22T01:31:40.848 INFO:journalctl@ceph.osd.7.smithi089.stdout:Mar 22 01:31:40 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-7[54352]: 2024-03-22T01:31:40.460+0000 7f09d6680700 -1 osd.7 0 waiting for initial osdmap 2024-03-22T01:31:40.848 INFO:journalctl@ceph.osd.7.smithi089.stdout:Mar 22 01:31:40 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-7[54352]: 2024-03-22T01:31:40.468+0000 7f09d2473700 -1 osd.7 51 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-22T01:31:40.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:40 smithi072 ceph-mon[32414]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-22T01:31:40.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:40 smithi072 ceph-mon[32414]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-22T01:31:40.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:40 smithi072 ceph-mon[32414]: osdmap e50: 8 total, 7 up, 8 in 2024-03-22T01:31:40.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:40 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T01:31:40.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:40 smithi072 ceph-mon[32414]: from='osd.7 [v2:172.21.15.89:6816/3075913234,v1:172.21.15.89:6817/3075913234]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-03-22T01:31:40.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:40 smithi072 ceph-mon[32414]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-03-22T01:31:41.478 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-22T01:31:41.479 INFO:teuthology.orchestra.run.smithi098.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-22T01:31:41.479 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10+0 records in 2024-03-22T01:31:41.479 INFO:teuthology.orchestra.run.smithi098.stderr:10+0 records out 2024-03-22T01:31:41.479 INFO:teuthology.orchestra.run.smithi098.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0119914 s, 874 MB/s 2024-03-22T01:31:41.479 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping successful for: 2024-03-22T01:31:41.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:41 smithi098 ceph-mon[36468]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2024-03-22T01:31:41.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:41 smithi098 ceph-mon[36468]: osdmap e51: 8 total, 7 up, 8 in 2024-03-22T01:31:41.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:41 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T01:31:41.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:41 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T01:31:41.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:41 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T01:31:41.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:41 smithi089 ceph-mon[35770]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2024-03-22T01:31:41.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:41 smithi089 ceph-mon[35770]: osdmap e51: 8 total, 7 up, 8 in 2024-03-22T01:31:41.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:41 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T01:31:41.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:41 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T01:31:41.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:41 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T01:31:41.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:41 smithi072 ceph-mon[32414]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2024-03-22T01:31:41.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:41 smithi072 ceph-mon[32414]: osdmap e51: 8 total, 7 up, 8 in 2024-03-22T01:31:41.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:41 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T01:31:41.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:41 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T01:31:41.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:41 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T01:31:42.108 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi098:vg_nvme/lv_2 2024-03-22T01:31:42.356 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.c/config 2024-03-22T01:31:42.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:42 smithi089 ceph-mon[35770]: purged_snaps scrub starts 2024-03-22T01:31:42.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:42 smithi089 ceph-mon[35770]: purged_snaps scrub ok 2024-03-22T01:31:42.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:42 smithi089 ceph-mon[35770]: pgmap v174: 1 pgs: 1 unknown; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-22T01:31:42.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:42 smithi089 ceph-mon[35770]: osd.7 [v2:172.21.15.89:6816/3075913234,v1:172.21.15.89:6817/3075913234] boot 2024-03-22T01:31:42.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:42 smithi089 ceph-mon[35770]: osdmap e52: 8 total, 8 up, 8 in 2024-03-22T01:31:42.609 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:42 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T01:31:42.726 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:42 smithi098 ceph-mon[36468]: purged_snaps scrub starts 2024-03-22T01:31:42.727 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:42 smithi098 ceph-mon[36468]: purged_snaps scrub ok 2024-03-22T01:31:42.727 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:42 smithi098 ceph-mon[36468]: pgmap v174: 1 pgs: 1 unknown; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-22T01:31:42.727 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:42 smithi098 ceph-mon[36468]: osd.7 [v2:172.21.15.89:6816/3075913234,v1:172.21.15.89:6817/3075913234] boot 2024-03-22T01:31:42.727 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:42 smithi098 ceph-mon[36468]: osdmap e52: 8 total, 8 up, 8 in 2024-03-22T01:31:42.727 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:42 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T01:31:42.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:42 smithi072 ceph-mon[32414]: purged_snaps scrub starts 2024-03-22T01:31:42.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:42 smithi072 ceph-mon[32414]: purged_snaps scrub ok 2024-03-22T01:31:42.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:42 smithi072 ceph-mon[32414]: pgmap v174: 1 pgs: 1 unknown; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-22T01:31:42.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:42 smithi072 ceph-mon[32414]: osd.7 [v2:172.21.15.89:6816/3075913234,v1:172.21.15.89:6817/3075913234] boot 2024-03-22T01:31:42.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:42 smithi072 ceph-mon[32414]: osdmap e52: 8 total, 8 up, 8 in 2024-03-22T01:31:42.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:42 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T01:31:43.842 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:43 smithi098 ceph-mon[36468]: osdmap e53: 8 total, 8 up, 8 in 2024-03-22T01:31:43.842 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:43 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:43.842 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:43 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:43.842 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:43 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:31:43.842 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:43 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:31:43.842 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:43 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:31:43.842 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:43 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:43.842 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:43 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:31:43.842 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:43 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:31:43.843 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:43 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:43.843 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:43 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:31:43.843 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:43 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:31:43.843 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:43 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:31:43.843 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:43 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:43.843 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:43 smithi098 ceph-mon[36468]: osdmap e54: 8 total, 8 up, 8 in 2024-03-22T01:31:43.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:43 smithi089 ceph-mon[35770]: osdmap e53: 8 total, 8 up, 8 in 2024-03-22T01:31:43.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:43 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:43.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:43 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:43.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:43 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:31:43.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:43 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:31:43.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:43 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:31:43.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:43 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:43.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:43 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:31:43.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:43 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:31:43.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:43 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:43.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:43 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:31:43.859 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:43 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:31:43.859 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:43 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:31:43.859 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:43 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:43.859 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:43 smithi089 ceph-mon[35770]: osdmap e54: 8 total, 8 up, 8 in 2024-03-22T01:31:43.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:43 smithi072 ceph-mon[32414]: osdmap e53: 8 total, 8 up, 8 in 2024-03-22T01:31:43.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:43 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:43.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:43 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:43.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:43 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:31:43.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:43 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:31:43.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:43 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:31:43.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:43 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:43.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:43 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:31:43.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:43 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:31:43.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:43 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:43.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:43 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:31:43.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:43 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:31:43.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:43 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:31:43.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:43 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:31:43.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:43 smithi072 ceph-mon[32414]: osdmap e54: 8 total, 8 up, 8 in 2024-03-22T01:31:44.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:31:44.021+0000 7fe482c65700 1 -- 172.21.15.98:0/1728794332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe47c102180 msgr2=0x7fe47c109a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:31:44.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:31:44.021+0000 7fe482c65700 1 --2- 172.21.15.98:0/1728794332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe47c102180 0x7fe47c109a20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe46c009990 tx=0x7fe46c009ca0 comp rx=0 tx=0).stop 2024-03-22T01:31:44.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:31:44.021+0000 7fe482c65700 1 -- 172.21.15.98:0/1728794332 shutdown_connections 2024-03-22T01:31:44.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:31:44.021+0000 7fe482c65700 1 --2- 172.21.15.98:0/1728794332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe47c102180 0x7fe47c109a20 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:31:44.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:31:44.021+0000 7fe482c65700 1 --2- 172.21.15.98:0/1728794332 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe47c102b60 0x7fe47c102f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:31:44.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:31:44.021+0000 7fe482c65700 1 --2- 172.21.15.98:0/1728794332 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe47c1017c0 0x7fe47c101c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:31:44.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:31:44.021+0000 7fe482c65700 1 -- 172.21.15.98:0/1728794332 >> 172.21.15.98:0/1728794332 conn(0x7fe47c0fc2f0 msgr2=0x7fe47c0fe710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:31:44.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:31:44.021+0000 7fe482c65700 1 -- 172.21.15.98:0/1728794332 shutdown_connections 2024-03-22T01:31:44.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:31:44.021+0000 7fe482c65700 1 -- 172.21.15.98:0/1728794332 wait complete. 2024-03-22T01:31:44.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:31:44.022+0000 7fe482c65700 1 Processor -- start 2024-03-22T01:31:44.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:31:44.022+0000 7fe482c65700 1 -- start start 2024-03-22T01:31:44.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:31:44.022+0000 7fe482c65700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe47c1017c0 0x7fe47c105560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:31:44.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:31:44.022+0000 7fe482c65700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe47c102180 0x7fe47c103bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:31:44.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:31:44.022+0000 7fe482c65700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe47c102b60 0x7fe47c1040f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:31:44.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:31:44.022+0000 7fe482c65700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fe47c10d180 con 0x7fe47c102b60 2024-03-22T01:31:44.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:31:44.022+0000 7fe482c65700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe47c10d000 con 0x7fe47c102180 2024-03-22T01:31:44.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:31:44.022+0000 7fe482c65700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe47c10d300 con 0x7fe47c1017c0 2024-03-22T01:31:44.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:31:44.022+0000 7fe480a01700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe47c1017c0 0x7fe47c105560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:31:44.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:31:44.022+0000 7fe480a01700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe47c1017c0 0x7fe47c105560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52600/0 (socket says 172.21.15.98:52600) 2024-03-22T01:31:44.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:31:44.022+0000 7fe480a01700 1 -- 172.21.15.98:0/1676517419 learned_addr learned my addr 172.21.15.98:0/1676517419 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-22T01:31:44.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:31:44.022+0000 7fe47bfff700 1 --2- 172.21.15.98:0/1676517419 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe47c102180 0x7fe47c103bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:31:44.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:31:44.022+0000 7fe481202700 1 --2- 172.21.15.98:0/1676517419 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe47c102b60 0x7fe47c1040f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:31:44.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:31:44.022+0000 7fe480a01700 1 -- 172.21.15.98:0/1676517419 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe47c102180 msgr2=0x7fe47c103bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:31:44.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:31:44.022+0000 7fe480a01700 1 --2- 172.21.15.98:0/1676517419 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe47c102180 0x7fe47c103bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:31:44.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:31:44.022+0000 7fe480a01700 1 -- 172.21.15.98:0/1676517419 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe47c102b60 msgr2=0x7fe47c1040f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:31:44.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:31:44.022+0000 7fe480a01700 1 --2- 172.21.15.98:0/1676517419 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe47c102b60 0x7fe47c1040f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:31:44.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:31:44.022+0000 7fe480a01700 1 -- 172.21.15.98:0/1676517419 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe47c1049a0 con 0x7fe47c1017c0 2024-03-22T01:31:44.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:31:44.023+0000 7fe480a01700 1 --2- 172.21.15.98:0/1676517419 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe47c1017c0 0x7fe47c105560 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe47400ef30 tx=0x7fe47400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:31:44.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:31:44.023+0000 7fe479ffb700 1 -- 172.21.15.98:0/1676517419 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4740092e0 con 0x7fe47c1017c0 2024-03-22T01:31:44.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:31:44.023+0000 7fe482c65700 1 -- 172.21.15.98:0/1676517419 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe47c1a2960 con 0x7fe47c1017c0 2024-03-22T01:31:44.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:31:44.023+0000 7fe479ffb700 1 -- 172.21.15.98:0/1676517419 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe474004bb0 con 0x7fe47c1017c0 2024-03-22T01:31:44.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:31:44.023+0000 7fe479ffb700 1 -- 172.21.15.98:0/1676517419 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe474004280 con 0x7fe47c1017c0 2024-03-22T01:31:44.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:31:44.023+0000 7fe482c65700 1 -- 172.21.15.98:0/1676517419 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe47c1a2e70 con 0x7fe47c1017c0 2024-03-22T01:31:44.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:31:44.024+0000 7fe482c65700 1 -- 172.21.15.98:0/1676517419 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe47c1a3040 con 0x7fe47c1017c0 2024-03-22T01:31:44.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:31:44.025+0000 7fe479ffb700 1 -- 172.21.15.98:0/1676517419 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7fe47401b030 con 0x7fe47c1017c0 2024-03-22T01:31:44.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:31:44.025+0000 7fe479ffb700 1 --2- 172.21.15.98:0/1676517419 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe4640a2fb0 0x7fe4640a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:31:44.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:31:44.026+0000 7fe479ffb700 1 -- 172.21.15.98:0/1676517419 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5426+0+0 (secure 0 0 0) 0x7fe4740ce400 con 0x7fe47c1017c0 2024-03-22T01:31:44.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:31:44.026+0000 7fe47bfff700 1 --2- 172.21.15.98:0/1676517419 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe4640a2fb0 0x7fe4640a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:31:44.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:31:44.026+0000 7fe47bfff700 1 --2- 172.21.15.98:0/1676517419 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe4640a2fb0 0x7fe4640a5470 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe470009c60 tx=0x7fe470009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:31:44.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:31:44.030+0000 7fe479ffb700 1 -- 172.21.15.98:0/1676517419 <== mon.1 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe474022020 con 0x7fe47c1017c0 2024-03-22T01:31:44.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:31:44.237+0000 7fe482c65700 1 -- 172.21.15.98:0/1676517419 --> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fe47c0fe3a0 con 0x7fe4640a2fb0 2024-03-22T01:31:44.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:44 smithi098 ceph-mon[36468]: Detected new or changed devices on smithi089 2024-03-22T01:31:44.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:44 smithi098 ceph-mon[36468]: Adjusting osd_memory_target on smithi089 to 5693M 2024-03-22T01:31:44.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:44 smithi098 ceph-mon[36468]: pgmap v177: 1 pgs: 1 unknown; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-22T01:31:44.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:44 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T01:31:44.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:44 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T01:31:44.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:44 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:31:44.773 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:44 smithi072 ceph-mon[32414]: Detected new or changed devices on smithi089 2024-03-22T01:31:44.774 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:44 smithi072 ceph-mon[32414]: Adjusting osd_memory_target on smithi089 to 5693M 2024-03-22T01:31:44.774 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:44 smithi072 ceph-mon[32414]: pgmap v177: 1 pgs: 1 unknown; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-22T01:31:44.774 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:44 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T01:31:44.774 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:44 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T01:31:44.774 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:44 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:31:44.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:44 smithi089 ceph-mon[35770]: Detected new or changed devices on smithi089 2024-03-22T01:31:44.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:44 smithi089 ceph-mon[35770]: Adjusting osd_memory_target on smithi089 to 5693M 2024-03-22T01:31:44.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:44 smithi089 ceph-mon[35770]: pgmap v177: 1 pgs: 1 unknown; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-22T01:31:44.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:44 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T01:31:44.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:44 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T01:31:44.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:44 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:31:45.733 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:45 smithi098 ceph-mon[36468]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:31:45.733 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:45 smithi098 ceph-mon[36468]: osdmap e55: 8 total, 8 up, 8 in 2024-03-22T01:31:45.785 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:45 smithi072 ceph-mon[32414]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:31:45.785 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:45 smithi072 ceph-mon[32414]: osdmap e55: 8 total, 8 up, 8 in 2024-03-22T01:31:45.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:45 smithi089 ceph-mon[35770]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:31:45.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:45 smithi089 ceph-mon[35770]: osdmap e55: 8 total, 8 up, 8 in 2024-03-22T01:31:46.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:46 smithi089 ceph-mon[35770]: pgmap v180: 1 pgs: 1 unknown; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-22T01:31:46.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:46 smithi089 ceph-mon[35770]: osdmap e56: 8 total, 8 up, 8 in 2024-03-22T01:31:46.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:46 smithi072 ceph-mon[32414]: pgmap v180: 1 pgs: 1 unknown; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-22T01:31:46.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:46 smithi072 ceph-mon[32414]: osdmap e56: 8 total, 8 up, 8 in 2024-03-22T01:31:46.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:46 smithi098 ceph-mon[36468]: pgmap v180: 1 pgs: 1 unknown; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-22T01:31:46.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:46 smithi098 ceph-mon[36468]: osdmap e56: 8 total, 8 up, 8 in 2024-03-22T01:31:47.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:47 smithi089 ceph-mon[35770]: from='client.? 172.21.15.98:0/994724256' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c4b13bee-ea4d-4344-95fb-2622ecbb8196"}]: dispatch 2024-03-22T01:31:47.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:47 smithi089 ceph-mon[35770]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c4b13bee-ea4d-4344-95fb-2622ecbb8196"}]: dispatch 2024-03-22T01:31:47.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:47 smithi089 ceph-mon[35770]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c4b13bee-ea4d-4344-95fb-2622ecbb8196"}]': finished 2024-03-22T01:31:47.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:47 smithi089 ceph-mon[35770]: osdmap e57: 9 total, 8 up, 9 in 2024-03-22T01:31:47.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:47 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-22T01:31:47.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:47 smithi072 ceph-mon[32414]: from='client.? 172.21.15.98:0/994724256' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c4b13bee-ea4d-4344-95fb-2622ecbb8196"}]: dispatch 2024-03-22T01:31:47.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:47 smithi072 ceph-mon[32414]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c4b13bee-ea4d-4344-95fb-2622ecbb8196"}]: dispatch 2024-03-22T01:31:47.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:47 smithi072 ceph-mon[32414]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c4b13bee-ea4d-4344-95fb-2622ecbb8196"}]': finished 2024-03-22T01:31:47.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:47 smithi072 ceph-mon[32414]: osdmap e57: 9 total, 8 up, 9 in 2024-03-22T01:31:47.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:47 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-22T01:31:47.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:47 smithi098 ceph-mon[36468]: from='client.? 172.21.15.98:0/994724256' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c4b13bee-ea4d-4344-95fb-2622ecbb8196"}]: dispatch 2024-03-22T01:31:47.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:47 smithi098 ceph-mon[36468]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c4b13bee-ea4d-4344-95fb-2622ecbb8196"}]: dispatch 2024-03-22T01:31:47.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:47 smithi098 ceph-mon[36468]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c4b13bee-ea4d-4344-95fb-2622ecbb8196"}]': finished 2024-03-22T01:31:47.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:47 smithi098 ceph-mon[36468]: osdmap e57: 9 total, 8 up, 9 in 2024-03-22T01:31:47.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:47 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-22T01:31:48.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:48 smithi089 ceph-mon[35770]: pgmap v183: 1 pgs: 1 unknown; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-22T01:31:48.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:48 smithi089 ceph-mon[35770]: from='client.? 172.21.15.98:0/3043464204' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T01:31:48.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:48 smithi072 ceph-mon[32414]: pgmap v183: 1 pgs: 1 unknown; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-22T01:31:48.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:48 smithi072 ceph-mon[32414]: from='client.? 172.21.15.98:0/3043464204' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T01:31:48.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:48 smithi098 ceph-mon[36468]: pgmap v183: 1 pgs: 1 unknown; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-22T01:31:48.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:48 smithi098 ceph-mon[36468]: from='client.? 172.21.15.98:0/3043464204' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T01:31:50.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:50 smithi089 ceph-mon[35770]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-22T01:31:50.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:50 smithi072 ceph-mon[32414]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-22T01:31:50.883 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:50 smithi098 ceph-mon[36468]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-22T01:31:52.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:52 smithi098 ceph-mon[36468]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-22T01:31:52.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:52 smithi089 ceph-mon[35770]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-22T01:31:52.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:52 smithi072 ceph-mon[32414]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-22T01:31:54.798 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:54 smithi098 ceph-mon[36468]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-22T01:31:54.799 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:54 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-22T01:31:54.799 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:54 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:31:54.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:54 smithi089 ceph-mon[35770]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-22T01:31:54.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:54 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-22T01:31:54.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:54 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:31:54.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:54 smithi072 ceph-mon[32414]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-22T01:31:54.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:54 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-22T01:31:54.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:54 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:31:55.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:55 smithi098 ceph-mon[36468]: Deploying daemon osd.8 on smithi098 2024-03-22T01:31:55.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:55 smithi089 ceph-mon[35770]: Deploying daemon osd.8 on smithi098 2024-03-22T01:31:55.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:55 smithi072 ceph-mon[32414]: Deploying daemon osd.8 on smithi098 2024-03-22T01:31:56.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:56 smithi098 ceph-mon[36468]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-22T01:31:56.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:56 smithi089 ceph-mon[35770]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-22T01:31:56.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:56 smithi072 ceph-mon[32414]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-22T01:31:58.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:31:58 smithi089 ceph-mon[35770]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-22T01:31:58.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:31:58 smithi072 ceph-mon[32414]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-22T01:31:58.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:31:58 smithi098 ceph-mon[36468]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-22T01:32:00.807 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:00 smithi098 ceph-mon[36468]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-22T01:32:00.807 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:00 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:00.807 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:00 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:00.808 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:00 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:32:00.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:00 smithi089 ceph-mon[35770]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-22T01:32:00.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:00 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:00.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:00 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:00.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:00 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:32:00.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:00 smithi072 ceph-mon[32414]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-22T01:32:00.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:00 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:00.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:00 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:00.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:00 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:32:02.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:02 smithi089 ceph-mon[35770]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-22T01:32:02.871 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:02 smithi072 ceph-mon[32414]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-22T01:32:02.890 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:02 smithi098 ceph-mon[36468]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-22T01:32:03.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:32:03.657+0000 7fe479ffb700 1 -- 172.21.15.98:0/1676517419 <== mgr.14152 v2:172.21.15.72:6800/401403200 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fe47c0fe3a0 con 0x7fe4640a2fb0 2024-03-22T01:32:03.656 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 8 on host 'smithi098' 2024-03-22T01:32:03.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:32:03.659+0000 7fe482c65700 1 -- 172.21.15.98:0/1676517419 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe4640a2fb0 msgr2=0x7fe4640a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:32:03.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:32:03.659+0000 7fe482c65700 1 --2- 172.21.15.98:0/1676517419 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe4640a2fb0 0x7fe4640a5470 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe470009c60 tx=0x7fe470009260 comp rx=0 tx=0).stop 2024-03-22T01:32:03.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:32:03.659+0000 7fe482c65700 1 -- 172.21.15.98:0/1676517419 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe47c1017c0 msgr2=0x7fe47c105560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:32:03.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:32:03.659+0000 7fe482c65700 1 --2- 172.21.15.98:0/1676517419 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe47c1017c0 0x7fe47c105560 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe47400ef30 tx=0x7fe47400c3a0 comp rx=0 tx=0).stop 2024-03-22T01:32:03.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:32:03.659+0000 7fe482c65700 1 -- 172.21.15.98:0/1676517419 shutdown_connections 2024-03-22T01:32:03.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:32:03.659+0000 7fe482c65700 1 --2- 172.21.15.98:0/1676517419 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe4640a2fb0 0x7fe4640a5470 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:32:03.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:32:03.659+0000 7fe482c65700 1 --2- 172.21.15.98:0/1676517419 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe47c102b60 0x7fe47c1040f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:32:03.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:32:03.659+0000 7fe482c65700 1 --2- 172.21.15.98:0/1676517419 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe47c1017c0 0x7fe47c105560 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:32:03.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:32:03.659+0000 7fe482c65700 1 --2- 172.21.15.98:0/1676517419 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe47c102180 0x7fe47c103bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:32:03.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:32:03.659+0000 7fe482c65700 1 -- 172.21.15.98:0/1676517419 >> 172.21.15.98:0/1676517419 conn(0x7fe47c0fc2f0 msgr2=0x7fe47c0fdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:32:03.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:32:03.659+0000 7fe482c65700 1 -- 172.21.15.98:0/1676517419 shutdown_connections 2024-03-22T01:32:03.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:32:03.659+0000 7fe482c65700 1 -- 172.21.15.98:0/1676517419 wait complete. 2024-03-22T01:32:03.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:03 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:03.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:03 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:03.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:03 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:32:03.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:03 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:32:03.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:03 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:03.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:03 smithi098 ceph-mon[36468]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-22T01:32:03.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:03 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:03.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:03 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:32:03.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:03 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:04.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:03 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:04.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:03 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:04.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:03 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:32:04.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:03 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:32:04.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:03 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:04.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:03 smithi089 ceph-mon[35770]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-22T01:32:04.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:03 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:04.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:03 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:32:04.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:03 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:04.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:03 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:04.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:03 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:04.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:03 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:32:04.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:03 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:32:04.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:03 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:04.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:03 smithi072 ceph-mon[32414]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-22T01:32:04.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:03 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:04.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:03 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:32:04.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:03 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:04.372 DEBUG:teuthology.orchestra.run.smithi098:osd.8> sudo journalctl -f -n 0 -u ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@osd.8.service 2024-03-22T01:32:04.378 INFO:tasks.cephadm:Deploying osd.9 on smithi072 with /dev/vg_nvme/lv_1... 2024-03-22T01:32:04.378 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-22T01:32:04.626 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:32:04.958 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:04 smithi072 ceph-mon[32414]: from='osd.8 [v2:172.21.15.98:6816/1813255577,v1:172.21.15.98:6817/1813255577]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-22T01:32:04.958 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:04 smithi072 ceph-mon[32414]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-22T01:32:05.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:04 smithi089 ceph-mon[35770]: from='osd.8 [v2:172.21.15.98:6816/1813255577,v1:172.21.15.98:6817/1813255577]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-22T01:32:05.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:04 smithi089 ceph-mon[35770]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-22T01:32:05.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:04 smithi098 ceph-mon[36468]: from='osd.8 [v2:172.21.15.98:6816/1813255577,v1:172.21.15.98:6817/1813255577]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-22T01:32:05.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:04 smithi098 ceph-mon[36468]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-22T01:32:05.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:05 smithi072 ceph-mon[32414]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-22T01:32:05.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:05 smithi072 ceph-mon[32414]: osdmap e58: 9 total, 8 up, 9 in 2024-03-22T01:32:05.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:05 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-22T01:32:05.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:05 smithi072 ceph-mon[32414]: from='osd.8 [v2:172.21.15.98:6816/1813255577,v1:172.21.15.98:6817/1813255577]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-22T01:32:05.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:05 smithi072 ceph-mon[32414]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-22T01:32:05.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:05 smithi072 ceph-mon[32414]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-22T01:32:05.910 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:05 smithi098 ceph-mon[36468]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-22T01:32:05.910 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:05 smithi098 ceph-mon[36468]: osdmap e58: 9 total, 8 up, 9 in 2024-03-22T01:32:05.910 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:05 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-22T01:32:05.911 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:05 smithi098 ceph-mon[36468]: from='osd.8 [v2:172.21.15.98:6816/1813255577,v1:172.21.15.98:6817/1813255577]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-22T01:32:05.911 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:05 smithi098 ceph-mon[36468]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-22T01:32:05.911 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:05 smithi098 ceph-mon[36468]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-22T01:32:06.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:05 smithi089 ceph-mon[35770]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-22T01:32:06.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:05 smithi089 ceph-mon[35770]: osdmap e58: 9 total, 8 up, 9 in 2024-03-22T01:32:06.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:05 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-22T01:32:06.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:05 smithi089 ceph-mon[35770]: from='osd.8 [v2:172.21.15.98:6816/1813255577,v1:172.21.15.98:6817/1813255577]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-22T01:32:06.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:05 smithi089 ceph-mon[35770]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-22T01:32:06.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:05 smithi089 ceph-mon[35770]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-22T01:32:06.671 INFO:journalctl@ceph.osd.8.smithi098.stdout:Mar 22 01:32:06 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-8[55604]: 2024-03-22T01:32:06.407+0000 7f98840bc700 -1 osd.8 0 waiting for initial osdmap 2024-03-22T01:32:06.672 INFO:journalctl@ceph.osd.8.smithi098.stdout:Mar 22 01:32:06 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-8[55604]: 2024-03-22T01:32:06.418+0000 7f987bea7700 -1 osd.8 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-22T01:32:06.780 INFO:teuthology.orchestra.run.smithi072.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-22T01:32:06.781 INFO:teuthology.orchestra.run.smithi072.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-22T01:32:06.781 INFO:teuthology.orchestra.run.smithi072.stderr: stderr: 10+0 records in 2024-03-22T01:32:06.781 INFO:teuthology.orchestra.run.smithi072.stderr:10+0 records out 2024-03-22T01:32:06.781 INFO:teuthology.orchestra.run.smithi072.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113432 s, 924 MB/s 2024-03-22T01:32:06.781 INFO:teuthology.orchestra.run.smithi072.stderr:--> Zapping successful for: 2024-03-22T01:32:07.085 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:06 smithi072 ceph-mon[32414]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-22T01:32:07.085 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:06 smithi072 ceph-mon[32414]: osdmap e59: 9 total, 8 up, 9 in 2024-03-22T01:32:07.085 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:06 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-22T01:32:07.085 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:06 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-22T01:32:07.085 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:06 smithi072 ceph-mon[32414]: from='osd.8 ' entity='osd.8' 2024-03-22T01:32:07.086 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:06 smithi072 ceph-mon[32414]: osd.8 [v2:172.21.15.98:6816/1813255577,v1:172.21.15.98:6817/1813255577] boot 2024-03-22T01:32:07.086 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:06 smithi072 ceph-mon[32414]: osdmap e60: 9 total, 9 up, 9 in 2024-03-22T01:32:07.086 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:06 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-22T01:32:07.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:06 smithi089 ceph-mon[35770]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-22T01:32:07.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:06 smithi089 ceph-mon[35770]: osdmap e59: 9 total, 8 up, 9 in 2024-03-22T01:32:07.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:06 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-22T01:32:07.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:06 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-22T01:32:07.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:06 smithi089 ceph-mon[35770]: from='osd.8 ' entity='osd.8' 2024-03-22T01:32:07.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:06 smithi089 ceph-mon[35770]: osd.8 [v2:172.21.15.98:6816/1813255577,v1:172.21.15.98:6817/1813255577] boot 2024-03-22T01:32:07.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:06 smithi089 ceph-mon[35770]: osdmap e60: 9 total, 9 up, 9 in 2024-03-22T01:32:07.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:06 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-22T01:32:07.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:06 smithi098 ceph-mon[36468]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-22T01:32:07.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:06 smithi098 ceph-mon[36468]: osdmap e59: 9 total, 8 up, 9 in 2024-03-22T01:32:07.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:06 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-22T01:32:07.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:06 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-22T01:32:07.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:06 smithi098 ceph-mon[36468]: from='osd.8 ' entity='osd.8' 2024-03-22T01:32:07.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:06 smithi098 ceph-mon[36468]: osd.8 [v2:172.21.15.98:6816/1813255577,v1:172.21.15.98:6817/1813255577] boot 2024-03-22T01:32:07.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:06 smithi098 ceph-mon[36468]: osdmap e60: 9 total, 9 up, 9 in 2024-03-22T01:32:07.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:06 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-22T01:32:07.528 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi072:vg_nvme/lv_1 2024-03-22T01:32:07.776 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:32:07.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:07 smithi072 ceph-mon[32414]: purged_snaps scrub starts 2024-03-22T01:32:07.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:07 smithi072 ceph-mon[32414]: purged_snaps scrub ok 2024-03-22T01:32:07.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:07 smithi072 ceph-mon[32414]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 804 GiB / 805 GiB avail 2024-03-22T01:32:07.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:07 smithi072 ceph-mon[32414]: osdmap e61: 9 total, 9 up, 9 in 2024-03-22T01:32:08.107 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:07 smithi089 ceph-mon[35770]: purged_snaps scrub starts 2024-03-22T01:32:08.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:07 smithi089 ceph-mon[35770]: purged_snaps scrub ok 2024-03-22T01:32:08.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:07 smithi089 ceph-mon[35770]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 804 GiB / 805 GiB avail 2024-03-22T01:32:08.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:07 smithi089 ceph-mon[35770]: osdmap e61: 9 total, 9 up, 9 in 2024-03-22T01:32:08.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:07 smithi098 ceph-mon[36468]: purged_snaps scrub starts 2024-03-22T01:32:08.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:07 smithi098 ceph-mon[36468]: purged_snaps scrub ok 2024-03-22T01:32:08.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:07 smithi098 ceph-mon[36468]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 804 GiB / 805 GiB avail 2024-03-22T01:32:08.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:07 smithi098 ceph-mon[36468]: osdmap e61: 9 total, 9 up, 9 in 2024-03-22T01:32:09.244 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:08 smithi072 ceph-mon[32414]: Detected new or changed devices on smithi098 2024-03-22T01:32:09.244 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:08 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:09.245 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:08 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:09.245 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:08 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:32:09.245 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:08 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:32:09.245 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:08 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:32:09.245 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:08 smithi072 ceph-mon[32414]: Adjusting osd_memory_target on smithi098 to 5693M 2024-03-22T01:32:09.245 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:08 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:09.245 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:08 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:32:09.245 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:08 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:32:09.245 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:08 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:09.245 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:08 smithi072 ceph-mon[32414]: osdmap e62: 9 total, 9 up, 9 in 2024-03-22T01:32:09.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:08 smithi089 ceph-mon[35770]: Detected new or changed devices on smithi098 2024-03-22T01:32:09.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:08 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:09.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:08 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:09.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:08 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:32:09.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:08 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:32:09.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:08 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:32:09.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:08 smithi089 ceph-mon[35770]: Adjusting osd_memory_target on smithi098 to 5693M 2024-03-22T01:32:09.359 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:08 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:09.359 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:08 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:32:09.359 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:08 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:32:09.359 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:08 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:09.359 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:08 smithi089 ceph-mon[35770]: osdmap e62: 9 total, 9 up, 9 in 2024-03-22T01:32:09.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:08 smithi098 ceph-mon[36468]: Detected new or changed devices on smithi098 2024-03-22T01:32:09.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:08 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:09.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:08 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:09.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:08 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:32:09.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:08 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:32:09.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:08 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:32:09.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:08 smithi098 ceph-mon[36468]: Adjusting osd_memory_target on smithi098 to 5693M 2024-03-22T01:32:09.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:08 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:09.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:08 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:32:09.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:08 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:32:09.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:08 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:09.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:08 smithi098 ceph-mon[36468]: osdmap e62: 9 total, 9 up, 9 in 2024-03-22T01:32:09.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:32:09.523+0000 7f0138511700 1 -- 172.21.15.72:0/2592596885 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01301027a0 msgr2=0x7f0130102c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:32:09.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:32:09.523+0000 7f0138511700 1 --2- 172.21.15.72:0/2592596885 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01301027a0 0x7f0130102c00 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f01200099c0 tx=0x7f0120009cd0 comp rx=0 tx=0).stop 2024-03-22T01:32:09.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:32:09.523+0000 7f0138511700 1 -- 172.21.15.72:0/2592596885 shutdown_connections 2024-03-22T01:32:09.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:32:09.523+0000 7f0138511700 1 --2- 172.21.15.72:0/2592596885 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0130103140 0x7f01301099f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:32:09.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:32:09.523+0000 7f0138511700 1 --2- 172.21.15.72:0/2592596885 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01301015a0 0x7f01301019c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:32:09.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:32:09.523+0000 7f0138511700 1 --2- 172.21.15.72:0/2592596885 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01301027a0 0x7f0130102c00 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:32:09.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:32:09.523+0000 7f0138511700 1 -- 172.21.15.72:0/2592596885 >> 172.21.15.72:0/2592596885 conn(0x7f01300fcb40 msgr2=0x7f01300fef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:32:09.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:32:09.523+0000 7f0138511700 1 -- 172.21.15.72:0/2592596885 shutdown_connections 2024-03-22T01:32:09.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:32:09.523+0000 7f0138511700 1 -- 172.21.15.72:0/2592596885 wait complete. 2024-03-22T01:32:09.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:32:09.524+0000 7f0138511700 1 Processor -- start 2024-03-22T01:32:09.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:32:09.524+0000 7f0138511700 1 -- start start 2024-03-22T01:32:09.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:32:09.524+0000 7f0138511700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01301015a0 0x7f0130197710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:32:09.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:32:09.524+0000 7f0138511700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01301027a0 0x7f0130197c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:32:09.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:32:09.524+0000 7f0138511700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0130103140 0x7f013019ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:32:09.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:32:09.524+0000 7f0138511700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f013010d190 con 0x7f01301027a0 2024-03-22T01:32:09.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:32:09.524+0000 7f0138511700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f013010d010 con 0x7f0130103140 2024-03-22T01:32:09.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:32:09.524+0000 7f0138511700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f013010d310 con 0x7f01301015a0 2024-03-22T01:32:09.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:32:09.524+0000 7f0135aac700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01301027a0 0x7f0130197c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:32:09.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:32:09.524+0000 7f0135aac700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01301027a0 0x7f0130197c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38010/0 (socket says 172.21.15.72:38010) 2024-03-22T01:32:09.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:32:09.524+0000 7f0135aac700 1 -- 172.21.15.72:0/1122230627 learned_addr learned my addr 172.21.15.72:0/1122230627 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:32:09.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:32:09.524+0000 7f01362ad700 1 --2- 172.21.15.72:0/1122230627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01301015a0 0x7f0130197710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:32:09.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:32:09.524+0000 7f0136aae700 1 --2- 172.21.15.72:0/1122230627 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0130103140 0x7f013019ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:32:09.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:32:09.524+0000 7f0135aac700 1 -- 172.21.15.72:0/1122230627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01301015a0 msgr2=0x7f0130197710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:32:09.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:32:09.524+0000 7f0135aac700 1 --2- 172.21.15.72:0/1122230627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01301015a0 0x7f0130197710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:32:09.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:32:09.524+0000 7f0135aac700 1 -- 172.21.15.72:0/1122230627 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0130103140 msgr2=0x7f013019ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:32:09.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:32:09.524+0000 7f0135aac700 1 --2- 172.21.15.72:0/1122230627 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0130103140 0x7f013019ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:32:09.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:32:09.524+0000 7f0135aac700 1 -- 172.21.15.72:0/1122230627 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f013019f370 con 0x7f01301027a0 2024-03-22T01:32:09.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:32:09.524+0000 7f01362ad700 1 --2- 172.21.15.72:0/1122230627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01301015a0 0x7f0130197710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:32:09.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:32:09.524+0000 7f0136aae700 1 --2- 172.21.15.72:0/1122230627 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0130103140 0x7f013019ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:32:09.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:32:09.524+0000 7f0135aac700 1 --2- 172.21.15.72:0/1122230627 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01301027a0 0x7f0130197c50 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f012000b580 tx=0x7f0120009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:32:09.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:32:09.525+0000 7f01277fe700 1 -- 172.21.15.72:0/1122230627 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0120027070 con 0x7f01301027a0 2024-03-22T01:32:09.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:32:09.525+0000 7f0138511700 1 -- 172.21.15.72:0/1122230627 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f013019f600 con 0x7f01301027a0 2024-03-22T01:32:09.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:32:09.525+0000 7f01277fe700 1 -- 172.21.15.72:0/1122230627 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0120004150 con 0x7f01301027a0 2024-03-22T01:32:09.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:32:09.525+0000 7f01277fe700 1 -- 172.21.15.72:0/1122230627 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f012000f460 con 0x7f01301027a0 2024-03-22T01:32:09.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:32:09.525+0000 7f0138511700 1 -- 172.21.15.72:0/1122230627 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f013019fa50 con 0x7f01301027a0 2024-03-22T01:32:09.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:32:09.526+0000 7f0138511700 1 -- 172.21.15.72:0/1122230627 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0130069340 con 0x7f01301027a0 2024-03-22T01:32:09.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:32:09.527+0000 7f01277fe700 1 -- 172.21.15.72:0/1122230627 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f012002a070 con 0x7f01301027a0 2024-03-22T01:32:09.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:32:09.527+0000 7f01277fe700 1 --2- 172.21.15.72:0/1122230627 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f011c0a2fb0 0x7f011c0a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:32:09.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:32:09.527+0000 7f01362ad700 1 --2- 172.21.15.72:0/1122230627 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f011c0a2fb0 0x7f011c0a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:32:09.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:32:09.527+0000 7f01277fe700 1 -- 172.21.15.72:0/1122230627 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5858+0+0 (secure 0 0 0) 0x7f01200cfe80 con 0x7f01301027a0 2024-03-22T01:32:09.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:32:09.528+0000 7f01362ad700 1 --2- 172.21.15.72:0/1122230627 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f011c0a2fb0 0x7f011c0a5470 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0130102600 tx=0x7f0128009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:32:09.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:32:09.531+0000 7f01277fe700 1 -- 172.21.15.72:0/1122230627 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0120031070 con 0x7f01301027a0 2024-03-22T01:32:09.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:32:09.738+0000 7f0138511700 1 -- 172.21.15.72:0/1122230627 --> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi072:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f0130062ac0 con 0x7f011c0a2fb0 2024-03-22T01:32:10.342 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:09 smithi072 ceph-mon[32414]: pgmap v199: 1 pgs: 1 remapped+peering; 577 KiB data, 368 MiB used, 804 GiB / 805 GiB avail 2024-03-22T01:32:10.342 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:09 smithi072 ceph-mon[32414]: osdmap e63: 9 total, 9 up, 9 in 2024-03-22T01:32:10.342 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:09 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T01:32:10.343 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:09 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T01:32:10.343 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:09 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:32:10.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:09 smithi089 ceph-mon[35770]: pgmap v199: 1 pgs: 1 remapped+peering; 577 KiB data, 368 MiB used, 804 GiB / 805 GiB avail 2024-03-22T01:32:10.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:09 smithi089 ceph-mon[35770]: osdmap e63: 9 total, 9 up, 9 in 2024-03-22T01:32:10.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:09 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T01:32:10.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:09 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T01:32:10.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:09 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:32:10.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:09 smithi098 ceph-mon[36468]: pgmap v199: 1 pgs: 1 remapped+peering; 577 KiB data, 368 MiB used, 804 GiB / 805 GiB avail 2024-03-22T01:32:10.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:09 smithi098 ceph-mon[36468]: osdmap e63: 9 total, 9 up, 9 in 2024-03-22T01:32:10.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:09 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T01:32:10.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:09 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T01:32:10.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:09 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:32:11.350 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:10 smithi072 ceph-mon[32414]: from='client.14421 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi072:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:32:11.350 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:10 smithi072 ceph-mon[32414]: osdmap e64: 9 total, 9 up, 9 in 2024-03-22T01:32:11.357 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:10 smithi089 ceph-mon[35770]: from='client.14421 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi072:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:32:11.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:10 smithi089 ceph-mon[35770]: osdmap e64: 9 total, 9 up, 9 in 2024-03-22T01:32:11.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:10 smithi098 ceph-mon[36468]: from='client.14421 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi072:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:32:11.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:10 smithi098 ceph-mon[36468]: osdmap e64: 9 total, 9 up, 9 in 2024-03-22T01:32:12.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:11 smithi089 ceph-mon[35770]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 368 MiB used, 804 GiB / 805 GiB avail 2024-03-22T01:32:12.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:11 smithi072 ceph-mon[32414]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 368 MiB used, 804 GiB / 805 GiB avail 2024-03-22T01:32:12.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:11 smithi098 ceph-mon[36468]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 368 MiB used, 804 GiB / 805 GiB avail 2024-03-22T01:32:13.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:12 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/564435866' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eefb0a44-4a36-4e55-9d04-0501bf718db6"}]: dispatch 2024-03-22T01:32:13.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:12 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/564435866' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eefb0a44-4a36-4e55-9d04-0501bf718db6"}]': finished 2024-03-22T01:32:13.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:12 smithi089 ceph-mon[35770]: osdmap e65: 10 total, 9 up, 10 in 2024-03-22T01:32:13.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:12 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-22T01:32:13.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:12 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/564435866' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eefb0a44-4a36-4e55-9d04-0501bf718db6"}]: dispatch 2024-03-22T01:32:13.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:12 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/564435866' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eefb0a44-4a36-4e55-9d04-0501bf718db6"}]': finished 2024-03-22T01:32:13.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:12 smithi072 ceph-mon[32414]: osdmap e65: 10 total, 9 up, 10 in 2024-03-22T01:32:13.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:12 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-22T01:32:13.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:12 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/564435866' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eefb0a44-4a36-4e55-9d04-0501bf718db6"}]: dispatch 2024-03-22T01:32:13.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:12 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/564435866' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eefb0a44-4a36-4e55-9d04-0501bf718db6"}]': finished 2024-03-22T01:32:13.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:12 smithi098 ceph-mon[36468]: osdmap e65: 10 total, 9 up, 10 in 2024-03-22T01:32:13.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:12 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-22T01:32:14.273 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:13 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/2275064066' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T01:32:14.273 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:13 smithi072 ceph-mon[32414]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail; 123 KiB/s, 0 objects/s recovering 2024-03-22T01:32:14.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:13 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/2275064066' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T01:32:14.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:13 smithi089 ceph-mon[35770]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail; 123 KiB/s, 0 objects/s recovering 2024-03-22T01:32:14.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:13 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/2275064066' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T01:32:14.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:13 smithi098 ceph-mon[36468]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail; 123 KiB/s, 0 objects/s recovering 2024-03-22T01:32:16.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:16 smithi089 ceph-mon[35770]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-22T01:32:16.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:16 smithi072 ceph-mon[32414]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-22T01:32:16.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:16 smithi098 ceph-mon[36468]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-22T01:32:18.787 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:18 smithi072 ceph-mon[32414]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-03-22T01:32:18.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:18 smithi089 ceph-mon[35770]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-03-22T01:32:18.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:18 smithi098 ceph-mon[36468]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-03-22T01:32:20.803 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:20 smithi072 ceph-mon[32414]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-03-22T01:32:20.803 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:20 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-22T01:32:20.803 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:20 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:32:20.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:20 smithi089 ceph-mon[35770]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-03-22T01:32:20.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:20 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-22T01:32:20.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:20 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:32:20.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:20 smithi098 ceph-mon[36468]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-03-22T01:32:20.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:20 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-22T01:32:20.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:20 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:32:21.742 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:21 smithi072 ceph-mon[32414]: Deploying daemon osd.9 on smithi072 2024-03-22T01:32:21.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:21 smithi089 ceph-mon[35770]: Deploying daemon osd.9 on smithi072 2024-03-22T01:32:21.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:21 smithi098 ceph-mon[36468]: Deploying daemon osd.9 on smithi072 2024-03-22T01:32:22.731 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:22 smithi072 ceph-mon[32414]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-22T01:32:22.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:22 smithi089 ceph-mon[35770]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-22T01:32:22.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:22 smithi098 ceph-mon[36468]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-22T01:32:24.724 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:24 smithi098 ceph-mon[36468]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail 2024-03-22T01:32:24.787 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:24 smithi072 ceph-mon[32414]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail 2024-03-22T01:32:24.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:24 smithi089 ceph-mon[35770]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail 2024-03-22T01:32:26.745 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:26 smithi072 ceph-mon[32414]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail 2024-03-22T01:32:26.745 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:26 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:26.745 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:26 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:26.745 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:26 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:32:26.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:26 smithi089 ceph-mon[35770]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail 2024-03-22T01:32:26.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:26 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:26.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:26 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:26.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:26 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:32:26.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:26 smithi098 ceph-mon[36468]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail 2024-03-22T01:32:26.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:26 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:26.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:26 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:26.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:26 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:32:28.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:28 smithi089 ceph-mon[35770]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail 2024-03-22T01:32:28.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:28 smithi072 ceph-mon[32414]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail 2024-03-22T01:32:28.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:28 smithi098 ceph-mon[36468]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail 2024-03-22T01:32:29.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:32:29.770+0000 7f01277fe700 1 -- 172.21.15.72:0/1122230627 <== mgr.14152 v2:172.21.15.72:6800/401403200 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f0130062ac0 con 0x7f011c0a2fb0 2024-03-22T01:32:29.771 INFO:teuthology.orchestra.run.smithi072.stdout:Created osd(s) 9 on host 'smithi072' 2024-03-22T01:32:29.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:32:29.772+0000 7f0138511700 1 -- 172.21.15.72:0/1122230627 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f011c0a2fb0 msgr2=0x7f011c0a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:32:29.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:32:29.772+0000 7f0138511700 1 --2- 172.21.15.72:0/1122230627 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f011c0a2fb0 0x7f011c0a5470 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0130102600 tx=0x7f0128009040 comp rx=0 tx=0).stop 2024-03-22T01:32:29.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:32:29.772+0000 7f0138511700 1 -- 172.21.15.72:0/1122230627 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01301027a0 msgr2=0x7f0130197c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:32:29.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:32:29.772+0000 7f0138511700 1 --2- 172.21.15.72:0/1122230627 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01301027a0 0x7f0130197c50 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f012000b580 tx=0x7f0120009580 comp rx=0 tx=0).stop 2024-03-22T01:32:29.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:32:29.772+0000 7f0138511700 1 -- 172.21.15.72:0/1122230627 shutdown_connections 2024-03-22T01:32:29.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:32:29.772+0000 7f0138511700 1 --2- 172.21.15.72:0/1122230627 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f011c0a2fb0 0x7f011c0a5470 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:32:29.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:32:29.772+0000 7f0138511700 1 --2- 172.21.15.72:0/1122230627 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0130103140 0x7f013019ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:32:29.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:32:29.772+0000 7f0138511700 1 --2- 172.21.15.72:0/1122230627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01301015a0 0x7f0130197710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:32:29.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:32:29.772+0000 7f0138511700 1 --2- 172.21.15.72:0/1122230627 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01301027a0 0x7f0130197c50 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:32:29.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:32:29.772+0000 7f0138511700 1 -- 172.21.15.72:0/1122230627 >> 172.21.15.72:0/1122230627 conn(0x7f01300fcb40 msgr2=0x7f0130105a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:32:29.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:32:29.773+0000 7f0138511700 1 -- 172.21.15.72:0/1122230627 shutdown_connections 2024-03-22T01:32:29.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:32:29.773+0000 7f0138511700 1 -- 172.21.15.72:0/1122230627 wait complete. 2024-03-22T01:32:30.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:29 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:30.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:29 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:30.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:29 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:32:30.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:29 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:32:30.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:29 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:30.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:29 smithi089 ceph-mon[35770]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail 2024-03-22T01:32:30.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:29 smithi089 ceph-mon[35770]: from='osd.9 [v2:172.21.15.72:6826/3377749206,v1:172.21.15.72:6827/3377749206]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-22T01:32:30.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:29 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:30.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:29 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:30.359 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:29 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:32:30.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:29 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:30.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:29 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:30.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:29 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:32:30.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:29 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:32:30.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:29 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:30.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:29 smithi072 ceph-mon[32414]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail 2024-03-22T01:32:30.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:29 smithi072 ceph-mon[32414]: from='osd.9 [v2:172.21.15.72:6826/3377749206,v1:172.21.15.72:6827/3377749206]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-22T01:32:30.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:29 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:30.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:29 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:30.373 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:29 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:32:30.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:29 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:30.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:29 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:30.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:29 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:32:30.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:29 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:32:30.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:29 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:30.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:29 smithi098 ceph-mon[36468]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail 2024-03-22T01:32:30.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:29 smithi098 ceph-mon[36468]: from='osd.9 [v2:172.21.15.72:6826/3377749206,v1:172.21.15.72:6827/3377749206]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-22T01:32:30.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:29 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:30.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:29 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:30.423 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:29 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:32:30.549 DEBUG:teuthology.orchestra.run.smithi072:osd.9> sudo journalctl -f -n 0 -u ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@osd.9.service 2024-03-22T01:32:30.551 INFO:tasks.cephadm:Deploying osd.10 on smithi089 with /dev/vg_nvme/lv_1... 2024-03-22T01:32:30.552 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-22T01:32:30.799 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.b/config 2024-03-22T01:32:31.106 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:30 smithi072 ceph-mon[32414]: from='osd.9 [v2:172.21.15.72:6826/3377749206,v1:172.21.15.72:6827/3377749206]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-22T01:32:31.106 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:30 smithi072 ceph-mon[32414]: osdmap e66: 10 total, 9 up, 10 in 2024-03-22T01:32:31.106 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:30 smithi072 ceph-mon[32414]: from='osd.9 [v2:172.21.15.72:6826/3377749206,v1:172.21.15.72:6827/3377749206]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]: dispatch 2024-03-22T01:32:31.106 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:30 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-22T01:32:31.107 INFO:journalctl@ceph.osd.9.smithi072.stdout:Mar 22 01:32:30 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-9[65321]: 2024-03-22T01:32:30.947+0000 7fcc0ca0d700 -1 osd.9 0 waiting for initial osdmap 2024-03-22T01:32:31.107 INFO:journalctl@ceph.osd.9.smithi072.stdout:Mar 22 01:32:30 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-9[65321]: 2024-03-22T01:32:30.958+0000 7fcc08800700 -1 osd.9 67 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-22T01:32:31.177 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:30 smithi089 ceph-mon[35770]: from='osd.9 [v2:172.21.15.72:6826/3377749206,v1:172.21.15.72:6827/3377749206]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-22T01:32:31.177 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:30 smithi089 ceph-mon[35770]: osdmap e66: 10 total, 9 up, 10 in 2024-03-22T01:32:31.177 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:30 smithi089 ceph-mon[35770]: from='osd.9 [v2:172.21.15.72:6826/3377749206,v1:172.21.15.72:6827/3377749206]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]: dispatch 2024-03-22T01:32:31.177 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:30 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-22T01:32:31.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:30 smithi098 ceph-mon[36468]: from='osd.9 [v2:172.21.15.72:6826/3377749206,v1:172.21.15.72:6827/3377749206]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-22T01:32:31.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:30 smithi098 ceph-mon[36468]: osdmap e66: 10 total, 9 up, 10 in 2024-03-22T01:32:31.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:30 smithi098 ceph-mon[36468]: from='osd.9 [v2:172.21.15.72:6826/3377749206,v1:172.21.15.72:6827/3377749206]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]: dispatch 2024-03-22T01:32:31.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:30 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-22T01:32:32.030 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:31 smithi072 ceph-mon[32414]: from='osd.9 [v2:172.21.15.72:6826/3377749206,v1:172.21.15.72:6827/3377749206]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]': finished 2024-03-22T01:32:32.030 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:31 smithi072 ceph-mon[32414]: osdmap e67: 10 total, 9 up, 10 in 2024-03-22T01:32:32.030 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:31 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-22T01:32:32.031 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:31 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-22T01:32:32.031 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:31 smithi072 ceph-mon[32414]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail 2024-03-22T01:32:32.307 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:31 smithi089 ceph-mon[35770]: from='osd.9 [v2:172.21.15.72:6826/3377749206,v1:172.21.15.72:6827/3377749206]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]': finished 2024-03-22T01:32:32.307 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:31 smithi089 ceph-mon[35770]: osdmap e67: 10 total, 9 up, 10 in 2024-03-22T01:32:32.307 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:31 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-22T01:32:32.307 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:31 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-22T01:32:32.307 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:31 smithi089 ceph-mon[35770]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail 2024-03-22T01:32:32.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:31 smithi098 ceph-mon[36468]: from='osd.9 [v2:172.21.15.72:6826/3377749206,v1:172.21.15.72:6827/3377749206]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]': finished 2024-03-22T01:32:32.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:31 smithi098 ceph-mon[36468]: osdmap e67: 10 total, 9 up, 10 in 2024-03-22T01:32:32.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:31 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-22T01:32:32.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:31 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-22T01:32:32.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:31 smithi098 ceph-mon[36468]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail 2024-03-22T01:32:32.870 INFO:teuthology.orchestra.run.smithi089.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-22T01:32:32.870 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-22T01:32:32.870 INFO:teuthology.orchestra.run.smithi089.stderr: stderr: 10+0 records in 2024-03-22T01:32:32.870 INFO:teuthology.orchestra.run.smithi089.stderr:10+0 records out 2024-03-22T01:32:32.870 INFO:teuthology.orchestra.run.smithi089.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0112783 s, 930 MB/s 2024-03-22T01:32:32.870 INFO:teuthology.orchestra.run.smithi089.stderr:--> Zapping successful for: 2024-03-22T01:32:33.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:32 smithi072 ceph-mon[32414]: purged_snaps scrub starts 2024-03-22T01:32:33.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:32 smithi072 ceph-mon[32414]: purged_snaps scrub ok 2024-03-22T01:32:33.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:32 smithi072 ceph-mon[32414]: osd.9 [v2:172.21.15.72:6826/3377749206,v1:172.21.15.72:6827/3377749206] boot 2024-03-22T01:32:33.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:32 smithi072 ceph-mon[32414]: osdmap e68: 10 total, 10 up, 10 in 2024-03-22T01:32:33.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:32 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-22T01:32:33.258 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:32 smithi089 ceph-mon[35770]: purged_snaps scrub starts 2024-03-22T01:32:33.258 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:32 smithi089 ceph-mon[35770]: purged_snaps scrub ok 2024-03-22T01:32:33.258 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:32 smithi089 ceph-mon[35770]: osd.9 [v2:172.21.15.72:6826/3377749206,v1:172.21.15.72:6827/3377749206] boot 2024-03-22T01:32:33.258 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:32 smithi089 ceph-mon[35770]: osdmap e68: 10 total, 10 up, 10 in 2024-03-22T01:32:33.258 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:32 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-22T01:32:33.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:32 smithi098 ceph-mon[36468]: purged_snaps scrub starts 2024-03-22T01:32:33.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:32 smithi098 ceph-mon[36468]: purged_snaps scrub ok 2024-03-22T01:32:33.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:32 smithi098 ceph-mon[36468]: osd.9 [v2:172.21.15.72:6826/3377749206,v1:172.21.15.72:6827/3377749206] boot 2024-03-22T01:32:33.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:32 smithi098 ceph-mon[36468]: osdmap e68: 10 total, 10 up, 10 in 2024-03-22T01:32:33.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:32 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-22T01:32:33.603 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi089:vg_nvme/lv_1 2024-03-22T01:32:33.847 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.b/config 2024-03-22T01:32:34.133 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:33 smithi089 ceph-mon[35770]: osdmap e69: 10 total, 10 up, 10 in 2024-03-22T01:32:34.133 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:33 smithi089 ceph-mon[35770]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 284 MiB used, 894 GiB / 894 GiB avail 2024-03-22T01:32:34.133 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:33 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:34.133 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:33 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:34.133 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:33 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:32:34.133 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:33 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:32:34.133 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:33 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:32:34.133 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:33 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:32:34.133 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:33 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:34.134 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:33 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:32:34.134 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:33 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:32:34.134 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:33 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:34.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:33 smithi072 ceph-mon[32414]: osdmap e69: 10 total, 10 up, 10 in 2024-03-22T01:32:34.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:33 smithi072 ceph-mon[32414]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 284 MiB used, 894 GiB / 894 GiB avail 2024-03-22T01:32:34.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:33 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:34.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:33 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:34.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:33 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:32:34.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:33 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:32:34.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:33 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:32:34.373 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:33 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:32:34.373 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:33 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:34.373 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:33 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:32:34.373 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:33 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:32:34.373 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:33 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:34.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:33 smithi098 ceph-mon[36468]: osdmap e69: 10 total, 10 up, 10 in 2024-03-22T01:32:34.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:33 smithi098 ceph-mon[36468]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 284 MiB used, 894 GiB / 894 GiB avail 2024-03-22T01:32:34.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:33 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:34.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:33 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:34.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:33 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:32:34.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:33 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:32:34.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:33 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:32:34.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:33 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:32:34.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:33 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:34.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:33 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:32:34.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:33 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:32:34.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:33 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:35.065 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:34 smithi089 ceph-mon[35770]: Detected new or changed devices on smithi072 2024-03-22T01:32:35.065 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:34 smithi089 ceph-mon[35770]: Adjusting osd_memory_target on smithi072 to 4272M 2024-03-22T01:32:35.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:34 smithi072 ceph-mon[32414]: Detected new or changed devices on smithi072 2024-03-22T01:32:35.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:34 smithi072 ceph-mon[32414]: Adjusting osd_memory_target on smithi072 to 4272M 2024-03-22T01:32:35.415 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:34 smithi098 ceph-mon[36468]: Detected new or changed devices on smithi072 2024-03-22T01:32:35.415 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:34 smithi098 ceph-mon[36468]: Adjusting osd_memory_target on smithi072 to 4272M 2024-03-22T01:32:35.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:32:35.613+0000 7f4631b91700 1 -- 172.21.15.89:0/3467246018 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f462c1031c0 msgr2=0x7f462c109a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:32:35.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:32:35.613+0000 7f4631b91700 1 --2- 172.21.15.89:0/3467246018 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f462c1031c0 0x7f462c109a70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f462400b0d0 tx=0x7f462400b3e0 comp rx=0 tx=0).stop 2024-03-22T01:32:35.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:32:35.613+0000 7f4631b91700 1 -- 172.21.15.89:0/3467246018 shutdown_connections 2024-03-22T01:32:35.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:32:35.613+0000 7f4631b91700 1 --2- 172.21.15.89:0/3467246018 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f462c1031c0 0x7f462c109a70 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:32:35.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:32:35.613+0000 7f4631b91700 1 --2- 172.21.15.89:0/3467246018 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f462c101620 0x7f462c101a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:32:35.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:32:35.613+0000 7f4631b91700 1 --2- 172.21.15.89:0/3467246018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f462c102820 0x7f462c102c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:32:35.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:32:35.613+0000 7f4631b91700 1 -- 172.21.15.89:0/3467246018 >> 172.21.15.89:0/3467246018 conn(0x7f462c0fcbe0 msgr2=0x7f462c0ff000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:32:35.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:32:35.613+0000 7f4631b91700 1 -- 172.21.15.89:0/3467246018 shutdown_connections 2024-03-22T01:32:35.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:32:35.613+0000 7f4631b91700 1 -- 172.21.15.89:0/3467246018 wait complete. 2024-03-22T01:32:35.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:32:35.614+0000 7f4631b91700 1 Processor -- start 2024-03-22T01:32:35.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:32:35.614+0000 7f4631b91700 1 -- start start 2024-03-22T01:32:35.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:32:35.614+0000 7f4631b91700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f462c101620 0x7f462c06b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:32:35.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:32:35.614+0000 7f4631b91700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f462c102820 0x7f462c19d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:32:35.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:32:35.614+0000 7f4631b91700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f462c1031c0 0x7f462c19fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:32:35.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:32:35.614+0000 7f4631b91700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f462c10d1f0 con 0x7f462c101620 2024-03-22T01:32:35.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:32:35.614+0000 7f4631b91700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f462c10d070 con 0x7f462c102820 2024-03-22T01:32:35.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:32:35.614+0000 7f4631b91700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f462c10d370 con 0x7f462c1031c0 2024-03-22T01:32:35.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:32:35.614+0000 7f462affd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f462c102820 0x7f462c19d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:32:35.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:32:35.614+0000 7f462affd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f462c102820 0x7f462c19d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34460/0 (socket says 172.21.15.89:34460) 2024-03-22T01:32:35.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:32:35.614+0000 7f462affd700 1 -- 172.21.15.89:0/1803677199 learned_addr learned my addr 172.21.15.89:0/1803677199 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-03-22T01:32:35.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:32:35.614+0000 7f462bfff700 1 --2- 172.21.15.89:0/1803677199 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f462c1031c0 0x7f462c19fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:32:35.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:32:35.614+0000 7f462affd700 1 -- 172.21.15.89:0/1803677199 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f462c1031c0 msgr2=0x7f462c19fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:32:35.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:32:35.614+0000 7f462affd700 1 --2- 172.21.15.89:0/1803677199 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f462c1031c0 0x7f462c19fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:32:35.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:32:35.614+0000 7f462affd700 1 -- 172.21.15.89:0/1803677199 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f462c101620 msgr2=0x7f462c06b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:32:35.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:32:35.614+0000 7f462b7fe700 1 --2- 172.21.15.89:0/1803677199 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f462c101620 0x7f462c06b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:32:35.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:32:35.615+0000 7f462affd700 1 --2- 172.21.15.89:0/1803677199 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f462c101620 0x7f462c06b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:32:35.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:32:35.615+0000 7f462affd700 1 -- 172.21.15.89:0/1803677199 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f462c1a0450 con 0x7f462c102820 2024-03-22T01:32:35.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:32:35.615+0000 7f462affd700 1 --2- 172.21.15.89:0/1803677199 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f462c102820 0x7f462c19d540 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f461c00b530 tx=0x7f461c00b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:32:35.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:32:35.615+0000 7f4628ff9700 1 -- 172.21.15.89:0/1803677199 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f461c015020 con 0x7f462c102820 2024-03-22T01:32:35.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:32:35.615+0000 7f4631b91700 1 -- 172.21.15.89:0/1803677199 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f462c1a0740 con 0x7f462c102820 2024-03-22T01:32:35.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:32:35.615+0000 7f4628ff9700 1 -- 172.21.15.89:0/1803677199 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f461c011d10 con 0x7f462c102820 2024-03-22T01:32:35.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:32:35.615+0000 7f4631b91700 1 -- 172.21.15.89:0/1803677199 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f462c1a0d50 con 0x7f462c102820 2024-03-22T01:32:35.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:32:35.615+0000 7f4628ff9700 1 -- 172.21.15.89:0/1803677199 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f461c010e50 con 0x7f462c102820 2024-03-22T01:32:35.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:32:35.616+0000 7f4631b91700 1 -- 172.21.15.89:0/1803677199 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f462c069340 con 0x7f462c102820 2024-03-22T01:32:35.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:32:35.617+0000 7f4628ff9700 1 -- 172.21.15.89:0/1803677199 <== mon.2 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f461c018020 con 0x7f462c102820 2024-03-22T01:32:35.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:32:35.618+0000 7f4628ff9700 1 --2- 172.21.15.89:0/1803677199 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f46140a3150 0x7f46140a5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:32:35.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:32:35.618+0000 7f4628ff9700 1 -- 172.21.15.89:0/1803677199 <== mon.2 v2:172.21.15.89:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 6261+0+0 (secure 0 0 0) 0x7f461c0ce9e0 con 0x7f462c102820 2024-03-22T01:32:35.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:32:35.618+0000 7f462b7fe700 1 --2- 172.21.15.89:0/1803677199 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f46140a3150 0x7f46140a5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:32:35.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:32:35.618+0000 7f462b7fe700 1 --2- 172.21.15.89:0/1803677199 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f46140a3150 0x7f46140a5610 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f462c102680 tx=0x7f4620005d70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:32:35.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:32:35.621+0000 7f4628ff9700 1 -- 172.21.15.89:0/1803677199 <== mon.2 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f461c01f020 con 0x7f462c102820 2024-03-22T01:32:35.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:32:35.824+0000 7f4631b91700 1 -- 172.21.15.89:0/1803677199 --> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f462c062ac0 con 0x7f46140a3150 2024-03-22T01:32:36.107 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:35 smithi089 ceph-mon[35770]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 284 MiB used, 894 GiB / 894 GiB avail 2024-03-22T01:32:36.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:35 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T01:32:36.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:35 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T01:32:36.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:35 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:32:36.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:35 smithi072 ceph-mon[32414]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 284 MiB used, 894 GiB / 894 GiB avail 2024-03-22T01:32:36.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:35 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T01:32:36.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:35 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T01:32:36.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:35 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:32:36.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:35 smithi098 ceph-mon[36468]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 284 MiB used, 894 GiB / 894 GiB avail 2024-03-22T01:32:36.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:35 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T01:32:36.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:35 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T01:32:36.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:35 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:32:37.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:36 smithi089 ceph-mon[35770]: from='client.24335 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:32:37.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:36 smithi072 ceph-mon[32414]: from='client.24335 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:32:37.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:36 smithi098 ceph-mon[36468]: from='client.24335 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:32:38.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:37 smithi089 ceph-mon[35770]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-03-22T01:32:38.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:37 smithi072 ceph-mon[32414]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-03-22T01:32:38.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:37 smithi098 ceph-mon[36468]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-03-22T01:32:39.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:38 smithi089 ceph-mon[35770]: from='client.? 172.21.15.89:0/1855901455' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8741d25f-40dd-40d7-b040-5cc5e411eb98"}]: dispatch 2024-03-22T01:32:39.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:38 smithi089 ceph-mon[35770]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8741d25f-40dd-40d7-b040-5cc5e411eb98"}]: dispatch 2024-03-22T01:32:39.359 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:38 smithi089 ceph-mon[35770]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8741d25f-40dd-40d7-b040-5cc5e411eb98"}]': finished 2024-03-22T01:32:39.359 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:38 smithi089 ceph-mon[35770]: osdmap e70: 11 total, 10 up, 11 in 2024-03-22T01:32:39.359 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:38 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-22T01:32:39.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:38 smithi072 ceph-mon[32414]: from='client.? 172.21.15.89:0/1855901455' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8741d25f-40dd-40d7-b040-5cc5e411eb98"}]: dispatch 2024-03-22T01:32:39.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:38 smithi072 ceph-mon[32414]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8741d25f-40dd-40d7-b040-5cc5e411eb98"}]: dispatch 2024-03-22T01:32:39.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:38 smithi072 ceph-mon[32414]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8741d25f-40dd-40d7-b040-5cc5e411eb98"}]': finished 2024-03-22T01:32:39.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:38 smithi072 ceph-mon[32414]: osdmap e70: 11 total, 10 up, 11 in 2024-03-22T01:32:39.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:38 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-22T01:32:39.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:38 smithi098 ceph-mon[36468]: from='client.? 172.21.15.89:0/1855901455' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8741d25f-40dd-40d7-b040-5cc5e411eb98"}]: dispatch 2024-03-22T01:32:39.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:38 smithi098 ceph-mon[36468]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8741d25f-40dd-40d7-b040-5cc5e411eb98"}]: dispatch 2024-03-22T01:32:39.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:38 smithi098 ceph-mon[36468]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8741d25f-40dd-40d7-b040-5cc5e411eb98"}]': finished 2024-03-22T01:32:39.423 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:38 smithi098 ceph-mon[36468]: osdmap e70: 11 total, 10 up, 11 in 2024-03-22T01:32:39.423 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:38 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-22T01:32:40.357 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:39 smithi089 ceph-mon[35770]: from='client.? 172.21.15.89:0/3211505551' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T01:32:40.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:39 smithi089 ceph-mon[35770]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-03-22T01:32:40.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:39 smithi072 ceph-mon[32414]: from='client.? 172.21.15.89:0/3211505551' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T01:32:40.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:39 smithi072 ceph-mon[32414]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-03-22T01:32:40.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:39 smithi098 ceph-mon[36468]: from='client.? 172.21.15.89:0/3211505551' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T01:32:40.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:39 smithi098 ceph-mon[36468]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-03-22T01:32:42.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:42 smithi089 ceph-mon[35770]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-03-22T01:32:42.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:42 smithi072 ceph-mon[32414]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-03-22T01:32:42.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:42 smithi098 ceph-mon[36468]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-03-22T01:32:44.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:44 smithi089 ceph-mon[35770]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-03-22T01:32:44.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:44 smithi072 ceph-mon[32414]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-03-22T01:32:44.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:44 smithi098 ceph-mon[36468]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-03-22T01:32:46.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:46 smithi089 ceph-mon[35770]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-03-22T01:32:46.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:46 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-22T01:32:46.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:46 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:32:46.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:46 smithi072 ceph-mon[32414]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-03-22T01:32:46.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:46 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-22T01:32:46.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:46 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:32:46.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:46 smithi098 ceph-mon[36468]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-03-22T01:32:46.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:46 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-22T01:32:46.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:46 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:32:47.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:47 smithi089 ceph-mon[35770]: Deploying daemon osd.10 on smithi089 2024-03-22T01:32:47.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:47 smithi072 ceph-mon[32414]: Deploying daemon osd.10 on smithi089 2024-03-22T01:32:47.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:47 smithi098 ceph-mon[36468]: Deploying daemon osd.10 on smithi089 2024-03-22T01:32:48.668 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:48 smithi089 ceph-mon[35770]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-03-22T01:32:48.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:48 smithi072 ceph-mon[32414]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-03-22T01:32:48.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:48 smithi098 ceph-mon[36468]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-03-22T01:32:50.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:50 smithi089 ceph-mon[35770]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-03-22T01:32:50.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:50 smithi072 ceph-mon[32414]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-03-22T01:32:50.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:50 smithi098 ceph-mon[36468]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-03-22T01:32:52.734 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:52 smithi089 ceph-mon[35770]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-03-22T01:32:52.734 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:52 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:52.734 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:52 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:52.734 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:52 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:32:52.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:52 smithi072 ceph-mon[32414]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-03-22T01:32:52.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:52 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:52.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:52 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:52.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:52 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:32:52.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:52 smithi098 ceph-mon[36468]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-03-22T01:32:52.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:52 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:52.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:52 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:52.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:52 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:32:54.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:54 smithi089 ceph-mon[35770]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-03-22T01:32:54.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:54 smithi072 ceph-mon[32414]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-03-22T01:32:54.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:54 smithi098 ceph-mon[36468]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-03-22T01:32:56.291 INFO:teuthology.orchestra.run.smithi089.stdout:Created osd(s) 10 on host 'smithi089' 2024-03-22T01:32:56.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:32:56.292+0000 7f4628ff9700 1 -- 172.21.15.89:0/1803677199 <== mgr.14152 v2:172.21.15.72:6800/401403200 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f462c062ac0 con 0x7f46140a3150 2024-03-22T01:32:56.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:32:56.294+0000 7f4631b91700 1 -- 172.21.15.89:0/1803677199 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f46140a3150 msgr2=0x7f46140a5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:32:56.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:32:56.294+0000 7f4631b91700 1 --2- 172.21.15.89:0/1803677199 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f46140a3150 0x7f46140a5610 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f462c102680 tx=0x7f4620005d70 comp rx=0 tx=0).stop 2024-03-22T01:32:56.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:32:56.294+0000 7f4631b91700 1 -- 172.21.15.89:0/1803677199 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f462c102820 msgr2=0x7f462c19d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:32:56.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:32:56.294+0000 7f4631b91700 1 --2- 172.21.15.89:0/1803677199 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f462c102820 0x7f462c19d540 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f461c00b530 tx=0x7f461c00b8f0 comp rx=0 tx=0).stop 2024-03-22T01:32:56.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:32:56.294+0000 7f4631b91700 1 -- 172.21.15.89:0/1803677199 shutdown_connections 2024-03-22T01:32:56.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:32:56.294+0000 7f4631b91700 1 --2- 172.21.15.89:0/1803677199 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f46140a3150 0x7f46140a5610 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:32:56.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:32:56.294+0000 7f4631b91700 1 --2- 172.21.15.89:0/1803677199 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f462c1031c0 0x7f462c19fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:32:56.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:32:56.294+0000 7f4631b91700 1 --2- 172.21.15.89:0/1803677199 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f462c101620 0x7f462c06b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:32:56.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:32:56.294+0000 7f4631b91700 1 --2- 172.21.15.89:0/1803677199 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f462c102820 0x7f462c19d540 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:32:56.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:32:56.294+0000 7f4631b91700 1 -- 172.21.15.89:0/1803677199 >> 172.21.15.89:0/1803677199 conn(0x7f462c0fcbe0 msgr2=0x7f462c105ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:32:56.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:32:56.295+0000 7f4631b91700 1 -- 172.21.15.89:0/1803677199 shutdown_connections 2024-03-22T01:32:56.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-22T01:32:56.295+0000 7f4631b91700 1 -- 172.21.15.89:0/1803677199 wait complete. 2024-03-22T01:32:56.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:56 smithi089 ceph-mon[35770]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-03-22T01:32:56.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:56 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:56.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:56 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:56.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:56 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:32:56.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:56 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:32:56.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:56 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:56.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:56 smithi089 ceph-mon[35770]: from='osd.10 [v2:172.21.15.89:6824/885058786,v1:172.21.15.89:6825/885058786]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-22T01:32:56.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:56 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:56.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:56 smithi089 ceph-mon[35770]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-22T01:32:56.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:56 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:56.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:56 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:32:56.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:56 smithi072 ceph-mon[32414]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-03-22T01:32:56.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:56 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:56.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:56 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:56.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:56 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:32:56.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:56 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:32:56.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:56 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:56.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:56 smithi072 ceph-mon[32414]: from='osd.10 [v2:172.21.15.89:6824/885058786,v1:172.21.15.89:6825/885058786]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-22T01:32:56.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:56 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:56.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:56 smithi072 ceph-mon[32414]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-22T01:32:56.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:56 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:56.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:56 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:32:56.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:56 smithi098 ceph-mon[36468]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-03-22T01:32:56.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:56 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:56.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:56 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:56.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:56 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:32:56.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:56 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:32:56.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:56 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:56.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:56 smithi098 ceph-mon[36468]: from='osd.10 [v2:172.21.15.89:6824/885058786,v1:172.21.15.89:6825/885058786]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-22T01:32:56.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:56 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:56.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:56 smithi098 ceph-mon[36468]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-22T01:32:56.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:56 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:32:56.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:56 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:32:57.062 DEBUG:teuthology.orchestra.run.smithi089:osd.10> sudo journalctl -f -n 0 -u ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@osd.10.service 2024-03-22T01:32:57.064 INFO:tasks.cephadm:Deploying osd.11 on smithi098 with /dev/vg_nvme/lv_1... 2024-03-22T01:32:57.065 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-22T01:32:57.309 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.c/config 2024-03-22T01:32:58.521 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:58 smithi089 ceph-mon[35770]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-22T01:32:58.521 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:58 smithi089 ceph-mon[35770]: osdmap e71: 11 total, 10 up, 11 in 2024-03-22T01:32:58.521 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:58 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-22T01:32:58.521 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:58 smithi089 ceph-mon[35770]: from='osd.10 [v2:172.21.15.89:6824/885058786,v1:172.21.15.89:6825/885058786]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-03-22T01:32:58.521 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:58 smithi089 ceph-mon[35770]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-03-22T01:32:58.521 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:58 smithi089 ceph-mon[35770]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-03-22T01:32:58.595 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:58 smithi098 ceph-mon[36468]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-22T01:32:58.596 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:58 smithi098 ceph-mon[36468]: osdmap e71: 11 total, 10 up, 11 in 2024-03-22T01:32:58.596 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:58 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-22T01:32:58.596 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:58 smithi098 ceph-mon[36468]: from='osd.10 [v2:172.21.15.89:6824/885058786,v1:172.21.15.89:6825/885058786]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-03-22T01:32:58.596 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:58 smithi098 ceph-mon[36468]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-03-22T01:32:58.596 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:58 smithi098 ceph-mon[36468]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-03-22T01:32:58.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:58 smithi072 ceph-mon[32414]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-22T01:32:58.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:58 smithi072 ceph-mon[32414]: osdmap e71: 11 total, 10 up, 11 in 2024-03-22T01:32:58.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:58 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-22T01:32:58.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:58 smithi072 ceph-mon[32414]: from='osd.10 [v2:172.21.15.89:6824/885058786,v1:172.21.15.89:6825/885058786]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-03-22T01:32:58.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:58 smithi072 ceph-mon[32414]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-03-22T01:32:58.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:58 smithi072 ceph-mon[32414]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-03-22T01:32:59.269 INFO:journalctl@ceph.osd.10.smithi089.stdout:Mar 22 01:32:59 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-10[61378]: 2024-03-22T01:32:59.007+0000 7f8b1f1d1700 -1 osd.10 0 waiting for initial osdmap 2024-03-22T01:32:59.269 INFO:journalctl@ceph.osd.10.smithi089.stdout:Mar 22 01:32:59 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-10[61378]: 2024-03-22T01:32:59.022+0000 7f8b16fbc700 -1 osd.10 73 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-22T01:32:59.464 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-22T01:32:59.464 INFO:teuthology.orchestra.run.smithi098.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-22T01:32:59.464 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10+0 records in 2024-03-22T01:32:59.464 INFO:teuthology.orchestra.run.smithi098.stderr:10+0 records out 2024-03-22T01:32:59.464 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0126805 s, 827 MB/s 2024-03-22T01:32:59.464 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping successful for: 2024-03-22T01:32:59.465 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:59 smithi098 ceph-mon[36468]: purged_snaps scrub starts 2024-03-22T01:32:59.465 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:59 smithi098 ceph-mon[36468]: purged_snaps scrub ok 2024-03-22T01:32:59.465 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:59 smithi098 ceph-mon[36468]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2024-03-22T01:32:59.465 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:59 smithi098 ceph-mon[36468]: osdmap e72: 11 total, 10 up, 11 in 2024-03-22T01:32:59.465 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:59 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-22T01:32:59.465 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:59 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-22T01:32:59.465 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:59 smithi098 ceph-mon[36468]: osdmap e73: 11 total, 10 up, 11 in 2024-03-22T01:32:59.465 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:59 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-22T01:32:59.465 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:32:59 smithi098 ceph-mon[36468]: from='osd.10 ' entity='osd.10' 2024-03-22T01:32:59.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:59 smithi089 ceph-mon[35770]: purged_snaps scrub starts 2024-03-22T01:32:59.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:59 smithi089 ceph-mon[35770]: purged_snaps scrub ok 2024-03-22T01:32:59.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:59 smithi089 ceph-mon[35770]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2024-03-22T01:32:59.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:59 smithi089 ceph-mon[35770]: osdmap e72: 11 total, 10 up, 11 in 2024-03-22T01:32:59.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:59 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-22T01:32:59.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:59 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-22T01:32:59.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:59 smithi089 ceph-mon[35770]: osdmap e73: 11 total, 10 up, 11 in 2024-03-22T01:32:59.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:59 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-22T01:32:59.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:32:59 smithi089 ceph-mon[35770]: from='osd.10 ' entity='osd.10' 2024-03-22T01:32:59.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:59 smithi072 ceph-mon[32414]: purged_snaps scrub starts 2024-03-22T01:32:59.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:59 smithi072 ceph-mon[32414]: purged_snaps scrub ok 2024-03-22T01:32:59.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:59 smithi072 ceph-mon[32414]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2024-03-22T01:32:59.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:59 smithi072 ceph-mon[32414]: osdmap e72: 11 total, 10 up, 11 in 2024-03-22T01:32:59.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:59 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-22T01:32:59.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:59 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-22T01:32:59.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:59 smithi072 ceph-mon[32414]: osdmap e73: 11 total, 10 up, 11 in 2024-03-22T01:32:59.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:59 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-22T01:32:59.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:32:59 smithi072 ceph-mon[32414]: from='osd.10 ' entity='osd.10' 2024-03-22T01:33:00.176 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi098:vg_nvme/lv_1 2024-03-22T01:33:00.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:00 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-22T01:33:00.359 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:00 smithi089 ceph-mon[35770]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-03-22T01:33:00.359 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:00 smithi089 ceph-mon[35770]: osd.10 [v2:172.21.15.89:6824/885058786,v1:172.21.15.89:6825/885058786] boot 2024-03-22T01:33:00.359 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:00 smithi089 ceph-mon[35770]: osdmap e74: 11 total, 11 up, 11 in 2024-03-22T01:33:00.359 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:00 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-22T01:33:00.359 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:00 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:00.359 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:00 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:00.359 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:00 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:33:00.359 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:00 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:33:00.359 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:00 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:33:00.359 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:00 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:33:00.359 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:00 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:00.359 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:00 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:33:00.360 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:00 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:33:00.360 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:00 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:00.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:00 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-22T01:33:00.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:00 smithi098 ceph-mon[36468]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-03-22T01:33:00.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:00 smithi098 ceph-mon[36468]: osd.10 [v2:172.21.15.89:6824/885058786,v1:172.21.15.89:6825/885058786] boot 2024-03-22T01:33:00.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:00 smithi098 ceph-mon[36468]: osdmap e74: 11 total, 11 up, 11 in 2024-03-22T01:33:00.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:00 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-22T01:33:00.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:00 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:00.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:00 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:00.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:00 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:33:00.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:00 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:33:00.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:00 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:33:00.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:00 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:33:00.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:00 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:00.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:00 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:33:00.423 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:00 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:33:00.423 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:00 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:00.439 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.c/config 2024-03-22T01:33:00.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:00 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-22T01:33:00.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:00 smithi072 ceph-mon[32414]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-03-22T01:33:00.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:00 smithi072 ceph-mon[32414]: osd.10 [v2:172.21.15.89:6824/885058786,v1:172.21.15.89:6825/885058786] boot 2024-03-22T01:33:00.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:00 smithi072 ceph-mon[32414]: osdmap e74: 11 total, 11 up, 11 in 2024-03-22T01:33:00.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:00 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-22T01:33:00.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:00 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:00.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:00 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:00.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:00 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:33:00.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:00 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:33:00.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:00 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:33:00.624 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:00 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:33:00.624 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:00 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:00.624 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:00 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:33:00.624 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:00 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:33:00.624 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:00 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:01.587 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:01 smithi098 ceph-mon[36468]: Detected new or changed devices on smithi089 2024-03-22T01:33:01.588 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:01 smithi098 ceph-mon[36468]: Adjusting osd_memory_target on smithi089 to 4270M 2024-03-22T01:33:01.588 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:01 smithi098 ceph-mon[36468]: osdmap e75: 11 total, 11 up, 11 in 2024-03-22T01:33:01.607 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:01 smithi089 ceph-mon[35770]: Detected new or changed devices on smithi089 2024-03-22T01:33:01.607 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:01 smithi089 ceph-mon[35770]: Adjusting osd_memory_target on smithi089 to 4270M 2024-03-22T01:33:01.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:01 smithi089 ceph-mon[35770]: osdmap e75: 11 total, 11 up, 11 in 2024-03-22T01:33:01.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:01 smithi072 ceph-mon[32414]: Detected new or changed devices on smithi089 2024-03-22T01:33:01.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:01 smithi072 ceph-mon[32414]: Adjusting osd_memory_target on smithi089 to 4270M 2024-03-22T01:33:01.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:01 smithi072 ceph-mon[32414]: osdmap e75: 11 total, 11 up, 11 in 2024-03-22T01:33:02.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:02.120+0000 7f1e82505700 1 -- 172.21.15.98:0/1815352054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e7c1018c0 msgr2=0x7f1e7c101d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:02.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:02.120+0000 7f1e82505700 1 --2- 172.21.15.98:0/1815352054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e7c1018c0 0x7f1e7c101d20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1e700099c0 tx=0x7f1e70009cd0 comp rx=0 tx=0).stop 2024-03-22T01:33:02.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:02.120+0000 7f1e82505700 1 -- 172.21.15.98:0/1815352054 shutdown_connections 2024-03-22T01:33:02.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:02.120+0000 7f1e82505700 1 --2- 172.21.15.98:0/1815352054 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1e7c102260 0x7f1e7c109b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:02.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:02.120+0000 7f1e82505700 1 --2- 172.21.15.98:0/1815352054 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e7c102b80 0x7f1e7c102fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:02.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:02.120+0000 7f1e82505700 1 --2- 172.21.15.98:0/1815352054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e7c1018c0 0x7f1e7c101d20 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:02.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:02.120+0000 7f1e82505700 1 -- 172.21.15.98:0/1815352054 >> 172.21.15.98:0/1815352054 conn(0x7f1e7c0fc310 msgr2=0x7f1e7c0fe750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:33:02.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:02.120+0000 7f1e82505700 1 -- 172.21.15.98:0/1815352054 shutdown_connections 2024-03-22T01:33:02.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:02.120+0000 7f1e82505700 1 -- 172.21.15.98:0/1815352054 wait complete. 2024-03-22T01:33:02.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:02.121+0000 7f1e82505700 1 Processor -- start 2024-03-22T01:33:02.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:02.121+0000 7f1e82505700 1 -- start start 2024-03-22T01:33:02.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:02.121+0000 7f1e82505700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e7c1018c0 0x7f1e7c197830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:02.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:02.121+0000 7f1e82505700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1e7c102260 0x7f1e7c197d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:02.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:02.121+0000 7f1e82505700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e7c102b80 0x7f1e7c19edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:02.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:02.121+0000 7f1e82505700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f1e7c10d250 con 0x7f1e7c102260 2024-03-22T01:33:02.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:02.121+0000 7f1e82505700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1e7c10d0d0 con 0x7f1e7c102b80 2024-03-22T01:33:02.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:02.121+0000 7f1e82505700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1e7c10d3d0 con 0x7f1e7c1018c0 2024-03-22T01:33:02.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:02.121+0000 7f1e7bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e7c1018c0 0x7f1e7c197830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:02.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:02.121+0000 7f1e7bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e7c1018c0 0x7f1e7c197830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37644/0 (socket says 172.21.15.98:37644) 2024-03-22T01:33:02.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:02.121+0000 7f1e7bfff700 1 -- 172.21.15.98:0/2992023244 learned_addr learned my addr 172.21.15.98:0/2992023244 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-22T01:33:02.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:02.122+0000 7f1e7b7fe700 1 --2- 172.21.15.98:0/2992023244 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1e7c102260 0x7f1e7c197d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:02.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:02.122+0000 7f1e7bfff700 1 -- 172.21.15.98:0/2992023244 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e7c102b80 msgr2=0x7f1e7c19edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:02.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:02.122+0000 7f1e7bfff700 1 --2- 172.21.15.98:0/2992023244 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e7c102b80 0x7f1e7c19edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:02.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:02.122+0000 7f1e7bfff700 1 -- 172.21.15.98:0/2992023244 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1e7c102260 msgr2=0x7f1e7c197d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:02.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:02.122+0000 7f1e7bfff700 1 --2- 172.21.15.98:0/2992023244 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1e7c102260 0x7f1e7c197d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:02.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:02.122+0000 7f1e7bfff700 1 -- 172.21.15.98:0/2992023244 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e7c19f490 con 0x7f1e7c1018c0 2024-03-22T01:33:02.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:02.122+0000 7f1e7b7fe700 1 --2- 172.21.15.98:0/2992023244 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1e7c102260 0x7f1e7c197d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T01:33:02.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:02.122+0000 7f1e7bfff700 1 --2- 172.21.15.98:0/2992023244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e7c1018c0 0x7f1e7c197830 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c00b530 tx=0x7f1e6c00b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:33:02.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:02.122+0000 7f1e797fa700 1 -- 172.21.15.98:0/2992023244 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e6c015020 con 0x7f1e7c1018c0 2024-03-22T01:33:02.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:02.122+0000 7f1e82505700 1 -- 172.21.15.98:0/2992023244 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e7c19f780 con 0x7f1e7c1018c0 2024-03-22T01:33:02.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:02.122+0000 7f1e797fa700 1 -- 172.21.15.98:0/2992023244 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e6c011d10 con 0x7f1e7c1018c0 2024-03-22T01:33:02.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:02.122+0000 7f1e797fa700 1 -- 172.21.15.98:0/2992023244 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e6c00f2e0 con 0x7f1e7c1018c0 2024-03-22T01:33:02.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:02.122+0000 7f1e82505700 1 -- 172.21.15.98:0/2992023244 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e7c19fd40 con 0x7f1e7c1018c0 2024-03-22T01:33:02.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:02.123+0000 7f1e82505700 1 -- 172.21.15.98:0/2992023244 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e7c069340 con 0x7f1e7c1018c0 2024-03-22T01:33:02.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:02.124+0000 7f1e797fa700 1 -- 172.21.15.98:0/2992023244 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f1e6c018020 con 0x7f1e7c1018c0 2024-03-22T01:33:02.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:02.124+0000 7f1e797fa700 1 --2- 172.21.15.98:0/2992023244 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f1e640a2fb0 0x7f1e640a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:02.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:02.125+0000 7f1e797fa700 1 -- 172.21.15.98:0/2992023244 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 6723+0+0 (secure 0 0 0) 0x7f1e6c0ce5e0 con 0x7f1e7c1018c0 2024-03-22T01:33:02.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:02.125+0000 7f1e7b7fe700 1 --2- 172.21.15.98:0/2992023244 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f1e640a2fb0 0x7f1e640a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:02.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:02.125+0000 7f1e7b7fe700 1 --2- 172.21.15.98:0/2992023244 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f1e640a2fb0 0x7f1e640a5470 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1e70005cd0 tx=0x7f1e70005c20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:33:02.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:02.129+0000 7f1e797fa700 1 -- 172.21.15.98:0/2992023244 <== mon.1 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1e6c01f020 con 0x7f1e7c1018c0 2024-03-22T01:33:02.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:02.335+0000 7f1e82505700 1 -- 172.21.15.98:0/2992023244 --> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f1e7c062ac0 con 0x7f1e640a2fb0 2024-03-22T01:33:02.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:02 smithi089 ceph-mon[35770]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-03-22T01:33:02.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:02 smithi089 ceph-mon[35770]: osdmap e76: 11 total, 11 up, 11 in 2024-03-22T01:33:02.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:02 smithi072 ceph-mon[32414]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-03-22T01:33:02.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:02 smithi072 ceph-mon[32414]: osdmap e76: 11 total, 11 up, 11 in 2024-03-22T01:33:02.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:02 smithi098 ceph-mon[36468]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-03-22T01:33:02.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:02 smithi098 ceph-mon[36468]: osdmap e76: 11 total, 11 up, 11 in 2024-03-22T01:33:03.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:03 smithi089 ceph-mon[35770]: from='client.24376 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:33:03.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:03 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T01:33:03.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:03 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T01:33:03.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:03 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:33:03.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:03 smithi089 ceph-mon[35770]: osdmap e77: 11 total, 11 up, 11 in 2024-03-22T01:33:03.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:03 smithi072 ceph-mon[32414]: from='client.24376 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:33:03.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:03 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T01:33:03.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:03 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T01:33:03.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:03 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:33:03.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:03 smithi072 ceph-mon[32414]: osdmap e77: 11 total, 11 up, 11 in 2024-03-22T01:33:03.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:03 smithi098 ceph-mon[36468]: from='client.24376 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:33:03.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:03 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T01:33:03.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:03 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T01:33:03.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:03 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:33:03.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:03 smithi098 ceph-mon[36468]: osdmap e77: 11 total, 11 up, 11 in 2024-03-22T01:33:04.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:04 smithi089 ceph-mon[35770]: pgmap v241: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-22T01:33:04.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:04 smithi089 ceph-mon[35770]: osdmap e78: 11 total, 11 up, 11 in 2024-03-22T01:33:04.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:04 smithi072 ceph-mon[32414]: pgmap v241: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-22T01:33:04.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:04 smithi072 ceph-mon[32414]: osdmap e78: 11 total, 11 up, 11 in 2024-03-22T01:33:04.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:04 smithi098 ceph-mon[36468]: pgmap v241: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-22T01:33:04.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:04 smithi098 ceph-mon[36468]: osdmap e78: 11 total, 11 up, 11 in 2024-03-22T01:33:05.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:05 smithi089 ceph-mon[35770]: from='client.? 172.21.15.98:0/2109809614' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8d02e957-eda4-4184-b5c6-5eb3b8b38060"}]: dispatch 2024-03-22T01:33:05.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:05 smithi089 ceph-mon[35770]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8d02e957-eda4-4184-b5c6-5eb3b8b38060"}]: dispatch 2024-03-22T01:33:05.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:05 smithi089 ceph-mon[35770]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8d02e957-eda4-4184-b5c6-5eb3b8b38060"}]': finished 2024-03-22T01:33:05.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:05 smithi089 ceph-mon[35770]: osdmap e79: 12 total, 11 up, 12 in 2024-03-22T01:33:05.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:05 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-22T01:33:05.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:05 smithi072 ceph-mon[32414]: from='client.? 172.21.15.98:0/2109809614' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8d02e957-eda4-4184-b5c6-5eb3b8b38060"}]: dispatch 2024-03-22T01:33:05.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:05 smithi072 ceph-mon[32414]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8d02e957-eda4-4184-b5c6-5eb3b8b38060"}]: dispatch 2024-03-22T01:33:05.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:05 smithi072 ceph-mon[32414]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8d02e957-eda4-4184-b5c6-5eb3b8b38060"}]': finished 2024-03-22T01:33:05.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:05 smithi072 ceph-mon[32414]: osdmap e79: 12 total, 11 up, 12 in 2024-03-22T01:33:05.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:05 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-22T01:33:05.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:05 smithi098 ceph-mon[36468]: from='client.? 172.21.15.98:0/2109809614' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8d02e957-eda4-4184-b5c6-5eb3b8b38060"}]: dispatch 2024-03-22T01:33:05.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:05 smithi098 ceph-mon[36468]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8d02e957-eda4-4184-b5c6-5eb3b8b38060"}]: dispatch 2024-03-22T01:33:05.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:05 smithi098 ceph-mon[36468]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8d02e957-eda4-4184-b5c6-5eb3b8b38060"}]': finished 2024-03-22T01:33:05.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:05 smithi098 ceph-mon[36468]: osdmap e79: 12 total, 11 up, 12 in 2024-03-22T01:33:05.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:05 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-22T01:33:06.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:06 smithi089 ceph-mon[35770]: pgmap v244: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-22T01:33:06.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:06 smithi089 ceph-mon[35770]: from='client.? 172.21.15.98:0/3645887779' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T01:33:06.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:06 smithi072 ceph-mon[32414]: pgmap v244: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-22T01:33:06.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:06 smithi072 ceph-mon[32414]: from='client.? 172.21.15.98:0/3645887779' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T01:33:06.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:06 smithi098 ceph-mon[36468]: pgmap v244: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-22T01:33:06.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:06 smithi098 ceph-mon[36468]: from='client.? 172.21.15.98:0/3645887779' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T01:33:08.713 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:08 smithi098 ceph-mon[36468]: pgmap v245: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-03-22T01:33:08.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:08 smithi089 ceph-mon[35770]: pgmap v245: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-03-22T01:33:08.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:08 smithi072 ceph-mon[32414]: pgmap v245: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-03-22T01:33:10.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:10 smithi098 ceph-mon[36468]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-22T01:33:10.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:10 smithi089 ceph-mon[35770]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-22T01:33:10.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:10 smithi072 ceph-mon[32414]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-22T01:33:12.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:12 smithi089 ceph-mon[35770]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-22T01:33:12.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:12 smithi072 ceph-mon[32414]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-22T01:33:12.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:12 smithi098 ceph-mon[36468]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-22T01:33:13.808 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:13 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-22T01:33:13.808 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:13 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:33:13.808 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:13 smithi098 ceph-mon[36468]: Deploying daemon osd.11 on smithi098 2024-03-22T01:33:13.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:13 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-22T01:33:13.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:13 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:33:13.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:13 smithi089 ceph-mon[35770]: Deploying daemon osd.11 on smithi098 2024-03-22T01:33:13.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:13 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-22T01:33:13.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:13 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:33:13.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:13 smithi072 ceph-mon[32414]: Deploying daemon osd.11 on smithi098 2024-03-22T01:33:14.725 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:14 smithi098 ceph-mon[36468]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-22T01:33:14.773 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:14 smithi072 ceph-mon[32414]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-22T01:33:14.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:14 smithi089 ceph-mon[35770]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-22T01:33:16.765 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:16 smithi098 ceph-mon[36468]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-22T01:33:16.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:16 smithi089 ceph-mon[35770]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-22T01:33:16.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:16 smithi072 ceph-mon[32414]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-22T01:33:18.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:18 smithi089 ceph-mon[35770]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-22T01:33:18.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:18 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:18.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:18 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:18.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:18 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:33:18.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:18 smithi072 ceph-mon[32414]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-22T01:33:18.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:18 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:18.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:18 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:18.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:18 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:33:18.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:18 smithi098 ceph-mon[36468]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-22T01:33:18.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:18 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:18.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:18 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:18.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:18 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:33:20.594 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:20 smithi098 ceph-mon[36468]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-22T01:33:20.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:20 smithi089 ceph-mon[35770]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-22T01:33:20.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:20 smithi072 ceph-mon[32414]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-22T01:33:22.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:22.401+0000 7f1e797fa700 1 -- 172.21.15.98:0/2992023244 <== mgr.14152 v2:172.21.15.72:6800/401403200 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f1e7c062ac0 con 0x7f1e640a2fb0 2024-03-22T01:33:22.401 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 11 on host 'smithi098' 2024-03-22T01:33:22.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:22.404+0000 7f1e82505700 1 -- 172.21.15.98:0/2992023244 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f1e640a2fb0 msgr2=0x7f1e640a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:22.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:22.404+0000 7f1e82505700 1 --2- 172.21.15.98:0/2992023244 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f1e640a2fb0 0x7f1e640a5470 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1e70005cd0 tx=0x7f1e70005c20 comp rx=0 tx=0).stop 2024-03-22T01:33:22.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:22.404+0000 7f1e82505700 1 -- 172.21.15.98:0/2992023244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e7c1018c0 msgr2=0x7f1e7c197830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:22.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:22.404+0000 7f1e82505700 1 --2- 172.21.15.98:0/2992023244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e7c1018c0 0x7f1e7c197830 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c00b530 tx=0x7f1e6c00b8f0 comp rx=0 tx=0).stop 2024-03-22T01:33:22.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:22.404+0000 7f1e82505700 1 -- 172.21.15.98:0/2992023244 shutdown_connections 2024-03-22T01:33:22.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:22.404+0000 7f1e82505700 1 --2- 172.21.15.98:0/2992023244 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f1e640a2fb0 0x7f1e640a5470 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:22.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:22.404+0000 7f1e82505700 1 --2- 172.21.15.98:0/2992023244 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e7c102b80 0x7f1e7c19edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:22.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:22.404+0000 7f1e82505700 1 --2- 172.21.15.98:0/2992023244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e7c1018c0 0x7f1e7c197830 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:22.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:22.404+0000 7f1e82505700 1 --2- 172.21.15.98:0/2992023244 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1e7c102260 0x7f1e7c197d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:22.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:22.404+0000 7f1e82505700 1 -- 172.21.15.98:0/2992023244 >> 172.21.15.98:0/2992023244 conn(0x7f1e7c0fc310 msgr2=0x7f1e7c105b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:33:22.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:22.404+0000 7f1e82505700 1 -- 172.21.15.98:0/2992023244 shutdown_connections 2024-03-22T01:33:22.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:22.404+0000 7f1e82505700 1 -- 172.21.15.98:0/2992023244 wait complete. 2024-03-22T01:33:22.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:22 smithi098 ceph-mon[36468]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-22T01:33:22.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:22 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:22.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:22 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:22.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:22 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:33:22.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:22 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:33:22.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:22 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:22.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:22 smithi098 ceph-mon[36468]: from='osd.11 [v2:172.21.15.98:6824/1517095838,v1:172.21.15.98:6825/1517095838]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-22T01:33:22.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:22 smithi098 ceph-mon[36468]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-22T01:33:22.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:22 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:22.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:22 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:22.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:22 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:33:22.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:22 smithi089 ceph-mon[35770]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-22T01:33:22.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:22 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:22.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:22 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:22.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:22 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:33:22.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:22 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:33:22.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:22 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:22.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:22 smithi089 ceph-mon[35770]: from='osd.11 [v2:172.21.15.98:6824/1517095838,v1:172.21.15.98:6825/1517095838]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-22T01:33:22.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:22 smithi089 ceph-mon[35770]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-22T01:33:22.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:22 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:22.859 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:22 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:22.859 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:22 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:33:22.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:22 smithi072 ceph-mon[32414]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-22T01:33:22.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:22 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:22.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:22 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:22.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:22 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:33:22.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:22 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:33:22.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:22 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:22.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:22 smithi072 ceph-mon[32414]: from='osd.11 [v2:172.21.15.98:6824/1517095838,v1:172.21.15.98:6825/1517095838]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-22T01:33:22.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:22 smithi072 ceph-mon[32414]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-22T01:33:22.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:22 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:22.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:22 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:22.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:22 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:33:23.255 DEBUG:teuthology.orchestra.run.smithi098:osd.11> sudo journalctl -f -n 0 -u ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@osd.11.service 2024-03-22T01:33:23.258 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-03-22T01:33:23.258 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph osd stat -f json 2024-03-22T01:33:23.504 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:33:23.823 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:23 smithi098 ceph-mon[36468]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-22T01:33:23.823 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:23 smithi098 ceph-mon[36468]: osdmap e80: 12 total, 11 up, 12 in 2024-03-22T01:33:23.823 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:23 smithi098 ceph-mon[36468]: from='osd.11 [v2:172.21.15.98:6824/1517095838,v1:172.21.15.98:6825/1517095838]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-22T01:33:23.823 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:23 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-22T01:33:23.823 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:23 smithi098 ceph-mon[36468]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-22T01:33:23.855 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:23 smithi072 ceph-mon[32414]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-22T01:33:23.855 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:23 smithi072 ceph-mon[32414]: osdmap e80: 12 total, 11 up, 12 in 2024-03-22T01:33:23.855 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:23 smithi072 ceph-mon[32414]: from='osd.11 [v2:172.21.15.98:6824/1517095838,v1:172.21.15.98:6825/1517095838]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-22T01:33:23.856 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:23 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-22T01:33:23.856 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:23 smithi072 ceph-mon[32414]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-22T01:33:23.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:23 smithi089 ceph-mon[35770]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-22T01:33:23.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:23 smithi089 ceph-mon[35770]: osdmap e80: 12 total, 11 up, 12 in 2024-03-22T01:33:23.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:23 smithi089 ceph-mon[35770]: from='osd.11 [v2:172.21.15.98:6824/1517095838,v1:172.21.15.98:6825/1517095838]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-22T01:33:23.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:23 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-22T01:33:23.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:23 smithi089 ceph-mon[35770]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-22T01:33:24.503 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:33:24 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:33:24.221+0000 7f6445fd8700 -1 osd.11 0 waiting for initial osdmap 2024-03-22T01:33:24.503 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:33:24 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:33:24.234+0000 7f643ddc3700 -1 osd.11 81 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-22T01:33:24.806 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:24 smithi098 ceph-mon[36468]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-22T01:33:24.806 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:24 smithi098 ceph-mon[36468]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-22T01:33:24.806 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:24 smithi098 ceph-mon[36468]: osdmap e81: 12 total, 11 up, 12 in 2024-03-22T01:33:24.806 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:24 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-22T01:33:24.806 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:24 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-22T01:33:24.806 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:24 smithi098 ceph-mon[36468]: from='osd.11 ' entity='osd.11' 2024-03-22T01:33:24.841 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:24 smithi072 ceph-mon[32414]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-22T01:33:24.841 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:24 smithi072 ceph-mon[32414]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-22T01:33:24.841 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:24 smithi072 ceph-mon[32414]: osdmap e81: 12 total, 11 up, 12 in 2024-03-22T01:33:24.841 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:24 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-22T01:33:24.841 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:24 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-22T01:33:24.841 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:24 smithi072 ceph-mon[32414]: from='osd.11 ' entity='osd.11' 2024-03-22T01:33:24.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:24 smithi089 ceph-mon[35770]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-22T01:33:24.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:24 smithi089 ceph-mon[35770]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-22T01:33:24.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:24 smithi089 ceph-mon[35770]: osdmap e81: 12 total, 11 up, 12 in 2024-03-22T01:33:24.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:24 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-22T01:33:24.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:24 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-22T01:33:24.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:24 smithi089 ceph-mon[35770]: from='osd.11 ' entity='osd.11' 2024-03-22T01:33:25.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:25.193+0000 7fdd69665700 1 -- 172.21.15.72:0/130852587 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd64102b40 msgr2=0x7fdd64102f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:25.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:25.193+0000 7fdd69665700 1 --2- 172.21.15.72:0/130852587 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd64102b40 0x7fdd64102f60 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fdd580099c0 tx=0x7fdd58009cd0 comp rx=0 tx=0).stop 2024-03-22T01:33:25.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:25.193+0000 7fdd69665700 1 -- 172.21.15.72:0/130852587 shutdown_connections 2024-03-22T01:33:25.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:25.193+0000 7fdd69665700 1 --2- 172.21.15.72:0/130852587 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd64102220 0x7fdd64109ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:25.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:25.193+0000 7fdd69665700 1 --2- 172.21.15.72:0/130852587 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd64102b40 0x7fdd64102f60 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:25.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:25.193+0000 7fdd69665700 1 --2- 172.21.15.72:0/130852587 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd64101880 0x7fdd64101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:25.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:25.193+0000 7fdd69665700 1 -- 172.21.15.72:0/130852587 >> 172.21.15.72:0/130852587 conn(0x7fdd640fc2f0 msgr2=0x7fdd640fe710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:33:25.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:25.193+0000 7fdd69665700 1 -- 172.21.15.72:0/130852587 shutdown_connections 2024-03-22T01:33:25.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:25.193+0000 7fdd69665700 1 -- 172.21.15.72:0/130852587 wait complete. 2024-03-22T01:33:25.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:25.193+0000 7fdd69665700 1 Processor -- start 2024-03-22T01:33:25.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:25.194+0000 7fdd69665700 1 -- start start 2024-03-22T01:33:25.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:25.194+0000 7fdd69665700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd64101880 0x7fdd641012d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:25.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:25.194+0000 7fdd69665700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd64102220 0x7fdd6419d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:25.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:25.194+0000 7fdd69665700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd64102b40 0x7fdd6419fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:25.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:25.194+0000 7fdd69665700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fdd6410d170 con 0x7fdd64101880 2024-03-22T01:33:25.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:25.194+0000 7fdd69665700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fdd6410cff0 con 0x7fdd64102220 2024-03-22T01:33:25.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:25.194+0000 7fdd69665700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdd6410d2f0 con 0x7fdd64102b40 2024-03-22T01:33:25.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:25.194+0000 7fdd62ffd700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd64101880 0x7fdd641012d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:25.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:25.194+0000 7fdd62ffd700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd64101880 0x7fdd641012d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:35232/0 (socket says 172.21.15.72:35232) 2024-03-22T01:33:25.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:25.194+0000 7fdd62ffd700 1 -- 172.21.15.72:0/1249221543 learned_addr learned my addr 172.21.15.72:0/1249221543 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:33:25.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:25.194+0000 7fdd637fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd64102b40 0x7fdd6419fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:25.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:25.194+0000 7fdd627fc700 1 --2- 172.21.15.72:0/1249221543 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd64102220 0x7fdd6419d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:25.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:25.194+0000 7fdd62ffd700 1 -- 172.21.15.72:0/1249221543 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd64102b40 msgr2=0x7fdd6419fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:25.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:25.194+0000 7fdd62ffd700 1 --2- 172.21.15.72:0/1249221543 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd64102b40 0x7fdd6419fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:25.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:25.194+0000 7fdd62ffd700 1 -- 172.21.15.72:0/1249221543 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd64102220 msgr2=0x7fdd6419d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:25.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:25.194+0000 7fdd62ffd700 1 --2- 172.21.15.72:0/1249221543 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd64102220 0x7fdd6419d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:25.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:25.194+0000 7fdd62ffd700 1 -- 172.21.15.72:0/1249221543 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd641a04c0 con 0x7fdd64101880 2024-03-22T01:33:25.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:25.195+0000 7fdd62ffd700 1 --2- 172.21.15.72:0/1249221543 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd64101880 0x7fdd641012d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fdd5800c010 tx=0x7fdd58009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:33:25.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:25.195+0000 7fdd4bfff700 1 -- 172.21.15.72:0/1249221543 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd58026070 con 0x7fdd64101880 2024-03-22T01:33:25.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:25.195+0000 7fdd69665700 1 -- 172.21.15.72:0/1249221543 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd641a0750 con 0x7fdd64101880 2024-03-22T01:33:25.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:25.195+0000 7fdd4bfff700 1 -- 172.21.15.72:0/1249221543 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd58004150 con 0x7fdd64101880 2024-03-22T01:33:25.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:25.195+0000 7fdd4bfff700 1 -- 172.21.15.72:0/1249221543 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd58021410 con 0x7fdd64101880 2024-03-22T01:33:25.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:25.195+0000 7fdd69665700 1 -- 172.21.15.72:0/1249221543 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd641a0c80 con 0x7fdd64101880 2024-03-22T01:33:25.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:25.196+0000 7fdd69665700 1 -- 172.21.15.72:0/1249221543 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd64069340 con 0x7fdd64101880 2024-03-22T01:33:25.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:25.197+0000 7fdd4bfff700 1 -- 172.21.15.72:0/1249221543 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7fdd58029070 con 0x7fdd64101880 2024-03-22T01:33:25.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:25.197+0000 7fdd4bfff700 1 --2- 172.21.15.72:0/1249221543 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fdd4c0a2fb0 0x7fdd4c0a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:25.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:25.197+0000 7fdd627fc700 1 --2- 172.21.15.72:0/1249221543 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fdd4c0a2fb0 0x7fdd4c0a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:25.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:25.197+0000 7fdd4bfff700 1 -- 172.21.15.72:0/1249221543 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 6847+0+0 (secure 0 0 0) 0x7fdd580cf8c0 con 0x7fdd64101880 2024-03-22T01:33:25.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:25.198+0000 7fdd627fc700 1 --2- 172.21.15.72:0/1249221543 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fdd4c0a2fb0 0x7fdd4c0a5470 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fdd54005ea0 tx=0x7fdd54005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:33:25.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:25.202+0000 7fdd4bfff700 1 -- 172.21.15.72:0/1249221543 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fdd5805d740 con 0x7fdd64101880 2024-03-22T01:33:25.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:25.405+0000 7fdd69665700 1 -- 172.21.15.72:0/1249221543 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7fdd6402dd70 con 0x7fdd64101880 2024-03-22T01:33:25.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:25.405+0000 7fdd4bfff700 1 -- 172.21.15.72:0/1249221543 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v82) ==== 74+0+133 (secure 0 0 0) 0x7fdd58098020 con 0x7fdd64101880 2024-03-22T01:33:25.405 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:33:25.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:25.407+0000 7fdd69665700 1 -- 172.21.15.72:0/1249221543 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fdd4c0a2fb0 msgr2=0x7fdd4c0a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:25.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:25.407+0000 7fdd69665700 1 --2- 172.21.15.72:0/1249221543 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fdd4c0a2fb0 0x7fdd4c0a5470 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fdd54005ea0 tx=0x7fdd54005e30 comp rx=0 tx=0).stop 2024-03-22T01:33:25.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:25.407+0000 7fdd69665700 1 -- 172.21.15.72:0/1249221543 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd64101880 msgr2=0x7fdd641012d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:25.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:25.407+0000 7fdd69665700 1 --2- 172.21.15.72:0/1249221543 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd64101880 0x7fdd641012d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fdd5800c010 tx=0x7fdd58009580 comp rx=0 tx=0).stop 2024-03-22T01:33:25.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:25.408+0000 7fdd69665700 1 -- 172.21.15.72:0/1249221543 shutdown_connections 2024-03-22T01:33:25.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:25.408+0000 7fdd69665700 1 --2- 172.21.15.72:0/1249221543 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fdd4c0a2fb0 0x7fdd4c0a5470 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:25.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:25.408+0000 7fdd69665700 1 --2- 172.21.15.72:0/1249221543 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd64102b40 0x7fdd6419fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:25.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:25.408+0000 7fdd69665700 1 --2- 172.21.15.72:0/1249221543 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd64101880 0x7fdd641012d0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:25.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:25.408+0000 7fdd69665700 1 --2- 172.21.15.72:0/1249221543 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd64102220 0x7fdd6419d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:25.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:25.408+0000 7fdd69665700 1 -- 172.21.15.72:0/1249221543 >> 172.21.15.72:0/1249221543 conn(0x7fdd640fc2f0 msgr2=0x7fdd64105b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:33:25.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:25.408+0000 7fdd69665700 1 -- 172.21.15.72:0/1249221543 shutdown_connections 2024-03-22T01:33:25.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:25.408+0000 7fdd69665700 1 -- 172.21.15.72:0/1249221543 wait complete. 2024-03-22T01:33:25.726 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:25 smithi072 ceph-mon[32414]: purged_snaps scrub starts 2024-03-22T01:33:25.726 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:25 smithi072 ceph-mon[32414]: purged_snaps scrub ok 2024-03-22T01:33:25.726 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:25 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-22T01:33:25.726 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:25 smithi072 ceph-mon[32414]: osd.11 [v2:172.21.15.98:6824/1517095838,v1:172.21.15.98:6825/1517095838] boot 2024-03-22T01:33:25.726 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:25 smithi072 ceph-mon[32414]: osdmap e82: 12 total, 12 up, 12 in 2024-03-22T01:33:25.726 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:25 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-22T01:33:25.726 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:25 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/1249221543' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-22T01:33:25.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:25 smithi089 ceph-mon[35770]: purged_snaps scrub starts 2024-03-22T01:33:25.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:25 smithi089 ceph-mon[35770]: purged_snaps scrub ok 2024-03-22T01:33:25.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:25 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-22T01:33:25.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:25 smithi089 ceph-mon[35770]: osd.11 [v2:172.21.15.98:6824/1517095838,v1:172.21.15.98:6825/1517095838] boot 2024-03-22T01:33:25.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:25 smithi089 ceph-mon[35770]: osdmap e82: 12 total, 12 up, 12 in 2024-03-22T01:33:25.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:25 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-22T01:33:25.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:25 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/1249221543' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-22T01:33:25.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:25 smithi098 ceph-mon[36468]: purged_snaps scrub starts 2024-03-22T01:33:25.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:25 smithi098 ceph-mon[36468]: purged_snaps scrub ok 2024-03-22T01:33:25.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:25 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-22T01:33:25.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:25 smithi098 ceph-mon[36468]: osd.11 [v2:172.21.15.98:6824/1517095838,v1:172.21.15.98:6825/1517095838] boot 2024-03-22T01:33:25.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:25 smithi098 ceph-mon[36468]: osdmap e82: 12 total, 12 up, 12 in 2024-03-22T01:33:25.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:25 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-22T01:33:25.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:25 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/1249221543' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-22T01:33:26.181 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":82,"num_osds":12,"num_up_osds":12,"osd_up_since":1711071205,"num_in_osds":12,"osd_in_since":1711071185,"num_remapped_pgs":0} 2024-03-22T01:33:26.182 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-22T01:33:26.421 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:33:26.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:26 smithi098 ceph-mon[36468]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-22T01:33:26.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:26 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:26.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:26 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:26.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:26 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:33:26.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:26 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:33:26.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:26 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:33:26.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:26 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:33:26.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:26 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:26.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:26 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:33:26.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:26 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:33:26.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:26 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:26.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:26 smithi089 ceph-mon[35770]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-22T01:33:26.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:26 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:26.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:26 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:26.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:26 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:33:26.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:26 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:33:26.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:26 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:33:26.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:26 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:33:26.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:26 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:26.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:26 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:33:26.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:26 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:33:26.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:26 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:26.866 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:26 smithi072 ceph-mon[32414]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-22T01:33:26.866 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:26 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:26.866 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:26 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:26.866 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:26 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:33:26.866 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:26 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:33:26.866 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:26 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:33:26.866 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:26 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:33:26.866 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:26 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:26.866 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:26 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:33:26.866 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:26 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:33:26.867 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:26 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:27.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:27 smithi089 ceph-mon[35770]: Detected new or changed devices on smithi098 2024-03-22T01:33:27.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:27 smithi089 ceph-mon[35770]: Adjusting osd_memory_target on smithi098 to 4270M 2024-03-22T01:33:27.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:27 smithi089 ceph-mon[35770]: osdmap e83: 12 total, 12 up, 12 in 2024-03-22T01:33:27.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:27 smithi072 ceph-mon[32414]: Detected new or changed devices on smithi098 2024-03-22T01:33:27.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:27 smithi072 ceph-mon[32414]: Adjusting osd_memory_target on smithi098 to 4270M 2024-03-22T01:33:27.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:27 smithi072 ceph-mon[32414]: osdmap e83: 12 total, 12 up, 12 in 2024-03-22T01:33:27.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:27 smithi098 ceph-mon[36468]: Detected new or changed devices on smithi098 2024-03-22T01:33:27.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:27 smithi098 ceph-mon[36468]: Adjusting osd_memory_target on smithi098 to 4270M 2024-03-22T01:33:27.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:27 smithi098 ceph-mon[36468]: osdmap e83: 12 total, 12 up, 12 in 2024-03-22T01:33:28.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:28.086+0000 7fc792c36700 1 -- 172.21.15.72:0/2858147997 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc78c108660 msgr2=0x7fc78c108a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:28.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:28.086+0000 7fc792c36700 1 --2- 172.21.15.72:0/2858147997 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc78c108660 0x7fc78c108a40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc78400b0a0 tx=0x7fc78400b3b0 comp rx=0 tx=0).stop 2024-03-22T01:33:28.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:28.086+0000 7fc792c36700 1 -- 172.21.15.72:0/2858147997 shutdown_connections 2024-03-22T01:33:28.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:28.086+0000 7fc792c36700 1 --2- 172.21.15.72:0/2858147997 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc78c103020 0x7fc78c10f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:28.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:28.086+0000 7fc792c36700 1 --2- 172.21.15.72:0/2858147997 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc78c108660 0x7fc78c108a40 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:28.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:28.086+0000 7fc792c36700 1 --2- 172.21.15.72:0/2858147997 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc78c102660 0x7fc78c102ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:28.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:28.086+0000 7fc792c36700 1 -- 172.21.15.72:0/2858147997 >> 172.21.15.72:0/2858147997 conn(0x7fc78c0fe130 msgr2=0x7fc78c100550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:33:28.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:28.086+0000 7fc792c36700 1 -- 172.21.15.72:0/2858147997 shutdown_connections 2024-03-22T01:33:28.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:28.086+0000 7fc792c36700 1 -- 172.21.15.72:0/2858147997 wait complete. 2024-03-22T01:33:28.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:28.087+0000 7fc792c36700 1 Processor -- start 2024-03-22T01:33:28.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:28.087+0000 7fc792c36700 1 -- start start 2024-03-22T01:33:28.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:28.087+0000 7fc792c36700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc78c102660 0x7fc78c198e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:28.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:28.087+0000 7fc792c36700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc78c103020 0x7fc78c199380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:28.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:28.087+0000 7fc792c36700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc78c108660 0x7fc78c19d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:28.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:28.087+0000 7fc792c36700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fc78c1129b0 con 0x7fc78c103020 2024-03-22T01:33:28.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:28.087+0000 7fc792c36700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc78c112830 con 0x7fc78c108660 2024-03-22T01:33:28.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:28.087+0000 7fc792c36700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc78c112b30 con 0x7fc78c102660 2024-03-22T01:33:28.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:28.087+0000 7fc78bfff700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc78c103020 0x7fc78c199380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:28.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:28.087+0000 7fc78bfff700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc78c103020 0x7fc78c199380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:35254/0 (socket says 172.21.15.72:35254) 2024-03-22T01:33:28.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:28.087+0000 7fc78bfff700 1 -- 172.21.15.72:0/3480302051 learned_addr learned my addr 172.21.15.72:0/3480302051 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:33:28.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:28.087+0000 7fc7911d3700 1 --2- 172.21.15.72:0/3480302051 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc78c108660 0x7fc78c19d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:28.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:28.087+0000 7fc7909d2700 1 --2- 172.21.15.72:0/3480302051 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc78c102660 0x7fc78c198e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:28.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:28.087+0000 7fc78bfff700 1 -- 172.21.15.72:0/3480302051 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc78c102660 msgr2=0x7fc78c198e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:28.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:28.087+0000 7fc78bfff700 1 --2- 172.21.15.72:0/3480302051 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc78c102660 0x7fc78c198e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:28.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:28.087+0000 7fc78bfff700 1 -- 172.21.15.72:0/3480302051 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc78c108660 msgr2=0x7fc78c19d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:28.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:28.087+0000 7fc78bfff700 1 --2- 172.21.15.72:0/3480302051 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc78c108660 0x7fc78c19d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:28.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:28.087+0000 7fc78bfff700 1 -- 172.21.15.72:0/3480302051 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc78c19de00 con 0x7fc78c103020 2024-03-22T01:33:28.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:28.088+0000 7fc78bfff700 1 --2- 172.21.15.72:0/3480302051 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc78c103020 0x7fc78c199380 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc78000d950 tx=0x7fc78000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:33:28.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:28.088+0000 7fc789ffb700 1 -- 172.21.15.72:0/3480302051 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc780016070 con 0x7fc78c103020 2024-03-22T01:33:28.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:28.088+0000 7fc792c36700 1 -- 172.21.15.72:0/3480302051 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc78c19e0f0 con 0x7fc78c103020 2024-03-22T01:33:28.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:28.088+0000 7fc789ffb700 1 -- 172.21.15.72:0/3480302051 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc780009e20 con 0x7fc78c103020 2024-03-22T01:33:28.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:28.088+0000 7fc789ffb700 1 -- 172.21.15.72:0/3480302051 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc78000f620 con 0x7fc78c103020 2024-03-22T01:33:28.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:28.088+0000 7fc792c36700 1 -- 172.21.15.72:0/3480302051 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc78c1a5970 con 0x7fc78c103020 2024-03-22T01:33:28.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:28.089+0000 7fc792c36700 1 -- 172.21.15.72:0/3480302051 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc78c069340 con 0x7fc78c103020 2024-03-22T01:33:28.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:28.090+0000 7fc789ffb700 1 -- 172.21.15.72:0/3480302051 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7fc780019070 con 0x7fc78c103020 2024-03-22T01:33:28.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:28.090+0000 7fc789ffb700 1 --2- 172.21.15.72:0/3480302051 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fc7740a2fb0 0x7fc7740a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:28.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:28.090+0000 7fc7909d2700 1 --2- 172.21.15.72:0/3480302051 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fc7740a2fb0 0x7fc7740a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:28.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:28.091+0000 7fc789ffb700 1 -- 172.21.15.72:0/3480302051 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fc7800cead0 con 0x7fc78c103020 2024-03-22T01:33:28.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:28.091+0000 7fc7909d2700 1 --2- 172.21.15.72:0/3480302051 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fc7740a2fb0 0x7fc7740a5470 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc784006130 tx=0x7fc7840060a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:33:28.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:28.095+0000 7fc789ffb700 1 -- 172.21.15.72:0/3480302051 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc78009bda0 con 0x7fc78c103020 2024-03-22T01:33:28.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:28.288+0000 7fc792c36700 1 -- 172.21.15.72:0/3480302051 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc78c02dd70 con 0x7fc78c103020 2024-03-22T01:33:28.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:28.290+0000 7fc789ffb700 1 -- 172.21.15.72:0/3480302051 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) ==== 74+0+18373 (secure 0 0 0) 0x7fc78009b6a0 con 0x7fc78c103020 2024-03-22T01:33:28.289 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:33:28.289 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":83,"fsid":"022a99bc-e7eb-11ee-95cd-87774f69a715","created":"2024-03-22T01:26:23.893672+0000","modified":"2024-03-22T01:33:26.552216+0000","last_up_change":"2024-03-22T01:33:25.223684+0000","last_in_change":"2024-03-22T01:33:05.010331+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T01:29:37.438503+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"16c15bb0-db92-4191-9063-38b4890e81e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6803","nonce":661906791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6805","nonce":661906791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6809","nonce":661906791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6807","nonce":661906791}]},"public_addr":"172.21.15.72:6803/661906791","cluster_addr":"172.21.15.72:6805/661906791","heartbeat_back_addr":"172.21.15.72:6809/661906791","heartbeat_front_addr":"172.21.15.72:6807/661906791","state":["exists","up"]},{"osd":1,"uuid":"981290ba-97dc-496b-aa22-d51704184b10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6801","nonce":2870590550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6803","nonce":2870590550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6807","nonce":2870590550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6805","nonce":2870590550}]},"public_addr":"172.21.15.89:6801/2870590550","cluster_addr":"172.21.15.89:6803/2870590550","heartbeat_back_addr":"172.21.15.89:6807/2870590550","heartbeat_front_addr":"172.21.15.89:6805/2870590550","state":["exists","up"]},{"osd":2,"uuid":"f07ed658-f545-46a1-8edd-8b1d5d49885c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6801","nonce":3865778118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6803","nonce":3865778118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6807","nonce":3865778118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6805","nonce":3865778118}]},"public_addr":"172.21.15.98:6801/3865778118","cluster_addr":"172.21.15.98:6803/3865778118","heartbeat_back_addr":"172.21.15.98:6807/3865778118","heartbeat_front_addr":"172.21.15.98:6805/3865778118","state":["exists","up"]},{"osd":3,"uuid":"ccc4bc40-2c48-46d2-8548-0c05b44f9f1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6811","nonce":1865951395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6813","nonce":1865951395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6817","nonce":1865951395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6815","nonce":1865951395}]},"public_addr":"172.21.15.72:6811/1865951395","cluster_addr":"172.21.15.72:6813/1865951395","heartbeat_back_addr":"172.21.15.72:6817/1865951395","heartbeat_front_addr":"172.21.15.72:6815/1865951395","state":["exists","up"]},{"osd":4,"uuid":"1812439f-810a-4ed2-9395-c00c9a085a13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6809","nonce":2683760795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6811","nonce":2683760795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6815","nonce":2683760795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6813","nonce":2683760795}]},"public_addr":"172.21.15.89:6809/2683760795","cluster_addr":"172.21.15.89:6811/2683760795","heartbeat_back_addr":"172.21.15.89:6815/2683760795","heartbeat_front_addr":"172.21.15.89:6813/2683760795","state":["exists","up"]},{"osd":5,"uuid":"e78954d8-f4e3-4d58-8536-d2d59c843386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6809","nonce":2635838572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6811","nonce":2635838572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6815","nonce":2635838572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6813","nonce":2635838572}]},"public_addr":"172.21.15.98:6809/2635838572","cluster_addr":"172.21.15.98:6811/2635838572","heartbeat_back_addr":"172.21.15.98:6815/2635838572","heartbeat_front_addr":"172.21.15.98:6813/2635838572","state":["exists","up"]},{"osd":6,"uuid":"dc6c8be2-b64c-4959-8015-7b0ea6d25599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6819","nonce":3158982661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6821","nonce":3158982661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6825","nonce":3158982661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6823","nonce":3158982661}]},"public_addr":"172.21.15.72:6819/3158982661","cluster_addr":"172.21.15.72:6821/3158982661","heartbeat_back_addr":"172.21.15.72:6825/3158982661","heartbeat_front_addr":"172.21.15.72:6823/3158982661","state":["exists","up"]},{"osd":7,"uuid":"7a73469f-cf3a-4306-ae93-ea46e21ebbbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6817","nonce":3075913234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6819","nonce":3075913234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6823","nonce":3075913234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6821","nonce":3075913234}]},"public_addr":"172.21.15.89:6817/3075913234","cluster_addr":"172.21.15.89:6819/3075913234","heartbeat_back_addr":"172.21.15.89:6823/3075913234","heartbeat_front_addr":"172.21.15.89:6821/3075913234","state":["exists","up"]},{"osd":8,"uuid":"c4b13bee-ea4d-4344-95fb-2622ecbb8196","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6817","nonce":1813255577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6819","nonce":1813255577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6823","nonce":1813255577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6821","nonce":1813255577}]},"public_addr":"172.21.15.98:6817/1813255577","cluster_addr":"172.21.15.98:6819/1813255577","heartbeat_back_addr":"172.21.15.98:6823/1813255577","heartbeat_front_addr":"172.21.15.98:6821/1813255577","state":["exists","up"]},{"osd":9,"uuid":"eefb0a44-4a36-4e55-9d04-0501bf718db6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6827","nonce":3377749206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6829","nonce":3377749206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6832","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6833","nonce":3377749206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6831","nonce":3377749206}]},"public_addr":"172.21.15.72:6827/3377749206","cluster_addr":"172.21.15.72:6829/3377749206","heartbeat_back_addr":"172.21.15.72:6833/3377749206","heartbeat_front_addr":"172.21.15.72:6831/3377749206","state":["exists","up"]},{"osd":10,"uuid":"8741d25f-40dd-40d7-b040-5cc5e411eb98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6825","nonce":885058786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6827","nonce":885058786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6831","nonce":885058786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6829","nonce":885058786}]},"public_addr":"172.21.15.89:6825/885058786","cluster_addr":"172.21.15.89:6827/885058786","heartbeat_back_addr":"172.21.15.89:6831/885058786","heartbeat_front_addr":"172.21.15.89:6829/885058786","state":["exists","up"]},{"osd":11,"uuid":"8d02e957-eda4-4184-b5c6-5eb3b8b38060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6825","nonce":1517095838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6827","nonce":1517095838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6831","nonce":1517095838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6829","nonce":1517095838}]},"public_addr":"172.21.15.98:6825/1517095838","cluster_addr":"172.21.15.98:6827/1517095838","heartbeat_back_addr":"172.21.15.98:6831/1517095838","heartbeat_front_addr":"172.21.15.98:6829/1517095838","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:28:47.307652+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:11.037135+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:34.619188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:59.350738+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:30:23.787337+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:30:48.994154+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:31:13.690350+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:31:39.628116+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:04.723912+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:30.689741+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:57.229508+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:33:23.154537+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:0/121087196":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/2451857761":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/1228477995":"2024-03-23T01:27:15.339809+0000","172.21.15.72:6801/1513232913":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/3558958264":"2024-03-23T01:26:50.074871+0000","172.21.15.72:0/1510187198":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6800/3518012916":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6801/3518012916":"2024-03-23T01:26:50.074871+0000","172.21.15.72:0/1372801041":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6800/1513232913":"2024-03-23T01:27:15.339809+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T01:33:28.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:28.292+0000 7fc792c36700 1 -- 172.21.15.72:0/3480302051 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fc7740a2fb0 msgr2=0x7fc7740a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:28.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:28.292+0000 7fc792c36700 1 --2- 172.21.15.72:0/3480302051 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fc7740a2fb0 0x7fc7740a5470 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc784006130 tx=0x7fc7840060a0 comp rx=0 tx=0).stop 2024-03-22T01:33:28.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:28.292+0000 7fc792c36700 1 -- 172.21.15.72:0/3480302051 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc78c103020 msgr2=0x7fc78c199380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:28.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:28.292+0000 7fc792c36700 1 --2- 172.21.15.72:0/3480302051 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc78c103020 0x7fc78c199380 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc78000d950 tx=0x7fc78000dc60 comp rx=0 tx=0).stop 2024-03-22T01:33:28.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:28.292+0000 7fc792c36700 1 -- 172.21.15.72:0/3480302051 shutdown_connections 2024-03-22T01:33:28.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:28.292+0000 7fc792c36700 1 --2- 172.21.15.72:0/3480302051 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fc7740a2fb0 0x7fc7740a5470 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:28.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:28.292+0000 7fc792c36700 1 --2- 172.21.15.72:0/3480302051 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc78c108660 0x7fc78c19d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:28.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:28.292+0000 7fc792c36700 1 --2- 172.21.15.72:0/3480302051 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc78c102660 0x7fc78c198e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:28.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:28.292+0000 7fc792c36700 1 --2- 172.21.15.72:0/3480302051 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc78c103020 0x7fc78c199380 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:28.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:28.292+0000 7fc792c36700 1 -- 172.21.15.72:0/3480302051 >> 172.21.15.72:0/3480302051 conn(0x7fc78c0fe130 msgr2=0x7fc78c0fe7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:33:28.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:28.292+0000 7fc792c36700 1 -- 172.21.15.72:0/3480302051 shutdown_connections 2024-03-22T01:33:28.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:28.292+0000 7fc792c36700 1 -- 172.21.15.72:0/3480302051 wait complete. 2024-03-22T01:33:28.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:28 smithi089 ceph-mon[35770]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:33:28.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:28 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/3480302051' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:33:28.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:28 smithi072 ceph-mon[32414]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:33:28.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:28 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/3480302051' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:33:28.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:28 smithi098 ceph-mon[36468]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:33:28.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:28 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/3480302051' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:33:28.933 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-22T01:29:37.438503+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-22T01:33:28.933 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-22T01:33:29.170 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:33:30.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:30.754+0000 7f94943c3700 1 -- 172.21.15.72:0/3756970372 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f948c107a40 msgr2=0x7f948c10f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:30.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:30.754+0000 7f94943c3700 1 --2- 172.21.15.72:0/3756970372 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f948c107a40 0x7f948c10f5c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f948400b0d0 tx=0x7f948400b3e0 comp rx=0 tx=0).stop 2024-03-22T01:33:30.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:30.755+0000 7f94943c3700 1 -- 172.21.15.72:0/3756970372 shutdown_connections 2024-03-22T01:33:30.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:30.755+0000 7f94943c3700 1 --2- 172.21.15.72:0/3756970372 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f948c107a40 0x7f948c10f5c0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:30.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:30.755+0000 7f94943c3700 1 --2- 172.21.15.72:0/3756970372 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f948c101220 0x7f948c101600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:30.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:30.755+0000 7f94943c3700 1 --2- 172.21.15.72:0/3756970372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f948c101bd0 0x7f948c107310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:30.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:30.755+0000 7f94943c3700 1 -- 172.21.15.72:0/3756970372 >> 172.21.15.72:0/3756970372 conn(0x7f948c0768e0 msgr2=0x7f948c076cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:33:30.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:30.755+0000 7f94943c3700 1 -- 172.21.15.72:0/3756970372 shutdown_connections 2024-03-22T01:33:30.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:30.755+0000 7f94943c3700 1 -- 172.21.15.72:0/3756970372 wait complete. 2024-03-22T01:33:30.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:30.755+0000 7f94943c3700 1 Processor -- start 2024-03-22T01:33:30.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:30.755+0000 7f94943c3700 1 -- start start 2024-03-22T01:33:30.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:30.755+0000 7f94943c3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f948c101220 0x7f948c198ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:30.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:30.755+0000 7f94943c3700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f948c101bd0 0x7f948c199400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:30.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:30.755+0000 7f94943c3700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f948c107a40 0x7f948c19d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:30.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:30.755+0000 7f94943c3700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f948c112a70 con 0x7f948c107a40 2024-03-22T01:33:30.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:30.756+0000 7f94943c3700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f948c1128f0 con 0x7f948c101bd0 2024-03-22T01:33:30.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:30.756+0000 7f94943c3700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f948c112bf0 con 0x7f948c101220 2024-03-22T01:33:30.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:30.756+0000 7f9492960700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f948c107a40 0x7f948c19d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:30.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:30.756+0000 7f9492960700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f948c107a40 0x7f948c19d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33314/0 (socket says 172.21.15.72:33314) 2024-03-22T01:33:30.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:30.756+0000 7f9492960700 1 -- 172.21.15.72:0/2936493327 learned_addr learned my addr 172.21.15.72:0/2936493327 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:33:30.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:30.756+0000 7f949215f700 1 --2- 172.21.15.72:0/2936493327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f948c101220 0x7f948c198ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:30.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:30.756+0000 7f949195e700 1 --2- 172.21.15.72:0/2936493327 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f948c101bd0 0x7f948c199400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:30.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:30.756+0000 7f9492960700 1 -- 172.21.15.72:0/2936493327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f948c101220 msgr2=0x7f948c198ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:30.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:30.756+0000 7f9492960700 1 --2- 172.21.15.72:0/2936493327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f948c101220 0x7f948c198ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:30.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:30.756+0000 7f9492960700 1 -- 172.21.15.72:0/2936493327 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f948c101bd0 msgr2=0x7f948c199400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:30.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:30.756+0000 7f9492960700 1 --2- 172.21.15.72:0/2936493327 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f948c101bd0 0x7f948c199400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:30.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:30.756+0000 7f9492960700 1 -- 172.21.15.72:0/2936493327 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f948c19de80 con 0x7f948c107a40 2024-03-22T01:33:30.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:30.756+0000 7f9492960700 1 --2- 172.21.15.72:0/2936493327 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f948c107a40 0x7f948c19d740 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f9484006070 tx=0x7f9484004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:33:30.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:30.756+0000 7f94837fe700 1 -- 172.21.15.72:0/2936493327 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9484021070 con 0x7f948c107a40 2024-03-22T01:33:30.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:30.756+0000 7f94837fe700 1 -- 172.21.15.72:0/2936493327 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9484007b80 con 0x7f948c107a40 2024-03-22T01:33:30.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:30.756+0000 7f94837fe700 1 -- 172.21.15.72:0/2936493327 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9484005330 con 0x7f948c107a40 2024-03-22T01:33:30.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:30.756+0000 7f94943c3700 1 -- 172.21.15.72:0/2936493327 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f948c19e110 con 0x7f948c107a40 2024-03-22T01:33:30.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:30.756+0000 7f94943c3700 1 -- 172.21.15.72:0/2936493327 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f948c1a59f0 con 0x7f948c107a40 2024-03-22T01:33:30.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:30.757+0000 7f94943c3700 1 -- 172.21.15.72:0/2936493327 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f948c10ac80 con 0x7f948c107a40 2024-03-22T01:33:30.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:30.758+0000 7f94837fe700 1 -- 172.21.15.72:0/2936493327 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f9484024070 con 0x7f948c107a40 2024-03-22T01:33:30.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:30.759+0000 7f94837fe700 1 --2- 172.21.15.72:0/2936493327 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f94780a3030 0x7f94780a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:30.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:30.759+0000 7f949215f700 1 --2- 172.21.15.72:0/2936493327 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f94780a3030 0x7f94780a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:30.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:30.759+0000 7f94837fe700 1 -- 172.21.15.72:0/2936493327 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f948400f020 con 0x7f948c107a40 2024-03-22T01:33:30.760 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:30.759+0000 7f949215f700 1 --2- 172.21.15.72:0/2936493327 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f94780a3030 0x7f94780a54f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9488009a10 tx=0x7f9488009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:33:30.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:30.762+0000 7f94837fe700 1 -- 172.21.15.72:0/2936493327 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f948402b020 con 0x7f948c107a40 2024-03-22T01:33:30.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:30 smithi089 ceph-mon[35770]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:33:30.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:30 smithi072 ceph-mon[32414]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:33:30.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:30 smithi098 ceph-mon[36468]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:33:30.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:30.956+0000 7f94943c3700 1 -- 172.21.15.72:0/2936493327 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f948c19a1d0 con 0x7f948c107a40 2024-03-22T01:33:30.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:30.957+0000 7f94837fe700 1 -- 172.21.15.72:0/2936493327 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) ==== 93+0+10 (secure 0 0 0) 0x7f94840bd0a0 con 0x7f948c107a40 2024-03-22T01:33:30.956 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-22T01:33:30.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:30.959+0000 7f94943c3700 1 -- 172.21.15.72:0/2936493327 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f94780a3030 msgr2=0x7f94780a54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:30.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:30.959+0000 7f94943c3700 1 --2- 172.21.15.72:0/2936493327 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f94780a3030 0x7f94780a54f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9488009a10 tx=0x7f9488009190 comp rx=0 tx=0).stop 2024-03-22T01:33:30.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:30.959+0000 7f94943c3700 1 -- 172.21.15.72:0/2936493327 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f948c107a40 msgr2=0x7f948c19d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:30.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:30.959+0000 7f94943c3700 1 --2- 172.21.15.72:0/2936493327 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f948c107a40 0x7f948c19d740 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f9484006070 tx=0x7f9484004d00 comp rx=0 tx=0).stop 2024-03-22T01:33:30.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:30.959+0000 7f94943c3700 1 -- 172.21.15.72:0/2936493327 shutdown_connections 2024-03-22T01:33:30.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:30.959+0000 7f94943c3700 1 --2- 172.21.15.72:0/2936493327 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f94780a3030 0x7f94780a54f0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:30.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:30.959+0000 7f94943c3700 1 --2- 172.21.15.72:0/2936493327 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f948c107a40 0x7f948c19d740 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:30.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:30.959+0000 7f94943c3700 1 --2- 172.21.15.72:0/2936493327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f948c101220 0x7f948c198ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:30.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:30.959+0000 7f94943c3700 1 --2- 172.21.15.72:0/2936493327 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f948c101bd0 0x7f948c199400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:30.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:30.959+0000 7f94943c3700 1 -- 172.21.15.72:0/2936493327 >> 172.21.15.72:0/2936493327 conn(0x7f948c0768e0 msgr2=0x7f948c0ff010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:33:30.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:30.959+0000 7f94943c3700 1 -- 172.21.15.72:0/2936493327 shutdown_connections 2024-03-22T01:33:30.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:30.959+0000 7f94943c3700 1 -- 172.21.15.72:0/2936493327 wait complete. 2024-03-22T01:33:31.617 INFO:tasks.cephadm:Adding mds.a on smithi072 2024-03-22T01:33:31.617 INFO:tasks.cephadm:Adding mds.d on smithi072 2024-03-22T01:33:31.617 INFO:tasks.cephadm:Adding mds.g on smithi072 2024-03-22T01:33:31.617 INFO:tasks.cephadm:Adding mds.j on smithi072 2024-03-22T01:33:31.617 INFO:tasks.cephadm:Adding mds.b on smithi089 2024-03-22T01:33:31.617 INFO:tasks.cephadm:Adding mds.e on smithi089 2024-03-22T01:33:31.618 INFO:tasks.cephadm:Adding mds.h on smithi089 2024-03-22T01:33:31.618 INFO:tasks.cephadm:Adding mds.k on smithi089 2024-03-22T01:33:31.618 INFO:tasks.cephadm:Adding mds.c on smithi098 2024-03-22T01:33:31.618 INFO:tasks.cephadm:Adding mds.f on smithi098 2024-03-22T01:33:31.618 INFO:tasks.cephadm:Adding mds.i on smithi098 2024-03-22T01:33:31.618 INFO:tasks.cephadm:Adding mds.l on smithi098 2024-03-22T01:33:31.619 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph orch apply mds all '12;smithi072=a;smithi072=d;smithi072=g;smithi072=j;smithi089=b;smithi089=e;smithi089=h;smithi089=k;smithi098=c;smithi098=f;smithi098=i;smithi098=l' 2024-03-22T01:33:31.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:31 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/2936493327' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T01:33:31.870 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.c/config 2024-03-22T01:33:31.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:31 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/2936493327' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T01:33:31.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:31 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/2936493327' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T01:33:32.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:32 smithi089 ceph-mon[35770]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:33:32.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:32 smithi072 ceph-mon[32414]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:33:32.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:32 smithi098 ceph-mon[36468]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:33:33.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:33.578+0000 7fa97d8c4700 1 -- 172.21.15.98:0/2096805795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9781026b0 msgr2=0x7fa978102ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:33.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:33.578+0000 7fa97d8c4700 1 --2- 172.21.15.98:0/2096805795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9781026b0 0x7fa978102ad0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa968009990 tx=0x7fa968009ca0 comp rx=0 tx=0).stop 2024-03-22T01:33:33.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:33.578+0000 7fa97d8c4700 1 -- 172.21.15.98:0/2096805795 shutdown_connections 2024-03-22T01:33:33.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:33.578+0000 7fa97d8c4700 1 --2- 172.21.15.98:0/2096805795 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa978104250 0x7fa97810ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:33.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:33.578+0000 7fa97d8c4700 1 --2- 172.21.15.98:0/2096805795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9781026b0 0x7fa978102ad0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:33.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:33.578+0000 7fa97d8c4700 1 --2- 172.21.15.98:0/2096805795 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9781038b0 0x7fa978103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:33.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:33.578+0000 7fa97d8c4700 1 -- 172.21.15.98:0/2096805795 >> 172.21.15.98:0/2096805795 conn(0x7fa9780fdc90 msgr2=0x7fa9781000b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:33:33.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:33.578+0000 7fa97d8c4700 1 -- 172.21.15.98:0/2096805795 shutdown_connections 2024-03-22T01:33:33.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:33.578+0000 7fa97d8c4700 1 -- 172.21.15.98:0/2096805795 wait complete. 2024-03-22T01:33:33.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:33.579+0000 7fa97d8c4700 1 Processor -- start 2024-03-22T01:33:33.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:33.579+0000 7fa97d8c4700 1 -- start start 2024-03-22T01:33:33.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:33.579+0000 7fa97d8c4700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa9781026b0 0x7fa9781988a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:33.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:33.579+0000 7fa97d8c4700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9781038b0 0x7fa978198de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:33.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:33.579+0000 7fa97d8c4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa978104250 0x7fa97819fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:33.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:33.579+0000 7fa97d8c4700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fa97810e2c0 con 0x7fa9781026b0 2024-03-22T01:33:33.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:33.579+0000 7fa97d8c4700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa97810e140 con 0x7fa9781038b0 2024-03-22T01:33:33.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:33.579+0000 7fa97d8c4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa97810e440 con 0x7fa978104250 2024-03-22T01:33:33.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:33.579+0000 7fa9777fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa978104250 0x7fa97819fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:33.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:33.579+0000 7fa9777fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa978104250 0x7fa97819fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36266/0 (socket says 172.21.15.98:36266) 2024-03-22T01:33:33.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:33.579+0000 7fa9777fe700 1 -- 172.21.15.98:0/1560023933 learned_addr learned my addr 172.21.15.98:0/1560023933 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-22T01:33:33.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:33.579+0000 7fa976ffd700 1 --2- 172.21.15.98:0/1560023933 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa9781026b0 0x7fa9781988a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:33.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:33.579+0000 7fa9767fc700 1 --2- 172.21.15.98:0/1560023933 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9781038b0 0x7fa978198de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:33.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:33.580+0000 7fa9777fe700 1 -- 172.21.15.98:0/1560023933 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9781038b0 msgr2=0x7fa978198de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:33.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:33.580+0000 7fa9777fe700 1 --2- 172.21.15.98:0/1560023933 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9781038b0 0x7fa978198de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:33.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:33.580+0000 7fa9777fe700 1 -- 172.21.15.98:0/1560023933 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa9781026b0 msgr2=0x7fa9781988a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:33.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:33.580+0000 7fa9777fe700 1 --2- 172.21.15.98:0/1560023933 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa9781026b0 0x7fa9781988a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:33.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:33.580+0000 7fa9777fe700 1 -- 172.21.15.98:0/1560023933 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9781a0500 con 0x7fa978104250 2024-03-22T01:33:33.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:33.580+0000 7fa9767fc700 1 --2- 172.21.15.98:0/1560023933 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9781038b0 0x7fa978198de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:33:33.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:33.580+0000 7fa9777fe700 1 --2- 172.21.15.98:0/1560023933 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa978104250 0x7fa97819fe60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa970007d40 tx=0x7fa97000a340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:33:33.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:33.580+0000 7fa97c8c2700 1 -- 172.21.15.98:0/1560023933 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa970016020 con 0x7fa978104250 2024-03-22T01:33:33.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:33.580+0000 7fa97d8c4700 1 -- 172.21.15.98:0/1560023933 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9781a07f0 con 0x7fa978104250 2024-03-22T01:33:33.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:33.580+0000 7fa97c8c2700 1 -- 172.21.15.98:0/1560023933 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa97000ad60 con 0x7fa978104250 2024-03-22T01:33:33.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:33.580+0000 7fa97c8c2700 1 -- 172.21.15.98:0/1560023933 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa970011600 con 0x7fa978104250 2024-03-22T01:33:33.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:33.580+0000 7fa97d8c4700 1 -- 172.21.15.98:0/1560023933 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9781a0d30 con 0x7fa978104250 2024-03-22T01:33:33.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:33.581+0000 7fa97d8c4700 1 -- 172.21.15.98:0/1560023933 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa978069340 con 0x7fa978104250 2024-03-22T01:33:33.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:33.582+0000 7fa97c8c2700 1 -- 172.21.15.98:0/1560023933 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7fa970019020 con 0x7fa978104250 2024-03-22T01:33:33.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:33.582+0000 7fa97c8c2700 1 --2- 172.21.15.98:0/1560023933 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fa9600a2fb0 0x7fa9600a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:33.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:33.583+0000 7fa976ffd700 1 --2- 172.21.15.98:0/1560023933 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fa9600a2fb0 0x7fa9600a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:33.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:33.583+0000 7fa97c8c2700 1 -- 172.21.15.98:0/1560023933 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fa9700cef20 con 0x7fa978104250 2024-03-22T01:33:33.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:33.583+0000 7fa976ffd700 1 --2- 172.21.15.98:0/1560023933 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fa9600a2fb0 0x7fa9600a5470 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa968000c00 tx=0x7fa968005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:33:33.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:33.588+0000 7fa97c8c2700 1 -- 172.21.15.98:0/1560023933 <== mon.1 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa970020020 con 0x7fa978104250 2024-03-22T01:33:33.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:33.796+0000 7fa97d8c4700 1 -- 172.21.15.98:0/1560023933 --> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi072=a;smithi072=d;smithi072=g;smithi072=j;smithi089=b;smithi089=e;smithi089=h;smithi089=k;smithi098=c;smithi098=f;smithi098=i;smithi098=l", "target": ["mon-mgr", ""]}) -- 0x7fa978062cc0 con 0x7fa9600a2fb0 2024-03-22T01:33:33.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:33.806+0000 7fa97c8c2700 1 -- 172.21.15.98:0/1560023933 <== mgr.14152 v2:172.21.15.72:6800/401403200 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7fa978062cc0 con 0x7fa9600a2fb0 2024-03-22T01:33:33.805 INFO:teuthology.orchestra.run.smithi098.stdout:Scheduled mds.all update... 2024-03-22T01:33:33.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:33.809+0000 7fa97d8c4700 1 -- 172.21.15.98:0/1560023933 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fa9600a2fb0 msgr2=0x7fa9600a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:33.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:33.809+0000 7fa97d8c4700 1 --2- 172.21.15.98:0/1560023933 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fa9600a2fb0 0x7fa9600a5470 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa968000c00 tx=0x7fa968005fb0 comp rx=0 tx=0).stop 2024-03-22T01:33:33.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:33.809+0000 7fa97d8c4700 1 -- 172.21.15.98:0/1560023933 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa978104250 msgr2=0x7fa97819fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:33.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:33.809+0000 7fa97d8c4700 1 --2- 172.21.15.98:0/1560023933 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa978104250 0x7fa97819fe60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa970007d40 tx=0x7fa97000a340 comp rx=0 tx=0).stop 2024-03-22T01:33:33.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:33.809+0000 7fa97d8c4700 1 -- 172.21.15.98:0/1560023933 shutdown_connections 2024-03-22T01:33:33.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:33.809+0000 7fa97d8c4700 1 --2- 172.21.15.98:0/1560023933 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fa9600a2fb0 0x7fa9600a5470 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:33.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:33.809+0000 7fa97d8c4700 1 --2- 172.21.15.98:0/1560023933 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa978104250 0x7fa97819fe60 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:33.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:33.809+0000 7fa97d8c4700 1 --2- 172.21.15.98:0/1560023933 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa9781026b0 0x7fa9781988a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:33.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:33.809+0000 7fa97d8c4700 1 --2- 172.21.15.98:0/1560023933 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9781038b0 0x7fa978198de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:33.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:33.809+0000 7fa97d8c4700 1 -- 172.21.15.98:0/1560023933 >> 172.21.15.98:0/1560023933 conn(0x7fa9780fdc90 msgr2=0x7fa978106bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:33:33.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:33.809+0000 7fa97d8c4700 1 -- 172.21.15.98:0/1560023933 shutdown_connections 2024-03-22T01:33:33.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-22T01:33:33.809+0000 7fa97d8c4700 1 -- 172.21.15.98:0/1560023933 wait complete. 2024-03-22T01:33:34.532 DEBUG:teuthology.orchestra.run.smithi072:mds.a> sudo journalctl -f -n 0 -u ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mds.a.service 2024-03-22T01:33:34.535 DEBUG:teuthology.orchestra.run.smithi072:mds.d> sudo journalctl -f -n 0 -u ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mds.d.service 2024-03-22T01:33:34.579 DEBUG:teuthology.orchestra.run.smithi072:mds.g> sudo journalctl -f -n 0 -u ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mds.g.service 2024-03-22T01:33:34.621 DEBUG:teuthology.orchestra.run.smithi072:mds.j> sudo journalctl -f -n 0 -u ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mds.j.service 2024-03-22T01:33:34.665 DEBUG:teuthology.orchestra.run.smithi089:mds.b> sudo journalctl -f -n 0 -u ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mds.b.service 2024-03-22T01:33:34.667 DEBUG:teuthology.orchestra.run.smithi089:mds.e> sudo journalctl -f -n 0 -u ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mds.e.service 2024-03-22T01:33:34.710 DEBUG:teuthology.orchestra.run.smithi089:mds.h> sudo journalctl -f -n 0 -u ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mds.h.service 2024-03-22T01:33:34.754 DEBUG:teuthology.orchestra.run.smithi089:mds.k> sudo journalctl -f -n 0 -u ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mds.k.service 2024-03-22T01:33:34.797 DEBUG:teuthology.orchestra.run.smithi098:mds.c> sudo journalctl -f -n 0 -u ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mds.c.service 2024-03-22T01:33:34.799 DEBUG:teuthology.orchestra.run.smithi098:mds.f> sudo journalctl -f -n 0 -u ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mds.f.service 2024-03-22T01:33:34.821 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:34 smithi098 ceph-mon[36468]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:33:34.821 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:34 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:34.821 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:34 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:33:34.822 DEBUG:teuthology.orchestra.run.smithi098:mds.i> sudo journalctl -f -n 0 -u ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mds.i.service 2024-03-22T01:33:34.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:34 smithi089 ceph-mon[35770]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:33:34.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:34 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:34.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:34 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:33:34.866 DEBUG:teuthology.orchestra.run.smithi098:mds.l> sudo journalctl -f -n 0 -u ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mds.l.service 2024-03-22T01:33:34.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:34 smithi072 ceph-mon[32414]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:33:34.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:34 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:34.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:34 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:33:34.909 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-03-22T01:33:34.909 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T01:33:35.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.041+0000 7f2f28ec3640 1 -- 172.21.15.72:0/966996141 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f24107910 msgr2=0x7f2f24109d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:35.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.041+0000 7f2f28ec3640 1 --2- 172.21.15.72:0/966996141 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f24107910 0x7f2f24109d60 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c009ca0 tx=0x7f2f1c02c0f0 comp rx=0 tx=0).stop 2024-03-22T01:33:35.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.041+0000 7f2f28ec3640 1 -- 172.21.15.72:0/966996141 shutdown_connections 2024-03-22T01:33:35.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.041+0000 7f2f28ec3640 1 --2- 172.21.15.72:0/966996141 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f24104fe0 0x7f2f241073d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:35.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.041+0000 7f2f28ec3640 1 --2- 172.21.15.72:0/966996141 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f24107910 0x7f2f24109d60 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:35.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.041+0000 7f2f28ec3640 1 --2- 172.21.15.72:0/966996141 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f241026b0 0x7f2f24104aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:35.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.041+0000 7f2f28ec3640 1 -- 172.21.15.72:0/966996141 >> 172.21.15.72:0/966996141 conn(0x7f2f240fc0a0 msgr2=0x7f2f240fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:33:35.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.041+0000 7f2f28ec3640 1 -- 172.21.15.72:0/966996141 shutdown_connections 2024-03-22T01:33:35.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.041+0000 7f2f28ec3640 1 -- 172.21.15.72:0/966996141 wait complete. 2024-03-22T01:33:35.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.041+0000 7f2f28ec3640 1 Processor -- start 2024-03-22T01:33:35.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.041+0000 7f2f28ec3640 1 -- start start 2024-03-22T01:33:35.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.042+0000 7f2f28ec3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f241026b0 0x7f2f24197dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:35.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.042+0000 7f2f28ec3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f24104fe0 0x7f2f24198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:35.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.042+0000 7f2f28ec3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f24107910 0x7f2f2419f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:35.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.042+0000 7f2f28ec3640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f2f2410d860 con 0x7f2f24104fe0 2024-03-22T01:33:35.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.042+0000 7f2f28ec3640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2f2410d6e0 con 0x7f2f24107910 2024-03-22T01:33:35.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.042+0000 7f2f28ec3640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2f2410d9e0 con 0x7f2f241026b0 2024-03-22T01:33:35.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.042+0000 7f2f21d9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f24104fe0 0x7f2f24198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:35.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.042+0000 7f2f21d9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f24104fe0 0x7f2f24198300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33344/0 (socket says 172.21.15.72:33344) 2024-03-22T01:33:35.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.042+0000 7f2f2259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f241026b0 0x7f2f24197dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:35.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.042+0000 7f2f21d9b640 1 -- 172.21.15.72:0/4039869606 learned_addr learned my addr 172.21.15.72:0/4039869606 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:33:35.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.042+0000 7f2f22d9d640 1 --2- 172.21.15.72:0/4039869606 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f24107910 0x7f2f2419f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:35.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.042+0000 7f2f21d9b640 1 -- 172.21.15.72:0/4039869606 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f241026b0 msgr2=0x7f2f24197dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:35.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.042+0000 7f2f21d9b640 1 --2- 172.21.15.72:0/4039869606 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f241026b0 0x7f2f24197dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:35.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.042+0000 7f2f21d9b640 1 -- 172.21.15.72:0/4039869606 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f24107910 msgr2=0x7f2f2419f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:35.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.042+0000 7f2f21d9b640 1 --2- 172.21.15.72:0/4039869606 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f24107910 0x7f2f2419f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:35.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.042+0000 7f2f21d9b640 1 -- 172.21.15.72:0/4039869606 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f2419f9e0 con 0x7f2f24104fe0 2024-03-22T01:33:35.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.042+0000 7f2f22d9d640 1 --2- 172.21.15.72:0/4039869606 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f24107910 0x7f2f2419f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:33:35.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.042+0000 7f2f2259c640 1 --2- 172.21.15.72:0/4039869606 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f241026b0 0x7f2f24197dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:33:35.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.042+0000 7f2f21d9b640 1 --2- 172.21.15.72:0/4039869606 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f24104fe0 0x7f2f24198300 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f2f1800c970 tx=0x7f2f1800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:33:35.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.042+0000 7f2f137fe640 1 -- 172.21.15.72:0/4039869606 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f18014070 con 0x7f2f24104fe0 2024-03-22T01:33:35.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.042+0000 7f2f28ec3640 1 -- 172.21.15.72:0/4039869606 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f2419fcd0 con 0x7f2f24104fe0 2024-03-22T01:33:35.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.042+0000 7f2f137fe640 1 -- 172.21.15.72:0/4039869606 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f18004440 con 0x7f2f24104fe0 2024-03-22T01:33:35.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.042+0000 7f2f137fe640 1 -- 172.21.15.72:0/4039869606 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f1800f450 con 0x7f2f24104fe0 2024-03-22T01:33:35.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.042+0000 7f2f28ec3640 1 -- 172.21.15.72:0/4039869606 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f241a0210 con 0x7f2f24104fe0 2024-03-22T01:33:35.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.043+0000 7f2f28ec3640 1 -- 172.21.15.72:0/4039869606 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f240fdb90 con 0x7f2f24104fe0 2024-03-22T01:33:35.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.044+0000 7f2f137fe640 1 -- 172.21.15.72:0/4039869606 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f2f18017070 con 0x7f2f24104fe0 2024-03-22T01:33:35.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.045+0000 7f2f137fe640 1 --2- 172.21.15.72:0/4039869606 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f2f0c0a2f20 0x7f2f0c0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:35.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.045+0000 7f2f2259c640 1 --2- 172.21.15.72:0/4039869606 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f2f0c0a2f20 0x7f2f0c0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:35.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.045+0000 7f2f137fe640 1 -- 172.21.15.72:0/4039869606 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f2f180cf180 con 0x7f2f24104fe0 2024-03-22T01:33:35.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.045+0000 7f2f2259c640 1 --2- 172.21.15.72:0/4039869606 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f2f0c0a2f20 0x7f2f0c0a53e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2f14009fc0 tx=0x7f2f14005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:33:35.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.049+0000 7f2f137fe640 1 -- 172.21.15.72:0/4039869606 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2f1801e020 con 0x7f2f24104fe0 2024-03-22T01:33:35.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.224+0000 7f2f28ec3640 1 -- 172.21.15.72:0/4039869606 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2f2402d530 con 0x7f2f24104fe0 2024-03-22T01:33:35.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.225+0000 7f2f137fe640 1 -- 172.21.15.72:0/4039869606 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) ==== 74+0+18373 (secure 0 0 0) 0x7f2f1809c450 con 0x7f2f24104fe0 2024-03-22T01:33:35.224 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:33:35.224 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":83,"fsid":"022a99bc-e7eb-11ee-95cd-87774f69a715","created":"2024-03-22T01:26:23.893672+0000","modified":"2024-03-22T01:33:26.552216+0000","last_up_change":"2024-03-22T01:33:25.223684+0000","last_in_change":"2024-03-22T01:33:05.010331+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T01:29:37.438503+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"16c15bb0-db92-4191-9063-38b4890e81e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6803","nonce":661906791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6805","nonce":661906791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6809","nonce":661906791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6807","nonce":661906791}]},"public_addr":"172.21.15.72:6803/661906791","cluster_addr":"172.21.15.72:6805/661906791","heartbeat_back_addr":"172.21.15.72:6809/661906791","heartbeat_front_addr":"172.21.15.72:6807/661906791","state":["exists","up"]},{"osd":1,"uuid":"981290ba-97dc-496b-aa22-d51704184b10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6801","nonce":2870590550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6803","nonce":2870590550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6807","nonce":2870590550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6805","nonce":2870590550}]},"public_addr":"172.21.15.89:6801/2870590550","cluster_addr":"172.21.15.89:6803/2870590550","heartbeat_back_addr":"172.21.15.89:6807/2870590550","heartbeat_front_addr":"172.21.15.89:6805/2870590550","state":["exists","up"]},{"osd":2,"uuid":"f07ed658-f545-46a1-8edd-8b1d5d49885c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6801","nonce":3865778118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6803","nonce":3865778118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6807","nonce":3865778118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6805","nonce":3865778118}]},"public_addr":"172.21.15.98:6801/3865778118","cluster_addr":"172.21.15.98:6803/3865778118","heartbeat_back_addr":"172.21.15.98:6807/3865778118","heartbeat_front_addr":"172.21.15.98:6805/3865778118","state":["exists","up"]},{"osd":3,"uuid":"ccc4bc40-2c48-46d2-8548-0c05b44f9f1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6811","nonce":1865951395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6813","nonce":1865951395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6817","nonce":1865951395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6815","nonce":1865951395}]},"public_addr":"172.21.15.72:6811/1865951395","cluster_addr":"172.21.15.72:6813/1865951395","heartbeat_back_addr":"172.21.15.72:6817/1865951395","heartbeat_front_addr":"172.21.15.72:6815/1865951395","state":["exists","up"]},{"osd":4,"uuid":"1812439f-810a-4ed2-9395-c00c9a085a13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6809","nonce":2683760795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6811","nonce":2683760795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6815","nonce":2683760795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6813","nonce":2683760795}]},"public_addr":"172.21.15.89:6809/2683760795","cluster_addr":"172.21.15.89:6811/2683760795","heartbeat_back_addr":"172.21.15.89:6815/2683760795","heartbeat_front_addr":"172.21.15.89:6813/2683760795","state":["exists","up"]},{"osd":5,"uuid":"e78954d8-f4e3-4d58-8536-d2d59c843386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6809","nonce":2635838572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6811","nonce":2635838572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6815","nonce":2635838572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6813","nonce":2635838572}]},"public_addr":"172.21.15.98:6809/2635838572","cluster_addr":"172.21.15.98:6811/2635838572","heartbeat_back_addr":"172.21.15.98:6815/2635838572","heartbeat_front_addr":"172.21.15.98:6813/2635838572","state":["exists","up"]},{"osd":6,"uuid":"dc6c8be2-b64c-4959-8015-7b0ea6d25599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6819","nonce":3158982661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6821","nonce":3158982661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6825","nonce":3158982661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6823","nonce":3158982661}]},"public_addr":"172.21.15.72:6819/3158982661","cluster_addr":"172.21.15.72:6821/3158982661","heartbeat_back_addr":"172.21.15.72:6825/3158982661","heartbeat_front_addr":"172.21.15.72:6823/3158982661","state":["exists","up"]},{"osd":7,"uuid":"7a73469f-cf3a-4306-ae93-ea46e21ebbbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6817","nonce":3075913234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6819","nonce":3075913234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6823","nonce":3075913234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6821","nonce":3075913234}]},"public_addr":"172.21.15.89:6817/3075913234","cluster_addr":"172.21.15.89:6819/3075913234","heartbeat_back_addr":"172.21.15.89:6823/3075913234","heartbeat_front_addr":"172.21.15.89:6821/3075913234","state":["exists","up"]},{"osd":8,"uuid":"c4b13bee-ea4d-4344-95fb-2622ecbb8196","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6817","nonce":1813255577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6819","nonce":1813255577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6823","nonce":1813255577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6821","nonce":1813255577}]},"public_addr":"172.21.15.98:6817/1813255577","cluster_addr":"172.21.15.98:6819/1813255577","heartbeat_back_addr":"172.21.15.98:6823/1813255577","heartbeat_front_addr":"172.21.15.98:6821/1813255577","state":["exists","up"]},{"osd":9,"uuid":"eefb0a44-4a36-4e55-9d04-0501bf718db6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6827","nonce":3377749206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6829","nonce":3377749206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6832","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6833","nonce":3377749206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6831","nonce":3377749206}]},"public_addr":"172.21.15.72:6827/3377749206","cluster_addr":"172.21.15.72:6829/3377749206","heartbeat_back_addr":"172.21.15.72:6833/3377749206","heartbeat_front_addr":"172.21.15.72:6831/3377749206","state":["exists","up"]},{"osd":10,"uuid":"8741d25f-40dd-40d7-b040-5cc5e411eb98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6825","nonce":885058786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6827","nonce":885058786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6831","nonce":885058786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6829","nonce":885058786}]},"public_addr":"172.21.15.89:6825/885058786","cluster_addr":"172.21.15.89:6827/885058786","heartbeat_back_addr":"172.21.15.89:6831/885058786","heartbeat_front_addr":"172.21.15.89:6829/885058786","state":["exists","up"]},{"osd":11,"uuid":"8d02e957-eda4-4184-b5c6-5eb3b8b38060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6825","nonce":1517095838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6827","nonce":1517095838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6831","nonce":1517095838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6829","nonce":1517095838}]},"public_addr":"172.21.15.98:6825/1517095838","cluster_addr":"172.21.15.98:6827/1517095838","heartbeat_back_addr":"172.21.15.98:6831/1517095838","heartbeat_front_addr":"172.21.15.98:6829/1517095838","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:28:47.307652+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:11.037135+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:34.619188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:59.350738+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:30:23.787337+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:30:48.994154+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:31:13.690350+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:31:39.628116+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:04.723912+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:30.689741+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:57.229508+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:33:23.154537+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:0/121087196":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/2451857761":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/1228477995":"2024-03-23T01:27:15.339809+0000","172.21.15.72:6801/1513232913":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/3558958264":"2024-03-23T01:26:50.074871+0000","172.21.15.72:0/1510187198":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6800/3518012916":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6801/3518012916":"2024-03-23T01:26:50.074871+0000","172.21.15.72:0/1372801041":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6800/1513232913":"2024-03-23T01:27:15.339809+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T01:33:35.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.227+0000 7f2f28ec3640 1 -- 172.21.15.72:0/4039869606 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f2f0c0a2f20 msgr2=0x7f2f0c0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:35.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.227+0000 7f2f28ec3640 1 --2- 172.21.15.72:0/4039869606 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f2f0c0a2f20 0x7f2f0c0a53e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2f14009fc0 tx=0x7f2f14005f90 comp rx=0 tx=0).stop 2024-03-22T01:33:35.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.227+0000 7f2f28ec3640 1 -- 172.21.15.72:0/4039869606 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f24104fe0 msgr2=0x7f2f24198300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:35.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.227+0000 7f2f28ec3640 1 --2- 172.21.15.72:0/4039869606 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f24104fe0 0x7f2f24198300 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f2f1800c970 tx=0x7f2f1800ce30 comp rx=0 tx=0).stop 2024-03-22T01:33:35.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.227+0000 7f2f28ec3640 1 -- 172.21.15.72:0/4039869606 shutdown_connections 2024-03-22T01:33:35.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.227+0000 7f2f28ec3640 1 --2- 172.21.15.72:0/4039869606 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f2f0c0a2f20 0x7f2f0c0a53e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:35.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.227+0000 7f2f28ec3640 1 --2- 172.21.15.72:0/4039869606 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f24107910 0x7f2f2419f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:35.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.227+0000 7f2f28ec3640 1 --2- 172.21.15.72:0/4039869606 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f24104fe0 0x7f2f24198300 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:35.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.227+0000 7f2f28ec3640 1 --2- 172.21.15.72:0/4039869606 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f241026b0 0x7f2f24197dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:35.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.227+0000 7f2f28ec3640 1 -- 172.21.15.72:0/4039869606 >> 172.21.15.72:0/4039869606 conn(0x7f2f240fc0a0 msgr2=0x7f2f24102ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:33:35.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.227+0000 7f2f28ec3640 1 -- 172.21.15.72:0/4039869606 shutdown_connections 2024-03-22T01:33:35.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.227+0000 7f2f28ec3640 1 -- 172.21.15.72:0/4039869606 wait complete. 2024-03-22T01:33:35.237 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-22T01:29:37.438503+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-22T01:33:35.238 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-22T01:33:35.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.343+0000 7f44b9fc9640 1 -- 172.21.15.72:0/604540003 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44b4103750 msgr2=0x7f44b4109fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:35.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.343+0000 7f44b9fc9640 1 --2- 172.21.15.72:0/604540003 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44b4103750 0x7f44b4109fe0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f44ac009ca0 tx=0x7f44ac02c0f0 comp rx=0 tx=0).stop 2024-03-22T01:33:35.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.343+0000 7f44b9fc9640 1 -- 172.21.15.72:0/604540003 shutdown_connections 2024-03-22T01:33:35.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.343+0000 7f44b9fc9640 1 --2- 172.21.15.72:0/604540003 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44b4102d90 0x7f44b4103210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:35.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.343+0000 7f44b9fc9640 1 --2- 172.21.15.72:0/604540003 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44b4103750 0x7f44b4109fe0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:35.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.343+0000 7f44b9fc9640 1 --2- 172.21.15.72:0/604540003 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44b4101ba0 0x7f44b4101fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:35.345 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.343+0000 7f44b9fc9640 1 -- 172.21.15.72:0/604540003 >> 172.21.15.72:0/604540003 conn(0x7f44b40fd110 msgr2=0x7f44b40ff550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:33:35.345 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.343+0000 7f44b9fc9640 1 -- 172.21.15.72:0/604540003 shutdown_connections 2024-03-22T01:33:35.345 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.343+0000 7f44b9fc9640 1 -- 172.21.15.72:0/604540003 wait complete. 2024-03-22T01:33:35.345 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.343+0000 7f44b9fc9640 1 Processor -- start 2024-03-22T01:33:35.345 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.343+0000 7f44b9fc9640 1 -- start start 2024-03-22T01:33:35.345 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.344+0000 7f44b9fc9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44b4101ba0 0x7f44b4075940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:35.345 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.344+0000 7f44b9fc9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44b4102d90 0x7f44b4073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:35.345 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.344+0000 7f44b9fc9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44b4103750 0x7f44b40744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:35.345 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.344+0000 7f44b9fc9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f44b410d820 con 0x7f44b4103750 2024-03-22T01:33:35.345 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.344+0000 7f44b9fc9640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f44b410d6a0 con 0x7f44b4101ba0 2024-03-22T01:33:35.345 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.344+0000 7f44b9fc9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f44b410d9a0 con 0x7f44b4102d90 2024-03-22T01:33:35.345 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.344+0000 7f44b3fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44b4103750 0x7f44b40744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:35.345 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.344+0000 7f44b3fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44b4103750 0x7f44b40744c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33354/0 (socket says 172.21.15.72:33354) 2024-03-22T01:33:35.345 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.344+0000 7f44b3fff640 1 -- 172.21.15.72:0/505497709 learned_addr learned my addr 172.21.15.72:0/505497709 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:33:35.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.344+0000 7f44aaffd640 1 --2- 172.21.15.72:0/505497709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44b4102d90 0x7f44b4073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:35.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.344+0000 7f44b37fe640 1 --2- 172.21.15.72:0/505497709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44b4101ba0 0x7f44b4075940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:35.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.344+0000 7f44b3fff640 1 -- 172.21.15.72:0/505497709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44b4102d90 msgr2=0x7f44b4073f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:35.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.344+0000 7f44b3fff640 1 --2- 172.21.15.72:0/505497709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44b4102d90 0x7f44b4073f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:35.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.344+0000 7f44b3fff640 1 -- 172.21.15.72:0/505497709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44b4101ba0 msgr2=0x7f44b4075940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:35.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.344+0000 7f44b3fff640 1 --2- 172.21.15.72:0/505497709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44b4101ba0 0x7f44b4075940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:35.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.344+0000 7f44b3fff640 1 -- 172.21.15.72:0/505497709 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44b4074d50 con 0x7f44b4103750 2024-03-22T01:33:35.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.344+0000 7f44b37fe640 1 --2- 172.21.15.72:0/505497709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44b4101ba0 0x7f44b4075940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:33:35.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.344+0000 7f44aaffd640 1 --2- 172.21.15.72:0/505497709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44b4102d90 0x7f44b4073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:33:35.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.344+0000 7f44b3fff640 1 --2- 172.21.15.72:0/505497709 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44b4103750 0x7f44b40744c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f44ac02c700 tx=0x7f44ac032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:33:35.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.344+0000 7f44b17fa640 1 -- 172.21.15.72:0/505497709 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44ac039040 con 0x7f44b4103750 2024-03-22T01:33:35.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.344+0000 7f44b17fa640 1 -- 172.21.15.72:0/505497709 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44ac044070 con 0x7f44b4103750 2024-03-22T01:33:35.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.344+0000 7f44b17fa640 1 -- 172.21.15.72:0/505497709 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44ac0042a0 con 0x7f44b4103750 2024-03-22T01:33:35.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.344+0000 7f44b9fc9640 1 -- 172.21.15.72:0/505497709 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44b41a1f30 con 0x7f44b4103750 2024-03-22T01:33:35.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.344+0000 7f44b9fc9640 1 -- 172.21.15.72:0/505497709 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44b41a2390 con 0x7f44b4103750 2024-03-22T01:33:35.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.345+0000 7f44b9fc9640 1 -- 172.21.15.72:0/505497709 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44b4108ec0 con 0x7f44b4103750 2024-03-22T01:33:35.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.346+0000 7f44b17fa640 1 -- 172.21.15.72:0/505497709 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f44ac003970 con 0x7f44b4103750 2024-03-22T01:33:35.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.347+0000 7f44b17fa640 1 --2- 172.21.15.72:0/505497709 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f44a00a2f20 0x7f44a00a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:35.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.347+0000 7f44b37fe640 1 --2- 172.21.15.72:0/505497709 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f44a00a2f20 0x7f44a00a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:35.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.347+0000 7f44b17fa640 1 -- 172.21.15.72:0/505497709 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f44ac0f1b40 con 0x7f44b4103750 2024-03-22T01:33:35.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.347+0000 7f44b37fe640 1 --2- 172.21.15.72:0/505497709 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f44a00a2f20 0x7f44a00a53e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f44a40058c0 tx=0x7f44a4005ec0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:33:35.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.351+0000 7f44b17fa640 1 -- 172.21.15.72:0/505497709 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f44ac0bee00 con 0x7f44b4103750 2024-03-22T01:33:35.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.525+0000 7f44b9fc9640 1 -- 172.21.15.72:0/505497709 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f44b402d3b0 con 0x7f44b4103750 2024-03-22T01:33:35.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.526+0000 7f44b17fa640 1 -- 172.21.15.72:0/505497709 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) ==== 93+0+10 (secure 0 0 0) 0x7f44ac0be700 con 0x7f44b4103750 2024-03-22T01:33:35.526 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-22T01:33:35.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.528+0000 7f44b9fc9640 1 -- 172.21.15.72:0/505497709 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f44a00a2f20 msgr2=0x7f44a00a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:35.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.528+0000 7f44b9fc9640 1 --2- 172.21.15.72:0/505497709 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f44a00a2f20 0x7f44a00a53e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f44a40058c0 tx=0x7f44a4005ec0 comp rx=0 tx=0).stop 2024-03-22T01:33:35.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.528+0000 7f44b9fc9640 1 -- 172.21.15.72:0/505497709 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44b4103750 msgr2=0x7f44b40744c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:35.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.528+0000 7f44b9fc9640 1 --2- 172.21.15.72:0/505497709 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44b4103750 0x7f44b40744c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f44ac02c700 tx=0x7f44ac032f20 comp rx=0 tx=0).stop 2024-03-22T01:33:35.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.528+0000 7f44b9fc9640 1 -- 172.21.15.72:0/505497709 shutdown_connections 2024-03-22T01:33:35.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.528+0000 7f44b9fc9640 1 --2- 172.21.15.72:0/505497709 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f44a00a2f20 0x7f44a00a53e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:35.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.528+0000 7f44b9fc9640 1 --2- 172.21.15.72:0/505497709 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44b4103750 0x7f44b40744c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:35.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.528+0000 7f44b9fc9640 1 --2- 172.21.15.72:0/505497709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44b4102d90 0x7f44b4073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:35.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.528+0000 7f44b9fc9640 1 --2- 172.21.15.72:0/505497709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44b4101ba0 0x7f44b4075940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:35.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.528+0000 7f44b9fc9640 1 -- 172.21.15.72:0/505497709 >> 172.21.15.72:0/505497709 conn(0x7f44b40fd110 msgr2=0x7f44b4106090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:33:35.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.528+0000 7f44b9fc9640 1 -- 172.21.15.72:0/505497709 shutdown_connections 2024-03-22T01:33:35.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.528+0000 7f44b9fc9640 1 -- 172.21.15.72:0/505497709 wait complete. 2024-03-22T01:33:35.538 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T01:33:35.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.642+0000 7f04ec9d7640 1 -- 172.21.15.72:0/3008684984 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04e8104f70 msgr2=0x7f04e8107360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:35.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.642+0000 7f04ec9d7640 1 --2- 172.21.15.72:0/3008684984 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04e8104f70 0x7f04e8107360 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f04d0009900 tx=0x7f04d002bc30 comp rx=0 tx=0).stop 2024-03-22T01:33:35.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.642+0000 7f04ec9d7640 1 -- 172.21.15.72:0/3008684984 shutdown_connections 2024-03-22T01:33:35.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.642+0000 7f04ec9d7640 1 --2- 172.21.15.72:0/3008684984 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04e81078a0 0x7f04e8109cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:35.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.642+0000 7f04ec9d7640 1 --2- 172.21.15.72:0/3008684984 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04e8104f70 0x7f04e8107360 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:35.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.642+0000 7f04ec9d7640 1 --2- 172.21.15.72:0/3008684984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04e8102640 0x7f04e8104a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:35.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.642+0000 7f04ec9d7640 1 -- 172.21.15.72:0/3008684984 >> 172.21.15.72:0/3008684984 conn(0x7f04e80fc030 msgr2=0x7f04e80fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:33:35.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.642+0000 7f04ec9d7640 1 -- 172.21.15.72:0/3008684984 shutdown_connections 2024-03-22T01:33:35.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.642+0000 7f04ec9d7640 1 -- 172.21.15.72:0/3008684984 wait complete. 2024-03-22T01:33:35.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.643+0000 7f04ec9d7640 1 Processor -- start 2024-03-22T01:33:35.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.643+0000 7f04ec9d7640 1 -- start start 2024-03-22T01:33:35.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.643+0000 7f04ec9d7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04e8102640 0x7f04e8197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:35.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.643+0000 7f04ec9d7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04e8104f70 0x7f04e8198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:35.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.643+0000 7f04ec9d7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04e81078a0 0x7f04e819f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:35.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.643+0000 7f04ec9d7640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f04e810d850 con 0x7f04e8104f70 2024-03-22T01:33:35.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.643+0000 7f04ec9d7640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f04e810d6d0 con 0x7f04e81078a0 2024-03-22T01:33:35.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.643+0000 7f04ec9d7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f04e810d9d0 con 0x7f04e8102640 2024-03-22T01:33:35.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.643+0000 7f04e5d9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04e8104f70 0x7f04e8198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:35.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.643+0000 7f04e5d9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04e8104f70 0x7f04e8198440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33386/0 (socket says 172.21.15.72:33386) 2024-03-22T01:33:35.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.643+0000 7f04e5d9b640 1 -- 172.21.15.72:0/2496718547 learned_addr learned my addr 172.21.15.72:0/2496718547 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:33:35.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.644+0000 7f04e659c640 1 --2- 172.21.15.72:0/2496718547 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04e8102640 0x7f04e8197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:35.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.644+0000 7f04e6d9d640 1 --2- 172.21.15.72:0/2496718547 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04e81078a0 0x7f04e819f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:35.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.644+0000 7f04e5d9b640 1 -- 172.21.15.72:0/2496718547 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04e8102640 msgr2=0x7f04e8197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:35.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.644+0000 7f04e5d9b640 1 --2- 172.21.15.72:0/2496718547 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04e8102640 0x7f04e8197f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:35.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.644+0000 7f04e5d9b640 1 -- 172.21.15.72:0/2496718547 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04e81078a0 msgr2=0x7f04e819f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:35.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.644+0000 7f04e5d9b640 1 --2- 172.21.15.72:0/2496718547 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04e81078a0 0x7f04e819f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:35.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.644+0000 7f04e5d9b640 1 -- 172.21.15.72:0/2496718547 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04e819fb20 con 0x7f04e8104f70 2024-03-22T01:33:35.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.644+0000 7f04e6d9d640 1 --2- 172.21.15.72:0/2496718547 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04e81078a0 0x7f04e819f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T01:33:35.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.644+0000 7f04e5d9b640 1 --2- 172.21.15.72:0/2496718547 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04e8104f70 0x7f04e8198440 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f04d0000c00 tx=0x7f04d0005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:33:35.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.644+0000 7f04df7fe640 1 -- 172.21.15.72:0/2496718547 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04d0041070 con 0x7f04e8104f70 2024-03-22T01:33:35.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.644+0000 7f04ec9d7640 1 -- 172.21.15.72:0/2496718547 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04e819fdb0 con 0x7f04e8104f70 2024-03-22T01:33:35.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.644+0000 7f04df7fe640 1 -- 172.21.15.72:0/2496718547 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04d00044d0 con 0x7f04e8104f70 2024-03-22T01:33:35.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.644+0000 7f04df7fe640 1 -- 172.21.15.72:0/2496718547 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04d0034440 con 0x7f04e8104f70 2024-03-22T01:33:35.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.644+0000 7f04ec9d7640 1 -- 172.21.15.72:0/2496718547 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04e81a0260 con 0x7f04e8104f70 2024-03-22T01:33:35.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.645+0000 7f04ec9d7640 1 -- 172.21.15.72:0/2496718547 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04e80fdb20 con 0x7f04e8104f70 2024-03-22T01:33:35.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.646+0000 7f04df7fe640 1 -- 172.21.15.72:0/2496718547 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f04d0044070 con 0x7f04e8104f70 2024-03-22T01:33:35.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.646+0000 7f04df7fe640 1 --2- 172.21.15.72:0/2496718547 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f04d40a2f20 0x7f04d40a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:35.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.647+0000 7f04e659c640 1 --2- 172.21.15.72:0/2496718547 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f04d40a2f20 0x7f04d40a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:35.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.647+0000 7f04df7fe640 1 -- 172.21.15.72:0/2496718547 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f04d00f0ac0 con 0x7f04e8104f70 2024-03-22T01:33:35.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.647+0000 7f04e659c640 1 --2- 172.21.15.72:0/2496718547 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f04d40a2f20 0x7f04d40a53e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f04d8005fd0 tx=0x7f04d8009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:33:35.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.651+0000 7f04df7fe640 1 -- 172.21.15.72:0/2496718547 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f04d00bdd90 con 0x7f04e8104f70 2024-03-22T01:33:35.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:35 smithi098 ceph-mon[36468]: from='client.24415 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi072=a;smithi072=d;smithi072=g;smithi072=j;smithi089=b;smithi089=e;smithi089=h;smithi089=k;smithi098=c;smithi098=f;smithi098=i;smithi098=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:33:35.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:35 smithi098 ceph-mon[36468]: Saving service mds.all spec with placement smithi072=a;smithi072=d;smithi072=g;smithi072=j;smithi089=b;smithi089=e;smithi089=h;smithi089=k;smithi098=c;smithi098=f;smithi098=i;smithi098=l;count:12 2024-03-22T01:33:35.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:35 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:33:35.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:35 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:33:35.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:35 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:35.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:35 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T01:33:35.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:35 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T01:33:35.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:35 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:33:35.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:35 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/4039869606' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:33:35.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:35 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/505497709' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T01:33:35.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:35 smithi089 ceph-mon[35770]: from='client.24415 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi072=a;smithi072=d;smithi072=g;smithi072=j;smithi089=b;smithi089=e;smithi089=h;smithi089=k;smithi098=c;smithi098=f;smithi098=i;smithi098=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:33:35.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:35 smithi089 ceph-mon[35770]: Saving service mds.all spec with placement smithi072=a;smithi072=d;smithi072=g;smithi072=j;smithi089=b;smithi089=e;smithi089=h;smithi089=k;smithi098=c;smithi098=f;smithi098=i;smithi098=l;count:12 2024-03-22T01:33:35.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:35 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:33:35.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:35 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:33:35.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:35 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:35.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:35 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T01:33:35.859 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:35 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T01:33:35.859 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:35 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:33:35.859 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:35 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/4039869606' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:33:35.859 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:35 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/505497709' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T01:33:35.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.864+0000 7f04ec9d7640 1 -- 172.21.15.72:0/2496718547 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f04e802d3b0 con 0x7f04e8104f70 2024-03-22T01:33:35.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.864+0000 7f04df7fe640 1 -- 172.21.15.72:0/2496718547 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f04d00bd690 con 0x7f04e8104f70 2024-03-22T01:33:35.863 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:33:35.864 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-22T01:33:35.864 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 1 2024-03-22T01:33:35.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.866+0000 7f04ec9d7640 1 -- 172.21.15.72:0/2496718547 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f04d40a2f20 msgr2=0x7f04d40a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:35.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.866+0000 7f04ec9d7640 1 --2- 172.21.15.72:0/2496718547 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f04d40a2f20 0x7f04d40a53e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f04d8005fd0 tx=0x7f04d8009500 comp rx=0 tx=0).stop 2024-03-22T01:33:35.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.867+0000 7f04ec9d7640 1 -- 172.21.15.72:0/2496718547 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04e8104f70 msgr2=0x7f04e8198440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:35.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.867+0000 7f04ec9d7640 1 --2- 172.21.15.72:0/2496718547 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04e8104f70 0x7f04e8198440 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f04d0000c00 tx=0x7f04d0005d50 comp rx=0 tx=0).stop 2024-03-22T01:33:35.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.867+0000 7f04ec9d7640 1 -- 172.21.15.72:0/2496718547 shutdown_connections 2024-03-22T01:33:35.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.867+0000 7f04ec9d7640 1 --2- 172.21.15.72:0/2496718547 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f04d40a2f20 0x7f04d40a53e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:35.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.867+0000 7f04ec9d7640 1 --2- 172.21.15.72:0/2496718547 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04e81078a0 0x7f04e819f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:35.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.867+0000 7f04ec9d7640 1 --2- 172.21.15.72:0/2496718547 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04e8104f70 0x7f04e8198440 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:35.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.867+0000 7f04ec9d7640 1 --2- 172.21.15.72:0/2496718547 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04e8102640 0x7f04e8197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:35.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.867+0000 7f04ec9d7640 1 -- 172.21.15.72:0/2496718547 >> 172.21.15.72:0/2496718547 conn(0x7f04e80fc030 msgr2=0x7f04e8103080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:33:35.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.867+0000 7f04ec9d7640 1 -- 172.21.15.72:0/2496718547 shutdown_connections 2024-03-22T01:33:35.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:35.867+0000 7f04ec9d7640 1 -- 172.21.15.72:0/2496718547 wait complete. 2024-03-22T01:33:35.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:35 smithi072 ceph-mon[32414]: from='client.24415 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi072=a;smithi072=d;smithi072=g;smithi072=j;smithi089=b;smithi089=e;smithi089=h;smithi089=k;smithi098=c;smithi098=f;smithi098=i;smithi098=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:33:35.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:35 smithi072 ceph-mon[32414]: Saving service mds.all spec with placement smithi072=a;smithi072=d;smithi072=g;smithi072=j;smithi089=b;smithi089=e;smithi089=h;smithi089=k;smithi098=c;smithi098=f;smithi098=i;smithi098=l;count:12 2024-03-22T01:33:35.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:35 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:33:35.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:35 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:33:35.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:35 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:35.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:35 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T01:33:35.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:35 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T01:33:35.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:35 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:33:35.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:35 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/4039869606' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:33:35.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:35 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/505497709' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T01:33:36.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:36 smithi098 ceph-mon[36468]: Deploying daemon mds.l on smithi098 2024-03-22T01:33:36.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:36 smithi098 ceph-mon[36468]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:33:36.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:36 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/2496718547' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:33:36.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:36 smithi089 ceph-mon[35770]: Deploying daemon mds.l on smithi098 2024-03-22T01:33:36.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:36 smithi089 ceph-mon[35770]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:33:36.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:36 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/2496718547' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:33:36.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:36 smithi072 ceph-mon[32414]: Deploying daemon mds.l on smithi098 2024-03-22T01:33:36.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:36 smithi072 ceph-mon[32414]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:33:36.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:36 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/2496718547' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:33:37.171 INFO:journalctl@ceph.mds.l.smithi098.stdout:Mar 22 01:33:37 smithi098 systemd[1]: Starting Ceph mds.l for 022a99bc-e7eb-11ee-95cd-87774f69a715... 2024-03-22T01:33:37.485 INFO:journalctl@ceph.mds.l.smithi098.stdout:Mar 22 01:33:37 smithi098 podman[67372]: 2024-03-22 01:33:37.138789346 +0000 UTC m=+0.017671405 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T01:33:37.486 INFO:journalctl@ceph.mds.l.smithi098.stdout:Mar 22 01:33:37 smithi098 podman[67372]: 2024-03-22 01:33:37.24323901 +0000 UTC m=+0.122121077 container create f5b0a765eaa3ada11ad05731813a5a84b01d60ad12699c16f0403bd50f1d2deb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-22T01:33:37.877 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T01:33:37.921 INFO:journalctl@ceph.mds.l.smithi098.stdout:Mar 22 01:33:37 smithi098 podman[67372]: 2024-03-22 01:33:37.487574663 +0000 UTC m=+0.366456723 container init f5b0a765eaa3ada11ad05731813a5a84b01d60ad12699c16f0403bd50f1d2deb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-l, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0) 2024-03-22T01:33:37.922 INFO:journalctl@ceph.mds.l.smithi098.stdout:Mar 22 01:33:37 smithi098 podman[67372]: 2024-03-22 01:33:37.490897646 +0000 UTC m=+0.369779697 container start f5b0a765eaa3ada11ad05731813a5a84b01d60ad12699c16f0403bd50f1d2deb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0) 2024-03-22T01:33:37.922 INFO:journalctl@ceph.mds.l.smithi098.stdout:Mar 22 01:33:37 smithi098 ceph-mds[67391]: -- 172.21.15.98:0/1577034876 <== mon.1 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5555d3303a40 con 0x5555d32d6800 2024-03-22T01:33:37.922 INFO:journalctl@ceph.mds.l.smithi098.stdout:Mar 22 01:33:37 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-l[67387]: starting mds.l at 2024-03-22T01:33:37.922 INFO:journalctl@ceph.mds.l.smithi098.stdout:Mar 22 01:33:37 smithi098 bash[67372]: f5b0a765eaa3ada11ad05731813a5a84b01d60ad12699c16f0403bd50f1d2deb 2024-03-22T01:33:37.922 INFO:journalctl@ceph.mds.l.smithi098.stdout:Mar 22 01:33:37 smithi098 systemd[1]: Started Ceph mds.l for 022a99bc-e7eb-11ee-95cd-87774f69a715. 2024-03-22T01:33:37.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:37.992+0000 7f2b3f44e640 1 -- 172.21.15.72:0/1699052207 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b38103790 msgr2=0x7f2b3810a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:37.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:37.992+0000 7f2b3f44e640 1 --2- 172.21.15.72:0/1699052207 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b38103790 0x7f2b3810a020 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2b30009ca0 tx=0x7f2b3002c0f0 comp rx=0 tx=0).stop 2024-03-22T01:33:37.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:37.992+0000 7f2b3f44e640 1 -- 172.21.15.72:0/1699052207 shutdown_connections 2024-03-22T01:33:37.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:37.992+0000 7f2b3f44e640 1 --2- 172.21.15.72:0/1699052207 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b38103790 0x7f2b3810a020 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:37.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:37.992+0000 7f2b3f44e640 1 --2- 172.21.15.72:0/1699052207 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b38102dd0 0x7f2b38103250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:37.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:37.992+0000 7f2b3f44e640 1 --2- 172.21.15.72:0/1699052207 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b38101be0 0x7f2b38102000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:37.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:37.992+0000 7f2b3f44e640 1 -- 172.21.15.72:0/1699052207 >> 172.21.15.72:0/1699052207 conn(0x7f2b380fd150 msgr2=0x7f2b380ff590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:33:37.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:37.992+0000 7f2b3f44e640 1 -- 172.21.15.72:0/1699052207 shutdown_connections 2024-03-22T01:33:37.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:37.992+0000 7f2b3f44e640 1 -- 172.21.15.72:0/1699052207 wait complete. 2024-03-22T01:33:37.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:37.992+0000 7f2b3f44e640 1 Processor -- start 2024-03-22T01:33:37.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:37.992+0000 7f2b3f44e640 1 -- start start 2024-03-22T01:33:37.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:37.993+0000 7f2b3f44e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b38101be0 0x7f2b38197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:37.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:37.993+0000 7f2b3f44e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b38102dd0 0x7f2b381983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:37.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:37.993+0000 7f2b3f44e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b38103790 0x7f2b3819f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:37.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:37.993+0000 7f2b3f44e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f2b3810d860 con 0x7f2b38103790 2024-03-22T01:33:37.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:37.993+0000 7f2b3f44e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2b3810d6e0 con 0x7f2b38102dd0 2024-03-22T01:33:37.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:37.993+0000 7f2b3f44e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2b3810d9e0 con 0x7f2b38101be0 2024-03-22T01:33:37.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:37.993+0000 7f2b3d9eb640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b38103790 0x7f2b3819f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:37.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:37.993+0000 7f2b3d9eb640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b38103790 0x7f2b3819f420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33412/0 (socket says 172.21.15.72:33412) 2024-03-22T01:33:37.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:37.993+0000 7f2b3d9eb640 1 -- 172.21.15.72:0/2066603957 learned_addr learned my addr 172.21.15.72:0/2066603957 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:33:37.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:37.993+0000 7f2b3c9e9640 1 --2- 172.21.15.72:0/2066603957 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b38102dd0 0x7f2b381983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:37.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:37.993+0000 7f2b3d1ea640 1 --2- 172.21.15.72:0/2066603957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b38101be0 0x7f2b38197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:37.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:37.993+0000 7f2b3d9eb640 1 -- 172.21.15.72:0/2066603957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b38101be0 msgr2=0x7f2b38197e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:37.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:37.993+0000 7f2b3d9eb640 1 --2- 172.21.15.72:0/2066603957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b38101be0 0x7f2b38197e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:37.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:37.993+0000 7f2b3d9eb640 1 -- 172.21.15.72:0/2066603957 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b38102dd0 msgr2=0x7f2b381983b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:37.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:37.993+0000 7f2b3d9eb640 1 --2- 172.21.15.72:0/2066603957 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b38102dd0 0x7f2b381983b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:37.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:37.993+0000 7f2b3d9eb640 1 -- 172.21.15.72:0/2066603957 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b3819fa90 con 0x7f2b38103790 2024-03-22T01:33:37.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:37.993+0000 7f2b3d9eb640 1 --2- 172.21.15.72:0/2066603957 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b38103790 0x7f2b3819f420 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2b30037040 tx=0x7f2b30032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:33:37.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:37.993+0000 7f2b2e7fc640 1 -- 172.21.15.72:0/2066603957 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b30041020 con 0x7f2b38103790 2024-03-22T01:33:37.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:37.993+0000 7f2b2e7fc640 1 -- 172.21.15.72:0/2066603957 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b30034d90 con 0x7f2b38103790 2024-03-22T01:33:37.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:37.993+0000 7f2b3f44e640 1 -- 172.21.15.72:0/2066603957 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b3819fd20 con 0x7f2b38103790 2024-03-22T01:33:37.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:37.993+0000 7f2b2e7fc640 1 -- 172.21.15.72:0/2066603957 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b30039040 con 0x7f2b38103790 2024-03-22T01:33:37.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:37.993+0000 7f2b3f44e640 1 -- 172.21.15.72:0/2066603957 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b381a01c0 con 0x7f2b38103790 2024-03-22T01:33:37.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:37.994+0000 7f2b3f44e640 1 -- 172.21.15.72:0/2066603957 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b38108ea0 con 0x7f2b38103790 2024-03-22T01:33:37.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:37.995+0000 7f2b2e7fc640 1 -- 172.21.15.72:0/2066603957 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f2b30044020 con 0x7f2b38103790 2024-03-22T01:33:37.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:37.996+0000 7f2b2e7fc640 1 --2- 172.21.15.72:0/2066603957 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f2b240a2f20 0x7f2b240a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:37.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:37.996+0000 7f2b3d1ea640 1 --2- 172.21.15.72:0/2066603957 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f2b240a2f20 0x7f2b240a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:37.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:37.996+0000 7f2b2e7fc640 1 -- 172.21.15.72:0/2066603957 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f2b300f0f10 con 0x7f2b38103790 2024-03-22T01:33:37.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:37.996+0000 7f2b3d1ea640 1 --2- 172.21.15.72:0/2066603957 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f2b240a2f20 0x7f2b240a53e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2b38102c30 tx=0x7f2b28005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:33:37.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:38.000+0000 7f2b2e7fc640 1 -- 172.21.15.72:0/2066603957 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2b3004b020 con 0x7f2b38103790 2024-03-22T01:33:38.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:38.214+0000 7f2b3f44e640 1 -- 172.21.15.72:0/2066603957 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2b3802d5d0 con 0x7f2b38103790 2024-03-22T01:33:38.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:38.214+0000 7f2b2e7fc640 1 -- 172.21.15.72:0/2066603957 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1393 (secure 0 0 0) 0x7f2b300be1e0 con 0x7f2b38103790 2024-03-22T01:33:38.213 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:33:38.213 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/2703047213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2703047213},{"type":"v1","addr":"172.21.15.98:6833","nonce":2703047213}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-22T01:33:38.214 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 2 2024-03-22T01:33:38.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:38.216+0000 7f2b3f44e640 1 -- 172.21.15.72:0/2066603957 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f2b240a2f20 msgr2=0x7f2b240a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:38.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:38.216+0000 7f2b3f44e640 1 --2- 172.21.15.72:0/2066603957 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f2b240a2f20 0x7f2b240a53e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2b38102c30 tx=0x7f2b28005f90 comp rx=0 tx=0).stop 2024-03-22T01:33:38.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:38.216+0000 7f2b3f44e640 1 -- 172.21.15.72:0/2066603957 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b38103790 msgr2=0x7f2b3819f420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:38.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:38.216+0000 7f2b3f44e640 1 --2- 172.21.15.72:0/2066603957 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b38103790 0x7f2b3819f420 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2b30037040 tx=0x7f2b30032f20 comp rx=0 tx=0).stop 2024-03-22T01:33:38.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:38.216+0000 7f2b3f44e640 1 -- 172.21.15.72:0/2066603957 shutdown_connections 2024-03-22T01:33:38.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:38.216+0000 7f2b3f44e640 1 --2- 172.21.15.72:0/2066603957 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f2b240a2f20 0x7f2b240a53e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:38.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:38.216+0000 7f2b3f44e640 1 --2- 172.21.15.72:0/2066603957 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b38102dd0 0x7f2b381983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:38.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:38.216+0000 7f2b3f44e640 1 --2- 172.21.15.72:0/2066603957 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b38103790 0x7f2b3819f420 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:38.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:38.216+0000 7f2b3f44e640 1 --2- 172.21.15.72:0/2066603957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b38101be0 0x7f2b38197e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:38.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:38.216+0000 7f2b3f44e640 1 -- 172.21.15.72:0/2066603957 >> 172.21.15.72:0/2066603957 conn(0x7f2b380fd150 msgr2=0x7f2b38106070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:33:38.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:38.216+0000 7f2b3f44e640 1 -- 172.21.15.72:0/2066603957 shutdown_connections 2024-03-22T01:33:38.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:38.216+0000 7f2b3f44e640 1 -- 172.21.15.72:0/2066603957 wait complete. 2024-03-22T01:33:38.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:38 smithi089 ceph-mon[35770]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:33:38.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:38 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.98:6832/2703047213,v1:172.21.15.98:6833/2703047213] up:boot 2024-03-22T01:33:38.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:38 smithi089 ceph-mon[35770]: fsmap 1 up:standby 2024-03-22T01:33:38.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:38 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-22T01:33:38.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:38 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:38.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:38 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:38.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:38 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:38.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:38 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T01:33:38.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:38 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T01:33:38.859 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:38 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:33:38.859 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:38 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/2066603957' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:33:38.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:38 smithi072 ceph-mon[32414]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:33:38.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:38 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.98:6832/2703047213,v1:172.21.15.98:6833/2703047213] up:boot 2024-03-22T01:33:38.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:38 smithi072 ceph-mon[32414]: fsmap 1 up:standby 2024-03-22T01:33:38.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:38 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-22T01:33:38.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:38 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:38.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:38 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:38.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:38 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:38.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:38 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T01:33:38.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:38 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T01:33:38.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:38 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:33:38.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:38 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/2066603957' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:33:38.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:38 smithi098 ceph-mon[36468]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:33:38.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:38 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.98:6832/2703047213,v1:172.21.15.98:6833/2703047213] up:boot 2024-03-22T01:33:38.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:38 smithi098 ceph-mon[36468]: fsmap 1 up:standby 2024-03-22T01:33:38.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:38 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-22T01:33:38.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:38 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:38.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:38 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:38.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:38 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:38.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:38 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T01:33:38.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:38 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T01:33:38.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:38 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:33:38.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:38 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/2066603957' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:33:39.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:39 smithi089 ceph-mon[35770]: Deploying daemon mds.k on smithi089 2024-03-22T01:33:39.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:39 smithi072 ceph-mon[32414]: Deploying daemon mds.k on smithi089 2024-03-22T01:33:39.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:39 smithi098 ceph-mon[36468]: Deploying daemon mds.k on smithi089 2024-03-22T01:33:40.227 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T01:33:40.284 INFO:journalctl@ceph.mds.k.smithi089.stdout:Mar 22 01:33:40 smithi089 systemd[1]: Starting Ceph mds.k for 022a99bc-e7eb-11ee-95cd-87774f69a715... 2024-03-22T01:33:40.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:40.341+0000 7fcde03bc640 1 -- 172.21.15.72:0/213452098 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcdd8102e60 msgr2=0x7fcdd81032e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:40.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:40.341+0000 7fcde03bc640 1 --2- 172.21.15.72:0/213452098 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcdd8102e60 0x7fcdd81032e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fcdd4009930 tx=0x7fcdd402bc30 comp rx=0 tx=0).stop 2024-03-22T01:33:40.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:40.341+0000 7fcde03bc640 1 -- 172.21.15.72:0/213452098 shutdown_connections 2024-03-22T01:33:40.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:40.341+0000 7fcde03bc640 1 --2- 172.21.15.72:0/213452098 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdd8103820 0x7fcdd810a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:40.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:40.341+0000 7fcde03bc640 1 --2- 172.21.15.72:0/213452098 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcdd8102e60 0x7fcdd81032e0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:40.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:40.341+0000 7fcde03bc640 1 --2- 172.21.15.72:0/213452098 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdd80ffe90 0x7fcdd81002b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:40.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:40.341+0000 7fcde03bc640 1 -- 172.21.15.72:0/213452098 >> 172.21.15.72:0/213452098 conn(0x7fcdd80fb7c0 msgr2=0x7fcdd80fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:33:40.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:40.341+0000 7fcde03bc640 1 -- 172.21.15.72:0/213452098 shutdown_connections 2024-03-22T01:33:40.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:40.341+0000 7fcde03bc640 1 -- 172.21.15.72:0/213452098 wait complete. 2024-03-22T01:33:40.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:40.342+0000 7fcde03bc640 1 Processor -- start 2024-03-22T01:33:40.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:40.342+0000 7fcde03bc640 1 -- start start 2024-03-22T01:33:40.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:40.342+0000 7fcde03bc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdd80ffe90 0x7fcdd8197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:40.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:40.342+0000 7fcde03bc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdd8102e60 0x7fcdd8198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:40.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:40.342+0000 7fcde03bc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcdd8103820 0x7fcdd819f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:40.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:40.342+0000 7fcde03bc640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd810d850 con 0x7fcdd8103820 2024-03-22T01:33:40.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:40.342+0000 7fcde03bc640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd810d6d0 con 0x7fcdd80ffe90 2024-03-22T01:33:40.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:40.342+0000 7fcde03bc640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd810d9d0 con 0x7fcdd8102e60 2024-03-22T01:33:40.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:40.342+0000 7fcdde959640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcdd8103820 0x7fcdd819f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:40.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:40.342+0000 7fcdde158640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdd80ffe90 0x7fcdd8197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:40.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:40.342+0000 7fcdde959640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcdd8103820 0x7fcdd819f4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33434/0 (socket says 172.21.15.72:33434) 2024-03-22T01:33:40.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:40.342+0000 7fcdde959640 1 -- 172.21.15.72:0/3134524303 learned_addr learned my addr 172.21.15.72:0/3134524303 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:33:40.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:40.342+0000 7fcddd957640 1 --2- 172.21.15.72:0/3134524303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdd8102e60 0x7fcdd8198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:40.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:40.342+0000 7fcdde959640 1 -- 172.21.15.72:0/3134524303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdd8102e60 msgr2=0x7fcdd8198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:40.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:40.342+0000 7fcdde959640 1 --2- 172.21.15.72:0/3134524303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdd8102e60 0x7fcdd8198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:40.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:40.342+0000 7fcdde959640 1 -- 172.21.15.72:0/3134524303 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdd80ffe90 msgr2=0x7fcdd8197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:40.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:40.342+0000 7fcdde959640 1 --2- 172.21.15.72:0/3134524303 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdd80ffe90 0x7fcdd8197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:40.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:40.342+0000 7fcdde959640 1 -- 172.21.15.72:0/3134524303 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdd819fb10 con 0x7fcdd8103820 2024-03-22T01:33:40.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:40.342+0000 7fcddd957640 1 --2- 172.21.15.72:0/3134524303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdd8102e60 0x7fcdd8198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:33:40.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:40.342+0000 7fcdde158640 1 --2- 172.21.15.72:0/3134524303 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdd80ffe90 0x7fcdd8197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:33:40.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:40.342+0000 7fcdde959640 1 --2- 172.21.15.72:0/3134524303 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcdd8103820 0x7fcdd819f4a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fcdd000eb10 tx=0x7fcdd000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:33:40.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:40.342+0000 7fcdcf7fe640 1 -- 172.21.15.72:0/3134524303 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdd0004b20 con 0x7fcdd8103820 2024-03-22T01:33:40.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:40.343+0000 7fcde03bc640 1 -- 172.21.15.72:0/3134524303 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdd819fe00 con 0x7fcdd8103820 2024-03-22T01:33:40.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:40.343+0000 7fcdcf7fe640 1 -- 172.21.15.72:0/3134524303 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcdd0004cc0 con 0x7fcdd8103820 2024-03-22T01:33:40.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:40.343+0000 7fcdcf7fe640 1 -- 172.21.15.72:0/3134524303 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdd0010760 con 0x7fcdd8103820 2024-03-22T01:33:40.345 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:40.343+0000 7fcde03bc640 1 -- 172.21.15.72:0/3134524303 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdd81a0270 con 0x7fcdd8103820 2024-03-22T01:33:40.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:40.344+0000 7fcde03bc640 1 -- 172.21.15.72:0/3134524303 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdd8109010 con 0x7fcdd8103820 2024-03-22T01:33:40.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:40.345+0000 7fcdcf7fe640 1 -- 172.21.15.72:0/3134524303 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7fcdd0005020 con 0x7fcdd8103820 2024-03-22T01:33:40.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:40.345+0000 7fcdcf7fe640 1 --2- 172.21.15.72:0/3134524303 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fcdc40a2f20 0x7fcdc40a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:40.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:40.345+0000 7fcdde158640 1 --2- 172.21.15.72:0/3134524303 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fcdc40a2f20 0x7fcdc40a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:40.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:40.345+0000 7fcdcf7fe640 1 -- 172.21.15.72:0/3134524303 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fcdd00cf4f0 con 0x7fcdd8103820 2024-03-22T01:33:40.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:40.346+0000 7fcdde158640 1 --2- 172.21.15.72:0/3134524303 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fcdc40a2f20 0x7fcdc40a53e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fcdc8005fd0 tx=0x7fcdc800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:33:40.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:40.350+0000 7fcdcf7fe640 1 -- 172.21.15.72:0/3134524303 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcdd009c7c0 con 0x7fcdd8103820 2024-03-22T01:33:40.539 INFO:journalctl@ceph.mds.k.smithi089.stdout:Mar 22 01:33:40 smithi089 podman[65849]: 2024-03-22 01:33:40.188355049 +0000 UTC m=+0.017719353 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T01:33:40.540 INFO:journalctl@ceph.mds.k.smithi089.stdout:Mar 22 01:33:40 smithi089 podman[65849]: 2024-03-22 01:33:40.28987305 +0000 UTC m=+0.119237335 container create de4cb18e4d6407484e1ef0e2e27bbc47b8dacbac71dfc800b2ac7d510a3b2fb5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-k, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.schema-version=1.0) 2024-03-22T01:33:40.540 INFO:journalctl@ceph.mds.k.smithi089.stdout:Mar 22 01:33:40 smithi089 podman[65849]: 2024-03-22 01:33:40.525970485 +0000 UTC m=+0.355334780 container init de4cb18e4d6407484e1ef0e2e27bbc47b8dacbac71dfc800b2ac7d510a3b2fb5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-k, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-22T01:33:40.540 INFO:journalctl@ceph.mds.k.smithi089.stdout:Mar 22 01:33:40 smithi089 podman[65849]: 2024-03-22 01:33:40.528952904 +0000 UTC m=+0.358317199 container start de4cb18e4d6407484e1ef0e2e27bbc47b8dacbac71dfc800b2ac7d510a3b2fb5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-k, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319) 2024-03-22T01:33:40.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:40.566+0000 7fcde03bc640 1 -- 172.21.15.72:0/3134524303 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcdd802d580 con 0x7fcdd8103820 2024-03-22T01:33:40.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:40.567+0000 7fcdcf7fe640 1 -- 172.21.15.72:0/3134524303 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1393 (secure 0 0 0) 0x7fcdd009c0c0 con 0x7fcdd8103820 2024-03-22T01:33:40.566 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:33:40.566 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/2703047213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2703047213},{"type":"v1","addr":"172.21.15.98:6833","nonce":2703047213}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-22T01:33:40.566 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 2 2024-03-22T01:33:40.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:40.569+0000 7fcde03bc640 1 -- 172.21.15.72:0/3134524303 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fcdc40a2f20 msgr2=0x7fcdc40a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:40.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:40.569+0000 7fcde03bc640 1 --2- 172.21.15.72:0/3134524303 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fcdc40a2f20 0x7fcdc40a53e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fcdc8005fd0 tx=0x7fcdc800b410 comp rx=0 tx=0).stop 2024-03-22T01:33:40.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:40.569+0000 7fcde03bc640 1 -- 172.21.15.72:0/3134524303 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcdd8103820 msgr2=0x7fcdd819f4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:40.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:40.569+0000 7fcde03bc640 1 --2- 172.21.15.72:0/3134524303 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcdd8103820 0x7fcdd819f4a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fcdd000eb10 tx=0x7fcdd000c2d0 comp rx=0 tx=0).stop 2024-03-22T01:33:40.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:40.569+0000 7fcde03bc640 1 -- 172.21.15.72:0/3134524303 shutdown_connections 2024-03-22T01:33:40.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:40.569+0000 7fcde03bc640 1 --2- 172.21.15.72:0/3134524303 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fcdc40a2f20 0x7fcdc40a53e0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:40.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:40.569+0000 7fcde03bc640 1 --2- 172.21.15.72:0/3134524303 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcdd8103820 0x7fcdd819f4a0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:40.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:40.569+0000 7fcde03bc640 1 --2- 172.21.15.72:0/3134524303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdd8102e60 0x7fcdd8198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:40.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:40.569+0000 7fcde03bc640 1 --2- 172.21.15.72:0/3134524303 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdd80ffe90 0x7fcdd8197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:40.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:40.569+0000 7fcde03bc640 1 -- 172.21.15.72:0/3134524303 >> 172.21.15.72:0/3134524303 conn(0x7fcdd80fb7c0 msgr2=0x7fcdd81061e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:33:40.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:40.569+0000 7fcde03bc640 1 -- 172.21.15.72:0/3134524303 shutdown_connections 2024-03-22T01:33:40.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:40.569+0000 7fcde03bc640 1 -- 172.21.15.72:0/3134524303 wait complete. 2024-03-22T01:33:40.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:40 smithi089 ceph-mon[35770]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:33:40.858 INFO:journalctl@ceph.mds.k.smithi089.stdout:Mar 22 01:33:40 smithi089 ceph-mds[65867]: -- 172.21.15.89:0/3267293183 <== mon.2 v2:172.21.15.89:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5598036f7a40 con 0x5598036cac00 2024-03-22T01:33:40.858 INFO:journalctl@ceph.mds.k.smithi089.stdout:Mar 22 01:33:40 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-k[65863]: starting mds.k at 2024-03-22T01:33:40.858 INFO:journalctl@ceph.mds.k.smithi089.stdout:Mar 22 01:33:40 smithi089 bash[65849]: de4cb18e4d6407484e1ef0e2e27bbc47b8dacbac71dfc800b2ac7d510a3b2fb5 2024-03-22T01:33:40.858 INFO:journalctl@ceph.mds.k.smithi089.stdout:Mar 22 01:33:40 smithi089 systemd[1]: Started Ceph mds.k for 022a99bc-e7eb-11ee-95cd-87774f69a715. 2024-03-22T01:33:40.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:40 smithi072 ceph-mon[32414]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:33:40.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:40 smithi098 ceph-mon[36468]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:33:41.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:41 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/3134524303' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:33:41.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:41 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:41.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:41 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:41.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:41 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:41.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:41 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T01:33:41.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:41 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T01:33:41.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:41 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:33:41.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:41 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/3134524303' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:33:41.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:41 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:41.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:41 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:41.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:41 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:41.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:41 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T01:33:41.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:41 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T01:33:41.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:41 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:33:41.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:41 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/3134524303' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:33:41.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:41 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:41.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:41 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:41.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:41 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:41.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:41 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T01:33:41.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:41 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T01:33:41.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:41 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:33:42.580 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T01:33:42.599 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:42 smithi072 ceph-mon[32414]: Deploying daemon mds.j on smithi072 2024-03-22T01:33:42.599 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:42 smithi072 ceph-mon[32414]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:33:42.599 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:42 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.89:6832/1578701171,v1:172.21.15.89:6833/1578701171] up:boot 2024-03-22T01:33:42.599 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:42 smithi072 ceph-mon[32414]: fsmap 2 up:standby 2024-03-22T01:33:42.599 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:42 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-22T01:33:42.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:42.836+0000 7f51ae772640 1 -- 172.21.15.72:0/4174187938 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51a8103820 msgr2=0x7f51a810a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:42.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:42.836+0000 7f51ae772640 1 --2- 172.21.15.72:0/4174187938 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51a8103820 0x7f51a810a0b0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f51a0009cd0 tx=0x7f51a002c0f0 comp rx=0 tx=0).stop 2024-03-22T01:33:42.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:42.837+0000 7f51ae772640 1 -- 172.21.15.72:0/4174187938 shutdown_connections 2024-03-22T01:33:42.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:42.837+0000 7f51ae772640 1 --2- 172.21.15.72:0/4174187938 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51a8103820 0x7f51a810a0b0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:42.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:42.837+0000 7f51ae772640 1 --2- 172.21.15.72:0/4174187938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51a8102e60 0x7f51a81032e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:42.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:42.837+0000 7f51ae772640 1 --2- 172.21.15.72:0/4174187938 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51a80ffe90 0x7f51a81002b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:42.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:42.837+0000 7f51ae772640 1 -- 172.21.15.72:0/4174187938 >> 172.21.15.72:0/4174187938 conn(0x7f51a80fb7c0 msgr2=0x7f51a80fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:33:42.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:42.837+0000 7f51ae772640 1 -- 172.21.15.72:0/4174187938 shutdown_connections 2024-03-22T01:33:42.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:42.837+0000 7f51ae772640 1 -- 172.21.15.72:0/4174187938 wait complete. 2024-03-22T01:33:42.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:42.837+0000 7f51ae772640 1 Processor -- start 2024-03-22T01:33:42.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:42.837+0000 7f51ae772640 1 -- start start 2024-03-22T01:33:42.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:42.837+0000 7f51ae772640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51a80ffe90 0x7f51a81027d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:42.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:42.837+0000 7f51ae772640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51a8102e60 0x7f51a8100e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:42.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:42.837+0000 7f51ae772640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51a8103820 0x7f51a8101350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:42.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:42.837+0000 7f51ae772640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f51a810d870 con 0x7f51a8103820 2024-03-22T01:33:42.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:42.837+0000 7f51ae772640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f51a810d6f0 con 0x7f51a8102e60 2024-03-22T01:33:42.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:42.837+0000 7f51ae772640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f51a810d9f0 con 0x7f51a80ffe90 2024-03-22T01:33:42.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:42.838+0000 7f51acd0f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51a8103820 0x7f51a8101350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:42.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:42.838+0000 7f51acd0f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51a8103820 0x7f51a8101350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:39908/0 (socket says 172.21.15.72:39908) 2024-03-22T01:33:42.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:42.838+0000 7f51acd0f640 1 -- 172.21.15.72:0/377850904 learned_addr learned my addr 172.21.15.72:0/377850904 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:33:42.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:42.838+0000 7f51a77fe640 1 --2- 172.21.15.72:0/377850904 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51a8102e60 0x7f51a8100e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:42.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:42.838+0000 7f51a7fff640 1 --2- 172.21.15.72:0/377850904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51a80ffe90 0x7f51a81027d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:42.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:42.838+0000 7f51acd0f640 1 -- 172.21.15.72:0/377850904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51a80ffe90 msgr2=0x7f51a81027d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:42.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:42.838+0000 7f51acd0f640 1 --2- 172.21.15.72:0/377850904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51a80ffe90 0x7f51a81027d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:42.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:42.838+0000 7f51acd0f640 1 -- 172.21.15.72:0/377850904 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51a8102e60 msgr2=0x7f51a8100e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:42.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:42.838+0000 7f51acd0f640 1 --2- 172.21.15.72:0/377850904 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51a8102e60 0x7f51a8100e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:42.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:42.838+0000 7f51acd0f640 1 -- 172.21.15.72:0/377850904 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51a8101be0 con 0x7f51a8103820 2024-03-22T01:33:42.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:42.838+0000 7f51acd0f640 1 --2- 172.21.15.72:0/377850904 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51a8103820 0x7f51a8101350 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f51a0005e30 tx=0x7f51a00346f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:33:42.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:42.838+0000 7f51a57fa640 1 -- 172.21.15.72:0/377850904 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51a0041020 con 0x7f51a8103820 2024-03-22T01:33:42.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:42.838+0000 7f51ae772640 1 -- 172.21.15.72:0/377850904 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51a81a1f50 con 0x7f51a8103820 2024-03-22T01:33:42.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:42.838+0000 7f51a57fa640 1 -- 172.21.15.72:0/377850904 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51a0032890 con 0x7f51a8103820 2024-03-22T01:33:42.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:42.838+0000 7f51ae772640 1 -- 172.21.15.72:0/377850904 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51a81a2410 con 0x7f51a8103820 2024-03-22T01:33:42.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:42.838+0000 7f51a57fa640 1 -- 172.21.15.72:0/377850904 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51a0039040 con 0x7f51a8103820 2024-03-22T01:33:42.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:42.840+0000 7f51ae772640 1 -- 172.21.15.72:0/377850904 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51940052c0 con 0x7f51a8103820 2024-03-22T01:33:42.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:42.840+0000 7f51a57fa640 1 -- 172.21.15.72:0/377850904 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f51a0044020 con 0x7f51a8103820 2024-03-22T01:33:42.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:42.841+0000 7f51a57fa640 1 --2- 172.21.15.72:0/377850904 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f51900a2fa0 0x7f51900a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:42.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:42.841+0000 7f51a57fa640 1 -- 172.21.15.72:0/377850904 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f51a00f0b30 con 0x7f51a8103820 2024-03-22T01:33:42.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:42.841+0000 7f51a7fff640 1 --2- 172.21.15.72:0/377850904 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f51900a2fa0 0x7f51900a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:42.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:42.842+0000 7f51a7fff640 1 --2- 172.21.15.72:0/377850904 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f51900a2fa0 0x7f51900a5460 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f51a806a090 tx=0x7f5198005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:33:42.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:42.847+0000 7f51a57fa640 1 -- 172.21.15.72:0/377850904 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f51a004b020 con 0x7f51a8103820 2024-03-22T01:33:42.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:42 smithi089 ceph-mon[35770]: Deploying daemon mds.j on smithi072 2024-03-22T01:33:42.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:42 smithi089 ceph-mon[35770]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:33:42.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:42 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.89:6832/1578701171,v1:172.21.15.89:6833/1578701171] up:boot 2024-03-22T01:33:42.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:42 smithi089 ceph-mon[35770]: fsmap 2 up:standby 2024-03-22T01:33:42.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:42 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-22T01:33:42.872 INFO:journalctl@ceph.mds.j.smithi072.stdout:Mar 22 01:33:42 smithi072 systemd[1]: Starting Ceph mds.j for 022a99bc-e7eb-11ee-95cd-87774f69a715... 2024-03-22T01:33:42.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:42 smithi098 ceph-mon[36468]: Deploying daemon mds.j on smithi072 2024-03-22T01:33:42.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:42 smithi098 ceph-mon[36468]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:33:42.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:42 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.89:6832/1578701171,v1:172.21.15.89:6833/1578701171] up:boot 2024-03-22T01:33:42.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:42 smithi098 ceph-mon[36468]: fsmap 2 up:standby 2024-03-22T01:33:42.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:42 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-22T01:33:43.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:43.080+0000 7f51ae772640 1 -- 172.21.15.72:0/377850904 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5194005880 con 0x7f51a8103820 2024-03-22T01:33:43.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:43.080+0000 7f51a57fa640 1 -- 172.21.15.72:0/377850904 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2218 (secure 0 0 0) 0x7f51a00bde00 con 0x7f51a8103820 2024-03-22T01:33:43.080 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:33:43.080 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6833/1578701171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":1578701171},{"type":"v1","addr":"172.21.15.89:6833","nonce":1578701171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/2703047213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2703047213},{"type":"v1","addr":"172.21.15.98:6833","nonce":2703047213}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-22T01:33:43.080 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 3 2024-03-22T01:33:43.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:43.082+0000 7f51ae772640 1 -- 172.21.15.72:0/377850904 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f51900a2fa0 msgr2=0x7f51900a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:43.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:43.082+0000 7f51ae772640 1 --2- 172.21.15.72:0/377850904 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f51900a2fa0 0x7f51900a5460 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f51a806a090 tx=0x7f5198005f90 comp rx=0 tx=0).stop 2024-03-22T01:33:43.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:43.082+0000 7f51ae772640 1 -- 172.21.15.72:0/377850904 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51a8103820 msgr2=0x7f51a8101350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:43.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:43.082+0000 7f51ae772640 1 --2- 172.21.15.72:0/377850904 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51a8103820 0x7f51a8101350 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f51a0005e30 tx=0x7f51a00346f0 comp rx=0 tx=0).stop 2024-03-22T01:33:43.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:43.082+0000 7f51ae772640 1 -- 172.21.15.72:0/377850904 shutdown_connections 2024-03-22T01:33:43.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:43.082+0000 7f51ae772640 1 --2- 172.21.15.72:0/377850904 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f51900a2fa0 0x7f51900a5460 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:43.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:43.082+0000 7f51ae772640 1 --2- 172.21.15.72:0/377850904 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51a8102e60 0x7f51a8100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:43.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:43.082+0000 7f51ae772640 1 --2- 172.21.15.72:0/377850904 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51a8103820 0x7f51a8101350 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:43.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:43.082+0000 7f51ae772640 1 --2- 172.21.15.72:0/377850904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51a80ffe90 0x7f51a81027d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:43.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:43.082+0000 7f51ae772640 1 -- 172.21.15.72:0/377850904 >> 172.21.15.72:0/377850904 conn(0x7f51a80fb7c0 msgr2=0x7f51a8106160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:33:43.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:43.082+0000 7f51ae772640 1 -- 172.21.15.72:0/377850904 shutdown_connections 2024-03-22T01:33:43.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:43.082+0000 7f51ae772640 1 -- 172.21.15.72:0/377850904 wait complete. 2024-03-22T01:33:43.260 INFO:journalctl@ceph.mds.j.smithi072.stdout:Mar 22 01:33:42 smithi072 podman[70505]: 2024-03-22 01:33:42.899460607 +0000 UTC m=+0.017008634 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T01:33:43.260 INFO:journalctl@ceph.mds.j.smithi072.stdout:Mar 22 01:33:43 smithi072 podman[70505]: 2024-03-22 01:33:43.002017355 +0000 UTC m=+0.119565378 container create 46300af5390d83c9f44f92b394f000ba7a1268cba3c990a9473b5641345c63c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-j, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240319) 2024-03-22T01:33:43.260 INFO:journalctl@ceph.mds.j.smithi072.stdout:Mar 22 01:33:43 smithi072 podman[70505]: 2024-03-22 01:33:43.234065419 +0000 UTC m=+0.351613452 container init 46300af5390d83c9f44f92b394f000ba7a1268cba3c990a9473b5641345c63c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-j, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240321.172515-c3a04c0) 2024-03-22T01:33:43.260 INFO:journalctl@ceph.mds.j.smithi072.stdout:Mar 22 01:33:43 smithi072 podman[70505]: 2024-03-22 01:33:43.236750705 +0000 UTC m=+0.354298733 container start 46300af5390d83c9f44f92b394f000ba7a1268cba3c990a9473b5641345c63c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-j, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-22T01:33:43.554 INFO:journalctl@ceph.mds.j.smithi072.stdout:Mar 22 01:33:43 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-j[70522]: starting mds.j at 2024-03-22T01:33:43.555 INFO:journalctl@ceph.mds.j.smithi072.stdout:Mar 22 01:33:43 smithi072 bash[70505]: 46300af5390d83c9f44f92b394f000ba7a1268cba3c990a9473b5641345c63c3 2024-03-22T01:33:43.555 INFO:journalctl@ceph.mds.j.smithi072.stdout:Mar 22 01:33:43 smithi072 systemd[1]: Started Ceph mds.j for 022a99bc-e7eb-11ee-95cd-87774f69a715. 2024-03-22T01:33:43.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:43 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/377850904' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:33:43.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:43 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:43.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:43 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:43.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:43 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:43.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:43 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T01:33:43.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:43 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T01:33:43.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:43 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:33:43.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:43 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/377850904' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:33:43.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:43 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:43.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:43 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:43.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:43 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:43.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:43 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T01:33:43.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:43 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T01:33:43.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:43 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:33:43.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:43 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/377850904' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:33:43.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:43 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:43.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:43 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:43.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:43 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:43.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:43 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T01:33:43.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:43 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T01:33:43.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:43 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:33:44.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:44 smithi072 ceph-mon[32414]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:33:44.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:44 smithi072 ceph-mon[32414]: Deploying daemon mds.d on smithi072 2024-03-22T01:33:44.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:44 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.72:6834/3077216308,v1:172.21.15.72:6835/3077216308] up:boot 2024-03-22T01:33:44.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:44 smithi072 ceph-mon[32414]: fsmap 3 up:standby 2024-03-22T01:33:44.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:44 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-22T01:33:44.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:44 smithi089 ceph-mon[35770]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:33:44.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:44 smithi089 ceph-mon[35770]: Deploying daemon mds.d on smithi072 2024-03-22T01:33:44.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:44 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.72:6834/3077216308,v1:172.21.15.72:6835/3077216308] up:boot 2024-03-22T01:33:44.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:44 smithi089 ceph-mon[35770]: fsmap 3 up:standby 2024-03-22T01:33:44.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:44 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-22T01:33:44.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:44 smithi098 ceph-mon[36468]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:33:44.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:44 smithi098 ceph-mon[36468]: Deploying daemon mds.d on smithi072 2024-03-22T01:33:44.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:44 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.72:6834/3077216308,v1:172.21.15.72:6835/3077216308] up:boot 2024-03-22T01:33:44.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:44 smithi098 ceph-mon[36468]: fsmap 3 up:standby 2024-03-22T01:33:44.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:44 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-22T01:33:45.093 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T01:33:45.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:45.211+0000 7fe716633640 1 -- 172.21.15.72:0/3257061921 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe710101cb0 msgr2=0x7fe710102130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:45.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:45.211+0000 7fe716633640 1 --2- 172.21.15.72:0/3257061921 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe710101cb0 0x7fe710102130 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe704009900 tx=0x7fe70402bc30 comp rx=0 tx=0).stop 2024-03-22T01:33:45.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:45.212+0000 7fe716633640 1 -- 172.21.15.72:0/3257061921 shutdown_connections 2024-03-22T01:33:45.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:45.212+0000 7fe716633640 1 --2- 172.21.15.72:0/3257061921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe710102670 0x7fe710108f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:45.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:45.212+0000 7fe716633640 1 --2- 172.21.15.72:0/3257061921 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe710101cb0 0x7fe710102130 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:45.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:45.212+0000 7fe716633640 1 --2- 172.21.15.72:0/3257061921 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe710100ac0 0x7fe710100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:45.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:45.212+0000 7fe716633640 1 -- 172.21.15.72:0/3257061921 >> 172.21.15.72:0/3257061921 conn(0x7fe7100fc030 msgr2=0x7fe7100fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:33:45.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:45.212+0000 7fe716633640 1 -- 172.21.15.72:0/3257061921 shutdown_connections 2024-03-22T01:33:45.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:45.212+0000 7fe716633640 1 -- 172.21.15.72:0/3257061921 wait complete. 2024-03-22T01:33:45.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:45.212+0000 7fe716633640 1 Processor -- start 2024-03-22T01:33:45.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:45.212+0000 7fe716633640 1 -- start start 2024-03-22T01:33:45.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:45.213+0000 7fe716633640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe710100ac0 0x7fe710197e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:45.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:45.213+0000 7fe716633640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe710101cb0 0x7fe710198370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:45.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:45.213+0000 7fe716633640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe710102670 0x7fe71019f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:45.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:45.213+0000 7fe716633640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fe71010c720 con 0x7fe710100ac0 2024-03-22T01:33:45.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:45.213+0000 7fe716633640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe71010c5a0 con 0x7fe710101cb0 2024-03-22T01:33:45.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:45.213+0000 7fe716633640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe71010c8a0 con 0x7fe710102670 2024-03-22T01:33:45.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:45.213+0000 7fe70ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe710100ac0 0x7fe710197e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:45.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:45.213+0000 7fe70ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe710100ac0 0x7fe710197e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:39960/0 (socket says 172.21.15.72:39960) 2024-03-22T01:33:45.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:45.213+0000 7fe70ffff640 1 -- 172.21.15.72:0/341696851 learned_addr learned my addr 172.21.15.72:0/341696851 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:33:45.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:45.213+0000 7fe714bd0640 1 --2- 172.21.15.72:0/341696851 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe710102670 0x7fe71019f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:45.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:45.213+0000 7fe70f7fe640 1 --2- 172.21.15.72:0/341696851 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe710101cb0 0x7fe710198370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:45.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:45.213+0000 7fe70ffff640 1 -- 172.21.15.72:0/341696851 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe710102670 msgr2=0x7fe71019f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:45.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:45.213+0000 7fe70ffff640 1 --2- 172.21.15.72:0/341696851 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe710102670 0x7fe71019f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:45.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:45.213+0000 7fe70ffff640 1 -- 172.21.15.72:0/341696851 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe710101cb0 msgr2=0x7fe710198370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:45.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:45.213+0000 7fe70ffff640 1 --2- 172.21.15.72:0/341696851 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe710101cb0 0x7fe710198370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:45.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:45.213+0000 7fe70ffff640 1 -- 172.21.15.72:0/341696851 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe71019fa50 con 0x7fe710100ac0 2024-03-22T01:33:45.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:45.213+0000 7fe70f7fe640 1 --2- 172.21.15.72:0/341696851 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe710101cb0 0x7fe710198370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T01:33:45.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:45.213+0000 7fe70ffff640 1 --2- 172.21.15.72:0/341696851 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe710100ac0 0x7fe710197e30 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe70000c970 tx=0x7fe70000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:33:45.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:45.213+0000 7fe70d7fa640 1 -- 172.21.15.72:0/341696851 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe700007c10 con 0x7fe710100ac0 2024-03-22T01:33:45.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:45.213+0000 7fe716633640 1 -- 172.21.15.72:0/341696851 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe71019fd40 con 0x7fe710100ac0 2024-03-22T01:33:45.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:45.213+0000 7fe70d7fa640 1 -- 172.21.15.72:0/341696851 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe700007db0 con 0x7fe710100ac0 2024-03-22T01:33:45.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:45.213+0000 7fe70d7fa640 1 -- 172.21.15.72:0/341696851 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe70000f8a0 con 0x7fe710100ac0 2024-03-22T01:33:45.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:45.213+0000 7fe716633640 1 -- 172.21.15.72:0/341696851 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7101a0220 con 0x7fe710100ac0 2024-03-22T01:33:45.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:45.215+0000 7fe716633640 1 -- 172.21.15.72:0/341696851 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7101a04b0 con 0x7fe710100ac0 2024-03-22T01:33:45.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:45.216+0000 7fe70d7fa640 1 -- 172.21.15.72:0/341696851 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7fe70000fa80 con 0x7fe710100ac0 2024-03-22T01:33:45.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:45.216+0000 7fe70d7fa640 1 --2- 172.21.15.72:0/341696851 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe6f80a2f20 0x7fe6f80a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:45.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:45.216+0000 7fe70f7fe640 1 --2- 172.21.15.72:0/341696851 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe6f80a2f20 0x7fe6f80a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:45.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:45.216+0000 7fe70d7fa640 1 -- 172.21.15.72:0/341696851 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fe7000cf320 con 0x7fe710100ac0 2024-03-22T01:33:45.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:45.217+0000 7fe70f7fe640 1 --2- 172.21.15.72:0/341696851 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe6f80a2f20 0x7fe6f80a53e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe704036040 tx=0x7fe704000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:33:45.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:45.221+0000 7fe70d7fa640 1 -- 172.21.15.72:0/341696851 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe70009c680 con 0x7fe710100ac0 2024-03-22T01:33:45.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:45.455+0000 7fe716633640 1 -- 172.21.15.72:0/341696851 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe7101a0680 con 0x7fe710100ac0 2024-03-22T01:33:45.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:45.456+0000 7fe70d7fa640 1 -- 172.21.15.72:0/341696851 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3043 (secure 0 0 0) 0x7fe70009bf80 con 0x7fe710100ac0 2024-03-22T01:33:45.455 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:33:45.455 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6835/3077216308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3077216308},{"type":"v1","addr":"172.21.15.72:6835","nonce":3077216308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6833/1578701171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":1578701171},{"type":"v1","addr":"172.21.15.89:6833","nonce":1578701171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/2703047213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2703047213},{"type":"v1","addr":"172.21.15.98:6833","nonce":2703047213}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-22T01:33:45.456 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 4 2024-03-22T01:33:45.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:45.458+0000 7fe716633640 1 -- 172.21.15.72:0/341696851 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe6f80a2f20 msgr2=0x7fe6f80a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:45.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:45.458+0000 7fe716633640 1 --2- 172.21.15.72:0/341696851 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe6f80a2f20 0x7fe6f80a53e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe704036040 tx=0x7fe704000bc0 comp rx=0 tx=0).stop 2024-03-22T01:33:45.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:45.458+0000 7fe716633640 1 -- 172.21.15.72:0/341696851 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe710100ac0 msgr2=0x7fe710197e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:45.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:45.458+0000 7fe716633640 1 --2- 172.21.15.72:0/341696851 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe710100ac0 0x7fe710197e30 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe70000c970 tx=0x7fe70000ce30 comp rx=0 tx=0).stop 2024-03-22T01:33:45.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:45.458+0000 7fe716633640 1 -- 172.21.15.72:0/341696851 shutdown_connections 2024-03-22T01:33:45.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:45.458+0000 7fe716633640 1 --2- 172.21.15.72:0/341696851 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe6f80a2f20 0x7fe6f80a53e0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:45.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:45.458+0000 7fe716633640 1 --2- 172.21.15.72:0/341696851 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe710101cb0 0x7fe710198370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:45.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:45.458+0000 7fe716633640 1 --2- 172.21.15.72:0/341696851 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe710102670 0x7fe71019f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:45.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:45.458+0000 7fe716633640 1 --2- 172.21.15.72:0/341696851 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe710100ac0 0x7fe710197e30 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:45.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:45.458+0000 7fe716633640 1 -- 172.21.15.72:0/341696851 >> 172.21.15.72:0/341696851 conn(0x7fe7100fc030 msgr2=0x7fe710068e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:33:45.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:45.458+0000 7fe716633640 1 -- 172.21.15.72:0/341696851 shutdown_connections 2024-03-22T01:33:45.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:45.458+0000 7fe716633640 1 -- 172.21.15.72:0/341696851 wait complete. 2024-03-22T01:33:45.786 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:45 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/341696851' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:33:45.786 INFO:journalctl@ceph.mds.d.smithi072.stdout:Mar 22 01:33:45 smithi072 systemd[1]: Starting Ceph mds.d for 022a99bc-e7eb-11ee-95cd-87774f69a715... 2024-03-22T01:33:45.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:45 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/341696851' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:33:45.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:45 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/341696851' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:33:46.049 INFO:journalctl@ceph.mds.d.smithi072.stdout:Mar 22 01:33:45 smithi072 podman[70856]: 2024-03-22 01:33:45.68999868 +0000 UTC m=+0.017594726 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T01:33:46.049 INFO:journalctl@ceph.mds.d.smithi072.stdout:Mar 22 01:33:45 smithi072 podman[70856]: 2024-03-22 01:33:45.795048498 +0000 UTC m=+0.122644536 container create bf8e80142e1c689e5edf05bba9698b82c27c6c662c25af2faef2c910b2979380 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-d, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-22T01:33:46.049 INFO:journalctl@ceph.mds.d.smithi072.stdout:Mar 22 01:33:46 smithi072 podman[70856]: 2024-03-22 01:33:46.024819817 +0000 UTC m=+0.352415859 container init bf8e80142e1c689e5edf05bba9698b82c27c6c662c25af2faef2c910b2979380 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-22T01:33:46.049 INFO:journalctl@ceph.mds.d.smithi072.stdout:Mar 22 01:33:46 smithi072 podman[70856]: 2024-03-22 01:33:46.02754004 +0000 UTC m=+0.355136082 container start bf8e80142e1c689e5edf05bba9698b82c27c6c662c25af2faef2c910b2979380 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-d, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-22T01:33:46.049 INFO:journalctl@ceph.mds.d.smithi072.stdout:Mar 22 01:33:46 smithi072 ceph-mds[70875]: -- 172.21.15.72:0/1825938737 <== mon.0 v2:172.21.15.72:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55bcc678ba40 con 0x55bcc675e800 2024-03-22T01:33:46.372 INFO:journalctl@ceph.mds.d.smithi072.stdout:Mar 22 01:33:46 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-d[70871]: starting mds.d at 2024-03-22T01:33:46.372 INFO:journalctl@ceph.mds.d.smithi072.stdout:Mar 22 01:33:46 smithi072 bash[70856]: bf8e80142e1c689e5edf05bba9698b82c27c6c662c25af2faef2c910b2979380 2024-03-22T01:33:46.372 INFO:journalctl@ceph.mds.d.smithi072.stdout:Mar 22 01:33:46 smithi072 systemd[1]: Started Ceph mds.d for 022a99bc-e7eb-11ee-95cd-87774f69a715. 2024-03-22T01:33:46.826 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:46 smithi089 ceph-mon[35770]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:33:46.826 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:46 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:46.826 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:46 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:46.826 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:46 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:46.826 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:46 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T01:33:46.826 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:46 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T01:33:46.826 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:46 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:33:46.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:46 smithi072 ceph-mon[32414]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:33:46.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:46 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:46.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:46 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:46.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:46 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:46.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:46 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T01:33:46.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:46 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T01:33:46.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:46 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:33:46.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:46 smithi098 ceph-mon[36468]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:33:46.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:46 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:46.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:46 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:46.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:46 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:46.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:46 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T01:33:46.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:46 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T01:33:46.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:46 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:33:47.469 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T01:33:47.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:47.577+0000 7f5338a4e640 1 -- 172.21.15.72:0/1155028521 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53341078a0 msgr2=0x7f5334109cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:47.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:47.577+0000 7f5338a4e640 1 --2- 172.21.15.72:0/1155028521 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53341078a0 0x7f5334109cf0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f532c009ca0 tx=0x7f532c02c0f0 comp rx=0 tx=0).stop 2024-03-22T01:33:47.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:47.577+0000 7f5338a4e640 1 -- 172.21.15.72:0/1155028521 shutdown_connections 2024-03-22T01:33:47.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:47.577+0000 7f5338a4e640 1 --2- 172.21.15.72:0/1155028521 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5334104f70 0x7f5334107360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:47.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:47.577+0000 7f5338a4e640 1 --2- 172.21.15.72:0/1155028521 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53341078a0 0x7f5334109cf0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:47.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:47.577+0000 7f5338a4e640 1 --2- 172.21.15.72:0/1155028521 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5334102640 0x7f5334104a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:47.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:47.577+0000 7f5338a4e640 1 -- 172.21.15.72:0/1155028521 >> 172.21.15.72:0/1155028521 conn(0x7f53340fc030 msgr2=0x7f53340fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:33:47.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:47.577+0000 7f5338a4e640 1 -- 172.21.15.72:0/1155028521 shutdown_connections 2024-03-22T01:33:47.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:47.577+0000 7f5338a4e640 1 -- 172.21.15.72:0/1155028521 wait complete. 2024-03-22T01:33:47.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:47.578+0000 7f5338a4e640 1 Processor -- start 2024-03-22T01:33:47.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:47.578+0000 7f5338a4e640 1 -- start start 2024-03-22T01:33:47.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:47.578+0000 7f5338a4e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5334102640 0x7f5334197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:47.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:47.578+0000 7f5338a4e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5334104f70 0x7f5334198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:47.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:47.578+0000 7f5338a4e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53341078a0 0x7f533419f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:47.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:47.578+0000 7f5338a4e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f533410d860 con 0x7f5334102640 2024-03-22T01:33:47.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:47.578+0000 7f5338a4e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f533410d6e0 con 0x7f5334104f70 2024-03-22T01:33:47.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:47.578+0000 7f5338a4e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f533410d9e0 con 0x7f53341078a0 2024-03-22T01:33:47.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:47.578+0000 7f533259c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5334102640 0x7f5334197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:47.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:47.578+0000 7f533259c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5334102640 0x7f5334197e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:39992/0 (socket says 172.21.15.72:39992) 2024-03-22T01:33:47.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:47.578+0000 7f533259c640 1 -- 172.21.15.72:0/2013827088 learned_addr learned my addr 172.21.15.72:0/2013827088 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:33:47.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:47.578+0000 7f5332d9d640 1 --2- 172.21.15.72:0/2013827088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53341078a0 0x7f533419f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:47.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:47.578+0000 7f5331d9b640 1 --2- 172.21.15.72:0/2013827088 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5334104f70 0x7f5334198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:47.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:47.579+0000 7f533259c640 1 -- 172.21.15.72:0/2013827088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53341078a0 msgr2=0x7f533419f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:47.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:47.579+0000 7f533259c640 1 --2- 172.21.15.72:0/2013827088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53341078a0 0x7f533419f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:47.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:47.579+0000 7f533259c640 1 -- 172.21.15.72:0/2013827088 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5334104f70 msgr2=0x7f5334198390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:47.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:47.579+0000 7f533259c640 1 --2- 172.21.15.72:0/2013827088 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5334104f70 0x7f5334198390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:47.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:47.579+0000 7f533259c640 1 -- 172.21.15.72:0/2013827088 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f533419fa70 con 0x7f5334102640 2024-03-22T01:33:47.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:47.579+0000 7f5332d9d640 1 --2- 172.21.15.72:0/2013827088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53341078a0 0x7f533419f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:33:47.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:47.579+0000 7f5331d9b640 1 --2- 172.21.15.72:0/2013827088 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5334104f70 0x7f5334198390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T01:33:47.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:47.579+0000 7f533259c640 1 --2- 172.21.15.72:0/2013827088 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5334102640 0x7f5334197e50 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f532400e9f0 tx=0x7f532400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:33:47.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:47.579+0000 7f53237fe640 1 -- 172.21.15.72:0/2013827088 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f532400cd80 con 0x7f5334102640 2024-03-22T01:33:47.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:47.579+0000 7f53237fe640 1 -- 172.21.15.72:0/2013827088 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5324005500 con 0x7f5334102640 2024-03-22T01:33:47.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:47.579+0000 7f5338a4e640 1 -- 172.21.15.72:0/2013827088 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f533419fd60 con 0x7f5334102640 2024-03-22T01:33:47.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:47.579+0000 7f53237fe640 1 -- 172.21.15.72:0/2013827088 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5324010690 con 0x7f5334102640 2024-03-22T01:33:47.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:47.579+0000 7f5338a4e640 1 -- 172.21.15.72:0/2013827088 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53341a01f0 con 0x7f5334102640 2024-03-22T01:33:47.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:47.580+0000 7f5338a4e640 1 -- 172.21.15.72:0/2013827088 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53340fdb20 con 0x7f5334102640 2024-03-22T01:33:47.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:47.581+0000 7f53237fe640 1 -- 172.21.15.72:0/2013827088 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f53240046c0 con 0x7f5334102640 2024-03-22T01:33:47.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:47.582+0000 7f53237fe640 1 --2- 172.21.15.72:0/2013827088 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f531c0a2fa0 0x7f531c0a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:47.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:47.582+0000 7f5331d9b640 1 --2- 172.21.15.72:0/2013827088 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f531c0a2fa0 0x7f531c0a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:47.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:47.582+0000 7f53237fe640 1 -- 172.21.15.72:0/2013827088 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f53240cf5a0 con 0x7f5334102640 2024-03-22T01:33:47.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:47.582+0000 7f5331d9b640 1 --2- 172.21.15.72:0/2013827088 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f531c0a2fa0 0x7f531c0a5460 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f5334199360 tx=0x7f5328005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:33:47.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:47.585+0000 7f53237fe640 1 -- 172.21.15.72:0/2013827088 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5324053740 con 0x7f5334102640 2024-03-22T01:33:47.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:47.796+0000 7f5338a4e640 1 -- 172.21.15.72:0/2013827088 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f533402d3b0 con 0x7f5334102640 2024-03-22T01:33:47.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:47.797+0000 7f53237fe640 1 -- 172.21.15.72:0/2013827088 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3868 (secure 0 0 0) 0x7f532409c870 con 0x7f5334102640 2024-03-22T01:33:47.796 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:33:47.796 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6835/3077216308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3077216308},{"type":"v1","addr":"172.21.15.72:6835","nonce":3077216308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2460254143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":2460254143},{"type":"v1","addr":"172.21.15.72:6837","nonce":2460254143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6833/1578701171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":1578701171},{"type":"v1","addr":"172.21.15.89:6833","nonce":1578701171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/2703047213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2703047213},{"type":"v1","addr":"172.21.15.98:6833","nonce":2703047213}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-22T01:33:47.796 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 5 2024-03-22T01:33:47.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:47.799+0000 7f5338a4e640 1 -- 172.21.15.72:0/2013827088 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f531c0a2fa0 msgr2=0x7f531c0a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:47.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:47.799+0000 7f5338a4e640 1 --2- 172.21.15.72:0/2013827088 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f531c0a2fa0 0x7f531c0a5460 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f5334199360 tx=0x7f5328005f90 comp rx=0 tx=0).stop 2024-03-22T01:33:47.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:47.799+0000 7f5338a4e640 1 -- 172.21.15.72:0/2013827088 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5334102640 msgr2=0x7f5334197e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:47.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:47.799+0000 7f5338a4e640 1 --2- 172.21.15.72:0/2013827088 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5334102640 0x7f5334197e50 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f532400e9f0 tx=0x7f532400edf0 comp rx=0 tx=0).stop 2024-03-22T01:33:47.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:47.799+0000 7f5338a4e640 1 -- 172.21.15.72:0/2013827088 shutdown_connections 2024-03-22T01:33:47.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:47.799+0000 7f5338a4e640 1 --2- 172.21.15.72:0/2013827088 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f531c0a2fa0 0x7f531c0a5460 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:47.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:47.799+0000 7f5338a4e640 1 --2- 172.21.15.72:0/2013827088 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5334104f70 0x7f5334198390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:47.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:47.799+0000 7f5338a4e640 1 --2- 172.21.15.72:0/2013827088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53341078a0 0x7f533419f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:47.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:47.799+0000 7f5338a4e640 1 --2- 172.21.15.72:0/2013827088 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5334102640 0x7f5334197e50 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:47.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:47.799+0000 7f5338a4e640 1 -- 172.21.15.72:0/2013827088 >> 172.21.15.72:0/2013827088 conn(0x7f53340fc030 msgr2=0x7f5334102f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:33:47.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:47.799+0000 7f5338a4e640 1 -- 172.21.15.72:0/2013827088 shutdown_connections 2024-03-22T01:33:47.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:47.799+0000 7f5338a4e640 1 -- 172.21.15.72:0/2013827088 wait complete. 2024-03-22T01:33:47.829 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:47 smithi089 ceph-mon[35770]: Deploying daemon mds.e on smithi089 2024-03-22T01:33:47.829 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:47 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.72:6836/2460254143,v1:172.21.15.72:6837/2460254143] up:boot 2024-03-22T01:33:47.829 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:47 smithi089 ceph-mon[35770]: fsmap 4 up:standby 2024-03-22T01:33:47.829 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:47 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-22T01:33:47.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:47 smithi072 ceph-mon[32414]: Deploying daemon mds.e on smithi089 2024-03-22T01:33:47.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:47 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.72:6836/2460254143,v1:172.21.15.72:6837/2460254143] up:boot 2024-03-22T01:33:47.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:47 smithi072 ceph-mon[32414]: fsmap 4 up:standby 2024-03-22T01:33:47.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:47 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-22T01:33:47.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:47 smithi098 ceph-mon[36468]: Deploying daemon mds.e on smithi089 2024-03-22T01:33:47.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:47 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.72:6836/2460254143,v1:172.21.15.72:6837/2460254143] up:boot 2024-03-22T01:33:47.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:47 smithi098 ceph-mon[36468]: fsmap 4 up:standby 2024-03-22T01:33:47.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:47 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-22T01:33:48.838 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:48 smithi089 ceph-mon[35770]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:33:48.838 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:48 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/2013827088' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:33:48.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:48 smithi072 ceph-mon[32414]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:33:48.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:48 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/2013827088' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:33:48.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:48 smithi098 ceph-mon[36468]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:33:48.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:48 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/2013827088' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:33:49.108 INFO:journalctl@ceph.mds.e.smithi089.stdout:Mar 22 01:33:49 smithi089 systemd[1]: Started Ceph mds.e for 022a99bc-e7eb-11ee-95cd-87774f69a715. 2024-03-22T01:33:49.811 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T01:33:49.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:49.926+0000 7ff2e097a640 1 -- 172.21.15.72:0/309303018 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2dc104f70 msgr2=0x7ff2dc107360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:49.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:49.926+0000 7ff2e097a640 1 --2- 172.21.15.72:0/309303018 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2dc104f70 0x7ff2dc107360 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff2d0009900 tx=0x7ff2d002bc30 comp rx=0 tx=0).stop 2024-03-22T01:33:49.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:49.926+0000 7ff2e097a640 1 -- 172.21.15.72:0/309303018 shutdown_connections 2024-03-22T01:33:49.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:49.926+0000 7ff2e097a640 1 --2- 172.21.15.72:0/309303018 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2dc1078a0 0x7ff2dc109cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:49.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:49.926+0000 7ff2e097a640 1 --2- 172.21.15.72:0/309303018 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2dc104f70 0x7ff2dc107360 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:49.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:49.926+0000 7ff2e097a640 1 --2- 172.21.15.72:0/309303018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2dc102640 0x7ff2dc104a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:49.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:49.926+0000 7ff2e097a640 1 -- 172.21.15.72:0/309303018 >> 172.21.15.72:0/309303018 conn(0x7ff2dc0fc030 msgr2=0x7ff2dc0fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:33:49.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:49.926+0000 7ff2e097a640 1 -- 172.21.15.72:0/309303018 shutdown_connections 2024-03-22T01:33:49.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:49.926+0000 7ff2e097a640 1 -- 172.21.15.72:0/309303018 wait complete. 2024-03-22T01:33:49.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:49.927+0000 7ff2e097a640 1 Processor -- start 2024-03-22T01:33:49.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:49.927+0000 7ff2e097a640 1 -- start start 2024-03-22T01:33:49.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:49.927+0000 7ff2e097a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2dc102640 0x7ff2dc197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:49.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:49.927+0000 7ff2e097a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2dc104f70 0x7ff2dc198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:49.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:49.927+0000 7ff2e097a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2dc1078a0 0x7ff2dc19f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:49.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:49.927+0000 7ff2e097a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7ff2dc10d850 con 0x7ff2dc1078a0 2024-03-22T01:33:49.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:49.927+0000 7ff2e097a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff2dc10d6d0 con 0x7ff2dc102640 2024-03-22T01:33:49.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:49.927+0000 7ff2e097a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff2dc10d9d0 con 0x7ff2dc104f70 2024-03-22T01:33:49.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:49.927+0000 7ff2dad9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2dc1078a0 0x7ff2dc19f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:49.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:49.927+0000 7ff2dad9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2dc1078a0 0x7ff2dc19f470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:40000/0 (socket says 172.21.15.72:40000) 2024-03-22T01:33:49.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:49.927+0000 7ff2da59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2dc102640 0x7ff2dc197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:49.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:49.927+0000 7ff2dad9d640 1 -- 172.21.15.72:0/2633895200 learned_addr learned my addr 172.21.15.72:0/2633895200 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:33:49.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:49.927+0000 7ff2d9d9b640 1 --2- 172.21.15.72:0/2633895200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2dc104f70 0x7ff2dc198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:49.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:49.927+0000 7ff2dad9d640 1 -- 172.21.15.72:0/2633895200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2dc104f70 msgr2=0x7ff2dc198400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:49.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:49.927+0000 7ff2dad9d640 1 --2- 172.21.15.72:0/2633895200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2dc104f70 0x7ff2dc198400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:49.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:49.927+0000 7ff2dad9d640 1 -- 172.21.15.72:0/2633895200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2dc102640 msgr2=0x7ff2dc197ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:49.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:49.927+0000 7ff2dad9d640 1 --2- 172.21.15.72:0/2633895200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2dc102640 0x7ff2dc197ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:49.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:49.927+0000 7ff2dad9d640 1 -- 172.21.15.72:0/2633895200 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2dc19fae0 con 0x7ff2dc1078a0 2024-03-22T01:33:49.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:49.927+0000 7ff2dad9d640 1 --2- 172.21.15.72:0/2633895200 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2dc1078a0 0x7ff2dc19f470 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff2d400eaa0 tx=0x7ff2d400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:33:49.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:49.927+0000 7ff2cb7fe640 1 -- 172.21.15.72:0/2633895200 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2d400cdb0 con 0x7ff2dc1078a0 2024-03-22T01:33:49.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:49.928+0000 7ff2cb7fe640 1 -- 172.21.15.72:0/2633895200 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff2d4005020 con 0x7ff2dc1078a0 2024-03-22T01:33:49.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:49.928+0000 7ff2e097a640 1 -- 172.21.15.72:0/2633895200 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2dc19fdd0 con 0x7ff2dc1078a0 2024-03-22T01:33:49.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:49.928+0000 7ff2cb7fe640 1 -- 172.21.15.72:0/2633895200 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2d40116b0 con 0x7ff2dc1078a0 2024-03-22T01:33:49.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:49.928+0000 7ff2e097a640 1 -- 172.21.15.72:0/2633895200 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2dc1a0240 con 0x7ff2dc1078a0 2024-03-22T01:33:49.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:49.929+0000 7ff2e097a640 1 -- 172.21.15.72:0/2633895200 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2dc0fdb20 con 0x7ff2dc1078a0 2024-03-22T01:33:49.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:49.930+0000 7ff2cb7fe640 1 -- 172.21.15.72:0/2633895200 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7ff2d4005560 con 0x7ff2dc1078a0 2024-03-22T01:33:49.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:49.930+0000 7ff2cb7fe640 1 --2- 172.21.15.72:0/2633895200 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7ff2c40a2f20 0x7ff2c40a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:49.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:49.930+0000 7ff2da59c640 1 --2- 172.21.15.72:0/2633895200 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7ff2c40a2f20 0x7ff2c40a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:49.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:49.930+0000 7ff2cb7fe640 1 -- 172.21.15.72:0/2633895200 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7ff2d40cf0f0 con 0x7ff2dc1078a0 2024-03-22T01:33:49.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:49.931+0000 7ff2da59c640 1 --2- 172.21.15.72:0/2633895200 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7ff2c40a2f20 0x7ff2c40a53e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc009bd0 tx=0x7ff2cc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:33:49.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:49.935+0000 7ff2cb7fe640 1 -- 172.21.15.72:0/2633895200 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff2d4016030 con 0x7ff2dc1078a0 2024-03-22T01:33:50.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:50.148+0000 7ff2e097a640 1 -- 172.21.15.72:0/2633895200 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff2dc02d560 con 0x7ff2dc1078a0 2024-03-22T01:33:50.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:50.149+0000 7ff2cb7fe640 1 -- 172.21.15.72:0/2633895200 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4693 (secure 0 0 0) 0x7ff2d409bd70 con 0x7ff2dc1078a0 2024-03-22T01:33:50.148 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:33:50.148 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6835/3077216308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3077216308},{"type":"v1","addr":"172.21.15.72:6835","nonce":3077216308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2460254143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":2460254143},{"type":"v1","addr":"172.21.15.72:6837","nonce":2460254143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6833/1578701171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":1578701171},{"type":"v1","addr":"172.21.15.89:6833","nonce":1578701171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/2703047213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2703047213},{"type":"v1","addr":"172.21.15.98:6833","nonce":2703047213}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2589468787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2589468787},{"type":"v1","addr":"172.21.15.89:6835","nonce":2589468787}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-03-22T01:33:50.148 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 6 2024-03-22T01:33:50.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:50.151+0000 7ff2e097a640 1 -- 172.21.15.72:0/2633895200 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7ff2c40a2f20 msgr2=0x7ff2c40a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:50.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:50.151+0000 7ff2e097a640 1 --2- 172.21.15.72:0/2633895200 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7ff2c40a2f20 0x7ff2c40a53e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc009bd0 tx=0x7ff2cc005f90 comp rx=0 tx=0).stop 2024-03-22T01:33:50.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:50.151+0000 7ff2e097a640 1 -- 172.21.15.72:0/2633895200 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2dc1078a0 msgr2=0x7ff2dc19f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:50.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:50.151+0000 7ff2e097a640 1 --2- 172.21.15.72:0/2633895200 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2dc1078a0 0x7ff2dc19f470 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff2d400eaa0 tx=0x7ff2d400ef60 comp rx=0 tx=0).stop 2024-03-22T01:33:50.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:50.151+0000 7ff2e097a640 1 -- 172.21.15.72:0/2633895200 shutdown_connections 2024-03-22T01:33:50.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:50.151+0000 7ff2e097a640 1 --2- 172.21.15.72:0/2633895200 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7ff2c40a2f20 0x7ff2c40a53e0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:50.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:50.151+0000 7ff2e097a640 1 --2- 172.21.15.72:0/2633895200 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2dc1078a0 0x7ff2dc19f470 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:50.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:50.151+0000 7ff2e097a640 1 --2- 172.21.15.72:0/2633895200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2dc104f70 0x7ff2dc198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:50.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:50.151+0000 7ff2e097a640 1 --2- 172.21.15.72:0/2633895200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2dc102640 0x7ff2dc197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:50.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:50.151+0000 7ff2e097a640 1 -- 172.21.15.72:0/2633895200 >> 172.21.15.72:0/2633895200 conn(0x7ff2dc0fc030 msgr2=0x7ff2dc102f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:33:50.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:50.151+0000 7ff2e097a640 1 -- 172.21.15.72:0/2633895200 shutdown_connections 2024-03-22T01:33:50.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:50.151+0000 7ff2e097a640 1 -- 172.21.15.72:0/2633895200 wait complete. 2024-03-22T01:33:50.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:50 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:50.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:50 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:50.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:50 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:50.373 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:50 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T01:33:50.373 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:50 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T01:33:50.373 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:50 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:33:50.373 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:50 smithi072 ceph-mon[32414]: Deploying daemon mds.f on smithi098 2024-03-22T01:33:50.373 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:50 smithi072 ceph-mon[32414]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:33:50.373 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:50 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.89:6834/2589468787,v1:172.21.15.89:6835/2589468787] up:boot 2024-03-22T01:33:50.373 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:50 smithi072 ceph-mon[32414]: fsmap 5 up:standby 2024-03-22T01:33:50.373 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:50 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-22T01:33:50.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:50 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:50.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:50 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:50.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:50 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:50.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:50 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T01:33:50.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:50 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T01:33:50.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:50 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:33:50.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:50 smithi098 ceph-mon[36468]: Deploying daemon mds.f on smithi098 2024-03-22T01:33:50.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:50 smithi098 ceph-mon[36468]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:33:50.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:50 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.89:6834/2589468787,v1:172.21.15.89:6835/2589468787] up:boot 2024-03-22T01:33:50.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:50 smithi098 ceph-mon[36468]: fsmap 5 up:standby 2024-03-22T01:33:50.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:50 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-22T01:33:50.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:50 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:50.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:50 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:50.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:50 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:50.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:50 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T01:33:50.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:50 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T01:33:50.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:50 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:33:50.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:50 smithi089 ceph-mon[35770]: Deploying daemon mds.f on smithi098 2024-03-22T01:33:50.609 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:50 smithi089 ceph-mon[35770]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:33:50.609 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:50 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.89:6834/2589468787,v1:172.21.15.89:6835/2589468787] up:boot 2024-03-22T01:33:50.609 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:50 smithi089 ceph-mon[35770]: fsmap 5 up:standby 2024-03-22T01:33:50.609 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:50 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-22T01:33:51.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:51 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/2633895200' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:33:51.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:51 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/2633895200' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:33:51.422 INFO:journalctl@ceph.mds.f.smithi098.stdout:Mar 22 01:33:51 smithi098 systemd[1]: Starting Ceph mds.f for 022a99bc-e7eb-11ee-95cd-87774f69a715... 2024-03-22T01:33:51.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:51 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/2633895200' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:33:51.825 INFO:journalctl@ceph.mds.f.smithi098.stdout:Mar 22 01:33:51 smithi098 podman[67684]: 2024-03-22 01:33:51.439656181 +0000 UTC m=+0.018684985 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T01:33:51.826 INFO:journalctl@ceph.mds.f.smithi098.stdout:Mar 22 01:33:51 smithi098 podman[67684]: 2024-03-22 01:33:51.549387914 +0000 UTC m=+0.128416712 container create 882fad7c69f457aa75d4d208150d69b9e03d721d8948e63079f7ab89f860027b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-f, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-22T01:33:51.826 INFO:journalctl@ceph.mds.f.smithi098.stdout:Mar 22 01:33:51 smithi098 podman[67684]: 2024-03-22 01:33:51.827291838 +0000 UTC m=+0.406320643 container init 882fad7c69f457aa75d4d208150d69b9e03d721d8948e63079f7ab89f860027b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-f, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-22T01:33:52.126 INFO:journalctl@ceph.mds.f.smithi098.stdout:Mar 22 01:33:51 smithi098 podman[67684]: 2024-03-22 01:33:51.830122582 +0000 UTC m=+0.409151378 container start 882fad7c69f457aa75d4d208150d69b9e03d721d8948e63079f7ab89f860027b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-f, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240319, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-22T01:33:52.126 INFO:journalctl@ceph.mds.f.smithi098.stdout:Mar 22 01:33:51 smithi098 ceph-mds[67703]: -- 172.21.15.98:0/1500928718 <== mon.1 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564c0baa7a40 con 0x564c0ba7a800 2024-03-22T01:33:52.126 INFO:journalctl@ceph.mds.f.smithi098.stdout:Mar 22 01:33:51 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-f[67699]: starting mds.f at 2024-03-22T01:33:52.126 INFO:journalctl@ceph.mds.f.smithi098.stdout:Mar 22 01:33:51 smithi098 bash[67684]: 882fad7c69f457aa75d4d208150d69b9e03d721d8948e63079f7ab89f860027b 2024-03-22T01:33:52.126 INFO:journalctl@ceph.mds.f.smithi098.stdout:Mar 22 01:33:51 smithi098 systemd[1]: Started Ceph mds.f for 022a99bc-e7eb-11ee-95cd-87774f69a715. 2024-03-22T01:33:52.161 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T01:33:52.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:52.274+0000 7f2eceeeb640 1 -- 172.21.15.72:0/3079699449 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2ec8104fe0 msgr2=0x7f2ec81073d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:52.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:52.274+0000 7f2eceeeb640 1 --2- 172.21.15.72:0/3079699449 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2ec8104fe0 0x7f2ec81073d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f2eb4009900 tx=0x7f2eb402bc30 comp rx=0 tx=0).stop 2024-03-22T01:33:52.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:52.274+0000 7f2eceeeb640 1 -- 172.21.15.72:0/3079699449 shutdown_connections 2024-03-22T01:33:52.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:52.274+0000 7f2eceeeb640 1 --2- 172.21.15.72:0/3079699449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ec8107910 0x7f2ec8109d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:52.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:52.274+0000 7f2eceeeb640 1 --2- 172.21.15.72:0/3079699449 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2ec8104fe0 0x7f2ec81073d0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:52.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:52.274+0000 7f2eceeeb640 1 --2- 172.21.15.72:0/3079699449 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ec81026b0 0x7f2ec8104aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:52.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:52.274+0000 7f2eceeeb640 1 -- 172.21.15.72:0/3079699449 >> 172.21.15.72:0/3079699449 conn(0x7f2ec80fc0a0 msgr2=0x7f2ec80fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:33:52.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:52.274+0000 7f2eceeeb640 1 -- 172.21.15.72:0/3079699449 shutdown_connections 2024-03-22T01:33:52.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:52.274+0000 7f2eceeeb640 1 -- 172.21.15.72:0/3079699449 wait complete. 2024-03-22T01:33:52.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:52.275+0000 7f2eceeeb640 1 Processor -- start 2024-03-22T01:33:52.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:52.275+0000 7f2eceeeb640 1 -- start start 2024-03-22T01:33:52.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:52.275+0000 7f2eceeeb640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ec81026b0 0x7f2ec8197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:52.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:52.275+0000 7f2eceeeb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ec8104fe0 0x7f2ec8198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:52.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:52.275+0000 7f2eceeeb640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2ec8107910 0x7f2ec819f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:52.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:52.275+0000 7f2eceeeb640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f2ec810d840 con 0x7f2ec8107910 2024-03-22T01:33:52.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:52.275+0000 7f2eceeeb640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2ec810d6c0 con 0x7f2ec81026b0 2024-03-22T01:33:52.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:52.275+0000 7f2eceeeb640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2ec810d9c0 con 0x7f2ec8104fe0 2024-03-22T01:33:52.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:52.275+0000 7f2ecd488640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2ec8107910 0x7f2ec819f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:52.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:52.275+0000 7f2ecd488640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2ec8107910 0x7f2ec819f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48582/0 (socket says 172.21.15.72:48582) 2024-03-22T01:33:52.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:52.275+0000 7f2ecd488640 1 -- 172.21.15.72:0/1818997665 learned_addr learned my addr 172.21.15.72:0/1818997665 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:33:52.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:52.275+0000 7f2ec7fff640 1 --2- 172.21.15.72:0/1818997665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ec8104fe0 0x7f2ec8198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:52.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:52.275+0000 7f2eccc87640 1 --2- 172.21.15.72:0/1818997665 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ec81026b0 0x7f2ec8197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:52.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:52.276+0000 7f2ecd488640 1 -- 172.21.15.72:0/1818997665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ec8104fe0 msgr2=0x7f2ec8198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:52.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:52.276+0000 7f2ecd488640 1 --2- 172.21.15.72:0/1818997665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ec8104fe0 0x7f2ec8198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:52.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:52.276+0000 7f2ecd488640 1 -- 172.21.15.72:0/1818997665 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ec81026b0 msgr2=0x7f2ec8197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:52.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:52.276+0000 7f2ecd488640 1 --2- 172.21.15.72:0/1818997665 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ec81026b0 0x7f2ec8197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:52.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:52.276+0000 7f2ecd488640 1 -- 172.21.15.72:0/1818997665 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ec819fb00 con 0x7f2ec8107910 2024-03-22T01:33:52.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:52.276+0000 7f2ec7fff640 1 --2- 172.21.15.72:0/1818997665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ec8104fe0 0x7f2ec8198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:33:52.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:52.276+0000 7f2ecd488640 1 --2- 172.21.15.72:0/1818997665 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2ec8107910 0x7f2ec819f490 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f2ec000eb10 tx=0x7f2ec000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:33:52.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:52.276+0000 7f2ec5ffb640 1 -- 172.21.15.72:0/1818997665 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ec0004b20 con 0x7f2ec8107910 2024-03-22T01:33:52.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:52.276+0000 7f2ec5ffb640 1 -- 172.21.15.72:0/1818997665 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ec0004cc0 con 0x7f2ec8107910 2024-03-22T01:33:52.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:52.276+0000 7f2eceeeb640 1 -- 172.21.15.72:0/1818997665 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ec819fdf0 con 0x7f2ec8107910 2024-03-22T01:33:52.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:52.276+0000 7f2ec5ffb640 1 -- 172.21.15.72:0/1818997665 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ec0010760 con 0x7f2ec8107910 2024-03-22T01:33:52.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:52.276+0000 7f2eceeeb640 1 -- 172.21.15.72:0/1818997665 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ec81a0300 con 0x7f2ec8107910 2024-03-22T01:33:52.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:52.277+0000 7f2eceeeb640 1 -- 172.21.15.72:0/1818997665 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ec80fdb90 con 0x7f2ec8107910 2024-03-22T01:33:52.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:52.278+0000 7f2ec5ffb640 1 -- 172.21.15.72:0/1818997665 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f2ec0005020 con 0x7f2ec8107910 2024-03-22T01:33:52.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:52.279+0000 7f2ec5ffb640 1 --2- 172.21.15.72:0/1818997665 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f2eb80a2fa0 0x7f2eb80a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:52.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:52.279+0000 7f2eccc87640 1 --2- 172.21.15.72:0/1818997665 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f2eb80a2fa0 0x7f2eb80a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:52.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:52.279+0000 7f2ec5ffb640 1 -- 172.21.15.72:0/1818997665 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f2ec00cf570 con 0x7f2ec8107910 2024-03-22T01:33:52.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:52.279+0000 7f2eccc87640 1 --2- 172.21.15.72:0/1818997665 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f2eb80a2fa0 0x7f2eb80a5460 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc00bbe0 tx=0x7f2ebc00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:33:52.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:52.283+0000 7f2ec5ffb640 1 -- 172.21.15.72:0/1818997665 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2ec009c840 con 0x7f2ec8107910 2024-03-22T01:33:52.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:52 smithi098 ceph-mon[36468]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:33:52.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:52 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:52.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:52 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:52.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:52 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:52.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:52 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T01:33:52.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:52 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T01:33:52.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:52 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:33:52.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:52.497+0000 7f2eceeeb640 1 -- 172.21.15.72:0/1818997665 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2ec802d580 con 0x7f2ec8107910 2024-03-22T01:33:52.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:52.498+0000 7f2ec5ffb640 1 -- 172.21.15.72:0/1818997665 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5512 (secure 0 0 0) 0x7f2ec009c140 con 0x7f2ec8107910 2024-03-22T01:33:52.497 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:33:52.497 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6835/3077216308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3077216308},{"type":"v1","addr":"172.21.15.72:6835","nonce":3077216308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2460254143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":2460254143},{"type":"v1","addr":"172.21.15.72:6837","nonce":2460254143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6833/1578701171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":1578701171},{"type":"v1","addr":"172.21.15.89:6833","nonce":1578701171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/2703047213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2703047213},{"type":"v1","addr":"172.21.15.98:6833","nonce":2703047213}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2589468787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2589468787},{"type":"v1","addr":"172.21.15.89:6835","nonce":2589468787}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/92657636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":92657636},{"type":"v1","addr":"172.21.15.98:6835","nonce":92657636}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-03-22T01:33:52.497 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 7 2024-03-22T01:33:52.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:52.500+0000 7f2eceeeb640 1 -- 172.21.15.72:0/1818997665 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f2eb80a2fa0 msgr2=0x7f2eb80a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:52.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:52.500+0000 7f2eceeeb640 1 --2- 172.21.15.72:0/1818997665 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f2eb80a2fa0 0x7f2eb80a5460 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc00bbe0 tx=0x7f2ebc00b390 comp rx=0 tx=0).stop 2024-03-22T01:33:52.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:52.500+0000 7f2eceeeb640 1 -- 172.21.15.72:0/1818997665 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2ec8107910 msgr2=0x7f2ec819f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:52.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:52.500+0000 7f2eceeeb640 1 --2- 172.21.15.72:0/1818997665 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2ec8107910 0x7f2ec819f490 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f2ec000eb10 tx=0x7f2ec000c2d0 comp rx=0 tx=0).stop 2024-03-22T01:33:52.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:52.500+0000 7f2eceeeb640 1 -- 172.21.15.72:0/1818997665 shutdown_connections 2024-03-22T01:33:52.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:52.500+0000 7f2eceeeb640 1 --2- 172.21.15.72:0/1818997665 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f2eb80a2fa0 0x7f2eb80a5460 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:52.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:52.500+0000 7f2eceeeb640 1 --2- 172.21.15.72:0/1818997665 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2ec8107910 0x7f2ec819f490 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:52.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:52.500+0000 7f2eceeeb640 1 --2- 172.21.15.72:0/1818997665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ec8104fe0 0x7f2ec8198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:52.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:52.500+0000 7f2eceeeb640 1 --2- 172.21.15.72:0/1818997665 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ec81026b0 0x7f2ec8197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:52.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:52.500+0000 7f2eceeeb640 1 -- 172.21.15.72:0/1818997665 >> 172.21.15.72:0/1818997665 conn(0x7f2ec80fc0a0 msgr2=0x7f2ec8102f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:33:52.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:52.500+0000 7f2eceeeb640 1 -- 172.21.15.72:0/1818997665 shutdown_connections 2024-03-22T01:33:52.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:52.500+0000 7f2eceeeb640 1 -- 172.21.15.72:0/1818997665 wait complete. 2024-03-22T01:33:52.508 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:52 smithi072 ceph-mon[32414]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:33:52.508 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:52 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:52.509 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:52 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:52.509 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:52 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:52.509 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:52 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T01:33:52.509 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:52 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T01:33:52.509 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:52 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:33:52.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:52 smithi089 ceph-mon[35770]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:33:52.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:52 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:52.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:52 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:52.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:52 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:52.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:52 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T01:33:52.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:52 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T01:33:52.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:52 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:33:53.387 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:53 smithi089 ceph-mon[35770]: Deploying daemon mds.c on smithi098 2024-03-22T01:33:53.388 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:53 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.98:6834/92657636,v1:172.21.15.98:6835/92657636] up:boot 2024-03-22T01:33:53.388 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:53 smithi089 ceph-mon[35770]: fsmap 6 up:standby 2024-03-22T01:33:53.388 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:53 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-22T01:33:53.388 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:53 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/1818997665' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:33:53.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:53 smithi098 ceph-mon[36468]: Deploying daemon mds.c on smithi098 2024-03-22T01:33:53.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:53 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.98:6834/92657636,v1:172.21.15.98:6835/92657636] up:boot 2024-03-22T01:33:53.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:53 smithi098 ceph-mon[36468]: fsmap 6 up:standby 2024-03-22T01:33:53.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:53 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-22T01:33:53.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:53 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/1818997665' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:33:53.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:53 smithi072 ceph-mon[32414]: Deploying daemon mds.c on smithi098 2024-03-22T01:33:53.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:53 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.98:6834/92657636,v1:172.21.15.98:6835/92657636] up:boot 2024-03-22T01:33:53.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:53 smithi072 ceph-mon[32414]: fsmap 6 up:standby 2024-03-22T01:33:53.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:53 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-22T01:33:53.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:53 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/1818997665' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:33:54.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:54 smithi098 ceph-mon[36468]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:33:54.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:54 smithi089 ceph-mon[35770]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:33:54.511 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T01:33:54.540 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:54 smithi072 ceph-mon[32414]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:33:54.582 INFO:journalctl@ceph.mds.c.smithi098.stdout:Mar 22 01:33:54 smithi098 podman[67992]: 2024-03-22 01:33:54.560989287 +0000 UTC m=+0.350056548 container start df75b211444a6295b2516116e51af97874cf8e763d1577df437b20c4dbf94012 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-c, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-22T01:33:54.583 INFO:journalctl@ceph.mds.c.smithi098.stdout:Mar 22 01:33:54 smithi098 ceph-mds[68011]: -- 172.21.15.98:0/2023709708 <== mon.1 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556a595c9a40 con 0x556a5959cc00 2024-03-22T01:33:54.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:54.626+0000 7fb525d5c640 1 -- 172.21.15.72:0/2116938257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb520102710 msgr2=0x7fb520104b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:54.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:54.626+0000 7fb525d5c640 1 --2- 172.21.15.72:0/2116938257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb520102710 0x7fb520104b00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb510009860 tx=0x7fb51002bb70 comp rx=0 tx=0).stop 2024-03-22T01:33:54.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:54.626+0000 7fb525d5c640 1 -- 172.21.15.72:0/2116938257 shutdown_connections 2024-03-22T01:33:54.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:54.626+0000 7fb525d5c640 1 --2- 172.21.15.72:0/2116938257 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb520107970 0x7fb520109d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:54.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:54.626+0000 7fb525d5c640 1 --2- 172.21.15.72:0/2116938257 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb520105040 0x7fb520107430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:54.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:54.626+0000 7fb525d5c640 1 --2- 172.21.15.72:0/2116938257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb520102710 0x7fb520104b00 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:54.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:54.626+0000 7fb525d5c640 1 -- 172.21.15.72:0/2116938257 >> 172.21.15.72:0/2116938257 conn(0x7fb5200fc0c0 msgr2=0x7fb5200fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:33:54.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:54.626+0000 7fb525d5c640 1 -- 172.21.15.72:0/2116938257 shutdown_connections 2024-03-22T01:33:54.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:54.626+0000 7fb525d5c640 1 -- 172.21.15.72:0/2116938257 wait complete. 2024-03-22T01:33:54.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:54.626+0000 7fb525d5c640 1 Processor -- start 2024-03-22T01:33:54.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:54.627+0000 7fb525d5c640 1 -- start start 2024-03-22T01:33:54.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:54.627+0000 7fb525d5c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb520102710 0x7fb520197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:54.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:54.627+0000 7fb525d5c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb520105040 0x7fb5201983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:54.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:54.627+0000 7fb525d5c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb520107970 0x7fb52019f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:54.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:54.627+0000 7fb525d5c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fb52010d860 con 0x7fb520105040 2024-03-22T01:33:54.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:54.627+0000 7fb525d5c640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb52010d6e0 con 0x7fb520107970 2024-03-22T01:33:54.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:54.627+0000 7fb525d5c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb52010d9e0 con 0x7fb520102710 2024-03-22T01:33:54.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:54.627+0000 7fb51effd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb520105040 0x7fb5201983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:54.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:54.627+0000 7fb51effd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb520105040 0x7fb5201983f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48598/0 (socket says 172.21.15.72:48598) 2024-03-22T01:33:54.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:54.627+0000 7fb51effd640 1 -- 172.21.15.72:0/3083396784 learned_addr learned my addr 172.21.15.72:0/3083396784 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:33:54.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:54.627+0000 7fb51f7fe640 1 --2- 172.21.15.72:0/3083396784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb520102710 0x7fb520197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:54.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:54.627+0000 7fb51ffff640 1 --2- 172.21.15.72:0/3083396784 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb520107970 0x7fb52019f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:54.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:54.627+0000 7fb51f7fe640 1 -- 172.21.15.72:0/3083396784 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb520107970 msgr2=0x7fb52019f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:54.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:54.627+0000 7fb51f7fe640 1 --2- 172.21.15.72:0/3083396784 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb520107970 0x7fb52019f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:54.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:54.627+0000 7fb51f7fe640 1 -- 172.21.15.72:0/3083396784 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb520105040 msgr2=0x7fb5201983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:54.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:54.627+0000 7fb51f7fe640 1 --2- 172.21.15.72:0/3083396784 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb520105040 0x7fb5201983f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:54.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:54.627+0000 7fb51f7fe640 1 -- 172.21.15.72:0/3083396784 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb52019fad0 con 0x7fb520102710 2024-03-22T01:33:54.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:54.627+0000 7fb51effd640 1 --2- 172.21.15.72:0/3083396784 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb520105040 0x7fb5201983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:33:54.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:54.628+0000 7fb51f7fe640 1 --2- 172.21.15.72:0/3083396784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb520102710 0x7fb520197eb0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb51002c0b0 tx=0x7fb510032430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:33:54.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:54.628+0000 7fb51cff9640 1 -- 172.21.15.72:0/3083396784 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb510041020 con 0x7fb520102710 2024-03-22T01:33:54.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:54.628+0000 7fb51cff9640 1 -- 172.21.15.72:0/3083396784 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb51002c9f0 con 0x7fb520102710 2024-03-22T01:33:54.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:54.628+0000 7fb525d5c640 1 -- 172.21.15.72:0/3083396784 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb52019fd60 con 0x7fb520102710 2024-03-22T01:33:54.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:54.628+0000 7fb51cff9640 1 -- 172.21.15.72:0/3083396784 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb510039040 con 0x7fb520102710 2024-03-22T01:33:54.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:54.628+0000 7fb525d5c640 1 -- 172.21.15.72:0/3083396784 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5201a01d0 con 0x7fb520102710 2024-03-22T01:33:54.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:54.629+0000 7fb525d5c640 1 -- 172.21.15.72:0/3083396784 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5200fdbb0 con 0x7fb520102710 2024-03-22T01:33:54.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:54.630+0000 7fb51cff9640 1 -- 172.21.15.72:0/3083396784 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7fb510044020 con 0x7fb520102710 2024-03-22T01:33:54.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:54.630+0000 7fb51cff9640 1 --2- 172.21.15.72:0/3083396784 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fb5080a2f20 0x7fb5080a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:54.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:54.630+0000 7fb51effd640 1 --2- 172.21.15.72:0/3083396784 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fb5080a2f20 0x7fb5080a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:54.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:54.630+0000 7fb51cff9640 1 -- 172.21.15.72:0/3083396784 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fb5100f0910 con 0x7fb520102710 2024-03-22T01:33:54.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:54.631+0000 7fb51effd640 1 --2- 172.21.15.72:0/3083396784 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fb5080a2f20 0x7fb5080a53e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb5140096c0 tx=0x7fb514009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:33:54.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:54.634+0000 7fb51cff9640 1 -- 172.21.15.72:0/3083396784 <== mon.1 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb51004b020 con 0x7fb520102710 2024-03-22T01:33:54.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:54.855+0000 7fb525d5c640 1 -- 172.21.15.72:0/3083396784 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb52002d3b0 con 0x7fb520102710 2024-03-22T01:33:54.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:54.856+0000 7fb51cff9640 1 -- 172.21.15.72:0/3083396784 <== mon.1 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5512 (secure 0 0 0) 0x7fb5100bdbe0 con 0x7fb520102710 2024-03-22T01:33:54.855 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:33:54.855 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6835/3077216308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3077216308},{"type":"v1","addr":"172.21.15.72:6835","nonce":3077216308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2460254143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":2460254143},{"type":"v1","addr":"172.21.15.72:6837","nonce":2460254143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6833/1578701171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":1578701171},{"type":"v1","addr":"172.21.15.89:6833","nonce":1578701171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/2703047213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2703047213},{"type":"v1","addr":"172.21.15.98:6833","nonce":2703047213}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2589468787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2589468787},{"type":"v1","addr":"172.21.15.89:6835","nonce":2589468787}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/92657636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":92657636},{"type":"v1","addr":"172.21.15.98:6835","nonce":92657636}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-03-22T01:33:54.855 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 7 2024-03-22T01:33:54.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:54.858+0000 7fb525d5c640 1 -- 172.21.15.72:0/3083396784 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fb5080a2f20 msgr2=0x7fb5080a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:54.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:54.858+0000 7fb525d5c640 1 --2- 172.21.15.72:0/3083396784 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fb5080a2f20 0x7fb5080a53e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb5140096c0 tx=0x7fb514009260 comp rx=0 tx=0).stop 2024-03-22T01:33:54.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:54.858+0000 7fb525d5c640 1 -- 172.21.15.72:0/3083396784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb520102710 msgr2=0x7fb520197eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:54.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:54.858+0000 7fb525d5c640 1 --2- 172.21.15.72:0/3083396784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb520102710 0x7fb520197eb0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb51002c0b0 tx=0x7fb510032430 comp rx=0 tx=0).stop 2024-03-22T01:33:54.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:54.858+0000 7fb525d5c640 1 -- 172.21.15.72:0/3083396784 shutdown_connections 2024-03-22T01:33:54.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:54.858+0000 7fb525d5c640 1 --2- 172.21.15.72:0/3083396784 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fb5080a2f20 0x7fb5080a53e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:54.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:54.858+0000 7fb525d5c640 1 --2- 172.21.15.72:0/3083396784 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb520107970 0x7fb52019f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:54.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:54.858+0000 7fb525d5c640 1 --2- 172.21.15.72:0/3083396784 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb520105040 0x7fb5201983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:54.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:54.858+0000 7fb525d5c640 1 --2- 172.21.15.72:0/3083396784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb520102710 0x7fb520197eb0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:54.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:54.858+0000 7fb525d5c640 1 -- 172.21.15.72:0/3083396784 >> 172.21.15.72:0/3083396784 conn(0x7fb5200fc0c0 msgr2=0x7fb5201030f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:33:54.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:54.858+0000 7fb525d5c640 1 -- 172.21.15.72:0/3083396784 shutdown_connections 2024-03-22T01:33:54.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:54.858+0000 7fb525d5c640 1 -- 172.21.15.72:0/3083396784 wait complete. 2024-03-22T01:33:54.921 INFO:journalctl@ceph.mds.c.smithi098.stdout:Mar 22 01:33:54 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-c[68007]: starting mds.c at 2024-03-22T01:33:54.922 INFO:journalctl@ceph.mds.c.smithi098.stdout:Mar 22 01:33:54 smithi098 bash[67992]: df75b211444a6295b2516116e51af97874cf8e763d1577df437b20c4dbf94012 2024-03-22T01:33:54.922 INFO:journalctl@ceph.mds.c.smithi098.stdout:Mar 22 01:33:54 smithi098 systemd[1]: Started Ceph mds.c for 022a99bc-e7eb-11ee-95cd-87774f69a715. 2024-03-22T01:33:56.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:55 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:56.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:55 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:56.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:55 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:56.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:55 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T01:33:56.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:55 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T01:33:56.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:55 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:33:56.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:55 smithi089 ceph-mon[35770]: Deploying daemon mds.h on smithi089 2024-03-22T01:33:56.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:55 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/3083396784' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:33:56.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:55 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.98:6836/781860371,v1:172.21.15.98:6837/781860371] up:boot 2024-03-22T01:33:56.109 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:55 smithi089 ceph-mon[35770]: fsmap 7 up:standby 2024-03-22T01:33:56.109 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:55 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-22T01:33:56.109 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:55 smithi089 ceph-mon[35770]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:33:56.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:55 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:56.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:55 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:56.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:55 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:56.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:55 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T01:33:56.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:55 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T01:33:56.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:55 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:33:56.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:55 smithi072 ceph-mon[32414]: Deploying daemon mds.h on smithi089 2024-03-22T01:33:56.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:55 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/3083396784' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:33:56.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:55 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.98:6836/781860371,v1:172.21.15.98:6837/781860371] up:boot 2024-03-22T01:33:56.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:55 smithi072 ceph-mon[32414]: fsmap 7 up:standby 2024-03-22T01:33:56.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:55 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-22T01:33:56.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:55 smithi072 ceph-mon[32414]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:33:56.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:55 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:56.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:55 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:56.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:55 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:56.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:55 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T01:33:56.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:55 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T01:33:56.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:55 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:33:56.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:55 smithi098 ceph-mon[36468]: Deploying daemon mds.h on smithi089 2024-03-22T01:33:56.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:55 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/3083396784' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:33:56.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:55 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.98:6836/781860371,v1:172.21.15.98:6837/781860371] up:boot 2024-03-22T01:33:56.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:55 smithi098 ceph-mon[36468]: fsmap 7 up:standby 2024-03-22T01:33:56.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:55 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-22T01:33:56.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:55 smithi098 ceph-mon[36468]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:33:56.868 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T01:33:56.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:56.979+0000 7fbaee5ac640 1 -- 172.21.15.72:0/2340159178 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbae8107970 msgr2=0x7fbae8109d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:56.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:56.979+0000 7fbaee5ac640 1 --2- 172.21.15.72:0/2340159178 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbae8107970 0x7fbae8109d60 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fbae0009ca0 tx=0x7fbae002c0f0 comp rx=0 tx=0).stop 2024-03-22T01:33:56.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:56.979+0000 7fbaee5ac640 1 -- 172.21.15.72:0/2340159178 shutdown_connections 2024-03-22T01:33:56.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:56.979+0000 7fbaee5ac640 1 --2- 172.21.15.72:0/2340159178 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbae8107970 0x7fbae8109d60 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:56.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:56.979+0000 7fbaee5ac640 1 --2- 172.21.15.72:0/2340159178 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbae8105040 0x7fbae8107430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:56.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:56.979+0000 7fbaee5ac640 1 --2- 172.21.15.72:0/2340159178 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbae8102710 0x7fbae8104b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:56.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:56.979+0000 7fbaee5ac640 1 -- 172.21.15.72:0/2340159178 >> 172.21.15.72:0/2340159178 conn(0x7fbae80fc0c0 msgr2=0x7fbae80fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:33:56.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:56.979+0000 7fbaee5ac640 1 -- 172.21.15.72:0/2340159178 shutdown_connections 2024-03-22T01:33:56.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:56.979+0000 7fbaee5ac640 1 -- 172.21.15.72:0/2340159178 wait complete. 2024-03-22T01:33:56.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:56.979+0000 7fbaee5ac640 1 Processor -- start 2024-03-22T01:33:56.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:56.979+0000 7fbaee5ac640 1 -- start start 2024-03-22T01:33:56.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:56.980+0000 7fbaee5ac640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbae8102710 0x7fbae8197e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:56.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:56.980+0000 7fbaee5ac640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbae8105040 0x7fbae81983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:56.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:56.980+0000 7fbaee5ac640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbae8107970 0x7fbae819f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:56.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:56.980+0000 7fbaee5ac640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fbae810d840 con 0x7fbae8105040 2024-03-22T01:33:56.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:56.980+0000 7fbaee5ac640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fbae810d6c0 con 0x7fbae8102710 2024-03-22T01:33:56.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:56.980+0000 7fbaee5ac640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbae810d9c0 con 0x7fbae8107970 2024-03-22T01:33:56.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:56.980+0000 7fbae77fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbae8105040 0x7fbae81983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:56.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:56.980+0000 7fbae77fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbae8105040 0x7fbae81983a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48620/0 (socket says 172.21.15.72:48620) 2024-03-22T01:33:56.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:56.980+0000 7fbae77fe640 1 -- 172.21.15.72:0/808064187 learned_addr learned my addr 172.21.15.72:0/808064187 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:33:56.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:56.980+0000 7fbae7fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbae8102710 0x7fbae8197e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:56.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:56.980+0000 7fbaecb49640 1 --2- 172.21.15.72:0/808064187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbae8107970 0x7fbae819f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:56.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:56.980+0000 7fbae77fe640 1 -- 172.21.15.72:0/808064187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbae8107970 msgr2=0x7fbae819f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:56.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:56.980+0000 7fbae77fe640 1 --2- 172.21.15.72:0/808064187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbae8107970 0x7fbae819f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:56.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:56.980+0000 7fbae77fe640 1 -- 172.21.15.72:0/808064187 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbae8102710 msgr2=0x7fbae8197e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:56.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:56.980+0000 7fbae77fe640 1 --2- 172.21.15.72:0/808064187 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbae8102710 0x7fbae8197e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:56.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:56.980+0000 7fbae77fe640 1 -- 172.21.15.72:0/808064187 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbae819fa80 con 0x7fbae8105040 2024-03-22T01:33:56.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:56.980+0000 7fbae7fff640 1 --2- 172.21.15.72:0/808064187 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbae8102710 0x7fbae8197e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:33:56.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:56.980+0000 7fbae77fe640 1 --2- 172.21.15.72:0/808064187 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbae8105040 0x7fbae81983a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fbadc00e9c0 tx=0x7fbadc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:33:56.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:56.980+0000 7fbae57fa640 1 -- 172.21.15.72:0/808064187 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbadc00cd80 con 0x7fbae8105040 2024-03-22T01:33:56.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:56.980+0000 7fbaee5ac640 1 -- 172.21.15.72:0/808064187 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbae819fd70 con 0x7fbae8105040 2024-03-22T01:33:56.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:56.980+0000 7fbae57fa640 1 -- 172.21.15.72:0/808064187 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbadc005500 con 0x7fbae8105040 2024-03-22T01:33:56.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:56.981+0000 7fbae57fa640 1 -- 172.21.15.72:0/808064187 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbadc010690 con 0x7fbae8105040 2024-03-22T01:33:56.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:56.981+0000 7fbaee5ac640 1 -- 172.21.15.72:0/808064187 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbae81a0280 con 0x7fbae8105040 2024-03-22T01:33:56.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:56.982+0000 7fbaee5ac640 1 -- 172.21.15.72:0/808064187 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbae80fdbb0 con 0x7fbae8105040 2024-03-22T01:33:56.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:56.983+0000 7fbae57fa640 1 -- 172.21.15.72:0/808064187 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7fbadc0046c0 con 0x7fbae8105040 2024-03-22T01:33:56.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:56.983+0000 7fbae57fa640 1 --2- 172.21.15.72:0/808064187 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fbad00a2f20 0x7fbad00a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:56.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:56.983+0000 7fbae7fff640 1 --2- 172.21.15.72:0/808064187 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fbad00a2f20 0x7fbad00a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:56.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:56.983+0000 7fbae57fa640 1 -- 172.21.15.72:0/808064187 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fbadc0cf290 con 0x7fbae8105040 2024-03-22T01:33:56.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:56.984+0000 7fbae7fff640 1 --2- 172.21.15.72:0/808064187 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fbad00a2f20 0x7fbad00a53e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fbad8009fc0 tx=0x7fbad8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:33:56.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:56.988+0000 7fbae57fa640 1 -- 172.21.15.72:0/808064187 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbadc09c560 con 0x7fbae8105040 2024-03-22T01:33:57.108 INFO:journalctl@ceph.mds.h.smithi089.stdout:Mar 22 01:33:57 smithi089 systemd[1]: Starting Ceph mds.h for 022a99bc-e7eb-11ee-95cd-87774f69a715... 2024-03-22T01:33:57.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:57.200+0000 7fbaee5ac640 1 -- 172.21.15.72:0/808064187 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbae802d3b0 con 0x7fbae8105040 2024-03-22T01:33:57.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:57.201+0000 7fbae57fa640 1 -- 172.21.15.72:0/808064187 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6334 (secure 0 0 0) 0x7fbadc09be60 con 0x7fbae8105040 2024-03-22T01:33:57.200 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:33:57.200 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6835/3077216308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3077216308},{"type":"v1","addr":"172.21.15.72:6835","nonce":3077216308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2460254143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":2460254143},{"type":"v1","addr":"172.21.15.72:6837","nonce":2460254143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6833/1578701171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":1578701171},{"type":"v1","addr":"172.21.15.89:6833","nonce":1578701171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/2703047213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2703047213},{"type":"v1","addr":"172.21.15.98:6833","nonce":2703047213}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2589468787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2589468787},{"type":"v1","addr":"172.21.15.89:6835","nonce":2589468787}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/92657636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":92657636},{"type":"v1","addr":"172.21.15.98:6835","nonce":92657636}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24490,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/781860371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":781860371},{"type":"v1","addr":"172.21.15.98:6837","nonce":781860371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-03-22T01:33:57.200 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 8 2024-03-22T01:33:57.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:57.203+0000 7fbaee5ac640 1 -- 172.21.15.72:0/808064187 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fbad00a2f20 msgr2=0x7fbad00a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:57.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:57.203+0000 7fbaee5ac640 1 --2- 172.21.15.72:0/808064187 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fbad00a2f20 0x7fbad00a53e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fbad8009fc0 tx=0x7fbad8005f90 comp rx=0 tx=0).stop 2024-03-22T01:33:57.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:57.203+0000 7fbaee5ac640 1 -- 172.21.15.72:0/808064187 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbae8105040 msgr2=0x7fbae81983a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:57.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:57.203+0000 7fbaee5ac640 1 --2- 172.21.15.72:0/808064187 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbae8105040 0x7fbae81983a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fbadc00e9c0 tx=0x7fbadc00ee80 comp rx=0 tx=0).stop 2024-03-22T01:33:57.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:57.203+0000 7fbaee5ac640 1 -- 172.21.15.72:0/808064187 shutdown_connections 2024-03-22T01:33:57.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:57.203+0000 7fbaee5ac640 1 --2- 172.21.15.72:0/808064187 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fbad00a2f20 0x7fbad00a53e0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:57.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:57.203+0000 7fbaee5ac640 1 --2- 172.21.15.72:0/808064187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbae8107970 0x7fbae819f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:57.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:57.203+0000 7fbaee5ac640 1 --2- 172.21.15.72:0/808064187 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbae8105040 0x7fbae81983a0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:57.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:57.203+0000 7fbaee5ac640 1 --2- 172.21.15.72:0/808064187 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbae8102710 0x7fbae8197e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:57.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:57.203+0000 7fbaee5ac640 1 -- 172.21.15.72:0/808064187 >> 172.21.15.72:0/808064187 conn(0x7fbae80fc0c0 msgr2=0x7fbae8103050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:33:57.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:57.203+0000 7fbaee5ac640 1 -- 172.21.15.72:0/808064187 shutdown_connections 2024-03-22T01:33:57.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:57.203+0000 7fbaee5ac640 1 -- 172.21.15.72:0/808064187 wait complete. 2024-03-22T01:33:57.521 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:57 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/808064187' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:33:57.521 INFO:journalctl@ceph.mds.h.smithi089.stdout:Mar 22 01:33:57 smithi089 podman[66489]: 2024-03-22 01:33:57.146207143 +0000 UTC m=+0.018200444 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T01:33:57.522 INFO:journalctl@ceph.mds.h.smithi089.stdout:Mar 22 01:33:57 smithi089 podman[66489]: 2024-03-22 01:33:57.261888676 +0000 UTC m=+0.133881967 container create 608794ace6ce8cb33d0b3d112f490c6d148e6c6677cc873f1ce596f34ba3319a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-h, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-22T01:33:57.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:57 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/808064187' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:33:57.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:57 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/808064187' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:33:57.858 INFO:journalctl@ceph.mds.h.smithi089.stdout:Mar 22 01:33:57 smithi089 podman[66489]: 2024-03-22 01:33:57.523213968 +0000 UTC m=+0.395207249 container init 608794ace6ce8cb33d0b3d112f490c6d148e6c6677cc873f1ce596f34ba3319a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-h, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True) 2024-03-22T01:33:57.858 INFO:journalctl@ceph.mds.h.smithi089.stdout:Mar 22 01:33:57 smithi089 podman[66489]: 2024-03-22 01:33:57.525852052 +0000 UTC m=+0.397845340 container start 608794ace6ce8cb33d0b3d112f490c6d148e6c6677cc873f1ce596f34ba3319a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-h, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-22T01:33:57.858 INFO:journalctl@ceph.mds.h.smithi089.stdout:Mar 22 01:33:57 smithi089 ceph-mds[66509]: -- 172.21.15.89:0/3294611700 <== mon.2 v2:172.21.15.89:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564fa07e5a40 con 0x564fa07b9000 2024-03-22T01:33:57.858 INFO:journalctl@ceph.mds.h.smithi089.stdout:Mar 22 01:33:57 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-h[66505]: starting mds.h at 2024-03-22T01:33:57.858 INFO:journalctl@ceph.mds.h.smithi089.stdout:Mar 22 01:33:57 smithi089 bash[66489]: 608794ace6ce8cb33d0b3d112f490c6d148e6c6677cc873f1ce596f34ba3319a 2024-03-22T01:33:57.858 INFO:journalctl@ceph.mds.h.smithi089.stdout:Mar 22 01:33:57 smithi089 systemd[1]: Started Ceph mds.h for 022a99bc-e7eb-11ee-95cd-87774f69a715. 2024-03-22T01:33:58.577 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:58 smithi098 ceph-mon[36468]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:33:58.577 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:58 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:58.577 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:58 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:58.577 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:58 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:58.577 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:58 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T01:33:58.577 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:58 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T01:33:58.577 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:58 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:33:58.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:58 smithi089 ceph-mon[35770]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:33:58.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:58 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:58.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:58 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:58.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:58 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:58.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:58 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T01:33:58.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:58 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T01:33:58.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:58 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:33:58.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:58 smithi072 ceph-mon[32414]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:33:58.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:58 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:58.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:58 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:58.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:58 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:33:58.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:58 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T01:33:58.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:58 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T01:33:58.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:58 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:33:59.213 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T01:33:59.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:59.327+0000 7f89be2df640 1 -- 172.21.15.72:0/404783179 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f89b8102640 msgr2=0x7f89b8104a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:59.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:59.327+0000 7f89be2df640 1 --2- 172.21.15.72:0/404783179 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f89b8102640 0x7f89b8104a30 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f89a8009900 tx=0x7f89a802bc30 comp rx=0 tx=0).stop 2024-03-22T01:33:59.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:59.327+0000 7f89be2df640 1 -- 172.21.15.72:0/404783179 shutdown_connections 2024-03-22T01:33:59.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:59.327+0000 7f89be2df640 1 --2- 172.21.15.72:0/404783179 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89b81078a0 0x7f89b8109cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:59.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:59.327+0000 7f89be2df640 1 --2- 172.21.15.72:0/404783179 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89b8104f70 0x7f89b8107360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:59.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:59.327+0000 7f89be2df640 1 --2- 172.21.15.72:0/404783179 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f89b8102640 0x7f89b8104a30 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:59.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:59.327+0000 7f89be2df640 1 -- 172.21.15.72:0/404783179 >> 172.21.15.72:0/404783179 conn(0x7f89b80fc030 msgr2=0x7f89b80fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:33:59.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:59.327+0000 7f89be2df640 1 -- 172.21.15.72:0/404783179 shutdown_connections 2024-03-22T01:33:59.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:59.327+0000 7f89be2df640 1 -- 172.21.15.72:0/404783179 wait complete. 2024-03-22T01:33:59.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:59.327+0000 7f89be2df640 1 Processor -- start 2024-03-22T01:33:59.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:59.327+0000 7f89be2df640 1 -- start start 2024-03-22T01:33:59.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:59.328+0000 7f89be2df640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89b8102640 0x7f89b8101fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:59.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:59.328+0000 7f89be2df640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f89b8104f70 0x7f89b81005f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:59.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:59.328+0000 7f89be2df640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89b81078a0 0x7f89b8100b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:59.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:59.328+0000 7f89be2df640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f89b810d840 con 0x7f89b8104f70 2024-03-22T01:33:59.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:59.328+0000 7f89be2df640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f89b810d6c0 con 0x7f89b81078a0 2024-03-22T01:33:59.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:59.328+0000 7f89be2df640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f89b810d9c0 con 0x7f89b8102640 2024-03-22T01:33:59.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:59.328+0000 7f89b77fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f89b8104f70 0x7f89b81005f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:59.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:59.328+0000 7f89b7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89b8102640 0x7f89b8101fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:59.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:59.328+0000 7f89b77fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f89b8104f70 0x7f89b81005f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48636/0 (socket says 172.21.15.72:48636) 2024-03-22T01:33:59.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:59.328+0000 7f89b77fe640 1 -- 172.21.15.72:0/3538000815 learned_addr learned my addr 172.21.15.72:0/3538000815 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:33:59.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:59.328+0000 7f89bc87c640 1 --2- 172.21.15.72:0/3538000815 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89b81078a0 0x7f89b8100b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:59.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:59.328+0000 7f89b77fe640 1 -- 172.21.15.72:0/3538000815 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89b8102640 msgr2=0x7f89b8101fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:59.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:59.328+0000 7f89b77fe640 1 --2- 172.21.15.72:0/3538000815 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89b8102640 0x7f89b8101fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:59.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:59.328+0000 7f89b77fe640 1 -- 172.21.15.72:0/3538000815 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89b81078a0 msgr2=0x7f89b8100b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:59.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:59.328+0000 7f89b77fe640 1 --2- 172.21.15.72:0/3538000815 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89b81078a0 0x7f89b8100b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:59.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:59.328+0000 7f89b77fe640 1 -- 172.21.15.72:0/3538000815 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89b81013c0 con 0x7f89b8104f70 2024-03-22T01:33:59.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:59.328+0000 7f89bc87c640 1 --2- 172.21.15.72:0/3538000815 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89b81078a0 0x7f89b8100b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:33:59.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:59.328+0000 7f89b77fe640 1 --2- 172.21.15.72:0/3538000815 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f89b8104f70 0x7f89b81005f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f89ac00c970 tx=0x7f89ac00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:33:59.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:59.328+0000 7f89b57fa640 1 -- 172.21.15.72:0/3538000815 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89ac007bb0 con 0x7f89b8104f70 2024-03-22T01:33:59.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:59.328+0000 7f89be2df640 1 -- 172.21.15.72:0/3538000815 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89b81a1fe0 con 0x7f89b8104f70 2024-03-22T01:33:59.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:59.328+0000 7f89b57fa640 1 -- 172.21.15.72:0/3538000815 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89ac007d50 con 0x7f89b8104f70 2024-03-22T01:33:59.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:59.328+0000 7f89b57fa640 1 -- 172.21.15.72:0/3538000815 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89ac00f450 con 0x7f89b8104f70 2024-03-22T01:33:59.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:59.328+0000 7f89be2df640 1 -- 172.21.15.72:0/3538000815 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89b81a2450 con 0x7f89b8104f70 2024-03-22T01:33:59.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:59.330+0000 7f89be2df640 1 -- 172.21.15.72:0/3538000815 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89b80fdb20 con 0x7f89b8104f70 2024-03-22T01:33:59.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:59.331+0000 7f89b57fa640 1 -- 172.21.15.72:0/3538000815 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f89ac004f90 con 0x7f89b8104f70 2024-03-22T01:33:59.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:59.331+0000 7f89b57fa640 1 --2- 172.21.15.72:0/3538000815 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f89a00a2f20 0x7f89a00a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:33:59.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:59.331+0000 7f89b7fff640 1 --2- 172.21.15.72:0/3538000815 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f89a00a2f20 0x7f89a00a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:33:59.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:59.331+0000 7f89b57fa640 1 -- 172.21.15.72:0/3538000815 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f89ac0cecc0 con 0x7f89b8104f70 2024-03-22T01:33:59.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:59.332+0000 7f89b7fff640 1 --2- 172.21.15.72:0/3538000815 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f89a00a2f20 0x7f89a00a53e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f89a8005ab0 tx=0x7f89a8004f10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:33:59.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:59.336+0000 7f89b57fa640 1 -- 172.21.15.72:0/3538000815 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f89ac014030 con 0x7f89b8104f70 2024-03-22T01:33:59.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:59.547+0000 7f89be2df640 1 -- 172.21.15.72:0/3538000815 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f89b802d5b0 con 0x7f89b8104f70 2024-03-22T01:33:59.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:59.547+0000 7f89b57fa640 1 -- 172.21.15.72:0/3538000815 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7159 (secure 0 0 0) 0x7f89ac084e20 con 0x7f89b8104f70 2024-03-22T01:33:59.547 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:33:59.547 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6835/3077216308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3077216308},{"type":"v1","addr":"172.21.15.72:6835","nonce":3077216308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2460254143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":2460254143},{"type":"v1","addr":"172.21.15.72:6837","nonce":2460254143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6833/1578701171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":1578701171},{"type":"v1","addr":"172.21.15.89:6833","nonce":1578701171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/2703047213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2703047213},{"type":"v1","addr":"172.21.15.98:6833","nonce":2703047213}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2589468787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2589468787},{"type":"v1","addr":"172.21.15.89:6835","nonce":2589468787}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/92657636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":92657636},{"type":"v1","addr":"172.21.15.98:6835","nonce":92657636}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1662941782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1662941782},{"type":"v1","addr":"172.21.15.89:6837","nonce":1662941782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24490,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/781860371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":781860371},{"type":"v1","addr":"172.21.15.98:6837","nonce":781860371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-03-22T01:33:59.548 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 9 2024-03-22T01:33:59.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:59.549+0000 7f89be2df640 1 -- 172.21.15.72:0/3538000815 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f89a00a2f20 msgr2=0x7f89a00a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:59.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:59.549+0000 7f89be2df640 1 --2- 172.21.15.72:0/3538000815 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f89a00a2f20 0x7f89a00a53e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f89a8005ab0 tx=0x7f89a8004f10 comp rx=0 tx=0).stop 2024-03-22T01:33:59.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:59.549+0000 7f89be2df640 1 -- 172.21.15.72:0/3538000815 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f89b8104f70 msgr2=0x7f89b81005f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:33:59.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:59.549+0000 7f89be2df640 1 --2- 172.21.15.72:0/3538000815 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f89b8104f70 0x7f89b81005f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f89ac00c970 tx=0x7f89ac00ce30 comp rx=0 tx=0).stop 2024-03-22T01:33:59.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:59.549+0000 7f89be2df640 1 -- 172.21.15.72:0/3538000815 shutdown_connections 2024-03-22T01:33:59.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:59.549+0000 7f89be2df640 1 --2- 172.21.15.72:0/3538000815 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f89a00a2f20 0x7f89a00a53e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:59.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:59.549+0000 7f89be2df640 1 --2- 172.21.15.72:0/3538000815 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89b81078a0 0x7f89b8100b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:59.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:59.549+0000 7f89be2df640 1 --2- 172.21.15.72:0/3538000815 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f89b8104f70 0x7f89b81005f0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:59.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:59.549+0000 7f89be2df640 1 --2- 172.21.15.72:0/3538000815 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89b8102640 0x7f89b8101fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:33:59.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:59.549+0000 7f89be2df640 1 -- 172.21.15.72:0/3538000815 >> 172.21.15.72:0/3538000815 conn(0x7f89b80fc030 msgr2=0x7f89b8102fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:33:59.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:59.549+0000 7f89be2df640 1 -- 172.21.15.72:0/3538000815 shutdown_connections 2024-03-22T01:33:59.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:33:59.549+0000 7f89be2df640 1 -- 172.21.15.72:0/3538000815 wait complete. 2024-03-22T01:33:59.559 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:59 smithi072 ceph-mon[32414]: Deploying daemon mds.i on smithi098 2024-03-22T01:33:59.559 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:59 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.89:6836/1662941782,v1:172.21.15.89:6837/1662941782] up:boot 2024-03-22T01:33:59.560 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:59 smithi072 ceph-mon[32414]: fsmap 8 up:standby 2024-03-22T01:33:59.560 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:33:59 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-22T01:33:59.605 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:59 smithi098 ceph-mon[36468]: Deploying daemon mds.i on smithi098 2024-03-22T01:33:59.605 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:59 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.89:6836/1662941782,v1:172.21.15.89:6837/1662941782] up:boot 2024-03-22T01:33:59.605 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:59 smithi098 ceph-mon[36468]: fsmap 8 up:standby 2024-03-22T01:33:59.605 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:33:59 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-22T01:33:59.607 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:59 smithi089 ceph-mon[35770]: Deploying daemon mds.i on smithi098 2024-03-22T01:33:59.607 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:59 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.89:6836/1662941782,v1:172.21.15.89:6837/1662941782] up:boot 2024-03-22T01:33:59.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:59 smithi089 ceph-mon[35770]: fsmap 8 up:standby 2024-03-22T01:33:59.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:33:59 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-22T01:33:59.922 INFO:journalctl@ceph.mds.i.smithi098.stdout:Mar 22 01:33:59 smithi098 systemd[1]: Starting Ceph mds.i for 022a99bc-e7eb-11ee-95cd-87774f69a715... 2024-03-22T01:34:00.267 INFO:journalctl@ceph.mds.i.smithi098.stdout:Mar 22 01:34:00 smithi098 podman[68316]: 2024-03-22 01:33:59.920090452 +0000 UTC m=+0.017447477 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T01:34:00.267 INFO:journalctl@ceph.mds.i.smithi098.stdout:Mar 22 01:34:00 smithi098 podman[68316]: 2024-03-22 01:34:00.030296046 +0000 UTC m=+0.127653057 container create 9ad857dd4375eb65cb9ab97e9e54c9d3028bbe3ea87193faf157ab633cdf5574 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-i, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-22T01:34:00.267 INFO:journalctl@ceph.mds.i.smithi098.stdout:Mar 22 01:34:00 smithi098 podman[68316]: 2024-03-22 01:34:00.258632252 +0000 UTC m=+0.355989264 container init 9ad857dd4375eb65cb9ab97e9e54c9d3028bbe3ea87193faf157ab633cdf5574 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-i, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0) 2024-03-22T01:34:00.267 INFO:journalctl@ceph.mds.i.smithi098.stdout:Mar 22 01:34:00 smithi098 podman[68316]: 2024-03-22 01:34:00.261443393 +0000 UTC m=+0.358800410 container start 9ad857dd4375eb65cb9ab97e9e54c9d3028bbe3ea87193faf157ab633cdf5574 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-i, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True) 2024-03-22T01:34:00.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:00 smithi089 ceph-mon[35770]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:34:00.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:00 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/3538000815' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:34:00.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:00 smithi072 ceph-mon[32414]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:34:00.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:00 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/3538000815' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:34:00.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:00 smithi098 ceph-mon[36468]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:34:00.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:00 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/3538000815' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:34:00.672 INFO:journalctl@ceph.mds.i.smithi098.stdout:Mar 22 01:34:00 smithi098 ceph-mds[68335]: -- 172.21.15.98:0/2500208959 <== mon.1 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e4e1791a40 con 0x55e4e1764800 2024-03-22T01:34:00.672 INFO:journalctl@ceph.mds.i.smithi098.stdout:Mar 22 01:34:00 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-i[68331]: starting mds.i at 2024-03-22T01:34:00.672 INFO:journalctl@ceph.mds.i.smithi098.stdout:Mar 22 01:34:00 smithi098 bash[68316]: 9ad857dd4375eb65cb9ab97e9e54c9d3028bbe3ea87193faf157ab633cdf5574 2024-03-22T01:34:00.672 INFO:journalctl@ceph.mds.i.smithi098.stdout:Mar 22 01:34:00 smithi098 systemd[1]: Started Ceph mds.i for 022a99bc-e7eb-11ee-95cd-87774f69a715. 2024-03-22T01:34:01.562 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T01:34:01.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:01.670+0000 7f0d79415640 1 -- 172.21.15.72:0/2595396073 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d74102e60 msgr2=0x7f0d741032e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:01.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:01.670+0000 7f0d79415640 1 --2- 172.21.15.72:0/2595396073 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d74102e60 0x7f0d741032e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0d60009930 tx=0x7f0d6002bc30 comp rx=0 tx=0).stop 2024-03-22T01:34:01.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:01.670+0000 7f0d79415640 1 -- 172.21.15.72:0/2595396073 shutdown_connections 2024-03-22T01:34:01.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:01.670+0000 7f0d79415640 1 --2- 172.21.15.72:0/2595396073 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d74103820 0x7f0d7410a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:01.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:01.670+0000 7f0d79415640 1 --2- 172.21.15.72:0/2595396073 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d74102e60 0x7f0d741032e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:01.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:01.670+0000 7f0d79415640 1 --2- 172.21.15.72:0/2595396073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d740ffe90 0x7f0d741002b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:01.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:01.670+0000 7f0d79415640 1 -- 172.21.15.72:0/2595396073 >> 172.21.15.72:0/2595396073 conn(0x7f0d740fb7c0 msgr2=0x7f0d740fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:01.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:01.670+0000 7f0d79415640 1 -- 172.21.15.72:0/2595396073 shutdown_connections 2024-03-22T01:34:01.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:01.670+0000 7f0d79415640 1 -- 172.21.15.72:0/2595396073 wait complete. 2024-03-22T01:34:01.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:01.670+0000 7f0d79415640 1 Processor -- start 2024-03-22T01:34:01.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:01.671+0000 7f0d79415640 1 -- start start 2024-03-22T01:34:01.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:01.671+0000 7f0d79415640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d740ffe90 0x7f0d74197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:01.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:01.671+0000 7f0d79415640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d74102e60 0x7f0d74198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:01.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:01.671+0000 7f0d79415640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d74103820 0x7f0d7419f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:01.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:01.671+0000 7f0d79415640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f0d7410d870 con 0x7f0d740ffe90 2024-03-22T01:34:01.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:01.671+0000 7f0d79415640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f0d7410d6f0 con 0x7f0d74103820 2024-03-22T01:34:01.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:01.671+0000 7f0d79415640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0d7410d9f0 con 0x7f0d74102e60 2024-03-22T01:34:01.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:01.671+0000 7f0d72ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d740ffe90 0x7f0d74197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:01.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:01.671+0000 7f0d72ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d740ffe90 0x7f0d74197f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54268/0 (socket says 172.21.15.72:54268) 2024-03-22T01:34:01.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:01.671+0000 7f0d72ffd640 1 -- 172.21.15.72:0/1301373488 learned_addr learned my addr 172.21.15.72:0/1301373488 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:34:01.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:01.671+0000 7f0d727fc640 1 --2- 172.21.15.72:0/1301373488 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d74102e60 0x7f0d74198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:01.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:01.671+0000 7f0d737fe640 1 --2- 172.21.15.72:0/1301373488 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d74103820 0x7f0d7419f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:01.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:01.671+0000 7f0d72ffd640 1 -- 172.21.15.72:0/1301373488 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d74102e60 msgr2=0x7f0d74198490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:01.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:01.671+0000 7f0d72ffd640 1 --2- 172.21.15.72:0/1301373488 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d74102e60 0x7f0d74198490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:01.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:01.671+0000 7f0d72ffd640 1 -- 172.21.15.72:0/1301373488 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d74103820 msgr2=0x7f0d7419f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:01.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:01.671+0000 7f0d72ffd640 1 --2- 172.21.15.72:0/1301373488 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d74103820 0x7f0d7419f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:01.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:01.671+0000 7f0d72ffd640 1 -- 172.21.15.72:0/1301373488 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d7419fb70 con 0x7f0d740ffe90 2024-03-22T01:34:01.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:01.671+0000 7f0d727fc640 1 --2- 172.21.15.72:0/1301373488 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d74102e60 0x7f0d74198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:34:01.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:01.671+0000 7f0d737fe640 1 --2- 172.21.15.72:0/1301373488 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d74103820 0x7f0d7419f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T01:34:01.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:01.671+0000 7f0d72ffd640 1 --2- 172.21.15.72:0/1301373488 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d740ffe90 0x7f0d74197f50 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c00eaa0 tx=0x7f0d6c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:01.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:01.671+0000 7f0d6bfff640 1 -- 172.21.15.72:0/1301373488 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d6c016070 con 0x7f0d740ffe90 2024-03-22T01:34:01.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:01.671+0000 7f0d79415640 1 -- 172.21.15.72:0/1301373488 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d7419fe60 con 0x7f0d740ffe90 2024-03-22T01:34:01.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:01.671+0000 7f0d6bfff640 1 -- 172.21.15.72:0/1301373488 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d6c003e50 con 0x7f0d740ffe90 2024-03-22T01:34:01.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:01.671+0000 7f0d6bfff640 1 -- 172.21.15.72:0/1301373488 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d6c0116d0 con 0x7f0d740ffe90 2024-03-22T01:34:01.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:01.671+0000 7f0d79415640 1 -- 172.21.15.72:0/1301373488 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d741a02c0 con 0x7f0d740ffe90 2024-03-22T01:34:01.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:01.672+0000 7f0d79415640 1 -- 172.21.15.72:0/1301373488 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d74109010 con 0x7f0d740ffe90 2024-03-22T01:34:01.675 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:01 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:01.675 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:01 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:01.675 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:01 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:01.676 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:01 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T01:34:01.676 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:01 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T01:34:01.676 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:01 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:34:01.676 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:01 smithi098 ceph-mon[36468]: Deploying daemon mds.g on smithi072 2024-03-22T01:34:01.676 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:01 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:boot 2024-03-22T01:34:01.676 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:01 smithi098 ceph-mon[36468]: fsmap 9 up:standby 2024-03-22T01:34:01.676 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:01 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-22T01:34:01.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:01.673+0000 7f0d6bfff640 1 -- 172.21.15.72:0/1301373488 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f0d6c019070 con 0x7f0d740ffe90 2024-03-22T01:34:01.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:01.674+0000 7f0d6bfff640 1 --2- 172.21.15.72:0/1301373488 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f0d580a2fa0 0x7f0d580a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:01.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:01.674+0000 7f0d727fc640 1 --2- 172.21.15.72:0/1301373488 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f0d580a2fa0 0x7f0d580a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:01.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:01.674+0000 7f0d6bfff640 1 -- 172.21.15.72:0/1301373488 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f0d6c0cf180 con 0x7f0d740ffe90 2024-03-22T01:34:01.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:01.674+0000 7f0d727fc640 1 --2- 172.21.15.72:0/1301373488 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f0d580a2fa0 0x7f0d580a5460 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f0d60009900 tx=0x7f0d60005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:01.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:01.678+0000 7f0d6bfff640 1 -- 172.21.15.72:0/1301373488 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0d6c09c3b0 con 0x7f0d740ffe90 2024-03-22T01:34:01.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:01 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:01.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:01 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:01.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:01 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:01.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:01 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T01:34:01.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:01 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T01:34:01.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:01 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:34:01.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:01 smithi089 ceph-mon[35770]: Deploying daemon mds.g on smithi072 2024-03-22T01:34:01.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:01 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:boot 2024-03-22T01:34:01.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:01 smithi089 ceph-mon[35770]: fsmap 9 up:standby 2024-03-22T01:34:01.859 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:01 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-22T01:34:01.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:01 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:01.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:01 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:01.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:01 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:01.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:01 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T01:34:01.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:01 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T01:34:01.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:01 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:34:01.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:01 smithi072 ceph-mon[32414]: Deploying daemon mds.g on smithi072 2024-03-22T01:34:01.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:01 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:boot 2024-03-22T01:34:01.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:01 smithi072 ceph-mon[32414]: fsmap 9 up:standby 2024-03-22T01:34:01.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:01 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-22T01:34:01.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:01.895+0000 7f0d79415640 1 -- 172.21.15.72:0/1301373488 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0d7402d3b0 con 0x7f0d740ffe90 2024-03-22T01:34:01.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:01.896+0000 7f0d6bfff640 1 -- 172.21.15.72:0/1301373488 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+7986 (secure 0 0 0) 0x7f0d6c09bcb0 con 0x7f0d740ffe90 2024-03-22T01:34:01.896 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:34:01.896 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6835/3077216308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3077216308},{"type":"v1","addr":"172.21.15.72:6835","nonce":3077216308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2460254143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":2460254143},{"type":"v1","addr":"172.21.15.72:6837","nonce":2460254143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6833/1578701171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":1578701171},{"type":"v1","addr":"172.21.15.89:6833","nonce":1578701171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/2703047213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2703047213},{"type":"v1","addr":"172.21.15.98:6833","nonce":2703047213}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2589468787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2589468787},{"type":"v1","addr":"172.21.15.89:6835","nonce":2589468787}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/92657636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":92657636},{"type":"v1","addr":"172.21.15.98:6835","nonce":92657636}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1662941782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1662941782},{"type":"v1","addr":"172.21.15.89:6837","nonce":1662941782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24490,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/781860371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":781860371},{"type":"v1","addr":"172.21.15.98:6837","nonce":781860371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24517,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/3192173650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3192173650},{"type":"v1","addr":"172.21.15.98:6839","nonce":3192173650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-22T01:34:01.896 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 10 2024-03-22T01:34:01.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:01.898+0000 7f0d79415640 1 -- 172.21.15.72:0/1301373488 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f0d580a2fa0 msgr2=0x7f0d580a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:01.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:01.898+0000 7f0d79415640 1 --2- 172.21.15.72:0/1301373488 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f0d580a2fa0 0x7f0d580a5460 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f0d60009900 tx=0x7f0d60005fd0 comp rx=0 tx=0).stop 2024-03-22T01:34:01.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:01.898+0000 7f0d79415640 1 -- 172.21.15.72:0/1301373488 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d740ffe90 msgr2=0x7f0d74197f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:01.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:01.898+0000 7f0d79415640 1 --2- 172.21.15.72:0/1301373488 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d740ffe90 0x7f0d74197f50 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c00eaa0 tx=0x7f0d6c00ef60 comp rx=0 tx=0).stop 2024-03-22T01:34:01.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:01.898+0000 7f0d79415640 1 -- 172.21.15.72:0/1301373488 shutdown_connections 2024-03-22T01:34:01.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:01.898+0000 7f0d79415640 1 --2- 172.21.15.72:0/1301373488 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f0d580a2fa0 0x7f0d580a5460 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:01.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:01.898+0000 7f0d79415640 1 --2- 172.21.15.72:0/1301373488 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d74103820 0x7f0d7419f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:01.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:01.898+0000 7f0d79415640 1 --2- 172.21.15.72:0/1301373488 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d74102e60 0x7f0d74198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:01.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:01.898+0000 7f0d79415640 1 --2- 172.21.15.72:0/1301373488 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d740ffe90 0x7f0d74197f50 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:01.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:01.898+0000 7f0d79415640 1 -- 172.21.15.72:0/1301373488 >> 172.21.15.72:0/1301373488 conn(0x7f0d740fb7c0 msgr2=0x7f0d741061e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:01.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:01.898+0000 7f0d79415640 1 -- 172.21.15.72:0/1301373488 shutdown_connections 2024-03-22T01:34:01.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:01.898+0000 7f0d79415640 1 -- 172.21.15.72:0/1301373488 wait complete. 2024-03-22T01:34:02.445 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:02 smithi072 ceph-mon[32414]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:34:02.445 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:02 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/1301373488' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:34:02.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:02 smithi098 ceph-mon[36468]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:34:02.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:02 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/1301373488' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:34:02.847 INFO:journalctl@ceph.mds.g.smithi072.stdout:Mar 22 01:34:02 smithi072 systemd[1]: Starting Ceph mds.g for 022a99bc-e7eb-11ee-95cd-87774f69a715... 2024-03-22T01:34:02.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:02 smithi089 ceph-mon[35770]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:34:02.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:02 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/1301373488' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:34:03.112 INFO:journalctl@ceph.mds.g.smithi072.stdout:Mar 22 01:34:02 smithi072 podman[71461]: 2024-03-22 01:34:02.75134031 +0000 UTC m=+0.017622654 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T01:34:03.112 INFO:journalctl@ceph.mds.g.smithi072.stdout:Mar 22 01:34:02 smithi072 podman[71461]: 2024-03-22 01:34:02.857596769 +0000 UTC m=+0.123879101 container create d15911c82385e48400369753629635b07c0dcae5f53f9bcd0e828289c984b562 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-g, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319) 2024-03-22T01:34:03.112 INFO:journalctl@ceph.mds.g.smithi072.stdout:Mar 22 01:34:03 smithi072 podman[71461]: 2024-03-22 01:34:03.087375397 +0000 UTC m=+0.353657727 container init d15911c82385e48400369753629635b07c0dcae5f53f9bcd0e828289c984b562 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-g, ceph=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-22T01:34:03.112 INFO:journalctl@ceph.mds.g.smithi072.stdout:Mar 22 01:34:03 smithi072 podman[71461]: 2024-03-22 01:34:03.089940664 +0000 UTC m=+0.356222994 container start d15911c82385e48400369753629635b07c0dcae5f53f9bcd0e828289c984b562 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-g, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-22T01:34:03.372 INFO:journalctl@ceph.mds.g.smithi072.stdout:Mar 22 01:34:03 smithi072 ceph-mds[71481]: -- 172.21.15.72:0/1989662393 <== mon.0 v2:172.21.15.72:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557e4511da40 con 0x557e450f0800 2024-03-22T01:34:03.372 INFO:journalctl@ceph.mds.g.smithi072.stdout:Mar 22 01:34:03 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-g[71477]: starting mds.g at 2024-03-22T01:34:03.372 INFO:journalctl@ceph.mds.g.smithi072.stdout:Mar 22 01:34:03 smithi072 bash[71461]: d15911c82385e48400369753629635b07c0dcae5f53f9bcd0e828289c984b562 2024-03-22T01:34:03.372 INFO:journalctl@ceph.mds.g.smithi072.stdout:Mar 22 01:34:03 smithi072 systemd[1]: Started Ceph mds.g for 022a99bc-e7eb-11ee-95cd-87774f69a715. 2024-03-22T01:34:03.910 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T01:34:04.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:04.017+0000 7f047ea39640 1 -- 172.21.15.72:0/277106688 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04781026a0 msgr2=0x7f0478104a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:04.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:04.017+0000 7f047ea39640 1 --2- 172.21.15.72:0/277106688 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04781026a0 0x7f0478104a90 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0468009900 tx=0x7f046802bc30 comp rx=0 tx=0).stop 2024-03-22T01:34:04.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:04.017+0000 7f047ea39640 1 -- 172.21.15.72:0/277106688 shutdown_connections 2024-03-22T01:34:04.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:04.017+0000 7f047ea39640 1 --2- 172.21.15.72:0/277106688 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0478104fd0 0x7f04781073c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:04.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:04.017+0000 7f047ea39640 1 --2- 172.21.15.72:0/277106688 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0478107900 0x7f0478109cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:04.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:04.017+0000 7f047ea39640 1 --2- 172.21.15.72:0/277106688 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04781026a0 0x7f0478104a90 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:04.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:04.017+0000 7f047ea39640 1 -- 172.21.15.72:0/277106688 >> 172.21.15.72:0/277106688 conn(0x7f04780fc030 msgr2=0x7f04780fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:04.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:04.018+0000 7f047ea39640 1 -- 172.21.15.72:0/277106688 shutdown_connections 2024-03-22T01:34:04.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:04.018+0000 7f047ea39640 1 -- 172.21.15.72:0/277106688 wait complete. 2024-03-22T01:34:04.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:04.018+0000 7f047ea39640 1 Processor -- start 2024-03-22T01:34:04.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:04.018+0000 7f047ea39640 1 -- start start 2024-03-22T01:34:04.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:04.018+0000 7f047ea39640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04781026a0 0x7f0478197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:04.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:04.018+0000 7f047ea39640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0478104fd0 0x7f04781983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:04.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:04.018+0000 7f047ea39640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0478107900 0x7f047819f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:04.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:04.018+0000 7f047ea39640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f047810d880 con 0x7f04781026a0 2024-03-22T01:34:04.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:04.018+0000 7f047ea39640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f047810d700 con 0x7f0478107900 2024-03-22T01:34:04.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:04.018+0000 7f047ea39640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f047810da00 con 0x7f0478104fd0 2024-03-22T01:34:04.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:04.019+0000 7f0477fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04781026a0 0x7f0478197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:04.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:04.019+0000 7f0477fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04781026a0 0x7f0478197ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54300/0 (socket says 172.21.15.72:54300) 2024-03-22T01:34:04.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:04.019+0000 7f0477fff640 1 -- 172.21.15.72:0/1626564051 learned_addr learned my addr 172.21.15.72:0/1626564051 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:34:04.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:04.019+0000 7f047cfd6640 1 --2- 172.21.15.72:0/1626564051 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0478107900 0x7f047819f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:04.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:04.019+0000 7f04777fe640 1 --2- 172.21.15.72:0/1626564051 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0478104fd0 0x7f04781983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:04.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:04.019+0000 7f0477fff640 1 -- 172.21.15.72:0/1626564051 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0478104fd0 msgr2=0x7f04781983e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:04.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:04.019+0000 7f0477fff640 1 --2- 172.21.15.72:0/1626564051 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0478104fd0 0x7f04781983e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:04.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:04.019+0000 7f0477fff640 1 -- 172.21.15.72:0/1626564051 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0478107900 msgr2=0x7f047819f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:04.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:04.019+0000 7f0477fff640 1 --2- 172.21.15.72:0/1626564051 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0478107900 0x7f047819f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:04.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:04.019+0000 7f0477fff640 1 -- 172.21.15.72:0/1626564051 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f047819fac0 con 0x7f04781026a0 2024-03-22T01:34:04.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:04.019+0000 7f0477fff640 1 --2- 172.21.15.72:0/1626564051 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04781026a0 0x7f0478197ea0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f0468000c00 tx=0x7f0468005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:04.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:04.019+0000 7f04757fa640 1 -- 172.21.15.72:0/1626564051 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0468043020 con 0x7f04781026a0 2024-03-22T01:34:04.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:04.019+0000 7f047ea39640 1 -- 172.21.15.72:0/1626564051 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f047819fd50 con 0x7f04781026a0 2024-03-22T01:34:04.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:04.019+0000 7f04757fa640 1 -- 172.21.15.72:0/1626564051 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04680054e0 con 0x7f04781026a0 2024-03-22T01:34:04.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:04.019+0000 7f04757fa640 1 -- 172.21.15.72:0/1626564051 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0468003e60 con 0x7f04781026a0 2024-03-22T01:34:04.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:04.019+0000 7f047ea39640 1 -- 172.21.15.72:0/1626564051 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04781a01e0 con 0x7f04781026a0 2024-03-22T01:34:04.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:04.021+0000 7f047ea39640 1 -- 172.21.15.72:0/1626564051 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04780fdb20 con 0x7f04781026a0 2024-03-22T01:34:04.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:04.021+0000 7f04757fa640 1 -- 172.21.15.72:0/1626564051 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f0468046020 con 0x7f04781026a0 2024-03-22T01:34:04.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:04.022+0000 7f04757fa640 1 --2- 172.21.15.72:0/1626564051 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f04600a2f20 0x7f04600a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:04.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:04.022+0000 7f04777fe640 1 --2- 172.21.15.72:0/1626564051 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f04600a2f20 0x7f04600a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:04.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:04.022+0000 7f04757fa640 1 -- 172.21.15.72:0/1626564051 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f04680f1190 con 0x7f04781026a0 2024-03-22T01:34:04.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:04.023+0000 7f04777fe640 1 --2- 172.21.15.72:0/1626564051 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f04600a2f20 0x7f04600a53e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f04781993b0 tx=0x7f046c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:04.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:04.026+0000 7f04757fa640 1 -- 172.21.15.72:0/1626564051 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f046804d020 con 0x7f04781026a0 2024-03-22T01:34:04.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:04.273+0000 7f047ea39640 1 -- 172.21.15.72:0/1626564051 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f04780627c0 con 0x7f04781026a0 2024-03-22T01:34:04.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:04.273+0000 7f04757fa640 1 -- 172.21.15.72:0/1626564051 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8812 (secure 0 0 0) 0x7f04680be4e0 con 0x7f04781026a0 2024-03-22T01:34:04.274 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:34:04.274 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6835/3077216308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3077216308},{"type":"v1","addr":"172.21.15.72:6835","nonce":3077216308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2460254143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":2460254143},{"type":"v1","addr":"172.21.15.72:6837","nonce":2460254143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6839/2026432819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6838","nonce":2026432819},{"type":"v1","addr":"172.21.15.72:6839","nonce":2026432819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6833/1578701171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":1578701171},{"type":"v1","addr":"172.21.15.89:6833","nonce":1578701171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/2703047213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2703047213},{"type":"v1","addr":"172.21.15.98:6833","nonce":2703047213}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2589468787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2589468787},{"type":"v1","addr":"172.21.15.89:6835","nonce":2589468787}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/92657636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":92657636},{"type":"v1","addr":"172.21.15.98:6835","nonce":92657636}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1662941782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1662941782},{"type":"v1","addr":"172.21.15.89:6837","nonce":1662941782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24490,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/781860371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":781860371},{"type":"v1","addr":"172.21.15.98:6837","nonce":781860371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24517,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/3192173650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3192173650},{"type":"v1","addr":"172.21.15.98:6839","nonce":3192173650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-22T01:34:04.275 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 11 2024-03-22T01:34:04.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:04.276+0000 7f047ea39640 1 -- 172.21.15.72:0/1626564051 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f04600a2f20 msgr2=0x7f04600a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:04.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:04.276+0000 7f047ea39640 1 --2- 172.21.15.72:0/1626564051 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f04600a2f20 0x7f04600a53e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f04781993b0 tx=0x7f046c00b410 comp rx=0 tx=0).stop 2024-03-22T01:34:04.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:04.276+0000 7f047ea39640 1 -- 172.21.15.72:0/1626564051 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04781026a0 msgr2=0x7f0478197ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:04.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:04.276+0000 7f047ea39640 1 --2- 172.21.15.72:0/1626564051 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04781026a0 0x7f0478197ea0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f0468000c00 tx=0x7f0468005dd0 comp rx=0 tx=0).stop 2024-03-22T01:34:04.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:04.276+0000 7f047ea39640 1 -- 172.21.15.72:0/1626564051 shutdown_connections 2024-03-22T01:34:04.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:04.276+0000 7f047ea39640 1 --2- 172.21.15.72:0/1626564051 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f04600a2f20 0x7f04600a53e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:04.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:04.276+0000 7f047ea39640 1 --2- 172.21.15.72:0/1626564051 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0478107900 0x7f047819f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:04.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:04.276+0000 7f047ea39640 1 --2- 172.21.15.72:0/1626564051 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0478104fd0 0x7f04781983e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:04.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:04.276+0000 7f047ea39640 1 --2- 172.21.15.72:0/1626564051 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04781026a0 0x7f0478197ea0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:04.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:04.276+0000 7f047ea39640 1 -- 172.21.15.72:0/1626564051 >> 172.21.15.72:0/1626564051 conn(0x7f04780fc030 msgr2=0x7f0478102f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:04.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:04.276+0000 7f047ea39640 1 -- 172.21.15.72:0/1626564051 shutdown_connections 2024-03-22T01:34:04.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:04.276+0000 7f047ea39640 1 -- 172.21.15.72:0/1626564051 wait complete. 2024-03-22T01:34:04.457 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:04 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:04.457 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:04 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:04.457 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:04 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:04.458 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:04 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T01:34:04.458 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:04 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T01:34:04.458 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:04 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:34:04.458 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:04 smithi072 ceph-mon[32414]: Deploying daemon mds.a on smithi072 2024-03-22T01:34:04.458 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:04 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.72:6838/2026432819,v1:172.21.15.72:6839/2026432819] up:boot 2024-03-22T01:34:04.458 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:04 smithi072 ceph-mon[32414]: fsmap 10 up:standby 2024-03-22T01:34:04.458 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:04 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-22T01:34:04.458 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:04 smithi072 ceph-mon[32414]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:34:04.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:04 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:04.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:04 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:04.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:04 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:04.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:04 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T01:34:04.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:04 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T01:34:04.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:04 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:34:04.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:04 smithi089 ceph-mon[35770]: Deploying daemon mds.a on smithi072 2024-03-22T01:34:04.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:04 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.72:6838/2026432819,v1:172.21.15.72:6839/2026432819] up:boot 2024-03-22T01:34:04.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:04 smithi089 ceph-mon[35770]: fsmap 10 up:standby 2024-03-22T01:34:04.609 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:04 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-22T01:34:04.609 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:04 smithi089 ceph-mon[35770]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:34:04.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:04 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:04.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:04 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:04.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:04 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:04.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:04 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T01:34:04.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:04 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T01:34:04.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:04 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:34:04.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:04 smithi098 ceph-mon[36468]: Deploying daemon mds.a on smithi072 2024-03-22T01:34:04.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:04 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.72:6838/2026432819,v1:172.21.15.72:6839/2026432819] up:boot 2024-03-22T01:34:04.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:04 smithi098 ceph-mon[36468]: fsmap 10 up:standby 2024-03-22T01:34:04.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:04 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-22T01:34:04.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:04 smithi098 ceph-mon[36468]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:34:05.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:05 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/1626564051' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:34:05.536 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:05 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/1626564051' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:34:05.536 INFO:journalctl@ceph.mds.a.smithi072.stdout:Mar 22 01:34:05 smithi072 systemd[1]: Starting Ceph mds.a for 022a99bc-e7eb-11ee-95cd-87774f69a715... 2024-03-22T01:34:05.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:05 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/1626564051' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:34:05.804 INFO:journalctl@ceph.mds.a.smithi072.stdout:Mar 22 01:34:05 smithi072 podman[71823]: 2024-03-22 01:34:05.440466847 +0000 UTC m=+0.017568032 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T01:34:05.804 INFO:journalctl@ceph.mds.a.smithi072.stdout:Mar 22 01:34:05 smithi072 podman[71823]: 2024-03-22 01:34:05.56523174 +0000 UTC m=+0.142332920 container create 88b4b3dad62127331e96eda429708ef2847ec7f742f33bf573b8b17b560913a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-a, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-22T01:34:05.804 INFO:journalctl@ceph.mds.a.smithi072.stdout:Mar 22 01:34:05 smithi072 podman[71823]: 2024-03-22 01:34:05.805761523 +0000 UTC m=+0.382862713 container init 88b4b3dad62127331e96eda429708ef2847ec7f742f33bf573b8b17b560913a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-a, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-22T01:34:06.122 INFO:journalctl@ceph.mds.a.smithi072.stdout:Mar 22 01:34:05 smithi072 podman[71823]: 2024-03-22 01:34:05.808379093 +0000 UTC m=+0.385480276 container start 88b4b3dad62127331e96eda429708ef2847ec7f742f33bf573b8b17b560913a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-22T01:34:06.122 INFO:journalctl@ceph.mds.a.smithi072.stdout:Mar 22 01:34:05 smithi072 ceph-mds[71842]: -- 172.21.15.72:0/3192729939 <== mon.0 v2:172.21.15.72:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5616c1aeba40 con 0x5616c1abf000 2024-03-22T01:34:06.122 INFO:journalctl@ceph.mds.a.smithi072.stdout:Mar 22 01:34:05 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-a[71838]: starting mds.a at 2024-03-22T01:34:06.122 INFO:journalctl@ceph.mds.a.smithi072.stdout:Mar 22 01:34:05 smithi072 bash[71823]: 88b4b3dad62127331e96eda429708ef2847ec7f742f33bf573b8b17b560913a9 2024-03-22T01:34:06.122 INFO:journalctl@ceph.mds.a.smithi072.stdout:Mar 22 01:34:05 smithi072 systemd[1]: Started Ceph mds.a for 022a99bc-e7eb-11ee-95cd-87774f69a715. 2024-03-22T01:34:06.288 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T01:34:06.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:06.395+0000 7f747573a640 1 -- 172.21.15.72:0/1824415413 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7470101cb0 msgr2=0x7f7470102130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:06.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:06.395+0000 7f747573a640 1 --2- 172.21.15.72:0/1824415413 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7470101cb0 0x7f7470102130 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7458009900 tx=0x7f745802bc30 comp rx=0 tx=0).stop 2024-03-22T01:34:06.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:06.395+0000 7f747573a640 1 -- 172.21.15.72:0/1824415413 shutdown_connections 2024-03-22T01:34:06.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:06.395+0000 7f747573a640 1 --2- 172.21.15.72:0/1824415413 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7470102670 0x7f7470108f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:06.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:06.395+0000 7f747573a640 1 --2- 172.21.15.72:0/1824415413 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7470101cb0 0x7f7470102130 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:06.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:06.395+0000 7f747573a640 1 --2- 172.21.15.72:0/1824415413 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7470100ac0 0x7f7470100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:06.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:06.395+0000 7f747573a640 1 -- 172.21.15.72:0/1824415413 >> 172.21.15.72:0/1824415413 conn(0x7f74700fc030 msgr2=0x7f74700fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:06.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:06.395+0000 7f747573a640 1 -- 172.21.15.72:0/1824415413 shutdown_connections 2024-03-22T01:34:06.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:06.395+0000 7f747573a640 1 -- 172.21.15.72:0/1824415413 wait complete. 2024-03-22T01:34:06.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:06.396+0000 7f747573a640 1 Processor -- start 2024-03-22T01:34:06.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:06.396+0000 7f747573a640 1 -- start start 2024-03-22T01:34:06.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:06.396+0000 7f747573a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7470100ac0 0x7f7470196e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:06.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:06.396+0000 7f747573a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7470101cb0 0x7f7470197390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:06.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:06.396+0000 7f747573a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7470102670 0x7f747019e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:06.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:06.396+0000 7f747573a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f747010c7b0 con 0x7f7470102670 2024-03-22T01:34:06.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:06.396+0000 7f747573a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f747010c630 con 0x7f7470101cb0 2024-03-22T01:34:06.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:06.396+0000 7f747573a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f747010c930 con 0x7f7470100ac0 2024-03-22T01:34:06.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:06.396+0000 7f746f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7470102670 0x7f747019e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:06.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:06.396+0000 7f746f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7470102670 0x7f747019e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54332/0 (socket says 172.21.15.72:54332) 2024-03-22T01:34:06.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:06.396+0000 7f746f7fe640 1 -- 172.21.15.72:0/433457317 learned_addr learned my addr 172.21.15.72:0/433457317 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:34:06.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:06.396+0000 7f7467fff640 1 --2- 172.21.15.72:0/433457317 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7470101cb0 0x7f7470197390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:06.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:06.396+0000 7f746effd640 1 --2- 172.21.15.72:0/433457317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7470100ac0 0x7f7470196e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:06.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:06.396+0000 7f746f7fe640 1 -- 172.21.15.72:0/433457317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7470100ac0 msgr2=0x7f7470196e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:06.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:06.396+0000 7f746f7fe640 1 --2- 172.21.15.72:0/433457317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7470100ac0 0x7f7470196e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:06.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:06.396+0000 7f746f7fe640 1 -- 172.21.15.72:0/433457317 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7470101cb0 msgr2=0x7f7470197390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:06.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:06.396+0000 7f746f7fe640 1 --2- 172.21.15.72:0/433457317 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7470101cb0 0x7f7470197390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:06.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:06.396+0000 7f746f7fe640 1 -- 172.21.15.72:0/433457317 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f747019ea70 con 0x7f7470102670 2024-03-22T01:34:06.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:06.396+0000 7f746f7fe640 1 --2- 172.21.15.72:0/433457317 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7470102670 0x7f747019e400 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f746800eb10 tx=0x7f746800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:06.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:06.396+0000 7f746cff9640 1 -- 172.21.15.72:0/433457317 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7468004b20 con 0x7f7470102670 2024-03-22T01:34:06.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:06.396+0000 7f746cff9640 1 -- 172.21.15.72:0/433457317 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7468004cc0 con 0x7f7470102670 2024-03-22T01:34:06.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:06.396+0000 7f747573a640 1 -- 172.21.15.72:0/433457317 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f747019ed60 con 0x7f7470102670 2024-03-22T01:34:06.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:06.396+0000 7f746cff9640 1 -- 172.21.15.72:0/433457317 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7468010760 con 0x7f7470102670 2024-03-22T01:34:06.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:06.396+0000 7f747573a640 1 -- 172.21.15.72:0/433457317 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f747019f1d0 con 0x7f7470102670 2024-03-22T01:34:06.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:06.397+0000 7f747573a640 1 -- 172.21.15.72:0/433457317 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7470107d80 con 0x7f7470102670 2024-03-22T01:34:06.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:06.398+0000 7f746cff9640 1 -- 172.21.15.72:0/433457317 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f7468005020 con 0x7f7470102670 2024-03-22T01:34:06.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:06.399+0000 7f746cff9640 1 --2- 172.21.15.72:0/433457317 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f745c0a2f20 0x7f745c0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:06.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:06.399+0000 7f746effd640 1 --2- 172.21.15.72:0/433457317 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f745c0a2f20 0x7f745c0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:06.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:06.399+0000 7f746cff9640 1 -- 172.21.15.72:0/433457317 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f74680cf550 con 0x7f7470102670 2024-03-22T01:34:06.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:06.399+0000 7f746effd640 1 --2- 172.21.15.72:0/433457317 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f745c0a2f20 0x7f745c0a53e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7470101b10 tx=0x7f7460008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:06.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:06.403+0000 7f746cff9640 1 -- 172.21.15.72:0/433457317 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f746809c820 con 0x7f7470102670 2024-03-22T01:34:06.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:06 smithi089 ceph-mon[35770]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:34:06.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:06 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:06.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:06 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:06.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:06 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:06.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:06 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T01:34:06.609 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:06 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T01:34:06.609 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:06 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:34:06.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:06.618+0000 7f747573a640 1 -- 172.21.15.72:0/433457317 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7470062830 con 0x7f7470102670 2024-03-22T01:34:06.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:06.619+0000 7f746cff9640 1 -- 172.21.15.72:0/433457317 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9635 (secure 0 0 0) 0x7f746809c120 con 0x7f7470102670 2024-03-22T01:34:06.618 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:34:06.619 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6835/3077216308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3077216308},{"type":"v1","addr":"172.21.15.72:6835","nonce":3077216308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2460254143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":2460254143},{"type":"v1","addr":"172.21.15.72:6837","nonce":2460254143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6839/2026432819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6838","nonce":2026432819},{"type":"v1","addr":"172.21.15.72:6839","nonce":2026432819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6841/400094545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6840","nonce":400094545},{"type":"v1","addr":"172.21.15.72:6841","nonce":400094545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6833/1578701171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":1578701171},{"type":"v1","addr":"172.21.15.89:6833","nonce":1578701171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/2703047213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2703047213},{"type":"v1","addr":"172.21.15.98:6833","nonce":2703047213}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2589468787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2589468787},{"type":"v1","addr":"172.21.15.89:6835","nonce":2589468787}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/92657636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":92657636},{"type":"v1","addr":"172.21.15.98:6835","nonce":92657636}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1662941782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1662941782},{"type":"v1","addr":"172.21.15.89:6837","nonce":1662941782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24490,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/781860371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":781860371},{"type":"v1","addr":"172.21.15.98:6837","nonce":781860371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24517,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/3192173650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3192173650},{"type":"v1","addr":"172.21.15.98:6839","nonce":3192173650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-22T01:34:06.620 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 12 2024-03-22T01:34:06.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:06.620+0000 7f747573a640 1 -- 172.21.15.72:0/433457317 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f745c0a2f20 msgr2=0x7f745c0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:06.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:06.620+0000 7f747573a640 1 --2- 172.21.15.72:0/433457317 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f745c0a2f20 0x7f745c0a53e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7470101b10 tx=0x7f7460008040 comp rx=0 tx=0).stop 2024-03-22T01:34:06.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:06.621+0000 7f747573a640 1 -- 172.21.15.72:0/433457317 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7470102670 msgr2=0x7f747019e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:06.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:06.621+0000 7f747573a640 1 --2- 172.21.15.72:0/433457317 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7470102670 0x7f747019e400 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f746800eb10 tx=0x7f746800c2d0 comp rx=0 tx=0).stop 2024-03-22T01:34:06.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:06.621+0000 7f747573a640 1 -- 172.21.15.72:0/433457317 shutdown_connections 2024-03-22T01:34:06.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:06.621+0000 7f747573a640 1 --2- 172.21.15.72:0/433457317 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f745c0a2f20 0x7f745c0a53e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:06.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:06.621+0000 7f747573a640 1 --2- 172.21.15.72:0/433457317 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7470101cb0 0x7f7470197390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:06.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:06.621+0000 7f747573a640 1 --2- 172.21.15.72:0/433457317 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7470102670 0x7f747019e400 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:06.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:06.621+0000 7f747573a640 1 --2- 172.21.15.72:0/433457317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7470100ac0 0x7f7470196e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:06.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:06.621+0000 7f747573a640 1 -- 172.21.15.72:0/433457317 >> 172.21.15.72:0/433457317 conn(0x7f74700fc030 msgr2=0x7f7470104f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:06.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:06.621+0000 7f747573a640 1 -- 172.21.15.72:0/433457317 shutdown_connections 2024-03-22T01:34:06.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:06.621+0000 7f747573a640 1 -- 172.21.15.72:0/433457317 wait complete. 2024-03-22T01:34:06.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:06 smithi072 ceph-mon[32414]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:34:06.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:06 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:06.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:06 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:06.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:06 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:06.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:06 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T01:34:06.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:06 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T01:34:06.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:06 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:34:06.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:06 smithi098 ceph-mon[36468]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:34:06.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:06 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:06.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:06 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:06.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:06 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:06.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:06 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T01:34:06.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:06 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T01:34:06.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:06 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:34:07.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:07 smithi089 ceph-mon[35770]: Deploying daemon mds.b on smithi089 2024-03-22T01:34:07.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:07 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.72:6840/400094545,v1:172.21.15.72:6841/400094545] up:boot 2024-03-22T01:34:07.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:07 smithi089 ceph-mon[35770]: fsmap 11 up:standby 2024-03-22T01:34:07.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:07 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-22T01:34:07.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:07 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/433457317' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:34:07.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:07 smithi072 ceph-mon[32414]: Deploying daemon mds.b on smithi089 2024-03-22T01:34:07.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:07 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.72:6840/400094545,v1:172.21.15.72:6841/400094545] up:boot 2024-03-22T01:34:07.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:07 smithi072 ceph-mon[32414]: fsmap 11 up:standby 2024-03-22T01:34:07.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:07 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-22T01:34:07.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:07 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/433457317' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:34:07.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:07 smithi098 ceph-mon[36468]: Deploying daemon mds.b on smithi089 2024-03-22T01:34:07.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:07 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.72:6840/400094545,v1:172.21.15.72:6841/400094545] up:boot 2024-03-22T01:34:07.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:07 smithi098 ceph-mon[36468]: fsmap 11 up:standby 2024-03-22T01:34:07.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:07 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-22T01:34:07.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:07 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/433457317' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:34:08.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:08 smithi089 ceph-mon[35770]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:34:08.608 INFO:journalctl@ceph.mds.b.smithi089.stdout:Mar 22 01:34:08 smithi089 systemd[1]: Starting Ceph mds.b for 022a99bc-e7eb-11ee-95cd-87774f69a715... 2024-03-22T01:34:08.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:08 smithi072 ceph-mon[32414]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:34:08.632 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T01:34:08.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:08 smithi098 ceph-mon[36468]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:34:08.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:08.748+0000 7f6140de9640 1 -- 172.21.15.72:0/870374390 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f613c101cb0 msgr2=0x7f613c102130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:08.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:08.748+0000 7f6140de9640 1 --2- 172.21.15.72:0/870374390 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f613c101cb0 0x7f613c102130 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f6130009900 tx=0x7f613002bc30 comp rx=0 tx=0).stop 2024-03-22T01:34:08.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:08.748+0000 7f6140de9640 1 -- 172.21.15.72:0/870374390 shutdown_connections 2024-03-22T01:34:08.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:08.748+0000 7f6140de9640 1 --2- 172.21.15.72:0/870374390 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f613c102670 0x7f613c108f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:08.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:08.748+0000 7f6140de9640 1 --2- 172.21.15.72:0/870374390 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f613c101cb0 0x7f613c102130 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:08.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:08.748+0000 7f6140de9640 1 --2- 172.21.15.72:0/870374390 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f613c100ac0 0x7f613c100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:08.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:08.748+0000 7f6140de9640 1 -- 172.21.15.72:0/870374390 >> 172.21.15.72:0/870374390 conn(0x7f613c0fc030 msgr2=0x7f613c0fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:08.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:08.749+0000 7f6140de9640 1 -- 172.21.15.72:0/870374390 shutdown_connections 2024-03-22T01:34:08.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:08.749+0000 7f6140de9640 1 -- 172.21.15.72:0/870374390 wait complete. 2024-03-22T01:34:08.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:08.749+0000 7f6140de9640 1 Processor -- start 2024-03-22T01:34:08.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:08.749+0000 7f6140de9640 1 -- start start 2024-03-22T01:34:08.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:08.749+0000 7f6140de9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f613c100ac0 0x7f613c198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:08.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:08.749+0000 7f6140de9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f613c101cb0 0x7f613c198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:08.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:08.749+0000 7f6140de9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f613c102670 0x7f613c19f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:08.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:08.749+0000 7f6140de9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f613c10c7b0 con 0x7f613c101cb0 2024-03-22T01:34:08.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:08.749+0000 7f6140de9640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f613c10c630 con 0x7f613c100ac0 2024-03-22T01:34:08.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:08.749+0000 7f6140de9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f613c10c930 con 0x7f613c102670 2024-03-22T01:34:08.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:08.750+0000 7f6139d9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f613c101cb0 0x7f613c198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:08.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:08.750+0000 7f6139d9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f613c101cb0 0x7f613c198540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54346/0 (socket says 172.21.15.72:54346) 2024-03-22T01:34:08.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:08.750+0000 7f6139d9b640 1 -- 172.21.15.72:0/1734696483 learned_addr learned my addr 172.21.15.72:0/1734696483 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:34:08.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:08.750+0000 7f613a59c640 1 --2- 172.21.15.72:0/1734696483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f613c100ac0 0x7f613c198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:08.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:08.750+0000 7f613ad9d640 1 --2- 172.21.15.72:0/1734696483 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f613c102670 0x7f613c19f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:08.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:08.750+0000 7f6139d9b640 1 -- 172.21.15.72:0/1734696483 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f613c102670 msgr2=0x7f613c19f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:08.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:08.750+0000 7f6139d9b640 1 --2- 172.21.15.72:0/1734696483 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f613c102670 0x7f613c19f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:08.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:08.750+0000 7f6139d9b640 1 -- 172.21.15.72:0/1734696483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f613c100ac0 msgr2=0x7f613c198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:08.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:08.750+0000 7f6139d9b640 1 --2- 172.21.15.72:0/1734696483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f613c100ac0 0x7f613c198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:08.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:08.750+0000 7f6139d9b640 1 -- 172.21.15.72:0/1734696483 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f613c19fbb0 con 0x7f613c101cb0 2024-03-22T01:34:08.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:08.750+0000 7f613ad9d640 1 --2- 172.21.15.72:0/1734696483 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f613c102670 0x7f613c19f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T01:34:08.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:08.750+0000 7f6139d9b640 1 --2- 172.21.15.72:0/1734696483 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f613c101cb0 0x7f613c198540 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6130009230 tx=0x7f6130005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:08.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:08.750+0000 7f612b7fe640 1 -- 172.21.15.72:0/1734696483 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6130041070 con 0x7f613c101cb0 2024-03-22T01:34:08.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:08.750+0000 7f612b7fe640 1 -- 172.21.15.72:0/1734696483 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61300054e0 con 0x7f613c101cb0 2024-03-22T01:34:08.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:08.750+0000 7f6140de9640 1 -- 172.21.15.72:0/1734696483 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f613c19fe40 con 0x7f613c101cb0 2024-03-22T01:34:08.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:08.750+0000 7f612b7fe640 1 -- 172.21.15.72:0/1734696483 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6130034440 con 0x7f613c101cb0 2024-03-22T01:34:08.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:08.750+0000 7f6140de9640 1 -- 172.21.15.72:0/1734696483 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f613c1a02a0 con 0x7f613c101cb0 2024-03-22T01:34:08.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:08.751+0000 7f6140de9640 1 -- 172.21.15.72:0/1734696483 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f613c1a04d0 con 0x7f613c101cb0 2024-03-22T01:34:08.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:08.752+0000 7f612b7fe640 1 -- 172.21.15.72:0/1734696483 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f6130044070 con 0x7f613c101cb0 2024-03-22T01:34:08.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:08.753+0000 7f612b7fe640 1 --2- 172.21.15.72:0/1734696483 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f61240a2f20 0x7f61240a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:08.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:08.753+0000 7f613a59c640 1 --2- 172.21.15.72:0/1734696483 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f61240a2f20 0x7f61240a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:08.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:08.753+0000 7f612b7fe640 1 -- 172.21.15.72:0/1734696483 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f61300f0fe0 con 0x7f613c101cb0 2024-03-22T01:34:08.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:08.754+0000 7f613a59c640 1 --2- 172.21.15.72:0/1734696483 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f61240a2f20 0x7f61240a53e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f613c101b10 tx=0x7f612c009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:08.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:08.757+0000 7f612b7fe640 1 -- 172.21.15.72:0/1734696483 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f61300be230 con 0x7f613c101cb0 2024-03-22T01:34:08.869 INFO:journalctl@ceph.mds.b.smithi089.stdout:Mar 22 01:34:08 smithi089 podman[66796]: 2024-03-22 01:34:08.519194666 +0000 UTC m=+0.018321403 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T01:34:08.869 INFO:journalctl@ceph.mds.b.smithi089.stdout:Mar 22 01:34:08 smithi089 podman[66796]: 2024-03-22 01:34:08.626718085 +0000 UTC m=+0.125844805 container create a7ed85c8dcb6a6b159f4fa17f3f7b297477ee90fc20e4c492a4f1ddae8278581 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-b, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-22T01:34:08.870 INFO:journalctl@ceph.mds.b.smithi089.stdout:Mar 22 01:34:08 smithi089 podman[66796]: 2024-03-22 01:34:08.871239044 +0000 UTC m=+0.370365790 container init a7ed85c8dcb6a6b159f4fa17f3f7b297477ee90fc20e4c492a4f1ddae8278581 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-b, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-22T01:34:08.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:08.970+0000 7f6140de9640 1 -- 172.21.15.72:0/1734696483 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f613c1a0810 con 0x7f613c101cb0 2024-03-22T01:34:08.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:08.971+0000 7f612b7fe640 1 -- 172.21.15.72:0/1734696483 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9635 (secure 0 0 0) 0x7f61300bdb30 con 0x7f613c101cb0 2024-03-22T01:34:08.970 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:34:08.970 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6835/3077216308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3077216308},{"type":"v1","addr":"172.21.15.72:6835","nonce":3077216308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2460254143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":2460254143},{"type":"v1","addr":"172.21.15.72:6837","nonce":2460254143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6839/2026432819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6838","nonce":2026432819},{"type":"v1","addr":"172.21.15.72:6839","nonce":2026432819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6841/400094545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6840","nonce":400094545},{"type":"v1","addr":"172.21.15.72:6841","nonce":400094545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6833/1578701171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":1578701171},{"type":"v1","addr":"172.21.15.89:6833","nonce":1578701171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/2703047213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2703047213},{"type":"v1","addr":"172.21.15.98:6833","nonce":2703047213}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2589468787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2589468787},{"type":"v1","addr":"172.21.15.89:6835","nonce":2589468787}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/92657636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":92657636},{"type":"v1","addr":"172.21.15.98:6835","nonce":92657636}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1662941782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1662941782},{"type":"v1","addr":"172.21.15.89:6837","nonce":1662941782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24490,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/781860371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":781860371},{"type":"v1","addr":"172.21.15.98:6837","nonce":781860371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24517,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/3192173650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3192173650},{"type":"v1","addr":"172.21.15.98:6839","nonce":3192173650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-22T01:34:08.970 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 12 2024-03-22T01:34:08.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:08.973+0000 7f6140de9640 1 -- 172.21.15.72:0/1734696483 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f61240a2f20 msgr2=0x7f61240a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:08.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:08.973+0000 7f6140de9640 1 --2- 172.21.15.72:0/1734696483 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f61240a2f20 0x7f61240a53e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f613c101b10 tx=0x7f612c009110 comp rx=0 tx=0).stop 2024-03-22T01:34:08.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:08.973+0000 7f6140de9640 1 -- 172.21.15.72:0/1734696483 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f613c101cb0 msgr2=0x7f613c198540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:08.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:08.973+0000 7f6140de9640 1 --2- 172.21.15.72:0/1734696483 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f613c101cb0 0x7f613c198540 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6130009230 tx=0x7f6130005dd0 comp rx=0 tx=0).stop 2024-03-22T01:34:08.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:08.973+0000 7f6140de9640 1 -- 172.21.15.72:0/1734696483 shutdown_connections 2024-03-22T01:34:08.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:08.973+0000 7f6140de9640 1 --2- 172.21.15.72:0/1734696483 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f61240a2f20 0x7f61240a53e0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:08.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:08.973+0000 7f6140de9640 1 --2- 172.21.15.72:0/1734696483 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f613c102670 0x7f613c19f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:08.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:08.973+0000 7f6140de9640 1 --2- 172.21.15.72:0/1734696483 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f613c101cb0 0x7f613c198540 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:08.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:08.973+0000 7f6140de9640 1 --2- 172.21.15.72:0/1734696483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f613c100ac0 0x7f613c198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:08.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:08.973+0000 7f6140de9640 1 -- 172.21.15.72:0/1734696483 >> 172.21.15.72:0/1734696483 conn(0x7f613c0fc030 msgr2=0x7f613c0fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:08.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:08.973+0000 7f6140de9640 1 -- 172.21.15.72:0/1734696483 shutdown_connections 2024-03-22T01:34:08.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:08.973+0000 7f6140de9640 1 -- 172.21.15.72:0/1734696483 wait complete. 2024-03-22T01:34:09.256 INFO:journalctl@ceph.mds.b.smithi089.stdout:Mar 22 01:34:08 smithi089 podman[66796]: 2024-03-22 01:34:08.873769597 +0000 UTC m=+0.372896317 container start a7ed85c8dcb6a6b159f4fa17f3f7b297477ee90fc20e4c492a4f1ddae8278581 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-b, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-22T01:34:09.256 INFO:journalctl@ceph.mds.b.smithi089.stdout:Mar 22 01:34:08 smithi089 ceph-mds[66815]: -- 172.21.15.89:0/569712360 <== mon.2 v2:172.21.15.89:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b8749e5a40 con 0x55b8749b8c00 2024-03-22T01:34:09.257 INFO:journalctl@ceph.mds.b.smithi089.stdout:Mar 22 01:34:08 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-b[66811]: starting mds.b at 2024-03-22T01:34:09.257 INFO:journalctl@ceph.mds.b.smithi089.stdout:Mar 22 01:34:08 smithi089 bash[66796]: a7ed85c8dcb6a6b159f4fa17f3f7b297477ee90fc20e4c492a4f1ddae8278581 2024-03-22T01:34:09.257 INFO:journalctl@ceph.mds.b.smithi089.stdout:Mar 22 01:34:08 smithi089 systemd[1]: Started Ceph mds.b for 022a99bc-e7eb-11ee-95cd-87774f69a715. 2024-03-22T01:34:09.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:09 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/1734696483' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:34:09.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:09 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:09.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:09 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:09.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:09 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:09.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:09 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:09.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:09 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:09.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:09 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:34:09.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:09 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/1734696483' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:34:09.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:09 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:09.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:09 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:09.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:09 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:09.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:09 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:09.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:09 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:09.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:09 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:34:09.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:09 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/1734696483' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:34:09.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:09 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:09.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:09 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:09.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:09 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:09.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:09 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:09.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:09 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:09.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:09 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:34:10.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:10 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:boot 2024-03-22T01:34:10.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:10 smithi089 ceph-mon[35770]: fsmap 12 up:standby 2024-03-22T01:34:10.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:10 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-22T01:34:10.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:10 smithi089 ceph-mon[35770]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:34:10.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:10 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:boot 2024-03-22T01:34:10.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:10 smithi072 ceph-mon[32414]: fsmap 12 up:standby 2024-03-22T01:34:10.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:10 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-22T01:34:10.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:10 smithi072 ceph-mon[32414]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:34:10.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:10 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:boot 2024-03-22T01:34:10.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:10 smithi098 ceph-mon[36468]: fsmap 12 up:standby 2024-03-22T01:34:10.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:10 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-22T01:34:10.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:10 smithi098 ceph-mon[36468]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:34:10.984 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T01:34:11.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.094+0000 7f79d27e2640 1 -- 172.21.15.72:0/3569072838 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79cc101d20 msgr2=0x7f79cc1021a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:11.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.094+0000 7f79d27e2640 1 --2- 172.21.15.72:0/3569072838 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79cc101d20 0x7f79cc1021a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f79c0009900 tx=0x7f79c002bc30 comp rx=0 tx=0).stop 2024-03-22T01:34:11.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.094+0000 7f79d27e2640 1 -- 172.21.15.72:0/3569072838 shutdown_connections 2024-03-22T01:34:11.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.094+0000 7f79d27e2640 1 --2- 172.21.15.72:0/3569072838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79cc1026e0 0x7f79cc108f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:11.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.094+0000 7f79d27e2640 1 --2- 172.21.15.72:0/3569072838 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79cc101d20 0x7f79cc1021a0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:11.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.094+0000 7f79d27e2640 1 --2- 172.21.15.72:0/3569072838 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79cc100b30 0x7f79cc100f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:11.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.094+0000 7f79d27e2640 1 -- 172.21.15.72:0/3569072838 >> 172.21.15.72:0/3569072838 conn(0x7f79cc0fc0a0 msgr2=0x7f79cc0fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:11.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.094+0000 7f79d27e2640 1 -- 172.21.15.72:0/3569072838 shutdown_connections 2024-03-22T01:34:11.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.094+0000 7f79d27e2640 1 -- 172.21.15.72:0/3569072838 wait complete. 2024-03-22T01:34:11.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.095+0000 7f79d27e2640 1 Processor -- start 2024-03-22T01:34:11.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.095+0000 7f79d27e2640 1 -- start start 2024-03-22T01:34:11.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.095+0000 7f79d27e2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79cc100b30 0x7f79cc196e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:11.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.095+0000 7f79d27e2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79cc101d20 0x7f79cc197370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:11.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.095+0000 7f79d27e2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79cc1026e0 0x7f79cc19e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:11.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.095+0000 7f79d27e2640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f79cc10c7b0 con 0x7f79cc101d20 2024-03-22T01:34:11.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.095+0000 7f79d27e2640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f79cc10c630 con 0x7f79cc1026e0 2024-03-22T01:34:11.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.095+0000 7f79d27e2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f79cc10c930 con 0x7f79cc100b30 2024-03-22T01:34:11.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.095+0000 7f79cb7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79cc101d20 0x7f79cc197370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:11.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.095+0000 7f79cb7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79cc101d20 0x7f79cc197370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:35786/0 (socket says 172.21.15.72:35786) 2024-03-22T01:34:11.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.095+0000 7f79cb7fe640 1 -- 172.21.15.72:0/393678833 learned_addr learned my addr 172.21.15.72:0/393678833 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:34:11.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.095+0000 7f79cbfff640 1 --2- 172.21.15.72:0/393678833 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79cc100b30 0x7f79cc196e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:11.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.095+0000 7f79d0d7f640 1 --2- 172.21.15.72:0/393678833 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79cc1026e0 0x7f79cc19e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:11.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.095+0000 7f79cb7fe640 1 -- 172.21.15.72:0/393678833 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79cc100b30 msgr2=0x7f79cc196e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:11.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.095+0000 7f79cb7fe640 1 --2- 172.21.15.72:0/393678833 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79cc100b30 0x7f79cc196e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:11.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.095+0000 7f79cb7fe640 1 -- 172.21.15.72:0/393678833 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79cc1026e0 msgr2=0x7f79cc19e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:11.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.095+0000 7f79cb7fe640 1 --2- 172.21.15.72:0/393678833 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79cc1026e0 0x7f79cc19e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:11.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.095+0000 7f79cb7fe640 1 -- 172.21.15.72:0/393678833 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79cc19ea50 con 0x7f79cc101d20 2024-03-22T01:34:11.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.095+0000 7f79d0d7f640 1 --2- 172.21.15.72:0/393678833 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79cc1026e0 0x7f79cc19e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T01:34:11.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.096+0000 7f79cb7fe640 1 --2- 172.21.15.72:0/393678833 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79cc101d20 0x7f79cc197370 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f79c0009230 tx=0x7f79c0004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:11.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.096+0000 7f79c97fa640 1 -- 172.21.15.72:0/393678833 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79c0041070 con 0x7f79cc101d20 2024-03-22T01:34:11.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.096+0000 7f79d27e2640 1 -- 172.21.15.72:0/393678833 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79cc19ece0 con 0x7f79cc101d20 2024-03-22T01:34:11.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.096+0000 7f79c97fa640 1 -- 172.21.15.72:0/393678833 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79c00054e0 con 0x7f79cc101d20 2024-03-22T01:34:11.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.096+0000 7f79c97fa640 1 -- 172.21.15.72:0/393678833 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79c00346a0 con 0x7f79cc101d20 2024-03-22T01:34:11.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.096+0000 7f79d27e2640 1 -- 172.21.15.72:0/393678833 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79cc19f1d0 con 0x7f79cc101d20 2024-03-22T01:34:11.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.097+0000 7f79d27e2640 1 -- 172.21.15.72:0/393678833 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79cc107ed0 con 0x7f79cc101d20 2024-03-22T01:34:11.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.098+0000 7f79c97fa640 1 -- 172.21.15.72:0/393678833 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f79c0044070 con 0x7f79cc101d20 2024-03-22T01:34:11.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.098+0000 7f79c97fa640 1 --2- 172.21.15.72:0/393678833 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f79b40a2f20 0x7f79b40a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:11.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.099+0000 7f79cbfff640 1 --2- 172.21.15.72:0/393678833 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f79b40a2f20 0x7f79b40a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:11.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.099+0000 7f79c97fa640 1 -- 172.21.15.72:0/393678833 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f79c00f0da0 con 0x7f79cc101d20 2024-03-22T01:34:11.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.099+0000 7f79cbfff640 1 --2- 172.21.15.72:0/393678833 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f79b40a2f20 0x7f79b40a53e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f79cc101b80 tx=0x7f79bc008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:11.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.103+0000 7f79c97fa640 1 -- 172.21.15.72:0/393678833 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f79c00be070 con 0x7f79cc101d20 2024-03-22T01:34:11.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.325+0000 7f79d27e2640 1 -- 172.21.15.72:0/393678833 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f79cc02d5d0 con 0x7f79cc101d20 2024-03-22T01:34:11.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.326+0000 7f79c97fa640 1 -- 172.21.15.72:0/393678833 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10458 (secure 0 0 0) 0x7f79c00bd970 con 0x7f79cc101d20 2024-03-22T01:34:11.325 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:34:11.325 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6835/3077216308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3077216308},{"type":"v1","addr":"172.21.15.72:6835","nonce":3077216308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2460254143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":2460254143},{"type":"v1","addr":"172.21.15.72:6837","nonce":2460254143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6839/2026432819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6838","nonce":2026432819},{"type":"v1","addr":"172.21.15.72:6839","nonce":2026432819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6841/400094545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6840","nonce":400094545},{"type":"v1","addr":"172.21.15.72:6841","nonce":400094545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6833/1578701171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":1578701171},{"type":"v1","addr":"172.21.15.89:6833","nonce":1578701171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/2703047213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2703047213},{"type":"v1","addr":"172.21.15.98:6833","nonce":2703047213}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2589468787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2589468787},{"type":"v1","addr":"172.21.15.89:6835","nonce":2589468787}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/92657636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":92657636},{"type":"v1","addr":"172.21.15.98:6835","nonce":92657636}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1662941782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1662941782},{"type":"v1","addr":"172.21.15.89:6837","nonce":1662941782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24490,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/781860371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":781860371},{"type":"v1","addr":"172.21.15.98:6837","nonce":781860371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24517,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/3192173650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3192173650},{"type":"v1","addr":"172.21.15.98:6839","nonce":3192173650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24524,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6839/149200280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":149200280},{"type":"v1","addr":"172.21.15.89:6839","nonce":149200280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-03-22T01:34:11.325 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 13 2024-03-22T01:34:11.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.328+0000 7f79d27e2640 1 -- 172.21.15.72:0/393678833 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f79b40a2f20 msgr2=0x7f79b40a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:11.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.328+0000 7f79d27e2640 1 --2- 172.21.15.72:0/393678833 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f79b40a2f20 0x7f79b40a53e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f79cc101b80 tx=0x7f79bc008040 comp rx=0 tx=0).stop 2024-03-22T01:34:11.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.328+0000 7f79d27e2640 1 -- 172.21.15.72:0/393678833 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79cc101d20 msgr2=0x7f79cc197370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:11.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.328+0000 7f79d27e2640 1 --2- 172.21.15.72:0/393678833 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79cc101d20 0x7f79cc197370 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f79c0009230 tx=0x7f79c0004040 comp rx=0 tx=0).stop 2024-03-22T01:34:11.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.328+0000 7f79d27e2640 1 -- 172.21.15.72:0/393678833 shutdown_connections 2024-03-22T01:34:11.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.328+0000 7f79d27e2640 1 --2- 172.21.15.72:0/393678833 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f79b40a2f20 0x7f79b40a53e0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:11.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.328+0000 7f79d27e2640 1 --2- 172.21.15.72:0/393678833 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79cc1026e0 0x7f79cc19e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:11.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.328+0000 7f79d27e2640 1 --2- 172.21.15.72:0/393678833 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79cc101d20 0x7f79cc197370 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:11.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.328+0000 7f79d27e2640 1 --2- 172.21.15.72:0/393678833 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79cc100b30 0x7f79cc196e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:11.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.328+0000 7f79d27e2640 1 -- 172.21.15.72:0/393678833 >> 172.21.15.72:0/393678833 conn(0x7f79cc0fc0a0 msgr2=0x7f79cc1050a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:11.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.328+0000 7f79d27e2640 1 -- 172.21.15.72:0/393678833 shutdown_connections 2024-03-22T01:34:11.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.328+0000 7f79d27e2640 1 -- 172.21.15.72:0/393678833 wait complete. 2024-03-22T01:34:11.339 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T01:34:11.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.444+0000 7fe305f81640 1 -- 172.21.15.72:0/2831374685 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe300100d60 msgr2=0x7fe3001011e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:11.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.444+0000 7fe305f81640 1 --2- 172.21.15.72:0/2831374685 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe300100d60 0x7fe3001011e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe2f4009930 tx=0x7fe2f402bc30 comp rx=0 tx=0).stop 2024-03-22T01:34:11.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.444+0000 7fe305f81640 1 -- 172.21.15.72:0/2831374685 shutdown_connections 2024-03-22T01:34:11.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.444+0000 7fe305f81640 1 --2- 172.21.15.72:0/2831374685 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe300101720 0x7fe300108fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:11.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.444+0000 7fe305f81640 1 --2- 172.21.15.72:0/2831374685 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe300100d60 0x7fe3001011e0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:11.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.444+0000 7fe305f81640 1 --2- 172.21.15.72:0/2831374685 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3001020c0 0x7fe3001024e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:11.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.444+0000 7fe305f81640 1 -- 172.21.15.72:0/2831374685 >> 172.21.15.72:0/2831374685 conn(0x7fe3000fb7b0 msgr2=0x7fe3000fdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:11.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.444+0000 7fe305f81640 1 -- 172.21.15.72:0/2831374685 shutdown_connections 2024-03-22T01:34:11.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.444+0000 7fe305f81640 1 -- 172.21.15.72:0/2831374685 wait complete. 2024-03-22T01:34:11.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.445+0000 7fe305f81640 1 Processor -- start 2024-03-22T01:34:11.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.445+0000 7fe305f81640 1 -- start start 2024-03-22T01:34:11.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.445+0000 7fe305f81640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe300100d60 0x7fe3001007c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:11.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.445+0000 7fe305f81640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe300101720 0x7fe30019cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:11.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.445+0000 7fe305f81640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3001020c0 0x7fe30019f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:11.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.445+0000 7fe305f81640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fe30010c730 con 0x7fe3001020c0 2024-03-22T01:34:11.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.445+0000 7fe305f81640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe30010c5b0 con 0x7fe300101720 2024-03-22T01:34:11.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.445+0000 7fe305f81640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe30010c8b0 con 0x7fe300100d60 2024-03-22T01:34:11.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.445+0000 7fe2fffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3001020c0 0x7fe30019f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:11.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.445+0000 7fe2fffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3001020c0 0x7fe30019f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:35808/0 (socket says 172.21.15.72:35808) 2024-03-22T01:34:11.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.445+0000 7fe2fffff640 1 -- 172.21.15.72:0/1295093852 learned_addr learned my addr 172.21.15.72:0/1295093852 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:34:11.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.445+0000 7fe2ff7fe640 1 --2- 172.21.15.72:0/1295093852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe300100d60 0x7fe3001007c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:11.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.445+0000 7fe2feffd640 1 --2- 172.21.15.72:0/1295093852 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe300101720 0x7fe30019cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:11.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.445+0000 7fe2fffff640 1 -- 172.21.15.72:0/1295093852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe300100d60 msgr2=0x7fe3001007c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:11.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.445+0000 7fe2fffff640 1 --2- 172.21.15.72:0/1295093852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe300100d60 0x7fe3001007c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:11.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.445+0000 7fe2fffff640 1 -- 172.21.15.72:0/1295093852 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe300101720 msgr2=0x7fe30019cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:11.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.445+0000 7fe2fffff640 1 --2- 172.21.15.72:0/1295093852 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe300101720 0x7fe30019cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:11.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.445+0000 7fe2fffff640 1 -- 172.21.15.72:0/1295093852 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe30019fb50 con 0x7fe3001020c0 2024-03-22T01:34:11.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.446+0000 7fe2fffff640 1 --2- 172.21.15.72:0/1295093852 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3001020c0 0x7fe30019f240 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe2f800eb10 tx=0x7fe2f800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:11.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.446+0000 7fe2fcff9640 1 -- 172.21.15.72:0/1295093852 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f8004b20 con 0x7fe3001020c0 2024-03-22T01:34:11.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.446+0000 7fe305f81640 1 -- 172.21.15.72:0/1295093852 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe30019fde0 con 0x7fe3001020c0 2024-03-22T01:34:11.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.446+0000 7fe2fcff9640 1 -- 172.21.15.72:0/1295093852 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe2f8004cc0 con 0x7fe3001020c0 2024-03-22T01:34:11.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.446+0000 7fe2fcff9640 1 -- 172.21.15.72:0/1295093852 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f8010760 con 0x7fe3001020c0 2024-03-22T01:34:11.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.446+0000 7fe305f81640 1 -- 172.21.15.72:0/1295093852 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3001a02a0 con 0x7fe3001020c0 2024-03-22T01:34:11.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.447+0000 7fe305f81640 1 -- 172.21.15.72:0/1295093852 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe300107f20 con 0x7fe3001020c0 2024-03-22T01:34:11.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.448+0000 7fe2fcff9640 1 -- 172.21.15.72:0/1295093852 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7fe2f8005020 con 0x7fe3001020c0 2024-03-22T01:34:11.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.449+0000 7fe2fcff9640 1 --2- 172.21.15.72:0/1295093852 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe2e80a2f20 0x7fe2e80a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:11.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.449+0000 7fe2ff7fe640 1 --2- 172.21.15.72:0/1295093852 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe2e80a2f20 0x7fe2e80a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:11.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.449+0000 7fe2fcff9640 1 -- 172.21.15.72:0/1295093852 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fe2f80cf4a0 con 0x7fe3001020c0 2024-03-22T01:34:11.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.449+0000 7fe2ff7fe640 1 --2- 172.21.15.72:0/1295093852 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe2e80a2f20 0x7fe2e80a53e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe2f0005fd0 tx=0x7fe2f000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:11.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.453+0000 7fe2fcff9640 1 -- 172.21.15.72:0/1295093852 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe2f809c770 con 0x7fe3001020c0 2024-03-22T01:34:11.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:11 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:11.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:11 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:11.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:11 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:11.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:11 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:11.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:11 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:11.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:11 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:11.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:11 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:11.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:11 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:34:11.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:11 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:34:11.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:11 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:11.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:11 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:34:11.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:11 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/393678833' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:34:11.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.651+0000 7fe305f81640 1 -- 172.21.15.72:0/1295093852 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe30002d580 con 0x7fe3001020c0 2024-03-22T01:34:11.652 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.652+0000 7fe2fcff9640 1 -- 172.21.15.72:0/1295093852 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) ==== 74+0+18373 (secure 0 0 0) 0x7fe2f809c070 con 0x7fe3001020c0 2024-03-22T01:34:11.652 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:34:11.652 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":83,"fsid":"022a99bc-e7eb-11ee-95cd-87774f69a715","created":"2024-03-22T01:26:23.893672+0000","modified":"2024-03-22T01:33:26.552216+0000","last_up_change":"2024-03-22T01:33:25.223684+0000","last_in_change":"2024-03-22T01:33:05.010331+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T01:29:37.438503+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"16c15bb0-db92-4191-9063-38b4890e81e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6803","nonce":661906791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6805","nonce":661906791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6809","nonce":661906791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6807","nonce":661906791}]},"public_addr":"172.21.15.72:6803/661906791","cluster_addr":"172.21.15.72:6805/661906791","heartbeat_back_addr":"172.21.15.72:6809/661906791","heartbeat_front_addr":"172.21.15.72:6807/661906791","state":["exists","up"]},{"osd":1,"uuid":"981290ba-97dc-496b-aa22-d51704184b10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6801","nonce":2870590550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6803","nonce":2870590550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6807","nonce":2870590550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6805","nonce":2870590550}]},"public_addr":"172.21.15.89:6801/2870590550","cluster_addr":"172.21.15.89:6803/2870590550","heartbeat_back_addr":"172.21.15.89:6807/2870590550","heartbeat_front_addr":"172.21.15.89:6805/2870590550","state":["exists","up"]},{"osd":2,"uuid":"f07ed658-f545-46a1-8edd-8b1d5d49885c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6801","nonce":3865778118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6803","nonce":3865778118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6807","nonce":3865778118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6805","nonce":3865778118}]},"public_addr":"172.21.15.98:6801/3865778118","cluster_addr":"172.21.15.98:6803/3865778118","heartbeat_back_addr":"172.21.15.98:6807/3865778118","heartbeat_front_addr":"172.21.15.98:6805/3865778118","state":["exists","up"]},{"osd":3,"uuid":"ccc4bc40-2c48-46d2-8548-0c05b44f9f1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6811","nonce":1865951395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6813","nonce":1865951395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6817","nonce":1865951395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6815","nonce":1865951395}]},"public_addr":"172.21.15.72:6811/1865951395","cluster_addr":"172.21.15.72:6813/1865951395","heartbeat_back_addr":"172.21.15.72:6817/1865951395","heartbeat_front_addr":"172.21.15.72:6815/1865951395","state":["exists","up"]},{"osd":4,"uuid":"1812439f-810a-4ed2-9395-c00c9a085a13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6809","nonce":2683760795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6811","nonce":2683760795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6815","nonce":2683760795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6813","nonce":2683760795}]},"public_addr":"172.21.15.89:6809/2683760795","cluster_addr":"172.21.15.89:6811/2683760795","heartbeat_back_addr":"172.21.15.89:6815/2683760795","heartbeat_front_addr":"172.21.15.89:6813/2683760795","state":["exists","up"]},{"osd":5,"uuid":"e78954d8-f4e3-4d58-8536-d2d59c843386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6809","nonce":2635838572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6811","nonce":2635838572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6815","nonce":2635838572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6813","nonce":2635838572}]},"public_addr":"172.21.15.98:6809/2635838572","cluster_addr":"172.21.15.98:6811/2635838572","heartbeat_back_addr":"172.21.15.98:6815/2635838572","heartbeat_front_addr":"172.21.15.98:6813/2635838572","state":["exists","up"]},{"osd":6,"uuid":"dc6c8be2-b64c-4959-8015-7b0ea6d25599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6819","nonce":3158982661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6821","nonce":3158982661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6825","nonce":3158982661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6823","nonce":3158982661}]},"public_addr":"172.21.15.72:6819/3158982661","cluster_addr":"172.21.15.72:6821/3158982661","heartbeat_back_addr":"172.21.15.72:6825/3158982661","heartbeat_front_addr":"172.21.15.72:6823/3158982661","state":["exists","up"]},{"osd":7,"uuid":"7a73469f-cf3a-4306-ae93-ea46e21ebbbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6817","nonce":3075913234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6819","nonce":3075913234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6823","nonce":3075913234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6821","nonce":3075913234}]},"public_addr":"172.21.15.89:6817/3075913234","cluster_addr":"172.21.15.89:6819/3075913234","heartbeat_back_addr":"172.21.15.89:6823/3075913234","heartbeat_front_addr":"172.21.15.89:6821/3075913234","state":["exists","up"]},{"osd":8,"uuid":"c4b13bee-ea4d-4344-95fb-2622ecbb8196","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6817","nonce":1813255577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6819","nonce":1813255577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6823","nonce":1813255577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6821","nonce":1813255577}]},"public_addr":"172.21.15.98:6817/1813255577","cluster_addr":"172.21.15.98:6819/1813255577","heartbeat_back_addr":"172.21.15.98:6823/1813255577","heartbeat_front_addr":"172.21.15.98:6821/1813255577","state":["exists","up"]},{"osd":9,"uuid":"eefb0a44-4a36-4e55-9d04-0501bf718db6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6827","nonce":3377749206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6829","nonce":3377749206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6832","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6833","nonce":3377749206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6831","nonce":3377749206}]},"public_addr":"172.21.15.72:6827/3377749206","cluster_addr":"172.21.15.72:6829/3377749206","heartbeat_back_addr":"172.21.15.72:6833/3377749206","heartbeat_front_addr":"172.21.15.72:6831/3377749206","state":["exists","up"]},{"osd":10,"uuid":"8741d25f-40dd-40d7-b040-5cc5e411eb98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6825","nonce":885058786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6827","nonce":885058786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6831","nonce":885058786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6829","nonce":885058786}]},"public_addr":"172.21.15.89:6825/885058786","cluster_addr":"172.21.15.89:6827/885058786","heartbeat_back_addr":"172.21.15.89:6831/885058786","heartbeat_front_addr":"172.21.15.89:6829/885058786","state":["exists","up"]},{"osd":11,"uuid":"8d02e957-eda4-4184-b5c6-5eb3b8b38060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6825","nonce":1517095838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6827","nonce":1517095838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6831","nonce":1517095838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6829","nonce":1517095838}]},"public_addr":"172.21.15.98:6825/1517095838","cluster_addr":"172.21.15.98:6827/1517095838","heartbeat_back_addr":"172.21.15.98:6831/1517095838","heartbeat_front_addr":"172.21.15.98:6829/1517095838","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:28:47.307652+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:11.037135+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:34.619188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:59.350738+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:30:23.787337+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:30:48.994154+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:31:13.690350+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:31:39.628116+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:04.723912+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:30.689741+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:57.229508+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:33:23.154537+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:0/121087196":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/2451857761":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/1228477995":"2024-03-23T01:27:15.339809+0000","172.21.15.72:6801/1513232913":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/3558958264":"2024-03-23T01:26:50.074871+0000","172.21.15.72:0/1510187198":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6800/3518012916":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6801/3518012916":"2024-03-23T01:26:50.074871+0000","172.21.15.72:0/1372801041":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6800/1513232913":"2024-03-23T01:27:15.339809+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T01:34:11.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.655+0000 7fe305f81640 1 -- 172.21.15.72:0/1295093852 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe2e80a2f20 msgr2=0x7fe2e80a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:11.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.655+0000 7fe305f81640 1 --2- 172.21.15.72:0/1295093852 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe2e80a2f20 0x7fe2e80a53e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe2f0005fd0 tx=0x7fe2f000b410 comp rx=0 tx=0).stop 2024-03-22T01:34:11.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.655+0000 7fe305f81640 1 -- 172.21.15.72:0/1295093852 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3001020c0 msgr2=0x7fe30019f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:11.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.655+0000 7fe305f81640 1 --2- 172.21.15.72:0/1295093852 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3001020c0 0x7fe30019f240 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe2f800eb10 tx=0x7fe2f800c2d0 comp rx=0 tx=0).stop 2024-03-22T01:34:11.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.655+0000 7fe305f81640 1 -- 172.21.15.72:0/1295093852 shutdown_connections 2024-03-22T01:34:11.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.655+0000 7fe305f81640 1 --2- 172.21.15.72:0/1295093852 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe2e80a2f20 0x7fe2e80a53e0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:11.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.655+0000 7fe305f81640 1 --2- 172.21.15.72:0/1295093852 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe300101720 0x7fe30019cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:11.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.655+0000 7fe305f81640 1 --2- 172.21.15.72:0/1295093852 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3001020c0 0x7fe30019f240 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:11.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.655+0000 7fe305f81640 1 --2- 172.21.15.72:0/1295093852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe300100d60 0x7fe3001007c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:11.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.655+0000 7fe305f81640 1 -- 172.21.15.72:0/1295093852 >> 172.21.15.72:0/1295093852 conn(0x7fe3000fb7b0 msgr2=0x7fe3001050f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:11.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.655+0000 7fe305f81640 1 -- 172.21.15.72:0/1295093852 shutdown_connections 2024-03-22T01:34:11.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.655+0000 7fe305f81640 1 -- 172.21.15.72:0/1295093852 wait complete. 2024-03-22T01:34:11.666 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-22T01:29:37.438503+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-22T01:34:11.666 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-22T01:34:11.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.782+0000 7f57a8d9a640 1 -- 172.21.15.72:0/3055592001 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f57a4100ab0 msgr2=0x7f57a4100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:11.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.782+0000 7f57a8d9a640 1 --2- 172.21.15.72:0/3055592001 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f57a4100ab0 0x7f57a4100ed0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5794009900 tx=0x7f579402bc30 comp rx=0 tx=0).stop 2024-03-22T01:34:11.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.782+0000 7f57a8d9a640 1 -- 172.21.15.72:0/3055592001 shutdown_connections 2024-03-22T01:34:11.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.782+0000 7f57a8d9a640 1 --2- 172.21.15.72:0/3055592001 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57a4102660 0x7f57a4108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:11.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.782+0000 7f57a8d9a640 1 --2- 172.21.15.72:0/3055592001 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57a4101ca0 0x7f57a4102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:11.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.782+0000 7f57a8d9a640 1 --2- 172.21.15.72:0/3055592001 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f57a4100ab0 0x7f57a4100ed0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:11.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.782+0000 7f57a8d9a640 1 -- 172.21.15.72:0/3055592001 >> 172.21.15.72:0/3055592001 conn(0x7f57a40fc020 msgr2=0x7f57a40fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:11.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.782+0000 7f57a8d9a640 1 -- 172.21.15.72:0/3055592001 shutdown_connections 2024-03-22T01:34:11.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.782+0000 7f57a8d9a640 1 -- 172.21.15.72:0/3055592001 wait complete. 2024-03-22T01:34:11.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.783+0000 7f57a8d9a640 1 Processor -- start 2024-03-22T01:34:11.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.783+0000 7f57a8d9a640 1 -- start start 2024-03-22T01:34:11.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.783+0000 7f57a8d9a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57a4100ab0 0x7f57a40699a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:11.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.783+0000 7f57a8d9a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57a4101ca0 0x7f57a419cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:11.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.783+0000 7f57a8d9a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f57a4102660 0x7f57a419f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:11.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.783+0000 7f57a8d9a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f57a410c720 con 0x7f57a4102660 2024-03-22T01:34:11.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.783+0000 7f57a8d9a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f57a410c5a0 con 0x7f57a4101ca0 2024-03-22T01:34:11.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.783+0000 7f57a8d9a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f57a410c8a0 con 0x7f57a4100ab0 2024-03-22T01:34:11.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.783+0000 7f57a2d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f57a4102660 0x7f57a419f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:11.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.783+0000 7f57a259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57a4100ab0 0x7f57a40699a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:11.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.783+0000 7f57a2d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f57a4102660 0x7f57a419f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:35812/0 (socket says 172.21.15.72:35812) 2024-03-22T01:34:11.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.783+0000 7f57a2d9d640 1 -- 172.21.15.72:0/4034132554 learned_addr learned my addr 172.21.15.72:0/4034132554 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:34:11.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.783+0000 7f57a259c640 1 --2- 172.21.15.72:0/4034132554 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57a4100ab0 0x7f57a40699a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.72:56300/0 (socket says 172.21.15.72:56300) 2024-03-22T01:34:11.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.783+0000 7f579bfff640 1 --2- 172.21.15.72:0/4034132554 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57a4101ca0 0x7f57a419cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:11.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.783+0000 7f57a2d9d640 1 -- 172.21.15.72:0/4034132554 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57a4100ab0 msgr2=0x7f57a40699a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:11.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.783+0000 7f57a2d9d640 1 --2- 172.21.15.72:0/4034132554 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57a4100ab0 0x7f57a40699a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:11.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.783+0000 7f57a2d9d640 1 -- 172.21.15.72:0/4034132554 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57a4101ca0 msgr2=0x7f57a419cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:11.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.783+0000 7f57a2d9d640 1 --2- 172.21.15.72:0/4034132554 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57a4101ca0 0x7f57a419cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:11.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.783+0000 7f57a2d9d640 1 -- 172.21.15.72:0/4034132554 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57a419fb40 con 0x7f57a4102660 2024-03-22T01:34:11.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.783+0000 7f57a2d9d640 1 --2- 172.21.15.72:0/4034132554 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f57a4102660 0x7f57a419f280 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f579c009fd0 tx=0x7f579c00b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:11.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.783+0000 7f579b7fe640 1 -- 172.21.15.72:0/4034132554 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f579c017070 con 0x7f57a4102660 2024-03-22T01:34:11.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.783+0000 7f57a8d9a640 1 -- 172.21.15.72:0/4034132554 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57a419fe30 con 0x7f57a4102660 2024-03-22T01:34:11.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.783+0000 7f579b7fe640 1 -- 172.21.15.72:0/4034132554 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f579c003e50 con 0x7f57a4102660 2024-03-22T01:34:11.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.783+0000 7f579b7fe640 1 -- 172.21.15.72:0/4034132554 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f579c0106c0 con 0x7f57a4102660 2024-03-22T01:34:11.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.784+0000 7f57a8d9a640 1 -- 172.21.15.72:0/4034132554 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57a41a02f0 con 0x7f57a4102660 2024-03-22T01:34:11.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.784+0000 7f57a8d9a640 1 -- 172.21.15.72:0/4034132554 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57a4107ef0 con 0x7f57a4102660 2024-03-22T01:34:11.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.785+0000 7f579b7fe640 1 -- 172.21.15.72:0/4034132554 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f579c003970 con 0x7f57a4102660 2024-03-22T01:34:11.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.786+0000 7f579b7fe640 1 --2- 172.21.15.72:0/4034132554 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f57900a2f20 0x7f57900a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:11.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.786+0000 7f57a259c640 1 --2- 172.21.15.72:0/4034132554 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f57900a2f20 0x7f57900a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:11.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.786+0000 7f579b7fe640 1 -- 172.21.15.72:0/4034132554 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f579c0ced20 con 0x7f57a4102660 2024-03-22T01:34:11.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.786+0000 7f57a259c640 1 --2- 172.21.15.72:0/4034132554 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f57900a2f20 0x7f57900a53e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5794000c00 tx=0x7f5794005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:11.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.790+0000 7f579b7fe640 1 -- 172.21.15.72:0/4034132554 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f579c0d3070 con 0x7f57a4102660 2024-03-22T01:34:11.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:11 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:11.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:11 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:11.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:11 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:11.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:11 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:11.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:11 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:11.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:11 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:11.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:11 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:11.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:11 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:34:11.859 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:11 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:34:11.859 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:11 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:11.859 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:11 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:34:11.859 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:11 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/393678833' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:34:11.905 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:11 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:11.905 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:11 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:11.905 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:11 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:11.905 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:11 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:11.905 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:11 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:11.905 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:11 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:11.905 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:11 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:11.906 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:11 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:34:11.906 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:11 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:34:11.906 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:11 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:11.906 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:11 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:34:11.906 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:11 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/393678833' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:34:11.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.971+0000 7f57a8d9a640 1 -- 172.21.15.72:0/4034132554 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f57a402d5b0 con 0x7f57a4102660 2024-03-22T01:34:11.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.972+0000 7f579b7fe640 1 -- 172.21.15.72:0/4034132554 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) ==== 93+0+10 (secure 0 0 0) 0x7f579c09bfd0 con 0x7f57a4102660 2024-03-22T01:34:11.971 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-22T01:34:11.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.974+0000 7f57a8d9a640 1 -- 172.21.15.72:0/4034132554 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f57900a2f20 msgr2=0x7f57900a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:11.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.974+0000 7f57a8d9a640 1 --2- 172.21.15.72:0/4034132554 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f57900a2f20 0x7f57900a53e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5794000c00 tx=0x7f5794005fd0 comp rx=0 tx=0).stop 2024-03-22T01:34:11.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.974+0000 7f57a8d9a640 1 -- 172.21.15.72:0/4034132554 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f57a4102660 msgr2=0x7f57a419f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:11.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.974+0000 7f57a8d9a640 1 --2- 172.21.15.72:0/4034132554 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f57a4102660 0x7f57a419f280 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f579c009fd0 tx=0x7f579c00b260 comp rx=0 tx=0).stop 2024-03-22T01:34:11.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.974+0000 7f57a8d9a640 1 -- 172.21.15.72:0/4034132554 shutdown_connections 2024-03-22T01:34:11.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.974+0000 7f57a8d9a640 1 --2- 172.21.15.72:0/4034132554 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f57900a2f20 0x7f57900a53e0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:11.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.974+0000 7f57a8d9a640 1 --2- 172.21.15.72:0/4034132554 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57a4101ca0 0x7f57a419cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:11.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.974+0000 7f57a8d9a640 1 --2- 172.21.15.72:0/4034132554 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f57a4102660 0x7f57a419f280 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:11.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.974+0000 7f57a8d9a640 1 --2- 172.21.15.72:0/4034132554 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57a4100ab0 0x7f57a40699a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:11.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.974+0000 7f57a8d9a640 1 -- 172.21.15.72:0/4034132554 >> 172.21.15.72:0/4034132554 conn(0x7f57a40fc020 msgr2=0x7f57a41050c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:11.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.974+0000 7f57a8d9a640 1 -- 172.21.15.72:0/4034132554 shutdown_connections 2024-03-22T01:34:11.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:11.974+0000 7f57a8d9a640 1 -- 172.21.15.72:0/4034132554 wait complete. 2024-03-22T01:34:11.985 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-22T01:34:12.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.090+0000 7f1526a88640 1 -- 172.21.15.72:0/1618094680 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f152010e1f0 msgr2=0x7f15201105e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:12.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.090+0000 7f1526a88640 1 --2- 172.21.15.72:0/1618094680 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f152010e1f0 0x7f15201105e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1518009cd0 tx=0x7f151802c0f0 comp rx=0 tx=0).stop 2024-03-22T01:34:12.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.090+0000 7f1526a88640 1 -- 172.21.15.72:0/1618094680 shutdown_connections 2024-03-22T01:34:12.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.090+0000 7f1526a88640 1 --2- 172.21.15.72:0/1618094680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15200745f0 0x7f1520074a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:12.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.090+0000 7f1526a88640 1 --2- 172.21.15.72:0/1618094680 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f152010e1f0 0x7f15201105e0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:12.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.090+0000 7f1526a88640 1 --2- 172.21.15.72:0/1618094680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1520075bd0 0x7f1520074020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:12.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.090+0000 7f1526a88640 1 -- 172.21.15.72:0/1618094680 >> 172.21.15.72:0/1618094680 conn(0x7f15200fd1b0 msgr2=0x7f15200ff5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:12.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.090+0000 7f1526a88640 1 -- 172.21.15.72:0/1618094680 shutdown_connections 2024-03-22T01:34:12.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.090+0000 7f1526a88640 1 -- 172.21.15.72:0/1618094680 wait complete. 2024-03-22T01:34:12.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.091+0000 7f1526a88640 1 Processor -- start 2024-03-22T01:34:12.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.091+0000 7f1526a88640 1 -- start start 2024-03-22T01:34:12.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.091+0000 7f1526a88640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f15200745f0 0x7f152019e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:12.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.091+0000 7f1526a88640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1520075bd0 0x7f152019ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:12.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.091+0000 7f1526a88640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f152010e1f0 0x7f15201a5b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:12.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.091+0000 7f1526a88640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f1520113f00 con 0x7f15200745f0 2024-03-22T01:34:12.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.091+0000 7f1526a88640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1520113d80 con 0x7f152010e1f0 2024-03-22T01:34:12.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.091+0000 7f1526a88640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1520114080 con 0x7f1520075bd0 2024-03-22T01:34:12.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.091+0000 7f1524824640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f15200745f0 0x7f152019e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:12.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.091+0000 7f1524824640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f15200745f0 0x7f152019e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:35828/0 (socket says 172.21.15.72:35828) 2024-03-22T01:34:12.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.091+0000 7f1524824640 1 -- 172.21.15.72:0/478934582 learned_addr learned my addr 172.21.15.72:0/478934582 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:34:12.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.091+0000 7f151ffff640 1 --2- 172.21.15.72:0/478934582 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1520075bd0 0x7f152019ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:12.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.091+0000 7f1524824640 1 -- 172.21.15.72:0/478934582 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1520075bd0 msgr2=0x7f152019ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:12.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.091+0000 7f1524824640 1 --2- 172.21.15.72:0/478934582 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1520075bd0 0x7f152019ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:12.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.091+0000 7f1524824640 1 -- 172.21.15.72:0/478934582 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f152010e1f0 msgr2=0x7f15201a5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:12.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.091+0000 7f1524824640 1 --2- 172.21.15.72:0/478934582 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f152010e1f0 0x7f15201a5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:12.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.091+0000 7f1524824640 1 -- 172.21.15.72:0/478934582 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15201a61b0 con 0x7f15200745f0 2024-03-22T01:34:12.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.092+0000 7f1524824640 1 --2- 172.21.15.72:0/478934582 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f15200745f0 0x7f152019e590 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f151400b4a0 tx=0x7f151400b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:12.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.092+0000 7f151dffb640 1 -- 172.21.15.72:0/478934582 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1514015020 con 0x7f15200745f0 2024-03-22T01:34:12.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.092+0000 7f151dffb640 1 -- 172.21.15.72:0/478934582 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1514011d10 con 0x7f15200745f0 2024-03-22T01:34:12.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.092+0000 7f1526a88640 1 -- 172.21.15.72:0/478934582 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15201a64a0 con 0x7f15200745f0 2024-03-22T01:34:12.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.092+0000 7f151dffb640 1 -- 172.21.15.72:0/478934582 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f151400f2e0 con 0x7f15200745f0 2024-03-22T01:34:12.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.092+0000 7f1526a88640 1 -- 172.21.15.72:0/478934582 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15201a6a70 con 0x7f15200745f0 2024-03-22T01:34:12.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.093+0000 7f1526a88640 1 -- 172.21.15.72:0/478934582 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15201043c0 con 0x7f15200745f0 2024-03-22T01:34:12.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.094+0000 7f151dffb640 1 -- 172.21.15.72:0/478934582 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f1514018020 con 0x7f15200745f0 2024-03-22T01:34:12.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.094+0000 7f151dffb640 1 --2- 172.21.15.72:0/478934582 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f15100a2fa0 0x7f15100a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:12.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.094+0000 7f151ffff640 1 --2- 172.21.15.72:0/478934582 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f15100a2fa0 0x7f15100a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:12.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.094+0000 7f151dffb640 1 -- 172.21.15.72:0/478934582 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f15140ce7d0 con 0x7f15200745f0 2024-03-22T01:34:12.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.095+0000 7f151ffff640 1 --2- 172.21.15.72:0/478934582 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f15100a2fa0 0x7f15100a5460 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f152019faa0 tx=0x7f150c009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:12.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.099+0000 7f151dffb640 1 -- 172.21.15.72:0/478934582 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f151401f020 con 0x7f15200745f0 2024-03-22T01:34:12.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.277+0000 7f1526a88640 1 -- 172.21.15.72:0/478934582 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f1520062790 con 0x7f15200745f0 2024-03-22T01:34:12.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.277+0000 7f151dffb640 1 -- 172.21.15.72:0/478934582 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v83) ==== 84+0+65 (secure 0 0 0) 0x7f151409baa0 con 0x7f15200745f0 2024-03-22T01:34:12.277 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:34:12.277 INFO:teuthology.orchestra.run.smithi072.stdout:[ 2024-03-22T01:34:12.277 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:34:12.278 INFO:teuthology.orchestra.run.smithi072.stdout: "poolnum": 1, 2024-03-22T01:34:12.278 INFO:teuthology.orchestra.run.smithi072.stdout: "poolname": ".mgr" 2024-03-22T01:34:12.278 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:34:12.278 INFO:teuthology.orchestra.run.smithi072.stdout:] 2024-03-22T01:34:12.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.279+0000 7f1526a88640 1 -- 172.21.15.72:0/478934582 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f15100a2fa0 msgr2=0x7f15100a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:12.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.279+0000 7f1526a88640 1 --2- 172.21.15.72:0/478934582 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f15100a2fa0 0x7f15100a5460 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f152019faa0 tx=0x7f150c009110 comp rx=0 tx=0).stop 2024-03-22T01:34:12.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.279+0000 7f1526a88640 1 -- 172.21.15.72:0/478934582 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f15200745f0 msgr2=0x7f152019e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:12.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.279+0000 7f1526a88640 1 --2- 172.21.15.72:0/478934582 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f15200745f0 0x7f152019e590 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f151400b4a0 tx=0x7f151400b8a0 comp rx=0 tx=0).stop 2024-03-22T01:34:12.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.279+0000 7f1526a88640 1 -- 172.21.15.72:0/478934582 shutdown_connections 2024-03-22T01:34:12.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.279+0000 7f1526a88640 1 --2- 172.21.15.72:0/478934582 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f15100a2fa0 0x7f15100a5460 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:12.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.279+0000 7f1526a88640 1 --2- 172.21.15.72:0/478934582 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f152010e1f0 0x7f15201a5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:12.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.279+0000 7f1526a88640 1 --2- 172.21.15.72:0/478934582 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1520075bd0 0x7f152019ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:12.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.279+0000 7f1526a88640 1 --2- 172.21.15.72:0/478934582 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f15200745f0 0x7f152019e590 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:12.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.279+0000 7f1526a88640 1 -- 172.21.15.72:0/478934582 >> 172.21.15.72:0/478934582 conn(0x7f15200fd1b0 msgr2=0x7f1520101590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:12.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.279+0000 7f1526a88640 1 -- 172.21.15.72:0/478934582 shutdown_connections 2024-03-22T01:34:12.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.279+0000 7f1526a88640 1 -- 172.21.15.72:0/478934582 wait complete. 2024-03-22T01:34:12.289 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-22T01:34:12.290 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-22T01:34:12.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.403+0000 7f48060c9640 1 -- 172.21.15.72:0/761163688 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4800101ca0 msgr2=0x7f4800102120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:12.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.403+0000 7f48060c9640 1 --2- 172.21.15.72:0/761163688 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4800101ca0 0x7f4800102120 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f47f4009900 tx=0x7f47f402bc30 comp rx=0 tx=0).stop 2024-03-22T01:34:12.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.403+0000 7f48060c9640 1 -- 172.21.15.72:0/761163688 shutdown_connections 2024-03-22T01:34:12.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.403+0000 7f48060c9640 1 --2- 172.21.15.72:0/761163688 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4800102660 0x7f4800108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:12.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.403+0000 7f48060c9640 1 --2- 172.21.15.72:0/761163688 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4800101ca0 0x7f4800102120 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:12.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.403+0000 7f48060c9640 1 --2- 172.21.15.72:0/761163688 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4800100ab0 0x7f4800100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:12.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.403+0000 7f48060c9640 1 -- 172.21.15.72:0/761163688 >> 172.21.15.72:0/761163688 conn(0x7f48000fc020 msgr2=0x7f48000fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:12.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.404+0000 7f48060c9640 1 -- 172.21.15.72:0/761163688 shutdown_connections 2024-03-22T01:34:12.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.404+0000 7f48060c9640 1 -- 172.21.15.72:0/761163688 wait complete. 2024-03-22T01:34:12.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.404+0000 7f48060c9640 1 Processor -- start 2024-03-22T01:34:12.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.404+0000 7f48060c9640 1 -- start start 2024-03-22T01:34:12.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.404+0000 7f48060c9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4800100ab0 0x7f4800197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:12.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.404+0000 7f48060c9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4800101ca0 0x7f4800198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:12.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.404+0000 7f48060c9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4800102660 0x7f480019f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:12.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.404+0000 7f48060c9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f480010c710 con 0x7f4800101ca0 2024-03-22T01:34:12.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.404+0000 7f48060c9640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f480010c590 con 0x7f4800100ab0 2024-03-22T01:34:12.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.404+0000 7f48060c9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f480010c890 con 0x7f4800102660 2024-03-22T01:34:12.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.405+0000 7f47feffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4800101ca0 0x7f4800198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:12.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.405+0000 7f47ff7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4800100ab0 0x7f4800197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:12.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.405+0000 7f47feffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4800101ca0 0x7f4800198410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:35854/0 (socket says 172.21.15.72:35854) 2024-03-22T01:34:12.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.405+0000 7f47feffd640 1 -- 172.21.15.72:0/388195924 learned_addr learned my addr 172.21.15.72:0/388195924 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:34:12.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.405+0000 7f47fffff640 1 --2- 172.21.15.72:0/388195924 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4800102660 0x7f480019f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:12.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.405+0000 7f47feffd640 1 -- 172.21.15.72:0/388195924 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4800102660 msgr2=0x7f480019f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:12.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.405+0000 7f47feffd640 1 --2- 172.21.15.72:0/388195924 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4800102660 0x7f480019f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:12.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.405+0000 7f47feffd640 1 -- 172.21.15.72:0/388195924 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4800100ab0 msgr2=0x7f4800197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:12.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.405+0000 7f47feffd640 1 --2- 172.21.15.72:0/388195924 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4800100ab0 0x7f4800197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:12.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.405+0000 7f47feffd640 1 -- 172.21.15.72:0/388195924 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f480019faf0 con 0x7f4800101ca0 2024-03-22T01:34:12.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.405+0000 7f47ff7fe640 1 --2- 172.21.15.72:0/388195924 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4800100ab0 0x7f4800197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:34:12.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.405+0000 7f47feffd640 1 --2- 172.21.15.72:0/388195924 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4800101ca0 0x7f4800198410 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f47f4009230 tx=0x7f47f4005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:12.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.405+0000 7f47fcff9640 1 -- 172.21.15.72:0/388195924 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f4041070 con 0x7f4800101ca0 2024-03-22T01:34:12.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.405+0000 7f47fcff9640 1 -- 172.21.15.72:0/388195924 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47f40044d0 con 0x7f4800101ca0 2024-03-22T01:34:12.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.405+0000 7f48060c9640 1 -- 172.21.15.72:0/388195924 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f480019fd80 con 0x7f4800101ca0 2024-03-22T01:34:12.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.405+0000 7f47fcff9640 1 -- 172.21.15.72:0/388195924 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f4034440 con 0x7f4800101ca0 2024-03-22T01:34:12.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.405+0000 7f48060c9640 1 -- 172.21.15.72:0/388195924 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48001a0210 con 0x7f4800101ca0 2024-03-22T01:34:12.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.406+0000 7f48060c9640 1 -- 172.21.15.72:0/388195924 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48001a0440 con 0x7f4800101ca0 2024-03-22T01:34:12.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.408+0000 7f47fcff9640 1 -- 172.21.15.72:0/388195924 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f47f4044070 con 0x7f4800101ca0 2024-03-22T01:34:12.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.408+0000 7f47fcff9640 1 --2- 172.21.15.72:0/388195924 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f47e80a2f20 0x7f47e80a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:12.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.408+0000 7f47fcff9640 1 -- 172.21.15.72:0/388195924 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f47f40f0b20 con 0x7f4800101ca0 2024-03-22T01:34:12.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.408+0000 7f47ff7fe640 1 --2- 172.21.15.72:0/388195924 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f47e80a2f20 0x7f47e80a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:12.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.409+0000 7f47ff7fe640 1 --2- 172.21.15.72:0/388195924 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f47e80a2f20 0x7f47e80a53e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4800101b00 tx=0x7f47f0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:12.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.414+0000 7f47fcff9640 1 -- 172.21.15.72:0/388195924 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f47f40bd7a0 con 0x7f4800101ca0 2024-03-22T01:34:12.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:12.605+0000 7f48060c9640 1 -- 172.21.15.72:0/388195924 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f48001a06b0 con 0x7f4800101ca0 2024-03-22T01:34:12.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:12 smithi072 ceph-mon[32414]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:34:12.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:12 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/1295093852' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:34:12.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:12 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/4034132554' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T01:34:12.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:12 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/478934582' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-22T01:34:12.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:12 smithi089 ceph-mon[35770]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:34:12.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:12 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/1295093852' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:34:12.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:12 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/4034132554' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T01:34:12.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:12 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/478934582' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-22T01:34:12.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:12 smithi098 ceph-mon[36468]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:34:12.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:12 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/1295093852' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:34:12.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:12 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/4034132554' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T01:34:12.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:12 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/478934582' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-22T01:34:13.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:13.444+0000 7f47fcff9640 1 -- 172.21.15.72:0/388195924 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v84) ==== 138+0+0 (secure 0 0 0) 0x7f47f40c1840 con 0x7f4800101ca0 2024-03-22T01:34:13.443 INFO:teuthology.orchestra.run.smithi072.stderr:pool 'cephfs_metadata' created 2024-03-22T01:34:13.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:13.446+0000 7f48060c9640 1 -- 172.21.15.72:0/388195924 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f47e80a2f20 msgr2=0x7f47e80a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:13.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:13.446+0000 7f48060c9640 1 --2- 172.21.15.72:0/388195924 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f47e80a2f20 0x7f47e80a53e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4800101b00 tx=0x7f47f0005f90 comp rx=0 tx=0).stop 2024-03-22T01:34:13.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:13.446+0000 7f48060c9640 1 -- 172.21.15.72:0/388195924 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4800101ca0 msgr2=0x7f4800198410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:13.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:13.446+0000 7f48060c9640 1 --2- 172.21.15.72:0/388195924 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4800101ca0 0x7f4800198410 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f47f4009230 tx=0x7f47f4005d50 comp rx=0 tx=0).stop 2024-03-22T01:34:13.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:13.446+0000 7f48060c9640 1 -- 172.21.15.72:0/388195924 shutdown_connections 2024-03-22T01:34:13.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:13.446+0000 7f48060c9640 1 --2- 172.21.15.72:0/388195924 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f47e80a2f20 0x7f47e80a53e0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:13.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:13.446+0000 7f48060c9640 1 --2- 172.21.15.72:0/388195924 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4800102660 0x7f480019f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:13.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:13.446+0000 7f48060c9640 1 --2- 172.21.15.72:0/388195924 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4800101ca0 0x7f4800198410 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:13.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:13.446+0000 7f48060c9640 1 --2- 172.21.15.72:0/388195924 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4800100ab0 0x7f4800197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:13.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:13.446+0000 7f48060c9640 1 -- 172.21.15.72:0/388195924 >> 172.21.15.72:0/388195924 conn(0x7f48000fc020 msgr2=0x7f4800069cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:13.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:13.447+0000 7f48060c9640 1 -- 172.21.15.72:0/388195924 shutdown_connections 2024-03-22T01:34:13.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:13.447+0000 7f48060c9640 1 -- 172.21.15.72:0/388195924 wait complete. 2024-03-22T01:34:13.459 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-03-22T01:34:13.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:13.570+0000 7f62c1d7a640 1 -- 172.21.15.72:0/3895779256 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f62bc101c90 msgr2=0x7f62bc102110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:13.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:13.570+0000 7f62c1d7a640 1 --2- 172.21.15.72:0/3895779256 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f62bc101c90 0x7f62bc102110 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f62b0009900 tx=0x7f62b002bc30 comp rx=0 tx=0).stop 2024-03-22T01:34:13.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:13.570+0000 7f62c1d7a640 1 -- 172.21.15.72:0/3895779256 shutdown_connections 2024-03-22T01:34:13.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:13.570+0000 7f62c1d7a640 1 --2- 172.21.15.72:0/3895779256 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62bc102650 0x7f62bc108ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:13.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:13.570+0000 7f62c1d7a640 1 --2- 172.21.15.72:0/3895779256 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f62bc101c90 0x7f62bc102110 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:13.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:13.570+0000 7f62c1d7a640 1 --2- 172.21.15.72:0/3895779256 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62bc100aa0 0x7f62bc100ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:13.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:13.570+0000 7f62c1d7a640 1 -- 172.21.15.72:0/3895779256 >> 172.21.15.72:0/3895779256 conn(0x7f62bc0fbff0 msgr2=0x7f62bc0fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:13.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:13.570+0000 7f62c1d7a640 1 -- 172.21.15.72:0/3895779256 shutdown_connections 2024-03-22T01:34:13.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:13.570+0000 7f62c1d7a640 1 -- 172.21.15.72:0/3895779256 wait complete. 2024-03-22T01:34:13.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:13.570+0000 7f62c1d7a640 1 Processor -- start 2024-03-22T01:34:13.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:13.570+0000 7f62c1d7a640 1 -- start start 2024-03-22T01:34:13.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:13.571+0000 7f62c1d7a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62bc100aa0 0x7f62bc197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:13.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:13.571+0000 7f62c1d7a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f62bc101c90 0x7f62bc198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:13.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:13.571+0000 7f62c1d7a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62bc102650 0x7f62bc19f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:13.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:13.571+0000 7f62c1d7a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f62bc10d880 con 0x7f62bc101c90 2024-03-22T01:34:13.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:13.571+0000 7f62c1d7a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f62bc10d700 con 0x7f62bc100aa0 2024-03-22T01:34:13.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:13.571+0000 7f62c1d7a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f62bc10da00 con 0x7f62bc102650 2024-03-22T01:34:13.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:13.571+0000 7f62baffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f62bc101c90 0x7f62bc198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:13.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:13.571+0000 7f62baffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f62bc101c90 0x7f62bc198480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:35880/0 (socket says 172.21.15.72:35880) 2024-03-22T01:34:13.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:13.571+0000 7f62baffd640 1 -- 172.21.15.72:0/3953349206 learned_addr learned my addr 172.21.15.72:0/3953349206 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:34:13.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:13.571+0000 7f62bbfff640 1 --2- 172.21.15.72:0/3953349206 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62bc102650 0x7f62bc19f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:13.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:13.571+0000 7f62bb7fe640 1 --2- 172.21.15.72:0/3953349206 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62bc100aa0 0x7f62bc197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:13.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:13.571+0000 7f62baffd640 1 -- 172.21.15.72:0/3953349206 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62bc102650 msgr2=0x7f62bc19f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:13.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:13.571+0000 7f62baffd640 1 --2- 172.21.15.72:0/3953349206 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62bc102650 0x7f62bc19f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:13.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:13.571+0000 7f62baffd640 1 -- 172.21.15.72:0/3953349206 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62bc100aa0 msgr2=0x7f62bc197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:13.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:13.571+0000 7f62baffd640 1 --2- 172.21.15.72:0/3953349206 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62bc100aa0 0x7f62bc197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:13.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:13.571+0000 7f62baffd640 1 -- 172.21.15.72:0/3953349206 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62bc19fb60 con 0x7f62bc101c90 2024-03-22T01:34:13.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:13.571+0000 7f62bb7fe640 1 --2- 172.21.15.72:0/3953349206 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62bc100aa0 0x7f62bc197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T01:34:13.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:13.571+0000 7f62baffd640 1 --2- 172.21.15.72:0/3953349206 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f62bc101c90 0x7f62bc198480 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f62b0009230 tx=0x7f62b0005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:13.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:13.571+0000 7f62b8ff9640 1 -- 172.21.15.72:0/3953349206 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62b0041070 con 0x7f62bc101c90 2024-03-22T01:34:13.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:13.571+0000 7f62c1d7a640 1 -- 172.21.15.72:0/3953349206 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62bc19fdf0 con 0x7f62bc101c90 2024-03-22T01:34:13.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:13.571+0000 7f62b8ff9640 1 -- 172.21.15.72:0/3953349206 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62b00054e0 con 0x7f62bc101c90 2024-03-22T01:34:13.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:13.571+0000 7f62c1d7a640 1 -- 172.21.15.72:0/3953349206 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62bc1a0260 con 0x7f62bc101c90 2024-03-22T01:34:13.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:13.571+0000 7f62b8ff9640 1 -- 172.21.15.72:0/3953349206 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62b0034440 con 0x7f62bc101c90 2024-03-22T01:34:13.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:13.573+0000 7f62c1d7a640 1 -- 172.21.15.72:0/3953349206 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62bc107dc0 con 0x7f62bc101c90 2024-03-22T01:34:13.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:13.573+0000 7f62b8ff9640 1 -- 172.21.15.72:0/3953349206 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f62b0044070 con 0x7f62bc101c90 2024-03-22T01:34:13.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:13.574+0000 7f62b8ff9640 1 --2- 172.21.15.72:0/3953349206 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f62a40a2f20 0x7f62a40a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:13.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:13.574+0000 7f62bb7fe640 1 --2- 172.21.15.72:0/3953349206 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f62a40a2f20 0x7f62a40a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:13.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:13.574+0000 7f62b8ff9640 1 -- 172.21.15.72:0/3953349206 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7497+0+0 (secure 0 0 0) 0x7f62b00f0e10 con 0x7f62bc101c90 2024-03-22T01:34:13.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:13.574+0000 7f62bb7fe640 1 --2- 172.21.15.72:0/3953349206 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f62a40a2f20 0x7f62a40a53e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f62bc101af0 tx=0x7f62ac005e70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:13.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:13.579+0000 7f62b8ff9640 1 -- 172.21.15.72:0/3953349206 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f62b00bdf70 con 0x7f62bc101c90 2024-03-22T01:34:13.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:13.757+0000 7f62c1d7a640 1 -- 172.21.15.72:0/3953349206 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7f62bc062790 con 0x7f62bc101c90 2024-03-22T01:34:13.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:13 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/388195924' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-22T01:34:13.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:13 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:13.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:13 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:13.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:13 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:13.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:13 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:13.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:13 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:13.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:13 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:13.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:13 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:34:13.859 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:13 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:34:13.859 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:13 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:13.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:13 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/388195924' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-22T01:34:13.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:13 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:13.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:13 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:13.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:13 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:13.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:13 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:13.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:13 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:13.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:13 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:13.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:13 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:34:13.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:13 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:34:13.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:13 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:13.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:13 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/388195924' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-22T01:34:13.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:13 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:13.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:13 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:13.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:13 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:13.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:13 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:13.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:13 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:13.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:13 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:13.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:13 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:34:13.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:13 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:34:13.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:13 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:34:14.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:14.447+0000 7f62b8ff9640 1 -- 172.21.15.72:0/3953349206 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v85) ==== 171+0+0 (secure 0 0 0) 0x7f62b00bd870 con 0x7f62bc101c90 2024-03-22T01:34:14.446 INFO:teuthology.orchestra.run.smithi072.stderr:pool 'cephfs_data' created 2024-03-22T01:34:14.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:14.449+0000 7f62c1d7a640 1 -- 172.21.15.72:0/3953349206 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f62a40a2f20 msgr2=0x7f62a40a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:14.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:14.449+0000 7f62c1d7a640 1 --2- 172.21.15.72:0/3953349206 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f62a40a2f20 0x7f62a40a53e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f62bc101af0 tx=0x7f62ac005e70 comp rx=0 tx=0).stop 2024-03-22T01:34:14.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:14.449+0000 7f62c1d7a640 1 -- 172.21.15.72:0/3953349206 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f62bc101c90 msgr2=0x7f62bc198480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:14.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:14.449+0000 7f62c1d7a640 1 --2- 172.21.15.72:0/3953349206 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f62bc101c90 0x7f62bc198480 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f62b0009230 tx=0x7f62b0005dd0 comp rx=0 tx=0).stop 2024-03-22T01:34:14.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:14.450+0000 7f62c1d7a640 1 -- 172.21.15.72:0/3953349206 shutdown_connections 2024-03-22T01:34:14.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:14.450+0000 7f62c1d7a640 1 --2- 172.21.15.72:0/3953349206 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f62a40a2f20 0x7f62a40a53e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:14.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:14.450+0000 7f62c1d7a640 1 --2- 172.21.15.72:0/3953349206 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62bc102650 0x7f62bc19f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:14.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:14.450+0000 7f62c1d7a640 1 --2- 172.21.15.72:0/3953349206 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f62bc101c90 0x7f62bc198480 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:14.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:14.450+0000 7f62c1d7a640 1 --2- 172.21.15.72:0/3953349206 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62bc100aa0 0x7f62bc197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:14.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:14.450+0000 7f62c1d7a640 1 -- 172.21.15.72:0/3953349206 >> 172.21.15.72:0/3953349206 conn(0x7f62bc0fbff0 msgr2=0x7f62bc104f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:14.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:14.450+0000 7f62c1d7a640 1 -- 172.21.15.72:0/3953349206 shutdown_connections 2024-03-22T01:34:14.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:14.450+0000 7f62c1d7a640 1 -- 172.21.15.72:0/3953349206 wait complete. 2024-03-22T01:34:14.473 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-22T01:34:14.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:14.583+0000 7f14cec47640 1 -- 172.21.15.72:0/667077504 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14c8100ab0 msgr2=0x7f14c8100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:14.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:14.583+0000 7f14cec47640 1 --2- 172.21.15.72:0/667077504 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14c8100ab0 0x7f14c8100ed0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f14bc009900 tx=0x7f14bc02bc30 comp rx=0 tx=0).stop 2024-03-22T01:34:14.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:14.583+0000 7f14cec47640 1 -- 172.21.15.72:0/667077504 shutdown_connections 2024-03-22T01:34:14.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:14.583+0000 7f14cec47640 1 --2- 172.21.15.72:0/667077504 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14c8102660 0x7f14c8108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:14.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:14.583+0000 7f14cec47640 1 --2- 172.21.15.72:0/667077504 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14c8101ca0 0x7f14c8102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:14.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:14.583+0000 7f14cec47640 1 --2- 172.21.15.72:0/667077504 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14c8100ab0 0x7f14c8100ed0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:14.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:14.583+0000 7f14cec47640 1 -- 172.21.15.72:0/667077504 >> 172.21.15.72:0/667077504 conn(0x7f14c80fc040 msgr2=0x7f14c80fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:14.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:14.583+0000 7f14cec47640 1 -- 172.21.15.72:0/667077504 shutdown_connections 2024-03-22T01:34:14.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:14.583+0000 7f14cec47640 1 -- 172.21.15.72:0/667077504 wait complete. 2024-03-22T01:34:14.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:14.583+0000 7f14cec47640 1 Processor -- start 2024-03-22T01:34:14.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:14.583+0000 7f14cec47640 1 -- start start 2024-03-22T01:34:14.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:14.584+0000 7f14cec47640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14c8100ab0 0x7f14c819a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:14.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:14.584+0000 7f14cec47640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14c8101ca0 0x7f14c819a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:14.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:14.584+0000 7f14cec47640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14c8102660 0x7f14c81a15f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:14.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:14.584+0000 7f14cec47640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f14c810b640 con 0x7f14c8101ca0 2024-03-22T01:34:14.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:14.584+0000 7f14cec47640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f14c810b4c0 con 0x7f14c8100ab0 2024-03-22T01:34:14.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:14.584+0000 7f14cec47640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f14c810b7c0 con 0x7f14c8102660 2024-03-22T01:34:14.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:14.584+0000 7f14c7fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14c8101ca0 0x7f14c819a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:14.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:14.584+0000 7f14c7fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14c8101ca0 0x7f14c819a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:35906/0 (socket says 172.21.15.72:35906) 2024-03-22T01:34:14.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:14.584+0000 7f14c7fff640 1 -- 172.21.15.72:0/1729468736 learned_addr learned my addr 172.21.15.72:0/1729468736 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:34:14.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:14.584+0000 7f14cc9e3640 1 --2- 172.21.15.72:0/1729468736 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14c8100ab0 0x7f14c819a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:14.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:14.584+0000 7f14cd1e4640 1 --2- 172.21.15.72:0/1729468736 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14c8102660 0x7f14c81a15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:14.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:14.584+0000 7f14c7fff640 1 -- 172.21.15.72:0/1729468736 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14c8102660 msgr2=0x7f14c81a15f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:14.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:14.584+0000 7f14c7fff640 1 --2- 172.21.15.72:0/1729468736 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14c8102660 0x7f14c81a15f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:14.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:14.584+0000 7f14c7fff640 1 -- 172.21.15.72:0/1729468736 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14c8100ab0 msgr2=0x7f14c819a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:14.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:14.584+0000 7f14c7fff640 1 --2- 172.21.15.72:0/1729468736 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14c8100ab0 0x7f14c819a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:14.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:14.584+0000 7f14c7fff640 1 -- 172.21.15.72:0/1729468736 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14c81a1c60 con 0x7f14c8101ca0 2024-03-22T01:34:14.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:14.584+0000 7f14c7fff640 1 --2- 172.21.15.72:0/1729468736 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14c8101ca0 0x7f14c819a580 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f14b400c920 tx=0x7f14b400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:14.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:14.584+0000 7f14c5ffb640 1 -- 172.21.15.72:0/1729468736 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b4007c70 con 0x7f14c8101ca0 2024-03-22T01:34:14.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:14.584+0000 7f14c5ffb640 1 -- 172.21.15.72:0/1729468736 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14b4007e10 con 0x7f14c8101ca0 2024-03-22T01:34:14.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:14.584+0000 7f14cec47640 1 -- 172.21.15.72:0/1729468736 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14c81a1f50 con 0x7f14c8101ca0 2024-03-22T01:34:14.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:14.584+0000 7f14c5ffb640 1 -- 172.21.15.72:0/1729468736 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b4018890 con 0x7f14c8101ca0 2024-03-22T01:34:14.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:14.585+0000 7f14cec47640 1 -- 172.21.15.72:0/1729468736 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14c81a2410 con 0x7f14c8101ca0 2024-03-22T01:34:14.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:14.585+0000 7f14cec47640 1 -- 172.21.15.72:0/1729468736 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14c8107dd0 con 0x7f14c8101ca0 2024-03-22T01:34:14.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:14.586+0000 7f14c5ffb640 1 -- 172.21.15.72:0/1729468736 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f14b4018a70 con 0x7f14c8101ca0 2024-03-22T01:34:14.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:14.587+0000 7f14c5ffb640 1 --2- 172.21.15.72:0/1729468736 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f14b80a2f20 0x7f14b80a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:14.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:14.587+0000 7f14cc9e3640 1 --2- 172.21.15.72:0/1729468736 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f14b80a2f20 0x7f14b80a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:14.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:14.587+0000 7f14c5ffb640 1 -- 172.21.15.72:0/1729468736 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7879+0+0 (secure 0 0 0) 0x7f14b40cf330 con 0x7f14c8101ca0 2024-03-22T01:34:14.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:14.588+0000 7f14cc9e3640 1 --2- 172.21.15.72:0/1729468736 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f14b80a2f20 0x7f14b80a53e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f14bc000c00 tx=0x7f14bc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:14.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:14.592+0000 7f14c5ffb640 1 -- 172.21.15.72:0/1729468736 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f14b4014030 con 0x7f14c8101ca0 2024-03-22T01:34:14.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:14 smithi072 ceph-mon[32414]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:34:14.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:14 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/388195924' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-22T01:34:14.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:14 smithi072 ceph-mon[32414]: osdmap e84: 12 total, 12 up, 12 in 2024-03-22T01:34:14.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:14 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/3953349206' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-22T01:34:14.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:14.814+0000 7f14cec47640 1 -- 172.21.15.72:0/1729468736 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f14c80628a0 con 0x7f14c8101ca0 2024-03-22T01:34:14.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:14 smithi089 ceph-mon[35770]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:34:14.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:14 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/388195924' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-22T01:34:14.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:14 smithi089 ceph-mon[35770]: osdmap e84: 12 total, 12 up, 12 in 2024-03-22T01:34:14.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:14 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/3953349206' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-22T01:34:14.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:14 smithi098 ceph-mon[36468]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:34:14.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:14 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/388195924' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-22T01:34:14.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:14 smithi098 ceph-mon[36468]: osdmap e84: 12 total, 12 up, 12 in 2024-03-22T01:34:14.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:14 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/3953349206' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-22T01:34:15.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:15.477+0000 7f14c5ffb640 1 -- 172.21.15.72:0/1729468736 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f14b409c310 con 0x7f14c8101ca0 2024-03-22T01:34:15.476 INFO:teuthology.orchestra.run.smithi072.stderr:new fs with metadata pool 2 and data pool 3 2024-03-22T01:34:15.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:15.481+0000 7f14cec47640 1 -- 172.21.15.72:0/1729468736 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f14b80a2f20 msgr2=0x7f14b80a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:15.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:15.481+0000 7f14cec47640 1 --2- 172.21.15.72:0/1729468736 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f14b80a2f20 0x7f14b80a53e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f14bc000c00 tx=0x7f14bc005fd0 comp rx=0 tx=0).stop 2024-03-22T01:34:15.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:15.481+0000 7f14cec47640 1 -- 172.21.15.72:0/1729468736 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14c8101ca0 msgr2=0x7f14c819a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:15.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:15.481+0000 7f14cec47640 1 --2- 172.21.15.72:0/1729468736 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14c8101ca0 0x7f14c819a580 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f14b400c920 tx=0x7f14b400cde0 comp rx=0 tx=0).stop 2024-03-22T01:34:15.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:15.481+0000 7f14cec47640 1 -- 172.21.15.72:0/1729468736 shutdown_connections 2024-03-22T01:34:15.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:15.481+0000 7f14cec47640 1 --2- 172.21.15.72:0/1729468736 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f14b80a2f20 0x7f14b80a53e0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:15.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:15.481+0000 7f14cec47640 1 --2- 172.21.15.72:0/1729468736 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14c8102660 0x7f14c81a15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:15.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:15.481+0000 7f14cec47640 1 --2- 172.21.15.72:0/1729468736 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14c8101ca0 0x7f14c819a580 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:15.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:15.481+0000 7f14cec47640 1 --2- 172.21.15.72:0/1729468736 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14c8100ab0 0x7f14c819a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:15.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:15.481+0000 7f14cec47640 1 -- 172.21.15.72:0/1729468736 >> 172.21.15.72:0/1729468736 conn(0x7f14c80fc040 msgr2=0x7f14c8104fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:15.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:15.481+0000 7f14cec47640 1 -- 172.21.15.72:0/1729468736 shutdown_connections 2024-03-22T01:34:15.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:15.481+0000 7f14cec47640 1 -- 172.21.15.72:0/1729468736 wait complete. 2024-03-22T01:34:15.499 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-03-22T01:34:15.499 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-03-22T01:34:15.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:15.609+0000 7fac3c8e1640 1 -- 172.21.15.72:0/2258325255 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fac380ffe80 msgr2=0x7fac381002a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:15.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:15.609+0000 7fac3c8e1640 1 --2- 172.21.15.72:0/2258325255 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fac380ffe80 0x7fac381002a0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fac28009900 tx=0x7fac2802bc80 comp rx=0 tx=0).stop 2024-03-22T01:34:15.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:15.609+0000 7fac3c8e1640 1 -- 172.21.15.72:0/2258325255 shutdown_connections 2024-03-22T01:34:15.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:15.609+0000 7fac3c8e1640 1 --2- 172.21.15.72:0/2258325255 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac38102ef0 0x7fac38103370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:15.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:15.609+0000 7fac3c8e1640 1 --2- 172.21.15.72:0/2258325255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac381038b0 0x7fac3810a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:15.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:15.609+0000 7fac3c8e1640 1 --2- 172.21.15.72:0/2258325255 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fac380ffe80 0x7fac381002a0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:15.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:15.609+0000 7fac3c8e1640 1 -- 172.21.15.72:0/2258325255 >> 172.21.15.72:0/2258325255 conn(0x7fac380fb7b0 msgr2=0x7fac380fdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:15.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:15.609+0000 7fac3c8e1640 1 -- 172.21.15.72:0/2258325255 shutdown_connections 2024-03-22T01:34:15.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:15.609+0000 7fac3c8e1640 1 -- 172.21.15.72:0/2258325255 wait complete. 2024-03-22T01:34:15.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:15.610+0000 7fac3c8e1640 1 Processor -- start 2024-03-22T01:34:15.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:15.610+0000 7fac3c8e1640 1 -- start start 2024-03-22T01:34:15.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:15.610+0000 7fac3c8e1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fac380ffe80 0x7fac38197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:15.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:15.610+0000 7fac3c8e1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac38102ef0 0x7fac38198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:15.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:15.610+0000 7fac3c8e1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac381038b0 0x7fac3819f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:15.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:15.610+0000 7fac3c8e1640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fac3810d8b0 con 0x7fac380ffe80 2024-03-22T01:34:15.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:15.610+0000 7fac3c8e1640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fac3810d730 con 0x7fac381038b0 2024-03-22T01:34:15.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:15.610+0000 7fac3c8e1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fac3810da30 con 0x7fac38102ef0 2024-03-22T01:34:15.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:15.610+0000 7fac3659c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fac380ffe80 0x7fac38197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:15.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:15.610+0000 7fac3659c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fac380ffe80 0x7fac38197ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:35938/0 (socket says 172.21.15.72:35938) 2024-03-22T01:34:15.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:15.610+0000 7fac3659c640 1 -- 172.21.15.72:0/1366268441 learned_addr learned my addr 172.21.15.72:0/1366268441 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:34:15.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:15.610+0000 7fac35d9b640 1 --2- 172.21.15.72:0/1366268441 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac38102ef0 0x7fac38198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:15.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:15.610+0000 7fac36d9d640 1 --2- 172.21.15.72:0/1366268441 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac381038b0 0x7fac3819f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:15.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:15.610+0000 7fac3659c640 1 -- 172.21.15.72:0/1366268441 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac38102ef0 msgr2=0x7fac38198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:15.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:15.610+0000 7fac3659c640 1 --2- 172.21.15.72:0/1366268441 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac38102ef0 0x7fac38198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:15.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:15.610+0000 7fac3659c640 1 -- 172.21.15.72:0/1366268441 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac381038b0 msgr2=0x7fac3819f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:15.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:15.610+0000 7fac3659c640 1 --2- 172.21.15.72:0/1366268441 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac381038b0 0x7fac3819f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:15.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:15.610+0000 7fac3659c640 1 -- 172.21.15.72:0/1366268441 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac3819faf0 con 0x7fac380ffe80 2024-03-22T01:34:15.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:15.610+0000 7fac36d9d640 1 --2- 172.21.15.72:0/1366268441 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac381038b0 0x7fac3819f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:34:15.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:15.610+0000 7fac3659c640 1 --2- 172.21.15.72:0/1366268441 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fac380ffe80 0x7fac38197ed0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fac28000c00 tx=0x7fac28004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:15.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:15.611+0000 7fac277fe640 1 -- 172.21.15.72:0/1366268441 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac28043020 con 0x7fac380ffe80 2024-03-22T01:34:15.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:15.611+0000 7fac277fe640 1 -- 172.21.15.72:0/1366268441 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac28004440 con 0x7fac380ffe80 2024-03-22T01:34:15.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:15.611+0000 7fac3c8e1640 1 -- 172.21.15.72:0/1366268441 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac3819fd80 con 0x7fac380ffe80 2024-03-22T01:34:15.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:15.611+0000 7fac277fe640 1 -- 172.21.15.72:0/1366268441 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac28003df0 con 0x7fac380ffe80 2024-03-22T01:34:15.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:15.611+0000 7fac3c8e1640 1 -- 172.21.15.72:0/1366268441 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac381a0210 con 0x7fac380ffe80 2024-03-22T01:34:15.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:15.612+0000 7fac3c8e1640 1 -- 172.21.15.72:0/1366268441 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac38109020 con 0x7fac380ffe80 2024-03-22T01:34:15.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:15.613+0000 7fac277fe640 1 -- 172.21.15.72:0/1366268441 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7fac28046020 con 0x7fac380ffe80 2024-03-22T01:34:15.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:15.613+0000 7fac277fe640 1 --2- 172.21.15.72:0/1366268441 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fac200a2f20 0x7fac200a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:15.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:15.614+0000 7fac35d9b640 1 --2- 172.21.15.72:0/1366268441 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fac200a2f20 0x7fac200a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:15.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:15.614+0000 7fac277fe640 1 -- 172.21.15.72:0/1366268441 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fac280f1080 con 0x7fac380ffe80 2024-03-22T01:34:15.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:15.614+0000 7fac35d9b640 1 --2- 172.21.15.72:0/1366268441 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fac200a2f20 0x7fac200a53e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fac381993e0 tx=0x7fac2c00a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:15.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:15.618+0000 7fac277fe640 1 -- 172.21.15.72:0/1366268441 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fac2804d070 con 0x7fac380ffe80 2024-03-22T01:34:15.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:15.802+0000 7fac3c8e1640 1 -- 172.21.15.72:0/1366268441 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7fac3802d3b0 con 0x7fac380ffe80 2024-03-22T01:34:15.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:15 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/3953349206' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-22T01:34:15.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:15 smithi089 ceph-mon[35770]: osdmap e85: 12 total, 12 up, 12 in 2024-03-22T01:34:15.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:15 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/1729468736' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-22T01:34:15.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:15 smithi089 ceph-mon[35770]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-22T01:34:15.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:15 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/3953349206' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-22T01:34:15.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:15 smithi072 ceph-mon[32414]: osdmap e85: 12 total, 12 up, 12 in 2024-03-22T01:34:15.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:15 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/1729468736' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-22T01:34:15.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:15 smithi072 ceph-mon[32414]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-22T01:34:15.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:15 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/3953349206' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-22T01:34:15.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:15 smithi098 ceph-mon[36468]: osdmap e85: 12 total, 12 up, 12 in 2024-03-22T01:34:15.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:15 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/1729468736' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-22T01:34:15.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:15 smithi098 ceph-mon[36468]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-22T01:34:16.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:16.467+0000 7fac277fe640 1 -- 172.21.15.72:0/1366268441 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v87) ==== 157+0+0 (secure 0 0 0) 0x7fac280be000 con 0x7fac380ffe80 2024-03-22T01:34:16.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:16.471+0000 7fac3c8e1640 1 -- 172.21.15.72:0/1366268441 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fac200a2f20 msgr2=0x7fac200a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:16.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:16.471+0000 7fac3c8e1640 1 --2- 172.21.15.72:0/1366268441 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fac200a2f20 0x7fac200a53e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fac381993e0 tx=0x7fac2c00a2d0 comp rx=0 tx=0).stop 2024-03-22T01:34:16.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:16.471+0000 7fac3c8e1640 1 -- 172.21.15.72:0/1366268441 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fac380ffe80 msgr2=0x7fac38197ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:16.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:16.471+0000 7fac3c8e1640 1 --2- 172.21.15.72:0/1366268441 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fac380ffe80 0x7fac38197ed0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fac28000c00 tx=0x7fac28004040 comp rx=0 tx=0).stop 2024-03-22T01:34:16.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:16.471+0000 7fac3c8e1640 1 -- 172.21.15.72:0/1366268441 shutdown_connections 2024-03-22T01:34:16.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:16.471+0000 7fac3c8e1640 1 --2- 172.21.15.72:0/1366268441 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fac200a2f20 0x7fac200a53e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:16.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:16.471+0000 7fac3c8e1640 1 --2- 172.21.15.72:0/1366268441 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac381038b0 0x7fac3819f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:16.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:16.471+0000 7fac3c8e1640 1 --2- 172.21.15.72:0/1366268441 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac38102ef0 0x7fac38198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:16.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:16.471+0000 7fac3c8e1640 1 --2- 172.21.15.72:0/1366268441 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fac380ffe80 0x7fac38197ed0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:16.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:16.471+0000 7fac3c8e1640 1 -- 172.21.15.72:0/1366268441 >> 172.21.15.72:0/1366268441 conn(0x7fac380fb7b0 msgr2=0x7fac381061f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:16.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:16.472+0000 7fac3c8e1640 1 -- 172.21.15.72:0/1366268441 shutdown_connections 2024-03-22T01:34:16.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:16.472+0000 7fac3c8e1640 1 -- 172.21.15.72:0/1366268441 wait complete. 2024-03-22T01:34:16.486 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-03-22T01:34:16.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:16.602+0000 7fa53e8df640 1 -- 172.21.15.72:0/82049523 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa538101c20 msgr2=0x7fa538102040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:16.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:16.602+0000 7fa53e8df640 1 --2- 172.21.15.72:0/82049523 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa538101c20 0x7fa538102040 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fa528009900 tx=0x7fa52802bc30 comp rx=0 tx=0).stop 2024-03-22T01:34:16.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:16.602+0000 7fa53e8df640 1 -- 172.21.15.72:0/82049523 shutdown_connections 2024-03-22T01:34:16.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:16.602+0000 7fa53e8df640 1 --2- 172.21.15.72:0/82049523 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5381037d0 0x7fa53810a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:16.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:16.602+0000 7fa53e8df640 1 --2- 172.21.15.72:0/82049523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa538102e10 0x7fa538103290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:16.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:16.602+0000 7fa53e8df640 1 --2- 172.21.15.72:0/82049523 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa538101c20 0x7fa538102040 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:16.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:16.602+0000 7fa53e8df640 1 -- 172.21.15.72:0/82049523 >> 172.21.15.72:0/82049523 conn(0x7fa5380fd170 msgr2=0x7fa5380ff5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:16.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:16.602+0000 7fa53e8df640 1 -- 172.21.15.72:0/82049523 shutdown_connections 2024-03-22T01:34:16.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:16.602+0000 7fa53e8df640 1 -- 172.21.15.72:0/82049523 wait complete. 2024-03-22T01:34:16.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:16.603+0000 7fa53e8df640 1 Processor -- start 2024-03-22T01:34:16.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:16.603+0000 7fa53e8df640 1 -- start start 2024-03-22T01:34:16.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:16.603+0000 7fa53e8df640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa538101c20 0x7fa538075940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:16.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:16.603+0000 7fa53e8df640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa538102e10 0x7fa538073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:16.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:16.603+0000 7fa53e8df640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5381037d0 0x7fa5380744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:16.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:16.603+0000 7fa53e8df640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fa53810d820 con 0x7fa538102e10 2024-03-22T01:34:16.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:16.603+0000 7fa53e8df640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa53810d6a0 con 0x7fa538101c20 2024-03-22T01:34:16.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:16.603+0000 7fa53e8df640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa53810d9a0 con 0x7fa5381037d0 2024-03-22T01:34:16.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:16.603+0000 7fa5377fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa538102e10 0x7fa538073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:16.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:16.603+0000 7fa5377fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa538102e10 0x7fa538073f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:35962/0 (socket says 172.21.15.72:35962) 2024-03-22T01:34:16.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:16.603+0000 7fa5377fe640 1 -- 172.21.15.72:0/517446742 learned_addr learned my addr 172.21.15.72:0/517446742 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:34:16.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:16.603+0000 7fa537fff640 1 --2- 172.21.15.72:0/517446742 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa538101c20 0x7fa538075940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:16.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:16.603+0000 7fa53ce7c640 1 --2- 172.21.15.72:0/517446742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5381037d0 0x7fa5380744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:16.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:16.603+0000 7fa5377fe640 1 -- 172.21.15.72:0/517446742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5381037d0 msgr2=0x7fa5380744c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:16.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:16.603+0000 7fa5377fe640 1 --2- 172.21.15.72:0/517446742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5381037d0 0x7fa5380744c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:16.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:16.603+0000 7fa5377fe640 1 -- 172.21.15.72:0/517446742 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa538101c20 msgr2=0x7fa538075940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:16.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:16.603+0000 7fa5377fe640 1 --2- 172.21.15.72:0/517446742 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa538101c20 0x7fa538075940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:16.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:16.604+0000 7fa5377fe640 1 -- 172.21.15.72:0/517446742 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa538074d50 con 0x7fa538102e10 2024-03-22T01:34:16.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:16.604+0000 7fa5377fe640 1 --2- 172.21.15.72:0/517446742 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa538102e10 0x7fa538073f80 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa52c00d770 tx=0x7fa52c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:16.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:16.604+0000 7fa5357fa640 1 -- 172.21.15.72:0/517446742 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa52c017070 con 0x7fa538102e10 2024-03-22T01:34:16.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:16.604+0000 7fa53e8df640 1 -- 172.21.15.72:0/517446742 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5381a1fc0 con 0x7fa538102e10 2024-03-22T01:34:16.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:16.604+0000 7fa5357fa640 1 -- 172.21.15.72:0/517446742 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa52c005500 con 0x7fa538102e10 2024-03-22T01:34:16.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:16.604+0000 7fa5357fa640 1 -- 172.21.15.72:0/517446742 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa52c004e00 con 0x7fa538102e10 2024-03-22T01:34:16.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:16.604+0000 7fa53e8df640 1 -- 172.21.15.72:0/517446742 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5381a23d0 con 0x7fa538102e10 2024-03-22T01:34:16.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:16.605+0000 7fa53e8df640 1 -- 172.21.15.72:0/517446742 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa538108f40 con 0x7fa538102e10 2024-03-22T01:34:16.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:16.606+0000 7fa5357fa640 1 -- 172.21.15.72:0/517446742 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7fa52c005000 con 0x7fa538102e10 2024-03-22T01:34:16.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:16.606+0000 7fa5357fa640 1 --2- 172.21.15.72:0/517446742 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fa5200a2f20 0x7fa5200a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:16.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:16.606+0000 7fa537fff640 1 --2- 172.21.15.72:0/517446742 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fa5200a2f20 0x7fa5200a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:16.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:16.606+0000 7fa5357fa640 1 -- 172.21.15.72:0/517446742 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8277+0+0 (secure 0 0 0) 0x7fa52c0ced10 con 0x7fa538102e10 2024-03-22T01:34:16.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:16.607+0000 7fa537fff640 1 --2- 172.21.15.72:0/517446742 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fa5200a2f20 0x7fa5200a53e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa528000c00 tx=0x7fa528005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:16.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:16.611+0000 7fa5357fa640 1 -- 172.21.15.72:0/517446742 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa52c09b4c0 con 0x7fa538102e10 2024-03-22T01:34:16.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:16.787+0000 7fa53e8df640 1 -- 172.21.15.72:0/517446742 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fa538062890 con 0x7fa538102e10 2024-03-22T01:34:16.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:16 smithi089 ceph-mon[35770]: pgmap v285: 97 pgs: 5 creating+peering, 4 active+clean, 88 unknown; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:34:16.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:16 smithi089 ceph-mon[35770]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-22T01:34:16.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:16 smithi089 ceph-mon[35770]: osdmap e86: 12 total, 12 up, 12 in 2024-03-22T01:34:16.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:16 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/1729468736' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-22T01:34:16.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:16 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-22T01:34:16.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:16 smithi089 ceph-mon[35770]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-22T01:34:16.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:16 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/1366268441' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-22T01:34:16.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:16 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/1366268441' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-22T01:34:16.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:16 smithi089 ceph-mon[35770]: osdmap e87: 12 total, 12 up, 12 in 2024-03-22T01:34:16.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:16 smithi072 ceph-mon[32414]: pgmap v285: 97 pgs: 5 creating+peering, 4 active+clean, 88 unknown; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:34:16.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:16 smithi072 ceph-mon[32414]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-22T01:34:16.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:16 smithi072 ceph-mon[32414]: osdmap e86: 12 total, 12 up, 12 in 2024-03-22T01:34:16.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:16 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/1729468736' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-22T01:34:16.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:16 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-22T01:34:16.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:16 smithi072 ceph-mon[32414]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-22T01:34:16.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:16 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/1366268441' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-22T01:34:16.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:16 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/1366268441' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-22T01:34:16.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:16 smithi072 ceph-mon[32414]: osdmap e87: 12 total, 12 up, 12 in 2024-03-22T01:34:16.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:16 smithi098 ceph-mon[36468]: pgmap v285: 97 pgs: 5 creating+peering, 4 active+clean, 88 unknown; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:34:16.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:16 smithi098 ceph-mon[36468]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-22T01:34:16.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:16 smithi098 ceph-mon[36468]: osdmap e86: 12 total, 12 up, 12 in 2024-03-22T01:34:16.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:16 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/1729468736' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-22T01:34:16.923 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:16 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-22T01:34:16.923 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:16 smithi098 ceph-mon[36468]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-22T01:34:16.923 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:16 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/1366268441' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-22T01:34:16.923 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:16 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/1366268441' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-22T01:34:16.923 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:16 smithi098 ceph-mon[36468]: osdmap e87: 12 total, 12 up, 12 in 2024-03-22T01:34:17.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:17 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:34:17.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:17 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-22T01:34:17.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:17 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-22T01:34:17.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:17 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/517446742' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-22T01:34:17.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:17 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:34:17.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:17 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-22T01:34:17.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:17 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-22T01:34:17.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:17 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/517446742' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-22T01:34:17.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:17 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:34:17.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:17 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-22T01:34:17.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:17 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-22T01:34:17.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:17 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/517446742' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-22T01:34:18.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:18.500+0000 7fa5357fa640 1 -- 172.21.15.72:0/517446742 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v89) ==== 228+0+0 (secure 0 0 0) 0x7fa52c09f560 con 0x7fa538102e10 2024-03-22T01:34:18.499 INFO:teuthology.orchestra.run.smithi072.stderr:pool 'cephfs_data_ec' created 2024-03-22T01:34:18.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:18.503+0000 7fa53e8df640 1 -- 172.21.15.72:0/517446742 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fa5200a2f20 msgr2=0x7fa5200a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:18.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:18.503+0000 7fa53e8df640 1 --2- 172.21.15.72:0/517446742 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fa5200a2f20 0x7fa5200a53e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa528000c00 tx=0x7fa528005fd0 comp rx=0 tx=0).stop 2024-03-22T01:34:18.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:18.503+0000 7fa53e8df640 1 -- 172.21.15.72:0/517446742 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa538102e10 msgr2=0x7fa538073f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:18.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:18.503+0000 7fa53e8df640 1 --2- 172.21.15.72:0/517446742 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa538102e10 0x7fa538073f80 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa52c00d770 tx=0x7fa52c00dc30 comp rx=0 tx=0).stop 2024-03-22T01:34:18.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:18.503+0000 7fa53e8df640 1 -- 172.21.15.72:0/517446742 shutdown_connections 2024-03-22T01:34:18.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:18.503+0000 7fa53e8df640 1 --2- 172.21.15.72:0/517446742 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fa5200a2f20 0x7fa5200a53e0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:18.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:18.503+0000 7fa53e8df640 1 --2- 172.21.15.72:0/517446742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5381037d0 0x7fa5380744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:18.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:18.503+0000 7fa53e8df640 1 --2- 172.21.15.72:0/517446742 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa538102e10 0x7fa538073f80 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:18.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:18.503+0000 7fa53e8df640 1 --2- 172.21.15.72:0/517446742 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa538101c20 0x7fa538075940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:18.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:18.503+0000 7fa53e8df640 1 -- 172.21.15.72:0/517446742 >> 172.21.15.72:0/517446742 conn(0x7fa5380fd170 msgr2=0x7fa538106110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:18.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:18.503+0000 7fa53e8df640 1 -- 172.21.15.72:0/517446742 shutdown_connections 2024-03-22T01:34:18.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:18.503+0000 7fa53e8df640 1 -- 172.21.15.72:0/517446742 wait complete. 2024-03-22T01:34:18.521 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-03-22T01:34:18.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:18.633+0000 7fcdfbff7640 1 -- 172.21.15.72:0/2599379053 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcdf4103810 msgr2=0x7fcdf410a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:18.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:18.633+0000 7fcdfbff7640 1 --2- 172.21.15.72:0/2599379053 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcdf4103810 0x7fcdf410a0a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fcdec009cd0 tx=0x7fcdec02c0f0 comp rx=0 tx=0).stop 2024-03-22T01:34:18.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:18.633+0000 7fcdfbff7640 1 -- 172.21.15.72:0/2599379053 shutdown_connections 2024-03-22T01:34:18.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:18.633+0000 7fcdfbff7640 1 --2- 172.21.15.72:0/2599379053 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcdf4103810 0x7fcdf410a0a0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:18.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:18.633+0000 7fcdfbff7640 1 --2- 172.21.15.72:0/2599379053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdf4102e50 0x7fcdf41032d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:18.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:18.633+0000 7fcdfbff7640 1 --2- 172.21.15.72:0/2599379053 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdf40ffe80 0x7fcdf41002a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:18.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:18.633+0000 7fcdfbff7640 1 -- 172.21.15.72:0/2599379053 >> 172.21.15.72:0/2599379053 conn(0x7fcdf40fb7b0 msgr2=0x7fcdf40fdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:18.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:18.633+0000 7fcdfbff7640 1 -- 172.21.15.72:0/2599379053 shutdown_connections 2024-03-22T01:34:18.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:18.633+0000 7fcdfbff7640 1 -- 172.21.15.72:0/2599379053 wait complete. 2024-03-22T01:34:18.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:18.633+0000 7fcdfbff7640 1 Processor -- start 2024-03-22T01:34:18.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:18.634+0000 7fcdfbff7640 1 -- start start 2024-03-22T01:34:18.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:18.634+0000 7fcdfbff7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdf40ffe80 0x7fcdf41027c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:18.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:18.634+0000 7fcdfbff7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdf4102e50 0x7fcdf4100e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:18.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:18.634+0000 7fcdfbff7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcdf4103810 0x7fcdf4101340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:18.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:18.634+0000 7fcdfbff7640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fcdf410d860 con 0x7fcdf4103810 2024-03-22T01:34:18.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:18.634+0000 7fcdfbff7640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fcdf410d6e0 con 0x7fcdf40ffe80 2024-03-22T01:34:18.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:18.634+0000 7fcdfbff7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcdf410d9e0 con 0x7fcdf4102e50 2024-03-22T01:34:18.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:18.634+0000 7fcdfa594640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcdf4103810 0x7fcdf4101340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:18.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:18.634+0000 7fcdf9d93640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdf40ffe80 0x7fcdf41027c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:18.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:18.634+0000 7fcdfa594640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcdf4103810 0x7fcdf4101340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:35970/0 (socket says 172.21.15.72:35970) 2024-03-22T01:34:18.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:18.634+0000 7fcdfa594640 1 -- 172.21.15.72:0/1154115290 learned_addr learned my addr 172.21.15.72:0/1154115290 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:34:18.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:18.634+0000 7fcdf9592640 1 --2- 172.21.15.72:0/1154115290 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdf4102e50 0x7fcdf4100e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:18.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:18.634+0000 7fcdfa594640 1 -- 172.21.15.72:0/1154115290 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdf4102e50 msgr2=0x7fcdf4100e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:18.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:18.634+0000 7fcdfa594640 1 --2- 172.21.15.72:0/1154115290 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdf4102e50 0x7fcdf4100e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:18.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:18.634+0000 7fcdfa594640 1 -- 172.21.15.72:0/1154115290 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdf40ffe80 msgr2=0x7fcdf41027c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:18.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:18.634+0000 7fcdfa594640 1 --2- 172.21.15.72:0/1154115290 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdf40ffe80 0x7fcdf41027c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:18.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:18.634+0000 7fcdfa594640 1 -- 172.21.15.72:0/1154115290 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdf4101bd0 con 0x7fcdf4103810 2024-03-22T01:34:18.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:18.634+0000 7fcdfa594640 1 --2- 172.21.15.72:0/1154115290 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcdf4103810 0x7fcdf4101340 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fcdec005e30 tx=0x7fcdec0346f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:18.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:18.634+0000 7fcde6ffd640 1 -- 172.21.15.72:0/1154115290 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdec041020 con 0x7fcdf4103810 2024-03-22T01:34:18.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:18.634+0000 7fcde6ffd640 1 -- 172.21.15.72:0/1154115290 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcdec032890 con 0x7fcdf4103810 2024-03-22T01:34:18.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:18.634+0000 7fcdfbff7640 1 -- 172.21.15.72:0/1154115290 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdf41a1f30 con 0x7fcdf4103810 2024-03-22T01:34:18.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:18.634+0000 7fcde6ffd640 1 -- 172.21.15.72:0/1154115290 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdec039040 con 0x7fcdf4103810 2024-03-22T01:34:18.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:18.634+0000 7fcdfbff7640 1 -- 172.21.15.72:0/1154115290 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdf41a2390 con 0x7fcdf4103810 2024-03-22T01:34:18.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:18.635+0000 7fcdfbff7640 1 -- 172.21.15.72:0/1154115290 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdf4108f20 con 0x7fcdf4103810 2024-03-22T01:34:18.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:18.636+0000 7fcde6ffd640 1 -- 172.21.15.72:0/1154115290 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7fcdec044020 con 0x7fcdf4103810 2024-03-22T01:34:18.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:18.637+0000 7fcde6ffd640 1 --2- 172.21.15.72:0/1154115290 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fcde00a2f20 0x7fcde00a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:18.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:18.637+0000 7fcdf9d93640 1 --2- 172.21.15.72:0/1154115290 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fcde00a2f20 0x7fcde00a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:18.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:18.637+0000 7fcde6ffd640 1 -- 172.21.15.72:0/1154115290 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8770+0+0 (secure 0 0 0) 0x7fcdec0f1740 con 0x7fcdf4103810 2024-03-22T01:34:18.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:18.637+0000 7fcdf9d93640 1 --2- 172.21.15.72:0/1154115290 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fcde00a2f20 0x7fcde00a53e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fcdf406a050 tx=0x7fcde8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:18.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:18.641+0000 7fcde6ffd640 1 -- 172.21.15.72:0/1154115290 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcdec04b020 con 0x7fcdf4103810 2024-03-22T01:34:18.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:18.815+0000 7fcdfbff7640 1 -- 172.21.15.72:0/1154115290 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7fcdf402d3b0 con 0x7fcdf4103810 2024-03-22T01:34:18.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:18 smithi089 ceph-mon[35770]: pgmap v288: 97 pgs: 5 creating+peering, 62 active+clean, 30 unknown; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 8 op/s 2024-03-22T01:34:18.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:18 smithi089 ceph-mon[35770]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-03-22T01:34:18.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:18 smithi089 ceph-mon[35770]: Cluster is now healthy 2024-03-22T01:34:18.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:18 smithi089 ceph-mon[35770]: osdmap e88: 12 total, 12 up, 12 in 2024-03-22T01:34:18.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:18 smithi072 ceph-mon[32414]: pgmap v288: 97 pgs: 5 creating+peering, 62 active+clean, 30 unknown; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 8 op/s 2024-03-22T01:34:18.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:18 smithi072 ceph-mon[32414]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-03-22T01:34:18.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:18 smithi072 ceph-mon[32414]: Cluster is now healthy 2024-03-22T01:34:18.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:18 smithi072 ceph-mon[32414]: osdmap e88: 12 total, 12 up, 12 in 2024-03-22T01:34:18.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:18 smithi098 ceph-mon[36468]: pgmap v288: 97 pgs: 5 creating+peering, 62 active+clean, 30 unknown; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 8 op/s 2024-03-22T01:34:18.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:18 smithi098 ceph-mon[36468]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-03-22T01:34:18.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:18 smithi098 ceph-mon[36468]: Cluster is now healthy 2024-03-22T01:34:18.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:18 smithi098 ceph-mon[36468]: osdmap e88: 12 total, 12 up, 12 in 2024-03-22T01:34:19.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:19.506+0000 7fcde6ffd640 1 -- 172.21.15.72:0/1154115290 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v90) ==== 169+0+0 (secure 0 0 0) 0x7fcdec0be350 con 0x7fcdf4103810 2024-03-22T01:34:19.505 INFO:teuthology.orchestra.run.smithi072.stderr:set pool 4 allow_ec_overwrites to true 2024-03-22T01:34:19.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:19.509+0000 7fcdfbff7640 1 -- 172.21.15.72:0/1154115290 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fcde00a2f20 msgr2=0x7fcde00a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:19.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:19.509+0000 7fcdfbff7640 1 --2- 172.21.15.72:0/1154115290 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fcde00a2f20 0x7fcde00a53e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fcdf406a050 tx=0x7fcde8005f90 comp rx=0 tx=0).stop 2024-03-22T01:34:19.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:19.509+0000 7fcdfbff7640 1 -- 172.21.15.72:0/1154115290 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcdf4103810 msgr2=0x7fcdf4101340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:19.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:19.509+0000 7fcdfbff7640 1 --2- 172.21.15.72:0/1154115290 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcdf4103810 0x7fcdf4101340 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fcdec005e30 tx=0x7fcdec0346f0 comp rx=0 tx=0).stop 2024-03-22T01:34:19.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:19.510+0000 7fcdfbff7640 1 -- 172.21.15.72:0/1154115290 shutdown_connections 2024-03-22T01:34:19.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:19.510+0000 7fcdfbff7640 1 --2- 172.21.15.72:0/1154115290 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fcde00a2f20 0x7fcde00a53e0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:19.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:19.510+0000 7fcdfbff7640 1 --2- 172.21.15.72:0/1154115290 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcdf4103810 0x7fcdf4101340 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:19.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:19.510+0000 7fcdfbff7640 1 --2- 172.21.15.72:0/1154115290 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdf4102e50 0x7fcdf4100e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:19.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:19.510+0000 7fcdfbff7640 1 --2- 172.21.15.72:0/1154115290 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdf40ffe80 0x7fcdf41027c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:19.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:19.510+0000 7fcdfbff7640 1 -- 172.21.15.72:0/1154115290 >> 172.21.15.72:0/1154115290 conn(0x7fcdf40fb7b0 msgr2=0x7fcdf41060f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:19.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:19.510+0000 7fcdfbff7640 1 -- 172.21.15.72:0/1154115290 shutdown_connections 2024-03-22T01:34:19.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:19.510+0000 7fcdfbff7640 1 -- 172.21.15.72:0/1154115290 wait complete. 2024-03-22T01:34:19.530 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-03-22T01:34:19.608 INFO:journalctl@ceph.mds.b.smithi089.stdout:Mar 22 01:34:19 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-b[66811]: 2024-03-22T01:34:19.268+0000 7ffb77952700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-22T01:34:19.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:19 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/517446742' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-22T01:34:19.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:19 smithi089 ceph-mon[35770]: osdmap e89: 12 total, 12 up, 12 in 2024-03-22T01:34:19.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:19 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/1154115290' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-22T01:34:19.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:19.639+0000 7f6fcd8a4640 1 -- 172.21.15.72:0/1050851379 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6fc8102690 msgr2=0x7f6fc8104a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:19.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:19.639+0000 7f6fcd8a4640 1 --2- 172.21.15.72:0/1050851379 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6fc8102690 0x7f6fc8104a80 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f6fb8009ca0 tx=0x7f6fb802c0f0 comp rx=0 tx=0).stop 2024-03-22T01:34:19.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:19.639+0000 7f6fcd8a4640 1 -- 172.21.15.72:0/1050851379 shutdown_connections 2024-03-22T01:34:19.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:19.639+0000 7f6fcd8a4640 1 --2- 172.21.15.72:0/1050851379 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fc81078f0 0x7f6fc8109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:19.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:19.639+0000 7f6fcd8a4640 1 --2- 172.21.15.72:0/1050851379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fc8104fc0 0x7f6fc81073b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:19.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:19.639+0000 7f6fcd8a4640 1 --2- 172.21.15.72:0/1050851379 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6fc8102690 0x7f6fc8104a80 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:19.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:19.639+0000 7f6fcd8a4640 1 -- 172.21.15.72:0/1050851379 >> 172.21.15.72:0/1050851379 conn(0x7f6fc80fc020 msgr2=0x7f6fc80fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:19.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:19.639+0000 7f6fcd8a4640 1 -- 172.21.15.72:0/1050851379 shutdown_connections 2024-03-22T01:34:19.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:19.639+0000 7f6fcd8a4640 1 -- 172.21.15.72:0/1050851379 wait complete. 2024-03-22T01:34:19.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:19.640+0000 7f6fcd8a4640 1 Processor -- start 2024-03-22T01:34:19.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:19.640+0000 7f6fcd8a4640 1 -- start start 2024-03-22T01:34:19.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:19.640+0000 7f6fcd8a4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fc8102690 0x7f6fc8197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:19.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:19.640+0000 7f6fcd8a4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fc8104fc0 0x7f6fc8198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:19.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:19.640+0000 7f6fcd8a4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6fc81078f0 0x7f6fc819f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:19.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:19.640+0000 7f6fcd8a4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f6fc810d850 con 0x7f6fc81078f0 2024-03-22T01:34:19.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:19.640+0000 7f6fcd8a4640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f6fc810d6d0 con 0x7f6fc8104fc0 2024-03-22T01:34:19.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:19.640+0000 7f6fcd8a4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6fc810d9d0 con 0x7f6fc8102690 2024-03-22T01:34:19.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:19.640+0000 7f6fc77fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6fc81078f0 0x7f6fc819f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:19.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:19.640+0000 7f6fc77fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6fc81078f0 0x7f6fc819f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:35982/0 (socket says 172.21.15.72:35982) 2024-03-22T01:34:19.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:19.640+0000 7f6fc77fe640 1 -- 172.21.15.72:0/3262355630 learned_addr learned my addr 172.21.15.72:0/3262355630 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:34:19.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:19.640+0000 7f6fc67fc640 1 --2- 172.21.15.72:0/3262355630 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fc8104fc0 0x7f6fc8198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:19.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:19.640+0000 7f6fc6ffd640 1 --2- 172.21.15.72:0/3262355630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fc8102690 0x7f6fc8197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:19.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:19.641+0000 7f6fc77fe640 1 -- 172.21.15.72:0/3262355630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fc8102690 msgr2=0x7f6fc8197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:19.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:19.641+0000 7f6fc77fe640 1 --2- 172.21.15.72:0/3262355630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fc8102690 0x7f6fc8197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:19.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:19.641+0000 7f6fc77fe640 1 -- 172.21.15.72:0/3262355630 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fc8104fc0 msgr2=0x7f6fc8198460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:19.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:19.641+0000 7f6fc77fe640 1 --2- 172.21.15.72:0/3262355630 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fc8104fc0 0x7f6fc8198460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:19.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:19.641+0000 7f6fc77fe640 1 -- 172.21.15.72:0/3262355630 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fc819fb40 con 0x7f6fc81078f0 2024-03-22T01:34:19.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:19.641+0000 7f6fc6ffd640 1 --2- 172.21.15.72:0/3262355630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fc8102690 0x7f6fc8197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:34:19.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:19.641+0000 7f6fc67fc640 1 --2- 172.21.15.72:0/3262355630 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fc8104fc0 0x7f6fc8198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:34:19.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:19.641+0000 7f6fc77fe640 1 --2- 172.21.15.72:0/3262355630 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6fc81078f0 0x7f6fc819f4d0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6fc000e970 tx=0x7f6fc000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:19.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:19.641+0000 7f6fcc8a2640 1 -- 172.21.15.72:0/3262355630 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fc000ccc0 con 0x7f6fc81078f0 2024-03-22T01:34:19.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:19.641+0000 7f6fcd8a4640 1 -- 172.21.15.72:0/3262355630 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fc819fe30 con 0x7f6fc81078f0 2024-03-22T01:34:19.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:19.641+0000 7f6fcc8a2640 1 -- 172.21.15.72:0/3262355630 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fc000ce60 con 0x7f6fc81078f0 2024-03-22T01:34:19.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:19.641+0000 7f6fcc8a2640 1 -- 172.21.15.72:0/3262355630 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fc0010490 con 0x7f6fc81078f0 2024-03-22T01:34:19.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:19.641+0000 7f6fcd8a4640 1 -- 172.21.15.72:0/3262355630 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fc81a02f0 con 0x7f6fc81078f0 2024-03-22T01:34:19.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:19.642+0000 7f6fcd8a4640 1 -- 172.21.15.72:0/3262355630 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fc80fdb10 con 0x7f6fc81078f0 2024-03-22T01:34:19.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:19.643+0000 7f6fcc8a2640 1 -- 172.21.15.72:0/3262355630 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f6fc0005020 con 0x7f6fc81078f0 2024-03-22T01:34:19.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:19.643+0000 7f6fcc8a2640 1 --2- 172.21.15.72:0/3262355630 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f6fb00a2f20 0x7f6fb00a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:19.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:19.644+0000 7f6fc6ffd640 1 --2- 172.21.15.72:0/3262355630 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f6fb00a2f20 0x7f6fb00a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:19.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:19.644+0000 7f6fcc8a2640 1 -- 172.21.15.72:0/3262355630 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8770+0+0 (secure 0 0 0) 0x7f6fc00cfb10 con 0x7f6fc81078f0 2024-03-22T01:34:19.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:19.644+0000 7f6fc6ffd640 1 --2- 172.21.15.72:0/3262355630 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f6fb00a2f20 0x7f6fb00a53e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6fb8006010 tx=0x7f6fb80325b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:19.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:19.648+0000 7f6fcc8a2640 1 -- 172.21.15.72:0/3262355630 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6fc009c720 con 0x7f6fc81078f0 2024-03-22T01:34:19.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:19.865+0000 7f6fcd8a4640 1 -- 172.21.15.72:0/3262355630 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7f6fc8062790 con 0x7f6fc81078f0 2024-03-22T01:34:19.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:19 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/517446742' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-22T01:34:19.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:19 smithi072 ceph-mon[32414]: osdmap e89: 12 total, 12 up, 12 in 2024-03-22T01:34:19.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:19 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/1154115290' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-22T01:34:19.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:19 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/517446742' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-22T01:34:19.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:19 smithi098 ceph-mon[36468]: osdmap e89: 12 total, 12 up, 12 in 2024-03-22T01:34:19.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:19 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/1154115290' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-22T01:34:20.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.513+0000 7f6fcc8a2640 1 -- 172.21.15.72:0/3262355630 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) ==== 137+0+0 (secure 0 0 0) 0x7f6fc009c020 con 0x7f6fc81078f0 2024-03-22T01:34:20.571 INFO:teuthology.orchestra.run.smithi072.stderr:added data pool 4 to fsmap 2024-03-22T01:34:20.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.517+0000 7f6fcd8a4640 1 -- 172.21.15.72:0/3262355630 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f6fb00a2f20 msgr2=0x7f6fb00a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:20.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.517+0000 7f6fcd8a4640 1 --2- 172.21.15.72:0/3262355630 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f6fb00a2f20 0x7f6fb00a53e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6fb8006010 tx=0x7f6fb80325b0 comp rx=0 tx=0).stop 2024-03-22T01:34:20.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.517+0000 7f6fcd8a4640 1 -- 172.21.15.72:0/3262355630 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6fc81078f0 msgr2=0x7f6fc819f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:20.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.517+0000 7f6fcd8a4640 1 --2- 172.21.15.72:0/3262355630 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6fc81078f0 0x7f6fc819f4d0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6fc000e970 tx=0x7f6fc000ed70 comp rx=0 tx=0).stop 2024-03-22T01:34:20.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.517+0000 7f6fcd8a4640 1 -- 172.21.15.72:0/3262355630 shutdown_connections 2024-03-22T01:34:20.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.517+0000 7f6fcd8a4640 1 --2- 172.21.15.72:0/3262355630 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f6fb00a2f20 0x7f6fb00a53e0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:20.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.517+0000 7f6fcd8a4640 1 --2- 172.21.15.72:0/3262355630 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fc8104fc0 0x7f6fc8198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:20.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.517+0000 7f6fcd8a4640 1 --2- 172.21.15.72:0/3262355630 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6fc81078f0 0x7f6fc819f4d0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:20.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.517+0000 7f6fcd8a4640 1 --2- 172.21.15.72:0/3262355630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fc8102690 0x7f6fc8197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:20.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.517+0000 7f6fcd8a4640 1 -- 172.21.15.72:0/3262355630 >> 172.21.15.72:0/3262355630 conn(0x7f6fc80fc020 msgr2=0x7f6fc8103110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:20.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.517+0000 7f6fcd8a4640 1 -- 172.21.15.72:0/3262355630 shutdown_connections 2024-03-22T01:34:20.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.517+0000 7f6fcd8a4640 1 -- 172.21.15.72:0/3262355630 wait complete. 2024-03-22T01:34:20.574 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T01:34:20.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.683+0000 7fe5dc4c3640 1 -- 172.21.15.72:0/2211446396 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe5d4100b30 msgr2=0x7fe5d4100f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:20.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.683+0000 7fe5dc4c3640 1 --2- 172.21.15.72:0/2211446396 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe5d4100b30 0x7fe5d4100f50 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fe5d0009900 tx=0x7fe5d002bc30 comp rx=0 tx=0).stop 2024-03-22T01:34:20.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.683+0000 7fe5dc4c3640 1 -- 172.21.15.72:0/2211446396 shutdown_connections 2024-03-22T01:34:20.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.683+0000 7fe5dc4c3640 1 --2- 172.21.15.72:0/2211446396 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5d41026e0 0x7fe5d4108f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:20.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.683+0000 7fe5dc4c3640 1 --2- 172.21.15.72:0/2211446396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5d4101d20 0x7fe5d41021a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:20.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.683+0000 7fe5dc4c3640 1 --2- 172.21.15.72:0/2211446396 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe5d4100b30 0x7fe5d4100f50 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:20.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.683+0000 7fe5dc4c3640 1 -- 172.21.15.72:0/2211446396 >> 172.21.15.72:0/2211446396 conn(0x7fe5d40fc0a0 msgr2=0x7fe5d40fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:20.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.683+0000 7fe5dc4c3640 1 -- 172.21.15.72:0/2211446396 shutdown_connections 2024-03-22T01:34:20.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.683+0000 7fe5dc4c3640 1 -- 172.21.15.72:0/2211446396 wait complete. 2024-03-22T01:34:20.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.684+0000 7fe5dc4c3640 1 Processor -- start 2024-03-22T01:34:20.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.684+0000 7fe5dc4c3640 1 -- start start 2024-03-22T01:34:20.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.684+0000 7fe5dc4c3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5d4100b30 0x7fe5d4197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:20.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.684+0000 7fe5dc4c3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe5d4101d20 0x7fe5d4198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:20.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.684+0000 7fe5dc4c3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5d41026e0 0x7fe5d419f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:20.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.684+0000 7fe5dc4c3640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fe5d410c7e0 con 0x7fe5d4101d20 2024-03-22T01:34:20.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.684+0000 7fe5dc4c3640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe5d410c660 con 0x7fe5d41026e0 2024-03-22T01:34:20.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.684+0000 7fe5dc4c3640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe5d410c960 con 0x7fe5d4100b30 2024-03-22T01:34:20.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.684+0000 7fe5d9a5e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe5d4101d20 0x7fe5d4198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:20.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.684+0000 7fe5d9a5e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe5d4101d20 0x7fe5d4198520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47746/0 (socket says 172.21.15.72:47746) 2024-03-22T01:34:20.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.684+0000 7fe5daa60640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5d41026e0 0x7fe5d419f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:20.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.684+0000 7fe5d9a5e640 1 -- 172.21.15.72:0/2791318743 learned_addr learned my addr 172.21.15.72:0/2791318743 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:34:20.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.684+0000 7fe5da25f640 1 --2- 172.21.15.72:0/2791318743 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5d4100b30 0x7fe5d4197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:20.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.684+0000 7fe5d9a5e640 1 -- 172.21.15.72:0/2791318743 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5d4100b30 msgr2=0x7fe5d4197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:20.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.684+0000 7fe5d9a5e640 1 --2- 172.21.15.72:0/2791318743 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5d4100b30 0x7fe5d4197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:20.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.684+0000 7fe5d9a5e640 1 -- 172.21.15.72:0/2791318743 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5d41026e0 msgr2=0x7fe5d419f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:20.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.684+0000 7fe5d9a5e640 1 --2- 172.21.15.72:0/2791318743 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5d41026e0 0x7fe5d419f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:20.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.684+0000 7fe5d9a5e640 1 -- 172.21.15.72:0/2791318743 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5d419fbb0 con 0x7fe5d4101d20 2024-03-22T01:34:20.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.684+0000 7fe5daa60640 1 --2- 172.21.15.72:0/2791318743 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5d41026e0 0x7fe5d419f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:34:20.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.684+0000 7fe5d9a5e640 1 --2- 172.21.15.72:0/2791318743 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe5d4101d20 0x7fe5d4198520 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fe5c400b4a0 tx=0x7fe5c400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:20.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.685+0000 7fe5cb7fe640 1 -- 172.21.15.72:0/2791318743 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5c4014070 con 0x7fe5d4101d20 2024-03-22T01:34:20.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.685+0000 7fe5cb7fe640 1 -- 172.21.15.72:0/2791318743 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe5c4010af0 con 0x7fe5d4101d20 2024-03-22T01:34:20.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.685+0000 7fe5dc4c3640 1 -- 172.21.15.72:0/2791318743 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5d419fea0 con 0x7fe5d4101d20 2024-03-22T01:34:20.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.685+0000 7fe5cb7fe640 1 -- 172.21.15.72:0/2791318743 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5c4017d10 con 0x7fe5d4101d20 2024-03-22T01:34:20.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.685+0000 7fe5dc4c3640 1 -- 172.21.15.72:0/2791318743 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5d41a0310 con 0x7fe5d4101d20 2024-03-22T01:34:20.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.686+0000 7fe5dc4c3640 1 -- 172.21.15.72:0/2791318743 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5d41a05a0 con 0x7fe5d4101d20 2024-03-22T01:34:20.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.687+0000 7fe5cb7fe640 1 -- 172.21.15.72:0/2791318743 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7fe5c4017070 con 0x7fe5d4101d20 2024-03-22T01:34:20.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.687+0000 7fe5cb7fe640 1 --2- 172.21.15.72:0/2791318743 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe5c00a2f20 0x7fe5c00a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:20.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.687+0000 7fe5da25f640 1 --2- 172.21.15.72:0/2791318743 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe5c00a2f20 0x7fe5c00a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:20.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.687+0000 7fe5cb7fe640 1 -- 172.21.15.72:0/2791318743 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fe5c40ced90 con 0x7fe5d4101d20 2024-03-22T01:34:20.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.688+0000 7fe5da25f640 1 --2- 172.21.15.72:0/2791318743 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe5c00a2f20 0x7fe5c00a53e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe5d002c170 tx=0x7fe5d002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:20.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.691+0000 7fe5cb7fe640 1 -- 172.21.15.72:0/2791318743 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe5c409b980 con 0x7fe5d4101d20 2024-03-22T01:34:20.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:20 smithi089 ceph-mon[35770]: pgmap v291: 129 pgs: 7 creating+peering, 93 active+clean, 29 unknown; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 9 op/s 2024-03-22T01:34:20.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:20 smithi089 ceph-mon[35770]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-22T01:34:20.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:20 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/1154115290' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-22T01:34:20.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:20 smithi089 ceph-mon[35770]: osdmap e90: 12 total, 12 up, 12 in 2024-03-22T01:34:20.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:20 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/3262355630' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-22T01:34:20.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:20 smithi072 ceph-mon[32414]: pgmap v291: 129 pgs: 7 creating+peering, 93 active+clean, 29 unknown; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 9 op/s 2024-03-22T01:34:20.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:20 smithi072 ceph-mon[32414]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-22T01:34:20.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:20 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/1154115290' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-22T01:34:20.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:20 smithi072 ceph-mon[32414]: osdmap e90: 12 total, 12 up, 12 in 2024-03-22T01:34:20.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:20 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/3262355630' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-22T01:34:20.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.904+0000 7fe5dc4c3640 1 -- 172.21.15.72:0/2791318743 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe5d41a0770 con 0x7fe5d4101d20 2024-03-22T01:34:20.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.905+0000 7fe5cb7fe640 1 -- 172.21.15.72:0/2791318743 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11776 (secure 0 0 0) 0x7fe5c409b280 con 0x7fe5d4101d20 2024-03-22T01:34:20.904 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:34:20.905 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6835/3077216308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3077216308},{"type":"v1","addr":"172.21.15.72:6835","nonce":3077216308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2460254143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":2460254143},{"type":"v1","addr":"172.21.15.72:6837","nonce":2460254143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6839/2026432819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6838","nonce":2026432819},{"type":"v1","addr":"172.21.15.72:6839","nonce":2026432819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6841/400094545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6840","nonce":400094545},{"type":"v1","addr":"172.21.15.72:6841","nonce":400094545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6833/1578701171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":1578701171},{"type":"v1","addr":"172.21.15.89:6833","nonce":1578701171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/2703047213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2703047213},{"type":"v1","addr":"172.21.15.98:6833","nonce":2703047213}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2589468787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2589468787},{"type":"v1","addr":"172.21.15.89:6835","nonce":2589468787}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/92657636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":92657636},{"type":"v1","addr":"172.21.15.98:6835","nonce":92657636}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1662941782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1662941782},{"type":"v1","addr":"172.21.15.89:6837","nonce":1662941782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24490,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/781860371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":781860371},{"type":"v1","addr":"172.21.15.98:6837","nonce":781860371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24517,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/3192173650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3192173650},{"type":"v1","addr":"172.21.15.98:6839","nonce":3192173650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T01:34:14.815677+0000","modified":"2024-03-22T01:34:19.866556+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24524},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6839/149200280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":149200280},{"type":"v1","addr":"172.21.15.89:6839","nonce":149200280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24524,"qdb_cluster":[24524]},"id":1}]} 2024-03-22T01:34:20.905 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 17 2024-03-22T01:34:20.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.907+0000 7fe5dc4c3640 1 -- 172.21.15.72:0/2791318743 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe5c00a2f20 msgr2=0x7fe5c00a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:20.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.907+0000 7fe5dc4c3640 1 --2- 172.21.15.72:0/2791318743 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe5c00a2f20 0x7fe5c00a53e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe5d002c170 tx=0x7fe5d002df90 comp rx=0 tx=0).stop 2024-03-22T01:34:20.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.907+0000 7fe5dc4c3640 1 -- 172.21.15.72:0/2791318743 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe5d4101d20 msgr2=0x7fe5d4198520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:20.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.907+0000 7fe5dc4c3640 1 --2- 172.21.15.72:0/2791318743 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe5d4101d20 0x7fe5d4198520 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fe5c400b4a0 tx=0x7fe5c400b960 comp rx=0 tx=0).stop 2024-03-22T01:34:20.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.907+0000 7fe5dc4c3640 1 -- 172.21.15.72:0/2791318743 shutdown_connections 2024-03-22T01:34:20.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.907+0000 7fe5dc4c3640 1 --2- 172.21.15.72:0/2791318743 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe5c00a2f20 0x7fe5c00a53e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:20.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.907+0000 7fe5dc4c3640 1 --2- 172.21.15.72:0/2791318743 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5d41026e0 0x7fe5d419f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:20.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.907+0000 7fe5dc4c3640 1 --2- 172.21.15.72:0/2791318743 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe5d4101d20 0x7fe5d4198520 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:20.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.907+0000 7fe5dc4c3640 1 --2- 172.21.15.72:0/2791318743 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5d4100b30 0x7fe5d4197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:20.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.908+0000 7fe5dc4c3640 1 -- 172.21.15.72:0/2791318743 >> 172.21.15.72:0/2791318743 conn(0x7fe5d40fc0a0 msgr2=0x7fe5d40fdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:20.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.908+0000 7fe5dc4c3640 1 -- 172.21.15.72:0/2791318743 shutdown_connections 2024-03-22T01:34:20.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:20.908+0000 7fe5dc4c3640 1 -- 172.21.15.72:0/2791318743 wait complete. 2024-03-22T01:34:20.918 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T01:34:20.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:20 smithi098 ceph-mon[36468]: pgmap v291: 129 pgs: 7 creating+peering, 93 active+clean, 29 unknown; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 9 op/s 2024-03-22T01:34:20.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:20 smithi098 ceph-mon[36468]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-22T01:34:20.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:20 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/1154115290' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-22T01:34:20.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:20 smithi098 ceph-mon[36468]: osdmap e90: 12 total, 12 up, 12 in 2024-03-22T01:34:20.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:20 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/3262355630' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-22T01:34:21.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.021+0000 7f99d745c640 1 -- 172.21.15.72:0/2747431677 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99d0101c20 msgr2=0x7f99d0102040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:21.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.021+0000 7f99d745c640 1 --2- 172.21.15.72:0/2747431677 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99d0101c20 0x7f99d0102040 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f99c4009900 tx=0x7f99c402bc30 comp rx=0 tx=0).stop 2024-03-22T01:34:21.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.021+0000 7f99d745c640 1 -- 172.21.15.72:0/2747431677 shutdown_connections 2024-03-22T01:34:21.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.021+0000 7f99d745c640 1 --2- 172.21.15.72:0/2747431677 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99d01037d0 0x7f99d010a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:21.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.021+0000 7f99d745c640 1 --2- 172.21.15.72:0/2747431677 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99d0102e10 0x7f99d0103290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:21.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.021+0000 7f99d745c640 1 --2- 172.21.15.72:0/2747431677 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99d0101c20 0x7f99d0102040 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:21.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.021+0000 7f99d745c640 1 -- 172.21.15.72:0/2747431677 >> 172.21.15.72:0/2747431677 conn(0x7f99d00fd1b0 msgr2=0x7f99d00ff5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:21.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.021+0000 7f99d745c640 1 -- 172.21.15.72:0/2747431677 shutdown_connections 2024-03-22T01:34:21.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.021+0000 7f99d745c640 1 -- 172.21.15.72:0/2747431677 wait complete. 2024-03-22T01:34:21.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.022+0000 7f99d745c640 1 Processor -- start 2024-03-22T01:34:21.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.022+0000 7f99d745c640 1 -- start start 2024-03-22T01:34:21.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.022+0000 7f99d745c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99d0101c20 0x7f99d0197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:21.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.022+0000 7f99d745c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99d0102e10 0x7f99d0198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:21.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.022+0000 7f99d745c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99d01037d0 0x7f99d019f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:21.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.022+0000 7f99d745c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f99d010d820 con 0x7f99d0101c20 2024-03-22T01:34:21.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.022+0000 7f99d745c640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f99d010d6a0 con 0x7f99d0102e10 2024-03-22T01:34:21.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.022+0000 7f99d745c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f99d010d9a0 con 0x7f99d01037d0 2024-03-22T01:34:21.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.022+0000 7f99d51f8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99d0101c20 0x7f99d0197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:21.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.022+0000 7f99d51f8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99d0101c20 0x7f99d0197ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47770/0 (socket says 172.21.15.72:47770) 2024-03-22T01:34:21.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.022+0000 7f99d51f8640 1 -- 172.21.15.72:0/374404390 learned_addr learned my addr 172.21.15.72:0/374404390 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:34:21.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.022+0000 7f99d49f7640 1 --2- 172.21.15.72:0/374404390 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99d0102e10 0x7f99d0198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:21.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.022+0000 7f99d59f9640 1 --2- 172.21.15.72:0/374404390 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99d01037d0 0x7f99d019f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:21.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.022+0000 7f99d51f8640 1 -- 172.21.15.72:0/374404390 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99d01037d0 msgr2=0x7f99d019f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:21.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.022+0000 7f99d51f8640 1 --2- 172.21.15.72:0/374404390 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99d01037d0 0x7f99d019f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:21.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.022+0000 7f99d51f8640 1 -- 172.21.15.72:0/374404390 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99d0102e10 msgr2=0x7f99d0198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:21.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.022+0000 7f99d51f8640 1 --2- 172.21.15.72:0/374404390 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99d0102e10 0x7f99d0198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:21.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.022+0000 7f99d51f8640 1 -- 172.21.15.72:0/374404390 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99d019fb10 con 0x7f99d0101c20 2024-03-22T01:34:21.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.022+0000 7f99d59f9640 1 --2- 172.21.15.72:0/374404390 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99d01037d0 0x7f99d019f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T01:34:21.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.022+0000 7f99d51f8640 1 --2- 172.21.15.72:0/374404390 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99d0101c20 0x7f99d0197ef0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f99c4000c00 tx=0x7f99c4005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:21.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.022+0000 7f99c27fc640 1 -- 172.21.15.72:0/374404390 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99c4041070 con 0x7f99d0101c20 2024-03-22T01:34:21.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.023+0000 7f99c27fc640 1 -- 172.21.15.72:0/374404390 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99c40044d0 con 0x7f99d0101c20 2024-03-22T01:34:21.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.023+0000 7f99d745c640 1 -- 172.21.15.72:0/374404390 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99d019fda0 con 0x7f99d0101c20 2024-03-22T01:34:21.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.023+0000 7f99c27fc640 1 -- 172.21.15.72:0/374404390 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99c4034440 con 0x7f99d0101c20 2024-03-22T01:34:21.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.023+0000 7f99d745c640 1 -- 172.21.15.72:0/374404390 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99d01a01d0 con 0x7f99d0101c20 2024-03-22T01:34:21.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.023+0000 7f99d745c640 1 -- 172.21.15.72:0/374404390 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99d0108f90 con 0x7f99d0101c20 2024-03-22T01:34:21.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.024+0000 7f99c27fc640 1 -- 172.21.15.72:0/374404390 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f99c4044070 con 0x7f99d0101c20 2024-03-22T01:34:21.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.025+0000 7f99c27fc640 1 --2- 172.21.15.72:0/374404390 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f99bc0a2f20 0x7f99bc0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:21.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.025+0000 7f99d49f7640 1 --2- 172.21.15.72:0/374404390 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f99bc0a2f20 0x7f99bc0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:21.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.025+0000 7f99c27fc640 1 -- 172.21.15.72:0/374404390 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f99c40f11d0 con 0x7f99d0101c20 2024-03-22T01:34:21.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.025+0000 7f99d49f7640 1 --2- 172.21.15.72:0/374404390 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f99bc0a2f20 0x7f99bc0a53e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f99d0199400 tx=0x7f99cc00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:21.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.029+0000 7f99c27fc640 1 -- 172.21.15.72:0/374404390 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f99c40bddc0 con 0x7f99d0101c20 2024-03-22T01:34:21.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.207+0000 7f99d745c640 1 -- 172.21.15.72:0/374404390 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f99d0062860 con 0x7f99d0101c20 2024-03-22T01:34:21.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.209+0000 7f99c27fc640 1 -- 172.21.15.72:0/374404390 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24045 (secure 0 0 0) 0x7f99c40bd6c0 con 0x7f99d0101c20 2024-03-22T01:34:21.208 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:34:21.208 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":91,"fsid":"022a99bc-e7eb-11ee-95cd-87774f69a715","created":"2024-03-22T01:26:23.893672+0000","modified":"2024-03-22T01:34:19.866367+0000","last_up_change":"2024-03-22T01:33:25.223684+0000","last_in_change":"2024-03-22T01:33:05.010331+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T01:29:37.438503+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-22T01:34:12.610422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-22T01:34:13.761686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-22T01:34:17.507039+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"16c15bb0-db92-4191-9063-38b4890e81e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6803","nonce":661906791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6805","nonce":661906791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6809","nonce":661906791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6807","nonce":661906791}]},"public_addr":"172.21.15.72:6803/661906791","cluster_addr":"172.21.15.72:6805/661906791","heartbeat_back_addr":"172.21.15.72:6809/661906791","heartbeat_front_addr":"172.21.15.72:6807/661906791","state":["exists","up"]},{"osd":1,"uuid":"981290ba-97dc-496b-aa22-d51704184b10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6801","nonce":2870590550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6803","nonce":2870590550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6807","nonce":2870590550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6805","nonce":2870590550}]},"public_addr":"172.21.15.89:6801/2870590550","cluster_addr":"172.21.15.89:6803/2870590550","heartbeat_back_addr":"172.21.15.89:6807/2870590550","heartbeat_front_addr":"172.21.15.89:6805/2870590550","state":["exists","up"]},{"osd":2,"uuid":"f07ed658-f545-46a1-8edd-8b1d5d49885c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6801","nonce":3865778118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6803","nonce":3865778118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6807","nonce":3865778118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6805","nonce":3865778118}]},"public_addr":"172.21.15.98:6801/3865778118","cluster_addr":"172.21.15.98:6803/3865778118","heartbeat_back_addr":"172.21.15.98:6807/3865778118","heartbeat_front_addr":"172.21.15.98:6805/3865778118","state":["exists","up"]},{"osd":3,"uuid":"ccc4bc40-2c48-46d2-8548-0c05b44f9f1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6811","nonce":1865951395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6813","nonce":1865951395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6817","nonce":1865951395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6815","nonce":1865951395}]},"public_addr":"172.21.15.72:6811/1865951395","cluster_addr":"172.21.15.72:6813/1865951395","heartbeat_back_addr":"172.21.15.72:6817/1865951395","heartbeat_front_addr":"172.21.15.72:6815/1865951395","state":["exists","up"]},{"osd":4,"uuid":"1812439f-810a-4ed2-9395-c00c9a085a13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6809","nonce":2683760795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6811","nonce":2683760795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6815","nonce":2683760795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6813","nonce":2683760795}]},"public_addr":"172.21.15.89:6809/2683760795","cluster_addr":"172.21.15.89:6811/2683760795","heartbeat_back_addr":"172.21.15.89:6815/2683760795","heartbeat_front_addr":"172.21.15.89:6813/2683760795","state":["exists","up"]},{"osd":5,"uuid":"e78954d8-f4e3-4d58-8536-d2d59c843386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6809","nonce":2635838572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6811","nonce":2635838572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6815","nonce":2635838572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6813","nonce":2635838572}]},"public_addr":"172.21.15.98:6809/2635838572","cluster_addr":"172.21.15.98:6811/2635838572","heartbeat_back_addr":"172.21.15.98:6815/2635838572","heartbeat_front_addr":"172.21.15.98:6813/2635838572","state":["exists","up"]},{"osd":6,"uuid":"dc6c8be2-b64c-4959-8015-7b0ea6d25599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6819","nonce":3158982661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6821","nonce":3158982661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6825","nonce":3158982661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6823","nonce":3158982661}]},"public_addr":"172.21.15.72:6819/3158982661","cluster_addr":"172.21.15.72:6821/3158982661","heartbeat_back_addr":"172.21.15.72:6825/3158982661","heartbeat_front_addr":"172.21.15.72:6823/3158982661","state":["exists","up"]},{"osd":7,"uuid":"7a73469f-cf3a-4306-ae93-ea46e21ebbbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6817","nonce":3075913234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6819","nonce":3075913234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6823","nonce":3075913234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6821","nonce":3075913234}]},"public_addr":"172.21.15.89:6817/3075913234","cluster_addr":"172.21.15.89:6819/3075913234","heartbeat_back_addr":"172.21.15.89:6823/3075913234","heartbeat_front_addr":"172.21.15.89:6821/3075913234","state":["exists","up"]},{"osd":8,"uuid":"c4b13bee-ea4d-4344-95fb-2622ecbb8196","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6817","nonce":1813255577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6819","nonce":1813255577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6823","nonce":1813255577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6821","nonce":1813255577}]},"public_addr":"172.21.15.98:6817/1813255577","cluster_addr":"172.21.15.98:6819/1813255577","heartbeat_back_addr":"172.21.15.98:6823/1813255577","heartbeat_front_addr":"172.21.15.98:6821/1813255577","state":["exists","up"]},{"osd":9,"uuid":"eefb0a44-4a36-4e55-9d04-0501bf718db6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6827","nonce":3377749206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6829","nonce":3377749206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6832","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6833","nonce":3377749206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6831","nonce":3377749206}]},"public_addr":"172.21.15.72:6827/3377749206","cluster_addr":"172.21.15.72:6829/3377749206","heartbeat_back_addr":"172.21.15.72:6833/3377749206","heartbeat_front_addr":"172.21.15.72:6831/3377749206","state":["exists","up"]},{"osd":10,"uuid":"8741d25f-40dd-40d7-b040-5cc5e411eb98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6825","nonce":885058786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6827","nonce":885058786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6831","nonce":885058786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6829","nonce":885058786}]},"public_addr":"172.21.15.89:6825/885058786","cluster_addr":"172.21.15.89:6827/885058786","heartbeat_back_addr":"172.21.15.89:6831/885058786","heartbeat_front_addr":"172.21.15.89:6829/885058786","state":["exists","up"]},{"osd":11,"uuid":"8d02e957-eda4-4184-b5c6-5eb3b8b38060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6825","nonce":1517095838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6827","nonce":1517095838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6831","nonce":1517095838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6829","nonce":1517095838}]},"public_addr":"172.21.15.98:6825/1517095838","cluster_addr":"172.21.15.98:6827/1517095838","heartbeat_back_addr":"172.21.15.98:6831/1517095838","heartbeat_front_addr":"172.21.15.98:6829/1517095838","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:28:47.307652+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:11.037135+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:34.619188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:59.350738+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:30:23.787337+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:30:48.994154+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:31:13.690350+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:31:39.628116+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:04.723912+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:30.689741+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:57.229508+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:33:23.154537+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:0/121087196":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/2451857761":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/1228477995":"2024-03-23T01:27:15.339809+0000","172.21.15.72:6801/1513232913":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/3558958264":"2024-03-23T01:26:50.074871+0000","172.21.15.72:0/1510187198":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6800/3518012916":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6801/3518012916":"2024-03-23T01:26:50.074871+0000","172.21.15.72:0/1372801041":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6800/1513232913":"2024-03-23T01:27:15.339809+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T01:34:21.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.211+0000 7f99d745c640 1 -- 172.21.15.72:0/374404390 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f99bc0a2f20 msgr2=0x7f99bc0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:21.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.211+0000 7f99d745c640 1 --2- 172.21.15.72:0/374404390 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f99bc0a2f20 0x7f99bc0a53e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f99d0199400 tx=0x7f99cc00b390 comp rx=0 tx=0).stop 2024-03-22T01:34:21.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.211+0000 7f99d745c640 1 -- 172.21.15.72:0/374404390 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99d0101c20 msgr2=0x7f99d0197ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:21.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.211+0000 7f99d745c640 1 --2- 172.21.15.72:0/374404390 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99d0101c20 0x7f99d0197ef0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f99c4000c00 tx=0x7f99c4005d50 comp rx=0 tx=0).stop 2024-03-22T01:34:21.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.211+0000 7f99d745c640 1 -- 172.21.15.72:0/374404390 shutdown_connections 2024-03-22T01:34:21.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.211+0000 7f99d745c640 1 --2- 172.21.15.72:0/374404390 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f99bc0a2f20 0x7f99bc0a53e0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:21.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.211+0000 7f99d745c640 1 --2- 172.21.15.72:0/374404390 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99d0102e10 0x7f99d0198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:21.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.211+0000 7f99d745c640 1 --2- 172.21.15.72:0/374404390 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99d01037d0 0x7f99d019f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:21.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.211+0000 7f99d745c640 1 --2- 172.21.15.72:0/374404390 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99d0101c20 0x7f99d0197ef0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:21.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.211+0000 7f99d745c640 1 -- 172.21.15.72:0/374404390 >> 172.21.15.72:0/374404390 conn(0x7f99d00fd1b0 msgr2=0x7f99d0106160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:21.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.211+0000 7f99d745c640 1 -- 172.21.15.72:0/374404390 shutdown_connections 2024-03-22T01:34:21.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.211+0000 7f99d745c640 1 -- 172.21.15.72:0/374404390 wait complete. 2024-03-22T01:34:21.222 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T01:34:21.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.323+0000 7f173be6b640 1 -- 172.21.15.72:0/1526123954 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1734100b30 msgr2=0x7f1734100f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:21.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.323+0000 7f173be6b640 1 --2- 172.21.15.72:0/1526123954 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1734100b30 0x7f1734100f50 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f1730009900 tx=0x7f173002bc30 comp rx=0 tx=0).stop 2024-03-22T01:34:21.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.323+0000 7f173be6b640 1 -- 172.21.15.72:0/1526123954 shutdown_connections 2024-03-22T01:34:21.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.323+0000 7f173be6b640 1 --2- 172.21.15.72:0/1526123954 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1734101d20 0x7f17341021a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:21.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.323+0000 7f173be6b640 1 --2- 172.21.15.72:0/1526123954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17341026e0 0x7f1734108f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:21.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.323+0000 7f173be6b640 1 --2- 172.21.15.72:0/1526123954 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1734100b30 0x7f1734100f50 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:21.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.323+0000 7f173be6b640 1 -- 172.21.15.72:0/1526123954 >> 172.21.15.72:0/1526123954 conn(0x7f17340fc0a0 msgr2=0x7f17340fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:21.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.323+0000 7f173be6b640 1 -- 172.21.15.72:0/1526123954 shutdown_connections 2024-03-22T01:34:21.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.323+0000 7f173be6b640 1 -- 172.21.15.72:0/1526123954 wait complete. 2024-03-22T01:34:21.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.324+0000 7f173be6b640 1 Processor -- start 2024-03-22T01:34:21.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.324+0000 7f173be6b640 1 -- start start 2024-03-22T01:34:21.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.324+0000 7f173be6b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1734100b30 0x7f1734197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:21.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.324+0000 7f173be6b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1734101d20 0x7f1734198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:21.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.324+0000 7f173be6b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17341026e0 0x7f173419f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:21.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.324+0000 7f173be6b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f173410c740 con 0x7f17341026e0 2024-03-22T01:34:21.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.324+0000 7f173be6b640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f173410c5c0 con 0x7f1734101d20 2024-03-22T01:34:21.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.324+0000 7f173be6b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f173410c8c0 con 0x7f1734100b30 2024-03-22T01:34:21.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.324+0000 7f173a408640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17341026e0 0x7f173419f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:21.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.324+0000 7f173a408640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17341026e0 0x7f173419f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47798/0 (socket says 172.21.15.72:47798) 2024-03-22T01:34:21.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.324+0000 7f173a408640 1 -- 172.21.15.72:0/1417488519 learned_addr learned my addr 172.21.15.72:0/1417488519 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:34:21.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.324+0000 7f1739406640 1 --2- 172.21.15.72:0/1417488519 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1734101d20 0x7f1734198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:21.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.324+0000 7f1739c07640 1 --2- 172.21.15.72:0/1417488519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1734100b30 0x7f1734197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:21.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.324+0000 7f173a408640 1 -- 172.21.15.72:0/1417488519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1734100b30 msgr2=0x7f1734197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:21.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.324+0000 7f173a408640 1 --2- 172.21.15.72:0/1417488519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1734100b30 0x7f1734197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:21.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.324+0000 7f173a408640 1 -- 172.21.15.72:0/1417488519 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1734101d20 msgr2=0x7f1734198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:21.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.324+0000 7f173a408640 1 --2- 172.21.15.72:0/1417488519 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1734101d20 0x7f1734198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:21.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.324+0000 7f173a408640 1 -- 172.21.15.72:0/1417488519 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f173419fb00 con 0x7f17341026e0 2024-03-22T01:34:21.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.324+0000 7f1739c07640 1 --2- 172.21.15.72:0/1417488519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1734100b30 0x7f1734197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:34:21.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.324+0000 7f173a408640 1 --2- 172.21.15.72:0/1417488519 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17341026e0 0x7f173419f490 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f172c00eaa0 tx=0x7f172c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:21.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.325+0000 7f172affd640 1 -- 172.21.15.72:0/1417488519 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f172c00cdb0 con 0x7f17341026e0 2024-03-22T01:34:21.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.325+0000 7f173be6b640 1 -- 172.21.15.72:0/1417488519 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f173419fdf0 con 0x7f17341026e0 2024-03-22T01:34:21.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.325+0000 7f172affd640 1 -- 172.21.15.72:0/1417488519 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f172c005020 con 0x7f17341026e0 2024-03-22T01:34:21.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.325+0000 7f172affd640 1 -- 172.21.15.72:0/1417488519 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f172c0116d0 con 0x7f17341026e0 2024-03-22T01:34:21.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.325+0000 7f173be6b640 1 -- 172.21.15.72:0/1417488519 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17341a02b0 con 0x7f17341026e0 2024-03-22T01:34:21.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.326+0000 7f173be6b640 1 -- 172.21.15.72:0/1417488519 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17341a0570 con 0x7f17341026e0 2024-03-22T01:34:21.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.327+0000 7f172affd640 1 -- 172.21.15.72:0/1417488519 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f172c0054d0 con 0x7f17341026e0 2024-03-22T01:34:21.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.327+0000 7f172affd640 1 --2- 172.21.15.72:0/1417488519 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f17200a2f20 0x7f17200a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:21.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.328+0000 7f1739c07640 1 --2- 172.21.15.72:0/1417488519 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f17200a2f20 0x7f17200a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:21.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.328+0000 7f172affd640 1 -- 172.21.15.72:0/1417488519 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f172c0cf830 con 0x7f17341026e0 2024-03-22T01:34:21.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.328+0000 7f1739c07640 1 --2- 172.21.15.72:0/1417488519 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f17200a2f20 0x7f17200a53e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f173002c7b0 tx=0x7f1730005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:21.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.332+0000 7f172affd640 1 -- 172.21.15.72:0/1417488519 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f172c016030 con 0x7f17341026e0 2024-03-22T01:34:21.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.509+0000 7f173be6b640 1 -- 172.21.15.72:0/1417488519 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f17341a07e0 con 0x7f17341026e0 2024-03-22T01:34:21.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.513+0000 7f172affd640 1 -- 172.21.15.72:0/1417488519 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24045 (secure 0 0 0) 0x7f172c09c420 con 0x7f17341026e0 2024-03-22T01:34:21.512 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:34:21.513 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":91,"fsid":"022a99bc-e7eb-11ee-95cd-87774f69a715","created":"2024-03-22T01:26:23.893672+0000","modified":"2024-03-22T01:34:19.866367+0000","last_up_change":"2024-03-22T01:33:25.223684+0000","last_in_change":"2024-03-22T01:33:05.010331+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T01:29:37.438503+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-22T01:34:12.610422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-22T01:34:13.761686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-22T01:34:17.507039+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"16c15bb0-db92-4191-9063-38b4890e81e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6803","nonce":661906791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6805","nonce":661906791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6809","nonce":661906791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6807","nonce":661906791}]},"public_addr":"172.21.15.72:6803/661906791","cluster_addr":"172.21.15.72:6805/661906791","heartbeat_back_addr":"172.21.15.72:6809/661906791","heartbeat_front_addr":"172.21.15.72:6807/661906791","state":["exists","up"]},{"osd":1,"uuid":"981290ba-97dc-496b-aa22-d51704184b10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6801","nonce":2870590550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6803","nonce":2870590550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6807","nonce":2870590550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6805","nonce":2870590550}]},"public_addr":"172.21.15.89:6801/2870590550","cluster_addr":"172.21.15.89:6803/2870590550","heartbeat_back_addr":"172.21.15.89:6807/2870590550","heartbeat_front_addr":"172.21.15.89:6805/2870590550","state":["exists","up"]},{"osd":2,"uuid":"f07ed658-f545-46a1-8edd-8b1d5d49885c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6801","nonce":3865778118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6803","nonce":3865778118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6807","nonce":3865778118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6805","nonce":3865778118}]},"public_addr":"172.21.15.98:6801/3865778118","cluster_addr":"172.21.15.98:6803/3865778118","heartbeat_back_addr":"172.21.15.98:6807/3865778118","heartbeat_front_addr":"172.21.15.98:6805/3865778118","state":["exists","up"]},{"osd":3,"uuid":"ccc4bc40-2c48-46d2-8548-0c05b44f9f1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6811","nonce":1865951395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6813","nonce":1865951395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6817","nonce":1865951395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6815","nonce":1865951395}]},"public_addr":"172.21.15.72:6811/1865951395","cluster_addr":"172.21.15.72:6813/1865951395","heartbeat_back_addr":"172.21.15.72:6817/1865951395","heartbeat_front_addr":"172.21.15.72:6815/1865951395","state":["exists","up"]},{"osd":4,"uuid":"1812439f-810a-4ed2-9395-c00c9a085a13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6809","nonce":2683760795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6811","nonce":2683760795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6815","nonce":2683760795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6813","nonce":2683760795}]},"public_addr":"172.21.15.89:6809/2683760795","cluster_addr":"172.21.15.89:6811/2683760795","heartbeat_back_addr":"172.21.15.89:6815/2683760795","heartbeat_front_addr":"172.21.15.89:6813/2683760795","state":["exists","up"]},{"osd":5,"uuid":"e78954d8-f4e3-4d58-8536-d2d59c843386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6809","nonce":2635838572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6811","nonce":2635838572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6815","nonce":2635838572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6813","nonce":2635838572}]},"public_addr":"172.21.15.98:6809/2635838572","cluster_addr":"172.21.15.98:6811/2635838572","heartbeat_back_addr":"172.21.15.98:6815/2635838572","heartbeat_front_addr":"172.21.15.98:6813/2635838572","state":["exists","up"]},{"osd":6,"uuid":"dc6c8be2-b64c-4959-8015-7b0ea6d25599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6819","nonce":3158982661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6821","nonce":3158982661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6825","nonce":3158982661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6823","nonce":3158982661}]},"public_addr":"172.21.15.72:6819/3158982661","cluster_addr":"172.21.15.72:6821/3158982661","heartbeat_back_addr":"172.21.15.72:6825/3158982661","heartbeat_front_addr":"172.21.15.72:6823/3158982661","state":["exists","up"]},{"osd":7,"uuid":"7a73469f-cf3a-4306-ae93-ea46e21ebbbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6817","nonce":3075913234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6819","nonce":3075913234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6823","nonce":3075913234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6821","nonce":3075913234}]},"public_addr":"172.21.15.89:6817/3075913234","cluster_addr":"172.21.15.89:6819/3075913234","heartbeat_back_addr":"172.21.15.89:6823/3075913234","heartbeat_front_addr":"172.21.15.89:6821/3075913234","state":["exists","up"]},{"osd":8,"uuid":"c4b13bee-ea4d-4344-95fb-2622ecbb8196","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6817","nonce":1813255577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6819","nonce":1813255577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6823","nonce":1813255577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6821","nonce":1813255577}]},"public_addr":"172.21.15.98:6817/1813255577","cluster_addr":"172.21.15.98:6819/1813255577","heartbeat_back_addr":"172.21.15.98:6823/1813255577","heartbeat_front_addr":"172.21.15.98:6821/1813255577","state":["exists","up"]},{"osd":9,"uuid":"eefb0a44-4a36-4e55-9d04-0501bf718db6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6827","nonce":3377749206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6829","nonce":3377749206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6832","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6833","nonce":3377749206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6831","nonce":3377749206}]},"public_addr":"172.21.15.72:6827/3377749206","cluster_addr":"172.21.15.72:6829/3377749206","heartbeat_back_addr":"172.21.15.72:6833/3377749206","heartbeat_front_addr":"172.21.15.72:6831/3377749206","state":["exists","up"]},{"osd":10,"uuid":"8741d25f-40dd-40d7-b040-5cc5e411eb98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6825","nonce":885058786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6827","nonce":885058786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6831","nonce":885058786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6829","nonce":885058786}]},"public_addr":"172.21.15.89:6825/885058786","cluster_addr":"172.21.15.89:6827/885058786","heartbeat_back_addr":"172.21.15.89:6831/885058786","heartbeat_front_addr":"172.21.15.89:6829/885058786","state":["exists","up"]},{"osd":11,"uuid":"8d02e957-eda4-4184-b5c6-5eb3b8b38060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6825","nonce":1517095838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6827","nonce":1517095838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6831","nonce":1517095838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6829","nonce":1517095838}]},"public_addr":"172.21.15.98:6825/1517095838","cluster_addr":"172.21.15.98:6827/1517095838","heartbeat_back_addr":"172.21.15.98:6831/1517095838","heartbeat_front_addr":"172.21.15.98:6829/1517095838","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:28:47.307652+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:11.037135+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:34.619188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:59.350738+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:30:23.787337+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:30:48.994154+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:31:13.690350+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:31:39.628116+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:04.723912+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:30.689741+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:57.229508+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:33:23.154537+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:0/121087196":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/2451857761":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/1228477995":"2024-03-23T01:27:15.339809+0000","172.21.15.72:6801/1513232913":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/3558958264":"2024-03-23T01:26:50.074871+0000","172.21.15.72:0/1510187198":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6800/3518012916":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6801/3518012916":"2024-03-23T01:26:50.074871+0000","172.21.15.72:0/1372801041":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6800/1513232913":"2024-03-23T01:27:15.339809+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T01:34:21.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.515+0000 7f173be6b640 1 -- 172.21.15.72:0/1417488519 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f17200a2f20 msgr2=0x7f17200a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:21.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.515+0000 7f173be6b640 1 --2- 172.21.15.72:0/1417488519 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f17200a2f20 0x7f17200a53e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f173002c7b0 tx=0x7f1730005fd0 comp rx=0 tx=0).stop 2024-03-22T01:34:21.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.515+0000 7f173be6b640 1 -- 172.21.15.72:0/1417488519 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17341026e0 msgr2=0x7f173419f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:21.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.515+0000 7f173be6b640 1 --2- 172.21.15.72:0/1417488519 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17341026e0 0x7f173419f490 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f172c00eaa0 tx=0x7f172c00ef60 comp rx=0 tx=0).stop 2024-03-22T01:34:21.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.515+0000 7f173be6b640 1 -- 172.21.15.72:0/1417488519 shutdown_connections 2024-03-22T01:34:21.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.515+0000 7f173be6b640 1 --2- 172.21.15.72:0/1417488519 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f17200a2f20 0x7f17200a53e0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:21.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.515+0000 7f173be6b640 1 --2- 172.21.15.72:0/1417488519 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1734101d20 0x7f1734198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:21.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.515+0000 7f173be6b640 1 --2- 172.21.15.72:0/1417488519 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17341026e0 0x7f173419f490 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:21.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.515+0000 7f173be6b640 1 --2- 172.21.15.72:0/1417488519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1734100b30 0x7f1734197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:21.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.515+0000 7f173be6b640 1 -- 172.21.15.72:0/1417488519 >> 172.21.15.72:0/1417488519 conn(0x7f17340fc0a0 msgr2=0x7f17340fdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:21.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.515+0000 7f173be6b640 1 -- 172.21.15.72:0/1417488519 shutdown_connections 2024-03-22T01:34:21.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.515+0000 7f173be6b640 1 -- 172.21.15.72:0/1417488519 wait complete. 2024-03-22T01:34:21.548 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T01:34:21.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.651+0000 7fe4994fa640 1 -- 172.21.15.72:0/1711321603 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe494102e60 msgr2=0x7fe4941032e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:21.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.651+0000 7fe4994fa640 1 --2- 172.21.15.72:0/1711321603 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe494102e60 0x7fe4941032e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fe47c009930 tx=0x7fe47c02bc30 comp rx=0 tx=0).stop 2024-03-22T01:34:21.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.651+0000 7fe4994fa640 1 -- 172.21.15.72:0/1711321603 shutdown_connections 2024-03-22T01:34:21.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.651+0000 7fe4994fa640 1 --2- 172.21.15.72:0/1711321603 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe494103820 0x7fe49410a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:21.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.651+0000 7fe4994fa640 1 --2- 172.21.15.72:0/1711321603 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe494102e60 0x7fe4941032e0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:21.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.651+0000 7fe4994fa640 1 --2- 172.21.15.72:0/1711321603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4940ffe90 0x7fe4941002b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:21.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.651+0000 7fe4994fa640 1 -- 172.21.15.72:0/1711321603 >> 172.21.15.72:0/1711321603 conn(0x7fe4940fb7c0 msgr2=0x7fe4940fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:21.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.651+0000 7fe4994fa640 1 -- 172.21.15.72:0/1711321603 shutdown_connections 2024-03-22T01:34:21.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.651+0000 7fe4994fa640 1 -- 172.21.15.72:0/1711321603 wait complete. 2024-03-22T01:34:21.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.652+0000 7fe4994fa640 1 Processor -- start 2024-03-22T01:34:21.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.652+0000 7fe4994fa640 1 -- start start 2024-03-22T01:34:21.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.652+0000 7fe4994fa640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe4940ffe90 0x7fe494197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:21.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.652+0000 7fe4994fa640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe494102e60 0x7fe494198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:21.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.652+0000 7fe4994fa640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe494103820 0x7fe49419f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:21.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.652+0000 7fe4994fa640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fe49410d850 con 0x7fe4940ffe90 2024-03-22T01:34:21.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.652+0000 7fe4994fa640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe49410d6d0 con 0x7fe494103820 2024-03-22T01:34:21.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.652+0000 7fe4994fa640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe49410d9d0 con 0x7fe494102e60 2024-03-22T01:34:21.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.652+0000 7fe492ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe4940ffe90 0x7fe494197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:21.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.652+0000 7fe492ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe4940ffe90 0x7fe494197f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47824/0 (socket says 172.21.15.72:47824) 2024-03-22T01:34:21.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.652+0000 7fe492ffd640 1 -- 172.21.15.72:0/3657003804 learned_addr learned my addr 172.21.15.72:0/3657003804 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:34:21.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.652+0000 7fe4937fe640 1 --2- 172.21.15.72:0/3657003804 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe494103820 0x7fe49419f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:21.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.652+0000 7fe48bfff640 1 --2- 172.21.15.72:0/3657003804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe494102e60 0x7fe494198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:21.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.652+0000 7fe492ffd640 1 -- 172.21.15.72:0/3657003804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe494102e60 msgr2=0x7fe494198490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:21.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.652+0000 7fe492ffd640 1 --2- 172.21.15.72:0/3657003804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe494102e60 0x7fe494198490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:21.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.652+0000 7fe492ffd640 1 -- 172.21.15.72:0/3657003804 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe494103820 msgr2=0x7fe49419f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:21.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.652+0000 7fe492ffd640 1 --2- 172.21.15.72:0/3657003804 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe494103820 0x7fe49419f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:21.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.652+0000 7fe492ffd640 1 -- 172.21.15.72:0/3657003804 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe49419fb70 con 0x7fe4940ffe90 2024-03-22T01:34:21.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.652+0000 7fe48bfff640 1 --2- 172.21.15.72:0/3657003804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe494102e60 0x7fe494198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T01:34:21.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.652+0000 7fe492ffd640 1 --2- 172.21.15.72:0/3657003804 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe4940ffe90 0x7fe494197f50 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe48400e9c0 tx=0x7fe48400edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:21.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.652+0000 7fe490ff9640 1 -- 172.21.15.72:0/3657003804 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe48400ccc0 con 0x7fe4940ffe90 2024-03-22T01:34:21.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.653+0000 7fe4994fa640 1 -- 172.21.15.72:0/3657003804 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe49419fe60 con 0x7fe4940ffe90 2024-03-22T01:34:21.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.653+0000 7fe490ff9640 1 -- 172.21.15.72:0/3657003804 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4840035f0 con 0x7fe4940ffe90 2024-03-22T01:34:21.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.653+0000 7fe490ff9640 1 -- 172.21.15.72:0/3657003804 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe48400ce60 con 0x7fe4940ffe90 2024-03-22T01:34:21.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.653+0000 7fe4994fa640 1 -- 172.21.15.72:0/3657003804 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4941a02d0 con 0x7fe4940ffe90 2024-03-22T01:34:21.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.653+0000 7fe4994fa640 1 -- 172.21.15.72:0/3657003804 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe494109000 con 0x7fe4940ffe90 2024-03-22T01:34:21.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.654+0000 7fe490ff9640 1 -- 172.21.15.72:0/3657003804 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7fe4840037a0 con 0x7fe4940ffe90 2024-03-22T01:34:21.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.655+0000 7fe490ff9640 1 --2- 172.21.15.72:0/3657003804 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe4800a2f20 0x7fe4800a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:21.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.655+0000 7fe48bfff640 1 --2- 172.21.15.72:0/3657003804 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe4800a2f20 0x7fe4800a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:21.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.655+0000 7fe490ff9640 1 -- 172.21.15.72:0/3657003804 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fe4840cfa80 con 0x7fe4940ffe90 2024-03-22T01:34:21.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.655+0000 7fe48bfff640 1 --2- 172.21.15.72:0/3657003804 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe4800a2f20 0x7fe4800a53e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe47c0096c0 tx=0x7fe47c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:21.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.659+0000 7fe490ff9640 1 -- 172.21.15.72:0/3657003804 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe48409c670 con 0x7fe4940ffe90 2024-03-22T01:34:21.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:21 smithi089 ceph-mon[35770]: osdmap e91: 12 total, 12 up, 12 in 2024-03-22T01:34:21.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:21 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/3262355630' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-22T01:34:21.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:21 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-22T01:34:21.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:21 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/2791318743' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:34:21.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:21 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/374404390' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:34:21.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:21 smithi072 ceph-mon[32414]: osdmap e91: 12 total, 12 up, 12 in 2024-03-22T01:34:21.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:21 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/3262355630' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-22T01:34:21.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:21 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-22T01:34:21.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:21 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/2791318743' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:34:21.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:21 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/374404390' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:34:21.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.874+0000 7fe4994fa640 1 -- 172.21.15.72:0/3657003804 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe494062830 con 0x7fe4940ffe90 2024-03-22T01:34:21.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.875+0000 7fe490ff9640 1 -- 172.21.15.72:0/3657003804 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11776 (secure 0 0 0) 0x7fe48409bf70 con 0x7fe4940ffe90 2024-03-22T01:34:21.875 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:34:21.875 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6835/3077216308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3077216308},{"type":"v1","addr":"172.21.15.72:6835","nonce":3077216308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2460254143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":2460254143},{"type":"v1","addr":"172.21.15.72:6837","nonce":2460254143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6839/2026432819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6838","nonce":2026432819},{"type":"v1","addr":"172.21.15.72:6839","nonce":2026432819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6841/400094545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6840","nonce":400094545},{"type":"v1","addr":"172.21.15.72:6841","nonce":400094545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6833/1578701171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":1578701171},{"type":"v1","addr":"172.21.15.89:6833","nonce":1578701171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/2703047213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2703047213},{"type":"v1","addr":"172.21.15.98:6833","nonce":2703047213}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2589468787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2589468787},{"type":"v1","addr":"172.21.15.89:6835","nonce":2589468787}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/92657636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":92657636},{"type":"v1","addr":"172.21.15.98:6835","nonce":92657636}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1662941782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1662941782},{"type":"v1","addr":"172.21.15.89:6837","nonce":1662941782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24490,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/781860371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":781860371},{"type":"v1","addr":"172.21.15.98:6837","nonce":781860371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24517,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/3192173650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3192173650},{"type":"v1","addr":"172.21.15.98:6839","nonce":3192173650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T01:34:14.815677+0000","modified":"2024-03-22T01:34:19.866556+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24524},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6839/149200280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":149200280},{"type":"v1","addr":"172.21.15.89:6839","nonce":149200280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24524,"qdb_cluster":[24524]},"id":1}]} 2024-03-22T01:34:21.875 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 17 2024-03-22T01:34:21.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.877+0000 7fe4994fa640 1 -- 172.21.15.72:0/3657003804 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe4800a2f20 msgr2=0x7fe4800a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:21.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.877+0000 7fe4994fa640 1 --2- 172.21.15.72:0/3657003804 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe4800a2f20 0x7fe4800a53e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe47c0096c0 tx=0x7fe47c005fd0 comp rx=0 tx=0).stop 2024-03-22T01:34:21.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.877+0000 7fe4994fa640 1 -- 172.21.15.72:0/3657003804 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe4940ffe90 msgr2=0x7fe494197f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:21.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.877+0000 7fe4994fa640 1 --2- 172.21.15.72:0/3657003804 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe4940ffe90 0x7fe494197f50 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe48400e9c0 tx=0x7fe48400edc0 comp rx=0 tx=0).stop 2024-03-22T01:34:21.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.877+0000 7fe4994fa640 1 -- 172.21.15.72:0/3657003804 shutdown_connections 2024-03-22T01:34:21.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.877+0000 7fe4994fa640 1 --2- 172.21.15.72:0/3657003804 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe4800a2f20 0x7fe4800a53e0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:21.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.877+0000 7fe4994fa640 1 --2- 172.21.15.72:0/3657003804 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe494103820 0x7fe49419f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:21.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.877+0000 7fe4994fa640 1 --2- 172.21.15.72:0/3657003804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe494102e60 0x7fe494198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:21.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.877+0000 7fe4994fa640 1 --2- 172.21.15.72:0/3657003804 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe4940ffe90 0x7fe494197f50 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:21.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.877+0000 7fe4994fa640 1 -- 172.21.15.72:0/3657003804 >> 172.21.15.72:0/3657003804 conn(0x7fe4940fb7c0 msgr2=0x7fe4941061d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:21.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.877+0000 7fe4994fa640 1 -- 172.21.15.72:0/3657003804 shutdown_connections 2024-03-22T01:34:21.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.877+0000 7fe4994fa640 1 -- 172.21.15.72:0/3657003804 wait complete. 2024-03-22T01:34:21.888 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T01:34:21.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:21 smithi098 ceph-mon[36468]: osdmap e91: 12 total, 12 up, 12 in 2024-03-22T01:34:21.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:21 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/3262355630' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-22T01:34:21.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:21 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-22T01:34:21.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:21 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/2791318743' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:34:21.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:21 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/374404390' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:34:21.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.991+0000 7fcdba1f6640 1 -- 172.21.15.72:0/702846153 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcdb4104f70 msgr2=0x7fcdb4107360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:21.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.991+0000 7fcdba1f6640 1 --2- 172.21.15.72:0/702846153 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcdb4104f70 0x7fcdb4107360 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c009900 tx=0x7fcd9c02bc30 comp rx=0 tx=0).stop 2024-03-22T01:34:21.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.991+0000 7fcdba1f6640 1 -- 172.21.15.72:0/702846153 shutdown_connections 2024-03-22T01:34:21.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.991+0000 7fcdba1f6640 1 --2- 172.21.15.72:0/702846153 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdb41078a0 0x7fcdb4109cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:21.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.991+0000 7fcdba1f6640 1 --2- 172.21.15.72:0/702846153 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcdb4104f70 0x7fcdb4107360 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:21.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.991+0000 7fcdba1f6640 1 --2- 172.21.15.72:0/702846153 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdb4102640 0x7fcdb4104a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:21.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.991+0000 7fcdba1f6640 1 -- 172.21.15.72:0/702846153 >> 172.21.15.72:0/702846153 conn(0x7fcdb40fc050 msgr2=0x7fcdb40fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:21.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.991+0000 7fcdba1f6640 1 -- 172.21.15.72:0/702846153 shutdown_connections 2024-03-22T01:34:21.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.991+0000 7fcdba1f6640 1 -- 172.21.15.72:0/702846153 wait complete. 2024-03-22T01:34:21.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.991+0000 7fcdba1f6640 1 Processor -- start 2024-03-22T01:34:21.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.991+0000 7fcdba1f6640 1 -- start start 2024-03-22T01:34:21.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.991+0000 7fcdba1f6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcdb4102640 0x7fcdb4197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:21.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.991+0000 7fcdba1f6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdb4104f70 0x7fcdb41984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:21.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.991+0000 7fcdba1f6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdb41078a0 0x7fcdb419f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:21.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.991+0000 7fcdba1f6640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fcdb410d8a0 con 0x7fcdb4102640 2024-03-22T01:34:21.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.991+0000 7fcdba1f6640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fcdb410d720 con 0x7fcdb4104f70 2024-03-22T01:34:21.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.991+0000 7fcdba1f6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcdb410da20 con 0x7fcdb41078a0 2024-03-22T01:34:21.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.992+0000 7fcdb37fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcdb4102640 0x7fcdb4197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:21.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.992+0000 7fcdb37fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcdb4102640 0x7fcdb4197f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47840/0 (socket says 172.21.15.72:47840) 2024-03-22T01:34:21.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.992+0000 7fcdb37fe640 1 -- 172.21.15.72:0/146534477 learned_addr learned my addr 172.21.15.72:0/146534477 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:34:21.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.992+0000 7fcdabfff640 1 --2- 172.21.15.72:0/146534477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdb4104f70 0x7fcdb41984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:21.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.992+0000 7fcdb37fe640 1 -- 172.21.15.72:0/146534477 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdb41078a0 msgr2=0x7fcdb419f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:21.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.992+0000 7fcdb3fff640 1 --2- 172.21.15.72:0/146534477 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdb41078a0 0x7fcdb419f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:21.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.992+0000 7fcdb37fe640 1 --2- 172.21.15.72:0/146534477 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdb41078a0 0x7fcdb419f520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:21.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.992+0000 7fcdb37fe640 1 -- 172.21.15.72:0/146534477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdb4104f70 msgr2=0x7fcdb41984b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:21.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.992+0000 7fcdb37fe640 1 --2- 172.21.15.72:0/146534477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdb4104f70 0x7fcdb41984b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:21.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.992+0000 7fcdb37fe640 1 -- 172.21.15.72:0/146534477 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdb419fb90 con 0x7fcdb4102640 2024-03-22T01:34:21.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.992+0000 7fcdb37fe640 1 --2- 172.21.15.72:0/146534477 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcdb4102640 0x7fcdb4197f70 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fcda400b4a0 tx=0x7fcda400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:21.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.992+0000 7fcdb17fa640 1 -- 172.21.15.72:0/146534477 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcda4014070 con 0x7fcdb4102640 2024-03-22T01:34:21.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.992+0000 7fcdb17fa640 1 -- 172.21.15.72:0/146534477 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcda4010af0 con 0x7fcdb4102640 2024-03-22T01:34:21.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.992+0000 7fcdba1f6640 1 -- 172.21.15.72:0/146534477 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdb419fe80 con 0x7fcdb4102640 2024-03-22T01:34:21.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.992+0000 7fcdb17fa640 1 -- 172.21.15.72:0/146534477 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcda4017d10 con 0x7fcdb4102640 2024-03-22T01:34:21.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.992+0000 7fcdba1f6640 1 -- 172.21.15.72:0/146534477 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdb41a0290 con 0x7fcdb4102640 2024-03-22T01:34:21.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.993+0000 7fcdba1f6640 1 -- 172.21.15.72:0/146534477 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdb40fdb40 con 0x7fcdb4102640 2024-03-22T01:34:21.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.994+0000 7fcdb17fa640 1 -- 172.21.15.72:0/146534477 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7fcda4017070 con 0x7fcdb4102640 2024-03-22T01:34:21.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.995+0000 7fcdb17fa640 1 --2- 172.21.15.72:0/146534477 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fcda00a2f20 0x7fcda00a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:21.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.995+0000 7fcdabfff640 1 --2- 172.21.15.72:0/146534477 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fcda00a2f20 0x7fcda00a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:21.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.995+0000 7fcdb17fa640 1 -- 172.21.15.72:0/146534477 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fcda40ce9d0 con 0x7fcdb4102640 2024-03-22T01:34:21.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:21.996+0000 7fcdabfff640 1 --2- 172.21.15.72:0/146534477 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fcda00a2f20 0x7fcda00a53e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c000c00 tx=0x7fcd9c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:21.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.000+0000 7fcdb17fa640 1 -- 172.21.15.72:0/146534477 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcda409b5c0 con 0x7fcdb4102640 2024-03-22T01:34:22.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.176+0000 7fcdba1f6640 1 -- 172.21.15.72:0/146534477 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcdb4062830 con 0x7fcdb4102640 2024-03-22T01:34:22.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.178+0000 7fcdb17fa640 1 -- 172.21.15.72:0/146534477 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24045 (secure 0 0 0) 0x7fcda409aec0 con 0x7fcdb4102640 2024-03-22T01:34:22.177 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:34:22.177 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":91,"fsid":"022a99bc-e7eb-11ee-95cd-87774f69a715","created":"2024-03-22T01:26:23.893672+0000","modified":"2024-03-22T01:34:19.866367+0000","last_up_change":"2024-03-22T01:33:25.223684+0000","last_in_change":"2024-03-22T01:33:05.010331+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T01:29:37.438503+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-22T01:34:12.610422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-22T01:34:13.761686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-22T01:34:17.507039+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"16c15bb0-db92-4191-9063-38b4890e81e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6803","nonce":661906791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6805","nonce":661906791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6809","nonce":661906791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6807","nonce":661906791}]},"public_addr":"172.21.15.72:6803/661906791","cluster_addr":"172.21.15.72:6805/661906791","heartbeat_back_addr":"172.21.15.72:6809/661906791","heartbeat_front_addr":"172.21.15.72:6807/661906791","state":["exists","up"]},{"osd":1,"uuid":"981290ba-97dc-496b-aa22-d51704184b10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6801","nonce":2870590550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6803","nonce":2870590550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6807","nonce":2870590550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6805","nonce":2870590550}]},"public_addr":"172.21.15.89:6801/2870590550","cluster_addr":"172.21.15.89:6803/2870590550","heartbeat_back_addr":"172.21.15.89:6807/2870590550","heartbeat_front_addr":"172.21.15.89:6805/2870590550","state":["exists","up"]},{"osd":2,"uuid":"f07ed658-f545-46a1-8edd-8b1d5d49885c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6801","nonce":3865778118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6803","nonce":3865778118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6807","nonce":3865778118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6805","nonce":3865778118}]},"public_addr":"172.21.15.98:6801/3865778118","cluster_addr":"172.21.15.98:6803/3865778118","heartbeat_back_addr":"172.21.15.98:6807/3865778118","heartbeat_front_addr":"172.21.15.98:6805/3865778118","state":["exists","up"]},{"osd":3,"uuid":"ccc4bc40-2c48-46d2-8548-0c05b44f9f1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6811","nonce":1865951395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6813","nonce":1865951395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6817","nonce":1865951395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6815","nonce":1865951395}]},"public_addr":"172.21.15.72:6811/1865951395","cluster_addr":"172.21.15.72:6813/1865951395","heartbeat_back_addr":"172.21.15.72:6817/1865951395","heartbeat_front_addr":"172.21.15.72:6815/1865951395","state":["exists","up"]},{"osd":4,"uuid":"1812439f-810a-4ed2-9395-c00c9a085a13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6809","nonce":2683760795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6811","nonce":2683760795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6815","nonce":2683760795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6813","nonce":2683760795}]},"public_addr":"172.21.15.89:6809/2683760795","cluster_addr":"172.21.15.89:6811/2683760795","heartbeat_back_addr":"172.21.15.89:6815/2683760795","heartbeat_front_addr":"172.21.15.89:6813/2683760795","state":["exists","up"]},{"osd":5,"uuid":"e78954d8-f4e3-4d58-8536-d2d59c843386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6809","nonce":2635838572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6811","nonce":2635838572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6815","nonce":2635838572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6813","nonce":2635838572}]},"public_addr":"172.21.15.98:6809/2635838572","cluster_addr":"172.21.15.98:6811/2635838572","heartbeat_back_addr":"172.21.15.98:6815/2635838572","heartbeat_front_addr":"172.21.15.98:6813/2635838572","state":["exists","up"]},{"osd":6,"uuid":"dc6c8be2-b64c-4959-8015-7b0ea6d25599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6819","nonce":3158982661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6821","nonce":3158982661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6825","nonce":3158982661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6823","nonce":3158982661}]},"public_addr":"172.21.15.72:6819/3158982661","cluster_addr":"172.21.15.72:6821/3158982661","heartbeat_back_addr":"172.21.15.72:6825/3158982661","heartbeat_front_addr":"172.21.15.72:6823/3158982661","state":["exists","up"]},{"osd":7,"uuid":"7a73469f-cf3a-4306-ae93-ea46e21ebbbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6817","nonce":3075913234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6819","nonce":3075913234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6823","nonce":3075913234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6821","nonce":3075913234}]},"public_addr":"172.21.15.89:6817/3075913234","cluster_addr":"172.21.15.89:6819/3075913234","heartbeat_back_addr":"172.21.15.89:6823/3075913234","heartbeat_front_addr":"172.21.15.89:6821/3075913234","state":["exists","up"]},{"osd":8,"uuid":"c4b13bee-ea4d-4344-95fb-2622ecbb8196","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6817","nonce":1813255577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6819","nonce":1813255577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6823","nonce":1813255577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6821","nonce":1813255577}]},"public_addr":"172.21.15.98:6817/1813255577","cluster_addr":"172.21.15.98:6819/1813255577","heartbeat_back_addr":"172.21.15.98:6823/1813255577","heartbeat_front_addr":"172.21.15.98:6821/1813255577","state":["exists","up"]},{"osd":9,"uuid":"eefb0a44-4a36-4e55-9d04-0501bf718db6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6827","nonce":3377749206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6829","nonce":3377749206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6832","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6833","nonce":3377749206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6831","nonce":3377749206}]},"public_addr":"172.21.15.72:6827/3377749206","cluster_addr":"172.21.15.72:6829/3377749206","heartbeat_back_addr":"172.21.15.72:6833/3377749206","heartbeat_front_addr":"172.21.15.72:6831/3377749206","state":["exists","up"]},{"osd":10,"uuid":"8741d25f-40dd-40d7-b040-5cc5e411eb98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6825","nonce":885058786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6827","nonce":885058786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6831","nonce":885058786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6829","nonce":885058786}]},"public_addr":"172.21.15.89:6825/885058786","cluster_addr":"172.21.15.89:6827/885058786","heartbeat_back_addr":"172.21.15.89:6831/885058786","heartbeat_front_addr":"172.21.15.89:6829/885058786","state":["exists","up"]},{"osd":11,"uuid":"8d02e957-eda4-4184-b5c6-5eb3b8b38060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6825","nonce":1517095838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6827","nonce":1517095838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6831","nonce":1517095838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6829","nonce":1517095838}]},"public_addr":"172.21.15.98:6825/1517095838","cluster_addr":"172.21.15.98:6827/1517095838","heartbeat_back_addr":"172.21.15.98:6831/1517095838","heartbeat_front_addr":"172.21.15.98:6829/1517095838","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:28:47.307652+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:11.037135+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:34.619188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:59.350738+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:30:23.787337+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:30:48.994154+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:31:13.690350+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:31:39.628116+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:04.723912+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:30.689741+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:57.229508+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:33:23.154537+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:0/121087196":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/2451857761":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/1228477995":"2024-03-23T01:27:15.339809+0000","172.21.15.72:6801/1513232913":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/3558958264":"2024-03-23T01:26:50.074871+0000","172.21.15.72:0/1510187198":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6800/3518012916":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6801/3518012916":"2024-03-23T01:26:50.074871+0000","172.21.15.72:0/1372801041":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6800/1513232913":"2024-03-23T01:27:15.339809+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T01:34:22.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.180+0000 7fcdba1f6640 1 -- 172.21.15.72:0/146534477 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fcda00a2f20 msgr2=0x7fcda00a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:22.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.180+0000 7fcdba1f6640 1 --2- 172.21.15.72:0/146534477 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fcda00a2f20 0x7fcda00a53e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c000c00 tx=0x7fcd9c005fd0 comp rx=0 tx=0).stop 2024-03-22T01:34:22.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.180+0000 7fcdba1f6640 1 -- 172.21.15.72:0/146534477 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcdb4102640 msgr2=0x7fcdb4197f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:22.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.180+0000 7fcdba1f6640 1 --2- 172.21.15.72:0/146534477 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcdb4102640 0x7fcdb4197f70 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fcda400b4a0 tx=0x7fcda400b960 comp rx=0 tx=0).stop 2024-03-22T01:34:22.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.180+0000 7fcdba1f6640 1 -- 172.21.15.72:0/146534477 shutdown_connections 2024-03-22T01:34:22.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.180+0000 7fcdba1f6640 1 --2- 172.21.15.72:0/146534477 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fcda00a2f20 0x7fcda00a53e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:22.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.180+0000 7fcdba1f6640 1 --2- 172.21.15.72:0/146534477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdb4104f70 0x7fcdb41984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:22.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.180+0000 7fcdba1f6640 1 --2- 172.21.15.72:0/146534477 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdb41078a0 0x7fcdb419f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:22.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.180+0000 7fcdba1f6640 1 --2- 172.21.15.72:0/146534477 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcdb4102640 0x7fcdb4197f70 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:22.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.180+0000 7fcdba1f6640 1 -- 172.21.15.72:0/146534477 >> 172.21.15.72:0/146534477 conn(0x7fcdb40fc050 msgr2=0x7fcdb4102ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:22.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.180+0000 7fcdba1f6640 1 -- 172.21.15.72:0/146534477 shutdown_connections 2024-03-22T01:34:22.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.180+0000 7fcdba1f6640 1 -- 172.21.15.72:0/146534477 wait complete. 2024-03-22T01:34:22.191 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T01:34:14.815677+0000', 'modified': '2024-03-22T01:34:19.866556+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24524}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24524': {'gid': 24524, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.89:6839/149200280', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6838', 'nonce': 149200280}, {'type': 'v1', 'addr': '172.21.15.89:6839', 'nonce': 149200280}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24524, 'qdb_cluster': [24524]} 2024-03-22T01:34:22.191 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-22T01:34:22.191 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-22T01:34:22.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.294+0000 7fdbe77c7640 1 -- 172.21.15.72:0/4193186996 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdbe0101780 msgr2=0x7fdbe0101c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:22.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.294+0000 7fdbe77c7640 1 --2- 172.21.15.72:0/4193186996 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdbe0101780 0x7fdbe0101c00 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fdbd8009ca0 tx=0x7fdbd802c0f0 comp rx=0 tx=0).stop 2024-03-22T01:34:22.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.294+0000 7fdbe77c7640 1 -- 172.21.15.72:0/4193186996 shutdown_connections 2024-03-22T01:34:22.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.294+0000 7fdbe77c7640 1 --2- 172.21.15.72:0/4193186996 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdbe0100de0 0x7fdbe0101240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:22.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.294+0000 7fdbe77c7640 1 --2- 172.21.15.72:0/4193186996 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdbe0101780 0x7fdbe0101c00 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:22.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.294+0000 7fdbe77c7640 1 --2- 172.21.15.72:0/4193186996 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbe0100480 0x7fdbe01008a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:22.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.294+0000 7fdbe77c7640 1 -- 172.21.15.72:0/4193186996 >> 172.21.15.72:0/4193186996 conn(0x7fdbe00fc020 msgr2=0x7fdbe00fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:22.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.294+0000 7fdbe77c7640 1 -- 172.21.15.72:0/4193186996 shutdown_connections 2024-03-22T01:34:22.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.294+0000 7fdbe77c7640 1 -- 172.21.15.72:0/4193186996 wait complete. 2024-03-22T01:34:22.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.295+0000 7fdbe77c7640 1 Processor -- start 2024-03-22T01:34:22.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.295+0000 7fdbe77c7640 1 -- start start 2024-03-22T01:34:22.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.295+0000 7fdbe77c7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbe0100480 0x7fdbe019a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:22.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.295+0000 7fdbe77c7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdbe0100de0 0x7fdbe019a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:22.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.295+0000 7fdbe77c7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdbe0101780 0x7fdbe01a1610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:22.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.295+0000 7fdbe77c7640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fdbe010fa20 con 0x7fdbe0100de0 2024-03-22T01:34:22.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.295+0000 7fdbe77c7640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fdbe010f8a0 con 0x7fdbe0101780 2024-03-22T01:34:22.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.295+0000 7fdbe77c7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdbe010fba0 con 0x7fdbe0100480 2024-03-22T01:34:22.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.295+0000 7fdbe4d62640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdbe0100de0 0x7fdbe019a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:22.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.295+0000 7fdbe4d62640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdbe0100de0 0x7fdbe019a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47856/0 (socket says 172.21.15.72:47856) 2024-03-22T01:34:22.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.295+0000 7fdbe4d62640 1 -- 172.21.15.72:0/4015338176 learned_addr learned my addr 172.21.15.72:0/4015338176 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:34:22.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.295+0000 7fdbe5d64640 1 --2- 172.21.15.72:0/4015338176 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdbe0101780 0x7fdbe01a1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:22.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.295+0000 7fdbe5563640 1 --2- 172.21.15.72:0/4015338176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbe0100480 0x7fdbe019a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:22.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.295+0000 7fdbe4d62640 1 -- 172.21.15.72:0/4015338176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbe0100480 msgr2=0x7fdbe019a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:22.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.295+0000 7fdbe4d62640 1 --2- 172.21.15.72:0/4015338176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbe0100480 0x7fdbe019a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:22.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.295+0000 7fdbe4d62640 1 -- 172.21.15.72:0/4015338176 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdbe0101780 msgr2=0x7fdbe01a1610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:22.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.295+0000 7fdbe4d62640 1 --2- 172.21.15.72:0/4015338176 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdbe0101780 0x7fdbe01a1610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:22.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.295+0000 7fdbe4d62640 1 -- 172.21.15.72:0/4015338176 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbe01a1c80 con 0x7fdbe0100de0 2024-03-22T01:34:22.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.296+0000 7fdbe4d62640 1 --2- 172.21.15.72:0/4015338176 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdbe0100de0 0x7fdbe019a5a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fdbd000ea60 tx=0x7fdbd000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:22.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.296+0000 7fdbd67fc640 1 -- 172.21.15.72:0/4015338176 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbd000cde0 con 0x7fdbe0100de0 2024-03-22T01:34:22.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.296+0000 7fdbd67fc640 1 -- 172.21.15.72:0/4015338176 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbd00035f0 con 0x7fdbe0100de0 2024-03-22T01:34:22.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.296+0000 7fdbe77c7640 1 -- 172.21.15.72:0/4015338176 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbe01a1f70 con 0x7fdbe0100de0 2024-03-22T01:34:22.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.296+0000 7fdbd67fc640 1 -- 172.21.15.72:0/4015338176 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbd0010670 con 0x7fdbe0100de0 2024-03-22T01:34:22.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.296+0000 7fdbe77c7640 1 -- 172.21.15.72:0/4015338176 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbe01a2430 con 0x7fdbe0100de0 2024-03-22T01:34:22.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.298+0000 7fdbd67fc640 1 -- 172.21.15.72:0/4015338176 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7fdbd00037a0 con 0x7fdbe0100de0 2024-03-22T01:34:22.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.298+0000 7fdbea3ac740 1 Processor -- start 2024-03-22T01:34:22.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.298+0000 7fdbea3ac740 1 -- start start 2024-03-22T01:34:22.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.298+0000 7fdbea3ac740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d2d6b61c80 0x55d2d6b620a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:22.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.298+0000 7fdbea3ac740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d2d6b62670 0x55d2d6b67170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:22.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.298+0000 7fdbea3ac740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2d6b677e0 0x55d2d6b69c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:22.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.298+0000 7fdbea3ac740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55d2d6ac6210 con 0x55d2d6b61c80 2024-03-22T01:34:22.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.298+0000 7fdbea3ac740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55d2d6a6ec80 con 0x55d2d6b62670 2024-03-22T01:34:22.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.298+0000 7fdbea3ac740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d2d6ac21c0 con 0x55d2d6b677e0 2024-03-22T01:34:22.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.298+0000 7fdbd67fc640 1 --2- 172.21.15.72:0/4015338176 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fdbcc0a2f20 0x7fdbcc0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:22.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.299+0000 7fdbe5d64640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d2d6b61c80 0x55d2d6b620a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:22.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.299+0000 7fdbe5563640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d2d6b62670 0x55d2d6b67170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:22.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.299+0000 7fdbe5d64640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d2d6b61c80 0x55d2d6b620a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47858/0 (socket says 172.21.15.72:47858) 2024-03-22T01:34:22.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.299+0000 7fdbe5d64640 1 -- 172.21.15.72:0/1265510774 learned_addr learned my addr 172.21.15.72:0/1265510774 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:34:22.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.299+0000 7fdbe5d64640 1 --2- 172.21.15.72:0/4015338176 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fdbcc0a2f20 0x7fdbcc0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:22.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.299+0000 7fdbd67fc640 1 -- 172.21.15.72:0/4015338176 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fdbd00cf990 con 0x7fdbe0100de0 2024-03-22T01:34:22.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.299+0000 7fdbe4d62640 1 --2- 172.21.15.72:0/1265510774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2d6b677e0 0x55d2d6b69c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:22.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.299+0000 7fdbe5563640 1 -- 172.21.15.72:0/1265510774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2d6b677e0 msgr2=0x55d2d6b69c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:34:22.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.299+0000 7fdbe5563640 1 --2- 172.21.15.72:0/1265510774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2d6b677e0 0x55d2d6b69c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:22.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.299+0000 7fdbe5563640 1 -- 172.21.15.72:0/1265510774 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d2d6b61c80 msgr2=0x55d2d6b620a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:34:22.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.299+0000 7fdbe5563640 1 --2- 172.21.15.72:0/1265510774 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d2d6b61c80 0x55d2d6b620a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:22.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.299+0000 7fdbe5563640 1 -- 172.21.15.72:0/1265510774 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d2d67da2d0 con 0x55d2d6b62670 2024-03-22T01:34:22.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.299+0000 7fdbe5d64640 1 --2- 172.21.15.72:0/4015338176 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fdbcc0a2f20 0x7fdbcc0a53e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fdbd8009a60 tx=0x7fdbd8032340 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:22.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.299+0000 7fdbe5563640 1 --2- 172.21.15.72:0/1265510774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d2d6b62670 0x55d2d6b67170 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fdbdc00b9e0 tx=0x7fdbdc00bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93d0d6b0183c820a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:22.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.299+0000 7fdbcb7fe640 1 -- 172.21.15.72:0/1265510774 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbdc00dda0 con 0x55d2d6b62670 2024-03-22T01:34:22.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.299+0000 7fdbcb7fe640 1 -- 172.21.15.72:0/1265510774 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbdc017900 con 0x55d2d6b62670 2024-03-22T01:34:22.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.299+0000 7fdbcb7fe640 1 -- 172.21.15.72:0/1265510774 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbdc00f9d0 con 0x55d2d6b62670 2024-03-22T01:34:22.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.300+0000 7fdbea3ac740 1 -- 172.21.15.72:0/1265510774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d2d6b62670 msgr2=0x55d2d6b67170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:22.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.300+0000 7fdbea3ac740 1 --2- 172.21.15.72:0/1265510774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d2d6b62670 0x55d2d6b67170 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fdbdc00b9e0 tx=0x7fdbdc00bea0 comp rx=0 tx=0).stop 2024-03-22T01:34:22.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.300+0000 7fdbea3ac740 1 -- 172.21.15.72:0/1265510774 shutdown_connections 2024-03-22T01:34:22.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.300+0000 7fdbea3ac740 1 --2- 172.21.15.72:0/1265510774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d2d6b62670 0x55d2d6b67170 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:22.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.300+0000 7fdbea3ac740 1 --2- 172.21.15.72:0/1265510774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2d6b677e0 0x55d2d6b69c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:22.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.300+0000 7fdbea3ac740 1 --2- 172.21.15.72:0/1265510774 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d2d6b61c80 0x55d2d6b620a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:22.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.300+0000 7fdbea3ac740 1 -- 172.21.15.72:0/1265510774 >> 172.21.15.72:0/1265510774 conn(0x55d2d6b5d1f0 msgr2=0x55d2d6b5d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:22.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.300+0000 7fdbea3ac740 1 -- 172.21.15.72:0/1265510774 shutdown_connections 2024-03-22T01:34:22.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.300+0000 7fdbea3ac740 1 -- 172.21.15.72:0/1265510774 wait complete. 2024-03-22T01:34:22.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.300+0000 7fdbea3ac740 1 -- start start 2024-03-22T01:34:22.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.300+0000 7fdbea3ac740 1 Processor -- start 2024-03-22T01:34:22.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.301+0000 7fdbea3ac740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2d6b61c80 0x55d2d6b6be60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:22.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.301+0000 7fdbea3ac740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d2d6b62670 0x55d2d6b6c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:22.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.301+0000 7fdbea3ac740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d2d6b677e0 0x55d2d6b733e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:22.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.301+0000 7fdbea3ac740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55d2d6b6a790 con 0x55d2d6b62670 2024-03-22T01:34:22.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.301+0000 7fdbea3ac740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55d2d6a81f80 con 0x55d2d6b677e0 2024-03-22T01:34:22.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.301+0000 7fdbea3ac740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d2d6b6a910 con 0x55d2d6b61c80 2024-03-22T01:34:22.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.301+0000 7fdbea3ac740 10 client.0 _ll_register_callbacks cb 0x55d2d6abe4f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:34:22.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.301+0000 7fdbe4d62640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d2d6b62670 0x55d2d6b6c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:22.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.301+0000 7fdbe4d62640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d2d6b62670 0x55d2d6b6c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47862/0 (socket says 172.21.15.72:47862) 2024-03-22T01:34:22.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.301+0000 7fdbe4d62640 1 -- 172.21.15.72:0/658604665 learned_addr learned my addr 172.21.15.72:0/658604665 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:34:22.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.301+0000 7fdbe5d64640 1 --2- 172.21.15.72:0/658604665 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d2d6b677e0 0x55d2d6b733e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:22.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.301+0000 7fdbe5563640 1 --2- 172.21.15.72:0/658604665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2d6b61c80 0x55d2d6b6be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:22.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.301+0000 7fdbe4d62640 1 -- 172.21.15.72:0/658604665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2d6b61c80 msgr2=0x55d2d6b6be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:34:22.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.301+0000 7fdbe4d62640 1 --2- 172.21.15.72:0/658604665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2d6b61c80 0x55d2d6b6be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:22.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.301+0000 7fdbe4d62640 1 -- 172.21.15.72:0/658604665 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d2d6b677e0 msgr2=0x55d2d6b733e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:34:22.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.301+0000 7fdbe4d62640 1 --2- 172.21.15.72:0/658604665 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d2d6b677e0 0x55d2d6b733e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:22.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.301+0000 7fdbe4d62640 1 -- 172.21.15.72:0/658604665 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d2d68a46b0 con 0x55d2d6b62670 2024-03-22T01:34:22.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.301+0000 7fdbe4d62640 1 --2- 172.21.15.72:0/658604665 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d2d6b62670 0x55d2d6b6c3a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fdbd0098850 tx=0x7fdbd009b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a543f2258a8fe60d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:22.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.301+0000 7fdbc8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:34:22.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.301+0000 7fdbc8ff9640 1 -- 172.21.15.72:0/658604665 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbd00b4070 con 0x55d2d6b62670 2024-03-22T01:34:22.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.301+0000 7fdbc8ff9640 1 -- 172.21.15.72:0/658604665 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fdbd00984f0 con 0x55d2d6b62670 2024-03-22T01:34:22.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.301+0000 7fdbea3ac740 1 -- 172.21.15.72:0/658604665 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdbdc017900 con 0x55d2d6b62670 2024-03-22T01:34:22.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.302+0000 7fdbc8ff9640 1 -- 172.21.15.72:0/658604665 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7fdbd00bc070 con 0x55d2d6b62670 2024-03-22T01:34:22.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.302+0000 7fdbea3ac740 10 client.14751 fetch_fsmap learned FSMap version 17 2024-03-22T01:34:22.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.302+0000 7fdbea3ac740 1 -- 172.21.15.72:0/658604665 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x55d2d6b76820 con 0x55d2d6b62670 2024-03-22T01:34:22.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.302+0000 7fdbc8ff9640 1 -- 172.21.15.72:0/658604665 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbdc017900 con 0x55d2d6b62670 2024-03-22T01:34:22.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.302+0000 7fdbc8ff9640 1 -- 172.21.15.72:0/658604665 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7fdbd0063030 con 0x55d2d6b62670 2024-03-22T01:34:22.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.302+0000 7fdbea3ac740 10 client.14751 fetch_fsmap finished waiting for FSMap version 17 2024-03-22T01:34:22.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.302+0000 7fdbea3ac740 10 client.14751 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-22T01:34:22.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.302+0000 7fdbea3ac740 20 client.14751 populate_metadata read hostname 'smithi072' 2024-03-22T01:34:22.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.302+0000 7fdbea3ac740 1 --2- 172.21.15.72:0/658604665 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55d2d6b78f00 0x55d2d6b7a370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:22.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.302+0000 7fdbea3ac740 4 client.14751 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T01:34:22.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.302+0000 7fdbea3ac740 1 -- 172.21.15.72:0/658604665 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d2d6ad3750 con 0x55d2d6b78f00 2024-03-22T01:34:22.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.302+0000 7fdbe5563640 1 --2- 172.21.15.72:0/658604665 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55d2d6b78f00 0x55d2d6b7a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:22.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.302+0000 7fdbc8ff9640 10 client.14751 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:34:22.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.302+0000 7fdbe5563640 1 --2- 172.21.15.72:0/658604665 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55d2d6b78f00 0x55d2d6b7a370 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ed2f2221ae26937 server_cookie=c712ad399a0ddbcf in_seq=0 out_seq=0 2024-03-22T01:34:22.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.303+0000 7fdbc8ff9640 1 -- 172.21.15.72:0/658604665 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d2d6ad3750 con 0x55d2d6b78f00 2024-03-22T01:34:22.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.303+0000 7fdbc8ff9640 10 client.14751 handle_command_reply: tid=0 2024-03-22T01:34:22.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.303+0000 7fdbea3ac740 1 client.14751 shutdown 2024-03-22T01:34:22.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.303+0000 7fdbea3ac740 1 -- 172.21.15.72:0/658604665 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d2d6b62670 msgr2=0x55d2d6b6c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:22.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.303+0000 7fdbea3ac740 1 --2- 172.21.15.72:0/658604665 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d2d6b62670 0x55d2d6b6c3a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fdbd0098850 tx=0x7fdbd009b8a0 comp rx=0 tx=0).stop 2024-03-22T01:34:22.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.303+0000 7fdbea3ac740 1 -- 172.21.15.72:0/658604665 shutdown_connections 2024-03-22T01:34:22.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.303+0000 7fdbea3ac740 1 --2- 172.21.15.72:0/658604665 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55d2d6b78f00 0x55d2d6b7a370 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:22.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.303+0000 7fdbea3ac740 1 --2- 172.21.15.72:0/658604665 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d2d6b677e0 0x55d2d6b733e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:22.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.303+0000 7fdbea3ac740 1 --2- 172.21.15.72:0/658604665 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d2d6b62670 0x55d2d6b6c3a0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:22.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.303+0000 7fdbc8ff9640 0 client.14751 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:34:22.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.303+0000 7fdbea3ac740 1 --2- 172.21.15.72:0/658604665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2d6b61c80 0x55d2d6b6be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:22.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.303+0000 7fdbea3ac740 1 -- 172.21.15.72:0/658604665 >> 172.21.15.72:0/658604665 conn(0x55d2d6b5d1f0 msgr2=0x55d2d6b5c4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:22.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.303+0000 7fdbea3ac740 1 -- 172.21.15.72:0/658604665 shutdown_connections 2024-03-22T01:34:22.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.303+0000 7fdbea3ac740 1 -- 172.21.15.72:0/658604665 wait complete. 2024-03-22T01:34:22.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.304+0000 7fdbea3ac740 20 client.14751 trim_cache size 0 max 16384 2024-03-22T01:34:22.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.321+0000 7fdbea3ac740 1 Processor -- start 2024-03-22T01:34:22.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.321+0000 7fdbea3ac740 1 -- start start 2024-03-22T01:34:22.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.321+0000 7fdbea3ac740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d2d6b61c80 0x55d2d6b66e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:22.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.321+0000 7fdbea3ac740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2d6b62670 0x55d2d6b733a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:22.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.321+0000 7fdbea3ac740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d2d6b677e0 0x55d2d6b63110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:22.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.321+0000 7fdbea3ac740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55d2d6b786b0 con 0x55d2d6b61c80 2024-03-22T01:34:22.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.321+0000 7fdbea3ac740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55d2d6b78d80 con 0x55d2d6b677e0 2024-03-22T01:34:22.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.321+0000 7fdbea3ac740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d2d6b79450 con 0x55d2d6b62670 2024-03-22T01:34:22.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.321+0000 7fdbe5563640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d2d6b61c80 0x55d2d6b66e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:22.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.321+0000 7fdbe5563640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d2d6b61c80 0x55d2d6b66e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47878/0 (socket says 172.21.15.72:47878) 2024-03-22T01:34:22.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.321+0000 7fdbe5563640 1 -- 172.21.15.72:0/3259885372 learned_addr learned my addr 172.21.15.72:0/3259885372 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:34:22.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.322+0000 7fdbe5d64640 1 --2- 172.21.15.72:0/3259885372 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d2d6b677e0 0x55d2d6b63110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:22.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.322+0000 7fdbe5563640 1 -- 172.21.15.72:0/3259885372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2d6b62670 msgr2=0x55d2d6b733a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:34:22.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.322+0000 7fdbe5563640 1 --2- 172.21.15.72:0/3259885372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2d6b62670 0x55d2d6b733a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:22.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.322+0000 7fdbe5563640 1 -- 172.21.15.72:0/3259885372 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d2d6b677e0 msgr2=0x55d2d6b63110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:34:22.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.322+0000 7fdbe5563640 1 --2- 172.21.15.72:0/3259885372 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d2d6b677e0 0x55d2d6b63110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:22.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.322+0000 7fdbe5563640 1 -- 172.21.15.72:0/3259885372 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d2d6b6cfe0 con 0x55d2d6b61c80 2024-03-22T01:34:22.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.322+0000 7fdbe5563640 1 --2- 172.21.15.72:0/3259885372 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d2d6b61c80 0x55d2d6b66e00 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fdbdc019040 tx=0x7fdbdc00b4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9395e8f3df8c9576 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:22.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.322+0000 7fdbcaffd640 1 -- 172.21.15.72:0/3259885372 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbdc032070 con 0x55d2d6b61c80 2024-03-22T01:34:22.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.322+0000 7fdbcaffd640 1 -- 172.21.15.72:0/3259885372 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbdc005cc0 con 0x55d2d6b61c80 2024-03-22T01:34:22.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.322+0000 7fdbcaffd640 1 -- 172.21.15.72:0/3259885372 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbdc0049e0 con 0x55d2d6b61c80 2024-03-22T01:34:22.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.323+0000 7fdbea3ac740 1 -- 172.21.15.72:0/3259885372 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d2d6b61c80 msgr2=0x55d2d6b66e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:22.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.323+0000 7fdbea3ac740 1 --2- 172.21.15.72:0/3259885372 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d2d6b61c80 0x55d2d6b66e00 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fdbdc019040 tx=0x7fdbdc00b4c0 comp rx=0 tx=0).stop 2024-03-22T01:34:22.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.323+0000 7fdbea3ac740 1 -- 172.21.15.72:0/3259885372 shutdown_connections 2024-03-22T01:34:22.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.323+0000 7fdbea3ac740 1 --2- 172.21.15.72:0/3259885372 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d2d6b677e0 0x55d2d6b63110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:22.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.323+0000 7fdbea3ac740 1 --2- 172.21.15.72:0/3259885372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2d6b62670 0x55d2d6b733a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:22.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.323+0000 7fdbea3ac740 1 --2- 172.21.15.72:0/3259885372 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d2d6b61c80 0x55d2d6b66e00 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:22.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.323+0000 7fdbea3ac740 1 -- 172.21.15.72:0/3259885372 >> 172.21.15.72:0/3259885372 conn(0x55d2d6b5d1f0 msgr2=0x55d2d6b60ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:22.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.323+0000 7fdbea3ac740 1 -- 172.21.15.72:0/3259885372 shutdown_connections 2024-03-22T01:34:22.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.323+0000 7fdbea3ac740 1 -- 172.21.15.72:0/3259885372 wait complete. 2024-03-22T01:34:22.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.323+0000 7fdbea3ac740 1 -- start start 2024-03-22T01:34:22.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.323+0000 7fdbea3ac740 1 Processor -- start 2024-03-22T01:34:22.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.323+0000 7fdbea3ac740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d2d6b61c80 0x55d2d6b6f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:22.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.323+0000 7fdbea3ac740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2d6b62670 0x55d2d6b6fb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:22.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.323+0000 7fdbea3ac740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d2d6b677e0 0x55d2d6b72540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:22.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.323+0000 7fdbea3ac740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55d2d6b7a1f0 con 0x55d2d6b61c80 2024-03-22T01:34:22.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.323+0000 7fdbea3ac740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55d2d6b79b20 con 0x55d2d6b677e0 2024-03-22T01:34:22.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.323+0000 7fdbea3ac740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d2d6b7a8c0 con 0x55d2d6b62670 2024-03-22T01:34:22.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.323+0000 7fdbea3ac740 10 client.0 _ll_register_callbacks cb 0x7fdbd000e9b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:34:22.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.323+0000 7fdbe5563640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d2d6b61c80 0x55d2d6b6f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:22.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.323+0000 7fdbe5563640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d2d6b61c80 0x55d2d6b6f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47888/0 (socket says 172.21.15.72:47888) 2024-03-22T01:34:22.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.323+0000 7fdbe5563640 1 -- 172.21.15.72:0/3210178641 learned_addr learned my addr 172.21.15.72:0/3210178641 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:34:22.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.324+0000 7fdbe5d64640 1 --2- 172.21.15.72:0/3210178641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d2d6b677e0 0x55d2d6b72540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:22.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.324+0000 7fdbe5563640 1 -- 172.21.15.72:0/3210178641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2d6b62670 msgr2=0x55d2d6b6fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:34:22.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.324+0000 7fdbe5563640 1 --2- 172.21.15.72:0/3210178641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2d6b62670 0x55d2d6b6fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:22.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.324+0000 7fdbe5563640 1 -- 172.21.15.72:0/3210178641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d2d6b677e0 msgr2=0x55d2d6b72540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:34:22.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.324+0000 7fdbe5563640 1 --2- 172.21.15.72:0/3210178641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d2d6b677e0 0x55d2d6b72540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:22.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.324+0000 7fdbe5563640 1 -- 172.21.15.72:0/3210178641 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d2d6b72b30 con 0x55d2d6b61c80 2024-03-22T01:34:22.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.324+0000 7fdbe5d64640 1 --2- 172.21.15.72:0/3210178641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d2d6b677e0 0x55d2d6b72540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:34:22.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.324+0000 7fdbe5563640 1 --2- 172.21.15.72:0/3210178641 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d2d6b61c80 0x55d2d6b6f5d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fdbdc00b3e0 tx=0x7fdbdc00b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6284274d1a51708e server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:22.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.324+0000 7fdbc8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:34:22.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.324+0000 7fdbc8ff9640 1 -- 172.21.15.72:0/3210178641 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbdc032070 con 0x55d2d6b61c80 2024-03-22T01:34:22.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.324+0000 7fdbc8ff9640 1 -- 172.21.15.72:0/3210178641 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fdbdc00f630 con 0x55d2d6b61c80 2024-03-22T01:34:22.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.324+0000 7fdbea3ac740 1 -- 172.21.15.72:0/3210178641 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdbdc005cc0 con 0x55d2d6b61c80 2024-03-22T01:34:22.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.324+0000 7fdbc8ff9640 1 -- 172.21.15.72:0/3210178641 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7fdbdc00f830 con 0x55d2d6b61c80 2024-03-22T01:34:22.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.324+0000 7fdbea3ac740 10 client.14757 fetch_fsmap learned FSMap version 17 2024-03-22T01:34:22.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.324+0000 7fdbea3ac740 1 -- 172.21.15.72:0/3210178641 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x55d2d6bdf120 con 0x55d2d6b61c80 2024-03-22T01:34:22.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.324+0000 7fdbc8ff9640 1 -- 172.21.15.72:0/3210178641 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbdc005cc0 con 0x55d2d6b61c80 2024-03-22T01:34:22.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.324+0000 7fdbc8ff9640 1 -- 172.21.15.72:0/3210178641 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7fdbdc024bf0 con 0x55d2d6b61c80 2024-03-22T01:34:22.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.324+0000 7fdbea3ac740 10 client.14757 fetch_fsmap finished waiting for FSMap version 17 2024-03-22T01:34:22.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.325+0000 7fdbea3ac740 10 client.14757 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-22T01:34:22.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.325+0000 7fdbea3ac740 20 client.14757 populate_metadata read hostname 'smithi072' 2024-03-22T01:34:22.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.325+0000 7fdbea3ac740 1 --2- 172.21.15.72:0/3210178641 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55d2d6be3650 0x55d2d6be2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:22.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.325+0000 7fdbea3ac740 4 client.14757 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-22T01:34:22.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.325+0000 7fdbea3ac740 1 -- 172.21.15.72:0/3210178641 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "status"}) -- 0x55d2d6acdaa0 con 0x55d2d6be3650 2024-03-22T01:34:22.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.325+0000 7fdbe4d62640 1 --2- 172.21.15.72:0/3210178641 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55d2d6be3650 0x55d2d6be2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:22.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.325+0000 7fdbc8ff9640 10 client.14757 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:34:22.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.325+0000 7fdbe4d62640 1 --2- 172.21.15.72:0/3210178641 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55d2d6be3650 0x55d2d6be2360 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=65d6c31a55ec6438 server_cookie=1e5965d384f30a6b in_seq=0 out_seq=0 2024-03-22T01:34:22.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.325+0000 7fdbc8ff9640 1 -- 172.21.15.72:0/3210178641 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55d2d6acdaa0 con 0x55d2d6be3650 2024-03-22T01:34:22.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.325+0000 7fdbc8ff9640 10 client.14757 handle_command_reply: tid=0 2024-03-22T01:34:22.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.325+0000 7fdbea3ac740 1 client.14757 shutdown 2024-03-22T01:34:22.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.325+0000 7fdbea3ac740 1 -- 172.21.15.72:0/3210178641 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d2d6b61c80 msgr2=0x55d2d6b6f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:22.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.325+0000 7fdbea3ac740 1 --2- 172.21.15.72:0/3210178641 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d2d6b61c80 0x55d2d6b6f5d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fdbdc00b3e0 tx=0x7fdbdc00b5b0 comp rx=0 tx=0).stop 2024-03-22T01:34:22.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.325+0000 7fdbea3ac740 1 -- 172.21.15.72:0/3210178641 shutdown_connections 2024-03-22T01:34:22.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.325+0000 7fdbea3ac740 1 --2- 172.21.15.72:0/3210178641 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55d2d6be3650 0x55d2d6be2360 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:22.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.325+0000 7fdbea3ac740 1 --2- 172.21.15.72:0/3210178641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d2d6b677e0 0x55d2d6b72540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:22.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.325+0000 7fdbc8ff9640 0 client.14757 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:34:22.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.325+0000 7fdbea3ac740 1 --2- 172.21.15.72:0/3210178641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2d6b62670 0x55d2d6b6fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:22.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.325+0000 7fdbea3ac740 1 --2- 172.21.15.72:0/3210178641 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d2d6b61c80 0x55d2d6b6f5d0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:22.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.325+0000 7fdbea3ac740 1 -- 172.21.15.72:0/3210178641 >> 172.21.15.72:0/3210178641 conn(0x55d2d6b5d1f0 msgr2=0x55d2d6b60ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:22.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.325+0000 7fdbea3ac740 1 -- 172.21.15.72:0/3210178641 shutdown_connections 2024-03-22T01:34:22.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.325+0000 7fdbea3ac740 1 -- 172.21.15.72:0/3210178641 wait complete. 2024-03-22T01:34:22.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.326+0000 7fdbea3ac740 20 client.14757 trim_cache size 0 max 16384 2024-03-22T01:34:22.335 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-22T01:34:22.336 INFO:teuthology.orchestra.run.smithi072.stdout: "cluster_fsid": "022a99bc-e7eb-11ee-95cd-87774f69a715", 2024-03-22T01:34:22.336 INFO:teuthology.orchestra.run.smithi072.stdout: "whoami": 0, 2024-03-22T01:34:22.336 INFO:teuthology.orchestra.run.smithi072.stdout: "id": 24524, 2024-03-22T01:34:22.336 INFO:teuthology.orchestra.run.smithi072.stdout: "want_state": "up:active", 2024-03-22T01:34:22.336 INFO:teuthology.orchestra.run.smithi072.stdout: "state": "up:active", 2024-03-22T01:34:22.336 INFO:teuthology.orchestra.run.smithi072.stdout: "fs_name": "cephfs", 2024-03-22T01:34:22.336 INFO:teuthology.orchestra.run.smithi072.stdout: "rank_uptime": 6.8454772850000003, 2024-03-22T01:34:22.336 INFO:teuthology.orchestra.run.smithi072.stdout: "mdsmap_epoch": 17, 2024-03-22T01:34:22.336 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_epoch": 91, 2024-03-22T01:34:22.336 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_epoch_barrier": 91, 2024-03-22T01:34:22.336 INFO:teuthology.orchestra.run.smithi072.stdout: "uptime": 13.427156672000001 2024-03-22T01:34:22.336 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-22T01:34:22.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.326+0000 7fdbe77c7640 1 -- 172.21.15.72:0/4015338176 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fdbcc0a2f20 msgr2=0x7fdbcc0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:22.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.326+0000 7fdbe77c7640 1 --2- 172.21.15.72:0/4015338176 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fdbcc0a2f20 0x7fdbcc0a53e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fdbd8009a60 tx=0x7fdbd8032340 comp rx=0 tx=0).stop 2024-03-22T01:34:22.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.326+0000 7fdbe77c7640 1 -- 172.21.15.72:0/4015338176 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdbe0100de0 msgr2=0x7fdbe019a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:22.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.326+0000 7fdbe77c7640 1 --2- 172.21.15.72:0/4015338176 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdbe0100de0 0x7fdbe019a5a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fdbd000ea60 tx=0x7fdbd000ef20 comp rx=0 tx=0).stop 2024-03-22T01:34:22.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.326+0000 7fdbe77c7640 1 -- 172.21.15.72:0/4015338176 shutdown_connections 2024-03-22T01:34:22.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.326+0000 7fdbe77c7640 1 --2- 172.21.15.72:0/4015338176 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fdbcc0a2f20 0x7fdbcc0a53e0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:22.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.326+0000 7fdbe77c7640 1 --2- 172.21.15.72:0/4015338176 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdbe0101780 0x7fdbe01a1610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:22.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.326+0000 7fdbe77c7640 1 --2- 172.21.15.72:0/4015338176 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdbe0100de0 0x7fdbe019a5a0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:22.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.326+0000 7fdbe77c7640 1 --2- 172.21.15.72:0/4015338176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbe0100480 0x7fdbe019a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:22.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.326+0000 7fdbe77c7640 1 -- 172.21.15.72:0/4015338176 >> 172.21.15.72:0/4015338176 conn(0x7fdbe00fc020 msgr2=0x7fdbe00fdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:22.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.326+0000 7fdbe77c7640 1 -- 172.21.15.72:0/4015338176 shutdown_connections 2024-03-22T01:34:22.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:22.326+0000 7fdbe77c7640 1 -- 172.21.15.72:0/4015338176 wait complete. 2024-03-22T01:34:22.339 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-22T01:34:22.339 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-22T01:34:22.339 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph fs ls 2024-03-22T01:34:22.577 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:34:22.829 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:22 smithi072 ceph-mon[32414]: pgmap v294: 129 pgs: 3 creating+peering, 106 active+clean, 20 unknown; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 2 op/s 2024-03-22T01:34:22.829 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:22 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/1417488519' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:34:22.829 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:22 smithi072 ceph-mon[32414]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-22T01:34:22.829 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:22 smithi072 ceph-mon[32414]: Cluster is now healthy 2024-03-22T01:34:22.829 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:22 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/3657003804' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:34:22.829 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:22 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/146534477' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:34:22.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:22 smithi089 ceph-mon[35770]: pgmap v294: 129 pgs: 3 creating+peering, 106 active+clean, 20 unknown; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 2 op/s 2024-03-22T01:34:22.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:22 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/1417488519' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:34:22.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:22 smithi089 ceph-mon[35770]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-22T01:34:22.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:22 smithi089 ceph-mon[35770]: Cluster is now healthy 2024-03-22T01:34:22.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:22 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/3657003804' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:34:22.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:22 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/146534477' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:34:22.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:22 smithi098 ceph-mon[36468]: pgmap v294: 129 pgs: 3 creating+peering, 106 active+clean, 20 unknown; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 2 op/s 2024-03-22T01:34:22.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:22 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/1417488519' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:34:22.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:22 smithi098 ceph-mon[36468]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-22T01:34:22.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:22 smithi098 ceph-mon[36468]: Cluster is now healthy 2024-03-22T01:34:22.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:22 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/3657003804' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:34:22.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:22 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/146534477' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:34:24.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:24.262+0000 7f726e3df700 1 -- 172.21.15.72:0/2694850542 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7268108630 msgr2=0x7f7268108a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:24.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:24.262+0000 7f726e3df700 1 --2- 172.21.15.72:0/2694850542 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7268108630 0x7f7268108a10 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f72580099c0 tx=0x7f7258009cd0 comp rx=0 tx=0).stop 2024-03-22T01:34:24.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:24.262+0000 7f726e3df700 1 -- 172.21.15.72:0/2694850542 shutdown_connections 2024-03-22T01:34:24.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:24.262+0000 7f726e3df700 1 --2- 172.21.15.72:0/2694850542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f726806b580 0x7f726810f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:24.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:24.262+0000 7f726e3df700 1 --2- 172.21.15.72:0/2694850542 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7268108630 0x7f7268108a10 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:24.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:24.262+0000 7f726e3df700 1 --2- 172.21.15.72:0/2694850542 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f726806abc0 0x7f726806b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:24.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:24.262+0000 7f726e3df700 1 -- 172.21.15.72:0/2694850542 >> 172.21.15.72:0/2694850542 conn(0x7f72680775d0 msgr2=0x7f72680779e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:24.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:24.262+0000 7f726e3df700 1 -- 172.21.15.72:0/2694850542 shutdown_connections 2024-03-22T01:34:24.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:24.262+0000 7f726e3df700 1 -- 172.21.15.72:0/2694850542 wait complete. 2024-03-22T01:34:24.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:24.263+0000 7f726e3df700 1 Processor -- start 2024-03-22T01:34:24.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:24.263+0000 7f726e3df700 1 -- start start 2024-03-22T01:34:24.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:24.263+0000 7f726e3df700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f726806abc0 0x7f7268198e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:24.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:24.263+0000 7f726e3df700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f726806b580 0x7f72681993c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:24.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:24.263+0000 7f726e3df700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7268108630 0x7f726819d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:24.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:24.263+0000 7f726e3df700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f7268112970 con 0x7f726806abc0 2024-03-22T01:34:24.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:24.263+0000 7f726e3df700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f72681127f0 con 0x7f7268108630 2024-03-22T01:34:24.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:24.263+0000 7f726e3df700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7268112af0 con 0x7f726806b580 2024-03-22T01:34:24.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:24.263+0000 7f7267fff700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f726806abc0 0x7f7268198e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:24.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:24.264+0000 7f7267fff700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f726806abc0 0x7f7268198e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47908/0 (socket says 172.21.15.72:47908) 2024-03-22T01:34:24.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:24.264+0000 7f7267fff700 1 -- 172.21.15.72:0/3696442332 learned_addr learned my addr 172.21.15.72:0/3696442332 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:34:24.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:24.264+0000 7f726c97c700 1 --2- 172.21.15.72:0/3696442332 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7268108630 0x7f726819d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:24.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:24.264+0000 7f72677fe700 1 --2- 172.21.15.72:0/3696442332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f726806b580 0x7f72681993c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:24.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:24.264+0000 7f726c97c700 1 -- 172.21.15.72:0/3696442332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f726806b580 msgr2=0x7f72681993c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:24.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:24.264+0000 7f726c97c700 1 --2- 172.21.15.72:0/3696442332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f726806b580 0x7f72681993c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:24.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:24.264+0000 7f726c97c700 1 -- 172.21.15.72:0/3696442332 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f726806abc0 msgr2=0x7f7268198e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:24.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:24.264+0000 7f726c97c700 1 --2- 172.21.15.72:0/3696442332 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f726806abc0 0x7f7268198e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:24.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:24.264+0000 7f726c97c700 1 -- 172.21.15.72:0/3696442332 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f726819de90 con 0x7f7268108630 2024-03-22T01:34:24.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:24.264+0000 7f7267fff700 1 --2- 172.21.15.72:0/3696442332 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f726806abc0 0x7f7268198e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:34:24.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:24.264+0000 7f726c97c700 1 --2- 172.21.15.72:0/3696442332 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7268108630 0x7f726819d750 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7260007f10 tx=0x7f726000d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:24.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:24.264+0000 7f72657fa700 1 -- 172.21.15.72:0/3696442332 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f726000ddd0 con 0x7f7268108630 2024-03-22T01:34:24.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:24.264+0000 7f726e3df700 1 -- 172.21.15.72:0/3696442332 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f726819e180 con 0x7f7268108630 2024-03-22T01:34:24.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:24.264+0000 7f72657fa700 1 -- 172.21.15.72:0/3696442332 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7260009e20 con 0x7f7268108630 2024-03-22T01:34:24.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:24.264+0000 7f72657fa700 1 -- 172.21.15.72:0/3696442332 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7260015660 con 0x7f7268108630 2024-03-22T01:34:24.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:24.264+0000 7f726e3df700 1 -- 172.21.15.72:0/3696442332 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72681a5a00 con 0x7f7268108630 2024-03-22T01:34:24.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:24.265+0000 7f726e3df700 1 -- 172.21.15.72:0/3696442332 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7268069340 con 0x7f7268108630 2024-03-22T01:34:24.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:24.266+0000 7f72657fa700 1 -- 172.21.15.72:0/3696442332 <== mon.2 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f7260004ad0 con 0x7f7268108630 2024-03-22T01:34:24.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:24.267+0000 7f72657fa700 1 --2- 172.21.15.72:0/3696442332 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f72500a2fb0 0x7f72500a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:24.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:24.267+0000 7f7267fff700 1 --2- 172.21.15.72:0/3696442332 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f72500a2fb0 0x7f72500a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:24.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:24.267+0000 7f72657fa700 1 -- 172.21.15.72:0/3696442332 <== mon.2 v2:172.21.15.89:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f72600cf8b0 con 0x7f7268108630 2024-03-22T01:34:24.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:24.267+0000 7f7267fff700 1 --2- 172.21.15.72:0/3696442332 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f72500a2fb0 0x7f72500a5470 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f7258009990 tx=0x7f7258005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:24.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:24.271+0000 7f72657fa700 1 -- 172.21.15.72:0/3696442332 <== mon.2 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7260010070 con 0x7f7268108630 2024-03-22T01:34:24.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:24.510+0000 7f726e3df700 1 -- 172.21.15.72:0/3696442332 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f726802dd70 con 0x7f7268108630 2024-03-22T01:34:24.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:24.511+0000 7f72657fa700 1 -- 172.21.15.72:0/3696442332 <== mon.2 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+88 (secure 0 0 0) 0x7f726009c440 con 0x7f7268108630 2024-03-22T01:34:24.510 INFO:teuthology.orchestra.run.smithi072.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-22T01:34:24.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:24.513+0000 7f726e3df700 1 -- 172.21.15.72:0/3696442332 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f72500a2fb0 msgr2=0x7f72500a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:24.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:24.513+0000 7f726e3df700 1 --2- 172.21.15.72:0/3696442332 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f72500a2fb0 0x7f72500a5470 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f7258009990 tx=0x7f7258005960 comp rx=0 tx=0).stop 2024-03-22T01:34:24.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:24.513+0000 7f726e3df700 1 -- 172.21.15.72:0/3696442332 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7268108630 msgr2=0x7f726819d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:24.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:24.513+0000 7f726e3df700 1 --2- 172.21.15.72:0/3696442332 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7268108630 0x7f726819d750 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7260007f10 tx=0x7f726000d3b0 comp rx=0 tx=0).stop 2024-03-22T01:34:24.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:24.513+0000 7f726e3df700 1 -- 172.21.15.72:0/3696442332 shutdown_connections 2024-03-22T01:34:24.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:24.513+0000 7f726e3df700 1 --2- 172.21.15.72:0/3696442332 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f72500a2fb0 0x7f72500a5470 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:24.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:24.513+0000 7f726e3df700 1 --2- 172.21.15.72:0/3696442332 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7268108630 0x7f726819d750 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:24.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:24.513+0000 7f726e3df700 1 --2- 172.21.15.72:0/3696442332 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f726806abc0 0x7f7268198e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:24.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:24.513+0000 7f726e3df700 1 --2- 172.21.15.72:0/3696442332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f726806b580 0x7f72681993c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:24.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:24.513+0000 7f726e3df700 1 -- 172.21.15.72:0/3696442332 >> 172.21.15.72:0/3696442332 conn(0x7f72680775d0 msgr2=0x7f7268100350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:24.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:24.513+0000 7f726e3df700 1 -- 172.21.15.72:0/3696442332 shutdown_connections 2024-03-22T01:34:24.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:24.513+0000 7f726e3df700 1 -- 172.21.15.72:0/3696442332 wait complete. 2024-03-22T01:34:24.833 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:24 smithi072 ceph-mon[32414]: pgmap v295: 129 pgs: 3 creating+peering, 126 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 691 B/s wr, 1 op/s 2024-03-22T01:34:24.833 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:24 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/3696442332' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-22T01:34:24.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:24 smithi089 ceph-mon[35770]: pgmap v295: 129 pgs: 3 creating+peering, 126 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 691 B/s wr, 1 op/s 2024-03-22T01:34:24.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:24 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/3696442332' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-22T01:34:24.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:24 smithi098 ceph-mon[36468]: pgmap v295: 129 pgs: 3 creating+peering, 126 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 691 B/s wr, 1 op/s 2024-03-22T01:34:24.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:24 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/3696442332' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-22T01:34:25.331 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-22T01:34:25.331 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-22T01:34:25.331 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi072.front.sepia.ceph.com 2024-03-22T01:34:25.331 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-03-22T01:34:25.331 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-22T01:34:25.331 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-22T01:34:25.331 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-22T01:34:25.331 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-22T01:34:25.331 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-03-22T01:34:25.331 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T01:34:25.332 DEBUG:teuthology.orchestra.run.smithi072:> ip addr 2024-03-22T01:34:25.347 INFO:teuthology.orchestra.run.smithi072.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-22T01:34:25.348 INFO:teuthology.orchestra.run.smithi072.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-22T01:34:25.348 INFO:teuthology.orchestra.run.smithi072.stdout: inet 127.0.0.1/8 scope host lo 2024-03-22T01:34:25.348 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-22T01:34:25.348 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 ::1/128 scope host 2024-03-22T01:34:25.348 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-22T01:34:25.348 INFO:teuthology.orchestra.run.smithi072.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-22T01:34:25.348 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:d8:da:f4 brd ff:ff:ff:ff:ff:ff 2024-03-22T01:34:25.349 INFO:teuthology.orchestra.run.smithi072.stdout: altname enp7s0f0 2024-03-22T01:34:25.349 INFO:teuthology.orchestra.run.smithi072.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-22T01:34:25.349 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:d8:da:f5 brd ff:ff:ff:ff:ff:ff 2024-03-22T01:34:25.349 INFO:teuthology.orchestra.run.smithi072.stdout: altname enp7s0f1 2024-03-22T01:34:25.349 INFO:teuthology.orchestra.run.smithi072.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-22T01:34:25.349 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:8f:b7:64 brd ff:ff:ff:ff:ff:ff 2024-03-22T01:34:25.349 INFO:teuthology.orchestra.run.smithi072.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-22T01:34:25.349 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:8f:b7:65 brd ff:ff:ff:ff:ff:ff 2024-03-22T01:34:25.350 INFO:teuthology.orchestra.run.smithi072.stdout: inet 172.21.15.72/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-22T01:34:25.350 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft 42198sec preferred_lft 42198sec 2024-03-22T01:34:25.350 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::ec4:7aff:fe8f:b765/64 scope link 2024-03-22T01:34:25.350 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-22T01:34:25.350 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-22T01:34:25.351 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-22T01:34:25.351 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-03-22T01:34:25.351 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link add name ceph-brx type bridge 2024-03-22T01:34:25.351 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip addr flush dev ceph-brx 2024-03-22T01:34:25.351 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link set ceph-brx up 2024-03-22T01:34:25.351 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-22T01:34:25.351 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-03-22T01:34:25.436 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:25 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-22T01:34:25.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-22T01:34:25.558 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T01:34:25.559 DEBUG:teuthology.orchestra.run.smithi072:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-22T01:34:25.590 INFO:teuthology.orchestra.run.smithi072.stdout:1 2024-03-22T01:34:25.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T01:34:25.592 DEBUG:teuthology.orchestra.run.smithi072:> route 2024-03-22T01:34:25.663 INFO:teuthology.orchestra.run.smithi072.stdout:Kernel IP routing table 2024-03-22T01:34:25.663 INFO:teuthology.orchestra.run.smithi072.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-22T01:34:25.663 INFO:teuthology.orchestra.run.smithi072.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-22T01:34:25.663 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-22T01:34:25.663 INFO:teuthology.orchestra.run.smithi072.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-22T01:34:25.664 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-22T01:34:25.664 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-03-22T01:34:25.664 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-22T01:34:25.664 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-22T01:34:25.664 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-22T01:34:25.664 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-03-22T01:34:25.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:25 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-22T01:34:25.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-22T01:34:25.973 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T01:34:25.973 DEBUG:teuthology.orchestra.run.smithi072:> ip netns list 2024-03-22T01:34:25.988 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T01:34:25.988 DEBUG:teuthology.orchestra.run.smithi072:> ip netns list-id 2024-03-22T01:34:26.043 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-22T01:34:26.043 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-03-22T01:34:26.043 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-22T01:34:26.043 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-03-22T01:34:26.043 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-03-22T01:34:26.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:26 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-22T01:34:26.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-22T01:34:26.166 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-03-22T01:34:26.167 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-22T01:34:26.167 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-03-22T01:34:26.167 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-03-22T01:34:26.167 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-22T01:34:26.167 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-03-22T01:34:26.167 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-03-22T01:34:26.167 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-03-22T01:34:26.167 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-03-22T01:34:26.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:26 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-22T01:34:26.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-22T01:34:26.307 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-22T01:34:26.307 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-03-22T01:34:26.308 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link set brx.0 up 2024-03-22T01:34:26.308 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link set dev brx.0 master ceph-brx 2024-03-22T01:34:26.308 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-03-22T01:34:26.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:26 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-22T01:34:26.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-22T01:34:26.391 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-03-22T01:34:26.391 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-22T01:34:26.391 DEBUG:teuthology.orchestra.run.smithi072:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-03-22T01:34:26.405 INFO:teuthology.orchestra.run.smithi072.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-03-22T01:34:26.406 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-22T01:34:26.406 DEBUG:teuthology.orchestra.run.smithi072:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-03-22T01:34:26.460 DEBUG:teuthology.orchestra.run.smithi072:> sudo modprobe fuse 2024-03-22T01:34:26.530 DEBUG:teuthology.orchestra.run.smithi072:> cat /proc/self/mounts | awk '{print $2}' 2024-03-22T01:34:26.585 INFO:teuthology.orchestra.run.smithi072.stdout:/proc 2024-03-22T01:34:26.586 INFO:teuthology.orchestra.run.smithi072.stdout:/sys 2024-03-22T01:34:26.586 INFO:teuthology.orchestra.run.smithi072.stdout:/dev 2024-03-22T01:34:26.586 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/security 2024-03-22T01:34:26.586 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/shm 2024-03-22T01:34:26.586 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/pts 2024-03-22T01:34:26.586 INFO:teuthology.orchestra.run.smithi072.stdout:/run 2024-03-22T01:34:26.586 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/cgroup 2024-03-22T01:34:26.586 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/pstore 2024-03-22T01:34:26.586 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/bpf 2024-03-22T01:34:26.586 INFO:teuthology.orchestra.run.smithi072.stdout:/ 2024-03-22T01:34:26.586 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/selinux 2024-03-22T01:34:26.587 INFO:teuthology.orchestra.run.smithi072.stdout:/proc/sys/fs/binfmt_misc 2024-03-22T01:34:26.587 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/mqueue 2024-03-22T01:34:26.587 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/hugepages 2024-03-22T01:34:26.587 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/debug 2024-03-22T01:34:26.587 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/tracing 2024-03-22T01:34:26.587 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-22T01:34:26.587 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/fuse/connections 2024-03-22T01:34:26.587 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/config 2024-03-22T01:34:26.587 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-sysctl.service 2024-03-22T01:34:26.587 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-22T01:34:26.587 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/nfs/rpc_pipefs 2024-03-22T01:34:26.587 INFO:teuthology.orchestra.run.smithi072.stdout:/run/user/1000 2024-03-22T01:34:26.587 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph 2024-03-22T01:34:26.588 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/containers/storage/overlay 2024-03-22T01:34:26.588 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/containers/storage/overlay/069117ffe870aadaee94635f35b7cf02aae0f606d9587cb0907b5d4dab63e946/merged 2024-03-22T01:34:26.588 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/containers/storage/overlay/6b1cf7b589b79c37eeeeb6b78f58c056fee85807d776163ee22a7a89de3ddf75/merged 2024-03-22T01:34:26.588 INFO:teuthology.orchestra.run.smithi072.stdout:/run/user/0 2024-03-22T01:34:26.588 INFO:teuthology.orchestra.run.smithi072.stdout:/proc/sys/fs/binfmt_misc 2024-03-22T01:34:26.588 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/containers/storage/overlay/b069fa7d57b8b76bc3d1c42abf6d7c1b5fb54ad1a0a29aa2dac69e9ab9916978/merged 2024-03-22T01:34:26.588 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/containers/storage/overlay/2f007650bd719ee2dd3b735bbae33dbcc3561e2865d342d06379848b1e7b80fe/merged 2024-03-22T01:34:26.588 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/containers/storage/overlay/ed2330155c50a791c4a8b49015124397c44acf96b694bce6b3035eaad27ceadf/merged 2024-03-22T01:34:26.588 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/containers/storage/overlay/de768fa5dc235fe12656be652916a87a4b0dddaca31df687197622b510116874/merged 2024-03-22T01:34:26.588 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/containers/storage/overlay/f2cf671e5b1e0026e223fe02882b788143825624e5b2f5c4fffc473f47f5a9c8/merged 2024-03-22T01:34:26.588 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/containers/storage/overlay/a2ca6ec688349bbe337aff3deb37a4c2706108c32e7b2f1dfa5b65f67ceaecb1/merged 2024-03-22T01:34:26.588 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/containers/storage/overlay/7e5b20051723b5937b23547234cd57ff1644844cd2cf7cf7e7dbe5939ac1f3f8/merged 2024-03-22T01:34:26.588 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/containers/storage/overlay/6337ea23c964add57bc86d4a987aed22ce9f35bfc1e9b2d0d153922d7017419d/merged 2024-03-22T01:34:26.588 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns 2024-03-22T01:34:26.589 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-22T01:34:26.589 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-22T01:34:26.589 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T01:34:26.589 DEBUG:teuthology.orchestra.run.smithi072:> ls /sys/fs/fuse/connections 2024-03-22T01:34:26.641 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-22T01:34:26.641 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-03-22T01:34:26.683 DEBUG:teuthology.orchestra.run.smithi072:> sudo modprobe fuse 2024-03-22T01:34:26.709 DEBUG:teuthology.orchestra.run.smithi072:> cat /proc/self/mounts | awk '{print $2}' 2024-03-22T01:34:26.766 INFO:teuthology.orchestra.run.smithi072.stdout:/proc 2024-03-22T01:34:26.766 INFO:teuthology.orchestra.run.smithi072.stdout:/sys 2024-03-22T01:34:26.766 INFO:teuthology.orchestra.run.smithi072.stdout:/dev 2024-03-22T01:34:26.766 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/security 2024-03-22T01:34:26.766 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/shm 2024-03-22T01:34:26.766 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/pts 2024-03-22T01:34:26.766 INFO:teuthology.orchestra.run.smithi072.stdout:/run 2024-03-22T01:34:26.766 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/cgroup 2024-03-22T01:34:26.766 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/pstore 2024-03-22T01:34:26.766 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/bpf 2024-03-22T01:34:26.766 INFO:teuthology.orchestra.run.smithi072.stdout:/ 2024-03-22T01:34:26.766 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/selinux 2024-03-22T01:34:26.767 INFO:teuthology.orchestra.run.smithi072.stdout:/proc/sys/fs/binfmt_misc 2024-03-22T01:34:26.767 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/mqueue 2024-03-22T01:34:26.767 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/hugepages 2024-03-22T01:34:26.767 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/debug 2024-03-22T01:34:26.767 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/tracing 2024-03-22T01:34:26.767 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-22T01:34:26.767 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/fuse/connections 2024-03-22T01:34:26.767 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/config 2024-03-22T01:34:26.767 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-sysctl.service 2024-03-22T01:34:26.767 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-22T01:34:26.767 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/nfs/rpc_pipefs 2024-03-22T01:34:26.767 INFO:teuthology.orchestra.run.smithi072.stdout:/run/user/1000 2024-03-22T01:34:26.767 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph 2024-03-22T01:34:26.767 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/containers/storage/overlay 2024-03-22T01:34:26.768 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/containers/storage/overlay/069117ffe870aadaee94635f35b7cf02aae0f606d9587cb0907b5d4dab63e946/merged 2024-03-22T01:34:26.768 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/containers/storage/overlay/6b1cf7b589b79c37eeeeb6b78f58c056fee85807d776163ee22a7a89de3ddf75/merged 2024-03-22T01:34:26.768 INFO:teuthology.orchestra.run.smithi072.stdout:/run/user/0 2024-03-22T01:34:26.768 INFO:teuthology.orchestra.run.smithi072.stdout:/proc/sys/fs/binfmt_misc 2024-03-22T01:34:26.768 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/containers/storage/overlay/b069fa7d57b8b76bc3d1c42abf6d7c1b5fb54ad1a0a29aa2dac69e9ab9916978/merged 2024-03-22T01:34:26.768 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/containers/storage/overlay/2f007650bd719ee2dd3b735bbae33dbcc3561e2865d342d06379848b1e7b80fe/merged 2024-03-22T01:34:26.768 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/containers/storage/overlay/ed2330155c50a791c4a8b49015124397c44acf96b694bce6b3035eaad27ceadf/merged 2024-03-22T01:34:26.768 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/containers/storage/overlay/de768fa5dc235fe12656be652916a87a4b0dddaca31df687197622b510116874/merged 2024-03-22T01:34:26.768 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/containers/storage/overlay/f2cf671e5b1e0026e223fe02882b788143825624e5b2f5c4fffc473f47f5a9c8/merged 2024-03-22T01:34:26.768 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/containers/storage/overlay/a2ca6ec688349bbe337aff3deb37a4c2706108c32e7b2f1dfa5b65f67ceaecb1/merged 2024-03-22T01:34:26.768 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/containers/storage/overlay/7e5b20051723b5937b23547234cd57ff1644844cd2cf7cf7e7dbe5939ac1f3f8/merged 2024-03-22T01:34:26.768 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/containers/storage/overlay/6337ea23c964add57bc86d4a987aed22ce9f35bfc1e9b2d0d153922d7017419d/merged 2024-03-22T01:34:26.768 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns 2024-03-22T01:34:26.768 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-22T01:34:26.768 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-22T01:34:26.769 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T01:34:26.769 DEBUG:teuthology.orchestra.run.smithi072:> ls /sys/fs/fuse/connections 2024-03-22T01:34:26.776 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi072.stderr:2024-03-22T01:34:26.776+0000 7fbc1ae0e480 -1 init, newargv = 0x55fbef8f98e0 newargc=15 2024-03-22T01:34:26.776 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi072.stderr:ceph-fuse[73989]: starting ceph client 2024-03-22T01:34:26.780 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi072.stderr:ceph-fuse[73989]: starting fuse 2024-03-22T01:34:26.791 INFO:teuthology.orchestra.run.smithi072.stdout:119 2024-03-22T01:34:26.791 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [119] 2024-03-22T01:34:26.791 DEBUG:teuthology.orchestra.run.smithi072:> sudo stdin-killer -- python3 -c ' 2024-03-22T01:34:26.791 DEBUG:teuthology.orchestra.run.smithi072:> import glob 2024-03-22T01:34:26.791 DEBUG:teuthology.orchestra.run.smithi072:> import re 2024-03-22T01:34:26.792 DEBUG:teuthology.orchestra.run.smithi072:> import os 2024-03-22T01:34:26.792 DEBUG:teuthology.orchestra.run.smithi072:> import subprocess 2024-03-22T01:34:26.792 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-22T01:34:26.792 DEBUG:teuthology.orchestra.run.smithi072:> def _find_admin_socket(client_name): 2024-03-22T01:34:26.792 DEBUG:teuthology.orchestra.run.smithi072:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-03-22T01:34:26.792 DEBUG:teuthology.orchestra.run.smithi072:> files = glob.glob(asok_path) 2024-03-22T01:34:26.792 DEBUG:teuthology.orchestra.run.smithi072:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-03-22T01:34:26.792 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-22T01:34:26.792 DEBUG:teuthology.orchestra.run.smithi072:> # Given a non-glob path, it better be there 2024-03-22T01:34:26.792 DEBUG:teuthology.orchestra.run.smithi072:> if "*" not in asok_path: 2024-03-22T01:34:26.792 DEBUG:teuthology.orchestra.run.smithi072:> assert(len(files) == 1) 2024-03-22T01:34:26.792 DEBUG:teuthology.orchestra.run.smithi072:> return files[0] 2024-03-22T01:34:26.792 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-22T01:34:26.792 DEBUG:teuthology.orchestra.run.smithi072:> for f in files: 2024-03-22T01:34:26.793 DEBUG:teuthology.orchestra.run.smithi072:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-22T01:34:26.793 DEBUG:teuthology.orchestra.run.smithi072:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-22T01:34:26.793 DEBUG:teuthology.orchestra.run.smithi072:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-22T01:34:26.793 DEBUG:teuthology.orchestra.run.smithi072:> contents = proc_f.read() 2024-03-22T01:34:26.793 DEBUG:teuthology.orchestra.run.smithi072:> if mountpoint in contents: 2024-03-22T01:34:26.793 DEBUG:teuthology.orchestra.run.smithi072:> return f 2024-03-22T01:34:26.793 DEBUG:teuthology.orchestra.run.smithi072:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-22T01:34:26.793 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-22T01:34:26.793 DEBUG:teuthology.orchestra.run.smithi072:> print(_find_admin_socket("client.admin")) 2024-03-22T01:34:26.793 DEBUG:teuthology.orchestra.run.smithi072:> ' 2024-03-22T01:34:26.856 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:26 smithi072 ceph-mon[32414]: pgmap v296: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 443 B/s wr, 0 op/s 2024-03-22T01:34:26.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:26 smithi089 ceph-mon[35770]: pgmap v296: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 443 B/s wr, 0 op/s 2024-03-22T01:34:26.913 INFO:teuthology.orchestra.run.smithi072.stdout:/var/run/ceph/ceph-client.admin.73989.asok 2024-03-22T01:34:26.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-22T01:34:26.916 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:26 smithi098 ceph-mon[36468]: pgmap v296: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 443 B/s wr, 0 op/s 2024-03-22T01:34:26.923 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.73989.asok 2024-03-22T01:34:26.923 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T01:34:26.923 DEBUG:teuthology.orchestra.run.smithi072:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.73989.asok status 2024-03-22T01:34:27.060 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-22T01:34:27.060 INFO:teuthology.orchestra.run.smithi072.stdout: "metadata": { 2024-03-22T01:34:27.060 INFO:teuthology.orchestra.run.smithi072.stdout: "ceph_sha1": "c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e", 2024-03-22T01:34:27.060 INFO:teuthology.orchestra.run.smithi072.stdout: "ceph_version": "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)", 2024-03-22T01:34:27.060 INFO:teuthology.orchestra.run.smithi072.stdout: "entity_id": "admin", 2024-03-22T01:34:27.060 INFO:teuthology.orchestra.run.smithi072.stdout: "hostname": "smithi072", 2024-03-22T01:34:27.060 INFO:teuthology.orchestra.run.smithi072.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-03-22T01:34:27.060 INFO:teuthology.orchestra.run.smithi072.stdout: "pid": "73989", 2024-03-22T01:34:27.060 INFO:teuthology.orchestra.run.smithi072.stdout: "root": "/" 2024-03-22T01:34:27.061 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:34:27.061 INFO:teuthology.orchestra.run.smithi072.stdout: "dentry_count": 0, 2024-03-22T01:34:27.061 INFO:teuthology.orchestra.run.smithi072.stdout: "dentry_pinned_count": 0, 2024-03-22T01:34:27.061 INFO:teuthology.orchestra.run.smithi072.stdout: "id": 14769, 2024-03-22T01:34:27.061 INFO:teuthology.orchestra.run.smithi072.stdout: "inst": { 2024-03-22T01:34:27.061 INFO:teuthology.orchestra.run.smithi072.stdout: "name": { 2024-03-22T01:34:27.061 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "client", 2024-03-22T01:34:27.061 INFO:teuthology.orchestra.run.smithi072.stdout: "num": 14769 2024-03-22T01:34:27.061 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:34:27.061 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": { 2024-03-22T01:34:27.061 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "v1", 2024-03-22T01:34:27.061 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": "192.168.0.1:0", 2024-03-22T01:34:27.061 INFO:teuthology.orchestra.run.smithi072.stdout: "nonce": 1446100583 2024-03-22T01:34:27.061 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:34:27.061 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:34:27.062 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": { 2024-03-22T01:34:27.062 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "v1", 2024-03-22T01:34:27.062 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": "192.168.0.1:0", 2024-03-22T01:34:27.062 INFO:teuthology.orchestra.run.smithi072.stdout: "nonce": 1446100583 2024-03-22T01:34:27.062 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:34:27.062 INFO:teuthology.orchestra.run.smithi072.stdout: "inst_str": "client.14769 192.168.0.1:0/1446100583", 2024-03-22T01:34:27.062 INFO:teuthology.orchestra.run.smithi072.stdout: "addr_str": "192.168.0.1:0/1446100583", 2024-03-22T01:34:27.062 INFO:teuthology.orchestra.run.smithi072.stdout: "inode_count": 1, 2024-03-22T01:34:27.062 INFO:teuthology.orchestra.run.smithi072.stdout: "mds_epoch": 17, 2024-03-22T01:34:27.062 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_epoch": 91, 2024-03-22T01:34:27.062 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_epoch_barrier": 0, 2024-03-22T01:34:27.062 INFO:teuthology.orchestra.run.smithi072.stdout: "blocklisted": false, 2024-03-22T01:34:27.062 INFO:teuthology.orchestra.run.smithi072.stdout: "fs_name": "cephfs" 2024-03-22T01:34:27.063 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-22T01:34:27.068 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T01:34:27.069 DEBUG:teuthology.orchestra.run.smithi072:> stat --file-system '--printf=%T 2024-03-22T01:34:27.069 DEBUG:teuthology.orchestra.run.smithi072:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-03-22T01:34:27.084 INFO:teuthology.orchestra.run.smithi072.stdout:fuseblk 2024-03-22T01:34:27.084 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-03-22T01:34:27.084 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T01:34:27.084 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-03-22T01:34:27.152 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-03-22T01:34:27.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-22T01:34:27.261 INFO:teuthology.orchestra.run.smithi072.stdout:# file: . 2024-03-22T01:34:27.261 INFO:teuthology.orchestra.run.smithi072.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-03-22T01:34:27.261 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:34:27.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-22T01:34:27.266 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-22T01:34:27.266 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-22T01:34:27.283 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T01:34:27.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.426+0000 7ffb3ff13640 1 -- 172.21.15.72:0/3416106274 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffb38107920 msgr2=0x7ffb38109d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:27.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.426+0000 7ffb3ff13640 1 --2- 172.21.15.72:0/3416106274 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffb38107920 0x7ffb38109d10 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ffb34009ca0 tx=0x7ffb3402c0f0 comp rx=0 tx=0).stop 2024-03-22T01:34:27.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.426+0000 7ffb3ff13640 1 -- 172.21.15.72:0/3416106274 shutdown_connections 2024-03-22T01:34:27.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.426+0000 7ffb3ff13640 1 --2- 172.21.15.72:0/3416106274 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffb38107920 0x7ffb38109d10 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:27.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.426+0000 7ffb3ff13640 1 --2- 172.21.15.72:0/3416106274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb38104ff0 0x7ffb381073e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:27.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.426+0000 7ffb3ff13640 1 --2- 172.21.15.72:0/3416106274 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb381026c0 0x7ffb38104ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:27.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.426+0000 7ffb3ff13640 1 -- 172.21.15.72:0/3416106274 >> 172.21.15.72:0/3416106274 conn(0x7ffb380fc030 msgr2=0x7ffb380fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:27.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.426+0000 7ffb3ff13640 1 -- 172.21.15.72:0/3416106274 shutdown_connections 2024-03-22T01:34:27.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.426+0000 7ffb3ff13640 1 -- 172.21.15.72:0/3416106274 wait complete. 2024-03-22T01:34:27.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.427+0000 7ffb3ff13640 1 Processor -- start 2024-03-22T01:34:27.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.427+0000 7ffb3ff13640 1 -- start start 2024-03-22T01:34:27.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.427+0000 7ffb3ff13640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffb381026c0 0x7ffb38197e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:27.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.427+0000 7ffb3ff13640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb38104ff0 0x7ffb38198360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:27.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.427+0000 7ffb3ff13640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb38107920 0x7ffb3819f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:27.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.427+0000 7ffb3ff13640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7ffb3810d840 con 0x7ffb381026c0 2024-03-22T01:34:27.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.427+0000 7ffb3ff13640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ffb3810d6c0 con 0x7ffb38104ff0 2024-03-22T01:34:27.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.427+0000 7ffb3ff13640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ffb3810d9c0 con 0x7ffb38107920 2024-03-22T01:34:27.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.427+0000 7ffb3dcaf640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffb381026c0 0x7ffb38197e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:27.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.427+0000 7ffb3dcaf640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffb381026c0 0x7ffb38197e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47938/0 (socket says 172.21.15.72:47938) 2024-03-22T01:34:27.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.427+0000 7ffb3dcaf640 1 -- 172.21.15.72:0/1087884979 learned_addr learned my addr 172.21.15.72:0/1087884979 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:34:27.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.427+0000 7ffb3d4ae640 1 --2- 172.21.15.72:0/1087884979 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb38104ff0 0x7ffb38198360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:27.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.427+0000 7ffb3e4b0640 1 --2- 172.21.15.72:0/1087884979 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb38107920 0x7ffb3819f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:27.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.427+0000 7ffb3dcaf640 1 -- 172.21.15.72:0/1087884979 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb38107920 msgr2=0x7ffb3819f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:27.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.427+0000 7ffb3dcaf640 1 --2- 172.21.15.72:0/1087884979 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb38107920 0x7ffb3819f3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:27.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.427+0000 7ffb3dcaf640 1 -- 172.21.15.72:0/1087884979 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb38104ff0 msgr2=0x7ffb38198360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:27.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.427+0000 7ffb3dcaf640 1 --2- 172.21.15.72:0/1087884979 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb38104ff0 0x7ffb38198360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:27.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.427+0000 7ffb3dcaf640 1 -- 172.21.15.72:0/1087884979 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb3819fa40 con 0x7ffb381026c0 2024-03-22T01:34:27.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.428+0000 7ffb3dcaf640 1 --2- 172.21.15.72:0/1087884979 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffb381026c0 0x7ffb38197e20 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ffb3000e9f0 tx=0x7ffb3000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:27.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.428+0000 7ffb2effd640 1 -- 172.21.15.72:0/1087884979 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb3000cd80 con 0x7ffb381026c0 2024-03-22T01:34:27.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.428+0000 7ffb2effd640 1 -- 172.21.15.72:0/1087884979 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb30010910 con 0x7ffb381026c0 2024-03-22T01:34:27.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.428+0000 7ffb3ff13640 1 -- 172.21.15.72:0/1087884979 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb3819fd30 con 0x7ffb381026c0 2024-03-22T01:34:27.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.428+0000 7ffb2effd640 1 -- 172.21.15.72:0/1087884979 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb30018990 con 0x7ffb381026c0 2024-03-22T01:34:27.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.428+0000 7ffb3ff13640 1 -- 172.21.15.72:0/1087884979 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb381a01d0 con 0x7ffb381026c0 2024-03-22T01:34:27.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.429+0000 7ffb3ff13640 1 -- 172.21.15.72:0/1087884979 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb380fdb20 con 0x7ffb381026c0 2024-03-22T01:34:27.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.430+0000 7ffb2effd640 1 -- 172.21.15.72:0/1087884979 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7ffb30010ac0 con 0x7ffb381026c0 2024-03-22T01:34:27.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.430+0000 7ffb2effd640 1 --2- 172.21.15.72:0/1087884979 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7ffb240a2f20 0x7ffb240a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:27.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.431+0000 7ffb2effd640 1 -- 172.21.15.72:0/1087884979 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7ffb300cfd60 con 0x7ffb381026c0 2024-03-22T01:34:27.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.431+0000 7ffb3d4ae640 1 --2- 172.21.15.72:0/1087884979 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7ffb240a2f20 0x7ffb240a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:27.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.431+0000 7ffb3d4ae640 1 --2- 172.21.15.72:0/1087884979 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7ffb240a2f20 0x7ffb240a53e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ffb38199330 tx=0x7ffb2800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:27.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.434+0000 7ffb2effd640 1 -- 172.21.15.72:0/1087884979 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ffb3009c300 con 0x7ffb381026c0 2024-03-22T01:34:27.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.612+0000 7ffb3ff13640 1 -- 172.21.15.72:0/1087884979 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ffb3802d530 con 0x7ffb381026c0 2024-03-22T01:34:27.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.614+0000 7ffb2effd640 1 -- 172.21.15.72:0/1087884979 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24045 (secure 0 0 0) 0x7ffb300a03a0 con 0x7ffb381026c0 2024-03-22T01:34:27.613 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:34:27.613 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":91,"fsid":"022a99bc-e7eb-11ee-95cd-87774f69a715","created":"2024-03-22T01:26:23.893672+0000","modified":"2024-03-22T01:34:19.866367+0000","last_up_change":"2024-03-22T01:33:25.223684+0000","last_in_change":"2024-03-22T01:33:05.010331+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T01:29:37.438503+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-22T01:34:12.610422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-22T01:34:13.761686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-22T01:34:17.507039+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"16c15bb0-db92-4191-9063-38b4890e81e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6803","nonce":661906791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6805","nonce":661906791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6809","nonce":661906791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6807","nonce":661906791}]},"public_addr":"172.21.15.72:6803/661906791","cluster_addr":"172.21.15.72:6805/661906791","heartbeat_back_addr":"172.21.15.72:6809/661906791","heartbeat_front_addr":"172.21.15.72:6807/661906791","state":["exists","up"]},{"osd":1,"uuid":"981290ba-97dc-496b-aa22-d51704184b10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6801","nonce":2870590550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6803","nonce":2870590550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6807","nonce":2870590550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6805","nonce":2870590550}]},"public_addr":"172.21.15.89:6801/2870590550","cluster_addr":"172.21.15.89:6803/2870590550","heartbeat_back_addr":"172.21.15.89:6807/2870590550","heartbeat_front_addr":"172.21.15.89:6805/2870590550","state":["exists","up"]},{"osd":2,"uuid":"f07ed658-f545-46a1-8edd-8b1d5d49885c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6801","nonce":3865778118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6803","nonce":3865778118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6807","nonce":3865778118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6805","nonce":3865778118}]},"public_addr":"172.21.15.98:6801/3865778118","cluster_addr":"172.21.15.98:6803/3865778118","heartbeat_back_addr":"172.21.15.98:6807/3865778118","heartbeat_front_addr":"172.21.15.98:6805/3865778118","state":["exists","up"]},{"osd":3,"uuid":"ccc4bc40-2c48-46d2-8548-0c05b44f9f1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6811","nonce":1865951395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6813","nonce":1865951395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6817","nonce":1865951395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6815","nonce":1865951395}]},"public_addr":"172.21.15.72:6811/1865951395","cluster_addr":"172.21.15.72:6813/1865951395","heartbeat_back_addr":"172.21.15.72:6817/1865951395","heartbeat_front_addr":"172.21.15.72:6815/1865951395","state":["exists","up"]},{"osd":4,"uuid":"1812439f-810a-4ed2-9395-c00c9a085a13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6809","nonce":2683760795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6811","nonce":2683760795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6815","nonce":2683760795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6813","nonce":2683760795}]},"public_addr":"172.21.15.89:6809/2683760795","cluster_addr":"172.21.15.89:6811/2683760795","heartbeat_back_addr":"172.21.15.89:6815/2683760795","heartbeat_front_addr":"172.21.15.89:6813/2683760795","state":["exists","up"]},{"osd":5,"uuid":"e78954d8-f4e3-4d58-8536-d2d59c843386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6809","nonce":2635838572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6811","nonce":2635838572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6815","nonce":2635838572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6813","nonce":2635838572}]},"public_addr":"172.21.15.98:6809/2635838572","cluster_addr":"172.21.15.98:6811/2635838572","heartbeat_back_addr":"172.21.15.98:6815/2635838572","heartbeat_front_addr":"172.21.15.98:6813/2635838572","state":["exists","up"]},{"osd":6,"uuid":"dc6c8be2-b64c-4959-8015-7b0ea6d25599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6819","nonce":3158982661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6821","nonce":3158982661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6825","nonce":3158982661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6823","nonce":3158982661}]},"public_addr":"172.21.15.72:6819/3158982661","cluster_addr":"172.21.15.72:6821/3158982661","heartbeat_back_addr":"172.21.15.72:6825/3158982661","heartbeat_front_addr":"172.21.15.72:6823/3158982661","state":["exists","up"]},{"osd":7,"uuid":"7a73469f-cf3a-4306-ae93-ea46e21ebbbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6817","nonce":3075913234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6819","nonce":3075913234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6823","nonce":3075913234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6821","nonce":3075913234}]},"public_addr":"172.21.15.89:6817/3075913234","cluster_addr":"172.21.15.89:6819/3075913234","heartbeat_back_addr":"172.21.15.89:6823/3075913234","heartbeat_front_addr":"172.21.15.89:6821/3075913234","state":["exists","up"]},{"osd":8,"uuid":"c4b13bee-ea4d-4344-95fb-2622ecbb8196","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6817","nonce":1813255577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6819","nonce":1813255577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6823","nonce":1813255577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6821","nonce":1813255577}]},"public_addr":"172.21.15.98:6817/1813255577","cluster_addr":"172.21.15.98:6819/1813255577","heartbeat_back_addr":"172.21.15.98:6823/1813255577","heartbeat_front_addr":"172.21.15.98:6821/1813255577","state":["exists","up"]},{"osd":9,"uuid":"eefb0a44-4a36-4e55-9d04-0501bf718db6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6827","nonce":3377749206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6829","nonce":3377749206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6832","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6833","nonce":3377749206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6831","nonce":3377749206}]},"public_addr":"172.21.15.72:6827/3377749206","cluster_addr":"172.21.15.72:6829/3377749206","heartbeat_back_addr":"172.21.15.72:6833/3377749206","heartbeat_front_addr":"172.21.15.72:6831/3377749206","state":["exists","up"]},{"osd":10,"uuid":"8741d25f-40dd-40d7-b040-5cc5e411eb98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6825","nonce":885058786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6827","nonce":885058786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6831","nonce":885058786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6829","nonce":885058786}]},"public_addr":"172.21.15.89:6825/885058786","cluster_addr":"172.21.15.89:6827/885058786","heartbeat_back_addr":"172.21.15.89:6831/885058786","heartbeat_front_addr":"172.21.15.89:6829/885058786","state":["exists","up"]},{"osd":11,"uuid":"8d02e957-eda4-4184-b5c6-5eb3b8b38060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6825","nonce":1517095838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6827","nonce":1517095838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6831","nonce":1517095838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6829","nonce":1517095838}]},"public_addr":"172.21.15.98:6825/1517095838","cluster_addr":"172.21.15.98:6827/1517095838","heartbeat_back_addr":"172.21.15.98:6831/1517095838","heartbeat_front_addr":"172.21.15.98:6829/1517095838","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:28:47.307652+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:11.037135+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:34.619188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:59.350738+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:30:23.787337+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:30:48.994154+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:31:13.690350+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:31:39.628116+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:04.723912+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:30.689741+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:57.229508+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:33:23.154537+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:0/121087196":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/2451857761":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/1228477995":"2024-03-23T01:27:15.339809+0000","172.21.15.72:6801/1513232913":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/3558958264":"2024-03-23T01:26:50.074871+0000","172.21.15.72:0/1510187198":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6800/3518012916":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6801/3518012916":"2024-03-23T01:26:50.074871+0000","172.21.15.72:0/1372801041":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6800/1513232913":"2024-03-23T01:27:15.339809+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T01:34:27.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.616+0000 7ffb3ff13640 1 -- 172.21.15.72:0/1087884979 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7ffb240a2f20 msgr2=0x7ffb240a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:27.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.616+0000 7ffb3ff13640 1 --2- 172.21.15.72:0/1087884979 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7ffb240a2f20 0x7ffb240a53e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ffb38199330 tx=0x7ffb2800b390 comp rx=0 tx=0).stop 2024-03-22T01:34:27.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.616+0000 7ffb3ff13640 1 -- 172.21.15.72:0/1087884979 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffb381026c0 msgr2=0x7ffb38197e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:27.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.616+0000 7ffb3ff13640 1 --2- 172.21.15.72:0/1087884979 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffb381026c0 0x7ffb38197e20 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ffb3000e9f0 tx=0x7ffb3000eeb0 comp rx=0 tx=0).stop 2024-03-22T01:34:27.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.616+0000 7ffb3ff13640 1 -- 172.21.15.72:0/1087884979 shutdown_connections 2024-03-22T01:34:27.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.616+0000 7ffb3ff13640 1 --2- 172.21.15.72:0/1087884979 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7ffb240a2f20 0x7ffb240a53e0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:27.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.616+0000 7ffb3ff13640 1 --2- 172.21.15.72:0/1087884979 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb38104ff0 0x7ffb38198360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:27.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.616+0000 7ffb3ff13640 1 --2- 172.21.15.72:0/1087884979 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb38107920 0x7ffb3819f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:27.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.616+0000 7ffb3ff13640 1 --2- 172.21.15.72:0/1087884979 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffb381026c0 0x7ffb38197e20 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:27.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.616+0000 7ffb3ff13640 1 -- 172.21.15.72:0/1087884979 >> 172.21.15.72:0/1087884979 conn(0x7ffb380fc030 msgr2=0x7ffb38103000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:27.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.616+0000 7ffb3ff13640 1 -- 172.21.15.72:0/1087884979 shutdown_connections 2024-03-22T01:34:27.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.616+0000 7ffb3ff13640 1 -- 172.21.15.72:0/1087884979 wait complete. 2024-03-22T01:34:27.626 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-22T01:29:37.438503+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-22T01:34:12.610422+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-22T01:34:13.761686+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-22T01:34:17.507039+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-22T01:34:27.626 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-22T01:34:27.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.732+0000 7f458195c640 1 -- 172.21.15.72:0/518371102 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f457c104f60 msgr2=0x7f457c107350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:27.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.732+0000 7f458195c640 1 --2- 172.21.15.72:0/518371102 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f457c104f60 0x7f457c107350 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f4570009900 tx=0x7f457002bc30 comp rx=0 tx=0).stop 2024-03-22T01:34:27.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.732+0000 7f458195c640 1 -- 172.21.15.72:0/518371102 shutdown_connections 2024-03-22T01:34:27.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.732+0000 7f458195c640 1 --2- 172.21.15.72:0/518371102 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f457c107890 0x7f457c109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:27.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.732+0000 7f458195c640 1 --2- 172.21.15.72:0/518371102 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f457c104f60 0x7f457c107350 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:27.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.732+0000 7f458195c640 1 --2- 172.21.15.72:0/518371102 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f457c102630 0x7f457c104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:27.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.732+0000 7f458195c640 1 -- 172.21.15.72:0/518371102 >> 172.21.15.72:0/518371102 conn(0x7f457c0fc020 msgr2=0x7f457c0fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:27.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.732+0000 7f458195c640 1 -- 172.21.15.72:0/518371102 shutdown_connections 2024-03-22T01:34:27.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.732+0000 7f458195c640 1 -- 172.21.15.72:0/518371102 wait complete. 2024-03-22T01:34:27.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.733+0000 7f458195c640 1 Processor -- start 2024-03-22T01:34:27.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.733+0000 7f458195c640 1 -- start start 2024-03-22T01:34:27.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.733+0000 7f458195c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f457c102630 0x7f457c197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:27.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.733+0000 7f458195c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f457c104f60 0x7f457c198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:27.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.733+0000 7f458195c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f457c107890 0x7f457c19f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:27.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.733+0000 7f458195c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f457c10d840 con 0x7f457c107890 2024-03-22T01:34:27.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.733+0000 7f458195c640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f457c10d6c0 con 0x7f457c104f60 2024-03-22T01:34:27.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.733+0000 7f458195c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f457c10d9c0 con 0x7f457c102630 2024-03-22T01:34:27.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.733+0000 7f457b7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f457c107890 0x7f457c19f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:27.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.733+0000 7f457b7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f457c107890 0x7f457c19f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47952/0 (socket says 172.21.15.72:47952) 2024-03-22T01:34:27.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.733+0000 7f457b7fe640 1 -- 172.21.15.72:0/2071956482 learned_addr learned my addr 172.21.15.72:0/2071956482 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:34:27.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.733+0000 7f457a7fc640 1 --2- 172.21.15.72:0/2071956482 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f457c104f60 0x7f457c198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:27.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.733+0000 7f457affd640 1 --2- 172.21.15.72:0/2071956482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f457c102630 0x7f457c197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:27.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.734+0000 7f457b7fe640 1 -- 172.21.15.72:0/2071956482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f457c102630 msgr2=0x7f457c197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:27.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.734+0000 7f457b7fe640 1 --2- 172.21.15.72:0/2071956482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f457c102630 0x7f457c197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:27.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.734+0000 7f457b7fe640 1 -- 172.21.15.72:0/2071956482 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f457c104f60 msgr2=0x7f457c198460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:27.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.734+0000 7f457b7fe640 1 --2- 172.21.15.72:0/2071956482 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f457c104f60 0x7f457c198460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:27.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.734+0000 7f457b7fe640 1 -- 172.21.15.72:0/2071956482 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f457c19fb40 con 0x7f457c107890 2024-03-22T01:34:27.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.734+0000 7f457a7fc640 1 --2- 172.21.15.72:0/2071956482 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f457c104f60 0x7f457c198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:34:27.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.734+0000 7f457b7fe640 1 --2- 172.21.15.72:0/2071956482 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f457c107890 0x7f457c19f4d0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f457400e970 tx=0x7f457400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:27.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.734+0000 7f458095a640 1 -- 172.21.15.72:0/2071956482 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f457400ccc0 con 0x7f457c107890 2024-03-22T01:34:27.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.734+0000 7f458095a640 1 -- 172.21.15.72:0/2071956482 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f457400ce60 con 0x7f457c107890 2024-03-22T01:34:27.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.734+0000 7f458195c640 1 -- 172.21.15.72:0/2071956482 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f457c19fe30 con 0x7f457c107890 2024-03-22T01:34:27.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.734+0000 7f458095a640 1 -- 172.21.15.72:0/2071956482 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4574010490 con 0x7f457c107890 2024-03-22T01:34:27.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.734+0000 7f458195c640 1 -- 172.21.15.72:0/2071956482 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f457c1a02a0 con 0x7f457c107890 2024-03-22T01:34:27.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.735+0000 7f458195c640 1 -- 172.21.15.72:0/2071956482 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f457c0fdb10 con 0x7f457c107890 2024-03-22T01:34:27.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.736+0000 7f458095a640 1 -- 172.21.15.72:0/2071956482 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f4574005000 con 0x7f457c107890 2024-03-22T01:34:27.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.737+0000 7f458095a640 1 --2- 172.21.15.72:0/2071956482 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f45640a2f20 0x7f45640a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:27.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.737+0000 7f457affd640 1 --2- 172.21.15.72:0/2071956482 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f45640a2f20 0x7f45640a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:27.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.737+0000 7f458095a640 1 -- 172.21.15.72:0/2071956482 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f45740cfd00 con 0x7f457c107890 2024-03-22T01:34:27.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.737+0000 7f457affd640 1 --2- 172.21.15.72:0/2071956482 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f45640a2f20 0x7f45640a53e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f456c005fd0 tx=0x7f456c009680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:27.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.740+0000 7f458095a640 1 -- 172.21.15.72:0/2071956482 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4574053740 con 0x7f457c107890 2024-03-22T01:34:27.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.913+0000 7f458195c640 1 -- 172.21.15.72:0/2071956482 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f457c02d3b0 con 0x7f457c107890 2024-03-22T01:34:27.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.913+0000 7f458095a640 1 -- 172.21.15.72:0/2071956482 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7f457409c8f0 con 0x7f457c107890 2024-03-22T01:34:27.912 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-22T01:34:27.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.915+0000 7f458195c640 1 -- 172.21.15.72:0/2071956482 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f45640a2f20 msgr2=0x7f45640a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:27.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.915+0000 7f458195c640 1 --2- 172.21.15.72:0/2071956482 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f45640a2f20 0x7f45640a53e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f456c005fd0 tx=0x7f456c009680 comp rx=0 tx=0).stop 2024-03-22T01:34:27.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.915+0000 7f458195c640 1 -- 172.21.15.72:0/2071956482 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f457c107890 msgr2=0x7f457c19f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:27.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.915+0000 7f458195c640 1 --2- 172.21.15.72:0/2071956482 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f457c107890 0x7f457c19f4d0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f457400e970 tx=0x7f457400ed70 comp rx=0 tx=0).stop 2024-03-22T01:34:27.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.915+0000 7f458195c640 1 -- 172.21.15.72:0/2071956482 shutdown_connections 2024-03-22T01:34:27.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.915+0000 7f458195c640 1 --2- 172.21.15.72:0/2071956482 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f45640a2f20 0x7f45640a53e0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:27.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.915+0000 7f458195c640 1 --2- 172.21.15.72:0/2071956482 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f457c104f60 0x7f457c198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:27.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.915+0000 7f458195c640 1 --2- 172.21.15.72:0/2071956482 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f457c107890 0x7f457c19f4d0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:27.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.915+0000 7f458195c640 1 --2- 172.21.15.72:0/2071956482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f457c102630 0x7f457c197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:27.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.915+0000 7f458195c640 1 -- 172.21.15.72:0/2071956482 >> 172.21.15.72:0/2071956482 conn(0x7f457c0fc020 msgr2=0x7f457c103090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:27.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.916+0000 7f458195c640 1 -- 172.21.15.72:0/2071956482 shutdown_connections 2024-03-22T01:34:27.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:27.916+0000 7f458195c640 1 -- 172.21.15.72:0/2071956482 wait complete. 2024-03-22T01:34:27.925 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-22T01:34:28.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.030+0000 7f4bd489d640 1 -- 172.21.15.72:0/4270729698 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4bd0102d90 msgr2=0x7f4bd0103210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:28.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.030+0000 7f4bd489d640 1 --2- 172.21.15.72:0/4270729698 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4bd0102d90 0x7f4bd0103210 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4bb8009860 tx=0x7f4bb802bb70 comp rx=0 tx=0).stop 2024-03-22T01:34:28.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.030+0000 7f4bd489d640 1 -- 172.21.15.72:0/4270729698 shutdown_connections 2024-03-22T01:34:28.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.030+0000 7f4bd489d640 1 --2- 172.21.15.72:0/4270729698 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4bd0102d90 0x7f4bd0103210 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:28.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.030+0000 7f4bd489d640 1 --2- 172.21.15.72:0/4270729698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bd0103750 0x7f4bd0109fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:28.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.030+0000 7f4bd489d640 1 --2- 172.21.15.72:0/4270729698 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4bd0101ba0 0x7f4bd0101fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:28.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.030+0000 7f4bd489d640 1 -- 172.21.15.72:0/4270729698 >> 172.21.15.72:0/4270729698 conn(0x7f4bd00fd110 msgr2=0x7f4bd00ff550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:28.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.030+0000 7f4bd489d640 1 -- 172.21.15.72:0/4270729698 shutdown_connections 2024-03-22T01:34:28.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.030+0000 7f4bd489d640 1 -- 172.21.15.72:0/4270729698 wait complete. 2024-03-22T01:34:28.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.031+0000 7f4bd489d640 1 Processor -- start 2024-03-22T01:34:28.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.031+0000 7f4bd489d640 1 -- start start 2024-03-22T01:34:28.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.031+0000 7f4bd489d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4bd0101ba0 0x7f4bd0197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:28.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.031+0000 7f4bd489d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4bd0102d90 0x7f4bd01983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:28.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.031+0000 7f4bd489d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bd0103750 0x7f4bd019f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:28.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.031+0000 7f4bd489d640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd010d800 con 0x7f4bd0101ba0 2024-03-22T01:34:28.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.031+0000 7f4bd489d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd010d680 con 0x7f4bd0102d90 2024-03-22T01:34:28.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.031+0000 7f4bd489d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd010d980 con 0x7f4bd0103750 2024-03-22T01:34:28.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.031+0000 7f4bce59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4bd0101ba0 0x7f4bd0197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:28.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.031+0000 7f4bce59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4bd0101ba0 0x7f4bd0197ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47962/0 (socket says 172.21.15.72:47962) 2024-03-22T01:34:28.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.031+0000 7f4bce59c640 1 -- 172.21.15.72:0/3098315673 learned_addr learned my addr 172.21.15.72:0/3098315673 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:34:28.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.031+0000 7f4bc7fff640 1 --2- 172.21.15.72:0/3098315673 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4bd0102d90 0x7f4bd01983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:28.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.031+0000 7f4bced9d640 1 --2- 172.21.15.72:0/3098315673 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bd0103750 0x7f4bd019f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:28.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.031+0000 7f4bce59c640 1 -- 172.21.15.72:0/3098315673 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bd0103750 msgr2=0x7f4bd019f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:28.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.031+0000 7f4bce59c640 1 --2- 172.21.15.72:0/3098315673 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bd0103750 0x7f4bd019f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:28.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.031+0000 7f4bce59c640 1 -- 172.21.15.72:0/3098315673 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4bd0102d90 msgr2=0x7f4bd01983e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:28.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.031+0000 7f4bce59c640 1 --2- 172.21.15.72:0/3098315673 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4bd0102d90 0x7f4bd01983e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:28.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.031+0000 7f4bce59c640 1 -- 172.21.15.72:0/3098315673 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bd019fac0 con 0x7f4bd0101ba0 2024-03-22T01:34:28.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.031+0000 7f4bc7fff640 1 --2- 172.21.15.72:0/3098315673 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4bd0102d90 0x7f4bd01983e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:34:28.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.032+0000 7f4bce59c640 1 --2- 172.21.15.72:0/3098315673 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4bd0101ba0 0x7f4bd0197ea0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f4bc000e9a0 tx=0x7f4bc000ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:28.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.032+0000 7f4bc77fe640 1 -- 172.21.15.72:0/3098315673 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc000cd80 con 0x7f4bd0101ba0 2024-03-22T01:34:28.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.032+0000 7f4bc77fe640 1 -- 172.21.15.72:0/3098315673 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4bc0010910 con 0x7f4bd0101ba0 2024-03-22T01:34:28.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.032+0000 7f4bd489d640 1 -- 172.21.15.72:0/3098315673 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bd019fdb0 con 0x7f4bd0101ba0 2024-03-22T01:34:28.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.032+0000 7f4bc77fe640 1 -- 172.21.15.72:0/3098315673 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc0018a30 con 0x7f4bd0101ba0 2024-03-22T01:34:28.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.032+0000 7f4bd489d640 1 -- 172.21.15.72:0/3098315673 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bd01a0210 con 0x7f4bd0101ba0 2024-03-22T01:34:28.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.033+0000 7f4bd489d640 1 -- 172.21.15.72:0/3098315673 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bd0108f10 con 0x7f4bd0101ba0 2024-03-22T01:34:28.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.034+0000 7f4bc77fe640 1 -- 172.21.15.72:0/3098315673 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f4bc0010ac0 con 0x7f4bd0101ba0 2024-03-22T01:34:28.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.034+0000 7f4bc77fe640 1 --2- 172.21.15.72:0/3098315673 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f4bbc0a2f20 0x7f4bbc0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:28.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.035+0000 7f4bc7fff640 1 --2- 172.21.15.72:0/3098315673 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f4bbc0a2f20 0x7f4bbc0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:28.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.035+0000 7f4bc77fe640 1 -- 172.21.15.72:0/3098315673 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f4bc00d0070 con 0x7f4bd0101ba0 2024-03-22T01:34:28.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.035+0000 7f4bc7fff640 1 --2- 172.21.15.72:0/3098315673 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f4bbc0a2f20 0x7f4bbc0a53e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f4bb802c180 tx=0x7f4bb802e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:28.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.038+0000 7f4bc77fe640 1 -- 172.21.15.72:0/3098315673 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4bc0026070 con 0x7f4bd0101ba0 2024-03-22T01:34:28.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.212+0000 7f4bd489d640 1 -- 172.21.15.72:0/3098315673 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4bd002d660 con 0x7f4bd0101ba0 2024-03-22T01:34:28.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.213+0000 7f4bc77fe640 1 -- 172.21.15.72:0/3098315673 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7f4bc009cc60 con 0x7f4bd0101ba0 2024-03-22T01:34:28.212 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-03-22T01:34:28.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.215+0000 7f4bd489d640 1 -- 172.21.15.72:0/3098315673 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f4bbc0a2f20 msgr2=0x7f4bbc0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:28.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.215+0000 7f4bd489d640 1 --2- 172.21.15.72:0/3098315673 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f4bbc0a2f20 0x7f4bbc0a53e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f4bb802c180 tx=0x7f4bb802e040 comp rx=0 tx=0).stop 2024-03-22T01:34:28.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.215+0000 7f4bd489d640 1 -- 172.21.15.72:0/3098315673 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4bd0101ba0 msgr2=0x7f4bd0197ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:28.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.215+0000 7f4bd489d640 1 --2- 172.21.15.72:0/3098315673 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4bd0101ba0 0x7f4bd0197ea0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f4bc000e9a0 tx=0x7f4bc000ee60 comp rx=0 tx=0).stop 2024-03-22T01:34:28.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.215+0000 7f4bd489d640 1 -- 172.21.15.72:0/3098315673 shutdown_connections 2024-03-22T01:34:28.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.215+0000 7f4bd489d640 1 --2- 172.21.15.72:0/3098315673 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f4bbc0a2f20 0x7f4bbc0a53e0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:28.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.215+0000 7f4bd489d640 1 --2- 172.21.15.72:0/3098315673 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4bd0102d90 0x7f4bd01983e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:28.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.215+0000 7f4bd489d640 1 --2- 172.21.15.72:0/3098315673 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4bd0103750 0x7f4bd019f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:28.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.215+0000 7f4bd489d640 1 --2- 172.21.15.72:0/3098315673 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4bd0101ba0 0x7f4bd0197ea0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:28.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.215+0000 7f4bd489d640 1 -- 172.21.15.72:0/3098315673 >> 172.21.15.72:0/3098315673 conn(0x7f4bd00fd110 msgr2=0x7f4bd01060e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:28.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.215+0000 7f4bd489d640 1 -- 172.21.15.72:0/3098315673 shutdown_connections 2024-03-22T01:34:28.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.215+0000 7f4bd489d640 1 -- 172.21.15.72:0/3098315673 wait complete. 2024-03-22T01:34:28.224 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-22T01:34:28.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.330+0000 7f89716f4640 1 -- 172.21.15.72:0/4201211034 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f896c104f50 msgr2=0x7f896c107340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:28.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.330+0000 7f89716f4640 1 --2- 172.21.15.72:0/4201211034 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f896c104f50 0x7f896c107340 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8954009860 tx=0x7f895402bb70 comp rx=0 tx=0).stop 2024-03-22T01:34:28.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.331+0000 7f89716f4640 1 -- 172.21.15.72:0/4201211034 shutdown_connections 2024-03-22T01:34:28.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.331+0000 7f89716f4640 1 --2- 172.21.15.72:0/4201211034 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f896c104f50 0x7f896c107340 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:28.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.331+0000 7f89716f4640 1 --2- 172.21.15.72:0/4201211034 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f896c107880 0x7f896c109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:28.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.331+0000 7f89716f4640 1 --2- 172.21.15.72:0/4201211034 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f896c102620 0x7f896c104a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:28.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.331+0000 7f89716f4640 1 -- 172.21.15.72:0/4201211034 >> 172.21.15.72:0/4201211034 conn(0x7f896c0fbff0 msgr2=0x7f896c0fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:28.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.331+0000 7f89716f4640 1 -- 172.21.15.72:0/4201211034 shutdown_connections 2024-03-22T01:34:28.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.331+0000 7f89716f4640 1 -- 172.21.15.72:0/4201211034 wait complete. 2024-03-22T01:34:28.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.331+0000 7f89716f4640 1 Processor -- start 2024-03-22T01:34:28.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.331+0000 7f89716f4640 1 -- start start 2024-03-22T01:34:28.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.331+0000 7f89716f4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f896c102620 0x7f896c197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:28.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.331+0000 7f89716f4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f896c104f50 0x7f896c1983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:28.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.331+0000 7f89716f4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f896c107880 0x7f896c19f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:28.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.331+0000 7f89716f4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f896c10d810 con 0x7f896c102620 2024-03-22T01:34:28.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.331+0000 7f89716f4640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f896c10d690 con 0x7f896c104f50 2024-03-22T01:34:28.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.331+0000 7f89716f4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f896c10d990 con 0x7f896c107880 2024-03-22T01:34:28.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.332+0000 7f896affd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f896c102620 0x7f896c197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:28.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.332+0000 7f896affd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f896c102620 0x7f896c197eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47986/0 (socket says 172.21.15.72:47986) 2024-03-22T01:34:28.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.332+0000 7f896affd640 1 -- 172.21.15.72:0/3480682007 learned_addr learned my addr 172.21.15.72:0/3480682007 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:34:28.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.332+0000 7f896a7fc640 1 --2- 172.21.15.72:0/3480682007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f896c104f50 0x7f896c1983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:28.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.332+0000 7f896b7fe640 1 --2- 172.21.15.72:0/3480682007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f896c107880 0x7f896c19f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:28.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.332+0000 7f896affd640 1 -- 172.21.15.72:0/3480682007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f896c107880 msgr2=0x7f896c19f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:28.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.332+0000 7f896affd640 1 --2- 172.21.15.72:0/3480682007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f896c107880 0x7f896c19f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:28.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.332+0000 7f896affd640 1 -- 172.21.15.72:0/3480682007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f896c104f50 msgr2=0x7f896c1983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:28.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.332+0000 7f896affd640 1 --2- 172.21.15.72:0/3480682007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f896c104f50 0x7f896c1983f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:28.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.332+0000 7f896affd640 1 -- 172.21.15.72:0/3480682007 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f896c19fad0 con 0x7f896c102620 2024-03-22T01:34:28.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.332+0000 7f896affd640 1 --2- 172.21.15.72:0/3480682007 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f896c102620 0x7f896c197eb0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f896400c420 tx=0x7f8964009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:28.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.332+0000 7f8963fff640 1 -- 172.21.15.72:0/3480682007 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8964003970 con 0x7f896c102620 2024-03-22T01:34:28.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.332+0000 7f89716f4640 1 -- 172.21.15.72:0/3480682007 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f896c19fdc0 con 0x7f896c102620 2024-03-22T01:34:28.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.332+0000 7f8963fff640 1 -- 172.21.15.72:0/3480682007 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8964004020 con 0x7f896c102620 2024-03-22T01:34:28.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.332+0000 7f8963fff640 1 -- 172.21.15.72:0/3480682007 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f896400bad0 con 0x7f896c102620 2024-03-22T01:34:28.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.332+0000 7f89716f4640 1 -- 172.21.15.72:0/3480682007 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f896c1a0220 con 0x7f896c102620 2024-03-22T01:34:28.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.333+0000 7f89716f4640 1 -- 172.21.15.72:0/3480682007 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f896c0fdae0 con 0x7f896c102620 2024-03-22T01:34:28.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.334+0000 7f8963fff640 1 -- 172.21.15.72:0/3480682007 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f896401d020 con 0x7f896c102620 2024-03-22T01:34:28.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.335+0000 7f8963fff640 1 --2- 172.21.15.72:0/3480682007 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f89580a2f20 0x7f89580a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:28.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.335+0000 7f896a7fc640 1 --2- 172.21.15.72:0/3480682007 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f89580a2f20 0x7f89580a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:28.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.335+0000 7f8963fff640 1 -- 172.21.15.72:0/3480682007 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f89640cf540 con 0x7f896c102620 2024-03-22T01:34:28.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.335+0000 7f896a7fc640 1 --2- 172.21.15.72:0/3480682007 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f89580a2f20 0x7f89580a53e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8954009830 tx=0x7f8954034ad0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:28.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.339+0000 7f8963fff640 1 -- 172.21.15.72:0/3480682007 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f896409c130 con 0x7f896c102620 2024-03-22T01:34:28.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.516+0000 7f89716f4640 1 -- 172.21.15.72:0/3480682007 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f896c062830 con 0x7f896c102620 2024-03-22T01:34:28.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.516+0000 7f8963fff640 1 -- 172.21.15.72:0/3480682007 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7f896409ba30 con 0x7f896c102620 2024-03-22T01:34:28.515 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-03-22T01:34:28.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.518+0000 7f89716f4640 1 -- 172.21.15.72:0/3480682007 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f89580a2f20 msgr2=0x7f89580a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:28.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.518+0000 7f89716f4640 1 --2- 172.21.15.72:0/3480682007 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f89580a2f20 0x7f89580a53e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8954009830 tx=0x7f8954034ad0 comp rx=0 tx=0).stop 2024-03-22T01:34:28.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.518+0000 7f89716f4640 1 -- 172.21.15.72:0/3480682007 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f896c102620 msgr2=0x7f896c197eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:28.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.518+0000 7f89716f4640 1 --2- 172.21.15.72:0/3480682007 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f896c102620 0x7f896c197eb0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f896400c420 tx=0x7f8964009f90 comp rx=0 tx=0).stop 2024-03-22T01:34:28.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.518+0000 7f89716f4640 1 -- 172.21.15.72:0/3480682007 shutdown_connections 2024-03-22T01:34:28.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.518+0000 7f89716f4640 1 --2- 172.21.15.72:0/3480682007 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f89580a2f20 0x7f89580a53e0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:28.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.518+0000 7f89716f4640 1 --2- 172.21.15.72:0/3480682007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f896c104f50 0x7f896c1983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:28.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.518+0000 7f89716f4640 1 --2- 172.21.15.72:0/3480682007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f896c107880 0x7f896c19f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:28.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.518+0000 7f89716f4640 1 --2- 172.21.15.72:0/3480682007 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f896c102620 0x7f896c197eb0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:28.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.518+0000 7f89716f4640 1 -- 172.21.15.72:0/3480682007 >> 172.21.15.72:0/3480682007 conn(0x7f896c0fbff0 msgr2=0x7f896c103040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:28.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.519+0000 7f89716f4640 1 -- 172.21.15.72:0/3480682007 shutdown_connections 2024-03-22T01:34:28.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.519+0000 7f89716f4640 1 -- 172.21.15.72:0/3480682007 wait complete. 2024-03-22T01:34:28.529 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-22T01:34:28.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.635+0000 7f41f6b56640 1 -- 172.21.15.72:0/295646541 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f41f0104fb0 msgr2=0x7f41f0105430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:28.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.635+0000 7f41f6b56640 1 --2- 172.21.15.72:0/295646541 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f41f0104fb0 0x7f41f0105430 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f41dc009900 tx=0x7f41dc02bc30 comp rx=0 tx=0).stop 2024-03-22T01:34:28.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.635+0000 7f41f6b56640 1 -- 172.21.15.72:0/295646541 shutdown_connections 2024-03-22T01:34:28.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.635+0000 7f41f6b56640 1 --2- 172.21.15.72:0/295646541 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41f0105970 0x7f41f010c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:28.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.635+0000 7f41f6b56640 1 --2- 172.21.15.72:0/295646541 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f41f0104fb0 0x7f41f0105430 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:28.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.635+0000 7f41f6b56640 1 --2- 172.21.15.72:0/295646541 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41f0103dc0 0x7f41f01041e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:28.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.635+0000 7f41f6b56640 1 -- 172.21.15.72:0/295646541 >> 172.21.15.72:0/295646541 conn(0x7f41f00ff350 msgr2=0x7f41f0101770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:28.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.635+0000 7f41f6b56640 1 -- 172.21.15.72:0/295646541 shutdown_connections 2024-03-22T01:34:28.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.635+0000 7f41f6b56640 1 -- 172.21.15.72:0/295646541 wait complete. 2024-03-22T01:34:28.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.635+0000 7f41f6b56640 1 Processor -- start 2024-03-22T01:34:28.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.635+0000 7f41f6b56640 1 -- start start 2024-03-22T01:34:28.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.636+0000 7f41f6b56640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f41f0103dc0 0x7f41f019a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:28.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.636+0000 7f41f6b56640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41f0104fb0 0x7f41f019a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:28.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.636+0000 7f41f6b56640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41f0105970 0x7f41f01a16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:28.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.636+0000 7f41f6b56640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f41f010fa40 con 0x7f41f0103dc0 2024-03-22T01:34:28.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.636+0000 7f41f6b56640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f41f010f8c0 con 0x7f41f0105970 2024-03-22T01:34:28.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.636+0000 7f41f6b56640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f41f010fbc0 con 0x7f41f0104fb0 2024-03-22T01:34:28.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.636+0000 7f41f48f2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f41f0103dc0 0x7f41f019a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:28.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.636+0000 7f41f48f2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f41f0103dc0 0x7f41f019a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48010/0 (socket says 172.21.15.72:48010) 2024-03-22T01:34:28.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.636+0000 7f41f48f2640 1 -- 172.21.15.72:0/766525942 learned_addr learned my addr 172.21.15.72:0/766525942 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:34:28.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.636+0000 7f41f50f3640 1 --2- 172.21.15.72:0/766525942 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41f0105970 0x7f41f01a16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:28.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.636+0000 7f41effff640 1 --2- 172.21.15.72:0/766525942 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41f0104fb0 0x7f41f019a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:28.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.636+0000 7f41f48f2640 1 -- 172.21.15.72:0/766525942 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41f0104fb0 msgr2=0x7f41f019a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:28.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.636+0000 7f41f48f2640 1 --2- 172.21.15.72:0/766525942 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41f0104fb0 0x7f41f019a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:28.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.636+0000 7f41f48f2640 1 -- 172.21.15.72:0/766525942 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41f0105970 msgr2=0x7f41f01a16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:28.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.636+0000 7f41f48f2640 1 --2- 172.21.15.72:0/766525942 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41f0105970 0x7f41f01a16c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:28.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.636+0000 7f41f48f2640 1 -- 172.21.15.72:0/766525942 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41f01a1d30 con 0x7f41f0103dc0 2024-03-22T01:34:28.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.636+0000 7f41effff640 1 --2- 172.21.15.72:0/766525942 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41f0104fb0 0x7f41f019a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:34:28.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.636+0000 7f41f48f2640 1 --2- 172.21.15.72:0/766525942 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f41f0103dc0 0x7f41f019a110 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f41e400c970 tx=0x7f41e400cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:28.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.636+0000 7f41edffb640 1 -- 172.21.15.72:0/766525942 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41e4004220 con 0x7f41f0103dc0 2024-03-22T01:34:28.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.636+0000 7f41edffb640 1 -- 172.21.15.72:0/766525942 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41e40043c0 con 0x7f41f0103dc0 2024-03-22T01:34:28.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.636+0000 7f41edffb640 1 -- 172.21.15.72:0/766525942 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41e4004a00 con 0x7f41f0103dc0 2024-03-22T01:34:28.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.636+0000 7f41f6b56640 1 -- 172.21.15.72:0/766525942 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41f01a1fc0 con 0x7f41f0103dc0 2024-03-22T01:34:28.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.637+0000 7f41f6b56640 1 -- 172.21.15.72:0/766525942 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41f01a2430 con 0x7f41f0103dc0 2024-03-22T01:34:28.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.637+0000 7f41f6b56640 1 -- 172.21.15.72:0/766525942 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41f010b200 con 0x7f41f0103dc0 2024-03-22T01:34:28.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.638+0000 7f41edffb640 1 -- 172.21.15.72:0/766525942 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f41e40079b0 con 0x7f41f0103dc0 2024-03-22T01:34:28.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.639+0000 7f41edffb640 1 --2- 172.21.15.72:0/766525942 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f41e00a2f20 0x7f41e00a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:28.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.639+0000 7f41effff640 1 --2- 172.21.15.72:0/766525942 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f41e00a2f20 0x7f41e00a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:28.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.639+0000 7f41edffb640 1 -- 172.21.15.72:0/766525942 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f41e40cf3a0 con 0x7f41f0103dc0 2024-03-22T01:34:28.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.639+0000 7f41effff640 1 --2- 172.21.15.72:0/766525942 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f41e00a2f20 0x7f41e00a53e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f41dc006010 tx=0x7f41dc02c8a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:28.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.643+0000 7f41edffb640 1 -- 172.21.15.72:0/766525942 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f41e4014030 con 0x7f41f0103dc0 2024-03-22T01:34:28.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.818+0000 7f41f6b56640 1 -- 172.21.15.72:0/766525942 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f41f002d770 con 0x7f41f0103dc0 2024-03-22T01:34:28.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.818+0000 7f41edffb640 1 -- 172.21.15.72:0/766525942 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) ==== 103+0+11 (secure 0 0 0) 0x7f41e4084e20 con 0x7f41f0103dc0 2024-03-22T01:34:28.817 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-03-22T01:34:28.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.820+0000 7f41f6b56640 1 -- 172.21.15.72:0/766525942 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f41e00a2f20 msgr2=0x7f41e00a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:28.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.820+0000 7f41f6b56640 1 --2- 172.21.15.72:0/766525942 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f41e00a2f20 0x7f41e00a53e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f41dc006010 tx=0x7f41dc02c8a0 comp rx=0 tx=0).stop 2024-03-22T01:34:28.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.820+0000 7f41f6b56640 1 -- 172.21.15.72:0/766525942 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f41f0103dc0 msgr2=0x7f41f019a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:28.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.820+0000 7f41f6b56640 1 --2- 172.21.15.72:0/766525942 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f41f0103dc0 0x7f41f019a110 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f41e400c970 tx=0x7f41e400cd70 comp rx=0 tx=0).stop 2024-03-22T01:34:28.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.820+0000 7f41f6b56640 1 -- 172.21.15.72:0/766525942 shutdown_connections 2024-03-22T01:34:28.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.820+0000 7f41f6b56640 1 --2- 172.21.15.72:0/766525942 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f41e00a2f20 0x7f41e00a53e0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:28.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.820+0000 7f41f6b56640 1 --2- 172.21.15.72:0/766525942 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41f0105970 0x7f41f01a16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:28.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.820+0000 7f41f6b56640 1 --2- 172.21.15.72:0/766525942 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41f0104fb0 0x7f41f019a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:28.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.820+0000 7f41f6b56640 1 --2- 172.21.15.72:0/766525942 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f41f0103dc0 0x7f41f019a110 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:28.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.820+0000 7f41f6b56640 1 -- 172.21.15.72:0/766525942 >> 172.21.15.72:0/766525942 conn(0x7f41f00ff350 msgr2=0x7f41f01083d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:28.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.820+0000 7f41f6b56640 1 -- 172.21.15.72:0/766525942 shutdown_connections 2024-03-22T01:34:28.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.820+0000 7f41f6b56640 1 -- 172.21.15.72:0/766525942 wait complete. 2024-03-22T01:34:28.829 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:28 smithi072 ceph-mon[32414]: pgmap v297: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 383 B/s wr, 0 op/s 2024-03-22T01:34:28.829 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:28 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/1087884979' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:34:28.829 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:28 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/2071956482' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T01:34:28.829 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:28 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/3098315673' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T01:34:28.829 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:28 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/3480682007' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T01:34:28.831 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-22T01:34:28.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:28 smithi089 ceph-mon[35770]: pgmap v297: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 383 B/s wr, 0 op/s 2024-03-22T01:34:28.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:28 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/1087884979' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:34:28.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:28 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/2071956482' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T01:34:28.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:28 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/3098315673' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T01:34:28.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:28 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/3480682007' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T01:34:28.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:28 smithi098 ceph-mon[36468]: pgmap v297: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 383 B/s wr, 0 op/s 2024-03-22T01:34:28.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:28 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/1087884979' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:34:28.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:28 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/2071956482' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T01:34:28.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:28 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/3098315673' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T01:34:28.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:28 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/3480682007' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T01:34:28.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.933+0000 7eff72793640 1 -- 172.21.15.72:0/91739837 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff6c103810 msgr2=0x7eff6c10a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:28.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.933+0000 7eff72793640 1 --2- 172.21.15.72:0/91739837 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff6c103810 0x7eff6c10a0a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7eff64009cd0 tx=0x7eff6402c0f0 comp rx=0 tx=0).stop 2024-03-22T01:34:28.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.933+0000 7eff72793640 1 -- 172.21.15.72:0/91739837 shutdown_connections 2024-03-22T01:34:28.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.933+0000 7eff72793640 1 --2- 172.21.15.72:0/91739837 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff6c103810 0x7eff6c10a0a0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:28.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.933+0000 7eff72793640 1 --2- 172.21.15.72:0/91739837 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff6c102e50 0x7eff6c1032d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:28.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.933+0000 7eff72793640 1 --2- 172.21.15.72:0/91739837 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff6c0ffe80 0x7eff6c1002a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:28.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.933+0000 7eff72793640 1 -- 172.21.15.72:0/91739837 >> 172.21.15.72:0/91739837 conn(0x7eff6c0fb7b0 msgr2=0x7eff6c0fdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:28.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.933+0000 7eff72793640 1 -- 172.21.15.72:0/91739837 shutdown_connections 2024-03-22T01:34:28.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.933+0000 7eff72793640 1 -- 172.21.15.72:0/91739837 wait complete. 2024-03-22T01:34:28.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.934+0000 7eff72793640 1 Processor -- start 2024-03-22T01:34:28.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.934+0000 7eff72793640 1 -- start start 2024-03-22T01:34:28.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.934+0000 7eff72793640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff6c0ffe80 0x7eff6c197db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:28.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.934+0000 7eff72793640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff6c102e50 0x7eff6c1982f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:28.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.934+0000 7eff72793640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff6c103810 0x7eff6c19f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:28.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.934+0000 7eff72793640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7eff6c10d830 con 0x7eff6c0ffe80 2024-03-22T01:34:28.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.934+0000 7eff72793640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7eff6c10d6b0 con 0x7eff6c103810 2024-03-22T01:34:28.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.934+0000 7eff72793640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7eff6c10d9b0 con 0x7eff6c102e50 2024-03-22T01:34:28.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.934+0000 7eff6bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff6c0ffe80 0x7eff6c197db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:28.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.934+0000 7eff6bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff6c0ffe80 0x7eff6c197db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48032/0 (socket says 172.21.15.72:48032) 2024-03-22T01:34:28.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.934+0000 7eff6bfff640 1 -- 172.21.15.72:0/402747336 learned_addr learned my addr 172.21.15.72:0/402747336 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:34:28.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.934+0000 7eff70d30640 1 --2- 172.21.15.72:0/402747336 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff6c103810 0x7eff6c19f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:28.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.935+0000 7eff6bfff640 1 -- 172.21.15.72:0/402747336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff6c102e50 msgr2=0x7eff6c1982f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:28.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.935+0000 7eff6bfff640 1 --2- 172.21.15.72:0/402747336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff6c102e50 0x7eff6c1982f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:28.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.935+0000 7eff6bfff640 1 -- 172.21.15.72:0/402747336 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff6c103810 msgr2=0x7eff6c19f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:28.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.935+0000 7eff6bfff640 1 --2- 172.21.15.72:0/402747336 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff6c103810 0x7eff6c19f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:28.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.935+0000 7eff6bfff640 1 -- 172.21.15.72:0/402747336 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff6c19f9d0 con 0x7eff6c0ffe80 2024-03-22T01:34:28.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.935+0000 7eff70d30640 1 --2- 172.21.15.72:0/402747336 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff6c103810 0x7eff6c19f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:34:28.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.935+0000 7eff6bfff640 1 --2- 172.21.15.72:0/402747336 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff6c0ffe80 0x7eff6c197db0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7eff5c00e9c0 tx=0x7eff5c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:28.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.935+0000 7eff697fa640 1 -- 172.21.15.72:0/402747336 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff5c00cd80 con 0x7eff6c0ffe80 2024-03-22T01:34:28.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.935+0000 7eff697fa640 1 -- 172.21.15.72:0/402747336 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff5c0035f0 con 0x7eff6c0ffe80 2024-03-22T01:34:28.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.935+0000 7eff72793640 1 -- 172.21.15.72:0/402747336 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff6c19fcc0 con 0x7eff6c0ffe80 2024-03-22T01:34:28.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.935+0000 7eff697fa640 1 -- 172.21.15.72:0/402747336 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff5c010690 con 0x7eff6c0ffe80 2024-03-22T01:34:28.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.935+0000 7eff72793640 1 -- 172.21.15.72:0/402747336 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff6c1a01c0 con 0x7eff6c0ffe80 2024-03-22T01:34:28.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.936+0000 7eff72793640 1 -- 172.21.15.72:0/402747336 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff6c108eb0 con 0x7eff6c0ffe80 2024-03-22T01:34:28.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.937+0000 7eff697fa640 1 -- 172.21.15.72:0/402747336 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7eff5c0037a0 con 0x7eff6c0ffe80 2024-03-22T01:34:28.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.937+0000 7eff697fa640 1 --2- 172.21.15.72:0/402747336 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7eff540a2f20 0x7eff540a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:28.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.938+0000 7eff6b7fe640 1 --2- 172.21.15.72:0/402747336 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7eff540a2f20 0x7eff540a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:28.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.938+0000 7eff697fa640 1 -- 172.21.15.72:0/402747336 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7eff5c0cfab0 con 0x7eff6c0ffe80 2024-03-22T01:34:28.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.938+0000 7eff6b7fe640 1 --2- 172.21.15.72:0/402747336 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7eff540a2f20 0x7eff540a53e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7eff6c1992c0 tx=0x7eff60005c70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:28.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:28.942+0000 7eff697fa640 1 -- 172.21.15.72:0/402747336 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7eff5c09c6a0 con 0x7eff6c0ffe80 2024-03-22T01:34:29.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.128+0000 7eff72793640 1 -- 172.21.15.72:0/402747336 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7eff6c062830 con 0x7eff6c0ffe80 2024-03-22T01:34:29.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.128+0000 7eff697fa640 1 -- 172.21.15.72:0/402747336 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v91) ==== 81+0+591 (secure 0 0 0) 0x7eff5c09bfa0 con 0x7eff6c0ffe80 2024-03-22T01:34:29.127 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:0/121087196 2024-03-23T01:27:15.339809+0000 2024-03-22T01:34:29.127 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:0/2451857761 2024-03-23T01:27:15.339809+0000 2024-03-22T01:34:29.127 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:0/1228477995 2024-03-23T01:27:15.339809+0000 2024-03-22T01:34:29.127 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:6801/1513232913 2024-03-23T01:27:15.339809+0000 2024-03-22T01:34:29.127 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:0/3558958264 2024-03-23T01:26:50.074871+0000 2024-03-22T01:34:29.128 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:0/1510187198 2024-03-23T01:26:50.074871+0000 2024-03-22T01:34:29.128 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:6800/3518012916 2024-03-23T01:26:50.074871+0000 2024-03-22T01:34:29.128 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:6801/3518012916 2024-03-23T01:26:50.074871+0000 2024-03-22T01:34:29.128 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:0/1372801041 2024-03-23T01:26:50.074871+0000 2024-03-22T01:34:29.128 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:6800/1513232913 2024-03-23T01:27:15.339809+0000 2024-03-22T01:34:29.128 INFO:teuthology.orchestra.run.smithi072.stderr:listed 10 entries 2024-03-22T01:34:29.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.130+0000 7eff72793640 1 -- 172.21.15.72:0/402747336 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7eff540a2f20 msgr2=0x7eff540a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:29.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.130+0000 7eff72793640 1 --2- 172.21.15.72:0/402747336 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7eff540a2f20 0x7eff540a53e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7eff6c1992c0 tx=0x7eff60005c70 comp rx=0 tx=0).stop 2024-03-22T01:34:29.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.130+0000 7eff72793640 1 -- 172.21.15.72:0/402747336 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff6c0ffe80 msgr2=0x7eff6c197db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:29.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.130+0000 7eff72793640 1 --2- 172.21.15.72:0/402747336 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff6c0ffe80 0x7eff6c197db0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7eff5c00e9c0 tx=0x7eff5c00ee80 comp rx=0 tx=0).stop 2024-03-22T01:34:29.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.130+0000 7eff72793640 1 -- 172.21.15.72:0/402747336 shutdown_connections 2024-03-22T01:34:29.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.130+0000 7eff72793640 1 --2- 172.21.15.72:0/402747336 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7eff540a2f20 0x7eff540a53e0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:29.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.130+0000 7eff72793640 1 --2- 172.21.15.72:0/402747336 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff6c103810 0x7eff6c19f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:29.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.130+0000 7eff72793640 1 --2- 172.21.15.72:0/402747336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff6c102e50 0x7eff6c1982f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:29.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.130+0000 7eff72793640 1 --2- 172.21.15.72:0/402747336 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff6c0ffe80 0x7eff6c197db0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:29.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.130+0000 7eff72793640 1 -- 172.21.15.72:0/402747336 >> 172.21.15.72:0/402747336 conn(0x7eff6c0fb7b0 msgr2=0x7eff6c106080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:29.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.130+0000 7eff72793640 1 -- 172.21.15.72:0/402747336 shutdown_connections 2024-03-22T01:34:29.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.130+0000 7eff72793640 1 -- 172.21.15.72:0/402747336 wait complete. 2024-03-22T01:34:29.140 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-22T01:34:29.140 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-22T01:34:29.156 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T01:34:29.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.298+0000 7fe3e799a640 1 -- 172.21.15.72:0/132701330 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3e0102670 msgr2=0x7fe3e0108f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:29.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.298+0000 7fe3e799a640 1 --2- 172.21.15.72:0/132701330 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3e0102670 0x7fe3e0108f00 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc009900 tx=0x7fe3dc02bc30 comp rx=0 tx=0).stop 2024-03-22T01:34:29.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.298+0000 7fe3e799a640 1 -- 172.21.15.72:0/132701330 shutdown_connections 2024-03-22T01:34:29.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.298+0000 7fe3e799a640 1 --2- 172.21.15.72:0/132701330 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3e0101cb0 0x7fe3e0102130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:29.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.298+0000 7fe3e799a640 1 --2- 172.21.15.72:0/132701330 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3e0102670 0x7fe3e0108f00 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:29.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.298+0000 7fe3e799a640 1 --2- 172.21.15.72:0/132701330 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3e0100ac0 0x7fe3e0100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:29.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.298+0000 7fe3e799a640 1 -- 172.21.15.72:0/132701330 >> 172.21.15.72:0/132701330 conn(0x7fe3e00fc030 msgr2=0x7fe3e00fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:29.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.299+0000 7fe3e799a640 1 -- 172.21.15.72:0/132701330 shutdown_connections 2024-03-22T01:34:29.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.299+0000 7fe3e799a640 1 -- 172.21.15.72:0/132701330 wait complete. 2024-03-22T01:34:29.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.299+0000 7fe3e799a640 1 Processor -- start 2024-03-22T01:34:29.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.299+0000 7fe3e799a640 1 -- start start 2024-03-22T01:34:29.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.299+0000 7fe3e799a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3e0100ac0 0x7fe3e0069a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:29.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.299+0000 7fe3e799a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3e0101cb0 0x7fe3e019ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:29.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.299+0000 7fe3e799a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3e0102670 0x7fe3e019f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:29.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.299+0000 7fe3e799a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fe3e010c720 con 0x7fe3e0100ac0 2024-03-22T01:34:29.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.299+0000 7fe3e799a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe3e010c5a0 con 0x7fe3e0101cb0 2024-03-22T01:34:29.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.299+0000 7fe3e799a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe3e010c8a0 con 0x7fe3e0102670 2024-03-22T01:34:29.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.299+0000 7fe3e5736640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3e0100ac0 0x7fe3e0069a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:29.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.299+0000 7fe3e5736640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3e0100ac0 0x7fe3e0069a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48044/0 (socket says 172.21.15.72:48044) 2024-03-22T01:34:29.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.299+0000 7fe3e5736640 1 -- 172.21.15.72:0/1356962531 learned_addr learned my addr 172.21.15.72:0/1356962531 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:34:29.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.299+0000 7fe3e4f35640 1 --2- 172.21.15.72:0/1356962531 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3e0101cb0 0x7fe3e019ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:29.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.300+0000 7fe3e5f37640 1 --2- 172.21.15.72:0/1356962531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3e0102670 0x7fe3e019f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:29.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.300+0000 7fe3e5736640 1 -- 172.21.15.72:0/1356962531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3e0102670 msgr2=0x7fe3e019f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:29.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.300+0000 7fe3e5736640 1 --2- 172.21.15.72:0/1356962531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3e0102670 0x7fe3e019f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:29.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.300+0000 7fe3e5736640 1 -- 172.21.15.72:0/1356962531 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3e0101cb0 msgr2=0x7fe3e019ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:29.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.300+0000 7fe3e5736640 1 --2- 172.21.15.72:0/1356962531 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3e0101cb0 0x7fe3e019ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:29.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.300+0000 7fe3e5736640 1 -- 172.21.15.72:0/1356962531 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3e019f990 con 0x7fe3e0100ac0 2024-03-22T01:34:29.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.300+0000 7fe3e5736640 1 --2- 172.21.15.72:0/1356962531 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3e0100ac0 0x7fe3e0069a30 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fe3d800ccb0 tx=0x7fe3d800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:29.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.300+0000 7fe3d67fc640 1 -- 172.21.15.72:0/1356962531 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3d8003b70 con 0x7fe3e0100ac0 2024-03-22T01:34:29.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.300+0000 7fe3d67fc640 1 -- 172.21.15.72:0/1356962531 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3d800ae50 con 0x7fe3e0100ac0 2024-03-22T01:34:29.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.300+0000 7fe3d67fc640 1 -- 172.21.15.72:0/1356962531 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3d800f810 con 0x7fe3e0100ac0 2024-03-22T01:34:29.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.300+0000 7fe3e799a640 1 -- 172.21.15.72:0/1356962531 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3e019fc80 con 0x7fe3e0100ac0 2024-03-22T01:34:29.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.300+0000 7fe3e799a640 1 -- 172.21.15.72:0/1356962531 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3e01a0180 con 0x7fe3e0100ac0 2024-03-22T01:34:29.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.301+0000 7fe3e799a640 1 -- 172.21.15.72:0/1356962531 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3e0107d10 con 0x7fe3e0100ac0 2024-03-22T01:34:29.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.302+0000 7fe3d67fc640 1 -- 172.21.15.72:0/1356962531 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7fe3d800f9b0 con 0x7fe3e0100ac0 2024-03-22T01:34:29.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.302+0000 7fe3d67fc640 1 --2- 172.21.15.72:0/1356962531 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe3cc0a2f20 0x7fe3cc0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:29.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.303+0000 7fe3d67fc640 1 -- 172.21.15.72:0/1356962531 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fe3d80cf750 con 0x7fe3e0100ac0 2024-03-22T01:34:29.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.303+0000 7fe3e4f35640 1 --2- 172.21.15.72:0/1356962531 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe3cc0a2f20 0x7fe3cc0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:29.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.303+0000 7fe3e4f35640 1 --2- 172.21.15.72:0/1356962531 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe3cc0a2f20 0x7fe3cc0a53e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe3d00095f0 tx=0x7fe3d0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:29.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.307+0000 7fe3d67fc640 1 -- 172.21.15.72:0/1356962531 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe3d8014070 con 0x7fe3e0100ac0 2024-03-22T01:34:29.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.483+0000 7fe3e799a640 1 -- 172.21.15.72:0/1356962531 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe3e002d660 con 0x7fe3e0100ac0 2024-03-22T01:34:29.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.485+0000 7fe3d67fc640 1 -- 172.21.15.72:0/1356962531 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24045 (secure 0 0 0) 0x7fe3d809c340 con 0x7fe3e0100ac0 2024-03-22T01:34:29.484 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:34:29.484 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":91,"fsid":"022a99bc-e7eb-11ee-95cd-87774f69a715","created":"2024-03-22T01:26:23.893672+0000","modified":"2024-03-22T01:34:19.866367+0000","last_up_change":"2024-03-22T01:33:25.223684+0000","last_in_change":"2024-03-22T01:33:05.010331+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T01:29:37.438503+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-22T01:34:12.610422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-22T01:34:13.761686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-22T01:34:17.507039+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"16c15bb0-db92-4191-9063-38b4890e81e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6803","nonce":661906791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6805","nonce":661906791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6809","nonce":661906791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6807","nonce":661906791}]},"public_addr":"172.21.15.72:6803/661906791","cluster_addr":"172.21.15.72:6805/661906791","heartbeat_back_addr":"172.21.15.72:6809/661906791","heartbeat_front_addr":"172.21.15.72:6807/661906791","state":["exists","up"]},{"osd":1,"uuid":"981290ba-97dc-496b-aa22-d51704184b10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6801","nonce":2870590550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6803","nonce":2870590550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6807","nonce":2870590550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6805","nonce":2870590550}]},"public_addr":"172.21.15.89:6801/2870590550","cluster_addr":"172.21.15.89:6803/2870590550","heartbeat_back_addr":"172.21.15.89:6807/2870590550","heartbeat_front_addr":"172.21.15.89:6805/2870590550","state":["exists","up"]},{"osd":2,"uuid":"f07ed658-f545-46a1-8edd-8b1d5d49885c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6801","nonce":3865778118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6803","nonce":3865778118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6807","nonce":3865778118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6805","nonce":3865778118}]},"public_addr":"172.21.15.98:6801/3865778118","cluster_addr":"172.21.15.98:6803/3865778118","heartbeat_back_addr":"172.21.15.98:6807/3865778118","heartbeat_front_addr":"172.21.15.98:6805/3865778118","state":["exists","up"]},{"osd":3,"uuid":"ccc4bc40-2c48-46d2-8548-0c05b44f9f1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6811","nonce":1865951395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6813","nonce":1865951395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6817","nonce":1865951395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6815","nonce":1865951395}]},"public_addr":"172.21.15.72:6811/1865951395","cluster_addr":"172.21.15.72:6813/1865951395","heartbeat_back_addr":"172.21.15.72:6817/1865951395","heartbeat_front_addr":"172.21.15.72:6815/1865951395","state":["exists","up"]},{"osd":4,"uuid":"1812439f-810a-4ed2-9395-c00c9a085a13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6809","nonce":2683760795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6811","nonce":2683760795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6815","nonce":2683760795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6813","nonce":2683760795}]},"public_addr":"172.21.15.89:6809/2683760795","cluster_addr":"172.21.15.89:6811/2683760795","heartbeat_back_addr":"172.21.15.89:6815/2683760795","heartbeat_front_addr":"172.21.15.89:6813/2683760795","state":["exists","up"]},{"osd":5,"uuid":"e78954d8-f4e3-4d58-8536-d2d59c843386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6809","nonce":2635838572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6811","nonce":2635838572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6815","nonce":2635838572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6813","nonce":2635838572}]},"public_addr":"172.21.15.98:6809/2635838572","cluster_addr":"172.21.15.98:6811/2635838572","heartbeat_back_addr":"172.21.15.98:6815/2635838572","heartbeat_front_addr":"172.21.15.98:6813/2635838572","state":["exists","up"]},{"osd":6,"uuid":"dc6c8be2-b64c-4959-8015-7b0ea6d25599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6819","nonce":3158982661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6821","nonce":3158982661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6825","nonce":3158982661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6823","nonce":3158982661}]},"public_addr":"172.21.15.72:6819/3158982661","cluster_addr":"172.21.15.72:6821/3158982661","heartbeat_back_addr":"172.21.15.72:6825/3158982661","heartbeat_front_addr":"172.21.15.72:6823/3158982661","state":["exists","up"]},{"osd":7,"uuid":"7a73469f-cf3a-4306-ae93-ea46e21ebbbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6817","nonce":3075913234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6819","nonce":3075913234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6823","nonce":3075913234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6821","nonce":3075913234}]},"public_addr":"172.21.15.89:6817/3075913234","cluster_addr":"172.21.15.89:6819/3075913234","heartbeat_back_addr":"172.21.15.89:6823/3075913234","heartbeat_front_addr":"172.21.15.89:6821/3075913234","state":["exists","up"]},{"osd":8,"uuid":"c4b13bee-ea4d-4344-95fb-2622ecbb8196","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6817","nonce":1813255577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6819","nonce":1813255577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6823","nonce":1813255577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6821","nonce":1813255577}]},"public_addr":"172.21.15.98:6817/1813255577","cluster_addr":"172.21.15.98:6819/1813255577","heartbeat_back_addr":"172.21.15.98:6823/1813255577","heartbeat_front_addr":"172.21.15.98:6821/1813255577","state":["exists","up"]},{"osd":9,"uuid":"eefb0a44-4a36-4e55-9d04-0501bf718db6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6827","nonce":3377749206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6829","nonce":3377749206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6832","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6833","nonce":3377749206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6831","nonce":3377749206}]},"public_addr":"172.21.15.72:6827/3377749206","cluster_addr":"172.21.15.72:6829/3377749206","heartbeat_back_addr":"172.21.15.72:6833/3377749206","heartbeat_front_addr":"172.21.15.72:6831/3377749206","state":["exists","up"]},{"osd":10,"uuid":"8741d25f-40dd-40d7-b040-5cc5e411eb98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6825","nonce":885058786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6827","nonce":885058786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6831","nonce":885058786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6829","nonce":885058786}]},"public_addr":"172.21.15.89:6825/885058786","cluster_addr":"172.21.15.89:6827/885058786","heartbeat_back_addr":"172.21.15.89:6831/885058786","heartbeat_front_addr":"172.21.15.89:6829/885058786","state":["exists","up"]},{"osd":11,"uuid":"8d02e957-eda4-4184-b5c6-5eb3b8b38060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6825","nonce":1517095838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6827","nonce":1517095838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6831","nonce":1517095838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6829","nonce":1517095838}]},"public_addr":"172.21.15.98:6825/1517095838","cluster_addr":"172.21.15.98:6827/1517095838","heartbeat_back_addr":"172.21.15.98:6831/1517095838","heartbeat_front_addr":"172.21.15.98:6829/1517095838","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:28:47.307652+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:11.037135+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:34.619188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:59.350738+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:30:23.787337+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:30:48.994154+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:31:13.690350+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:31:39.628116+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:04.723912+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:30.689741+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:57.229508+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:33:23.154537+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:0/121087196":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/2451857761":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/1228477995":"2024-03-23T01:27:15.339809+0000","172.21.15.72:6801/1513232913":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/3558958264":"2024-03-23T01:26:50.074871+0000","172.21.15.72:0/1510187198":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6800/3518012916":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6801/3518012916":"2024-03-23T01:26:50.074871+0000","172.21.15.72:0/1372801041":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6800/1513232913":"2024-03-23T01:27:15.339809+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T01:34:29.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.487+0000 7fe3e799a640 1 -- 172.21.15.72:0/1356962531 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe3cc0a2f20 msgr2=0x7fe3cc0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:29.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.487+0000 7fe3e799a640 1 --2- 172.21.15.72:0/1356962531 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe3cc0a2f20 0x7fe3cc0a53e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe3d00095f0 tx=0x7fe3d0005f90 comp rx=0 tx=0).stop 2024-03-22T01:34:29.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.487+0000 7fe3e799a640 1 -- 172.21.15.72:0/1356962531 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3e0100ac0 msgr2=0x7fe3e0069a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:29.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.487+0000 7fe3e799a640 1 --2- 172.21.15.72:0/1356962531 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3e0100ac0 0x7fe3e0069a30 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fe3d800ccb0 tx=0x7fe3d800a620 comp rx=0 tx=0).stop 2024-03-22T01:34:29.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.487+0000 7fe3e799a640 1 -- 172.21.15.72:0/1356962531 shutdown_connections 2024-03-22T01:34:29.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.487+0000 7fe3e799a640 1 --2- 172.21.15.72:0/1356962531 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe3cc0a2f20 0x7fe3cc0a53e0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:29.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.487+0000 7fe3e799a640 1 --2- 172.21.15.72:0/1356962531 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3e0101cb0 0x7fe3e019ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:29.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.487+0000 7fe3e799a640 1 --2- 172.21.15.72:0/1356962531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3e0102670 0x7fe3e019f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:29.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.487+0000 7fe3e799a640 1 --2- 172.21.15.72:0/1356962531 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3e0100ac0 0x7fe3e0069a30 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:29.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.487+0000 7fe3e799a640 1 -- 172.21.15.72:0/1356962531 >> 172.21.15.72:0/1356962531 conn(0x7fe3e00fc030 msgr2=0x7fe3e0104ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:29.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.487+0000 7fe3e799a640 1 -- 172.21.15.72:0/1356962531 shutdown_connections 2024-03-22T01:34:29.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.487+0000 7fe3e799a640 1 -- 172.21.15.72:0/1356962531 wait complete. 2024-03-22T01:34:29.498 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-22T01:29:37.438503+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-22T01:34:12.610422+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-22T01:34:13.761686+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-22T01:34:17.507039+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-22T01:34:29.498 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-22T01:34:29.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.604+0000 7f85921f6640 1 -- 172.21.15.72:0/3752508504 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f858c100ab0 msgr2=0x7f858c100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:29.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.604+0000 7f85921f6640 1 --2- 172.21.15.72:0/3752508504 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f858c100ab0 0x7f858c100ed0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f857c009900 tx=0x7f857c02bc30 comp rx=0 tx=0).stop 2024-03-22T01:34:29.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.604+0000 7f85921f6640 1 -- 172.21.15.72:0/3752508504 shutdown_connections 2024-03-22T01:34:29.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.604+0000 7f85921f6640 1 --2- 172.21.15.72:0/3752508504 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f858c102660 0x7f858c108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:29.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.604+0000 7f85921f6640 1 --2- 172.21.15.72:0/3752508504 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f858c101ca0 0x7f858c102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:29.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.604+0000 7f85921f6640 1 --2- 172.21.15.72:0/3752508504 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f858c100ab0 0x7f858c100ed0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:29.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.604+0000 7f85921f6640 1 -- 172.21.15.72:0/3752508504 >> 172.21.15.72:0/3752508504 conn(0x7f858c0fc020 msgr2=0x7f858c0fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:29.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.604+0000 7f85921f6640 1 -- 172.21.15.72:0/3752508504 shutdown_connections 2024-03-22T01:34:29.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.604+0000 7f85921f6640 1 -- 172.21.15.72:0/3752508504 wait complete. 2024-03-22T01:34:29.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.605+0000 7f85921f6640 1 Processor -- start 2024-03-22T01:34:29.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.605+0000 7f85921f6640 1 -- start start 2024-03-22T01:34:29.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.605+0000 7f85921f6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f858c100ab0 0x7f858c197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:29.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.605+0000 7f85921f6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f858c101ca0 0x7f858c198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:29.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.605+0000 7f85921f6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f858c102660 0x7f858c19f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:29.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.605+0000 7f85921f6640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f858c10d890 con 0x7f858c102660 2024-03-22T01:34:29.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.605+0000 7f85921f6640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f858c10d710 con 0x7f858c101ca0 2024-03-22T01:34:29.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.605+0000 7f85921f6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f858c10da10 con 0x7f858c100ab0 2024-03-22T01:34:29.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.605+0000 7f858bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f858c102660 0x7f858c19f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:29.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.605+0000 7f858bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f858c102660 0x7f858c19f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48068/0 (socket says 172.21.15.72:48068) 2024-03-22T01:34:29.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.605+0000 7f858bfff640 1 -- 172.21.15.72:0/1869260608 learned_addr learned my addr 172.21.15.72:0/1869260608 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:34:29.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.605+0000 7f858affd640 1 --2- 172.21.15.72:0/1869260608 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f858c101ca0 0x7f858c198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:29.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.605+0000 7f858bfff640 1 -- 172.21.15.72:0/1869260608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f858c100ab0 msgr2=0x7f858c197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:29.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.605+0000 7f858b7fe640 1 --2- 172.21.15.72:0/1869260608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f858c100ab0 0x7f858c197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:29.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.605+0000 7f858bfff640 1 --2- 172.21.15.72:0/1869260608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f858c100ab0 0x7f858c197f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:29.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.605+0000 7f858bfff640 1 -- 172.21.15.72:0/1869260608 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f858c101ca0 msgr2=0x7f858c198460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:29.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.605+0000 7f858bfff640 1 --2- 172.21.15.72:0/1869260608 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f858c101ca0 0x7f858c198460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:29.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.605+0000 7f858bfff640 1 -- 172.21.15.72:0/1869260608 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f858c19fb40 con 0x7f858c102660 2024-03-22T01:34:29.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.606+0000 7f858bfff640 1 --2- 172.21.15.72:0/1869260608 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f858c102660 0x7f858c19f4d0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f858400eb10 tx=0x7f858400ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:29.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.606+0000 7f8588ff9640 1 -- 172.21.15.72:0/1869260608 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f858400ce20 con 0x7f858c102660 2024-03-22T01:34:29.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.606+0000 7f8588ff9640 1 -- 172.21.15.72:0/1869260608 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8584005020 con 0x7f858c102660 2024-03-22T01:34:29.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.606+0000 7f85921f6640 1 -- 172.21.15.72:0/1869260608 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f858c19fe30 con 0x7f858c102660 2024-03-22T01:34:29.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.606+0000 7f8588ff9640 1 -- 172.21.15.72:0/1869260608 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8584010690 con 0x7f858c102660 2024-03-22T01:34:29.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.606+0000 7f85921f6640 1 -- 172.21.15.72:0/1869260608 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f858c1a02a0 con 0x7f858c102660 2024-03-22T01:34:29.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.607+0000 7f85921f6640 1 -- 172.21.15.72:0/1869260608 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f858c107e70 con 0x7f858c102660 2024-03-22T01:34:29.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.608+0000 7f8588ff9640 1 -- 172.21.15.72:0/1869260608 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f8584005540 con 0x7f858c102660 2024-03-22T01:34:29.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.608+0000 7f8588ff9640 1 --2- 172.21.15.72:0/1869260608 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f85740a2f20 0x7f85740a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:29.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.608+0000 7f858b7fe640 1 --2- 172.21.15.72:0/1869260608 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f85740a2f20 0x7f85740a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:29.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.609+0000 7f8588ff9640 1 -- 172.21.15.72:0/1869260608 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f85840cfc40 con 0x7f858c102660 2024-03-22T01:34:29.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.609+0000 7f858b7fe640 1 --2- 172.21.15.72:0/1869260608 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f85740a2f20 0x7f85740a53e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f857c02c8e0 tx=0x7f857c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:29.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.613+0000 7f8588ff9640 1 -- 172.21.15.72:0/1869260608 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f858409c830 con 0x7f858c102660 2024-03-22T01:34:29.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.788+0000 7f85921f6640 1 -- 172.21.15.72:0/1869260608 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f858c062830 con 0x7f858c102660 2024-03-22T01:34:29.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.788+0000 7f8588ff9640 1 -- 172.21.15.72:0/1869260608 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7f858409c130 con 0x7f858c102660 2024-03-22T01:34:29.787 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-22T01:34:29.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.790+0000 7f85921f6640 1 -- 172.21.15.72:0/1869260608 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f85740a2f20 msgr2=0x7f85740a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:29.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.790+0000 7f85921f6640 1 --2- 172.21.15.72:0/1869260608 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f85740a2f20 0x7f85740a53e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f857c02c8e0 tx=0x7f857c005fd0 comp rx=0 tx=0).stop 2024-03-22T01:34:29.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.790+0000 7f85921f6640 1 -- 172.21.15.72:0/1869260608 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f858c102660 msgr2=0x7f858c19f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:29.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.790+0000 7f85921f6640 1 --2- 172.21.15.72:0/1869260608 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f858c102660 0x7f858c19f4d0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f858400eb10 tx=0x7f858400ef10 comp rx=0 tx=0).stop 2024-03-22T01:34:29.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.790+0000 7f85921f6640 1 -- 172.21.15.72:0/1869260608 shutdown_connections 2024-03-22T01:34:29.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.790+0000 7f85921f6640 1 --2- 172.21.15.72:0/1869260608 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f85740a2f20 0x7f85740a53e0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:29.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.790+0000 7f85921f6640 1 --2- 172.21.15.72:0/1869260608 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f858c101ca0 0x7f858c198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:29.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.790+0000 7f85921f6640 1 --2- 172.21.15.72:0/1869260608 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f858c102660 0x7f858c19f4d0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:29.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.790+0000 7f85921f6640 1 --2- 172.21.15.72:0/1869260608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f858c100ab0 0x7f858c197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:29.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.790+0000 7f85921f6640 1 -- 172.21.15.72:0/1869260608 >> 172.21.15.72:0/1869260608 conn(0x7f858c0fc020 msgr2=0x7f858c105040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:29.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.790+0000 7f85921f6640 1 -- 172.21.15.72:0/1869260608 shutdown_connections 2024-03-22T01:34:29.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.790+0000 7f85921f6640 1 -- 172.21.15.72:0/1869260608 wait complete. 2024-03-22T01:34:29.798 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:29 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/766525942' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T01:34:29.799 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:29 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/402747336' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-22T01:34:29.799 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:29 smithi072 ceph-mon[32414]: pgmap v298: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 309 B/s wr, 0 op/s 2024-03-22T01:34:29.799 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:29 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/1356962531' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:34:29.800 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-22T01:34:29.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:29 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/766525942' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T01:34:29.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:29 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/402747336' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-22T01:34:29.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:29 smithi089 ceph-mon[35770]: pgmap v298: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 309 B/s wr, 0 op/s 2024-03-22T01:34:29.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:29 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/1356962531' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:34:29.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.905+0000 7f3745a55640 1 -- 172.21.15.72:0/3677576853 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3740101c90 msgr2=0x7f3740102110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:29.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.905+0000 7f3745a55640 1 --2- 172.21.15.72:0/3677576853 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3740101c90 0x7f3740102110 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3728009860 tx=0x7f372802bb70 comp rx=0 tx=0).stop 2024-03-22T01:34:29.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.905+0000 7f3745a55640 1 -- 172.21.15.72:0/3677576853 shutdown_connections 2024-03-22T01:34:29.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.905+0000 7f3745a55640 1 --2- 172.21.15.72:0/3677576853 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3740101c90 0x7f3740102110 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:29.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.905+0000 7f3745a55640 1 --2- 172.21.15.72:0/3677576853 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3740102650 0x7f3740108ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:29.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.905+0000 7f3745a55640 1 --2- 172.21.15.72:0/3677576853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3740100aa0 0x7f3740100ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:29.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.905+0000 7f3745a55640 1 -- 172.21.15.72:0/3677576853 >> 172.21.15.72:0/3677576853 conn(0x7f37400fbff0 msgr2=0x7f37400fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:29.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.905+0000 7f3745a55640 1 -- 172.21.15.72:0/3677576853 shutdown_connections 2024-03-22T01:34:29.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.905+0000 7f3745a55640 1 -- 172.21.15.72:0/3677576853 wait complete. 2024-03-22T01:34:29.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.906+0000 7f3745a55640 1 Processor -- start 2024-03-22T01:34:29.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.906+0000 7f3745a55640 1 -- start start 2024-03-22T01:34:29.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.906+0000 7f3745a55640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3740100aa0 0x7f3740197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:29.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.906+0000 7f3745a55640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3740101c90 0x7f3740198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:29.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.906+0000 7f3745a55640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3740102650 0x7f374019f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:29.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.906+0000 7f3745a55640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f374010c700 con 0x7f3740101c90 2024-03-22T01:34:29.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.906+0000 7f3745a55640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f374010c580 con 0x7f3740100aa0 2024-03-22T01:34:29.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.906+0000 7f3745a55640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f374010c880 con 0x7f3740102650 2024-03-22T01:34:29.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.906+0000 7f373e7fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3740101c90 0x7f3740198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:29.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.906+0000 7f373e7fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3740101c90 0x7f3740198460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48086/0 (socket says 172.21.15.72:48086) 2024-03-22T01:34:29.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.906+0000 7f373e7fc640 1 -- 172.21.15.72:0/3953082493 learned_addr learned my addr 172.21.15.72:0/3953082493 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:34:29.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.906+0000 7f373effd640 1 --2- 172.21.15.72:0/3953082493 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3740100aa0 0x7f3740197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:29.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.906+0000 7f373f7fe640 1 --2- 172.21.15.72:0/3953082493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3740102650 0x7f374019f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:29.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.907+0000 7f373e7fc640 1 -- 172.21.15.72:0/3953082493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3740102650 msgr2=0x7f374019f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:29.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.907+0000 7f373e7fc640 1 --2- 172.21.15.72:0/3953082493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3740102650 0x7f374019f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:29.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.907+0000 7f373e7fc640 1 -- 172.21.15.72:0/3953082493 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3740100aa0 msgr2=0x7f3740197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:29.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.907+0000 7f373e7fc640 1 --2- 172.21.15.72:0/3953082493 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3740100aa0 0x7f3740197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:29.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.907+0000 7f373e7fc640 1 -- 172.21.15.72:0/3953082493 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f374019faf0 con 0x7f3740101c90 2024-03-22T01:34:29.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.907+0000 7f373f7fe640 1 --2- 172.21.15.72:0/3953082493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3740102650 0x7f374019f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T01:34:29.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.907+0000 7f373e7fc640 1 --2- 172.21.15.72:0/3953082493 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3740101c90 0x7f3740198460 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f37280349e0 tx=0x7f37280042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:29.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.907+0000 7f3744a53640 1 -- 172.21.15.72:0/3953082493 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3728041030 con 0x7f3740101c90 2024-03-22T01:34:29.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.907+0000 7f3745a55640 1 -- 172.21.15.72:0/3953082493 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f374019fd80 con 0x7f3740101c90 2024-03-22T01:34:29.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.907+0000 7f3744a53640 1 -- 172.21.15.72:0/3953082493 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37280347b0 con 0x7f3740101c90 2024-03-22T01:34:29.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.907+0000 7f3745a55640 1 -- 172.21.15.72:0/3953082493 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37401a0210 con 0x7f3740101c90 2024-03-22T01:34:29.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.907+0000 7f3744a53640 1 -- 172.21.15.72:0/3953082493 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3728039040 con 0x7f3740101c90 2024-03-22T01:34:29.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.908+0000 7f3745a55640 1 -- 172.21.15.72:0/3953082493 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37401a0490 con 0x7f3740101c90 2024-03-22T01:34:29.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.909+0000 7f3744a53640 1 -- 172.21.15.72:0/3953082493 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f3728044070 con 0x7f3740101c90 2024-03-22T01:34:29.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.910+0000 7f3744a53640 1 --2- 172.21.15.72:0/3953082493 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f372c0a2f20 0x7f372c0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:29.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.910+0000 7f373effd640 1 --2- 172.21.15.72:0/3953082493 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f372c0a2f20 0x7f372c0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:29.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.910+0000 7f3744a53640 1 -- 172.21.15.72:0/3953082493 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f37280f0c50 con 0x7f3740101c90 2024-03-22T01:34:29.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.910+0000 7f373effd640 1 --2- 172.21.15.72:0/3953082493 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f372c0a2f20 0x7f372c0a53e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f3738005fd0 tx=0x7f373800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:29.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:29.914+0000 7f3744a53640 1 -- 172.21.15.72:0/3953082493 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f37280f6020 con 0x7f3740101c90 2024-03-22T01:34:29.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:29 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/766525942' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T01:34:29.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:29 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/402747336' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-22T01:34:29.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:29 smithi098 ceph-mon[36468]: pgmap v298: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 309 B/s wr, 0 op/s 2024-03-22T01:34:29.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:29 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/1356962531' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:34:30.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.090+0000 7f3745a55640 1 -- 172.21.15.72:0/3953082493 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f37401a0660 con 0x7f3740101c90 2024-03-22T01:34:30.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.090+0000 7f3744a53640 1 -- 172.21.15.72:0/3953082493 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7f37280bd840 con 0x7f3740101c90 2024-03-22T01:34:30.089 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-03-22T01:34:30.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.092+0000 7f3745a55640 1 -- 172.21.15.72:0/3953082493 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f372c0a2f20 msgr2=0x7f372c0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:30.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.092+0000 7f3745a55640 1 --2- 172.21.15.72:0/3953082493 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f372c0a2f20 0x7f372c0a53e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f3738005fd0 tx=0x7f373800b410 comp rx=0 tx=0).stop 2024-03-22T01:34:30.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.092+0000 7f3745a55640 1 -- 172.21.15.72:0/3953082493 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3740101c90 msgr2=0x7f3740198460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:30.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.092+0000 7f3745a55640 1 --2- 172.21.15.72:0/3953082493 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3740101c90 0x7f3740198460 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f37280349e0 tx=0x7f37280042f0 comp rx=0 tx=0).stop 2024-03-22T01:34:30.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.092+0000 7f3745a55640 1 -- 172.21.15.72:0/3953082493 shutdown_connections 2024-03-22T01:34:30.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.092+0000 7f3745a55640 1 --2- 172.21.15.72:0/3953082493 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f372c0a2f20 0x7f372c0a53e0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:30.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.092+0000 7f3745a55640 1 --2- 172.21.15.72:0/3953082493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3740102650 0x7f374019f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:30.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.092+0000 7f3745a55640 1 --2- 172.21.15.72:0/3953082493 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3740101c90 0x7f3740198460 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:30.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.092+0000 7f3745a55640 1 --2- 172.21.15.72:0/3953082493 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3740100aa0 0x7f3740197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:30.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.092+0000 7f3745a55640 1 -- 172.21.15.72:0/3953082493 >> 172.21.15.72:0/3953082493 conn(0x7f37400fbff0 msgr2=0x7f37400fdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:30.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.093+0000 7f3745a55640 1 -- 172.21.15.72:0/3953082493 shutdown_connections 2024-03-22T01:34:30.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.093+0000 7f3745a55640 1 -- 172.21.15.72:0/3953082493 wait complete. 2024-03-22T01:34:30.103 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-22T01:34:30.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.213+0000 7fc20b98a640 1 -- 172.21.15.72:0/3067123421 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc2041020c0 msgr2=0x7fc2041024e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:30.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.213+0000 7fc20b98a640 1 --2- 172.21.15.72:0/3067123421 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc2041020c0 0x7fc2041024e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fc1f8009900 tx=0x7fc1f802bc30 comp rx=0 tx=0).stop 2024-03-22T01:34:30.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.213+0000 7fc20b98a640 1 -- 172.21.15.72:0/3067123421 shutdown_connections 2024-03-22T01:34:30.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.213+0000 7fc20b98a640 1 --2- 172.21.15.72:0/3067123421 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc204100d10 0x7fc204101190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:30.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.213+0000 7fc20b98a640 1 --2- 172.21.15.72:0/3067123421 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2041016d0 0x7fc204108f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:30.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.213+0000 7fc20b98a640 1 --2- 172.21.15.72:0/3067123421 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc2041020c0 0x7fc2041024e0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:30.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.213+0000 7fc20b98a640 1 -- 172.21.15.72:0/3067123421 >> 172.21.15.72:0/3067123421 conn(0x7fc2040fb7a0 msgr2=0x7fc2040fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:30.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.213+0000 7fc20b98a640 1 -- 172.21.15.72:0/3067123421 shutdown_connections 2024-03-22T01:34:30.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.213+0000 7fc20b98a640 1 -- 172.21.15.72:0/3067123421 wait complete. 2024-03-22T01:34:30.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.213+0000 7fc20b98a640 1 Processor -- start 2024-03-22T01:34:30.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.213+0000 7fc20b98a640 1 -- start start 2024-03-22T01:34:30.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.213+0000 7fc20b98a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc204100d10 0x7fc204197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:30.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.214+0000 7fc20b98a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2041016d0 0x7fc204198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:30.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.214+0000 7fc20b98a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2041020c0 0x7fc20419f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:30.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.214+0000 7fc20b98a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fc20410d8b0 con 0x7fc204100d10 2024-03-22T01:34:30.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.214+0000 7fc20b98a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc20410d730 con 0x7fc2041020c0 2024-03-22T01:34:30.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.214+0000 7fc20b98a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc20410da30 con 0x7fc2041016d0 2024-03-22T01:34:30.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.214+0000 7fc209726640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc204100d10 0x7fc204197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:30.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.214+0000 7fc209726640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc204100d10 0x7fc204197f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48102/0 (socket says 172.21.15.72:48102) 2024-03-22T01:34:30.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.214+0000 7fc209726640 1 -- 172.21.15.72:0/176679730 learned_addr learned my addr 172.21.15.72:0/176679730 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:34:30.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.214+0000 7fc209f27640 1 --2- 172.21.15.72:0/176679730 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2041020c0 0x7fc20419f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:30.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.214+0000 7fc208f25640 1 --2- 172.21.15.72:0/176679730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2041016d0 0x7fc204198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:30.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.214+0000 7fc209726640 1 -- 172.21.15.72:0/176679730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2041016d0 msgr2=0x7fc204198480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:30.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.214+0000 7fc209726640 1 --2- 172.21.15.72:0/176679730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2041016d0 0x7fc204198480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:30.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.214+0000 7fc209726640 1 -- 172.21.15.72:0/176679730 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2041020c0 msgr2=0x7fc20419f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:30.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.214+0000 7fc209726640 1 --2- 172.21.15.72:0/176679730 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2041020c0 0x7fc20419f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:30.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.214+0000 7fc209726640 1 -- 172.21.15.72:0/176679730 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc20419fb60 con 0x7fc204100d10 2024-03-22T01:34:30.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.214+0000 7fc208f25640 1 --2- 172.21.15.72:0/176679730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2041016d0 0x7fc204198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T01:34:30.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.214+0000 7fc209726640 1 --2- 172.21.15.72:0/176679730 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc204100d10 0x7fc204197f40 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fc1f802efd0 tx=0x7fc1f802c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:30.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.214+0000 7fc1f67fc640 1 -- 172.21.15.72:0/176679730 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1f8041070 con 0x7fc204100d10 2024-03-22T01:34:30.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.214+0000 7fc1f67fc640 1 -- 172.21.15.72:0/176679730 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1f8004450 con 0x7fc204100d10 2024-03-22T01:34:30.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.214+0000 7fc20b98a640 1 -- 172.21.15.72:0/176679730 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc20419fdf0 con 0x7fc204100d10 2024-03-22T01:34:30.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.214+0000 7fc1f67fc640 1 -- 172.21.15.72:0/176679730 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1f8034440 con 0x7fc204100d10 2024-03-22T01:34:30.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.214+0000 7fc20b98a640 1 -- 172.21.15.72:0/176679730 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2041a0220 con 0x7fc204100d10 2024-03-22T01:34:30.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.215+0000 7fc20b98a640 1 -- 172.21.15.72:0/176679730 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc204107f20 con 0x7fc204100d10 2024-03-22T01:34:30.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.216+0000 7fc1f67fc640 1 -- 172.21.15.72:0/176679730 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7fc1f8044070 con 0x7fc204100d10 2024-03-22T01:34:30.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.217+0000 7fc1f67fc640 1 --2- 172.21.15.72:0/176679730 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fc1f00a2f20 0x7fc1f00a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:30.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.217+0000 7fc208f25640 1 --2- 172.21.15.72:0/176679730 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fc1f00a2f20 0x7fc1f00a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:30.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.217+0000 7fc1f67fc640 1 -- 172.21.15.72:0/176679730 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fc1f80f1270 con 0x7fc204100d10 2024-03-22T01:34:30.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.217+0000 7fc208f25640 1 --2- 172.21.15.72:0/176679730 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fc1f00a2f20 0x7fc1f00a53e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc204199450 tx=0x7fc200005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:30.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.222+0000 7fc1f67fc640 1 -- 172.21.15.72:0/176679730 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc1f80bdea0 con 0x7fc204100d10 2024-03-22T01:34:30.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.397+0000 7fc20b98a640 1 -- 172.21.15.72:0/176679730 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc204062890 con 0x7fc204100d10 2024-03-22T01:34:30.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.398+0000 7fc1f67fc640 1 -- 172.21.15.72:0/176679730 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7fc1f80bd7a0 con 0x7fc204100d10 2024-03-22T01:34:30.397 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-03-22T01:34:30.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.400+0000 7fc20b98a640 1 -- 172.21.15.72:0/176679730 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fc1f00a2f20 msgr2=0x7fc1f00a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:30.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.400+0000 7fc20b98a640 1 --2- 172.21.15.72:0/176679730 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fc1f00a2f20 0x7fc1f00a53e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc204199450 tx=0x7fc200005f90 comp rx=0 tx=0).stop 2024-03-22T01:34:30.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.400+0000 7fc20b98a640 1 -- 172.21.15.72:0/176679730 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc204100d10 msgr2=0x7fc204197f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:30.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.400+0000 7fc20b98a640 1 --2- 172.21.15.72:0/176679730 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc204100d10 0x7fc204197f40 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fc1f802efd0 tx=0x7fc1f802c6c0 comp rx=0 tx=0).stop 2024-03-22T01:34:30.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.400+0000 7fc20b98a640 1 -- 172.21.15.72:0/176679730 shutdown_connections 2024-03-22T01:34:30.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.400+0000 7fc20b98a640 1 --2- 172.21.15.72:0/176679730 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fc1f00a2f20 0x7fc1f00a53e0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:30.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.400+0000 7fc20b98a640 1 --2- 172.21.15.72:0/176679730 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2041020c0 0x7fc20419f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:30.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.400+0000 7fc20b98a640 1 --2- 172.21.15.72:0/176679730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2041016d0 0x7fc204198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:30.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.400+0000 7fc20b98a640 1 --2- 172.21.15.72:0/176679730 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc204100d10 0x7fc204197f40 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:30.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.400+0000 7fc20b98a640 1 -- 172.21.15.72:0/176679730 >> 172.21.15.72:0/176679730 conn(0x7fc2040fb7a0 msgr2=0x7fc2041050f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:30.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.400+0000 7fc20b98a640 1 -- 172.21.15.72:0/176679730 shutdown_connections 2024-03-22T01:34:30.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.400+0000 7fc20b98a640 1 -- 172.21.15.72:0/176679730 wait complete. 2024-03-22T01:34:30.410 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-22T01:34:30.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.515+0000 7f6d069e9640 1 -- 172.21.15.72:0/3176154344 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d00101ca0 msgr2=0x7f6d00102120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:30.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.515+0000 7f6d069e9640 1 --2- 172.21.15.72:0/3176154344 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d00101ca0 0x7f6d00102120 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4009860 tx=0x7f6cf402bb70 comp rx=0 tx=0).stop 2024-03-22T01:34:30.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.515+0000 7f6d069e9640 1 -- 172.21.15.72:0/3176154344 shutdown_connections 2024-03-22T01:34:30.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.515+0000 7f6d069e9640 1 --2- 172.21.15.72:0/3176154344 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d00101ca0 0x7f6d00102120 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:30.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.515+0000 7f6d069e9640 1 --2- 172.21.15.72:0/3176154344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d00102660 0x7f6d00108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:30.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.515+0000 7f6d069e9640 1 --2- 172.21.15.72:0/3176154344 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d00100ab0 0x7f6d00100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:30.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.515+0000 7f6d069e9640 1 -- 172.21.15.72:0/3176154344 >> 172.21.15.72:0/3176154344 conn(0x7f6d000fc020 msgr2=0x7f6d000fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:30.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.515+0000 7f6d069e9640 1 -- 172.21.15.72:0/3176154344 shutdown_connections 2024-03-22T01:34:30.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.515+0000 7f6d069e9640 1 -- 172.21.15.72:0/3176154344 wait complete. 2024-03-22T01:34:30.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.515+0000 7f6d069e9640 1 Processor -- start 2024-03-22T01:34:30.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.516+0000 7f6d069e9640 1 -- start start 2024-03-22T01:34:30.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.516+0000 7f6d069e9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d00100ab0 0x7f6d00197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:30.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.516+0000 7f6d069e9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d00101ca0 0x7f6d001983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:30.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.516+0000 7f6d069e9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d00102660 0x7f6d0019f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:30.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.516+0000 7f6d069e9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f6d0010c710 con 0x7f6d00100ab0 2024-03-22T01:34:30.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.516+0000 7f6d069e9640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f6d0010c590 con 0x7f6d00102660 2024-03-22T01:34:30.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.516+0000 7f6d069e9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6d0010c890 con 0x7f6d00101ca0 2024-03-22T01:34:30.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.516+0000 7f6cfffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d00100ab0 0x7f6d00197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:30.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.516+0000 7f6cfffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d00100ab0 0x7f6d00197ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48110/0 (socket says 172.21.15.72:48110) 2024-03-22T01:34:30.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.516+0000 7f6cfffff640 1 -- 172.21.15.72:0/798886543 learned_addr learned my addr 172.21.15.72:0/798886543 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:34:30.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.516+0000 7f6d04f86640 1 --2- 172.21.15.72:0/798886543 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d00102660 0x7f6d0019f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:30.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.516+0000 7f6cff7fe640 1 --2- 172.21.15.72:0/798886543 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d00101ca0 0x7f6d001983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:30.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.516+0000 7f6cfffff640 1 -- 172.21.15.72:0/798886543 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d00101ca0 msgr2=0x7f6d001983e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:30.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.516+0000 7f6cfffff640 1 --2- 172.21.15.72:0/798886543 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d00101ca0 0x7f6d001983e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:30.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.516+0000 7f6cfffff640 1 -- 172.21.15.72:0/798886543 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d00102660 msgr2=0x7f6d0019f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:30.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.516+0000 7f6cfffff640 1 --2- 172.21.15.72:0/798886543 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d00102660 0x7f6d0019f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:30.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.516+0000 7f6cfffff640 1 -- 172.21.15.72:0/798886543 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d0019fac0 con 0x7f6d00100ab0 2024-03-22T01:34:30.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.516+0000 7f6cff7fe640 1 --2- 172.21.15.72:0/798886543 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d00101ca0 0x7f6d001983e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T01:34:30.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.516+0000 7f6cfffff640 1 --2- 172.21.15.72:0/798886543 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d00100ab0 0x7f6d00197ea0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6cf000e9a0 tx=0x7f6cf000ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:30.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.517+0000 7f6cfd7fa640 1 -- 172.21.15.72:0/798886543 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf000cd80 con 0x7f6d00100ab0 2024-03-22T01:34:30.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.517+0000 7f6d069e9640 1 -- 172.21.15.72:0/798886543 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d0019fdb0 con 0x7f6d00100ab0 2024-03-22T01:34:30.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.517+0000 7f6cfd7fa640 1 -- 172.21.15.72:0/798886543 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6cf0010910 con 0x7f6d00100ab0 2024-03-22T01:34:30.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.517+0000 7f6cfd7fa640 1 -- 172.21.15.72:0/798886543 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf0018a40 con 0x7f6d00100ab0 2024-03-22T01:34:30.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.517+0000 7f6d069e9640 1 -- 172.21.15.72:0/798886543 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d001a0220 con 0x7f6d00100ab0 2024-03-22T01:34:30.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.518+0000 7f6d069e9640 1 -- 172.21.15.72:0/798886543 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d001a04b0 con 0x7f6d00100ab0 2024-03-22T01:34:30.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.519+0000 7f6cfd7fa640 1 -- 172.21.15.72:0/798886543 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f6cf0010ac0 con 0x7f6d00100ab0 2024-03-22T01:34:30.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.519+0000 7f6cfd7fa640 1 --2- 172.21.15.72:0/798886543 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f6ce80a2f20 0x7f6ce80a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:30.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.520+0000 7f6cff7fe640 1 --2- 172.21.15.72:0/798886543 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f6ce80a2f20 0x7f6ce80a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:30.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.520+0000 7f6cfd7fa640 1 -- 172.21.15.72:0/798886543 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f6cf00cf9d0 con 0x7f6d00100ab0 2024-03-22T01:34:30.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.520+0000 7f6cff7fe640 1 --2- 172.21.15.72:0/798886543 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f6ce80a2f20 0x7f6ce80a53e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4005940 tx=0x7f6cf40058b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:30.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.524+0000 7f6cfd7fa640 1 -- 172.21.15.72:0/798886543 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6cf0053740 con 0x7f6d00100ab0 2024-03-22T01:34:30.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.702+0000 7f6d069e9640 1 -- 172.21.15.72:0/798886543 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f6d001a0810 con 0x7f6d00100ab0 2024-03-22T01:34:30.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.702+0000 7f6cfd7fa640 1 -- 172.21.15.72:0/798886543 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) ==== 103+0+11 (secure 0 0 0) 0x7f6cf009c5c0 con 0x7f6d00100ab0 2024-03-22T01:34:30.701 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-03-22T01:34:30.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.704+0000 7f6d069e9640 1 -- 172.21.15.72:0/798886543 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f6ce80a2f20 msgr2=0x7f6ce80a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:30.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.704+0000 7f6d069e9640 1 --2- 172.21.15.72:0/798886543 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f6ce80a2f20 0x7f6ce80a53e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4005940 tx=0x7f6cf40058b0 comp rx=0 tx=0).stop 2024-03-22T01:34:30.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.704+0000 7f6d069e9640 1 -- 172.21.15.72:0/798886543 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d00100ab0 msgr2=0x7f6d00197ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:30.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.704+0000 7f6d069e9640 1 --2- 172.21.15.72:0/798886543 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d00100ab0 0x7f6d00197ea0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6cf000e9a0 tx=0x7f6cf000ee60 comp rx=0 tx=0).stop 2024-03-22T01:34:30.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.704+0000 7f6d069e9640 1 -- 172.21.15.72:0/798886543 shutdown_connections 2024-03-22T01:34:30.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.704+0000 7f6d069e9640 1 --2- 172.21.15.72:0/798886543 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f6ce80a2f20 0x7f6ce80a53e0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:30.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.704+0000 7f6d069e9640 1 --2- 172.21.15.72:0/798886543 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d00102660 0x7f6d0019f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:30.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.704+0000 7f6d069e9640 1 --2- 172.21.15.72:0/798886543 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d00101ca0 0x7f6d001983e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:30.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.704+0000 7f6d069e9640 1 --2- 172.21.15.72:0/798886543 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d00100ab0 0x7f6d00197ea0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:30.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.704+0000 7f6d069e9640 1 -- 172.21.15.72:0/798886543 >> 172.21.15.72:0/798886543 conn(0x7f6d000fc020 msgr2=0x7f6d000fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:30.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.704+0000 7f6d069e9640 1 -- 172.21.15.72:0/798886543 shutdown_connections 2024-03-22T01:34:30.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.704+0000 7f6d069e9640 1 -- 172.21.15.72:0/798886543 wait complete. 2024-03-22T01:34:30.713 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:30 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/1869260608' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T01:34:30.713 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:30 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/3953082493' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T01:34:30.713 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:30 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/176679730' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T01:34:30.714 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-22T01:34:30.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.819+0000 7f3ec533e640 1 -- 172.21.15.72:0/1262454755 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3ec01037f0 msgr2=0x7f3ec010a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:30.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.819+0000 7f3ec533e640 1 --2- 172.21.15.72:0/1262454755 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3ec01037f0 0x7f3ec010a080 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f3eb8009cd0 tx=0x7f3eb802c0f0 comp rx=0 tx=0).stop 2024-03-22T01:34:30.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.819+0000 7f3ec533e640 1 -- 172.21.15.72:0/1262454755 shutdown_connections 2024-03-22T01:34:30.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.819+0000 7f3ec533e640 1 --2- 172.21.15.72:0/1262454755 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3ec01037f0 0x7f3ec010a080 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:30.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.819+0000 7f3ec533e640 1 --2- 172.21.15.72:0/1262454755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ec0102e30 0x7f3ec01032b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:30.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.819+0000 7f3ec533e640 1 --2- 172.21.15.72:0/1262454755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ec00ffe60 0x7f3ec0100280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:30.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.819+0000 7f3ec533e640 1 -- 172.21.15.72:0/1262454755 >> 172.21.15.72:0/1262454755 conn(0x7f3ec00fb7b0 msgr2=0x7f3ec00fdbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:30.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.819+0000 7f3ec533e640 1 -- 172.21.15.72:0/1262454755 shutdown_connections 2024-03-22T01:34:30.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.819+0000 7f3ec533e640 1 -- 172.21.15.72:0/1262454755 wait complete. 2024-03-22T01:34:30.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.820+0000 7f3ec533e640 1 Processor -- start 2024-03-22T01:34:30.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.820+0000 7f3ec533e640 1 -- start start 2024-03-22T01:34:30.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.820+0000 7f3ec533e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ec00ffe60 0x7f3ec0197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:30.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.820+0000 7f3ec533e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ec0102e30 0x7f3ec01983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:30.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.820+0000 7f3ec533e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3ec01037f0 0x7f3ec019f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:30.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.820+0000 7f3ec533e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f3ec010d810 con 0x7f3ec01037f0 2024-03-22T01:34:30.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.820+0000 7f3ec533e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3ec010d690 con 0x7f3ec0102e30 2024-03-22T01:34:30.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.820+0000 7f3ec533e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3ec010d990 con 0x7f3ec00ffe60 2024-03-22T01:34:30.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.820+0000 7f3ebf7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3ec01037f0 0x7f3ec019f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:30.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.820+0000 7f3ebf7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3ec01037f0 0x7f3ec019f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45702/0 (socket says 172.21.15.72:45702) 2024-03-22T01:34:30.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.820+0000 7f3ebf7fe640 1 -- 172.21.15.72:0/3525332296 learned_addr learned my addr 172.21.15.72:0/3525332296 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:34:30.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.820+0000 7f3eb7fff640 1 --2- 172.21.15.72:0/3525332296 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ec0102e30 0x7f3ec01983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:30.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.820+0000 7f3ebeffd640 1 --2- 172.21.15.72:0/3525332296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ec00ffe60 0x7f3ec0197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:30.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.820+0000 7f3ebf7fe640 1 -- 172.21.15.72:0/3525332296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ec00ffe60 msgr2=0x7f3ec0197eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:30.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.820+0000 7f3ebf7fe640 1 --2- 172.21.15.72:0/3525332296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ec00ffe60 0x7f3ec0197eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:30.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.820+0000 7f3ebf7fe640 1 -- 172.21.15.72:0/3525332296 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ec0102e30 msgr2=0x7f3ec01983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:30.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.820+0000 7f3ebf7fe640 1 --2- 172.21.15.72:0/3525332296 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ec0102e30 0x7f3ec01983f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:30.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.820+0000 7f3ebf7fe640 1 -- 172.21.15.72:0/3525332296 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ec019fad0 con 0x7f3ec01037f0 2024-03-22T01:34:30.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.820+0000 7f3ebf7fe640 1 --2- 172.21.15.72:0/3525332296 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3ec01037f0 0x7f3ec019f460 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f3eb8005e30 tx=0x7f3eb80346f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:30.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.821+0000 7f3ebcff9640 1 -- 172.21.15.72:0/3525332296 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eb8041020 con 0x7f3ec01037f0 2024-03-22T01:34:30.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.821+0000 7f3ec533e640 1 -- 172.21.15.72:0/3525332296 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ec019fd60 con 0x7f3ec01037f0 2024-03-22T01:34:30.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.821+0000 7f3ebcff9640 1 -- 172.21.15.72:0/3525332296 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3eb8032890 con 0x7f3ec01037f0 2024-03-22T01:34:30.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.821+0000 7f3ebcff9640 1 -- 172.21.15.72:0/3525332296 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eb8039040 con 0x7f3ec01037f0 2024-03-22T01:34:30.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.821+0000 7f3ec533e640 1 -- 172.21.15.72:0/3525332296 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ec01a01f0 con 0x7f3ec01037f0 2024-03-22T01:34:30.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.822+0000 7f3ec533e640 1 -- 172.21.15.72:0/3525332296 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ec0108fb0 con 0x7f3ec01037f0 2024-03-22T01:34:30.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.823+0000 7f3ebcff9640 1 -- 172.21.15.72:0/3525332296 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f3eb8044020 con 0x7f3ec01037f0 2024-03-22T01:34:30.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.823+0000 7f3ebcff9640 1 --2- 172.21.15.72:0/3525332296 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f3eac0a2f20 0x7f3eac0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:30.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.823+0000 7f3ebeffd640 1 --2- 172.21.15.72:0/3525332296 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f3eac0a2f20 0x7f3eac0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:30.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.823+0000 7f3ebcff9640 1 -- 172.21.15.72:0/3525332296 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f3eb80f1880 con 0x7f3ec01037f0 2024-03-22T01:34:30.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.824+0000 7f3ebeffd640 1 --2- 172.21.15.72:0/3525332296 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f3eac0a2f20 0x7f3eac0a53e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f3eb00095f0 tx=0x7f3eb0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:30.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:30.826+0000 7f3ebcff9640 1 -- 172.21.15.72:0/3525332296 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3eb804b020 con 0x7f3ec01037f0 2024-03-22T01:34:30.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:30 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/1869260608' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T01:34:30.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:30 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/3953082493' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T01:34:30.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:30 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/176679730' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T01:34:30.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:30 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/1869260608' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T01:34:30.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:30 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/3953082493' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T01:34:30.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:30 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/176679730' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T01:34:31.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:31.013+0000 7f3ec533e640 1 -- 172.21.15.72:0/3525332296 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f3ec002d3b0 con 0x7f3ec01037f0 2024-03-22T01:34:31.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:31.014+0000 7f3ebcff9640 1 -- 172.21.15.72:0/3525332296 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v91) ==== 81+0+591 (secure 0 0 0) 0x7f3eb80be470 con 0x7f3ec01037f0 2024-03-22T01:34:31.013 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:0/121087196 2024-03-23T01:27:15.339809+0000 2024-03-22T01:34:31.013 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:0/2451857761 2024-03-23T01:27:15.339809+0000 2024-03-22T01:34:31.013 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:0/1228477995 2024-03-23T01:27:15.339809+0000 2024-03-22T01:34:31.013 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:6801/1513232913 2024-03-23T01:27:15.339809+0000 2024-03-22T01:34:31.013 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:0/3558958264 2024-03-23T01:26:50.074871+0000 2024-03-22T01:34:31.013 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:0/1510187198 2024-03-23T01:26:50.074871+0000 2024-03-22T01:34:31.014 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:6800/3518012916 2024-03-23T01:26:50.074871+0000 2024-03-22T01:34:31.014 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:6801/3518012916 2024-03-23T01:26:50.074871+0000 2024-03-22T01:34:31.014 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:0/1372801041 2024-03-23T01:26:50.074871+0000 2024-03-22T01:34:31.014 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:6800/1513232913 2024-03-23T01:27:15.339809+0000 2024-03-22T01:34:31.014 INFO:teuthology.orchestra.run.smithi072.stderr:listed 10 entries 2024-03-22T01:34:31.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:31.016+0000 7f3ec533e640 1 -- 172.21.15.72:0/3525332296 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f3eac0a2f20 msgr2=0x7f3eac0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:31.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:31.016+0000 7f3ec533e640 1 --2- 172.21.15.72:0/3525332296 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f3eac0a2f20 0x7f3eac0a53e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f3eb00095f0 tx=0x7f3eb0005f90 comp rx=0 tx=0).stop 2024-03-22T01:34:31.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:31.016+0000 7f3ec533e640 1 -- 172.21.15.72:0/3525332296 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3ec01037f0 msgr2=0x7f3ec019f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:31.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:31.016+0000 7f3ec533e640 1 --2- 172.21.15.72:0/3525332296 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3ec01037f0 0x7f3ec019f460 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f3eb8005e30 tx=0x7f3eb80346f0 comp rx=0 tx=0).stop 2024-03-22T01:34:31.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:31.016+0000 7f3ec533e640 1 -- 172.21.15.72:0/3525332296 shutdown_connections 2024-03-22T01:34:31.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:31.016+0000 7f3ec533e640 1 --2- 172.21.15.72:0/3525332296 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f3eac0a2f20 0x7f3eac0a53e0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:31.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:31.016+0000 7f3ec533e640 1 --2- 172.21.15.72:0/3525332296 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ec0102e30 0x7f3ec01983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:31.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:31.016+0000 7f3ec533e640 1 --2- 172.21.15.72:0/3525332296 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3ec01037f0 0x7f3ec019f460 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:31.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:31.016+0000 7f3ec533e640 1 --2- 172.21.15.72:0/3525332296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ec00ffe60 0x7f3ec0197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:31.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:31.016+0000 7f3ec533e640 1 -- 172.21.15.72:0/3525332296 >> 172.21.15.72:0/3525332296 conn(0x7f3ec00fb7b0 msgr2=0x7f3ec0106180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:31.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:31.016+0000 7f3ec533e640 1 -- 172.21.15.72:0/3525332296 shutdown_connections 2024-03-22T01:34:31.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:31.016+0000 7f3ec533e640 1 -- 172.21.15.72:0/3525332296 wait complete. 2024-03-22T01:34:31.026 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi072.front.sepia.ceph.com... 2024-03-22T01:34:31.026 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T01:34:31.027 DEBUG:teuthology.orchestra.run.smithi072:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-03-22T01:34:31.052 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi072.stderr:ceph-fuse[73989]: fuse finished with error 0 and tester_r 0 2024-03-22T01:34:31.054 INFO:teuthology.orchestra.run:waiting for 300 2024-03-22T01:34:31.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:31 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/798886543' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T01:34:31.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:31 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/3525332296' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-22T01:34:31.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:31 smithi089 ceph-mon[35770]: pgmap v299: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 281 B/s wr, 0 op/s 2024-03-22T01:34:31.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:31 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/798886543' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T01:34:31.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:31 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/3525332296' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-22T01:34:31.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:31 smithi072 ceph-mon[32414]: pgmap v299: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 281 B/s wr, 0 op/s 2024-03-22T01:34:31.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:31 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/798886543' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T01:34:31.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:31 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/3525332296' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-22T01:34:31.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:31 smithi098 ceph-mon[36468]: pgmap v299: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 281 B/s wr, 0 op/s 2024-03-22T01:34:34.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:34 smithi089 ceph-mon[35770]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-22T01:34:34.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:34 smithi072 ceph-mon[32414]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-22T01:34:34.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:34 smithi098 ceph-mon[36468]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-22T01:34:36.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:36 smithi098 ceph-mon[36468]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-22T01:34:36.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:36 smithi089 ceph-mon[35770]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-22T01:34:36.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:36 smithi072 ceph-mon[32414]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-22T01:34:37.155 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi072.front.sepia.ceph.com 2024-03-22T01:34:37.156 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T01:34:37.156 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-03-22T01:34:37.181 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T01:34:37.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.336+0000 7f371bc2a640 1 -- 172.21.15.72:0/1719162770 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3714100ac0 msgr2=0x7f3714100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:37.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.336+0000 7f371bc2a640 1 --2- 172.21.15.72:0/1719162770 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3714100ac0 0x7f3714100ee0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f3710009900 tx=0x7f371002bc30 comp rx=0 tx=0).stop 2024-03-22T01:34:37.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.336+0000 7f371bc2a640 1 -- 172.21.15.72:0/1719162770 shutdown_connections 2024-03-22T01:34:37.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.336+0000 7f371bc2a640 1 --2- 172.21.15.72:0/1719162770 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3714101cb0 0x7f3714102130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:37.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.336+0000 7f371bc2a640 1 --2- 172.21.15.72:0/1719162770 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3714102670 0x7f3714108f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:37.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.336+0000 7f371bc2a640 1 --2- 172.21.15.72:0/1719162770 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3714100ac0 0x7f3714100ee0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:37.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.336+0000 7f371bc2a640 1 -- 172.21.15.72:0/1719162770 >> 172.21.15.72:0/1719162770 conn(0x7f37140fc050 msgr2=0x7f37140fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:37.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.336+0000 7f371bc2a640 1 -- 172.21.15.72:0/1719162770 shutdown_connections 2024-03-22T01:34:37.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.336+0000 7f371bc2a640 1 -- 172.21.15.72:0/1719162770 wait complete. 2024-03-22T01:34:37.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.336+0000 7f371bc2a640 1 Processor -- start 2024-03-22T01:34:37.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.336+0000 7f371bc2a640 1 -- start start 2024-03-22T01:34:37.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.336+0000 7f371bc2a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3714100ac0 0x7f3714196e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:37.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.336+0000 7f371bc2a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3714101cb0 0x7f3714197370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:37.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.336+0000 7f371bc2a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3714102670 0x7f371419e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:37.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.336+0000 7f371bc2a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f371410c740 con 0x7f3714101cb0 2024-03-22T01:34:37.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.336+0000 7f371bc2a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f371410c5c0 con 0x7f3714102670 2024-03-22T01:34:37.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.336+0000 7f371bc2a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f371410c8c0 con 0x7f3714100ac0 2024-03-22T01:34:37.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.337+0000 7f37191c5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3714101cb0 0x7f3714197370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:37.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.337+0000 7f37191c5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3714101cb0 0x7f3714197370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45720/0 (socket says 172.21.15.72:45720) 2024-03-22T01:34:37.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.337+0000 7f37191c5640 1 -- 172.21.15.72:0/4147022655 learned_addr learned my addr 172.21.15.72:0/4147022655 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:34:37.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.337+0000 7f371a1c7640 1 --2- 172.21.15.72:0/4147022655 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3714102670 0x7f371419e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:37.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.337+0000 7f37199c6640 1 --2- 172.21.15.72:0/4147022655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3714100ac0 0x7f3714196e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:37.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.337+0000 7f37191c5640 1 -- 172.21.15.72:0/4147022655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3714100ac0 msgr2=0x7f3714196e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:37.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.337+0000 7f37191c5640 1 --2- 172.21.15.72:0/4147022655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3714100ac0 0x7f3714196e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:37.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.337+0000 7f37191c5640 1 -- 172.21.15.72:0/4147022655 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3714102670 msgr2=0x7f371419e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:37.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.337+0000 7f37191c5640 1 --2- 172.21.15.72:0/4147022655 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3714102670 0x7f371419e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:37.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.337+0000 7f37191c5640 1 -- 172.21.15.72:0/4147022655 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f371419ea50 con 0x7f3714101cb0 2024-03-22T01:34:37.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.337+0000 7f371a1c7640 1 --2- 172.21.15.72:0/4147022655 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3714102670 0x7f371419e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:34:37.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.337+0000 7f37191c5640 1 --2- 172.21.15.72:0/4147022655 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3714101cb0 0x7f3714197370 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f370800eb00 tx=0x7f3708009f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:37.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.337+0000 7f3706ffd640 1 -- 172.21.15.72:0/4147022655 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f370801c070 con 0x7f3714101cb0 2024-03-22T01:34:37.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.337+0000 7f3706ffd640 1 -- 172.21.15.72:0/4147022655 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f370800bc30 con 0x7f3714101cb0 2024-03-22T01:34:37.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.337+0000 7f371bc2a640 1 -- 172.21.15.72:0/4147022655 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f371406a2b0 con 0x7f3714101cb0 2024-03-22T01:34:37.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.337+0000 7f3706ffd640 1 -- 172.21.15.72:0/4147022655 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37080166a0 con 0x7f3714101cb0 2024-03-22T01:34:37.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.337+0000 7f371bc2a640 1 -- 172.21.15.72:0/4147022655 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f371406a850 con 0x7f3714101cb0 2024-03-22T01:34:37.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.338+0000 7f371bc2a640 1 -- 172.21.15.72:0/4147022655 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3714107850 con 0x7f3714101cb0 2024-03-22T01:34:37.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.340+0000 7f3706ffd640 1 -- 172.21.15.72:0/4147022655 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f370801f070 con 0x7f3714101cb0 2024-03-22T01:34:37.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.340+0000 7f3706ffd640 1 --2- 172.21.15.72:0/4147022655 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f37000a2fa0 0x7f37000a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:37.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.340+0000 7f37199c6640 1 --2- 172.21.15.72:0/4147022655 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f37000a2fa0 0x7f37000a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:37.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.340+0000 7f3706ffd640 1 -- 172.21.15.72:0/4147022655 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f37080d0bc0 con 0x7f3714101cb0 2024-03-22T01:34:37.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.341+0000 7f37199c6640 1 --2- 172.21.15.72:0/4147022655 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f37000a2fa0 0x7f37000a5460 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f371002c7b0 tx=0x7f3710005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:37.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.345+0000 7f3706ffd640 1 -- 172.21.15.72:0/4147022655 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f370800bde0 con 0x7f3714101cb0 2024-03-22T01:34:37.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.521+0000 7f371bc2a640 1 -- 172.21.15.72:0/4147022655 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f371419ec20 con 0x7f3714101cb0 2024-03-22T01:34:37.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.523+0000 7f3706ffd640 1 -- 172.21.15.72:0/4147022655 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24045 (secure 0 0 0) 0x7f370809d100 con 0x7f3714101cb0 2024-03-22T01:34:37.522 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:34:37.522 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":91,"fsid":"022a99bc-e7eb-11ee-95cd-87774f69a715","created":"2024-03-22T01:26:23.893672+0000","modified":"2024-03-22T01:34:19.866367+0000","last_up_change":"2024-03-22T01:33:25.223684+0000","last_in_change":"2024-03-22T01:33:05.010331+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T01:29:37.438503+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-22T01:34:12.610422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-22T01:34:13.761686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-22T01:34:17.507039+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"16c15bb0-db92-4191-9063-38b4890e81e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6803","nonce":661906791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6805","nonce":661906791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6809","nonce":661906791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6807","nonce":661906791}]},"public_addr":"172.21.15.72:6803/661906791","cluster_addr":"172.21.15.72:6805/661906791","heartbeat_back_addr":"172.21.15.72:6809/661906791","heartbeat_front_addr":"172.21.15.72:6807/661906791","state":["exists","up"]},{"osd":1,"uuid":"981290ba-97dc-496b-aa22-d51704184b10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6801","nonce":2870590550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6803","nonce":2870590550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6807","nonce":2870590550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6805","nonce":2870590550}]},"public_addr":"172.21.15.89:6801/2870590550","cluster_addr":"172.21.15.89:6803/2870590550","heartbeat_back_addr":"172.21.15.89:6807/2870590550","heartbeat_front_addr":"172.21.15.89:6805/2870590550","state":["exists","up"]},{"osd":2,"uuid":"f07ed658-f545-46a1-8edd-8b1d5d49885c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6801","nonce":3865778118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6803","nonce":3865778118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6807","nonce":3865778118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6805","nonce":3865778118}]},"public_addr":"172.21.15.98:6801/3865778118","cluster_addr":"172.21.15.98:6803/3865778118","heartbeat_back_addr":"172.21.15.98:6807/3865778118","heartbeat_front_addr":"172.21.15.98:6805/3865778118","state":["exists","up"]},{"osd":3,"uuid":"ccc4bc40-2c48-46d2-8548-0c05b44f9f1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6811","nonce":1865951395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6813","nonce":1865951395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6817","nonce":1865951395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6815","nonce":1865951395}]},"public_addr":"172.21.15.72:6811/1865951395","cluster_addr":"172.21.15.72:6813/1865951395","heartbeat_back_addr":"172.21.15.72:6817/1865951395","heartbeat_front_addr":"172.21.15.72:6815/1865951395","state":["exists","up"]},{"osd":4,"uuid":"1812439f-810a-4ed2-9395-c00c9a085a13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6809","nonce":2683760795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6811","nonce":2683760795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6815","nonce":2683760795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6813","nonce":2683760795}]},"public_addr":"172.21.15.89:6809/2683760795","cluster_addr":"172.21.15.89:6811/2683760795","heartbeat_back_addr":"172.21.15.89:6815/2683760795","heartbeat_front_addr":"172.21.15.89:6813/2683760795","state":["exists","up"]},{"osd":5,"uuid":"e78954d8-f4e3-4d58-8536-d2d59c843386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6809","nonce":2635838572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6811","nonce":2635838572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6815","nonce":2635838572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6813","nonce":2635838572}]},"public_addr":"172.21.15.98:6809/2635838572","cluster_addr":"172.21.15.98:6811/2635838572","heartbeat_back_addr":"172.21.15.98:6815/2635838572","heartbeat_front_addr":"172.21.15.98:6813/2635838572","state":["exists","up"]},{"osd":6,"uuid":"dc6c8be2-b64c-4959-8015-7b0ea6d25599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6819","nonce":3158982661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6821","nonce":3158982661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6825","nonce":3158982661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6823","nonce":3158982661}]},"public_addr":"172.21.15.72:6819/3158982661","cluster_addr":"172.21.15.72:6821/3158982661","heartbeat_back_addr":"172.21.15.72:6825/3158982661","heartbeat_front_addr":"172.21.15.72:6823/3158982661","state":["exists","up"]},{"osd":7,"uuid":"7a73469f-cf3a-4306-ae93-ea46e21ebbbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6817","nonce":3075913234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6819","nonce":3075913234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6823","nonce":3075913234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6821","nonce":3075913234}]},"public_addr":"172.21.15.89:6817/3075913234","cluster_addr":"172.21.15.89:6819/3075913234","heartbeat_back_addr":"172.21.15.89:6823/3075913234","heartbeat_front_addr":"172.21.15.89:6821/3075913234","state":["exists","up"]},{"osd":8,"uuid":"c4b13bee-ea4d-4344-95fb-2622ecbb8196","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6817","nonce":1813255577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6819","nonce":1813255577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6823","nonce":1813255577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6821","nonce":1813255577}]},"public_addr":"172.21.15.98:6817/1813255577","cluster_addr":"172.21.15.98:6819/1813255577","heartbeat_back_addr":"172.21.15.98:6823/1813255577","heartbeat_front_addr":"172.21.15.98:6821/1813255577","state":["exists","up"]},{"osd":9,"uuid":"eefb0a44-4a36-4e55-9d04-0501bf718db6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6827","nonce":3377749206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6829","nonce":3377749206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6832","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6833","nonce":3377749206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6831","nonce":3377749206}]},"public_addr":"172.21.15.72:6827/3377749206","cluster_addr":"172.21.15.72:6829/3377749206","heartbeat_back_addr":"172.21.15.72:6833/3377749206","heartbeat_front_addr":"172.21.15.72:6831/3377749206","state":["exists","up"]},{"osd":10,"uuid":"8741d25f-40dd-40d7-b040-5cc5e411eb98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6825","nonce":885058786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6827","nonce":885058786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6831","nonce":885058786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6829","nonce":885058786}]},"public_addr":"172.21.15.89:6825/885058786","cluster_addr":"172.21.15.89:6827/885058786","heartbeat_back_addr":"172.21.15.89:6831/885058786","heartbeat_front_addr":"172.21.15.89:6829/885058786","state":["exists","up"]},{"osd":11,"uuid":"8d02e957-eda4-4184-b5c6-5eb3b8b38060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6825","nonce":1517095838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6827","nonce":1517095838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6831","nonce":1517095838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6829","nonce":1517095838}]},"public_addr":"172.21.15.98:6825/1517095838","cluster_addr":"172.21.15.98:6827/1517095838","heartbeat_back_addr":"172.21.15.98:6831/1517095838","heartbeat_front_addr":"172.21.15.98:6829/1517095838","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:28:47.307652+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:11.037135+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:34.619188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:59.350738+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:30:23.787337+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:30:48.994154+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:31:13.690350+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:31:39.628116+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:04.723912+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:30.689741+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:57.229508+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:33:23.154537+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:0/121087196":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/2451857761":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/1228477995":"2024-03-23T01:27:15.339809+0000","172.21.15.72:6801/1513232913":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/3558958264":"2024-03-23T01:26:50.074871+0000","172.21.15.72:0/1510187198":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6800/3518012916":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6801/3518012916":"2024-03-23T01:26:50.074871+0000","172.21.15.72:0/1372801041":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6800/1513232913":"2024-03-23T01:27:15.339809+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T01:34:37.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.524+0000 7f371bc2a640 1 -- 172.21.15.72:0/4147022655 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f37000a2fa0 msgr2=0x7f37000a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:37.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.524+0000 7f371bc2a640 1 --2- 172.21.15.72:0/4147022655 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f37000a2fa0 0x7f37000a5460 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f371002c7b0 tx=0x7f3710005fd0 comp rx=0 tx=0).stop 2024-03-22T01:34:37.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.524+0000 7f371bc2a640 1 -- 172.21.15.72:0/4147022655 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3714101cb0 msgr2=0x7f3714197370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:37.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.524+0000 7f371bc2a640 1 --2- 172.21.15.72:0/4147022655 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3714101cb0 0x7f3714197370 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f370800eb00 tx=0x7f3708009f20 comp rx=0 tx=0).stop 2024-03-22T01:34:37.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.525+0000 7f371bc2a640 1 -- 172.21.15.72:0/4147022655 shutdown_connections 2024-03-22T01:34:37.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.525+0000 7f371bc2a640 1 --2- 172.21.15.72:0/4147022655 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f37000a2fa0 0x7f37000a5460 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:37.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.525+0000 7f371bc2a640 1 --2- 172.21.15.72:0/4147022655 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3714102670 0x7f371419e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:37.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.525+0000 7f371bc2a640 1 --2- 172.21.15.72:0/4147022655 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3714101cb0 0x7f3714197370 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:37.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.525+0000 7f371bc2a640 1 --2- 172.21.15.72:0/4147022655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3714100ac0 0x7f3714196e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:37.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.525+0000 7f371bc2a640 1 -- 172.21.15.72:0/4147022655 >> 172.21.15.72:0/4147022655 conn(0x7f37140fc050 msgr2=0x7f3714104fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:37.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.525+0000 7f371bc2a640 1 -- 172.21.15.72:0/4147022655 shutdown_connections 2024-03-22T01:34:37.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.525+0000 7f371bc2a640 1 -- 172.21.15.72:0/4147022655 wait complete. 2024-03-22T01:34:37.535 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T01:34:37.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.637+0000 7f8437581640 1 -- 172.21.15.72:0/3358635017 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8430100b10 msgr2=0x7f8430100f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:37.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.637+0000 7f8437581640 1 --2- 172.21.15.72:0/3358635017 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8430100b10 0x7f8430100f30 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f842c009900 tx=0x7f842c02bc30 comp rx=0 tx=0).stop 2024-03-22T01:34:37.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.637+0000 7f8437581640 1 -- 172.21.15.72:0/3358635017 shutdown_connections 2024-03-22T01:34:37.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.637+0000 7f8437581640 1 --2- 172.21.15.72:0/3358635017 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8430101d00 0x7f8430102180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:37.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.637+0000 7f8437581640 1 --2- 172.21.15.72:0/3358635017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84301026c0 0x7f8430108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:37.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.637+0000 7f8437581640 1 --2- 172.21.15.72:0/3358635017 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8430100b10 0x7f8430100f30 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:37.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.637+0000 7f8437581640 1 -- 172.21.15.72:0/3358635017 >> 172.21.15.72:0/3358635017 conn(0x7f84300fc0a0 msgr2=0x7f84300fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:37.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.637+0000 7f8437581640 1 -- 172.21.15.72:0/3358635017 shutdown_connections 2024-03-22T01:34:37.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.637+0000 7f8437581640 1 -- 172.21.15.72:0/3358635017 wait complete. 2024-03-22T01:34:37.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.638+0000 7f8437581640 1 Processor -- start 2024-03-22T01:34:37.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.638+0000 7f8437581640 1 -- start start 2024-03-22T01:34:37.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.638+0000 7f8437581640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8430100b10 0x7f8430196db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:37.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.638+0000 7f8437581640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8430101d00 0x7f84301972f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:37.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.638+0000 7f8437581640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84301026c0 0x7f843019e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:37.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.638+0000 7f8437581640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f843010c720 con 0x7f8430100b10 2024-03-22T01:34:37.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.638+0000 7f8437581640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f843010c5a0 con 0x7f8430101d00 2024-03-22T01:34:37.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.638+0000 7f8437581640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f843010c8a0 con 0x7f84301026c0 2024-03-22T01:34:37.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.638+0000 7f843531d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8430100b10 0x7f8430196db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:37.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.638+0000 7f843531d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8430100b10 0x7f8430196db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45740/0 (socket says 172.21.15.72:45740) 2024-03-22T01:34:37.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.638+0000 7f843531d640 1 -- 172.21.15.72:0/3435442723 learned_addr learned my addr 172.21.15.72:0/3435442723 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:34:37.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.638+0000 7f8434b1c640 1 --2- 172.21.15.72:0/3435442723 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8430101d00 0x7f84301972f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:37.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.638+0000 7f8435b1e640 1 --2- 172.21.15.72:0/3435442723 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84301026c0 0x7f843019e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:37.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.638+0000 7f843531d640 1 -- 172.21.15.72:0/3435442723 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84301026c0 msgr2=0x7f843019e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:37.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.638+0000 7f843531d640 1 --2- 172.21.15.72:0/3435442723 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84301026c0 0x7f843019e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:37.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.638+0000 7f843531d640 1 -- 172.21.15.72:0/3435442723 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8430101d00 msgr2=0x7f84301972f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:37.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.638+0000 7f843531d640 1 --2- 172.21.15.72:0/3435442723 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8430101d00 0x7f84301972f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:37.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.638+0000 7f843531d640 1 -- 172.21.15.72:0/3435442723 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f843019e9c0 con 0x7f8430100b10 2024-03-22T01:34:37.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.639+0000 7f843531d640 1 --2- 172.21.15.72:0/3435442723 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8430100b10 0x7f8430196db0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f842c000c00 tx=0x7f842c005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:37.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.639+0000 7f84267fc640 1 -- 172.21.15.72:0/3435442723 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f842c041070 con 0x7f8430100b10 2024-03-22T01:34:37.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.639+0000 7f84267fc640 1 -- 172.21.15.72:0/3435442723 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f842c0044d0 con 0x7f8430100b10 2024-03-22T01:34:37.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.639+0000 7f8437581640 1 -- 172.21.15.72:0/3435442723 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f843019ec50 con 0x7f8430100b10 2024-03-22T01:34:37.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.639+0000 7f84267fc640 1 -- 172.21.15.72:0/3435442723 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f842c034440 con 0x7f8430100b10 2024-03-22T01:34:37.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.639+0000 7f8437581640 1 -- 172.21.15.72:0/3435442723 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f843019f140 con 0x7f8430100b10 2024-03-22T01:34:37.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.640+0000 7f8437581640 1 -- 172.21.15.72:0/3435442723 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8430107eb0 con 0x7f8430100b10 2024-03-22T01:34:37.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.641+0000 7f84267fc640 1 -- 172.21.15.72:0/3435442723 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f842c044070 con 0x7f8430100b10 2024-03-22T01:34:37.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.641+0000 7f84267fc640 1 --2- 172.21.15.72:0/3435442723 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f841c0a2f20 0x7f841c0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:37.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.641+0000 7f8434b1c640 1 --2- 172.21.15.72:0/3435442723 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f841c0a2f20 0x7f841c0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:37.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.641+0000 7f84267fc640 1 -- 172.21.15.72:0/3435442723 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f842c0f11b0 con 0x7f8430100b10 2024-03-22T01:34:37.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.642+0000 7f8434b1c640 1 --2- 172.21.15.72:0/3435442723 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f841c0a2f20 0x7f841c0a53e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f84200095f0 tx=0x7f8420005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:37.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.646+0000 7f84267fc640 1 -- 172.21.15.72:0/3435442723 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f842c0bd750 con 0x7f8430100b10 2024-03-22T01:34:37.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.823+0000 7f8437581640 1 -- 172.21.15.72:0/3435442723 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f843002d5d0 con 0x7f8430100b10 2024-03-22T01:34:37.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.824+0000 7f84267fc640 1 -- 172.21.15.72:0/3435442723 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24045 (secure 0 0 0) 0x7f842c0c17f0 con 0x7f8430100b10 2024-03-22T01:34:37.827 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:34:37.827 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":91,"fsid":"022a99bc-e7eb-11ee-95cd-87774f69a715","created":"2024-03-22T01:26:23.893672+0000","modified":"2024-03-22T01:34:19.866367+0000","last_up_change":"2024-03-22T01:33:25.223684+0000","last_in_change":"2024-03-22T01:33:05.010331+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T01:29:37.438503+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-22T01:34:12.610422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-22T01:34:13.761686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-22T01:34:17.507039+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"16c15bb0-db92-4191-9063-38b4890e81e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6803","nonce":661906791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6805","nonce":661906791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6809","nonce":661906791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6807","nonce":661906791}]},"public_addr":"172.21.15.72:6803/661906791","cluster_addr":"172.21.15.72:6805/661906791","heartbeat_back_addr":"172.21.15.72:6809/661906791","heartbeat_front_addr":"172.21.15.72:6807/661906791","state":["exists","up"]},{"osd":1,"uuid":"981290ba-97dc-496b-aa22-d51704184b10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6801","nonce":2870590550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6803","nonce":2870590550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6807","nonce":2870590550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6805","nonce":2870590550}]},"public_addr":"172.21.15.89:6801/2870590550","cluster_addr":"172.21.15.89:6803/2870590550","heartbeat_back_addr":"172.21.15.89:6807/2870590550","heartbeat_front_addr":"172.21.15.89:6805/2870590550","state":["exists","up"]},{"osd":2,"uuid":"f07ed658-f545-46a1-8edd-8b1d5d49885c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6801","nonce":3865778118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6803","nonce":3865778118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6807","nonce":3865778118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6805","nonce":3865778118}]},"public_addr":"172.21.15.98:6801/3865778118","cluster_addr":"172.21.15.98:6803/3865778118","heartbeat_back_addr":"172.21.15.98:6807/3865778118","heartbeat_front_addr":"172.21.15.98:6805/3865778118","state":["exists","up"]},{"osd":3,"uuid":"ccc4bc40-2c48-46d2-8548-0c05b44f9f1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6811","nonce":1865951395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6813","nonce":1865951395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6817","nonce":1865951395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6815","nonce":1865951395}]},"public_addr":"172.21.15.72:6811/1865951395","cluster_addr":"172.21.15.72:6813/1865951395","heartbeat_back_addr":"172.21.15.72:6817/1865951395","heartbeat_front_addr":"172.21.15.72:6815/1865951395","state":["exists","up"]},{"osd":4,"uuid":"1812439f-810a-4ed2-9395-c00c9a085a13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6809","nonce":2683760795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6811","nonce":2683760795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6815","nonce":2683760795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6813","nonce":2683760795}]},"public_addr":"172.21.15.89:6809/2683760795","cluster_addr":"172.21.15.89:6811/2683760795","heartbeat_back_addr":"172.21.15.89:6815/2683760795","heartbeat_front_addr":"172.21.15.89:6813/2683760795","state":["exists","up"]},{"osd":5,"uuid":"e78954d8-f4e3-4d58-8536-d2d59c843386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6809","nonce":2635838572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6811","nonce":2635838572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6815","nonce":2635838572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6813","nonce":2635838572}]},"public_addr":"172.21.15.98:6809/2635838572","cluster_addr":"172.21.15.98:6811/2635838572","heartbeat_back_addr":"172.21.15.98:6815/2635838572","heartbeat_front_addr":"172.21.15.98:6813/2635838572","state":["exists","up"]},{"osd":6,"uuid":"dc6c8be2-b64c-4959-8015-7b0ea6d25599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6819","nonce":3158982661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6821","nonce":3158982661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6825","nonce":3158982661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6823","nonce":3158982661}]},"public_addr":"172.21.15.72:6819/3158982661","cluster_addr":"172.21.15.72:6821/3158982661","heartbeat_back_addr":"172.21.15.72:6825/3158982661","heartbeat_front_addr":"172.21.15.72:6823/3158982661","state":["exists","up"]},{"osd":7,"uuid":"7a73469f-cf3a-4306-ae93-ea46e21ebbbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6817","nonce":3075913234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6819","nonce":3075913234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6823","nonce":3075913234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6821","nonce":3075913234}]},"public_addr":"172.21.15.89:6817/3075913234","cluster_addr":"172.21.15.89:6819/3075913234","heartbeat_back_addr":"172.21.15.89:6823/3075913234","heartbeat_front_addr":"172.21.15.89:6821/3075913234","state":["exists","up"]},{"osd":8,"uuid":"c4b13bee-ea4d-4344-95fb-2622ecbb8196","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6817","nonce":1813255577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6819","nonce":1813255577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6823","nonce":1813255577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6821","nonce":1813255577}]},"public_addr":"172.21.15.98:6817/1813255577","cluster_addr":"172.21.15.98:6819/1813255577","heartbeat_back_addr":"172.21.15.98:6823/1813255577","heartbeat_front_addr":"172.21.15.98:6821/1813255577","state":["exists","up"]},{"osd":9,"uuid":"eefb0a44-4a36-4e55-9d04-0501bf718db6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6827","nonce":3377749206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6829","nonce":3377749206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6832","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6833","nonce":3377749206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6831","nonce":3377749206}]},"public_addr":"172.21.15.72:6827/3377749206","cluster_addr":"172.21.15.72:6829/3377749206","heartbeat_back_addr":"172.21.15.72:6833/3377749206","heartbeat_front_addr":"172.21.15.72:6831/3377749206","state":["exists","up"]},{"osd":10,"uuid":"8741d25f-40dd-40d7-b040-5cc5e411eb98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6825","nonce":885058786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6827","nonce":885058786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6831","nonce":885058786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6829","nonce":885058786}]},"public_addr":"172.21.15.89:6825/885058786","cluster_addr":"172.21.15.89:6827/885058786","heartbeat_back_addr":"172.21.15.89:6831/885058786","heartbeat_front_addr":"172.21.15.89:6829/885058786","state":["exists","up"]},{"osd":11,"uuid":"8d02e957-eda4-4184-b5c6-5eb3b8b38060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6825","nonce":1517095838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6827","nonce":1517095838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6831","nonce":1517095838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6829","nonce":1517095838}]},"public_addr":"172.21.15.98:6825/1517095838","cluster_addr":"172.21.15.98:6827/1517095838","heartbeat_back_addr":"172.21.15.98:6831/1517095838","heartbeat_front_addr":"172.21.15.98:6829/1517095838","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:28:47.307652+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:11.037135+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:34.619188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:59.350738+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:30:23.787337+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:30:48.994154+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:31:13.690350+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:31:39.628116+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:04.723912+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:30.689741+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:57.229508+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:33:23.154537+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:0/121087196":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/2451857761":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/1228477995":"2024-03-23T01:27:15.339809+0000","172.21.15.72:6801/1513232913":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/3558958264":"2024-03-23T01:26:50.074871+0000","172.21.15.72:0/1510187198":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6800/3518012916":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6801/3518012916":"2024-03-23T01:26:50.074871+0000","172.21.15.72:0/1372801041":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6800/1513232913":"2024-03-23T01:27:15.339809+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T01:34:37.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.826+0000 7f8437581640 1 -- 172.21.15.72:0/3435442723 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f841c0a2f20 msgr2=0x7f841c0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:37.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.826+0000 7f8437581640 1 --2- 172.21.15.72:0/3435442723 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f841c0a2f20 0x7f841c0a53e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f84200095f0 tx=0x7f8420005f90 comp rx=0 tx=0).stop 2024-03-22T01:34:37.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.826+0000 7f8437581640 1 -- 172.21.15.72:0/3435442723 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8430100b10 msgr2=0x7f8430196db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:37.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.826+0000 7f8437581640 1 --2- 172.21.15.72:0/3435442723 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8430100b10 0x7f8430196db0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f842c000c00 tx=0x7f842c005d50 comp rx=0 tx=0).stop 2024-03-22T01:34:37.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.826+0000 7f8437581640 1 -- 172.21.15.72:0/3435442723 shutdown_connections 2024-03-22T01:34:37.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.826+0000 7f8437581640 1 --2- 172.21.15.72:0/3435442723 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f841c0a2f20 0x7f841c0a53e0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:37.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.826+0000 7f8437581640 1 --2- 172.21.15.72:0/3435442723 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8430101d00 0x7f84301972f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:37.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.826+0000 7f8437581640 1 --2- 172.21.15.72:0/3435442723 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84301026c0 0x7f843019e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:37.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.826+0000 7f8437581640 1 --2- 172.21.15.72:0/3435442723 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8430100b10 0x7f8430196db0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:37.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.826+0000 7f8437581640 1 -- 172.21.15.72:0/3435442723 >> 172.21.15.72:0/3435442723 conn(0x7f84300fc0a0 msgr2=0x7f8430105080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:37.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.827+0000 7f8437581640 1 -- 172.21.15.72:0/3435442723 shutdown_connections 2024-03-22T01:34:37.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.827+0000 7f8437581640 1 -- 172.21.15.72:0/3435442723 wait complete. 2024-03-22T01:34:37.837 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-22T01:34:37.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.942+0000 7f6a71a0f640 1 -- 172.21.15.72:0/2728602140 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6a6c107870 msgr2=0x7f6a6c109cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:37.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.942+0000 7f6a71a0f640 1 --2- 172.21.15.72:0/2728602140 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6a6c107870 0x7f6a6c109cc0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f6a64009ca0 tx=0x7f6a6402c0f0 comp rx=0 tx=0).stop 2024-03-22T01:34:37.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.942+0000 7f6a71a0f640 1 -- 172.21.15.72:0/2728602140 shutdown_connections 2024-03-22T01:34:37.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.942+0000 7f6a71a0f640 1 --2- 172.21.15.72:0/2728602140 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a6c104f40 0x7f6a6c107330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:37.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.942+0000 7f6a71a0f640 1 --2- 172.21.15.72:0/2728602140 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6a6c107870 0x7f6a6c109cc0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:37.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.942+0000 7f6a71a0f640 1 --2- 172.21.15.72:0/2728602140 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a6c102610 0x7f6a6c104a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:37.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.942+0000 7f6a71a0f640 1 -- 172.21.15.72:0/2728602140 >> 172.21.15.72:0/2728602140 conn(0x7f6a6c0fbfc0 msgr2=0x7f6a6c0fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:37.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.942+0000 7f6a71a0f640 1 -- 172.21.15.72:0/2728602140 shutdown_connections 2024-03-22T01:34:37.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.942+0000 7f6a71a0f640 1 -- 172.21.15.72:0/2728602140 wait complete. 2024-03-22T01:34:37.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.943+0000 7f6a71a0f640 1 Processor -- start 2024-03-22T01:34:37.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.943+0000 7f6a71a0f640 1 -- start start 2024-03-22T01:34:37.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.943+0000 7f6a71a0f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a6c102610 0x7f6a6c197dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:37.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.943+0000 7f6a71a0f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6a6c104f40 0x7f6a6c198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:37.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.943+0000 7f6a71a0f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a6c107870 0x7f6a6c19f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:37.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.943+0000 7f6a71a0f640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f6a6c10d810 con 0x7f6a6c104f40 2024-03-22T01:34:37.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.943+0000 7f6a71a0f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f6a6c10d690 con 0x7f6a6c102610 2024-03-22T01:34:37.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.943+0000 7f6a71a0f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6a6c10d990 con 0x7f6a6c107870 2024-03-22T01:34:37.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.943+0000 7f6a6a7fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6a6c104f40 0x7f6a6c198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:37.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.943+0000 7f6a6a7fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6a6c104f40 0x7f6a6c198300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45758/0 (socket says 172.21.15.72:45758) 2024-03-22T01:34:37.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.943+0000 7f6a6a7fc640 1 -- 172.21.15.72:0/3967518793 learned_addr learned my addr 172.21.15.72:0/3967518793 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:34:37.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.943+0000 7f6a6affd640 1 --2- 172.21.15.72:0/3967518793 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a6c102610 0x7f6a6c197dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:37.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.944+0000 7f6a6a7fc640 1 -- 172.21.15.72:0/3967518793 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a6c107870 msgr2=0x7f6a6c19f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:37.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.944+0000 7f6a6a7fc640 1 --2- 172.21.15.72:0/3967518793 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a6c107870 0x7f6a6c19f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:37.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.944+0000 7f6a6a7fc640 1 -- 172.21.15.72:0/3967518793 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a6c102610 msgr2=0x7f6a6c197dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:37.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.944+0000 7f6a6a7fc640 1 --2- 172.21.15.72:0/3967518793 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a6c102610 0x7f6a6c197dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:37.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.944+0000 7f6a6a7fc640 1 -- 172.21.15.72:0/3967518793 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a6c19f9d0 con 0x7f6a6c104f40 2024-03-22T01:34:37.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.944+0000 7f6a6a7fc640 1 --2- 172.21.15.72:0/3967518793 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6a6c104f40 0x7f6a6c198300 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f6a6000e9a0 tx=0x7f6a6000eda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:37.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.944+0000 7f6a70a0d640 1 -- 172.21.15.72:0/3967518793 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a6000cd80 con 0x7f6a6c104f40 2024-03-22T01:34:37.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.944+0000 7f6a71a0f640 1 -- 172.21.15.72:0/3967518793 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a6c19fcc0 con 0x7f6a6c104f40 2024-03-22T01:34:37.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.944+0000 7f6a70a0d640 1 -- 172.21.15.72:0/3967518793 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a600035f0 con 0x7f6a6c104f40 2024-03-22T01:34:37.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.944+0000 7f6a70a0d640 1 -- 172.21.15.72:0/3967518793 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a60010690 con 0x7f6a6c104f40 2024-03-22T01:34:37.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.944+0000 7f6a71a0f640 1 -- 172.21.15.72:0/3967518793 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a6c1a0200 con 0x7f6a6c104f40 2024-03-22T01:34:37.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.945+0000 7f6a71a0f640 1 -- 172.21.15.72:0/3967518793 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a6c0fdab0 con 0x7f6a6c104f40 2024-03-22T01:34:37.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.946+0000 7f6a70a0d640 1 -- 172.21.15.72:0/3967518793 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f6a600037a0 con 0x7f6a6c104f40 2024-03-22T01:34:37.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.946+0000 7f6a70a0d640 1 --2- 172.21.15.72:0/3967518793 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f6a540a2f20 0x7f6a540a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:37.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.947+0000 7f6a6affd640 1 --2- 172.21.15.72:0/3967518793 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f6a540a2f20 0x7f6a540a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:37.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.947+0000 7f6a70a0d640 1 -- 172.21.15.72:0/3967518793 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f6a600cfa30 con 0x7f6a6c104f40 2024-03-22T01:34:37.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.947+0000 7f6a6affd640 1 --2- 172.21.15.72:0/3967518793 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f6a540a2f20 0x7f6a540a53e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f6a6c199290 tx=0x7f6a5c00a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:37.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:37.951+0000 7f6a70a0d640 1 -- 172.21.15.72:0/3967518793 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6a6009c620 con 0x7f6a6c104f40 2024-03-22T01:34:38.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:38.164+0000 7f6a71a0f640 1 -- 172.21.15.72:0/3967518793 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f6a6c02d3b0 con 0x7f6a6c104f40 2024-03-22T01:34:38.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:38.490+0000 7f6a70a0d640 1 -- 172.21.15.72:0/3967518793 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) ==== 118+0+0 (secure 0 0 0) 0x7f6a6009bf20 con 0x7f6a6c104f40 2024-03-22T01:34:38.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:38.493+0000 7f6a71a0f640 1 -- 172.21.15.72:0/3967518793 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f6a540a2f20 msgr2=0x7f6a540a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:38.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:38.493+0000 7f6a71a0f640 1 --2- 172.21.15.72:0/3967518793 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f6a540a2f20 0x7f6a540a53e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f6a6c199290 tx=0x7f6a5c00a2d0 comp rx=0 tx=0).stop 2024-03-22T01:34:38.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:38.493+0000 7f6a71a0f640 1 -- 172.21.15.72:0/3967518793 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6a6c104f40 msgr2=0x7f6a6c198300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:38.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:38.493+0000 7f6a71a0f640 1 --2- 172.21.15.72:0/3967518793 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6a6c104f40 0x7f6a6c198300 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f6a6000e9a0 tx=0x7f6a6000eda0 comp rx=0 tx=0).stop 2024-03-22T01:34:38.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:38.493+0000 7f6a71a0f640 1 -- 172.21.15.72:0/3967518793 shutdown_connections 2024-03-22T01:34:38.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:38.493+0000 7f6a71a0f640 1 --2- 172.21.15.72:0/3967518793 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f6a540a2f20 0x7f6a540a53e0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:38.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:38.493+0000 7f6a71a0f640 1 --2- 172.21.15.72:0/3967518793 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a6c107870 0x7f6a6c19f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:38.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:38.493+0000 7f6a71a0f640 1 --2- 172.21.15.72:0/3967518793 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6a6c104f40 0x7f6a6c198300 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:38.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:38.493+0000 7f6a71a0f640 1 --2- 172.21.15.72:0/3967518793 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a6c102610 0x7f6a6c197dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:38.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:38.493+0000 7f6a71a0f640 1 -- 172.21.15.72:0/3967518793 >> 172.21.15.72:0/3967518793 conn(0x7f6a6c0fbfc0 msgr2=0x7f6a6c102f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:38.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:38.493+0000 7f6a71a0f640 1 -- 172.21.15.72:0/3967518793 shutdown_connections 2024-03-22T01:34:38.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:38.493+0000 7f6a71a0f640 1 -- 172.21.15.72:0/3967518793 wait complete. 2024-03-22T01:34:38.503 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 1, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-03-22T01:34:38.503 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-03-22T01:34:38.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:38.609+0000 7f098e3b6640 1 -- 172.21.15.72:0/4189209832 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0988101460 msgr2=0x7f09881018e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:38.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:38.609+0000 7f098e3b6640 1 --2- 172.21.15.72:0/4189209832 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0988101460 0x7f09881018e0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f097c00aa10 tx=0x7f097c02cf70 comp rx=0 tx=0).stop 2024-03-22T01:34:38.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:38.609+0000 7f098e3b6640 1 -- 172.21.15.72:0/4189209832 shutdown_connections 2024-03-22T01:34:38.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:38.609+0000 7f098e3b6640 1 --2- 172.21.15.72:0/4189209832 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0988101e20 0x7f09881086b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:38.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:38.609+0000 7f098e3b6640 1 --2- 172.21.15.72:0/4189209832 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0988101460 0x7f09881018e0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:38.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:38.609+0000 7f098e3b6640 1 --2- 172.21.15.72:0/4189209832 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0988100270 0x7f0988100690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:38.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:38.609+0000 7f098e3b6640 1 -- 172.21.15.72:0/4189209832 >> 172.21.15.72:0/4189209832 conn(0x7f09880fb7e0 msgr2=0x7f09880fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:38.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:38.609+0000 7f098e3b6640 1 -- 172.21.15.72:0/4189209832 shutdown_connections 2024-03-22T01:34:38.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:38.609+0000 7f098e3b6640 1 -- 172.21.15.72:0/4189209832 wait complete. 2024-03-22T01:34:38.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:38.609+0000 7f098e3b6640 1 Processor -- start 2024-03-22T01:34:38.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:38.610+0000 7f098e3b6640 1 -- start start 2024-03-22T01:34:38.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:38.610+0000 7f098e3b6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0988100270 0x7f0988197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:38.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:38.610+0000 7f098e3b6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0988101460 0x7f09881984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:38.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:38.610+0000 7f098e3b6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0988101e20 0x7f098819f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:38.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:38.610+0000 7f098e3b6640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f098810b6e0 con 0x7f0988101e20 2024-03-22T01:34:38.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:38.610+0000 7f098e3b6640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f098810b560 con 0x7f0988101460 2024-03-22T01:34:38.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:38.610+0000 7f098e3b6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f098810b860 con 0x7f0988100270 2024-03-22T01:34:38.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:38.610+0000 7f098c953640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0988101e20 0x7f098819f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:38.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:38.610+0000 7f098c953640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0988101e20 0x7f098819f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45782/0 (socket says 172.21.15.72:45782) 2024-03-22T01:34:38.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:38.610+0000 7f098c953640 1 -- 172.21.15.72:0/3305626895 learned_addr learned my addr 172.21.15.72:0/3305626895 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:34:38.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:38.610+0000 7f09877fe640 1 --2- 172.21.15.72:0/3305626895 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0988101460 0x7f09881984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:38.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:38.610+0000 7f0987fff640 1 --2- 172.21.15.72:0/3305626895 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0988100270 0x7f0988197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:38.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:38.610+0000 7f098c953640 1 -- 172.21.15.72:0/3305626895 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0988100270 msgr2=0x7f0988197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:38.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:38.610+0000 7f098c953640 1 --2- 172.21.15.72:0/3305626895 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0988100270 0x7f0988197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:38.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:38.610+0000 7f098c953640 1 -- 172.21.15.72:0/3305626895 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0988101460 msgr2=0x7f09881984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:38.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:38.610+0000 7f098c953640 1 --2- 172.21.15.72:0/3305626895 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0988101460 0x7f09881984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:38.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:38.610+0000 7f098c953640 1 -- 172.21.15.72:0/3305626895 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f098819fb80 con 0x7f0988101e20 2024-03-22T01:34:38.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:38.610+0000 7f09877fe640 1 --2- 172.21.15.72:0/3305626895 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0988101460 0x7f09881984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:34:38.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:38.610+0000 7f098c953640 1 --2- 172.21.15.72:0/3305626895 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0988101e20 0x7f098819f510 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f098000eb10 tx=0x7f098000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:38.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:38.611+0000 7f09857fa640 1 -- 172.21.15.72:0/3305626895 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0980004b20 con 0x7f0988101e20 2024-03-22T01:34:38.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:38.611+0000 7f09857fa640 1 -- 172.21.15.72:0/3305626895 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0980004cc0 con 0x7f0988101e20 2024-03-22T01:34:38.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:38.611+0000 7f09857fa640 1 -- 172.21.15.72:0/3305626895 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0980010760 con 0x7f0988101e20 2024-03-22T01:34:38.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:38.611+0000 7f098e3b6640 1 -- 172.21.15.72:0/3305626895 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f098819fe70 con 0x7f0988101e20 2024-03-22T01:34:38.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:38.611+0000 7f098e3b6640 1 -- 172.21.15.72:0/3305626895 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09881a0360 con 0x7f0988101e20 2024-03-22T01:34:38.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:38.612+0000 7f098e3b6640 1 -- 172.21.15.72:0/3305626895 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09881a05f0 con 0x7f0988101e20 2024-03-22T01:34:38.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:38.613+0000 7f09857fa640 1 -- 172.21.15.72:0/3305626895 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f0980010900 con 0x7f0988101e20 2024-03-22T01:34:38.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:38.613+0000 7f09857fa640 1 --2- 172.21.15.72:0/3305626895 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f09700a2f20 0x7f09700a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:38.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:38.613+0000 7f0987fff640 1 --2- 172.21.15.72:0/3305626895 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f09700a2f20 0x7f09700a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:38.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:38.613+0000 7f09857fa640 1 -- 172.21.15.72:0/3305626895 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f09800cfe70 con 0x7f0988101e20 2024-03-22T01:34:38.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:38.614+0000 7f0987fff640 1 --2- 172.21.15.72:0/3305626895 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f09700a2f20 0x7f09700a53e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f09881012c0 tx=0x7f0978008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:38.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:38.617+0000 7f09857fa640 1 -- 172.21.15.72:0/3305626895 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0980053740 con 0x7f0988101e20 2024-03-22T01:34:38.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:38.831+0000 7f098e3b6640 1 -- 172.21.15.72:0/3305626895 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f09881a09e0 con 0x7f0988101e20 2024-03-22T01:34:38.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:38 smithi089 ceph-mon[35770]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-22T01:34:38.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:38 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/4147022655' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:34:38.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:38 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/3435442723' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:34:38.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:38 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/3967518793' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-22T01:34:38.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:38 smithi072 ceph-mon[32414]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-22T01:34:38.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:38 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/4147022655' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:34:38.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:38 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/3435442723' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:34:38.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:38 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/3967518793' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-22T01:34:38.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:38 smithi098 ceph-mon[36468]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-22T01:34:38.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:38 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/4147022655' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:34:38.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:38 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/3435442723' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:34:38.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:38 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/3967518793' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-22T01:34:39.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:39.496+0000 7f09857fa640 1 -- 172.21.15.72:0/3305626895 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v19) ==== 121+0+0 (secure 0 0 0) 0x7f098009ca60 con 0x7f0988101e20 2024-03-22T01:34:39.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:39.499+0000 7f098e3b6640 1 -- 172.21.15.72:0/3305626895 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f09700a2f20 msgr2=0x7f09700a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:39.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:39.499+0000 7f098e3b6640 1 --2- 172.21.15.72:0/3305626895 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f09700a2f20 0x7f09700a53e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f09881012c0 tx=0x7f0978008040 comp rx=0 tx=0).stop 2024-03-22T01:34:39.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:39.499+0000 7f098e3b6640 1 -- 172.21.15.72:0/3305626895 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0988101e20 msgr2=0x7f098819f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:39.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:39.499+0000 7f098e3b6640 1 --2- 172.21.15.72:0/3305626895 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0988101e20 0x7f098819f510 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f098000eb10 tx=0x7f098000c2d0 comp rx=0 tx=0).stop 2024-03-22T01:34:39.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:39.499+0000 7f098e3b6640 1 -- 172.21.15.72:0/3305626895 shutdown_connections 2024-03-22T01:34:39.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:39.499+0000 7f098e3b6640 1 --2- 172.21.15.72:0/3305626895 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f09700a2f20 0x7f09700a53e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:39.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:39.499+0000 7f098e3b6640 1 --2- 172.21.15.72:0/3305626895 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0988101460 0x7f09881984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:39.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:39.499+0000 7f098e3b6640 1 --2- 172.21.15.72:0/3305626895 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0988101e20 0x7f098819f510 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:39.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:39.499+0000 7f098e3b6640 1 --2- 172.21.15.72:0/3305626895 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0988100270 0x7f0988197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:39.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:39.499+0000 7f098e3b6640 1 -- 172.21.15.72:0/3305626895 >> 172.21.15.72:0/3305626895 conn(0x7f09880fb7e0 msgr2=0x7f0988106970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:39.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:39.499+0000 7f098e3b6640 1 -- 172.21.15.72:0/3305626895 shutdown_connections 2024-03-22T01:34:39.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:39.499+0000 7f098e3b6640 1 -- 172.21.15.72:0/3305626895 wait complete. 2024-03-22T01:34:39.513 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-22T01:34:39.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:39.623+0000 7f6eb6637640 1 -- 172.21.15.72:0/3699952591 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6eb0101ba0 msgr2=0x7f6eb0101fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:39.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:39.623+0000 7f6eb6637640 1 --2- 172.21.15.72:0/3699952591 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6eb0101ba0 0x7f6eb0101fc0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6ea0009900 tx=0x7f6ea002bc30 comp rx=0 tx=0).stop 2024-03-22T01:34:39.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:39.623+0000 7f6eb6637640 1 -- 172.21.15.72:0/3699952591 shutdown_connections 2024-03-22T01:34:39.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:39.623+0000 7f6eb6637640 1 --2- 172.21.15.72:0/3699952591 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6eb0102d90 0x7f6eb0103210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:39.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:39.623+0000 7f6eb6637640 1 --2- 172.21.15.72:0/3699952591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6eb0103750 0x7f6eb0109fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:39.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:39.623+0000 7f6eb6637640 1 --2- 172.21.15.72:0/3699952591 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6eb0101ba0 0x7f6eb0101fc0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:39.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:39.623+0000 7f6eb6637640 1 -- 172.21.15.72:0/3699952591 >> 172.21.15.72:0/3699952591 conn(0x7f6eb00fd110 msgr2=0x7f6eb00ff550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:39.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:39.623+0000 7f6eb6637640 1 -- 172.21.15.72:0/3699952591 shutdown_connections 2024-03-22T01:34:39.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:39.623+0000 7f6eb6637640 1 -- 172.21.15.72:0/3699952591 wait complete. 2024-03-22T01:34:39.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:39.623+0000 7f6eb6637640 1 Processor -- start 2024-03-22T01:34:39.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:39.623+0000 7f6eb6637640 1 -- start start 2024-03-22T01:34:39.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:39.624+0000 7f6eb6637640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6eb0101ba0 0x7f6eb0197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:39.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:39.624+0000 7f6eb6637640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6eb0102d90 0x7f6eb01984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:39.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:39.624+0000 7f6eb6637640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6eb0103750 0x7f6eb019f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:39.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:39.624+0000 7f6eb6637640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f6eb010d870 con 0x7f6eb0102d90 2024-03-22T01:34:39.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:39.624+0000 7f6eb6637640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f6eb010d6f0 con 0x7f6eb0103750 2024-03-22T01:34:39.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:39.624+0000 7f6eb6637640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6eb010d9f0 con 0x7f6eb0101ba0 2024-03-22T01:34:39.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:39.624+0000 7f6eaf7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6eb0102d90 0x7f6eb01984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:39.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:39.624+0000 7f6eaf7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6eb0102d90 0x7f6eb01984b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45808/0 (socket says 172.21.15.72:45808) 2024-03-22T01:34:39.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:39.624+0000 7f6eaf7fe640 1 -- 172.21.15.72:0/2925059628 learned_addr learned my addr 172.21.15.72:0/2925059628 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:34:39.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:39.624+0000 7f6eb4bd4640 1 --2- 172.21.15.72:0/2925059628 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6eb0103750 0x7f6eb019f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:39.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:39.624+0000 7f6eaffff640 1 --2- 172.21.15.72:0/2925059628 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6eb0101ba0 0x7f6eb0197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:39.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:39.624+0000 7f6eaf7fe640 1 -- 172.21.15.72:0/2925059628 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6eb0101ba0 msgr2=0x7f6eb0197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:39.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:39.624+0000 7f6eaf7fe640 1 --2- 172.21.15.72:0/2925059628 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6eb0101ba0 0x7f6eb0197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:39.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:39.624+0000 7f6eaf7fe640 1 -- 172.21.15.72:0/2925059628 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6eb0103750 msgr2=0x7f6eb019f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:39.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:39.624+0000 7f6eaf7fe640 1 --2- 172.21.15.72:0/2925059628 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6eb0103750 0x7f6eb019f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:39.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:39.624+0000 7f6eaf7fe640 1 -- 172.21.15.72:0/2925059628 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6eb019fb90 con 0x7f6eb0102d90 2024-03-22T01:34:39.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:39.624+0000 7f6eaffff640 1 --2- 172.21.15.72:0/2925059628 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6eb0101ba0 0x7f6eb0197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T01:34:39.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:39.624+0000 7f6eaf7fe640 1 --2- 172.21.15.72:0/2925059628 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6eb0102d90 0x7f6eb01984b0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f6e9800e9f0 tx=0x7f6e9800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:39.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:39.624+0000 7f6ead7fa640 1 -- 172.21.15.72:0/2925059628 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e9800cd80 con 0x7f6eb0102d90 2024-03-22T01:34:39.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:39.624+0000 7f6eb6637640 1 -- 172.21.15.72:0/2925059628 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6eb019fe80 con 0x7f6eb0102d90 2024-03-22T01:34:39.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:39.624+0000 7f6ead7fa640 1 -- 172.21.15.72:0/2925059628 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e98010910 con 0x7f6eb0102d90 2024-03-22T01:34:39.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:39.624+0000 7f6ead7fa640 1 -- 172.21.15.72:0/2925059628 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e980188f0 con 0x7f6eb0102d90 2024-03-22T01:34:39.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:39.624+0000 7f6eb6637640 1 -- 172.21.15.72:0/2925059628 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6eb01a0340 con 0x7f6eb0102d90 2024-03-22T01:34:39.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:39.625+0000 7f6eb6637640 1 -- 172.21.15.72:0/2925059628 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6eb0108f10 con 0x7f6eb0102d90 2024-03-22T01:34:39.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:39.626+0000 7f6ead7fa640 1 -- 172.21.15.72:0/2925059628 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f6e98010ac0 con 0x7f6eb0102d90 2024-03-22T01:34:39.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:39.627+0000 7f6ead7fa640 1 --2- 172.21.15.72:0/2925059628 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f6e9c0a2f20 0x7f6e9c0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:39.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:39.627+0000 7f6eaffff640 1 --2- 172.21.15.72:0/2925059628 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f6e9c0a2f20 0x7f6e9c0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:39.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:39.627+0000 7f6ead7fa640 1 -- 172.21.15.72:0/2925059628 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f6e980cfa70 con 0x7f6eb0102d90 2024-03-22T01:34:39.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:39.627+0000 7f6eaffff640 1 --2- 172.21.15.72:0/2925059628 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f6e9c0a2f20 0x7f6e9c0a53e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6ea0006010 tx=0x7f6ea0000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:39.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:39.631+0000 7f6ead7fa640 1 -- 172.21.15.72:0/2925059628 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6e9809c660 con 0x7f6eb0102d90 2024-03-22T01:34:39.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:39.841+0000 7f6eb6637640 1 -- 172.21.15.72:0/2925059628 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f6eb0062790 con 0x7f6eb0102d90 2024-03-22T01:34:39.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:39 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/3967518793' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-22T01:34:39.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:39 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-22T01:34:39.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:39 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/3305626895' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-22T01:34:39.871 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:39 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/3967518793' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-22T01:34:39.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:39 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-22T01:34:39.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:39 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/3305626895' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-22T01:34:39.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:39 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/3967518793' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-22T01:34:39.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:39 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-22T01:34:39.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:39 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/3305626895' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-22T01:34:40.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.505+0000 7f6ead7fa640 1 -- 172.21.15.72:0/2925059628 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v21) ==== 115+0+0 (secure 0 0 0) 0x7f6e9809bf60 con 0x7f6eb0102d90 2024-03-22T01:34:40.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.507+0000 7f6eb6637640 1 -- 172.21.15.72:0/2925059628 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f6e9c0a2f20 msgr2=0x7f6e9c0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:40.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.507+0000 7f6eb6637640 1 --2- 172.21.15.72:0/2925059628 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f6e9c0a2f20 0x7f6e9c0a53e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6ea0006010 tx=0x7f6ea0000bc0 comp rx=0 tx=0).stop 2024-03-22T01:34:40.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.507+0000 7f6eb6637640 1 -- 172.21.15.72:0/2925059628 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6eb0102d90 msgr2=0x7f6eb01984b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:40.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.507+0000 7f6eb6637640 1 --2- 172.21.15.72:0/2925059628 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6eb0102d90 0x7f6eb01984b0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f6e9800e9f0 tx=0x7f6e9800eeb0 comp rx=0 tx=0).stop 2024-03-22T01:34:40.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.508+0000 7f6eb6637640 1 -- 172.21.15.72:0/2925059628 shutdown_connections 2024-03-22T01:34:40.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.508+0000 7f6eb6637640 1 --2- 172.21.15.72:0/2925059628 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f6e9c0a2f20 0x7f6e9c0a53e0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:40.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.508+0000 7f6eb6637640 1 --2- 172.21.15.72:0/2925059628 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6eb0103750 0x7f6eb019f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:40.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.508+0000 7f6eb6637640 1 --2- 172.21.15.72:0/2925059628 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6eb0102d90 0x7f6eb01984b0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:40.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.508+0000 7f6eb6637640 1 --2- 172.21.15.72:0/2925059628 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6eb0101ba0 0x7f6eb0197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:40.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.508+0000 7f6eb6637640 1 -- 172.21.15.72:0/2925059628 >> 172.21.15.72:0/2925059628 conn(0x7f6eb00fd110 msgr2=0x7f6eb01060e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:40.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.508+0000 7f6eb6637640 1 -- 172.21.15.72:0/2925059628 shutdown_connections 2024-03-22T01:34:40.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.508+0000 7f6eb6637640 1 -- 172.21.15.72:0/2925059628 wait complete. 2024-03-22T01:34:40.519 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'} subvols for filesystem 'cephfs' 2024-03-22T01:34:40.519 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-03-22T01:34:40.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.626+0000 7ff21e837640 1 -- 172.21.15.72:0/120128771 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2181026b0 msgr2=0x7ff218104aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:40.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.626+0000 7ff21e837640 1 --2- 172.21.15.72:0/120128771 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2181026b0 0x7ff218104aa0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ff208009900 tx=0x7ff20802bc30 comp rx=0 tx=0).stop 2024-03-22T01:34:40.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.626+0000 7ff21e837640 1 -- 172.21.15.72:0/120128771 shutdown_connections 2024-03-22T01:34:40.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.626+0000 7ff21e837640 1 --2- 172.21.15.72:0/120128771 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff218104fe0 0x7ff2181073d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:40.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.626+0000 7ff21e837640 1 --2- 172.21.15.72:0/120128771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff218107910 0x7ff218109d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:40.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.626+0000 7ff21e837640 1 --2- 172.21.15.72:0/120128771 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2181026b0 0x7ff218104aa0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:40.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.626+0000 7ff21e837640 1 -- 172.21.15.72:0/120128771 >> 172.21.15.72:0/120128771 conn(0x7ff2180fbfc0 msgr2=0x7ff2180fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:40.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.626+0000 7ff21e837640 1 -- 172.21.15.72:0/120128771 shutdown_connections 2024-03-22T01:34:40.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.626+0000 7ff21e837640 1 -- 172.21.15.72:0/120128771 wait complete. 2024-03-22T01:34:40.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.627+0000 7ff21e837640 1 Processor -- start 2024-03-22T01:34:40.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.627+0000 7ff21e837640 1 -- start start 2024-03-22T01:34:40.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.627+0000 7ff21e837640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2181026b0 0x7ff218197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:40.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.627+0000 7ff21e837640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff218104fe0 0x7ff218198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:40.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.627+0000 7ff21e837640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff218107910 0x7ff21819f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:40.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.627+0000 7ff21e837640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7ff21810d820 con 0x7ff218107910 2024-03-22T01:34:40.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.627+0000 7ff21e837640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff21810d6a0 con 0x7ff2181026b0 2024-03-22T01:34:40.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.627+0000 7ff21e837640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff21810d9a0 con 0x7ff218104fe0 2024-03-22T01:34:40.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.627+0000 7ff21cdd4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff218107910 0x7ff21819f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:40.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.627+0000 7ff21cdd4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff218107910 0x7ff21819f470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45814/0 (socket says 172.21.15.72:45814) 2024-03-22T01:34:40.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.627+0000 7ff21cdd4640 1 -- 172.21.15.72:0/2993280045 learned_addr learned my addr 172.21.15.72:0/2993280045 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:34:40.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.627+0000 7ff217fff640 1 --2- 172.21.15.72:0/2993280045 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2181026b0 0x7ff218197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:40.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.628+0000 7ff2177fe640 1 --2- 172.21.15.72:0/2993280045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff218104fe0 0x7ff218198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:40.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.628+0000 7ff21cdd4640 1 -- 172.21.15.72:0/2993280045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff218104fe0 msgr2=0x7ff218198400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:40.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.628+0000 7ff21cdd4640 1 --2- 172.21.15.72:0/2993280045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff218104fe0 0x7ff218198400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:40.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.628+0000 7ff21cdd4640 1 -- 172.21.15.72:0/2993280045 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2181026b0 msgr2=0x7ff218197ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:40.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.628+0000 7ff21cdd4640 1 --2- 172.21.15.72:0/2993280045 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2181026b0 0x7ff218197ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:40.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.628+0000 7ff21cdd4640 1 -- 172.21.15.72:0/2993280045 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff21819fae0 con 0x7ff218107910 2024-03-22T01:34:40.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.628+0000 7ff21cdd4640 1 --2- 172.21.15.72:0/2993280045 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff218107910 0x7ff21819f470 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ff21000cad0 tx=0x7ff21000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:40.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.628+0000 7ff2157fa640 1 -- 172.21.15.72:0/2993280045 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff21000ae20 con 0x7ff218107910 2024-03-22T01:34:40.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.628+0000 7ff2157fa640 1 -- 172.21.15.72:0/2993280045 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff210005020 con 0x7ff218107910 2024-03-22T01:34:40.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.628+0000 7ff21e837640 1 -- 172.21.15.72:0/2993280045 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff21819fdd0 con 0x7ff218107910 2024-03-22T01:34:40.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.628+0000 7ff2157fa640 1 -- 172.21.15.72:0/2993280045 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff21000f6f0 con 0x7ff218107910 2024-03-22T01:34:40.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.628+0000 7ff21e837640 1 -- 172.21.15.72:0/2993280045 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2181a0290 con 0x7ff218107910 2024-03-22T01:34:40.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.629+0000 7ff21e837640 1 -- 172.21.15.72:0/2993280045 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2180fdab0 con 0x7ff218107910 2024-03-22T01:34:40.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.630+0000 7ff2157fa640 1 -- 172.21.15.72:0/2993280045 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7ff2100051d0 con 0x7ff218107910 2024-03-22T01:34:40.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.630+0000 7ff2157fa640 1 --2- 172.21.15.72:0/2993280045 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7ff2000a2f20 0x7ff2000a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:40.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.631+0000 7ff217fff640 1 --2- 172.21.15.72:0/2993280045 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7ff2000a2f20 0x7ff2000a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:40.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.631+0000 7ff2157fa640 1 -- 172.21.15.72:0/2993280045 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7ff2100cf9c0 con 0x7ff218107910 2024-03-22T01:34:40.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.631+0000 7ff217fff640 1 --2- 172.21.15.72:0/2993280045 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7ff2000a2f20 0x7ff2000a53e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff20802c7b0 tx=0x7ff208005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:40.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.635+0000 7ff2157fa640 1 -- 172.21.15.72:0/2993280045 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff2100d1e20 con 0x7ff218107910 2024-03-22T01:34:40.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.834+0000 7ff21e837640 1 -- 172.21.15.72:0/2993280045 --> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7ff218061fd0 con 0x7ff2000a2f20 2024-03-22T01:34:40.838 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:40 smithi072 ceph-mon[32414]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-22T01:34:40.838 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:40 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/3305626895' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-22T01:34:40.839 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:40 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-22T01:34:40.839 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:40 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:34:40.839 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:40 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/2925059628' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-22T01:34:40.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.855+0000 7ff2157fa640 1 -- 172.21.15.72:0/2993280045 <== mgr.14152 v2:172.21.15.72:6800/401403200 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7ff218061fd0 con 0x7ff2000a2f20 2024-03-22T01:34:40.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.857+0000 7ff21e837640 1 -- 172.21.15.72:0/2993280045 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7ff2000a2f20 msgr2=0x7ff2000a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:40.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.857+0000 7ff21e837640 1 --2- 172.21.15.72:0/2993280045 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7ff2000a2f20 0x7ff2000a53e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff20802c7b0 tx=0x7ff208005fd0 comp rx=0 tx=0).stop 2024-03-22T01:34:40.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.857+0000 7ff21e837640 1 -- 172.21.15.72:0/2993280045 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff218107910 msgr2=0x7ff21819f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:40.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.857+0000 7ff21e837640 1 --2- 172.21.15.72:0/2993280045 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff218107910 0x7ff21819f470 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ff21000cad0 tx=0x7ff21000cf90 comp rx=0 tx=0).stop 2024-03-22T01:34:40.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.857+0000 7ff21e837640 1 -- 172.21.15.72:0/2993280045 shutdown_connections 2024-03-22T01:34:40.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.857+0000 7ff21e837640 1 --2- 172.21.15.72:0/2993280045 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7ff2000a2f20 0x7ff2000a53e0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:40.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.857+0000 7ff21e837640 1 --2- 172.21.15.72:0/2993280045 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff218107910 0x7ff21819f470 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:40.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.857+0000 7ff21e837640 1 --2- 172.21.15.72:0/2993280045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff218104fe0 0x7ff218198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:40.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.857+0000 7ff21e837640 1 --2- 172.21.15.72:0/2993280045 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2181026b0 0x7ff218197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:40.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.857+0000 7ff21e837640 1 -- 172.21.15.72:0/2993280045 >> 172.21.15.72:0/2993280045 conn(0x7ff2180fbfc0 msgr2=0x7ff218102f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:40.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.857+0000 7ff21e837640 1 -- 172.21.15.72:0/2993280045 shutdown_connections 2024-03-22T01:34:40.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.857+0000 7ff21e837640 1 -- 172.21.15.72:0/2993280045 wait complete. 2024-03-22T01:34:40.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:40 smithi089 ceph-mon[35770]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-22T01:34:40.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:40 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/3305626895' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-22T01:34:40.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:40 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-22T01:34:40.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:40 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:34:40.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:40 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/2925059628' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-22T01:34:40.867 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-03-22T01:34:40.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:40 smithi098 ceph-mon[36468]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-22T01:34:40.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:40 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/3305626895' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-22T01:34:40.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:40 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-22T01:34:40.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:40 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:34:40.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:40 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/2925059628' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-22T01:34:40.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.975+0000 7ff2c4847640 1 -- 172.21.15.72:0/3349223582 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2c0102630 msgr2=0x7ff2c0104a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:40.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.975+0000 7ff2c4847640 1 --2- 172.21.15.72:0/3349223582 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2c0102630 0x7ff2c0104a20 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7ff2b0009900 tx=0x7ff2b002bc30 comp rx=0 tx=0).stop 2024-03-22T01:34:40.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.975+0000 7ff2c4847640 1 -- 172.21.15.72:0/3349223582 shutdown_connections 2024-03-22T01:34:40.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.975+0000 7ff2c4847640 1 --2- 172.21.15.72:0/3349223582 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2c0104f60 0x7ff2c0107350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:40.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.975+0000 7ff2c4847640 1 --2- 172.21.15.72:0/3349223582 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2c0107890 0x7ff2c0109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:40.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.975+0000 7ff2c4847640 1 --2- 172.21.15.72:0/3349223582 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2c0102630 0x7ff2c0104a20 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:40.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.975+0000 7ff2c4847640 1 -- 172.21.15.72:0/3349223582 >> 172.21.15.72:0/3349223582 conn(0x7ff2c00fc020 msgr2=0x7ff2c00fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:40.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.975+0000 7ff2c4847640 1 -- 172.21.15.72:0/3349223582 shutdown_connections 2024-03-22T01:34:40.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.975+0000 7ff2c4847640 1 -- 172.21.15.72:0/3349223582 wait complete. 2024-03-22T01:34:40.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.976+0000 7ff2c4847640 1 Processor -- start 2024-03-22T01:34:40.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.976+0000 7ff2c4847640 1 -- start start 2024-03-22T01:34:40.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.976+0000 7ff2c4847640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2c0102630 0x7ff2c0197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:40.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.976+0000 7ff2c4847640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2c0104f60 0x7ff2c0198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:40.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.976+0000 7ff2c4847640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2c0107890 0x7ff2c019f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:40.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.976+0000 7ff2c4847640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7ff2c010d850 con 0x7ff2c0102630 2024-03-22T01:34:40.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.976+0000 7ff2c4847640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff2c010d6d0 con 0x7ff2c0107890 2024-03-22T01:34:40.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.976+0000 7ff2c4847640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff2c010d9d0 con 0x7ff2c0104f60 2024-03-22T01:34:40.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.976+0000 7ff2be59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2c0102630 0x7ff2c0197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:40.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.976+0000 7ff2be59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2c0102630 0x7ff2c0197f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37364/0 (socket says 172.21.15.72:37364) 2024-03-22T01:34:40.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.976+0000 7ff2be59c640 1 -- 172.21.15.72:0/3726894977 learned_addr learned my addr 172.21.15.72:0/3726894977 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:34:40.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.976+0000 7ff2bed9d640 1 --2- 172.21.15.72:0/3726894977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2c0107890 0x7ff2c019f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:40.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.977+0000 7ff2bdd9b640 1 --2- 172.21.15.72:0/3726894977 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2c0104f60 0x7ff2c0198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:40.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.977+0000 7ff2be59c640 1 -- 172.21.15.72:0/3726894977 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2c0104f60 msgr2=0x7ff2c0198460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:40.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.977+0000 7ff2be59c640 1 --2- 172.21.15.72:0/3726894977 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2c0104f60 0x7ff2c0198460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:40.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.977+0000 7ff2be59c640 1 -- 172.21.15.72:0/3726894977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2c0107890 msgr2=0x7ff2c019f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:40.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.977+0000 7ff2be59c640 1 --2- 172.21.15.72:0/3726894977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2c0107890 0x7ff2c019f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:40.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.977+0000 7ff2be59c640 1 -- 172.21.15.72:0/3726894977 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2c019fb40 con 0x7ff2c0102630 2024-03-22T01:34:40.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.977+0000 7ff2be59c640 1 --2- 172.21.15.72:0/3726894977 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2c0102630 0x7ff2c0197f20 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7ff2b0009230 tx=0x7ff2b0005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:40.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.977+0000 7ff2af7fe640 1 -- 172.21.15.72:0/3726894977 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2b0041070 con 0x7ff2c0102630 2024-03-22T01:34:40.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.977+0000 7ff2af7fe640 1 -- 172.21.15.72:0/3726894977 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff2b00044d0 con 0x7ff2c0102630 2024-03-22T01:34:40.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.977+0000 7ff2af7fe640 1 -- 172.21.15.72:0/3726894977 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2b0034440 con 0x7ff2c0102630 2024-03-22T01:34:40.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.977+0000 7ff2c4847640 1 -- 172.21.15.72:0/3726894977 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2c019fdd0 con 0x7ff2c0102630 2024-03-22T01:34:40.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.977+0000 7ff2c4847640 1 -- 172.21.15.72:0/3726894977 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2c01a0200 con 0x7ff2c0102630 2024-03-22T01:34:40.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.978+0000 7ff2c4847640 1 -- 172.21.15.72:0/3726894977 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2c00fdb10 con 0x7ff2c0102630 2024-03-22T01:34:40.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.979+0000 7ff2af7fe640 1 -- 172.21.15.72:0/3726894977 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7ff2b0044070 con 0x7ff2c0102630 2024-03-22T01:34:40.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.980+0000 7ff2af7fe640 1 --2- 172.21.15.72:0/3726894977 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7ff2a80a2f20 0x7ff2a80a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:40.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.980+0000 7ff2bdd9b640 1 --2- 172.21.15.72:0/3726894977 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7ff2a80a2f20 0x7ff2a80a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:40.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.980+0000 7ff2af7fe640 1 -- 172.21.15.72:0/3726894977 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7ff2b00f1200 con 0x7ff2c0102630 2024-03-22T01:34:40.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.980+0000 7ff2bdd9b640 1 --2- 172.21.15.72:0/3726894977 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7ff2a80a2f20 0x7ff2a80a53e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ff2c0199430 tx=0x7ff2b4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:40.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:40.984+0000 7ff2af7fe640 1 -- 172.21.15.72:0/3726894977 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff2b00bd7a0 con 0x7ff2c0102630 2024-03-22T01:34:41.122 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:34:40 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:34:40.838+0000 7f617b17f700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-22T01:34:41.122 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:34:40 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:34:40.838+0000 7f617b17f700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-22T01:34:41.122 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:34:40 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:34:40.838+0000 7f617b17f700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-22T01:34:41.122 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:34:40 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:34:40.838+0000 7f617b17f700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-22T01:34:41.122 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:34:40 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:34:40.838+0000 7f617b17f700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-22T01:34:41.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.189+0000 7ff2c4847640 1 -- 172.21.15.72:0/3726894977 --> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7ff2c0062060 con 0x7ff2a80a2f20 2024-03-22T01:34:41.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.213+0000 7ff2af7fe640 1 -- 172.21.15.72:0/3726894977 <== mgr.14152 v2:172.21.15.72:6800/401403200 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7ff2c0062060 con 0x7ff2a80a2f20 2024-03-22T01:34:41.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.215+0000 7ff2c4847640 1 -- 172.21.15.72:0/3726894977 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7ff2a80a2f20 msgr2=0x7ff2a80a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:41.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.215+0000 7ff2c4847640 1 --2- 172.21.15.72:0/3726894977 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7ff2a80a2f20 0x7ff2a80a53e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ff2c0199430 tx=0x7ff2b4005f90 comp rx=0 tx=0).stop 2024-03-22T01:34:41.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.215+0000 7ff2c4847640 1 -- 172.21.15.72:0/3726894977 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2c0102630 msgr2=0x7ff2c0197f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:41.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.215+0000 7ff2c4847640 1 --2- 172.21.15.72:0/3726894977 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2c0102630 0x7ff2c0197f20 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7ff2b0009230 tx=0x7ff2b0005d50 comp rx=0 tx=0).stop 2024-03-22T01:34:41.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.215+0000 7ff2c4847640 1 -- 172.21.15.72:0/3726894977 shutdown_connections 2024-03-22T01:34:41.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.215+0000 7ff2c4847640 1 --2- 172.21.15.72:0/3726894977 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7ff2a80a2f20 0x7ff2a80a53e0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:41.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.215+0000 7ff2c4847640 1 --2- 172.21.15.72:0/3726894977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2c0107890 0x7ff2c019f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:41.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.215+0000 7ff2c4847640 1 --2- 172.21.15.72:0/3726894977 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2c0104f60 0x7ff2c0198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:41.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.215+0000 7ff2c4847640 1 --2- 172.21.15.72:0/3726894977 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2c0102630 0x7ff2c0197f20 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:41.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.215+0000 7ff2c4847640 1 -- 172.21.15.72:0/3726894977 >> 172.21.15.72:0/3726894977 conn(0x7ff2c00fc020 msgr2=0x7ff2c01030f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:41.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.215+0000 7ff2c4847640 1 -- 172.21.15.72:0/3726894977 shutdown_connections 2024-03-22T01:34:41.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.215+0000 7ff2c4847640 1 -- 172.21.15.72:0/3726894977 wait complete. 2024-03-22T01:34:41.225 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-03-22T01:34:41.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.333+0000 7fd95eb8c640 1 -- 172.21.15.72:0/3464196621 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9581016e0 msgr2=0x7fd958108f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:41.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.333+0000 7fd95eb8c640 1 --2- 172.21.15.72:0/3464196621 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9581016e0 0x7fd958108f80 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fd950009ca0 tx=0x7fd95002c0f0 comp rx=0 tx=0).stop 2024-03-22T01:34:41.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.333+0000 7fd95eb8c640 1 -- 172.21.15.72:0/3464196621 shutdown_connections 2024-03-22T01:34:41.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.333+0000 7fd95eb8c640 1 --2- 172.21.15.72:0/3464196621 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9581016e0 0x7fd958108f80 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:41.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.333+0000 7fd95eb8c640 1 --2- 172.21.15.72:0/3464196621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd958100d20 0x7fd9581011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:41.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.333+0000 7fd95eb8c640 1 --2- 172.21.15.72:0/3464196621 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9581020d0 0x7fd9581024f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:41.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.333+0000 7fd95eb8c640 1 -- 172.21.15.72:0/3464196621 >> 172.21.15.72:0/3464196621 conn(0x7fd9580fb7b0 msgr2=0x7fd9580fdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:41.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.333+0000 7fd95eb8c640 1 -- 172.21.15.72:0/3464196621 shutdown_connections 2024-03-22T01:34:41.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.333+0000 7fd95eb8c640 1 -- 172.21.15.72:0/3464196621 wait complete. 2024-03-22T01:34:41.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.334+0000 7fd95eb8c640 1 Processor -- start 2024-03-22T01:34:41.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.334+0000 7fd95eb8c640 1 -- start start 2024-03-22T01:34:41.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.334+0000 7fd95eb8c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd958100d20 0x7fd958104ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:41.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.334+0000 7fd95eb8c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9581016e0 0x7fd958103110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:41.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.334+0000 7fd95eb8c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9581020d0 0x7fd958103650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:41.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.334+0000 7fd95eb8c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fd95810c730 con 0x7fd9581016e0 2024-03-22T01:34:41.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.334+0000 7fd95eb8c640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd95810c5b0 con 0x7fd958100d20 2024-03-22T01:34:41.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.334+0000 7fd95eb8c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd95810c8b0 con 0x7fd9581020d0 2024-03-22T01:34:41.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.334+0000 7fd957fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9581016e0 0x7fd958103110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:41.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.334+0000 7fd957fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9581016e0 0x7fd958103110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37368/0 (socket says 172.21.15.72:37368) 2024-03-22T01:34:41.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.334+0000 7fd957fff640 1 -- 172.21.15.72:0/3582889971 learned_addr learned my addr 172.21.15.72:0/3582889971 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:34:41.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.334+0000 7fd95c928640 1 --2- 172.21.15.72:0/3582889971 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd958100d20 0x7fd958104ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:41.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.334+0000 7fd95d129640 1 --2- 172.21.15.72:0/3582889971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9581020d0 0x7fd958103650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:41.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.334+0000 7fd957fff640 1 -- 172.21.15.72:0/3582889971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9581020d0 msgr2=0x7fd958103650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:41.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.334+0000 7fd957fff640 1 --2- 172.21.15.72:0/3582889971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9581020d0 0x7fd958103650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:41.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.334+0000 7fd957fff640 1 -- 172.21.15.72:0/3582889971 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd958100d20 msgr2=0x7fd958104ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:41.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.334+0000 7fd957fff640 1 --2- 172.21.15.72:0/3582889971 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd958100d20 0x7fd958104ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:41.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.334+0000 7fd957fff640 1 -- 172.21.15.72:0/3582889971 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd958103ee0 con 0x7fd9581016e0 2024-03-22T01:34:41.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.334+0000 7fd957fff640 1 --2- 172.21.15.72:0/3582889971 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9581016e0 0x7fd958103110 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fd94400c970 tx=0x7fd94400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:41.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.335+0000 7fd955ffb640 1 -- 172.21.15.72:0/3582889971 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd944007c10 con 0x7fd9581016e0 2024-03-22T01:34:41.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.335+0000 7fd955ffb640 1 -- 172.21.15.72:0/3582889971 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd944007db0 con 0x7fd9581016e0 2024-03-22T01:34:41.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.335+0000 7fd95eb8c640 1 -- 172.21.15.72:0/3582889971 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9581a1f20 con 0x7fd9581016e0 2024-03-22T01:34:41.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.335+0000 7fd955ffb640 1 -- 172.21.15.72:0/3582889971 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd94400f940 con 0x7fd9581016e0 2024-03-22T01:34:41.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.335+0000 7fd95eb8c640 1 -- 172.21.15.72:0/3582889971 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9581a2390 con 0x7fd9581016e0 2024-03-22T01:34:41.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.336+0000 7fd95eb8c640 1 -- 172.21.15.72:0/3582889971 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9581a26b0 con 0x7fd9581016e0 2024-03-22T01:34:41.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.337+0000 7fd955ffb640 1 -- 172.21.15.72:0/3582889971 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7fd94400fae0 con 0x7fd9581016e0 2024-03-22T01:34:41.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.337+0000 7fd955ffb640 1 --2- 172.21.15.72:0/3582889971 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fd9480a2fa0 0x7fd9480a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:41.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.337+0000 7fd95c928640 1 --2- 172.21.15.72:0/3582889971 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fd9480a2fa0 0x7fd9480a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:41.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.337+0000 7fd955ffb640 1 -- 172.21.15.72:0/3582889971 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fd9440cf930 con 0x7fd9581016e0 2024-03-22T01:34:41.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.338+0000 7fd95c928640 1 --2- 172.21.15.72:0/3582889971 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fd9480a2fa0 0x7fd9480a5460 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd95806a070 tx=0x7fd94c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:41.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.342+0000 7fd955ffb640 1 -- 172.21.15.72:0/3582889971 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd94409c5b0 con 0x7fd9581016e0 2024-03-22T01:34:41.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.547+0000 7fd95eb8c640 1 -- 172.21.15.72:0/3582889971 --> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7fd9580fd8a0 con 0x7fd9480a2fa0 2024-03-22T01:34:41.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.553+0000 7fd955ffb640 1 -- 172.21.15.72:0/3582889971 <== mgr.14152 v2:172.21.15.72:6800/401403200 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7fd9580fd8a0 con 0x7fd9480a2fa0 2024-03-22T01:34:41.552 INFO:teuthology.orchestra.run.smithi072.stdout:/volumes/qa/sv_0/cfced9fa-cf60-4248-a2d0-33320cad6f83 2024-03-22T01:34:41.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.555+0000 7fd95eb8c640 1 -- 172.21.15.72:0/3582889971 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fd9480a2fa0 msgr2=0x7fd9480a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:41.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.555+0000 7fd95eb8c640 1 --2- 172.21.15.72:0/3582889971 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fd9480a2fa0 0x7fd9480a5460 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd95806a070 tx=0x7fd94c005f90 comp rx=0 tx=0).stop 2024-03-22T01:34:41.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.555+0000 7fd95eb8c640 1 -- 172.21.15.72:0/3582889971 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9581016e0 msgr2=0x7fd958103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:41.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.555+0000 7fd95eb8c640 1 --2- 172.21.15.72:0/3582889971 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9581016e0 0x7fd958103110 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fd94400c970 tx=0x7fd94400ce30 comp rx=0 tx=0).stop 2024-03-22T01:34:41.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.555+0000 7fd95eb8c640 1 -- 172.21.15.72:0/3582889971 shutdown_connections 2024-03-22T01:34:41.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.555+0000 7fd95eb8c640 1 --2- 172.21.15.72:0/3582889971 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fd9480a2fa0 0x7fd9480a5460 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:41.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.555+0000 7fd95eb8c640 1 --2- 172.21.15.72:0/3582889971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9581020d0 0x7fd958103650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:41.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.555+0000 7fd95eb8c640 1 --2- 172.21.15.72:0/3582889971 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9581016e0 0x7fd958103110 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:41.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.555+0000 7fd95eb8c640 1 --2- 172.21.15.72:0/3582889971 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd958100d20 0x7fd958104ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:41.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.555+0000 7fd95eb8c640 1 -- 172.21.15.72:0/3582889971 >> 172.21.15.72:0/3582889971 conn(0x7fd9580fb7b0 msgr2=0x7fd9580fd190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:41.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.555+0000 7fd95eb8c640 1 -- 172.21.15.72:0/3582889971 shutdown_connections 2024-03-22T01:34:41.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.555+0000 7fd95eb8c640 1 -- 172.21.15.72:0/3582889971 wait complete. 2024-03-22T01:34:41.565 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-03-22T01:34:41.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.672+0000 7f6561a8f640 1 -- 172.21.15.72:0/2258992379 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f655c100ab0 msgr2=0x7f655c100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:41.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.672+0000 7f6561a8f640 1 --2- 172.21.15.72:0/2258992379 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f655c100ab0 0x7f655c100ed0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f654c009900 tx=0x7f654c02bc30 comp rx=0 tx=0).stop 2024-03-22T01:34:41.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.672+0000 7f6561a8f640 1 -- 172.21.15.72:0/2258992379 shutdown_connections 2024-03-22T01:34:41.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.672+0000 7f6561a8f640 1 --2- 172.21.15.72:0/2258992379 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f655c102660 0x7f655c108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:41.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.672+0000 7f6561a8f640 1 --2- 172.21.15.72:0/2258992379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f655c101ca0 0x7f655c102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:41.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.672+0000 7f6561a8f640 1 --2- 172.21.15.72:0/2258992379 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f655c100ab0 0x7f655c100ed0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:41.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.672+0000 7f6561a8f640 1 -- 172.21.15.72:0/2258992379 >> 172.21.15.72:0/2258992379 conn(0x7f655c0fc020 msgr2=0x7f655c0fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:41.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.672+0000 7f6561a8f640 1 -- 172.21.15.72:0/2258992379 shutdown_connections 2024-03-22T01:34:41.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.672+0000 7f6561a8f640 1 -- 172.21.15.72:0/2258992379 wait complete. 2024-03-22T01:34:41.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.672+0000 7f6561a8f640 1 Processor -- start 2024-03-22T01:34:41.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.672+0000 7f6561a8f640 1 -- start start 2024-03-22T01:34:41.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.673+0000 7f6561a8f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f655c100ab0 0x7f655c06ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:41.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.673+0000 7f6561a8f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f655c101ca0 0x7f655c19cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:41.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.673+0000 7f6561a8f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f655c102660 0x7f655c19f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:41.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.673+0000 7f6561a8f640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f655c10c720 con 0x7f655c100ab0 2024-03-22T01:34:41.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.673+0000 7f6561a8f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f655c10c5a0 con 0x7f655c102660 2024-03-22T01:34:41.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.673+0000 7f6561a8f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f655c10c8a0 con 0x7f655c101ca0 2024-03-22T01:34:41.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.673+0000 7f655b7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f655c100ab0 0x7f655c06ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:41.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.673+0000 7f655b7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f655c100ab0 0x7f655c06ae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37386/0 (socket says 172.21.15.72:37386) 2024-03-22T01:34:41.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.673+0000 7f655b7fe640 1 -- 172.21.15.72:0/1700689160 learned_addr learned my addr 172.21.15.72:0/1700689160 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:34:41.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.673+0000 7f655bfff640 1 --2- 172.21.15.72:0/1700689160 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f655c102660 0x7f655c19f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:41.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.673+0000 7f655affd640 1 --2- 172.21.15.72:0/1700689160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f655c101ca0 0x7f655c19cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:41.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.673+0000 7f655b7fe640 1 -- 172.21.15.72:0/1700689160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f655c101ca0 msgr2=0x7f655c19cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:41.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.673+0000 7f655b7fe640 1 --2- 172.21.15.72:0/1700689160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f655c101ca0 0x7f655c19cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:41.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.673+0000 7f655b7fe640 1 -- 172.21.15.72:0/1700689160 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f655c102660 msgr2=0x7f655c19f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:41.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.673+0000 7f655b7fe640 1 --2- 172.21.15.72:0/1700689160 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f655c102660 0x7f655c19f280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:41.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.673+0000 7f655b7fe640 1 -- 172.21.15.72:0/1700689160 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f655c19fb40 con 0x7f655c100ab0 2024-03-22T01:34:41.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.673+0000 7f655affd640 1 --2- 172.21.15.72:0/1700689160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f655c101ca0 0x7f655c19cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:34:41.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.673+0000 7f655b7fe640 1 --2- 172.21.15.72:0/1700689160 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f655c100ab0 0x7f655c06ae00 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f654c000c00 tx=0x7f654c02c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:41.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.673+0000 7f6558ff9640 1 -- 172.21.15.72:0/1700689160 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f654c041070 con 0x7f655c100ab0 2024-03-22T01:34:41.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.673+0000 7f6558ff9640 1 -- 172.21.15.72:0/1700689160 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f654c004450 con 0x7f655c100ab0 2024-03-22T01:34:41.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.673+0000 7f6558ff9640 1 -- 172.21.15.72:0/1700689160 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f654c034440 con 0x7f655c100ab0 2024-03-22T01:34:41.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.673+0000 7f6561a8f640 1 -- 172.21.15.72:0/1700689160 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f655c19fdd0 con 0x7f655c100ab0 2024-03-22T01:34:41.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.673+0000 7f6561a8f640 1 -- 172.21.15.72:0/1700689160 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f655c1a0200 con 0x7f655c100ab0 2024-03-22T01:34:41.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.674+0000 7f6561a8f640 1 -- 172.21.15.72:0/1700689160 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f655c107e40 con 0x7f655c100ab0 2024-03-22T01:34:41.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.675+0000 7f6558ff9640 1 -- 172.21.15.72:0/1700689160 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f654c044070 con 0x7f655c100ab0 2024-03-22T01:34:41.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.676+0000 7f6558ff9640 1 --2- 172.21.15.72:0/1700689160 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f65440a2f20 0x7f65440a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:41.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.676+0000 7f655affd640 1 --2- 172.21.15.72:0/1700689160 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f65440a2f20 0x7f65440a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:41.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.676+0000 7f6558ff9640 1 -- 172.21.15.72:0/1700689160 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f654c0f1180 con 0x7f655c100ab0 2024-03-22T01:34:41.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.676+0000 7f655affd640 1 --2- 172.21.15.72:0/1700689160 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f65440a2f20 0x7f65440a53e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f655c06ab70 tx=0x7f6550005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:41.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.680+0000 7f6558ff9640 1 -- 172.21.15.72:0/1700689160 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f654c0bde00 con 0x7f655c100ab0 2024-03-22T01:34:41.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:41 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/2925059628' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-22T01:34:41.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:41 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:34:41.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:41 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/2925059628' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-22T01:34:41.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:41 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:34:41.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.882+0000 7f6561a8f640 1 -- 172.21.15.72:0/1700689160 --> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f655c062060 con 0x7f65440a2f20 2024-03-22T01:34:41.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.910+0000 7f6558ff9640 1 -- 172.21.15.72:0/1700689160 <== mgr.14152 v2:172.21.15.72:6800/401403200 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f655c062060 con 0x7f65440a2f20 2024-03-22T01:34:41.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.912+0000 7f6561a8f640 1 -- 172.21.15.72:0/1700689160 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f65440a2f20 msgr2=0x7f65440a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:41.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.912+0000 7f6561a8f640 1 --2- 172.21.15.72:0/1700689160 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f65440a2f20 0x7f65440a53e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f655c06ab70 tx=0x7f6550005f90 comp rx=0 tx=0).stop 2024-03-22T01:34:41.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.912+0000 7f6561a8f640 1 -- 172.21.15.72:0/1700689160 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f655c100ab0 msgr2=0x7f655c06ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:41.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.912+0000 7f6561a8f640 1 --2- 172.21.15.72:0/1700689160 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f655c100ab0 0x7f655c06ae00 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f654c000c00 tx=0x7f654c02c640 comp rx=0 tx=0).stop 2024-03-22T01:34:41.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.912+0000 7f6561a8f640 1 -- 172.21.15.72:0/1700689160 shutdown_connections 2024-03-22T01:34:41.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.912+0000 7f6561a8f640 1 --2- 172.21.15.72:0/1700689160 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f65440a2f20 0x7f65440a53e0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:41.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.912+0000 7f6561a8f640 1 --2- 172.21.15.72:0/1700689160 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f655c102660 0x7f655c19f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:41.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.912+0000 7f6561a8f640 1 --2- 172.21.15.72:0/1700689160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f655c101ca0 0x7f655c19cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:41.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.912+0000 7f6561a8f640 1 --2- 172.21.15.72:0/1700689160 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f655c100ab0 0x7f655c06ae00 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:41.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.912+0000 7f6561a8f640 1 -- 172.21.15.72:0/1700689160 >> 172.21.15.72:0/1700689160 conn(0x7f655c0fc020 msgr2=0x7f655c105010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:41.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.912+0000 7f6561a8f640 1 -- 172.21.15.72:0/1700689160 shutdown_connections 2024-03-22T01:34:41.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:41.913+0000 7f6561a8f640 1 -- 172.21.15.72:0/1700689160 wait complete. 2024-03-22T01:34:41.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:41 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/2925059628' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-22T01:34:41.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:41 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:34:41.924 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-03-22T01:34:42.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.031+0000 7f14bd319640 1 -- 172.21.15.72:0/2030113381 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14b8102630 msgr2=0x7f14b8104a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:42.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.031+0000 7f14bd319640 1 --2- 172.21.15.72:0/2030113381 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14b8102630 0x7f14b8104a20 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f14a8009ca0 tx=0x7f14a802c0f0 comp rx=0 tx=0).stop 2024-03-22T01:34:42.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.031+0000 7f14bd319640 1 -- 172.21.15.72:0/2030113381 shutdown_connections 2024-03-22T01:34:42.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.031+0000 7f14bd319640 1 --2- 172.21.15.72:0/2030113381 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14b8104f60 0x7f14b8107350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:42.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.031+0000 7f14bd319640 1 --2- 172.21.15.72:0/2030113381 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14b8107890 0x7f14b8109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:42.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.031+0000 7f14bd319640 1 --2- 172.21.15.72:0/2030113381 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14b8102630 0x7f14b8104a20 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:42.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.031+0000 7f14bd319640 1 -- 172.21.15.72:0/2030113381 >> 172.21.15.72:0/2030113381 conn(0x7f14b80fc020 msgr2=0x7f14b80fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:42.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.031+0000 7f14bd319640 1 -- 172.21.15.72:0/2030113381 shutdown_connections 2024-03-22T01:34:42.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.031+0000 7f14bd319640 1 -- 172.21.15.72:0/2030113381 wait complete. 2024-03-22T01:34:42.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.032+0000 7f14bd319640 1 Processor -- start 2024-03-22T01:34:42.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.032+0000 7f14bd319640 1 -- start start 2024-03-22T01:34:42.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.032+0000 7f14bd319640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14b8102630 0x7f14b8197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:42.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.032+0000 7f14bd319640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14b8104f60 0x7f14b81984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:42.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.032+0000 7f14bd319640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14b8107890 0x7f14b819f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:42.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.032+0000 7f14bd319640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f14b810d8a0 con 0x7f14b8107890 2024-03-22T01:34:42.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.032+0000 7f14bd319640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f14b810d720 con 0x7f14b8104f60 2024-03-22T01:34:42.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.032+0000 7f14bd319640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f14b810da20 con 0x7f14b8102630 2024-03-22T01:34:42.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.032+0000 7f14b77fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14b8107890 0x7f14b819f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:42.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.032+0000 7f14b77fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14b8107890 0x7f14b819f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37410/0 (socket says 172.21.15.72:37410) 2024-03-22T01:34:42.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.032+0000 7f14b77fe640 1 -- 172.21.15.72:0/365302483 learned_addr learned my addr 172.21.15.72:0/365302483 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:34:42.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.032+0000 7f14b67fc640 1 --2- 172.21.15.72:0/365302483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14b8104f60 0x7f14b81984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:42.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.033+0000 7f14b6ffd640 1 --2- 172.21.15.72:0/365302483 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14b8102630 0x7f14b8197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:42.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.033+0000 7f14b77fe640 1 -- 172.21.15.72:0/365302483 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14b8102630 msgr2=0x7f14b8197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:42.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.033+0000 7f14b77fe640 1 --2- 172.21.15.72:0/365302483 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14b8102630 0x7f14b8197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:42.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.033+0000 7f14b77fe640 1 -- 172.21.15.72:0/365302483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14b8104f60 msgr2=0x7f14b81984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:42.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.033+0000 7f14b77fe640 1 --2- 172.21.15.72:0/365302483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14b8104f60 0x7f14b81984c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:42.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.033+0000 7f14b77fe640 1 -- 172.21.15.72:0/365302483 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14b819fba0 con 0x7f14b8107890 2024-03-22T01:34:42.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.033+0000 7f14b6ffd640 1 --2- 172.21.15.72:0/365302483 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14b8102630 0x7f14b8197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:34:42.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.033+0000 7f14b67fc640 1 --2- 172.21.15.72:0/365302483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14b8104f60 0x7f14b81984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:34:42.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.033+0000 7f14b77fe640 1 --2- 172.21.15.72:0/365302483 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14b8107890 0x7f14b819f530 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f14b0009fd0 tx=0x7f14b000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:42.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.033+0000 7f149ffff640 1 -- 172.21.15.72:0/365302483 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b0011830 con 0x7f14b8107890 2024-03-22T01:34:42.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.033+0000 7f149ffff640 1 -- 172.21.15.72:0/365302483 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14b0016070 con 0x7f14b8107890 2024-03-22T01:34:42.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.033+0000 7f14bd319640 1 -- 172.21.15.72:0/365302483 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14b819fe90 con 0x7f14b8107890 2024-03-22T01:34:42.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.033+0000 7f149ffff640 1 -- 172.21.15.72:0/365302483 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b000f2e0 con 0x7f14b8107890 2024-03-22T01:34:42.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.033+0000 7f14bd319640 1 -- 172.21.15.72:0/365302483 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14b81a0350 con 0x7f14b8107890 2024-03-22T01:34:42.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.034+0000 7f14bd319640 1 -- 172.21.15.72:0/365302483 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14b80fdb10 con 0x7f14b8107890 2024-03-22T01:34:42.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.035+0000 7f149ffff640 1 -- 172.21.15.72:0/365302483 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f14b00119d0 con 0x7f14b8107890 2024-03-22T01:34:42.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.036+0000 7f149ffff640 1 --2- 172.21.15.72:0/365302483 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f14a00a2f20 0x7f14a00a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:42.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.036+0000 7f14b6ffd640 1 --2- 172.21.15.72:0/365302483 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f14a00a2f20 0x7f14a00a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:42.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.036+0000 7f149ffff640 1 -- 172.21.15.72:0/365302483 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f14b00cec10 con 0x7f14b8107890 2024-03-22T01:34:42.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.036+0000 7f14b6ffd640 1 --2- 172.21.15.72:0/365302483 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f14a00a2f20 0x7f14a00a53e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f14a8006010 tx=0x7f14a80325b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:42.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.040+0000 7f149ffff640 1 -- 172.21.15.72:0/365302483 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f14b009b800 con 0x7f14b8107890 2024-03-22T01:34:42.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.243+0000 7f14bd319640 1 -- 172.21.15.72:0/365302483 --> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f14b8061fd0 con 0x7f14a00a2f20 2024-03-22T01:34:42.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.249+0000 7f149ffff640 1 -- 172.21.15.72:0/365302483 <== mgr.14152 v2:172.21.15.72:6800/401403200 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f14b8061fd0 con 0x7f14a00a2f20 2024-03-22T01:34:42.248 INFO:teuthology.orchestra.run.smithi072.stdout:/volumes/qa/sv_1/63be0226-262f-43ca-bc55-6e0603eb71d2 2024-03-22T01:34:42.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.251+0000 7f14bd319640 1 -- 172.21.15.72:0/365302483 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f14a00a2f20 msgr2=0x7f14a00a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:42.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.251+0000 7f14bd319640 1 --2- 172.21.15.72:0/365302483 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f14a00a2f20 0x7f14a00a53e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f14a8006010 tx=0x7f14a80325b0 comp rx=0 tx=0).stop 2024-03-22T01:34:42.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.251+0000 7f14bd319640 1 -- 172.21.15.72:0/365302483 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14b8107890 msgr2=0x7f14b819f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:42.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.251+0000 7f14bd319640 1 --2- 172.21.15.72:0/365302483 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14b8107890 0x7f14b819f530 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f14b0009fd0 tx=0x7f14b000b8e0 comp rx=0 tx=0).stop 2024-03-22T01:34:42.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.251+0000 7f14bd319640 1 -- 172.21.15.72:0/365302483 shutdown_connections 2024-03-22T01:34:42.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.251+0000 7f14bd319640 1 --2- 172.21.15.72:0/365302483 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f14a00a2f20 0x7f14a00a53e0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:42.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.251+0000 7f14bd319640 1 --2- 172.21.15.72:0/365302483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14b8104f60 0x7f14b81984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:42.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.251+0000 7f14bd319640 1 --2- 172.21.15.72:0/365302483 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14b8107890 0x7f14b819f530 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:42.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.251+0000 7f14bd319640 1 --2- 172.21.15.72:0/365302483 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14b8102630 0x7f14b8197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:42.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.251+0000 7f14bd319640 1 -- 172.21.15.72:0/365302483 >> 172.21.15.72:0/365302483 conn(0x7f14b80fc020 msgr2=0x7f14b8102f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:42.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.251+0000 7f14bd319640 1 -- 172.21.15.72:0/365302483 shutdown_connections 2024-03-22T01:34:42.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.251+0000 7f14bd319640 1 -- 172.21.15.72:0/365302483 wait complete. 2024-03-22T01:34:42.261 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T01:34:42.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.363+0000 7f5fd2cdf640 1 -- 172.21.15.72:0/809719628 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fcc101c20 msgr2=0x7f5fcc102040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:42.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.363+0000 7f5fd2cdf640 1 --2- 172.21.15.72:0/809719628 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fcc101c20 0x7f5fcc102040 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5fc0009860 tx=0x7f5fc002bb70 comp rx=0 tx=0).stop 2024-03-22T01:34:42.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.363+0000 7f5fd2cdf640 1 -- 172.21.15.72:0/809719628 shutdown_connections 2024-03-22T01:34:42.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.363+0000 7f5fd2cdf640 1 --2- 172.21.15.72:0/809719628 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5fcc1037d0 0x7f5fcc10a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:42.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.363+0000 7f5fd2cdf640 1 --2- 172.21.15.72:0/809719628 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fcc102e10 0x7f5fcc103290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:42.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.363+0000 7f5fd2cdf640 1 --2- 172.21.15.72:0/809719628 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fcc101c20 0x7f5fcc102040 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:42.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.363+0000 7f5fd2cdf640 1 -- 172.21.15.72:0/809719628 >> 172.21.15.72:0/809719628 conn(0x7f5fcc0fd1b0 msgr2=0x7f5fcc0ff5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:42.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.363+0000 7f5fd2cdf640 1 -- 172.21.15.72:0/809719628 shutdown_connections 2024-03-22T01:34:42.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.363+0000 7f5fd2cdf640 1 -- 172.21.15.72:0/809719628 wait complete. 2024-03-22T01:34:42.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.364+0000 7f5fd2cdf640 1 Processor -- start 2024-03-22T01:34:42.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.364+0000 7f5fd2cdf640 1 -- start start 2024-03-22T01:34:42.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.364+0000 7f5fd2cdf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fcc101c20 0x7f5fcc197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:42.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.364+0000 7f5fd2cdf640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fcc102e10 0x7f5fcc198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:42.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.364+0000 7f5fd2cdf640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5fcc1037d0 0x7f5fcc19f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:42.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.364+0000 7f5fd2cdf640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f5fcc10d820 con 0x7f5fcc1037d0 2024-03-22T01:34:42.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.364+0000 7f5fd2cdf640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5fcc10d6a0 con 0x7f5fcc102e10 2024-03-22T01:34:42.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.364+0000 7f5fd2cdf640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5fcc10d9a0 con 0x7f5fcc101c20 2024-03-22T01:34:42.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.364+0000 7f5fd127c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5fcc1037d0 0x7f5fcc19f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:42.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.364+0000 7f5fd127c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5fcc1037d0 0x7f5fcc19f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37424/0 (socket says 172.21.15.72:37424) 2024-03-22T01:34:42.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.364+0000 7f5fd127c640 1 -- 172.21.15.72:0/1244924754 learned_addr learned my addr 172.21.15.72:0/1244924754 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:34:42.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.364+0000 7f5fcbfff640 1 --2- 172.21.15.72:0/1244924754 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fcc102e10 0x7f5fcc198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:42.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.364+0000 7f5fd0a7b640 1 --2- 172.21.15.72:0/1244924754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fcc101c20 0x7f5fcc197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:42.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.364+0000 7f5fd127c640 1 -- 172.21.15.72:0/1244924754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fcc101c20 msgr2=0x7f5fcc197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:42.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.364+0000 7f5fd127c640 1 --2- 172.21.15.72:0/1244924754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fcc101c20 0x7f5fcc197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:42.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.364+0000 7f5fd127c640 1 -- 172.21.15.72:0/1244924754 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fcc102e10 msgr2=0x7f5fcc198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:42.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.364+0000 7f5fd127c640 1 --2- 172.21.15.72:0/1244924754 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fcc102e10 0x7f5fcc198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:42.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.364+0000 7f5fd127c640 1 -- 172.21.15.72:0/1244924754 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fcc19fb00 con 0x7f5fcc1037d0 2024-03-22T01:34:42.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.364+0000 7f5fcbfff640 1 --2- 172.21.15.72:0/1244924754 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fcc102e10 0x7f5fcc198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:34:42.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.364+0000 7f5fd127c640 1 --2- 172.21.15.72:0/1244924754 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5fcc1037d0 0x7f5fcc19f490 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f5fc400efc0 tx=0x7f5fc400c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:42.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.365+0000 7f5fc9ffb640 1 -- 172.21.15.72:0/1244924754 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fc4003970 con 0x7f5fcc1037d0 2024-03-22T01:34:42.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.365+0000 7f5fd2cdf640 1 -- 172.21.15.72:0/1244924754 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fcc19fdf0 con 0x7f5fcc1037d0 2024-03-22T01:34:42.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.365+0000 7f5fc9ffb640 1 -- 172.21.15.72:0/1244924754 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5fc4004020 con 0x7f5fcc1037d0 2024-03-22T01:34:42.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.365+0000 7f5fc9ffb640 1 -- 172.21.15.72:0/1244924754 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fc400b9f0 con 0x7f5fcc1037d0 2024-03-22T01:34:42.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.365+0000 7f5fd2cdf640 1 -- 172.21.15.72:0/1244924754 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fcc1a0260 con 0x7f5fcc1037d0 2024-03-22T01:34:42.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.366+0000 7f5fd2cdf640 1 -- 172.21.15.72:0/1244924754 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fcc109010 con 0x7f5fcc1037d0 2024-03-22T01:34:42.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.367+0000 7f5fc9ffb640 1 -- 172.21.15.72:0/1244924754 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f5fc401d020 con 0x7f5fcc1037d0 2024-03-22T01:34:42.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.367+0000 7f5fc9ffb640 1 --2- 172.21.15.72:0/1244924754 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f5fbc0a2f20 0x7f5fbc0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:42.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.367+0000 7f5fd0a7b640 1 --2- 172.21.15.72:0/1244924754 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f5fbc0a2f20 0x7f5fbc0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:42.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.367+0000 7f5fc9ffb640 1 -- 172.21.15.72:0/1244924754 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f5fc40cf570 con 0x7f5fcc1037d0 2024-03-22T01:34:42.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.368+0000 7f5fd0a7b640 1 --2- 172.21.15.72:0/1244924754 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f5fbc0a2f20 0x7f5fbc0a53e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5fc0009830 tx=0x7f5fc0032480 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:42.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.372+0000 7f5fc9ffb640 1 -- 172.21.15.72:0/1244924754 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5fc409c160 con 0x7f5fcc1037d0 2024-03-22T01:34:42.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.526+0000 7f5fc9ffb640 1 -- 172.21.15.72:0/1244924754 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7f5fc40980d0 con 0x7f5fcc1037d0 2024-03-22T01:34:42.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.588+0000 7f5fd2cdf640 1 -- 172.21.15.72:0/1244924754 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5fcc062790 con 0x7f5fcc1037d0 2024-03-22T01:34:42.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.589+0000 7f5fc9ffb640 1 -- 172.21.15.72:0/1244924754 <== mon.0 v2:172.21.15.72:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11785 (secure 0 0 0) 0x7f5fc40870a0 con 0x7f5fcc1037d0 2024-03-22T01:34:42.588 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:34:42.589 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6835/3077216308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3077216308},{"type":"v1","addr":"172.21.15.72:6835","nonce":3077216308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2460254143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":2460254143},{"type":"v1","addr":"172.21.15.72:6837","nonce":2460254143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6839/2026432819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6838","nonce":2026432819},{"type":"v1","addr":"172.21.15.72:6839","nonce":2026432819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6841/400094545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6840","nonce":400094545},{"type":"v1","addr":"172.21.15.72:6841","nonce":400094545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6833/1578701171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":1578701171},{"type":"v1","addr":"172.21.15.89:6833","nonce":1578701171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/2703047213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2703047213},{"type":"v1","addr":"172.21.15.98:6833","nonce":2703047213}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2589468787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2589468787},{"type":"v1","addr":"172.21.15.89:6835","nonce":2589468787}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/92657636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":92657636},{"type":"v1","addr":"172.21.15.98:6835","nonce":92657636}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1662941782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1662941782},{"type":"v1","addr":"172.21.15.89:6837","nonce":1662941782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24490,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/781860371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":781860371},{"type":"v1","addr":"172.21.15.98:6837","nonce":781860371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-22T01:34:14.815677+0000","modified":"2024-03-22T01:34:39.842959+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24524},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3192173650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3192173650},{"type":"v1","addr":"172.21.15.98:6839","nonce":3192173650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6839/149200280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":149200280},{"type":"v1","addr":"172.21.15.89:6839","nonce":149200280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24524]},"id":1}]} 2024-03-22T01:34:42.589 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 21 2024-03-22T01:34:42.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.591+0000 7f5fd2cdf640 1 -- 172.21.15.72:0/1244924754 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f5fbc0a2f20 msgr2=0x7f5fbc0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:42.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.591+0000 7f5fd2cdf640 1 --2- 172.21.15.72:0/1244924754 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f5fbc0a2f20 0x7f5fbc0a53e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5fc0009830 tx=0x7f5fc0032480 comp rx=0 tx=0).stop 2024-03-22T01:34:42.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.591+0000 7f5fd2cdf640 1 -- 172.21.15.72:0/1244924754 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5fcc1037d0 msgr2=0x7f5fcc19f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:42.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.591+0000 7f5fd2cdf640 1 --2- 172.21.15.72:0/1244924754 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5fcc1037d0 0x7f5fcc19f490 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f5fc400efc0 tx=0x7f5fc400c800 comp rx=0 tx=0).stop 2024-03-22T01:34:42.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.591+0000 7f5fd2cdf640 1 -- 172.21.15.72:0/1244924754 shutdown_connections 2024-03-22T01:34:42.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.591+0000 7f5fd2cdf640 1 --2- 172.21.15.72:0/1244924754 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f5fbc0a2f20 0x7f5fbc0a53e0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:42.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.591+0000 7f5fd2cdf640 1 --2- 172.21.15.72:0/1244924754 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fcc102e10 0x7f5fcc198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:42.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.591+0000 7f5fd2cdf640 1 --2- 172.21.15.72:0/1244924754 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5fcc1037d0 0x7f5fcc19f490 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:42.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.591+0000 7f5fd2cdf640 1 --2- 172.21.15.72:0/1244924754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fcc101c20 0x7f5fcc197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:42.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.591+0000 7f5fd2cdf640 1 -- 172.21.15.72:0/1244924754 >> 172.21.15.72:0/1244924754 conn(0x7f5fcc0fd1b0 msgr2=0x7f5fcc1061e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:42.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.591+0000 7f5fd2cdf640 1 -- 172.21.15.72:0/1244924754 shutdown_connections 2024-03-22T01:34:42.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.591+0000 7f5fd2cdf640 1 -- 172.21.15.72:0/1244924754 wait complete. 2024-03-22T01:34:42.602 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T01:34:42.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.704+0000 7fad0233e640 1 -- 172.21.15.72:0/3953763519 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7facfc105060 msgr2=0x7facfc107450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:42.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.704+0000 7fad0233e640 1 --2- 172.21.15.72:0/3953763519 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7facfc105060 0x7facfc107450 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7facf0009900 tx=0x7facf002bc30 comp rx=0 tx=0).stop 2024-03-22T01:34:42.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.704+0000 7fad0233e640 1 -- 172.21.15.72:0/3953763519 shutdown_connections 2024-03-22T01:34:42.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.704+0000 7fad0233e640 1 --2- 172.21.15.72:0/3953763519 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facfc107990 0x7facfc109d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:42.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.704+0000 7fad0233e640 1 --2- 172.21.15.72:0/3953763519 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7facfc105060 0x7facfc107450 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:42.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.704+0000 7fad0233e640 1 --2- 172.21.15.72:0/3953763519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facfc102730 0x7facfc104b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:42.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.704+0000 7fad0233e640 1 -- 172.21.15.72:0/3953763519 >> 172.21.15.72:0/3953763519 conn(0x7facfc0fc0c0 msgr2=0x7facfc0fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:42.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.704+0000 7fad0233e640 1 -- 172.21.15.72:0/3953763519 shutdown_connections 2024-03-22T01:34:42.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.704+0000 7fad0233e640 1 -- 172.21.15.72:0/3953763519 wait complete. 2024-03-22T01:34:42.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.705+0000 7fad0233e640 1 Processor -- start 2024-03-22T01:34:42.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.705+0000 7fad0233e640 1 -- start start 2024-03-22T01:34:42.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.705+0000 7fad0233e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7facfc102730 0x7facfc197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:42.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.705+0000 7fad0233e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facfc105060 0x7facfc198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:42.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.705+0000 7fad0233e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facfc107990 0x7facfc19f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:42.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.705+0000 7fad0233e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7facfc10d810 con 0x7facfc102730 2024-03-22T01:34:42.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.705+0000 7fad0233e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7facfc10d690 con 0x7facfc107990 2024-03-22T01:34:42.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.705+0000 7fad0233e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7facfc10d990 con 0x7facfc105060 2024-03-22T01:34:42.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.705+0000 7facfbfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7facfc102730 0x7facfc197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:42.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.705+0000 7facfbfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7facfc102730 0x7facfc197e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37456/0 (socket says 172.21.15.72:37456) 2024-03-22T01:34:42.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.705+0000 7facfbfff640 1 -- 172.21.15.72:0/693387139 learned_addr learned my addr 172.21.15.72:0/693387139 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:34:42.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.705+0000 7fad008db640 1 --2- 172.21.15.72:0/693387139 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facfc107990 0x7facfc19f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:42.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.705+0000 7facfb7fe640 1 --2- 172.21.15.72:0/693387139 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facfc105060 0x7facfc198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:42.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.705+0000 7facfbfff640 1 -- 172.21.15.72:0/693387139 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facfc105060 msgr2=0x7facfc198390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:42.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.705+0000 7facfbfff640 1 --2- 172.21.15.72:0/693387139 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facfc105060 0x7facfc198390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:42.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.706+0000 7facfbfff640 1 -- 172.21.15.72:0/693387139 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facfc107990 msgr2=0x7facfc19f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:42.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.706+0000 7facfbfff640 1 --2- 172.21.15.72:0/693387139 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facfc107990 0x7facfc19f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:42.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.706+0000 7facfbfff640 1 -- 172.21.15.72:0/693387139 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facfc19fa70 con 0x7facfc102730 2024-03-22T01:34:42.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.706+0000 7facfb7fe640 1 --2- 172.21.15.72:0/693387139 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facfc105060 0x7facfc198390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:34:42.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.706+0000 7facfbfff640 1 --2- 172.21.15.72:0/693387139 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7facfc102730 0x7facfc197e50 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7facec00c920 tx=0x7facec00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:42.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.706+0000 7facf97fa640 1 -- 172.21.15.72:0/693387139 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facec007c70 con 0x7facfc102730 2024-03-22T01:34:42.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.706+0000 7facf97fa640 1 -- 172.21.15.72:0/693387139 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7facec007e10 con 0x7facfc102730 2024-03-22T01:34:42.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.706+0000 7fad0233e640 1 -- 172.21.15.72:0/693387139 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facfc19fd60 con 0x7facfc102730 2024-03-22T01:34:42.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.706+0000 7facf97fa640 1 -- 172.21.15.72:0/693387139 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facec018890 con 0x7facfc102730 2024-03-22T01:34:42.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.706+0000 7fad0233e640 1 -- 172.21.15.72:0/693387139 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facfc1a01d0 con 0x7facfc102730 2024-03-22T01:34:42.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.707+0000 7fad0233e640 1 -- 172.21.15.72:0/693387139 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facfc0fdbb0 con 0x7facfc102730 2024-03-22T01:34:42.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.708+0000 7facf97fa640 1 -- 172.21.15.72:0/693387139 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7facec018a50 con 0x7facfc102730 2024-03-22T01:34:42.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.708+0000 7facf97fa640 1 --2- 172.21.15.72:0/693387139 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7face40a2fb0 0x7face40a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:42.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.709+0000 7facfb7fe640 1 --2- 172.21.15.72:0/693387139 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7face40a2fb0 0x7face40a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:42.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.709+0000 7facf97fa640 1 -- 172.21.15.72:0/693387139 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7facec0cfbb0 con 0x7facfc102730 2024-03-22T01:34:42.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.709+0000 7facfb7fe640 1 --2- 172.21.15.72:0/693387139 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7face40a2fb0 0x7face40a5470 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7facf002c7b0 tx=0x7facf0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:42.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.712+0000 7facf97fa640 1 -- 172.21.15.72:0/693387139 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7facec014030 con 0x7facfc102730 2024-03-22T01:34:42.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:42 smithi089 ceph-mon[35770]: from='client.14877 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:34:42.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:42 smithi089 ceph-mon[35770]: from='client.14889 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:34:42.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:42 smithi089 ceph-mon[35770]: pgmap v304: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 597 B/s rd, 426 B/s wr, 0 op/s 2024-03-22T01:34:42.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:42 smithi089 ceph-mon[35770]: from='client.14895 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:34:42.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:42 smithi072 ceph-mon[32414]: from='client.14877 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:34:42.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:42 smithi072 ceph-mon[32414]: from='client.14889 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:34:42.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:42 smithi072 ceph-mon[32414]: pgmap v304: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 597 B/s rd, 426 B/s wr, 0 op/s 2024-03-22T01:34:42.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:42 smithi072 ceph-mon[32414]: from='client.14895 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:34:42.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.889+0000 7fad0233e640 1 -- 172.21.15.72:0/693387139 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7facfc062890 con 0x7facfc102730 2024-03-22T01:34:42.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.891+0000 7facf97fa640 1 -- 172.21.15.72:0/693387139 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24045 (secure 0 0 0) 0x7facec09c6d0 con 0x7facfc102730 2024-03-22T01:34:42.890 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:34:42.890 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":91,"fsid":"022a99bc-e7eb-11ee-95cd-87774f69a715","created":"2024-03-22T01:26:23.893672+0000","modified":"2024-03-22T01:34:19.866367+0000","last_up_change":"2024-03-22T01:33:25.223684+0000","last_in_change":"2024-03-22T01:33:05.010331+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T01:29:37.438503+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-22T01:34:12.610422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-22T01:34:13.761686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-22T01:34:17.507039+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"16c15bb0-db92-4191-9063-38b4890e81e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6803","nonce":661906791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6805","nonce":661906791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6809","nonce":661906791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6807","nonce":661906791}]},"public_addr":"172.21.15.72:6803/661906791","cluster_addr":"172.21.15.72:6805/661906791","heartbeat_back_addr":"172.21.15.72:6809/661906791","heartbeat_front_addr":"172.21.15.72:6807/661906791","state":["exists","up"]},{"osd":1,"uuid":"981290ba-97dc-496b-aa22-d51704184b10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6801","nonce":2870590550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6803","nonce":2870590550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6807","nonce":2870590550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6805","nonce":2870590550}]},"public_addr":"172.21.15.89:6801/2870590550","cluster_addr":"172.21.15.89:6803/2870590550","heartbeat_back_addr":"172.21.15.89:6807/2870590550","heartbeat_front_addr":"172.21.15.89:6805/2870590550","state":["exists","up"]},{"osd":2,"uuid":"f07ed658-f545-46a1-8edd-8b1d5d49885c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6801","nonce":3865778118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6803","nonce":3865778118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6807","nonce":3865778118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6805","nonce":3865778118}]},"public_addr":"172.21.15.98:6801/3865778118","cluster_addr":"172.21.15.98:6803/3865778118","heartbeat_back_addr":"172.21.15.98:6807/3865778118","heartbeat_front_addr":"172.21.15.98:6805/3865778118","state":["exists","up"]},{"osd":3,"uuid":"ccc4bc40-2c48-46d2-8548-0c05b44f9f1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6811","nonce":1865951395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6813","nonce":1865951395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6817","nonce":1865951395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6815","nonce":1865951395}]},"public_addr":"172.21.15.72:6811/1865951395","cluster_addr":"172.21.15.72:6813/1865951395","heartbeat_back_addr":"172.21.15.72:6817/1865951395","heartbeat_front_addr":"172.21.15.72:6815/1865951395","state":["exists","up"]},{"osd":4,"uuid":"1812439f-810a-4ed2-9395-c00c9a085a13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6809","nonce":2683760795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6811","nonce":2683760795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6815","nonce":2683760795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6813","nonce":2683760795}]},"public_addr":"172.21.15.89:6809/2683760795","cluster_addr":"172.21.15.89:6811/2683760795","heartbeat_back_addr":"172.21.15.89:6815/2683760795","heartbeat_front_addr":"172.21.15.89:6813/2683760795","state":["exists","up"]},{"osd":5,"uuid":"e78954d8-f4e3-4d58-8536-d2d59c843386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6809","nonce":2635838572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6811","nonce":2635838572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6815","nonce":2635838572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6813","nonce":2635838572}]},"public_addr":"172.21.15.98:6809/2635838572","cluster_addr":"172.21.15.98:6811/2635838572","heartbeat_back_addr":"172.21.15.98:6815/2635838572","heartbeat_front_addr":"172.21.15.98:6813/2635838572","state":["exists","up"]},{"osd":6,"uuid":"dc6c8be2-b64c-4959-8015-7b0ea6d25599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6819","nonce":3158982661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6821","nonce":3158982661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6825","nonce":3158982661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6823","nonce":3158982661}]},"public_addr":"172.21.15.72:6819/3158982661","cluster_addr":"172.21.15.72:6821/3158982661","heartbeat_back_addr":"172.21.15.72:6825/3158982661","heartbeat_front_addr":"172.21.15.72:6823/3158982661","state":["exists","up"]},{"osd":7,"uuid":"7a73469f-cf3a-4306-ae93-ea46e21ebbbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6817","nonce":3075913234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6819","nonce":3075913234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6823","nonce":3075913234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6821","nonce":3075913234}]},"public_addr":"172.21.15.89:6817/3075913234","cluster_addr":"172.21.15.89:6819/3075913234","heartbeat_back_addr":"172.21.15.89:6823/3075913234","heartbeat_front_addr":"172.21.15.89:6821/3075913234","state":["exists","up"]},{"osd":8,"uuid":"c4b13bee-ea4d-4344-95fb-2622ecbb8196","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6817","nonce":1813255577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6819","nonce":1813255577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6823","nonce":1813255577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6821","nonce":1813255577}]},"public_addr":"172.21.15.98:6817/1813255577","cluster_addr":"172.21.15.98:6819/1813255577","heartbeat_back_addr":"172.21.15.98:6823/1813255577","heartbeat_front_addr":"172.21.15.98:6821/1813255577","state":["exists","up"]},{"osd":9,"uuid":"eefb0a44-4a36-4e55-9d04-0501bf718db6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6827","nonce":3377749206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6829","nonce":3377749206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6832","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6833","nonce":3377749206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6831","nonce":3377749206}]},"public_addr":"172.21.15.72:6827/3377749206","cluster_addr":"172.21.15.72:6829/3377749206","heartbeat_back_addr":"172.21.15.72:6833/3377749206","heartbeat_front_addr":"172.21.15.72:6831/3377749206","state":["exists","up"]},{"osd":10,"uuid":"8741d25f-40dd-40d7-b040-5cc5e411eb98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6825","nonce":885058786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6827","nonce":885058786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6831","nonce":885058786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6829","nonce":885058786}]},"public_addr":"172.21.15.89:6825/885058786","cluster_addr":"172.21.15.89:6827/885058786","heartbeat_back_addr":"172.21.15.89:6831/885058786","heartbeat_front_addr":"172.21.15.89:6829/885058786","state":["exists","up"]},{"osd":11,"uuid":"8d02e957-eda4-4184-b5c6-5eb3b8b38060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6825","nonce":1517095838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6827","nonce":1517095838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6831","nonce":1517095838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6829","nonce":1517095838}]},"public_addr":"172.21.15.98:6825/1517095838","cluster_addr":"172.21.15.98:6827/1517095838","heartbeat_back_addr":"172.21.15.98:6831/1517095838","heartbeat_front_addr":"172.21.15.98:6829/1517095838","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:28:47.307652+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:11.037135+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:34.619188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:59.350738+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:30:23.787337+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:30:48.994154+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:31:13.690350+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:31:39.628116+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:04.723912+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:30.689741+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:57.229508+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:33:23.154537+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:0/121087196":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/2451857761":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/1228477995":"2024-03-23T01:27:15.339809+0000","172.21.15.72:6801/1513232913":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/3558958264":"2024-03-23T01:26:50.074871+0000","172.21.15.72:0/1510187198":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6800/3518012916":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6801/3518012916":"2024-03-23T01:26:50.074871+0000","172.21.15.72:0/1372801041":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6800/1513232913":"2024-03-23T01:27:15.339809+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T01:34:42.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.893+0000 7fad0233e640 1 -- 172.21.15.72:0/693387139 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7face40a2fb0 msgr2=0x7face40a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:42.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.893+0000 7fad0233e640 1 --2- 172.21.15.72:0/693387139 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7face40a2fb0 0x7face40a5470 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7facf002c7b0 tx=0x7facf0005fd0 comp rx=0 tx=0).stop 2024-03-22T01:34:42.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.893+0000 7fad0233e640 1 -- 172.21.15.72:0/693387139 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7facfc102730 msgr2=0x7facfc197e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:42.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.893+0000 7fad0233e640 1 --2- 172.21.15.72:0/693387139 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7facfc102730 0x7facfc197e50 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7facec00c920 tx=0x7facec00cde0 comp rx=0 tx=0).stop 2024-03-22T01:34:42.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.893+0000 7fad0233e640 1 -- 172.21.15.72:0/693387139 shutdown_connections 2024-03-22T01:34:42.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.893+0000 7fad0233e640 1 --2- 172.21.15.72:0/693387139 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7face40a2fb0 0x7face40a5470 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:42.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.893+0000 7fad0233e640 1 --2- 172.21.15.72:0/693387139 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facfc107990 0x7facfc19f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:42.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.893+0000 7fad0233e640 1 --2- 172.21.15.72:0/693387139 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facfc105060 0x7facfc198390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:42.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.893+0000 7fad0233e640 1 --2- 172.21.15.72:0/693387139 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7facfc102730 0x7facfc197e50 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:42.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.893+0000 7fad0233e640 1 -- 172.21.15.72:0/693387139 >> 172.21.15.72:0/693387139 conn(0x7facfc0fc0c0 msgr2=0x7facfc103190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:42.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.893+0000 7fad0233e640 1 -- 172.21.15.72:0/693387139 shutdown_connections 2024-03-22T01:34:42.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:42.893+0000 7fad0233e640 1 -- 172.21.15.72:0/693387139 wait complete. 2024-03-22T01:34:42.903 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-22T01:34:42.904 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-22T01:34:42.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:42 smithi098 ceph-mon[36468]: from='client.14877 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:34:42.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:42 smithi098 ceph-mon[36468]: from='client.14889 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:34:42.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:42 smithi098 ceph-mon[36468]: pgmap v304: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 597 B/s rd, 426 B/s wr, 0 op/s 2024-03-22T01:34:42.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:42 smithi098 ceph-mon[36468]: from='client.14895 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:34:43.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.007+0000 7fdc2558a640 1 -- 172.21.15.72:0/66318320 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc20101be0 msgr2=0x7fdc20102000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:43.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.007+0000 7fdc2558a640 1 --2- 172.21.15.72:0/66318320 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc20101be0 0x7fdc20102000 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fdc18009900 tx=0x7fdc1802bc30 comp rx=0 tx=0).stop 2024-03-22T01:34:43.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.007+0000 7fdc2558a640 1 -- 172.21.15.72:0/66318320 shutdown_connections 2024-03-22T01:34:43.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.007+0000 7fdc2558a640 1 --2- 172.21.15.72:0/66318320 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc20102dd0 0x7fdc20103250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:43.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.007+0000 7fdc2558a640 1 --2- 172.21.15.72:0/66318320 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc20103790 0x7fdc2010a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:43.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.007+0000 7fdc2558a640 1 --2- 172.21.15.72:0/66318320 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc20101be0 0x7fdc20102000 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:43.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.007+0000 7fdc2558a640 1 -- 172.21.15.72:0/66318320 >> 172.21.15.72:0/66318320 conn(0x7fdc200fd150 msgr2=0x7fdc200ff590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:43.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.007+0000 7fdc2558a640 1 -- 172.21.15.72:0/66318320 shutdown_connections 2024-03-22T01:34:43.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.007+0000 7fdc2558a640 1 -- 172.21.15.72:0/66318320 wait complete. 2024-03-22T01:34:43.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.008+0000 7fdc2558a640 1 Processor -- start 2024-03-22T01:34:43.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.008+0000 7fdc2558a640 1 -- start start 2024-03-22T01:34:43.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.008+0000 7fdc2558a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc20101be0 0x7fdc20197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:43.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.008+0000 7fdc2558a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc20102dd0 0x7fdc20198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:43.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.008+0000 7fdc2558a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc20103790 0x7fdc2019f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:43.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.008+0000 7fdc2558a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fdc2010d8b0 con 0x7fdc20101be0 2024-03-22T01:34:43.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.008+0000 7fdc2558a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fdc2010d730 con 0x7fdc20102dd0 2024-03-22T01:34:43.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.008+0000 7fdc2558a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdc2010da30 con 0x7fdc20103790 2024-03-22T01:34:43.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.008+0000 7fdc1effd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc20101be0 0x7fdc20197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:43.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.008+0000 7fdc1effd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc20101be0 0x7fdc20197f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37464/0 (socket says 172.21.15.72:37464) 2024-03-22T01:34:43.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.008+0000 7fdc1effd640 1 -- 172.21.15.72:0/2233002194 learned_addr learned my addr 172.21.15.72:0/2233002194 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:34:43.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.008+0000 7fdc1e7fc640 1 --2- 172.21.15.72:0/2233002194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc20102dd0 0x7fdc20198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:43.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.008+0000 7fdc1f7fe640 1 --2- 172.21.15.72:0/2233002194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc20103790 0x7fdc2019f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:43.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.008+0000 7fdc1effd640 1 -- 172.21.15.72:0/2233002194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc20103790 msgr2=0x7fdc2019f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:43.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.008+0000 7fdc1effd640 1 --2- 172.21.15.72:0/2233002194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc20103790 0x7fdc2019f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:43.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.008+0000 7fdc1effd640 1 -- 172.21.15.72:0/2233002194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc20102dd0 msgr2=0x7fdc20198460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:43.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.008+0000 7fdc1effd640 1 --2- 172.21.15.72:0/2233002194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc20102dd0 0x7fdc20198460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:43.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.008+0000 7fdc1effd640 1 -- 172.21.15.72:0/2233002194 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc2019fb40 con 0x7fdc20101be0 2024-03-22T01:34:43.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.008+0000 7fdc1f7fe640 1 --2- 172.21.15.72:0/2233002194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc20103790 0x7fdc2019f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T01:34:43.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.009+0000 7fdc1effd640 1 --2- 172.21.15.72:0/2233002194 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc20101be0 0x7fdc20197f20 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fdc18000c00 tx=0x7fdc18005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:43.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.009+0000 7fdc17fff640 1 -- 172.21.15.72:0/2233002194 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc18041070 con 0x7fdc20101be0 2024-03-22T01:34:43.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.009+0000 7fdc17fff640 1 -- 172.21.15.72:0/2233002194 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc180044d0 con 0x7fdc20101be0 2024-03-22T01:34:43.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.009+0000 7fdc2558a640 1 -- 172.21.15.72:0/2233002194 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc2019fdd0 con 0x7fdc20101be0 2024-03-22T01:34:43.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.009+0000 7fdc17fff640 1 -- 172.21.15.72:0/2233002194 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc18034440 con 0x7fdc20101be0 2024-03-22T01:34:43.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.009+0000 7fdc2558a640 1 -- 172.21.15.72:0/2233002194 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc201a0200 con 0x7fdc20101be0 2024-03-22T01:34:43.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.010+0000 7fdc2558a640 1 -- 172.21.15.72:0/2233002194 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc20108f00 con 0x7fdc20101be0 2024-03-22T01:34:43.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.011+0000 7fdc17fff640 1 -- 172.21.15.72:0/2233002194 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7fdc18044070 con 0x7fdc20101be0 2024-03-22T01:34:43.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.011+0000 7fdc17fff640 1 --2- 172.21.15.72:0/2233002194 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fdc0c0a2fb0 0x7fdc0c0a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:43.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.011+0000 7fdc1e7fc640 1 --2- 172.21.15.72:0/2233002194 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fdc0c0a2fb0 0x7fdc0c0a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:43.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.011+0000 7fdc17fff640 1 -- 172.21.15.72:0/2233002194 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fdc180f12f0 con 0x7fdc20101be0 2024-03-22T01:34:43.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.012+0000 7fdc1e7fc640 1 --2- 172.21.15.72:0/2233002194 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fdc0c0a2fb0 0x7fdc0c0a5470 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fdc20199430 tx=0x7fdc08005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:43.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.016+0000 7fdc17fff640 1 -- 172.21.15.72:0/2233002194 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fdc180bde90 con 0x7fdc20101be0 2024-03-22T01:34:43.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.187+0000 7fdc2558a640 1 -- 172.21.15.72:0/2233002194 --> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdc20061fd0 con 0x7fdc0c0a2fb0 2024-03-22T01:34:43.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.197+0000 7fdc17fff640 1 -- 172.21.15.72:0/2233002194 <== mgr.14152 v2:172.21.15.72:6800/401403200 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340201 (secure 0 0 0) 0x7fdc20061fd0 con 0x7fdc0c0a2fb0 2024-03-22T01:34:43.203 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:34:43.205 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-03-22T01:34:43.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.200+0000 7fdc2558a640 1 -- 172.21.15.72:0/2233002194 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fdc0c0a2fb0 msgr2=0x7fdc0c0a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:43.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.200+0000 7fdc2558a640 1 --2- 172.21.15.72:0/2233002194 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fdc0c0a2fb0 0x7fdc0c0a5470 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fdc20199430 tx=0x7fdc08005f90 comp rx=0 tx=0).stop 2024-03-22T01:34:43.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.200+0000 7fdc2558a640 1 -- 172.21.15.72:0/2233002194 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc20101be0 msgr2=0x7fdc20197f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:43.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.200+0000 7fdc2558a640 1 --2- 172.21.15.72:0/2233002194 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc20101be0 0x7fdc20197f20 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fdc18000c00 tx=0x7fdc18005d50 comp rx=0 tx=0).stop 2024-03-22T01:34:43.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.200+0000 7fdc2558a640 1 -- 172.21.15.72:0/2233002194 shutdown_connections 2024-03-22T01:34:43.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.200+0000 7fdc2558a640 1 --2- 172.21.15.72:0/2233002194 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fdc0c0a2fb0 0x7fdc0c0a5470 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:43.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.200+0000 7fdc2558a640 1 --2- 172.21.15.72:0/2233002194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc20102dd0 0x7fdc20198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:43.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.200+0000 7fdc2558a640 1 --2- 172.21.15.72:0/2233002194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc20103790 0x7fdc2019f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:43.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.200+0000 7fdc2558a640 1 --2- 172.21.15.72:0/2233002194 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc20101be0 0x7fdc20197f20 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:43.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.200+0000 7fdc2558a640 1 -- 172.21.15.72:0/2233002194 >> 172.21.15.72:0/2233002194 conn(0x7fdc200fd150 msgr2=0x7fdc201060d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:43.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.200+0000 7fdc2558a640 1 -- 172.21.15.72:0/2233002194 shutdown_connections 2024-03-22T01:34:43.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.200+0000 7fdc2558a640 1 -- 172.21.15.72:0/2233002194 wait complete. 2024-03-22T01:34:43.211 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":304,"stamp":"2024-03-22T01:34:41.427751+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":141,"num_read_kb":126,"num_write":230,"num_write_kb":2957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":180,"ondisk_log_size":180,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":366988,"kb_used_data":13596,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124574836,"statfs":{"total":1151940427776,"available":1151564632064,"internally_reserved":0,"allocated":13922304,"data_stored":7628076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":361674042},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2230,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":7,"num_write":4,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003439"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.191830+0000","last_change":"2024-03-22T01:34:15.487496+0000","last_active":"2024-03-22T01:34:21.191830+0000","last_peered":"2024-03-22T01:34:21.191830+0000","last_clean":"2024-03-22T01:34:21.191830+0000","last_became_active":"2024-03-22T01:34:15.486700+0000","last_became_peered":"2024-03-22T01:34:15.486700+0000","last_unstale":"2024-03-22T01:34:21.191830+0000","last_undegraded":"2024-03-22T01:34:21.191830+0000","last_fullsized":"2024-03-22T01:34:21.191830+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:03:19.146811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.192092+0000","last_change":"2024-03-22T01:34:15.489791+0000","last_active":"2024-03-22T01:34:21.192092+0000","last_peered":"2024-03-22T01:34:21.192092+0000","last_clean":"2024-03-22T01:34:21.192092+0000","last_became_active":"2024-03-22T01:34:15.489493+0000","last_became_peered":"2024-03-22T01:34:15.489493+0000","last_unstale":"2024-03-22T01:34:21.192092+0000","last_undegraded":"2024-03-22T01:34:21.192092+0000","last_fullsized":"2024-03-22T01:34:21.192092+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:45:05.060454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.191049+0000","last_change":"2024-03-22T01:34:15.491567+0000","last_active":"2024-03-22T01:34:21.191049+0000","last_peered":"2024-03-22T01:34:21.191049+0000","last_clean":"2024-03-22T01:34:21.191049+0000","last_became_active":"2024-03-22T01:34:15.491275+0000","last_became_peered":"2024-03-22T01:34:15.491275+0000","last_unstale":"2024-03-22T01:34:21.191049+0000","last_undegraded":"2024-03-22T01:34:21.191049+0000","last_fullsized":"2024-03-22T01:34:21.191049+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:26:36.630811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.511199+0000","last_change":"2024-03-22T01:34:15.488579+0000","last_active":"2024-03-22T01:34:20.511199+0000","last_peered":"2024-03-22T01:34:20.511199+0000","last_clean":"2024-03-22T01:34:20.511199+0000","last_became_active":"2024-03-22T01:34:15.488262+0000","last_became_peered":"2024-03-22T01:34:15.488262+0000","last_unstale":"2024-03-22T01:34:20.511199+0000","last_undegraded":"2024-03-22T01:34:20.511199+0000","last_fullsized":"2024-03-22T01:34:20.511199+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:58:21.788420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.876769+0000","last_change":"2024-03-22T01:34:15.485744+0000","last_active":"2024-03-22T01:34:20.876769+0000","last_peered":"2024-03-22T01:34:20.876769+0000","last_clean":"2024-03-22T01:34:20.876769+0000","last_became_active":"2024-03-22T01:34:15.485378+0000","last_became_peered":"2024-03-22T01:34:15.485378+0000","last_unstale":"2024-03-22T01:34:20.876769+0000","last_undegraded":"2024-03-22T01:34:20.876769+0000","last_fullsized":"2024-03-22T01:34:20.876769+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:06:40.979475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.191301+0000","last_change":"2024-03-22T01:34:15.495051+0000","last_active":"2024-03-22T01:34:21.191301+0000","last_peered":"2024-03-22T01:34:21.191301+0000","last_clean":"2024-03-22T01:34:21.191301+0000","last_became_active":"2024-03-22T01:34:15.494691+0000","last_became_peered":"2024-03-22T01:34:15.494691+0000","last_unstale":"2024-03-22T01:34:21.191301+0000","last_undegraded":"2024-03-22T01:34:21.191301+0000","last_fullsized":"2024-03-22T01:34:21.191301+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T01:44:28.336545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.083941+0000","last_change":"2024-03-22T01:34:15.493911+0000","last_active":"2024-03-22T01:34:21.083941+0000","last_peered":"2024-03-22T01:34:21.083941+0000","last_clean":"2024-03-22T01:34:21.083941+0000","last_became_active":"2024-03-22T01:34:15.493586+0000","last_became_peered":"2024-03-22T01:34:15.493586+0000","last_unstale":"2024-03-22T01:34:21.083941+0000","last_undegraded":"2024-03-22T01:34:21.083941+0000","last_fullsized":"2024-03-22T01:34:21.083941+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:22:41.391855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.191086+0000","last_change":"2024-03-22T01:34:15.481410+0000","last_active":"2024-03-22T01:34:21.191086+0000","last_peered":"2024-03-22T01:34:21.191086+0000","last_clean":"2024-03-22T01:34:21.191086+0000","last_became_active":"2024-03-22T01:34:15.481109+0000","last_became_peered":"2024-03-22T01:34:15.481109+0000","last_unstale":"2024-03-22T01:34:21.191086+0000","last_undegraded":"2024-03-22T01:34:21.191086+0000","last_fullsized":"2024-03-22T01:34:21.191086+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:27:10.004584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.600278+0000","last_change":"2024-03-22T01:34:15.485173+0000","last_active":"2024-03-22T01:34:20.600278+0000","last_peered":"2024-03-22T01:34:20.600278+0000","last_clean":"2024-03-22T01:34:20.600278+0000","last_became_active":"2024-03-22T01:34:15.484812+0000","last_became_peered":"2024-03-22T01:34:15.484812+0000","last_unstale":"2024-03-22T01:34:20.600278+0000","last_undegraded":"2024-03-22T01:34:20.600278+0000","last_fullsized":"2024-03-22T01:34:20.600278+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:44:06.383568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.191572+0000","last_change":"2024-03-22T01:34:15.497964+0000","last_active":"2024-03-22T01:34:21.191572+0000","last_peered":"2024-03-22T01:34:21.191572+0000","last_clean":"2024-03-22T01:34:21.191572+0000","last_became_active":"2024-03-22T01:34:15.496887+0000","last_became_peered":"2024-03-22T01:34:15.496887+0000","last_unstale":"2024-03-22T01:34:21.191572+0000","last_undegraded":"2024-03-22T01:34:21.191572+0000","last_fullsized":"2024-03-22T01:34:21.191572+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:13:40.420667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.084353+0000","last_change":"2024-03-22T01:34:15.496310+0000","last_active":"2024-03-22T01:34:21.084353+0000","last_peered":"2024-03-22T01:34:21.084353+0000","last_clean":"2024-03-22T01:34:21.084353+0000","last_became_active":"2024-03-22T01:34:15.494196+0000","last_became_peered":"2024-03-22T01:34:15.494196+0000","last_unstale":"2024-03-22T01:34:21.084353+0000","last_undegraded":"2024-03-22T01:34:21.084353+0000","last_fullsized":"2024-03-22T01:34:21.084353+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:33:02.798310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.510976+0000","last_change":"2024-03-22T01:34:15.495380+0000","last_active":"2024-03-22T01:34:20.510976+0000","last_peered":"2024-03-22T01:34:20.510976+0000","last_clean":"2024-03-22T01:34:20.510976+0000","last_became_active":"2024-03-22T01:34:15.495063+0000","last_became_peered":"2024-03-22T01:34:15.495063+0000","last_unstale":"2024-03-22T01:34:20.510976+0000","last_undegraded":"2024-03-22T01:34:20.510976+0000","last_fullsized":"2024-03-22T01:34:20.510976+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:46:29.497329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.990160+0000","last_change":"2024-03-22T01:34:15.497533+0000","last_active":"2024-03-22T01:34:20.990160+0000","last_peered":"2024-03-22T01:34:20.990160+0000","last_clean":"2024-03-22T01:34:20.990160+0000","last_became_active":"2024-03-22T01:34:15.497223+0000","last_became_peered":"2024-03-22T01:34:15.497223+0000","last_unstale":"2024-03-22T01:34:20.990160+0000","last_undegraded":"2024-03-22T01:34:20.990160+0000","last_fullsized":"2024-03-22T01:34:20.990160+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:17:49.202842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.190865+0000","last_change":"2024-03-22T01:34:15.491317+0000","last_active":"2024-03-22T01:34:21.190865+0000","last_peered":"2024-03-22T01:34:21.190865+0000","last_clean":"2024-03-22T01:34:21.190865+0000","last_became_active":"2024-03-22T01:34:15.490982+0000","last_became_peered":"2024-03-22T01:34:15.490982+0000","last_unstale":"2024-03-22T01:34:21.190865+0000","last_undegraded":"2024-03-22T01:34:21.190865+0000","last_fullsized":"2024-03-22T01:34:21.190865+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:05:39.889964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.190892+0000","last_change":"2024-03-22T01:34:15.488502+0000","last_active":"2024-03-22T01:34:21.190892+0000","last_peered":"2024-03-22T01:34:21.190892+0000","last_clean":"2024-03-22T01:34:21.190892+0000","last_became_active":"2024-03-22T01:34:15.488200+0000","last_became_peered":"2024-03-22T01:34:15.488200+0000","last_unstale":"2024-03-22T01:34:21.190892+0000","last_undegraded":"2024-03-22T01:34:21.190892+0000","last_fullsized":"2024-03-22T01:34:21.190892+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:52:59.991591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.599708+0000","last_change":"2024-03-22T01:34:15.489991+0000","last_active":"2024-03-22T01:34:20.599708+0000","last_peered":"2024-03-22T01:34:20.599708+0000","last_clean":"2024-03-22T01:34:20.599708+0000","last_became_active":"2024-03-22T01:34:15.489696+0000","last_became_peered":"2024-03-22T01:34:15.489696+0000","last_unstale":"2024-03-22T01:34:20.599708+0000","last_undegraded":"2024-03-22T01:34:20.599708+0000","last_fullsized":"2024-03-22T01:34:20.599708+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:10:45.316422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.191721+0000","last_change":"2024-03-22T01:34:15.488011+0000","last_active":"2024-03-22T01:34:21.191721+0000","last_peered":"2024-03-22T01:34:21.191721+0000","last_clean":"2024-03-22T01:34:21.191721+0000","last_became_active":"2024-03-22T01:34:15.487282+0000","last_became_peered":"2024-03-22T01:34:15.487282+0000","last_unstale":"2024-03-22T01:34:21.191721+0000","last_undegraded":"2024-03-22T01:34:21.191721+0000","last_fullsized":"2024-03-22T01:34:21.191721+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:16:21.396273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.510761+0000","last_change":"2024-03-22T01:34:15.488046+0000","last_active":"2024-03-22T01:34:20.510761+0000","last_peered":"2024-03-22T01:34:20.510761+0000","last_clean":"2024-03-22T01:34:20.510761+0000","last_became_active":"2024-03-22T01:34:15.487731+0000","last_became_peered":"2024-03-22T01:34:15.487731+0000","last_unstale":"2024-03-22T01:34:20.510761+0000","last_undegraded":"2024-03-22T01:34:20.510761+0000","last_fullsized":"2024-03-22T01:34:20.510761+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:44:16.931544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.512006+0000","last_change":"2024-03-22T01:34:15.503737+0000","last_active":"2024-03-22T01:34:20.512006+0000","last_peered":"2024-03-22T01:34:20.512006+0000","last_clean":"2024-03-22T01:34:20.512006+0000","last_became_active":"2024-03-22T01:34:15.502169+0000","last_became_peered":"2024-03-22T01:34:15.502169+0000","last_unstale":"2024-03-22T01:34:20.512006+0000","last_undegraded":"2024-03-22T01:34:20.512006+0000","last_fullsized":"2024-03-22T01:34:20.512006+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:05:53.027318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.511598+0000","last_change":"2024-03-22T01:34:19.527964+0000","last_active":"2024-03-22T01:34:20.511598+0000","last_peered":"2024-03-22T01:34:20.511598+0000","last_clean":"2024-03-22T01:34:20.511598+0000","last_became_active":"2024-03-22T01:34:19.527526+0000","last_became_peered":"2024-03-22T01:34:19.527526+0000","last_unstale":"2024-03-22T01:34:20.511598+0000","last_undegraded":"2024-03-22T01:34:20.511598+0000","last_fullsized":"2024-03-22T01:34:20.511598+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:35:39.404439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.511085+0000","last_change":"2024-03-22T01:34:14.473065+0000","last_active":"2024-03-22T01:34:20.511085+0000","last_peered":"2024-03-22T01:34:20.511085+0000","last_clean":"2024-03-22T01:34:20.511085+0000","last_became_active":"2024-03-22T01:34:14.472653+0000","last_became_peered":"2024-03-22T01:34:14.472653+0000","last_unstale":"2024-03-22T01:34:20.511085+0000","last_undegraded":"2024-03-22T01:34:20.511085+0000","last_fullsized":"2024-03-22T01:34:20.511085+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:11:09.525438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.191967+0000","last_change":"2024-03-22T01:34:15.497818+0000","last_active":"2024-03-22T01:34:21.191967+0000","last_peered":"2024-03-22T01:34:21.191967+0000","last_clean":"2024-03-22T01:34:21.191967+0000","last_became_active":"2024-03-22T01:34:15.497488+0000","last_became_peered":"2024-03-22T01:34:15.497488+0000","last_unstale":"2024-03-22T01:34:21.191967+0000","last_undegraded":"2024-03-22T01:34:21.191967+0000","last_fullsized":"2024-03-22T01:34:21.191967+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:21:09.340557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.510720+0000","last_change":"2024-03-22T01:34:19.528435+0000","last_active":"2024-03-22T01:34:20.510720+0000","last_peered":"2024-03-22T01:34:20.510720+0000","last_clean":"2024-03-22T01:34:20.510720+0000","last_became_active":"2024-03-22T01:34:19.528044+0000","last_became_peered":"2024-03-22T01:34:19.528044+0000","last_unstale":"2024-03-22T01:34:20.510720+0000","last_undegraded":"2024-03-22T01:34:20.510720+0000","last_fullsized":"2024-03-22T01:34:20.510720+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:52:05.682173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"86'1","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.192154+0000","last_change":"2024-03-22T01:34:14.465817+0000","last_active":"2024-03-22T01:34:21.192154+0000","last_peered":"2024-03-22T01:34:21.192154+0000","last_clean":"2024-03-22T01:34:21.192154+0000","last_became_active":"2024-03-22T01:34:14.464338+0000","last_became_peered":"2024-03-22T01:34:14.464338+0000","last_unstale":"2024-03-22T01:34:21.192154+0000","last_undegraded":"2024-03-22T01:34:21.192154+0000","last_fullsized":"2024-03-22T01:34:21.192154+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:04:47.457092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.191311+0000","last_change":"2024-03-22T01:34:15.481988+0000","last_active":"2024-03-22T01:34:21.191311+0000","last_peered":"2024-03-22T01:34:21.191311+0000","last_clean":"2024-03-22T01:34:21.191311+0000","last_became_active":"2024-03-22T01:34:15.481676+0000","last_became_peered":"2024-03-22T01:34:15.481676+0000","last_unstale":"2024-03-22T01:34:21.191311+0000","last_undegraded":"2024-03-22T01:34:21.191311+0000","last_fullsized":"2024-03-22T01:34:21.191311+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:37:54.609731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.511003+0000","last_change":"2024-03-22T01:34:19.528349+0000","last_active":"2024-03-22T01:34:20.511003+0000","last_peered":"2024-03-22T01:34:20.511003+0000","last_clean":"2024-03-22T01:34:20.511003+0000","last_became_active":"2024-03-22T01:34:19.527937+0000","last_became_peered":"2024-03-22T01:34:19.527937+0000","last_unstale":"2024-03-22T01:34:20.511003+0000","last_undegraded":"2024-03-22T01:34:20.511003+0000","last_fullsized":"2024-03-22T01:34:20.511003+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:48:34.242104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"86'2","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.877331+0000","last_change":"2024-03-22T01:34:14.472905+0000","last_active":"2024-03-22T01:34:20.877331+0000","last_peered":"2024-03-22T01:34:20.877331+0000","last_clean":"2024-03-22T01:34:20.877331+0000","last_became_active":"2024-03-22T01:34:14.472560+0000","last_became_peered":"2024-03-22T01:34:14.472560+0000","last_unstale":"2024-03-22T01:34:20.877331+0000","last_undegraded":"2024-03-22T01:34:20.877331+0000","last_fullsized":"2024-03-22T01:34:20.877331+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:19:41.363630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.599949+0000","last_change":"2024-03-22T01:34:15.491345+0000","last_active":"2024-03-22T01:34:20.599949+0000","last_peered":"2024-03-22T01:34:20.599949+0000","last_clean":"2024-03-22T01:34:20.599949+0000","last_became_active":"2024-03-22T01:34:15.490833+0000","last_became_peered":"2024-03-22T01:34:15.490833+0000","last_unstale":"2024-03-22T01:34:20.599949+0000","last_undegraded":"2024-03-22T01:34:20.599949+0000","last_fullsized":"2024-03-22T01:34:20.599949+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:30:43.950679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.510730+0000","last_change":"2024-03-22T01:34:19.528351+0000","last_active":"2024-03-22T01:34:20.510730+0000","last_peered":"2024-03-22T01:34:20.510730+0000","last_clean":"2024-03-22T01:34:20.510730+0000","last_became_active":"2024-03-22T01:34:19.527939+0000","last_became_peered":"2024-03-22T01:34:19.527939+0000","last_unstale":"2024-03-22T01:34:20.510730+0000","last_undegraded":"2024-03-22T01:34:20.510730+0000","last_fullsized":"2024-03-22T01:34:20.510730+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:30:21.055848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.877161+0000","last_change":"2024-03-22T01:34:14.467743+0000","last_active":"2024-03-22T01:34:20.877161+0000","last_peered":"2024-03-22T01:34:20.877161+0000","last_clean":"2024-03-22T01:34:20.877161+0000","last_became_active":"2024-03-22T01:34:14.467421+0000","last_became_peered":"2024-03-22T01:34:14.467421+0000","last_unstale":"2024-03-22T01:34:20.877161+0000","last_undegraded":"2024-03-22T01:34:20.877161+0000","last_fullsized":"2024-03-22T01:34:20.877161+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:26:41.770050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.092335+0000","last_change":"2024-03-22T01:34:15.479530+0000","last_active":"2024-03-22T01:34:21.092335+0000","last_peered":"2024-03-22T01:34:21.092335+0000","last_clean":"2024-03-22T01:34:21.092335+0000","last_became_active":"2024-03-22T01:34:15.479198+0000","last_became_peered":"2024-03-22T01:34:15.479198+0000","last_unstale":"2024-03-22T01:34:21.092335+0000","last_undegraded":"2024-03-22T01:34:21.092335+0000","last_fullsized":"2024-03-22T01:34:21.092335+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:28:51.624369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.511923+0000","last_change":"2024-03-22T01:34:19.524486+0000","last_active":"2024-03-22T01:34:20.511923+0000","last_peered":"2024-03-22T01:34:20.511923+0000","last_clean":"2024-03-22T01:34:20.511923+0000","last_became_active":"2024-03-22T01:34:19.524030+0000","last_became_peered":"2024-03-22T01:34:19.524030+0000","last_unstale":"2024-03-22T01:34:20.511923+0000","last_undegraded":"2024-03-22T01:34:20.511923+0000","last_fullsized":"2024-03-22T01:34:20.511923+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:31:10.157431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"86'1","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.091981+0000","last_change":"2024-03-22T01:34:14.469191+0000","last_active":"2024-03-22T01:34:21.091981+0000","last_peered":"2024-03-22T01:34:21.091981+0000","last_clean":"2024-03-22T01:34:21.091981+0000","last_became_active":"2024-03-22T01:34:14.468842+0000","last_became_peered":"2024-03-22T01:34:14.468842+0000","last_unstale":"2024-03-22T01:34:21.091981+0000","last_undegraded":"2024-03-22T01:34:21.091981+0000","last_fullsized":"2024-03-22T01:34:21.091981+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:29:56.566256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.876794+0000","last_change":"2024-03-22T01:34:15.483736+0000","last_active":"2024-03-22T01:34:20.876794+0000","last_peered":"2024-03-22T01:34:20.876794+0000","last_clean":"2024-03-22T01:34:20.876794+0000","last_became_active":"2024-03-22T01:34:15.483401+0000","last_became_peered":"2024-03-22T01:34:15.483401+0000","last_unstale":"2024-03-22T01:34:20.876794+0000","last_undegraded":"2024-03-22T01:34:20.876794+0000","last_fullsized":"2024-03-22T01:34:20.876794+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:26:51.974932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.511890+0000","last_change":"2024-03-22T01:34:19.534542+0000","last_active":"2024-03-22T01:34:20.511890+0000","last_peered":"2024-03-22T01:34:20.511890+0000","last_clean":"2024-03-22T01:34:20.511890+0000","last_became_active":"2024-03-22T01:34:19.534039+0000","last_became_peered":"2024-03-22T01:34:19.534039+0000","last_unstale":"2024-03-22T01:34:20.511890+0000","last_undegraded":"2024-03-22T01:34:20.511890+0000","last_fullsized":"2024-03-22T01:34:20.511890+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:59:37.363972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"86'1","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.191265+0000","last_change":"2024-03-22T01:34:14.468382+0000","last_active":"2024-03-22T01:34:21.191265+0000","last_peered":"2024-03-22T01:34:21.191265+0000","last_clean":"2024-03-22T01:34:21.191265+0000","last_became_active":"2024-03-22T01:34:14.467939+0000","last_became_peered":"2024-03-22T01:34:14.467939+0000","last_unstale":"2024-03-22T01:34:21.191265+0000","last_undegraded":"2024-03-22T01:34:21.191265+0000","last_fullsized":"2024-03-22T01:34:21.191265+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:57:17.031893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.092181+0000","last_change":"2024-03-22T01:34:15.484361+0000","last_active":"2024-03-22T01:34:21.092181+0000","last_peered":"2024-03-22T01:34:21.092181+0000","last_clean":"2024-03-22T01:34:21.092181+0000","last_became_active":"2024-03-22T01:34:15.483894+0000","last_became_peered":"2024-03-22T01:34:15.483894+0000","last_unstale":"2024-03-22T01:34:21.092181+0000","last_undegraded":"2024-03-22T01:34:21.092181+0000","last_fullsized":"2024-03-22T01:34:21.092181+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:18:53.527145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.191786+0000","last_change":"2024-03-22T01:34:19.531532+0000","last_active":"2024-03-22T01:34:21.191786+0000","last_peered":"2024-03-22T01:34:21.191786+0000","last_clean":"2024-03-22T01:34:21.191786+0000","last_became_active":"2024-03-22T01:34:19.531202+0000","last_became_peered":"2024-03-22T01:34:19.531202+0000","last_unstale":"2024-03-22T01:34:21.191786+0000","last_undegraded":"2024-03-22T01:34:21.191786+0000","last_fullsized":"2024-03-22T01:34:21.191786+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:52:12.823806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.989912+0000","last_change":"2024-03-22T01:34:14.464485+0000","last_active":"2024-03-22T01:34:20.989912+0000","last_peered":"2024-03-22T01:34:20.989912+0000","last_clean":"2024-03-22T01:34:20.989912+0000","last_became_active":"2024-03-22T01:34:14.463973+0000","last_became_peered":"2024-03-22T01:34:14.463973+0000","last_unstale":"2024-03-22T01:34:20.989912+0000","last_undegraded":"2024-03-22T01:34:20.989912+0000","last_fullsized":"2024-03-22T01:34:20.989912+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:11:13.265346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.191338+0000","last_change":"2024-03-22T01:34:15.483893+0000","last_active":"2024-03-22T01:34:21.191338+0000","last_peered":"2024-03-22T01:34:21.191338+0000","last_clean":"2024-03-22T01:34:21.191338+0000","last_became_active":"2024-03-22T01:34:15.482295+0000","last_became_peered":"2024-03-22T01:34:15.482295+0000","last_unstale":"2024-03-22T01:34:21.191338+0000","last_undegraded":"2024-03-22T01:34:21.191338+0000","last_fullsized":"2024-03-22T01:34:21.191338+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:17:33.274900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.192160+0000","last_change":"2024-03-22T01:34:19.522567+0000","last_active":"2024-03-22T01:34:21.192160+0000","last_peered":"2024-03-22T01:34:21.192160+0000","last_clean":"2024-03-22T01:34:21.192160+0000","last_became_active":"2024-03-22T01:34:19.522165+0000","last_became_peered":"2024-03-22T01:34:19.522165+0000","last_unstale":"2024-03-22T01:34:21.192160+0000","last_undegraded":"2024-03-22T01:34:21.192160+0000","last_fullsized":"2024-03-22T01:34:21.192160+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:41:37.829622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.091814+0000","last_change":"2024-03-22T01:34:14.463824+0000","last_active":"2024-03-22T01:34:21.091814+0000","last_peered":"2024-03-22T01:34:21.091814+0000","last_clean":"2024-03-22T01:34:21.091814+0000","last_became_active":"2024-03-22T01:34:14.463484+0000","last_became_peered":"2024-03-22T01:34:14.463484+0000","last_unstale":"2024-03-22T01:34:21.091814+0000","last_undegraded":"2024-03-22T01:34:21.091814+0000","last_fullsized":"2024-03-22T01:34:21.091814+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:19:31.568853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.510383+0000","last_change":"2024-03-22T01:34:15.488883+0000","last_active":"2024-03-22T01:34:20.510383+0000","last_peered":"2024-03-22T01:34:20.510383+0000","last_clean":"2024-03-22T01:34:20.510383+0000","last_became_active":"2024-03-22T01:34:15.488573+0000","last_became_peered":"2024-03-22T01:34:15.488573+0000","last_unstale":"2024-03-22T01:34:20.510383+0000","last_undegraded":"2024-03-22T01:34:20.510383+0000","last_fullsized":"2024-03-22T01:34:20.510383+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:28:46.388930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.192007+0000","last_change":"2024-03-22T01:34:19.527321+0000","last_active":"2024-03-22T01:34:21.192007+0000","last_peered":"2024-03-22T01:34:21.192007+0000","last_clean":"2024-03-22T01:34:21.192007+0000","last_became_active":"2024-03-22T01:34:19.526905+0000","last_became_peered":"2024-03-22T01:34:19.526905+0000","last_unstale":"2024-03-22T01:34:21.192007+0000","last_undegraded":"2024-03-22T01:34:21.192007+0000","last_fullsized":"2024-03-22T01:34:21.192007+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:49:31.023938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.191570+0000","last_change":"2024-03-22T01:34:14.471415+0000","last_active":"2024-03-22T01:34:21.191570+0000","last_peered":"2024-03-22T01:34:21.191570+0000","last_clean":"2024-03-22T01:34:21.191570+0000","last_became_active":"2024-03-22T01:34:14.470928+0000","last_became_peered":"2024-03-22T01:34:14.470928+0000","last_unstale":"2024-03-22T01:34:21.191570+0000","last_undegraded":"2024-03-22T01:34:21.191570+0000","last_fullsized":"2024-03-22T01:34:21.191570+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:09:10.286549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.600275+0000","last_change":"2024-03-22T01:34:15.489656+0000","last_active":"2024-03-22T01:34:20.600275+0000","last_peered":"2024-03-22T01:34:20.600275+0000","last_clean":"2024-03-22T01:34:20.600275+0000","last_became_active":"2024-03-22T01:34:15.488933+0000","last_became_peered":"2024-03-22T01:34:15.488933+0000","last_unstale":"2024-03-22T01:34:20.600275+0000","last_undegraded":"2024-03-22T01:34:20.600275+0000","last_fullsized":"2024-03-22T01:34:20.600275+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:17:38.789414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.510408+0000","last_change":"2024-03-22T01:34:19.534174+0000","last_active":"2024-03-22T01:34:20.510408+0000","last_peered":"2024-03-22T01:34:20.510408+0000","last_clean":"2024-03-22T01:34:20.510408+0000","last_became_active":"2024-03-22T01:34:19.533759+0000","last_became_peered":"2024-03-22T01:34:19.533759+0000","last_unstale":"2024-03-22T01:34:20.510408+0000","last_undegraded":"2024-03-22T01:34:20.510408+0000","last_fullsized":"2024-03-22T01:34:20.510408+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:24:50.363984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.084219+0000","last_change":"2024-03-22T01:34:14.465755+0000","last_active":"2024-03-22T01:34:21.084219+0000","last_peered":"2024-03-22T01:34:21.084219+0000","last_clean":"2024-03-22T01:34:21.084219+0000","last_became_active":"2024-03-22T01:34:14.465230+0000","last_became_peered":"2024-03-22T01:34:14.465230+0000","last_unstale":"2024-03-22T01:34:21.084219+0000","last_undegraded":"2024-03-22T01:34:21.084219+0000","last_fullsized":"2024-03-22T01:34:21.084219+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:56:29.286551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.191316+0000","last_change":"2024-03-22T01:34:15.476563+0000","last_active":"2024-03-22T01:34:21.191316+0000","last_peered":"2024-03-22T01:34:21.191316+0000","last_clean":"2024-03-22T01:34:21.191316+0000","last_became_active":"2024-03-22T01:34:15.476250+0000","last_became_peered":"2024-03-22T01:34:15.476250+0000","last_unstale":"2024-03-22T01:34:21.191316+0000","last_undegraded":"2024-03-22T01:34:21.191316+0000","last_fullsized":"2024-03-22T01:34:21.191316+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:45:59.605528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.511303+0000","last_change":"2024-03-22T01:34:19.527218+0000","last_active":"2024-03-22T01:34:20.511303+0000","last_peered":"2024-03-22T01:34:20.511303+0000","last_clean":"2024-03-22T01:34:20.511303+0000","last_became_active":"2024-03-22T01:34:19.526856+0000","last_became_peered":"2024-03-22T01:34:19.526856+0000","last_unstale":"2024-03-22T01:34:20.511303+0000","last_undegraded":"2024-03-22T01:34:20.511303+0000","last_fullsized":"2024-03-22T01:34:20.511303+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:25:33.878015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"86'3","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.092241+0000","last_change":"2024-03-22T01:34:14.463365+0000","last_active":"2024-03-22T01:34:21.092241+0000","last_peered":"2024-03-22T01:34:21.092241+0000","last_clean":"2024-03-22T01:34:21.092241+0000","last_became_active":"2024-03-22T01:34:14.462984+0000","last_became_peered":"2024-03-22T01:34:14.462984+0000","last_unstale":"2024-03-22T01:34:21.092241+0000","last_undegraded":"2024-03-22T01:34:21.092241+0000","last_fullsized":"2024-03-22T01:34:21.092241+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:55:27.318854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.600003+0000","last_change":"2024-03-22T01:34:15.485309+0000","last_active":"2024-03-22T01:34:20.600003+0000","last_peered":"2024-03-22T01:34:20.600003+0000","last_clean":"2024-03-22T01:34:20.600003+0000","last_became_active":"2024-03-22T01:34:15.484964+0000","last_became_peered":"2024-03-22T01:34:15.484964+0000","last_unstale":"2024-03-22T01:34:20.600003+0000","last_undegraded":"2024-03-22T01:34:20.600003+0000","last_fullsized":"2024-03-22T01:34:20.600003+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:25:57.911623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.510773+0000","last_change":"2024-03-22T01:34:19.529346+0000","last_active":"2024-03-22T01:34:20.510773+0000","last_peered":"2024-03-22T01:34:20.510773+0000","last_clean":"2024-03-22T01:34:20.510773+0000","last_became_active":"2024-03-22T01:34:19.525999+0000","last_became_peered":"2024-03-22T01:34:19.525999+0000","last_unstale":"2024-03-22T01:34:20.510773+0000","last_undegraded":"2024-03-22T01:34:20.510773+0000","last_fullsized":"2024-03-22T01:34:20.510773+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:00:21.868812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'10","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:39.513848+0000","last_change":"2024-03-22T01:34:14.469378+0000","last_active":"2024-03-22T01:34:39.513848+0000","last_peered":"2024-03-22T01:34:39.513848+0000","last_clean":"2024-03-22T01:34:39.513848+0000","last_became_active":"2024-03-22T01:34:14.468956+0000","last_became_peered":"2024-03-22T01:34:14.468956+0000","last_unstale":"2024-03-22T01:34:39.513848+0000","last_undegraded":"2024-03-22T01:34:39.513848+0000","last_fullsized":"2024-03-22T01:34:39.513848+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:40:49.048793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":6,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.877123+0000","last_change":"2024-03-22T01:34:15.488497+0000","last_active":"2024-03-22T01:34:20.877123+0000","last_peered":"2024-03-22T01:34:20.877123+0000","last_clean":"2024-03-22T01:34:20.877123+0000","last_became_active":"2024-03-22T01:34:15.488134+0000","last_became_peered":"2024-03-22T01:34:15.488134+0000","last_unstale":"2024-03-22T01:34:20.877123+0000","last_undegraded":"2024-03-22T01:34:20.877123+0000","last_fullsized":"2024-03-22T01:34:20.877123+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:50:41.206135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.092612+0000","last_change":"2024-03-22T01:34:19.528744+0000","last_active":"2024-03-22T01:34:21.092612+0000","last_peered":"2024-03-22T01:34:21.092612+0000","last_clean":"2024-03-22T01:34:21.092612+0000","last_became_active":"2024-03-22T01:34:19.528375+0000","last_became_peered":"2024-03-22T01:34:19.528375+0000","last_unstale":"2024-03-22T01:34:21.092612+0000","last_undegraded":"2024-03-22T01:34:21.092612+0000","last_fullsized":"2024-03-22T01:34:21.092612+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:47:53.753923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"86'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.511589+0000","last_change":"2024-03-22T01:34:14.463289+0000","last_active":"2024-03-22T01:34:20.511589+0000","last_peered":"2024-03-22T01:34:20.511589+0000","last_clean":"2024-03-22T01:34:20.511589+0000","last_became_active":"2024-03-22T01:34:14.462883+0000","last_became_peered":"2024-03-22T01:34:14.462883+0000","last_unstale":"2024-03-22T01:34:20.511589+0000","last_undegraded":"2024-03-22T01:34:20.511589+0000","last_fullsized":"2024-03-22T01:34:20.511589+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:06:14.452674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.990049+0000","last_change":"2024-03-22T01:34:15.484785+0000","last_active":"2024-03-22T01:34:20.990049+0000","last_peered":"2024-03-22T01:34:20.990049+0000","last_clean":"2024-03-22T01:34:20.990049+0000","last_became_active":"2024-03-22T01:34:15.484474+0000","last_became_peered":"2024-03-22T01:34:15.484474+0000","last_unstale":"2024-03-22T01:34:20.990049+0000","last_undegraded":"2024-03-22T01:34:20.990049+0000","last_fullsized":"2024-03-22T01:34:20.990049+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:11:13.338836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.191504+0000","last_change":"2024-03-22T01:34:19.527379+0000","last_active":"2024-03-22T01:34:21.191504+0000","last_peered":"2024-03-22T01:34:21.191504+0000","last_clean":"2024-03-22T01:34:21.191504+0000","last_became_active":"2024-03-22T01:34:19.526957+0000","last_became_peered":"2024-03-22T01:34:19.526957+0000","last_unstale":"2024-03-22T01:34:21.191504+0000","last_undegraded":"2024-03-22T01:34:21.191504+0000","last_fullsized":"2024-03-22T01:34:21.191504+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:03:09.538334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.091981+0000","last_change":"2024-03-22T01:34:14.467372+0000","last_active":"2024-03-22T01:34:21.091981+0000","last_peered":"2024-03-22T01:34:21.091981+0000","last_clean":"2024-03-22T01:34:21.091981+0000","last_became_active":"2024-03-22T01:34:14.466124+0000","last_became_peered":"2024-03-22T01:34:14.466124+0000","last_unstale":"2024-03-22T01:34:21.091981+0000","last_undegraded":"2024-03-22T01:34:21.091981+0000","last_fullsized":"2024-03-22T01:34:21.091981+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:15:41.804735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.084102+0000","last_change":"2024-03-22T01:34:15.491533+0000","last_active":"2024-03-22T01:34:21.084102+0000","last_peered":"2024-03-22T01:34:21.084102+0000","last_clean":"2024-03-22T01:34:21.084102+0000","last_became_active":"2024-03-22T01:34:15.491208+0000","last_became_peered":"2024-03-22T01:34:15.491208+0000","last_unstale":"2024-03-22T01:34:21.084102+0000","last_undegraded":"2024-03-22T01:34:21.084102+0000","last_fullsized":"2024-03-22T01:34:21.084102+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:03:07.800607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.511063+0000","last_change":"2024-03-22T01:34:19.533419+0000","last_active":"2024-03-22T01:34:20.511063+0000","last_peered":"2024-03-22T01:34:20.511063+0000","last_clean":"2024-03-22T01:34:20.511063+0000","last_became_active":"2024-03-22T01:34:19.530042+0000","last_became_peered":"2024-03-22T01:34:19.530042+0000","last_unstale":"2024-03-22T01:34:20.511063+0000","last_undegraded":"2024-03-22T01:34:20.511063+0000","last_fullsized":"2024-03-22T01:34:20.511063+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:46:59.859742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.510719+0000","last_change":"2024-03-22T01:34:14.467033+0000","last_active":"2024-03-22T01:34:20.510719+0000","last_peered":"2024-03-22T01:34:20.510719+0000","last_clean":"2024-03-22T01:34:20.510719+0000","last_became_active":"2024-03-22T01:34:14.466649+0000","last_became_peered":"2024-03-22T01:34:14.466649+0000","last_unstale":"2024-03-22T01:34:20.510719+0000","last_undegraded":"2024-03-22T01:34:20.510719+0000","last_fullsized":"2024-03-22T01:34:20.510719+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:38:44.890899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.083839+0000","last_change":"2024-03-22T01:34:15.502422+0000","last_active":"2024-03-22T01:34:21.083839+0000","last_peered":"2024-03-22T01:34:21.083839+0000","last_clean":"2024-03-22T01:34:21.083839+0000","last_became_active":"2024-03-22T01:34:15.501782+0000","last_became_peered":"2024-03-22T01:34:15.501782+0000","last_unstale":"2024-03-22T01:34:21.083839+0000","last_undegraded":"2024-03-22T01:34:21.083839+0000","last_fullsized":"2024-03-22T01:34:21.083839+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:09:06.408318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.990486+0000","last_change":"2024-03-22T01:34:19.531417+0000","last_active":"2024-03-22T01:34:20.990486+0000","last_peered":"2024-03-22T01:34:20.990486+0000","last_clean":"2024-03-22T01:34:20.990486+0000","last_became_active":"2024-03-22T01:34:19.531003+0000","last_became_peered":"2024-03-22T01:34:19.531003+0000","last_unstale":"2024-03-22T01:34:20.990486+0000","last_undegraded":"2024-03-22T01:34:20.990486+0000","last_fullsized":"2024-03-22T01:34:20.990486+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:33:52.865110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"86'8","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.876692+0000","last_change":"2024-03-22T01:34:14.472904+0000","last_active":"2024-03-22T01:34:20.876692+0000","last_peered":"2024-03-22T01:34:20.876692+0000","last_clean":"2024-03-22T01:34:20.876692+0000","last_became_active":"2024-03-22T01:34:14.472442+0000","last_became_peered":"2024-03-22T01:34:14.472442+0000","last_unstale":"2024-03-22T01:34:20.876692+0000","last_undegraded":"2024-03-22T01:34:20.876692+0000","last_fullsized":"2024-03-22T01:34:20.876692+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:57:45.478316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.091885+0000","last_change":"2024-03-22T01:34:15.477411+0000","last_active":"2024-03-22T01:34:21.091885+0000","last_peered":"2024-03-22T01:34:21.091885+0000","last_clean":"2024-03-22T01:34:21.091885+0000","last_became_active":"2024-03-22T01:34:15.477035+0000","last_became_peered":"2024-03-22T01:34:15.477035+0000","last_unstale":"2024-03-22T01:34:21.091885+0000","last_undegraded":"2024-03-22T01:34:21.091885+0000","last_fullsized":"2024-03-22T01:34:21.091885+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:33:13.410872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.091888+0000","last_change":"2024-03-22T01:34:19.528258+0000","last_active":"2024-03-22T01:34:21.091888+0000","last_peered":"2024-03-22T01:34:21.091888+0000","last_clean":"2024-03-22T01:34:21.091888+0000","last_became_active":"2024-03-22T01:34:19.527668+0000","last_became_peered":"2024-03-22T01:34:19.527668+0000","last_unstale":"2024-03-22T01:34:21.091888+0000","last_undegraded":"2024-03-22T01:34:21.091888+0000","last_fullsized":"2024-03-22T01:34:21.091888+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:34:19.162609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'1","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.877235+0000","last_change":"2024-03-22T01:34:14.464735+0000","last_active":"2024-03-22T01:34:20.877235+0000","last_peered":"2024-03-22T01:34:20.877235+0000","last_clean":"2024-03-22T01:34:20.877235+0000","last_became_active":"2024-03-22T01:34:14.464359+0000","last_became_peered":"2024-03-22T01:34:14.464359+0000","last_unstale":"2024-03-22T01:34:20.877235+0000","last_undegraded":"2024-03-22T01:34:20.877235+0000","last_fullsized":"2024-03-22T01:34:20.877235+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:21:18.354922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.191611+0000","last_change":"2024-03-22T01:34:15.493904+0000","last_active":"2024-03-22T01:34:21.191611+0000","last_peered":"2024-03-22T01:34:21.191611+0000","last_clean":"2024-03-22T01:34:21.191611+0000","last_became_active":"2024-03-22T01:34:15.493557+0000","last_became_peered":"2024-03-22T01:34:15.493557+0000","last_unstale":"2024-03-22T01:34:21.191611+0000","last_undegraded":"2024-03-22T01:34:21.191611+0000","last_fullsized":"2024-03-22T01:34:21.191611+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:52:56.295047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.876991+0000","last_change":"2024-03-22T01:34:19.668136+0000","last_active":"2024-03-22T01:34:20.876991+0000","last_peered":"2024-03-22T01:34:20.876991+0000","last_clean":"2024-03-22T01:34:20.876991+0000","last_became_active":"2024-03-22T01:34:19.667633+0000","last_became_peered":"2024-03-22T01:34:19.667633+0000","last_unstale":"2024-03-22T01:34:20.876991+0000","last_undegraded":"2024-03-22T01:34:20.876991+0000","last_fullsized":"2024-03-22T01:34:20.876991+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:07:26.455373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.510615+0000","last_change":"2024-03-22T01:34:14.471959+0000","last_active":"2024-03-22T01:34:20.510615+0000","last_peered":"2024-03-22T01:34:20.510615+0000","last_clean":"2024-03-22T01:34:20.510615+0000","last_became_active":"2024-03-22T01:34:14.471618+0000","last_became_peered":"2024-03-22T01:34:14.471618+0000","last_unstale":"2024-03-22T01:34:20.510615+0000","last_undegraded":"2024-03-22T01:34:20.510615+0000","last_fullsized":"2024-03-22T01:34:20.510615+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:45:03.537013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.191540+0000","last_change":"2024-03-22T01:34:15.495770+0000","last_active":"2024-03-22T01:34:21.191540+0000","last_peered":"2024-03-22T01:34:21.191540+0000","last_clean":"2024-03-22T01:34:21.191540+0000","last_became_active":"2024-03-22T01:34:15.495403+0000","last_became_peered":"2024-03-22T01:34:15.495403+0000","last_unstale":"2024-03-22T01:34:21.191540+0000","last_undegraded":"2024-03-22T01:34:21.191540+0000","last_fullsized":"2024-03-22T01:34:21.191540+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:48:05.184305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.990326+0000","last_change":"2024-03-22T01:34:19.520204+0000","last_active":"2024-03-22T01:34:20.990326+0000","last_peered":"2024-03-22T01:34:20.990326+0000","last_clean":"2024-03-22T01:34:20.990326+0000","last_became_active":"2024-03-22T01:34:19.519847+0000","last_became_peered":"2024-03-22T01:34:19.519847+0000","last_unstale":"2024-03-22T01:34:20.990326+0000","last_undegraded":"2024-03-22T01:34:20.990326+0000","last_fullsized":"2024-03-22T01:34:20.990326+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:06:21.389636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.084029+0000","last_change":"2024-03-22T01:34:14.464162+0000","last_active":"2024-03-22T01:34:21.084029+0000","last_peered":"2024-03-22T01:34:21.084029+0000","last_clean":"2024-03-22T01:34:21.084029+0000","last_became_active":"2024-03-22T01:34:14.462700+0000","last_became_peered":"2024-03-22T01:34:14.462700+0000","last_unstale":"2024-03-22T01:34:21.084029+0000","last_undegraded":"2024-03-22T01:34:21.084029+0000","last_fullsized":"2024-03-22T01:34:21.084029+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:45:32.262466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.191342+0000","last_change":"2024-03-22T01:34:15.488256+0000","last_active":"2024-03-22T01:34:21.191342+0000","last_peered":"2024-03-22T01:34:21.191342+0000","last_clean":"2024-03-22T01:34:21.191342+0000","last_became_active":"2024-03-22T01:34:15.487939+0000","last_became_peered":"2024-03-22T01:34:15.487939+0000","last_unstale":"2024-03-22T01:34:21.191342+0000","last_undegraded":"2024-03-22T01:34:21.191342+0000","last_fullsized":"2024-03-22T01:34:21.191342+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:30:49.323531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.990386+0000","last_change":"2024-03-22T01:34:19.667633+0000","last_active":"2024-03-22T01:34:20.990386+0000","last_peered":"2024-03-22T01:34:20.990386+0000","last_clean":"2024-03-22T01:34:20.990386+0000","last_became_active":"2024-03-22T01:34:19.667199+0000","last_became_peered":"2024-03-22T01:34:19.667199+0000","last_unstale":"2024-03-22T01:34:20.990386+0000","last_undegraded":"2024-03-22T01:34:20.990386+0000","last_fullsized":"2024-03-22T01:34:20.990386+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T01:58:13.459475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.091992+0000","last_change":"2024-03-22T01:34:15.489468+0000","last_active":"2024-03-22T01:34:21.091992+0000","last_peered":"2024-03-22T01:34:21.091992+0000","last_clean":"2024-03-22T01:34:21.091992+0000","last_became_active":"2024-03-22T01:34:15.489029+0000","last_became_peered":"2024-03-22T01:34:15.489029+0000","last_unstale":"2024-03-22T01:34:21.091992+0000","last_undegraded":"2024-03-22T01:34:21.091992+0000","last_fullsized":"2024-03-22T01:34:21.091992+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:00:01.370161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.989799+0000","last_change":"2024-03-22T01:34:14.464552+0000","last_active":"2024-03-22T01:34:20.989799+0000","last_peered":"2024-03-22T01:34:20.989799+0000","last_clean":"2024-03-22T01:34:20.989799+0000","last_became_active":"2024-03-22T01:34:14.463554+0000","last_became_peered":"2024-03-22T01:34:14.463554+0000","last_unstale":"2024-03-22T01:34:20.989799+0000","last_undegraded":"2024-03-22T01:34:20.989799+0000","last_fullsized":"2024-03-22T01:34:20.989799+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:51:16.709278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.084337+0000","last_change":"2024-03-22T01:34:19.526066+0000","last_active":"2024-03-22T01:34:21.084337+0000","last_peered":"2024-03-22T01:34:21.084337+0000","last_clean":"2024-03-22T01:34:21.084337+0000","last_became_active":"2024-03-22T01:34:19.525659+0000","last_became_peered":"2024-03-22T01:34:19.525659+0000","last_unstale":"2024-03-22T01:34:21.084337+0000","last_undegraded":"2024-03-22T01:34:21.084337+0000","last_fullsized":"2024-03-22T01:34:21.084337+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:06:26.287971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.191440+0000","last_change":"2024-03-22T01:34:15.477018+0000","last_active":"2024-03-22T01:34:21.191440+0000","last_peered":"2024-03-22T01:34:21.191440+0000","last_clean":"2024-03-22T01:34:21.191440+0000","last_became_active":"2024-03-22T01:34:15.476613+0000","last_became_peered":"2024-03-22T01:34:15.476613+0000","last_unstale":"2024-03-22T01:34:21.191440+0000","last_undegraded":"2024-03-22T01:34:21.191440+0000","last_fullsized":"2024-03-22T01:34:21.191440+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:10:07.980123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.510790+0000","last_change":"2024-03-22T01:34:14.470922+0000","last_active":"2024-03-22T01:34:20.510790+0000","last_peered":"2024-03-22T01:34:20.510790+0000","last_clean":"2024-03-22T01:34:20.510790+0000","last_became_active":"2024-03-22T01:34:14.470515+0000","last_became_peered":"2024-03-22T01:34:14.470515+0000","last_unstale":"2024-03-22T01:34:20.510790+0000","last_undegraded":"2024-03-22T01:34:20.510790+0000","last_fullsized":"2024-03-22T01:34:20.510790+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:52:50.203631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":354,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.083835+0000","last_change":"2024-03-22T01:33:03.615234+0000","last_active":"2024-03-22T01:34:21.083835+0000","last_peered":"2024-03-22T01:34:21.083835+0000","last_clean":"2024-03-22T01:34:21.083835+0000","last_became_active":"2024-03-22T01:33:03.614710+0000","last_became_peered":"2024-03-22T01:33:03.614710+0000","last_unstale":"2024-03-22T01:34:21.083835+0000","last_undegraded":"2024-03-22T01:34:21.083835+0000","last_fullsized":"2024-03-22T01:34:21.083835+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:29:38.312292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:29:38.312292+0000","last_clean_scrub_stamp":"2024-03-22T01:29:38.312292+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:56:42.348119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.092499+0000","last_change":"2024-03-22T01:34:19.668565+0000","last_active":"2024-03-22T01:34:21.092499+0000","last_peered":"2024-03-22T01:34:21.092499+0000","last_clean":"2024-03-22T01:34:21.092499+0000","last_became_active":"2024-03-22T01:34:19.667959+0000","last_became_peered":"2024-03-22T01:34:19.667959+0000","last_unstale":"2024-03-22T01:34:21.092499+0000","last_undegraded":"2024-03-22T01:34:21.092499+0000","last_fullsized":"2024-03-22T01:34:21.092499+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:13:04.447486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.989797+0000","last_change":"2024-03-22T01:34:15.489513+0000","last_active":"2024-03-22T01:34:20.989797+0000","last_peered":"2024-03-22T01:34:20.989797+0000","last_clean":"2024-03-22T01:34:20.989797+0000","last_became_active":"2024-03-22T01:34:15.489188+0000","last_became_peered":"2024-03-22T01:34:15.489188+0000","last_unstale":"2024-03-22T01:34:20.989797+0000","last_undegraded":"2024-03-22T01:34:20.989797+0000","last_fullsized":"2024-03-22T01:34:20.989797+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:08:24.859350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.083994+0000","last_change":"2024-03-22T01:34:14.466305+0000","last_active":"2024-03-22T01:34:21.083994+0000","last_peered":"2024-03-22T01:34:21.083994+0000","last_clean":"2024-03-22T01:34:21.083994+0000","last_became_active":"2024-03-22T01:34:14.465495+0000","last_became_peered":"2024-03-22T01:34:14.465495+0000","last_unstale":"2024-03-22T01:34:21.083994+0000","last_undegraded":"2024-03-22T01:34:21.083994+0000","last_fullsized":"2024-03-22T01:34:21.083994+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:45:13.481187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.191481+0000","last_change":"2024-03-22T01:34:19.526970+0000","last_active":"2024-03-22T01:34:21.191481+0000","last_peered":"2024-03-22T01:34:21.191481+0000","last_clean":"2024-03-22T01:34:21.191481+0000","last_became_active":"2024-03-22T01:34:19.526462+0000","last_became_peered":"2024-03-22T01:34:19.526462+0000","last_unstale":"2024-03-22T01:34:21.191481+0000","last_undegraded":"2024-03-22T01:34:21.191481+0000","last_fullsized":"2024-03-22T01:34:21.191481+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:04:18.417358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.877214+0000","last_change":"2024-03-22T01:34:14.473771+0000","last_active":"2024-03-22T01:34:20.877214+0000","last_peered":"2024-03-22T01:34:20.877214+0000","last_clean":"2024-03-22T01:34:20.877214+0000","last_became_active":"2024-03-22T01:34:14.473188+0000","last_became_peered":"2024-03-22T01:34:14.473188+0000","last_unstale":"2024-03-22T01:34:20.877214+0000","last_undegraded":"2024-03-22T01:34:20.877214+0000","last_fullsized":"2024-03-22T01:34:20.877214+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:25:19.541443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.990444+0000","last_change":"2024-03-22T01:34:15.490405+0000","last_active":"2024-03-22T01:34:20.990444+0000","last_peered":"2024-03-22T01:34:20.990444+0000","last_clean":"2024-03-22T01:34:20.990444+0000","last_became_active":"2024-03-22T01:34:15.489921+0000","last_became_peered":"2024-03-22T01:34:15.489921+0000","last_unstale":"2024-03-22T01:34:20.990444+0000","last_undegraded":"2024-03-22T01:34:20.990444+0000","last_fullsized":"2024-03-22T01:34:20.990444+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:27:28.804649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.191289+0000","last_change":"2024-03-22T01:34:19.667651+0000","last_active":"2024-03-22T01:34:21.191289+0000","last_peered":"2024-03-22T01:34:21.191289+0000","last_clean":"2024-03-22T01:34:21.191289+0000","last_became_active":"2024-03-22T01:34:19.667197+0000","last_became_peered":"2024-03-22T01:34:19.667197+0000","last_unstale":"2024-03-22T01:34:21.191289+0000","last_undegraded":"2024-03-22T01:34:21.191289+0000","last_fullsized":"2024-03-22T01:34:21.191289+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:32:55.874098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"87'1","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.510898+0000","last_change":"2024-03-22T01:34:14.474988+0000","last_active":"2024-03-22T01:34:20.510898+0000","last_peered":"2024-03-22T01:34:20.510898+0000","last_clean":"2024-03-22T01:34:20.510898+0000","last_became_active":"2024-03-22T01:34:14.474651+0000","last_became_peered":"2024-03-22T01:34:14.474651+0000","last_unstale":"2024-03-22T01:34:20.510898+0000","last_undegraded":"2024-03-22T01:34:20.510898+0000","last_fullsized":"2024-03-22T01:34:20.510898+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:14:59.846727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.191516+0000","last_change":"2024-03-22T01:34:15.485209+0000","last_active":"2024-03-22T01:34:21.191516+0000","last_peered":"2024-03-22T01:34:21.191516+0000","last_clean":"2024-03-22T01:34:21.191516+0000","last_became_active":"2024-03-22T01:34:15.484898+0000","last_became_peered":"2024-03-22T01:34:15.484898+0000","last_unstale":"2024-03-22T01:34:21.191516+0000","last_undegraded":"2024-03-22T01:34:21.191516+0000","last_fullsized":"2024-03-22T01:34:21.191516+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:40:01.362735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.511340+0000","last_change":"2024-03-22T01:34:19.523959+0000","last_active":"2024-03-22T01:34:20.511340+0000","last_peered":"2024-03-22T01:34:20.511340+0000","last_clean":"2024-03-22T01:34:20.511340+0000","last_became_active":"2024-03-22T01:34:19.523556+0000","last_became_peered":"2024-03-22T01:34:19.523556+0000","last_unstale":"2024-03-22T01:34:20.511340+0000","last_undegraded":"2024-03-22T01:34:20.511340+0000","last_fullsized":"2024-03-22T01:34:20.511340+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:22:21.623042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"86'2","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.511115+0000","last_change":"2024-03-22T01:34:14.463434+0000","last_active":"2024-03-22T01:34:20.511115+0000","last_peered":"2024-03-22T01:34:20.511115+0000","last_clean":"2024-03-22T01:34:20.511115+0000","last_became_active":"2024-03-22T01:34:14.463075+0000","last_became_peered":"2024-03-22T01:34:14.463075+0000","last_unstale":"2024-03-22T01:34:20.511115+0000","last_undegraded":"2024-03-22T01:34:20.511115+0000","last_fullsized":"2024-03-22T01:34:20.511115+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:43:18.223766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.192191+0000","last_change":"2024-03-22T01:34:15.482695+0000","last_active":"2024-03-22T01:34:21.192191+0000","last_peered":"2024-03-22T01:34:21.192191+0000","last_clean":"2024-03-22T01:34:21.192191+0000","last_became_active":"2024-03-22T01:34:15.481881+0000","last_became_peered":"2024-03-22T01:34:15.481881+0000","last_unstale":"2024-03-22T01:34:21.192191+0000","last_undegraded":"2024-03-22T01:34:21.192191+0000","last_fullsized":"2024-03-22T01:34:21.192191+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:44:26.290714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.511053+0000","last_change":"2024-03-22T01:34:19.668252+0000","last_active":"2024-03-22T01:34:20.511053+0000","last_peered":"2024-03-22T01:34:20.511053+0000","last_clean":"2024-03-22T01:34:20.511053+0000","last_became_active":"2024-03-22T01:34:19.667830+0000","last_became_peered":"2024-03-22T01:34:19.667830+0000","last_unstale":"2024-03-22T01:34:20.511053+0000","last_undegraded":"2024-03-22T01:34:20.511053+0000","last_fullsized":"2024-03-22T01:34:20.511053+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:08:29.602899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.599766+0000","last_change":"2024-03-22T01:34:14.473020+0000","last_active":"2024-03-22T01:34:20.599766+0000","last_peered":"2024-03-22T01:34:20.599766+0000","last_clean":"2024-03-22T01:34:20.599766+0000","last_became_active":"2024-03-22T01:34:14.472689+0000","last_became_peered":"2024-03-22T01:34:14.472689+0000","last_unstale":"2024-03-22T01:34:20.599766+0000","last_undegraded":"2024-03-22T01:34:20.599766+0000","last_fullsized":"2024-03-22T01:34:20.599766+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:04:19.724685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.876686+0000","last_change":"2024-03-22T01:34:15.491613+0000","last_active":"2024-03-22T01:34:20.876686+0000","last_peered":"2024-03-22T01:34:20.876686+0000","last_clean":"2024-03-22T01:34:20.876686+0000","last_became_active":"2024-03-22T01:34:15.490922+0000","last_became_peered":"2024-03-22T01:34:15.490922+0000","last_unstale":"2024-03-22T01:34:20.876686+0000","last_undegraded":"2024-03-22T01:34:20.876686+0000","last_fullsized":"2024-03-22T01:34:20.876686+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:09:33.177646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.511039+0000","last_change":"2024-03-22T01:34:19.534537+0000","last_active":"2024-03-22T01:34:20.511039+0000","last_peered":"2024-03-22T01:34:20.511039+0000","last_clean":"2024-03-22T01:34:20.511039+0000","last_became_active":"2024-03-22T01:34:19.534038+0000","last_became_peered":"2024-03-22T01:34:19.534038+0000","last_unstale":"2024-03-22T01:34:20.511039+0000","last_undegraded":"2024-03-22T01:34:20.511039+0000","last_fullsized":"2024-03-22T01:34:20.511039+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:59:26.020000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.191607+0000","last_change":"2024-03-22T01:34:14.464724+0000","last_active":"2024-03-22T01:34:21.191607+0000","last_peered":"2024-03-22T01:34:21.191607+0000","last_clean":"2024-03-22T01:34:21.191607+0000","last_became_active":"2024-03-22T01:34:14.464056+0000","last_became_peered":"2024-03-22T01:34:14.464056+0000","last_unstale":"2024-03-22T01:34:21.191607+0000","last_undegraded":"2024-03-22T01:34:21.191607+0000","last_fullsized":"2024-03-22T01:34:21.191607+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:49:39.098126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.191340+0000","last_change":"2024-03-22T01:34:15.488934+0000","last_active":"2024-03-22T01:34:21.191340+0000","last_peered":"2024-03-22T01:34:21.191340+0000","last_clean":"2024-03-22T01:34:21.191340+0000","last_became_active":"2024-03-22T01:34:15.488585+0000","last_became_peered":"2024-03-22T01:34:15.488585+0000","last_unstale":"2024-03-22T01:34:21.191340+0000","last_undegraded":"2024-03-22T01:34:21.191340+0000","last_fullsized":"2024-03-22T01:34:21.191340+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:59:04.554614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.511140+0000","last_change":"2024-03-22T01:34:19.534096+0000","last_active":"2024-03-22T01:34:20.511140+0000","last_peered":"2024-03-22T01:34:20.511140+0000","last_clean":"2024-03-22T01:34:20.511140+0000","last_became_active":"2024-03-22T01:34:19.533645+0000","last_became_peered":"2024-03-22T01:34:19.533645+0000","last_unstale":"2024-03-22T01:34:20.511140+0000","last_undegraded":"2024-03-22T01:34:20.511140+0000","last_fullsized":"2024-03-22T01:34:20.511140+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:50:21.259146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.511279+0000","last_change":"2024-03-22T01:34:14.469529+0000","last_active":"2024-03-22T01:34:20.511279+0000","last_peered":"2024-03-22T01:34:20.511279+0000","last_clean":"2024-03-22T01:34:20.511279+0000","last_became_active":"2024-03-22T01:34:14.469204+0000","last_became_peered":"2024-03-22T01:34:14.469204+0000","last_unstale":"2024-03-22T01:34:20.511279+0000","last_undegraded":"2024-03-22T01:34:20.511279+0000","last_fullsized":"2024-03-22T01:34:20.511279+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:39:57.048537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.191758+0000","last_change":"2024-03-22T01:34:15.501324+0000","last_active":"2024-03-22T01:34:21.191758+0000","last_peered":"2024-03-22T01:34:21.191758+0000","last_clean":"2024-03-22T01:34:21.191758+0000","last_became_active":"2024-03-22T01:34:15.501015+0000","last_became_peered":"2024-03-22T01:34:15.501015+0000","last_unstale":"2024-03-22T01:34:21.191758+0000","last_undegraded":"2024-03-22T01:34:21.191758+0000","last_fullsized":"2024-03-22T01:34:21.191758+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:35:27.673019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.511064+0000","last_change":"2024-03-22T01:34:19.667806+0000","last_active":"2024-03-22T01:34:20.511064+0000","last_peered":"2024-03-22T01:34:20.511064+0000","last_clean":"2024-03-22T01:34:20.511064+0000","last_became_active":"2024-03-22T01:34:19.667249+0000","last_became_peered":"2024-03-22T01:34:19.667249+0000","last_unstale":"2024-03-22T01:34:20.511064+0000","last_undegraded":"2024-03-22T01:34:20.511064+0000","last_fullsized":"2024-03-22T01:34:20.511064+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:44:22.476882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"86'2","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.876800+0000","last_change":"2024-03-22T01:34:14.471800+0000","last_active":"2024-03-22T01:34:20.876800+0000","last_peered":"2024-03-22T01:34:20.876800+0000","last_clean":"2024-03-22T01:34:20.876800+0000","last_became_active":"2024-03-22T01:34:14.471422+0000","last_became_peered":"2024-03-22T01:34:14.471422+0000","last_unstale":"2024-03-22T01:34:20.876800+0000","last_undegraded":"2024-03-22T01:34:20.876800+0000","last_fullsized":"2024-03-22T01:34:20.876800+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:25:58.546331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.083985+0000","last_change":"2024-03-22T01:34:15.487348+0000","last_active":"2024-03-22T01:34:21.083985+0000","last_peered":"2024-03-22T01:34:21.083985+0000","last_clean":"2024-03-22T01:34:21.083985+0000","last_became_active":"2024-03-22T01:34:15.486788+0000","last_became_peered":"2024-03-22T01:34:15.486788+0000","last_unstale":"2024-03-22T01:34:21.083985+0000","last_undegraded":"2024-03-22T01:34:21.083985+0000","last_fullsized":"2024-03-22T01:34:21.083985+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:10:31.489021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.599883+0000","last_change":"2024-03-22T01:34:19.521490+0000","last_active":"2024-03-22T01:34:20.599883+0000","last_peered":"2024-03-22T01:34:20.599883+0000","last_clean":"2024-03-22T01:34:20.599883+0000","last_became_active":"2024-03-22T01:34:19.521047+0000","last_became_peered":"2024-03-22T01:34:19.521047+0000","last_unstale":"2024-03-22T01:34:20.599883+0000","last_undegraded":"2024-03-22T01:34:20.599883+0000","last_fullsized":"2024-03-22T01:34:20.599883+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:36:50.176756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.511888+0000","last_change":"2024-03-22T01:34:14.457729+0000","last_active":"2024-03-22T01:34:20.511888+0000","last_peered":"2024-03-22T01:34:20.511888+0000","last_clean":"2024-03-22T01:34:20.511888+0000","last_became_active":"2024-03-22T01:34:14.457336+0000","last_became_peered":"2024-03-22T01:34:14.457336+0000","last_unstale":"2024-03-22T01:34:20.511888+0000","last_undegraded":"2024-03-22T01:34:20.511888+0000","last_fullsized":"2024-03-22T01:34:20.511888+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:11:51.184034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.084390+0000","last_change":"2024-03-22T01:34:15.491683+0000","last_active":"2024-03-22T01:34:21.084390+0000","last_peered":"2024-03-22T01:34:21.084390+0000","last_clean":"2024-03-22T01:34:21.084390+0000","last_became_active":"2024-03-22T01:34:15.491030+0000","last_became_peered":"2024-03-22T01:34:15.491030+0000","last_unstale":"2024-03-22T01:34:21.084390+0000","last_undegraded":"2024-03-22T01:34:21.084390+0000","last_fullsized":"2024-03-22T01:34:21.084390+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:02:10.115971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.190882+0000","last_change":"2024-03-22T01:34:19.528608+0000","last_active":"2024-03-22T01:34:21.190882+0000","last_peered":"2024-03-22T01:34:21.190882+0000","last_clean":"2024-03-22T01:34:21.190882+0000","last_became_active":"2024-03-22T01:34:19.528173+0000","last_became_peered":"2024-03-22T01:34:19.528173+0000","last_unstale":"2024-03-22T01:34:21.190882+0000","last_undegraded":"2024-03-22T01:34:21.190882+0000","last_fullsized":"2024-03-22T01:34:21.190882+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:36:02.193187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"86'1","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.092410+0000","last_change":"2024-03-22T01:34:14.467755+0000","last_active":"2024-03-22T01:34:21.092410+0000","last_peered":"2024-03-22T01:34:21.092410+0000","last_clean":"2024-03-22T01:34:21.092410+0000","last_became_active":"2024-03-22T01:34:14.467426+0000","last_became_peered":"2024-03-22T01:34:14.467426+0000","last_unstale":"2024-03-22T01:34:21.092410+0000","last_undegraded":"2024-03-22T01:34:21.092410+0000","last_fullsized":"2024-03-22T01:34:21.092410+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:16:35.436872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.084257+0000","last_change":"2024-03-22T01:34:15.492437+0000","last_active":"2024-03-22T01:34:21.084257+0000","last_peered":"2024-03-22T01:34:21.084257+0000","last_clean":"2024-03-22T01:34:21.084257+0000","last_became_active":"2024-03-22T01:34:15.492110+0000","last_became_peered":"2024-03-22T01:34:15.492110+0000","last_unstale":"2024-03-22T01:34:21.084257+0000","last_undegraded":"2024-03-22T01:34:21.084257+0000","last_fullsized":"2024-03-22T01:34:21.084257+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:58:57.863065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.091668+0000","last_change":"2024-03-22T01:34:19.523631+0000","last_active":"2024-03-22T01:34:21.091668+0000","last_peered":"2024-03-22T01:34:21.091668+0000","last_clean":"2024-03-22T01:34:21.091668+0000","last_became_active":"2024-03-22T01:34:19.522968+0000","last_became_peered":"2024-03-22T01:34:19.522968+0000","last_unstale":"2024-03-22T01:34:21.091668+0000","last_undegraded":"2024-03-22T01:34:21.091668+0000","last_fullsized":"2024-03-22T01:34:21.091668+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:27:24.182886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.990402+0000","last_change":"2024-03-22T01:34:14.464113+0000","last_active":"2024-03-22T01:34:20.990402+0000","last_peered":"2024-03-22T01:34:20.990402+0000","last_clean":"2024-03-22T01:34:20.990402+0000","last_became_active":"2024-03-22T01:34:14.463389+0000","last_became_peered":"2024-03-22T01:34:14.463389+0000","last_unstale":"2024-03-22T01:34:20.990402+0000","last_undegraded":"2024-03-22T01:34:20.990402+0000","last_fullsized":"2024-03-22T01:34:20.990402+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:08:42.376753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.084319+0000","last_change":"2024-03-22T01:34:15.492941+0000","last_active":"2024-03-22T01:34:21.084319+0000","last_peered":"2024-03-22T01:34:21.084319+0000","last_clean":"2024-03-22T01:34:21.084319+0000","last_became_active":"2024-03-22T01:34:15.492598+0000","last_became_peered":"2024-03-22T01:34:15.492598+0000","last_unstale":"2024-03-22T01:34:21.084319+0000","last_undegraded":"2024-03-22T01:34:21.084319+0000","last_fullsized":"2024-03-22T01:34:21.084319+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:22:35.502404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.191449+0000","last_change":"2024-03-22T01:34:15.494548+0000","last_active":"2024-03-22T01:34:21.191449+0000","last_peered":"2024-03-22T01:34:21.191449+0000","last_clean":"2024-03-22T01:34:21.191449+0000","last_became_active":"2024-03-22T01:34:15.494228+0000","last_became_peered":"2024-03-22T01:34:15.494228+0000","last_unstale":"2024-03-22T01:34:21.191449+0000","last_undegraded":"2024-03-22T01:34:21.191449+0000","last_fullsized":"2024-03-22T01:34:21.191449+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:43:26.011030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.511375+0000","last_change":"2024-03-22T01:34:15.490843+0000","last_active":"2024-03-22T01:34:20.511375+0000","last_peered":"2024-03-22T01:34:20.511375+0000","last_clean":"2024-03-22T01:34:20.511375+0000","last_became_active":"2024-03-22T01:34:15.490474+0000","last_became_peered":"2024-03-22T01:34:15.490474+0000","last_unstale":"2024-03-22T01:34:20.511375+0000","last_undegraded":"2024-03-22T01:34:20.511375+0000","last_fullsized":"2024-03-22T01:34:20.511375+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:11:31.488132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.511255+0000","last_change":"2024-03-22T01:34:15.502213+0000","last_active":"2024-03-22T01:34:20.511255+0000","last_peered":"2024-03-22T01:34:20.511255+0000","last_clean":"2024-03-22T01:34:20.511255+0000","last_became_active":"2024-03-22T01:34:15.501464+0000","last_became_peered":"2024-03-22T01:34:15.501464+0000","last_unstale":"2024-03-22T01:34:20.511255+0000","last_undegraded":"2024-03-22T01:34:20.511255+0000","last_fullsized":"2024-03-22T01:34:20.511255+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:38:02.090365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.191071+0000","last_change":"2024-03-22T01:34:15.481282+0000","last_active":"2024-03-22T01:34:21.191071+0000","last_peered":"2024-03-22T01:34:21.191071+0000","last_clean":"2024-03-22T01:34:21.191071+0000","last_became_active":"2024-03-22T01:34:15.480465+0000","last_became_peered":"2024-03-22T01:34:15.480465+0000","last_unstale":"2024-03-22T01:34:21.191071+0000","last_undegraded":"2024-03-22T01:34:21.191071+0000","last_fullsized":"2024-03-22T01:34:21.191071+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:55:59.398543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.990394+0000","last_change":"2024-03-22T01:34:15.480910+0000","last_active":"2024-03-22T01:34:20.990394+0000","last_peered":"2024-03-22T01:34:20.990394+0000","last_clean":"2024-03-22T01:34:20.990394+0000","last_became_active":"2024-03-22T01:34:15.480601+0000","last_became_peered":"2024-03-22T01:34:15.480601+0000","last_unstale":"2024-03-22T01:34:20.990394+0000","last_undegraded":"2024-03-22T01:34:20.990394+0000","last_fullsized":"2024-03-22T01:34:20.990394+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:19:51.563598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.191743+0000","last_change":"2024-03-22T01:34:15.490859+0000","last_active":"2024-03-22T01:34:21.191743+0000","last_peered":"2024-03-22T01:34:21.191743+0000","last_clean":"2024-03-22T01:34:21.191743+0000","last_became_active":"2024-03-22T01:34:15.490337+0000","last_became_peered":"2024-03-22T01:34:15.490337+0000","last_unstale":"2024-03-22T01:34:21.191743+0000","last_undegraded":"2024-03-22T01:34:21.191743+0000","last_fullsized":"2024-03-22T01:34:21.191743+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:46:07.072015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.512014+0000","last_change":"2024-03-22T01:34:15.495531+0000","last_active":"2024-03-22T01:34:20.512014+0000","last_peered":"2024-03-22T01:34:20.512014+0000","last_clean":"2024-03-22T01:34:20.512014+0000","last_became_active":"2024-03-22T01:34:15.495216+0000","last_became_peered":"2024-03-22T01:34:15.495216+0000","last_unstale":"2024-03-22T01:34:20.512014+0000","last_undegraded":"2024-03-22T01:34:20.512014+0000","last_fullsized":"2024-03-22T01:34:20.512014+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:39:22.575926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.191691+0000","last_change":"2024-03-22T01:34:15.499430+0000","last_active":"2024-03-22T01:34:21.191691+0000","last_peered":"2024-03-22T01:34:21.191691+0000","last_clean":"2024-03-22T01:34:21.191691+0000","last_became_active":"2024-03-22T01:34:15.499114+0000","last_became_peered":"2024-03-22T01:34:15.499114+0000","last_unstale":"2024-03-22T01:34:21.191691+0000","last_undegraded":"2024-03-22T01:34:21.191691+0000","last_fullsized":"2024-03-22T01:34:21.191691+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:23:01.781963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.091940+0000","last_change":"2024-03-22T01:34:15.485449+0000","last_active":"2024-03-22T01:34:21.091940+0000","last_peered":"2024-03-22T01:34:21.091940+0000","last_clean":"2024-03-22T01:34:21.091940+0000","last_became_active":"2024-03-22T01:34:15.485084+0000","last_became_peered":"2024-03-22T01:34:15.485084+0000","last_unstale":"2024-03-22T01:34:21.091940+0000","last_undegraded":"2024-03-22T01:34:21.091940+0000","last_fullsized":"2024-03-22T01:34:21.091940+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:53:25.632719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.191699+0000","last_change":"2024-03-22T01:34:15.495521+0000","last_active":"2024-03-22T01:34:21.191699+0000","last_peered":"2024-03-22T01:34:21.191699+0000","last_clean":"2024-03-22T01:34:21.191699+0000","last_became_active":"2024-03-22T01:34:15.495218+0000","last_became_peered":"2024-03-22T01:34:15.495218+0000","last_unstale":"2024-03-22T01:34:21.191699+0000","last_undegraded":"2024-03-22T01:34:21.191699+0000","last_fullsized":"2024-03-22T01:34:21.191699+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:04:32.569934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.599930+0000","last_change":"2024-03-22T01:34:15.489213+0000","last_active":"2024-03-22T01:34:20.599930+0000","last_peered":"2024-03-22T01:34:20.599930+0000","last_clean":"2024-03-22T01:34:20.599930+0000","last_became_active":"2024-03-22T01:34:15.488887+0000","last_became_peered":"2024-03-22T01:34:15.488887+0000","last_unstale":"2024-03-22T01:34:20.599930+0000","last_undegraded":"2024-03-22T01:34:20.599930+0000","last_fullsized":"2024-03-22T01:34:20.599930+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:53:39.983144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.990185+0000","last_change":"2024-03-22T01:34:15.485632+0000","last_active":"2024-03-22T01:34:20.990185+0000","last_peered":"2024-03-22T01:34:20.990185+0000","last_clean":"2024-03-22T01:34:20.990185+0000","last_became_active":"2024-03-22T01:34:15.485341+0000","last_became_peered":"2024-03-22T01:34:15.485341+0000","last_unstale":"2024-03-22T01:34:20.990185+0000","last_undegraded":"2024-03-22T01:34:20.990185+0000","last_fullsized":"2024-03-22T01:34:20.990185+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:33:00.248555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.191597+0000","last_change":"2024-03-22T01:34:15.483593+0000","last_active":"2024-03-22T01:34:21.191597+0000","last_peered":"2024-03-22T01:34:21.191597+0000","last_clean":"2024-03-22T01:34:21.191597+0000","last_became_active":"2024-03-22T01:34:15.483201+0000","last_became_peered":"2024-03-22T01:34:15.483201+0000","last_unstale":"2024-03-22T01:34:21.191597+0000","last_undegraded":"2024-03-22T01:34:21.191597+0000","last_fullsized":"2024-03-22T01:34:21.191597+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:03:10.955120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":5,"num_read_kb":7,"num_write":49,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":13920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318290,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1003520,"data_stored":486921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579927,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36920,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708232,"statfs":{"total":95995035648,"available":95957229568,"internally_reserved":0,"allocated":1622016,"data_stored":1081325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776156,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1007616,"data_stored":487503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037792,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1011712,"data_stored":487525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299430,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28592,"kb_used_data":1564,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716560,"statfs":{"total":95995035648,"available":95965757440,"internally_reserved":0,"allocated":1601536,"data_stored":1077871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462954,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1003520,"data_stored":486921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574915,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":490763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738440,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1007616,"data_stored":487011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411390,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1019904,"data_stored":490797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149754,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1011712,"data_stored":487025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888117,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1007616,"data_stored":486943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724592,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36920,"kb_used_data":1572,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708232,"statfs":{"total":95995035648,"available":95957229568,"internally_reserved":0,"allocated":1609728,"data_stored":1077471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-22T01:34:43.217 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-22T01:34:43.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.354+0000 7f49cc04e640 1 -- 172.21.15.72:0/522116822 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49c4102640 msgr2=0x7f49c4104a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:43.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.354+0000 7f49cc04e640 1 --2- 172.21.15.72:0/522116822 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49c4102640 0x7f49c4104a30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f49c0009900 tx=0x7f49c002bc30 comp rx=0 tx=0).stop 2024-03-22T01:34:43.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.354+0000 7f49cc04e640 1 -- 172.21.15.72:0/522116822 shutdown_connections 2024-03-22T01:34:43.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.354+0000 7f49cc04e640 1 --2- 172.21.15.72:0/522116822 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f49c41078a0 0x7f49c4109cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:43.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.354+0000 7f49cc04e640 1 --2- 172.21.15.72:0/522116822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49c4104f70 0x7f49c4107360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:43.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.354+0000 7f49cc04e640 1 --2- 172.21.15.72:0/522116822 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49c4102640 0x7f49c4104a30 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:43.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.354+0000 7f49cc04e640 1 -- 172.21.15.72:0/522116822 >> 172.21.15.72:0/522116822 conn(0x7f49c40fc050 msgr2=0x7f49c40fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:43.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.355+0000 7f49cc04e640 1 -- 172.21.15.72:0/522116822 shutdown_connections 2024-03-22T01:34:43.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.355+0000 7f49cc04e640 1 -- 172.21.15.72:0/522116822 wait complete. 2024-03-22T01:34:43.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.355+0000 7f49cc04e640 1 Processor -- start 2024-03-22T01:34:43.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.355+0000 7f49cc04e640 1 -- start start 2024-03-22T01:34:43.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.355+0000 7f49cc04e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49c4102640 0x7f49c4197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:43.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.355+0000 7f49cc04e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f49c4104f70 0x7f49c4198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:43.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.355+0000 7f49cc04e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49c41078a0 0x7f49c419f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:43.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.355+0000 7f49cc04e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f49c410d840 con 0x7f49c4104f70 2024-03-22T01:34:43.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.355+0000 7f49cc04e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f49c410d6c0 con 0x7f49c41078a0 2024-03-22T01:34:43.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.355+0000 7f49cc04e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f49c410d9c0 con 0x7f49c4102640 2024-03-22T01:34:43.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.355+0000 7f49c95e9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f49c4104f70 0x7f49c4198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:43.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.355+0000 7f49c95e9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f49c4104f70 0x7f49c4198420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37482/0 (socket says 172.21.15.72:37482) 2024-03-22T01:34:43.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.355+0000 7f49c95e9640 1 -- 172.21.15.72:0/2013023717 learned_addr learned my addr 172.21.15.72:0/2013023717 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:34:43.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.355+0000 7f49ca5eb640 1 --2- 172.21.15.72:0/2013023717 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49c41078a0 0x7f49c419f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:43.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.356+0000 7f49c95e9640 1 -- 172.21.15.72:0/2013023717 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49c4102640 msgr2=0x7f49c4197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:43.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.356+0000 7f49c95e9640 1 --2- 172.21.15.72:0/2013023717 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49c4102640 0x7f49c4197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:43.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.356+0000 7f49c95e9640 1 -- 172.21.15.72:0/2013023717 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49c41078a0 msgr2=0x7f49c419f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:43.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.356+0000 7f49c95e9640 1 --2- 172.21.15.72:0/2013023717 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49c41078a0 0x7f49c419f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:43.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.356+0000 7f49c95e9640 1 -- 172.21.15.72:0/2013023717 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49c419fb00 con 0x7f49c4104f70 2024-03-22T01:34:43.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.356+0000 7f49c95e9640 1 --2- 172.21.15.72:0/2013023717 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f49c4104f70 0x7f49c4198420 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f49b400d770 tx=0x7f49b400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:43.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.356+0000 7f49baffd640 1 -- 172.21.15.72:0/2013023717 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49b4017070 con 0x7f49c4104f70 2024-03-22T01:34:43.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.356+0000 7f49cc04e640 1 -- 172.21.15.72:0/2013023717 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49c419fdf0 con 0x7f49c4104f70 2024-03-22T01:34:43.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.356+0000 7f49baffd640 1 -- 172.21.15.72:0/2013023717 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49b4009c30 con 0x7f49c4104f70 2024-03-22T01:34:43.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.356+0000 7f49baffd640 1 -- 172.21.15.72:0/2013023717 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49b400f620 con 0x7f49c4104f70 2024-03-22T01:34:43.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.356+0000 7f49cc04e640 1 -- 172.21.15.72:0/2013023717 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49c41a02b0 con 0x7f49c4104f70 2024-03-22T01:34:43.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.357+0000 7f49cc04e640 1 -- 172.21.15.72:0/2013023717 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49c40fdb40 con 0x7f49c4104f70 2024-03-22T01:34:43.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.358+0000 7f49baffd640 1 -- 172.21.15.72:0/2013023717 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7f49b401a070 con 0x7f49c4104f70 2024-03-22T01:34:43.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.358+0000 7f49baffd640 1 --2- 172.21.15.72:0/2013023717 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f49b00a2fb0 0x7f49b00a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:43.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.358+0000 7f49c9dea640 1 --2- 172.21.15.72:0/2013023717 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f49b00a2fb0 0x7f49b00a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:43.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.359+0000 7f49baffd640 1 -- 172.21.15.72:0/2013023717 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f49b40cf650 con 0x7f49c4104f70 2024-03-22T01:34:43.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.359+0000 7f49c9dea640 1 --2- 172.21.15.72:0/2013023717 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f49b00a2fb0 0x7f49b00a5470 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f49c0005c20 tx=0x7f49c0000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:43.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.363+0000 7f49baffd640 1 -- 172.21.15.72:0/2013023717 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f49b4009de0 con 0x7f49c4104f70 2024-03-22T01:34:43.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.538+0000 7f49cc04e640 1 -- 172.21.15.72:0/2013023717 --> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f49c4061fd0 con 0x7f49b00a2fb0 2024-03-22T01:34:43.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.547+0000 7f49baffd640 1 -- 172.21.15.72:0/2013023717 <== mgr.14152 v2:172.21.15.72:6800/401403200 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340232 (secure 0 0 0) 0x7f49c4061fd0 con 0x7f49b00a2fb0 2024-03-22T01:34:43.550 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:34:43.552 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-03-22T01:34:43.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.550+0000 7f49cc04e640 1 -- 172.21.15.72:0/2013023717 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f49b00a2fb0 msgr2=0x7f49b00a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:43.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.550+0000 7f49cc04e640 1 --2- 172.21.15.72:0/2013023717 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f49b00a2fb0 0x7f49b00a5470 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f49c0005c20 tx=0x7f49c0000bc0 comp rx=0 tx=0).stop 2024-03-22T01:34:43.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.550+0000 7f49cc04e640 1 -- 172.21.15.72:0/2013023717 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f49c4104f70 msgr2=0x7f49c4198420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:43.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.550+0000 7f49cc04e640 1 --2- 172.21.15.72:0/2013023717 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f49c4104f70 0x7f49c4198420 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f49b400d770 tx=0x7f49b400dc30 comp rx=0 tx=0).stop 2024-03-22T01:34:43.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.550+0000 7f49cc04e640 1 -- 172.21.15.72:0/2013023717 shutdown_connections 2024-03-22T01:34:43.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.550+0000 7f49cc04e640 1 --2- 172.21.15.72:0/2013023717 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f49b00a2fb0 0x7f49b00a5470 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:43.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.550+0000 7f49cc04e640 1 --2- 172.21.15.72:0/2013023717 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49c41078a0 0x7f49c419f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:43.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.550+0000 7f49cc04e640 1 --2- 172.21.15.72:0/2013023717 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f49c4104f70 0x7f49c4198420 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:43.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.550+0000 7f49cc04e640 1 --2- 172.21.15.72:0/2013023717 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49c4102640 0x7f49c4197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:43.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.550+0000 7f49cc04e640 1 -- 172.21.15.72:0/2013023717 >> 172.21.15.72:0/2013023717 conn(0x7f49c40fc050 msgr2=0x7f49c4102f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:43.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.550+0000 7f49cc04e640 1 -- 172.21.15.72:0/2013023717 shutdown_connections 2024-03-22T01:34:43.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:43.550+0000 7f49cc04e640 1 -- 172.21.15.72:0/2013023717 wait complete. 2024-03-22T01:34:43.560 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":305,"stamp":"2024-03-22T01:34:43.428218+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595125,"num_objects":25,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":146,"num_read_kb":131,"num_write":235,"num_write_kb":2960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":183,"ondisk_log_size":183,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":367012,"kb_used_data":13644,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124574812,"statfs":{"total":1151940427776,"available":1151564607488,"internally_reserved":0,"allocated":13971456,"data_stored":7676993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":361674042},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":258,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":12,"num_write":7,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003444"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.191830+0000","last_change":"2024-03-22T01:34:15.487496+0000","last_active":"2024-03-22T01:34:21.191830+0000","last_peered":"2024-03-22T01:34:21.191830+0000","last_clean":"2024-03-22T01:34:21.191830+0000","last_became_active":"2024-03-22T01:34:15.486700+0000","last_became_peered":"2024-03-22T01:34:15.486700+0000","last_unstale":"2024-03-22T01:34:21.191830+0000","last_undegraded":"2024-03-22T01:34:21.191830+0000","last_fullsized":"2024-03-22T01:34:21.191830+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:03:19.146811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.192092+0000","last_change":"2024-03-22T01:34:15.489791+0000","last_active":"2024-03-22T01:34:21.192092+0000","last_peered":"2024-03-22T01:34:21.192092+0000","last_clean":"2024-03-22T01:34:21.192092+0000","last_became_active":"2024-03-22T01:34:15.489493+0000","last_became_peered":"2024-03-22T01:34:15.489493+0000","last_unstale":"2024-03-22T01:34:21.192092+0000","last_undegraded":"2024-03-22T01:34:21.192092+0000","last_fullsized":"2024-03-22T01:34:21.192092+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:45:05.060454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.191049+0000","last_change":"2024-03-22T01:34:15.491567+0000","last_active":"2024-03-22T01:34:21.191049+0000","last_peered":"2024-03-22T01:34:21.191049+0000","last_clean":"2024-03-22T01:34:21.191049+0000","last_became_active":"2024-03-22T01:34:15.491275+0000","last_became_peered":"2024-03-22T01:34:15.491275+0000","last_unstale":"2024-03-22T01:34:21.191049+0000","last_undegraded":"2024-03-22T01:34:21.191049+0000","last_fullsized":"2024-03-22T01:34:21.191049+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:26:36.630811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.511199+0000","last_change":"2024-03-22T01:34:15.488579+0000","last_active":"2024-03-22T01:34:20.511199+0000","last_peered":"2024-03-22T01:34:20.511199+0000","last_clean":"2024-03-22T01:34:20.511199+0000","last_became_active":"2024-03-22T01:34:15.488262+0000","last_became_peered":"2024-03-22T01:34:15.488262+0000","last_unstale":"2024-03-22T01:34:20.511199+0000","last_undegraded":"2024-03-22T01:34:20.511199+0000","last_fullsized":"2024-03-22T01:34:20.511199+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:58:21.788420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.876769+0000","last_change":"2024-03-22T01:34:15.485744+0000","last_active":"2024-03-22T01:34:20.876769+0000","last_peered":"2024-03-22T01:34:20.876769+0000","last_clean":"2024-03-22T01:34:20.876769+0000","last_became_active":"2024-03-22T01:34:15.485378+0000","last_became_peered":"2024-03-22T01:34:15.485378+0000","last_unstale":"2024-03-22T01:34:20.876769+0000","last_undegraded":"2024-03-22T01:34:20.876769+0000","last_fullsized":"2024-03-22T01:34:20.876769+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:06:40.979475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.191301+0000","last_change":"2024-03-22T01:34:15.495051+0000","last_active":"2024-03-22T01:34:21.191301+0000","last_peered":"2024-03-22T01:34:21.191301+0000","last_clean":"2024-03-22T01:34:21.191301+0000","last_became_active":"2024-03-22T01:34:15.494691+0000","last_became_peered":"2024-03-22T01:34:15.494691+0000","last_unstale":"2024-03-22T01:34:21.191301+0000","last_undegraded":"2024-03-22T01:34:21.191301+0000","last_fullsized":"2024-03-22T01:34:21.191301+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T01:44:28.336545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.083941+0000","last_change":"2024-03-22T01:34:15.493911+0000","last_active":"2024-03-22T01:34:21.083941+0000","last_peered":"2024-03-22T01:34:21.083941+0000","last_clean":"2024-03-22T01:34:21.083941+0000","last_became_active":"2024-03-22T01:34:15.493586+0000","last_became_peered":"2024-03-22T01:34:15.493586+0000","last_unstale":"2024-03-22T01:34:21.083941+0000","last_undegraded":"2024-03-22T01:34:21.083941+0000","last_fullsized":"2024-03-22T01:34:21.083941+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:22:41.391855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.191086+0000","last_change":"2024-03-22T01:34:15.481410+0000","last_active":"2024-03-22T01:34:21.191086+0000","last_peered":"2024-03-22T01:34:21.191086+0000","last_clean":"2024-03-22T01:34:21.191086+0000","last_became_active":"2024-03-22T01:34:15.481109+0000","last_became_peered":"2024-03-22T01:34:15.481109+0000","last_unstale":"2024-03-22T01:34:21.191086+0000","last_undegraded":"2024-03-22T01:34:21.191086+0000","last_fullsized":"2024-03-22T01:34:21.191086+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:27:10.004584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.600278+0000","last_change":"2024-03-22T01:34:15.485173+0000","last_active":"2024-03-22T01:34:20.600278+0000","last_peered":"2024-03-22T01:34:20.600278+0000","last_clean":"2024-03-22T01:34:20.600278+0000","last_became_active":"2024-03-22T01:34:15.484812+0000","last_became_peered":"2024-03-22T01:34:15.484812+0000","last_unstale":"2024-03-22T01:34:20.600278+0000","last_undegraded":"2024-03-22T01:34:20.600278+0000","last_fullsized":"2024-03-22T01:34:20.600278+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:44:06.383568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.191572+0000","last_change":"2024-03-22T01:34:15.497964+0000","last_active":"2024-03-22T01:34:21.191572+0000","last_peered":"2024-03-22T01:34:21.191572+0000","last_clean":"2024-03-22T01:34:21.191572+0000","last_became_active":"2024-03-22T01:34:15.496887+0000","last_became_peered":"2024-03-22T01:34:15.496887+0000","last_unstale":"2024-03-22T01:34:21.191572+0000","last_undegraded":"2024-03-22T01:34:21.191572+0000","last_fullsized":"2024-03-22T01:34:21.191572+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:13:40.420667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.084353+0000","last_change":"2024-03-22T01:34:15.496310+0000","last_active":"2024-03-22T01:34:21.084353+0000","last_peered":"2024-03-22T01:34:21.084353+0000","last_clean":"2024-03-22T01:34:21.084353+0000","last_became_active":"2024-03-22T01:34:15.494196+0000","last_became_peered":"2024-03-22T01:34:15.494196+0000","last_unstale":"2024-03-22T01:34:21.084353+0000","last_undegraded":"2024-03-22T01:34:21.084353+0000","last_fullsized":"2024-03-22T01:34:21.084353+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:33:02.798310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.510976+0000","last_change":"2024-03-22T01:34:15.495380+0000","last_active":"2024-03-22T01:34:20.510976+0000","last_peered":"2024-03-22T01:34:20.510976+0000","last_clean":"2024-03-22T01:34:20.510976+0000","last_became_active":"2024-03-22T01:34:15.495063+0000","last_became_peered":"2024-03-22T01:34:15.495063+0000","last_unstale":"2024-03-22T01:34:20.510976+0000","last_undegraded":"2024-03-22T01:34:20.510976+0000","last_fullsized":"2024-03-22T01:34:20.510976+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:46:29.497329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.990160+0000","last_change":"2024-03-22T01:34:15.497533+0000","last_active":"2024-03-22T01:34:20.990160+0000","last_peered":"2024-03-22T01:34:20.990160+0000","last_clean":"2024-03-22T01:34:20.990160+0000","last_became_active":"2024-03-22T01:34:15.497223+0000","last_became_peered":"2024-03-22T01:34:15.497223+0000","last_unstale":"2024-03-22T01:34:20.990160+0000","last_undegraded":"2024-03-22T01:34:20.990160+0000","last_fullsized":"2024-03-22T01:34:20.990160+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:17:49.202842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.190865+0000","last_change":"2024-03-22T01:34:15.491317+0000","last_active":"2024-03-22T01:34:21.190865+0000","last_peered":"2024-03-22T01:34:21.190865+0000","last_clean":"2024-03-22T01:34:21.190865+0000","last_became_active":"2024-03-22T01:34:15.490982+0000","last_became_peered":"2024-03-22T01:34:15.490982+0000","last_unstale":"2024-03-22T01:34:21.190865+0000","last_undegraded":"2024-03-22T01:34:21.190865+0000","last_fullsized":"2024-03-22T01:34:21.190865+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:05:39.889964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.190892+0000","last_change":"2024-03-22T01:34:15.488502+0000","last_active":"2024-03-22T01:34:21.190892+0000","last_peered":"2024-03-22T01:34:21.190892+0000","last_clean":"2024-03-22T01:34:21.190892+0000","last_became_active":"2024-03-22T01:34:15.488200+0000","last_became_peered":"2024-03-22T01:34:15.488200+0000","last_unstale":"2024-03-22T01:34:21.190892+0000","last_undegraded":"2024-03-22T01:34:21.190892+0000","last_fullsized":"2024-03-22T01:34:21.190892+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:52:59.991591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.599708+0000","last_change":"2024-03-22T01:34:15.489991+0000","last_active":"2024-03-22T01:34:20.599708+0000","last_peered":"2024-03-22T01:34:20.599708+0000","last_clean":"2024-03-22T01:34:20.599708+0000","last_became_active":"2024-03-22T01:34:15.489696+0000","last_became_peered":"2024-03-22T01:34:15.489696+0000","last_unstale":"2024-03-22T01:34:20.599708+0000","last_undegraded":"2024-03-22T01:34:20.599708+0000","last_fullsized":"2024-03-22T01:34:20.599708+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:10:45.316422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.191721+0000","last_change":"2024-03-22T01:34:15.488011+0000","last_active":"2024-03-22T01:34:21.191721+0000","last_peered":"2024-03-22T01:34:21.191721+0000","last_clean":"2024-03-22T01:34:21.191721+0000","last_became_active":"2024-03-22T01:34:15.487282+0000","last_became_peered":"2024-03-22T01:34:15.487282+0000","last_unstale":"2024-03-22T01:34:21.191721+0000","last_undegraded":"2024-03-22T01:34:21.191721+0000","last_fullsized":"2024-03-22T01:34:21.191721+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:16:21.396273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.510761+0000","last_change":"2024-03-22T01:34:15.488046+0000","last_active":"2024-03-22T01:34:20.510761+0000","last_peered":"2024-03-22T01:34:20.510761+0000","last_clean":"2024-03-22T01:34:20.510761+0000","last_became_active":"2024-03-22T01:34:15.487731+0000","last_became_peered":"2024-03-22T01:34:15.487731+0000","last_unstale":"2024-03-22T01:34:20.510761+0000","last_undegraded":"2024-03-22T01:34:20.510761+0000","last_fullsized":"2024-03-22T01:34:20.510761+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:44:16.931544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.512006+0000","last_change":"2024-03-22T01:34:15.503737+0000","last_active":"2024-03-22T01:34:20.512006+0000","last_peered":"2024-03-22T01:34:20.512006+0000","last_clean":"2024-03-22T01:34:20.512006+0000","last_became_active":"2024-03-22T01:34:15.502169+0000","last_became_peered":"2024-03-22T01:34:15.502169+0000","last_unstale":"2024-03-22T01:34:20.512006+0000","last_undegraded":"2024-03-22T01:34:20.512006+0000","last_fullsized":"2024-03-22T01:34:20.512006+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:05:53.027318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.511598+0000","last_change":"2024-03-22T01:34:19.527964+0000","last_active":"2024-03-22T01:34:20.511598+0000","last_peered":"2024-03-22T01:34:20.511598+0000","last_clean":"2024-03-22T01:34:20.511598+0000","last_became_active":"2024-03-22T01:34:19.527526+0000","last_became_peered":"2024-03-22T01:34:19.527526+0000","last_unstale":"2024-03-22T01:34:20.511598+0000","last_undegraded":"2024-03-22T01:34:20.511598+0000","last_fullsized":"2024-03-22T01:34:20.511598+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:35:39.404439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'4","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.511118+0000","last_change":"2024-03-22T01:34:14.473065+0000","last_active":"2024-03-22T01:34:20.511118+0000","last_peered":"2024-03-22T01:34:20.511118+0000","last_clean":"2024-03-22T01:34:20.511118+0000","last_became_active":"2024-03-22T01:34:14.472653+0000","last_became_peered":"2024-03-22T01:34:14.472653+0000","last_unstale":"2024-03-22T01:34:20.511118+0000","last_undegraded":"2024-03-22T01:34:20.511118+0000","last_fullsized":"2024-03-22T01:34:20.511118+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:11:09.525438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.191967+0000","last_change":"2024-03-22T01:34:15.497818+0000","last_active":"2024-03-22T01:34:21.191967+0000","last_peered":"2024-03-22T01:34:21.191967+0000","last_clean":"2024-03-22T01:34:21.191967+0000","last_became_active":"2024-03-22T01:34:15.497488+0000","last_became_peered":"2024-03-22T01:34:15.497488+0000","last_unstale":"2024-03-22T01:34:21.191967+0000","last_undegraded":"2024-03-22T01:34:21.191967+0000","last_fullsized":"2024-03-22T01:34:21.191967+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:21:09.340557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.510720+0000","last_change":"2024-03-22T01:34:19.528435+0000","last_active":"2024-03-22T01:34:20.510720+0000","last_peered":"2024-03-22T01:34:20.510720+0000","last_clean":"2024-03-22T01:34:20.510720+0000","last_became_active":"2024-03-22T01:34:19.528044+0000","last_became_peered":"2024-03-22T01:34:19.528044+0000","last_unstale":"2024-03-22T01:34:20.510720+0000","last_undegraded":"2024-03-22T01:34:20.510720+0000","last_fullsized":"2024-03-22T01:34:20.510720+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:52:05.682173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"86'1","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.192154+0000","last_change":"2024-03-22T01:34:14.465817+0000","last_active":"2024-03-22T01:34:21.192154+0000","last_peered":"2024-03-22T01:34:21.192154+0000","last_clean":"2024-03-22T01:34:21.192154+0000","last_became_active":"2024-03-22T01:34:14.464338+0000","last_became_peered":"2024-03-22T01:34:14.464338+0000","last_unstale":"2024-03-22T01:34:21.192154+0000","last_undegraded":"2024-03-22T01:34:21.192154+0000","last_fullsized":"2024-03-22T01:34:21.192154+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:04:47.457092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.191311+0000","last_change":"2024-03-22T01:34:15.481988+0000","last_active":"2024-03-22T01:34:21.191311+0000","last_peered":"2024-03-22T01:34:21.191311+0000","last_clean":"2024-03-22T01:34:21.191311+0000","last_became_active":"2024-03-22T01:34:15.481676+0000","last_became_peered":"2024-03-22T01:34:15.481676+0000","last_unstale":"2024-03-22T01:34:21.191311+0000","last_undegraded":"2024-03-22T01:34:21.191311+0000","last_fullsized":"2024-03-22T01:34:21.191311+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:37:54.609731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.511003+0000","last_change":"2024-03-22T01:34:19.528349+0000","last_active":"2024-03-22T01:34:20.511003+0000","last_peered":"2024-03-22T01:34:20.511003+0000","last_clean":"2024-03-22T01:34:20.511003+0000","last_became_active":"2024-03-22T01:34:19.527937+0000","last_became_peered":"2024-03-22T01:34:19.527937+0000","last_unstale":"2024-03-22T01:34:20.511003+0000","last_undegraded":"2024-03-22T01:34:20.511003+0000","last_fullsized":"2024-03-22T01:34:20.511003+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:48:34.242104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"86'2","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.877331+0000","last_change":"2024-03-22T01:34:14.472905+0000","last_active":"2024-03-22T01:34:20.877331+0000","last_peered":"2024-03-22T01:34:20.877331+0000","last_clean":"2024-03-22T01:34:20.877331+0000","last_became_active":"2024-03-22T01:34:14.472560+0000","last_became_peered":"2024-03-22T01:34:14.472560+0000","last_unstale":"2024-03-22T01:34:20.877331+0000","last_undegraded":"2024-03-22T01:34:20.877331+0000","last_fullsized":"2024-03-22T01:34:20.877331+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:19:41.363630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.599949+0000","last_change":"2024-03-22T01:34:15.491345+0000","last_active":"2024-03-22T01:34:20.599949+0000","last_peered":"2024-03-22T01:34:20.599949+0000","last_clean":"2024-03-22T01:34:20.599949+0000","last_became_active":"2024-03-22T01:34:15.490833+0000","last_became_peered":"2024-03-22T01:34:15.490833+0000","last_unstale":"2024-03-22T01:34:20.599949+0000","last_undegraded":"2024-03-22T01:34:20.599949+0000","last_fullsized":"2024-03-22T01:34:20.599949+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:30:43.950679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.510730+0000","last_change":"2024-03-22T01:34:19.528351+0000","last_active":"2024-03-22T01:34:20.510730+0000","last_peered":"2024-03-22T01:34:20.510730+0000","last_clean":"2024-03-22T01:34:20.510730+0000","last_became_active":"2024-03-22T01:34:19.527939+0000","last_became_peered":"2024-03-22T01:34:19.527939+0000","last_unstale":"2024-03-22T01:34:20.510730+0000","last_undegraded":"2024-03-22T01:34:20.510730+0000","last_fullsized":"2024-03-22T01:34:20.510730+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:30:21.055848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.877161+0000","last_change":"2024-03-22T01:34:14.467743+0000","last_active":"2024-03-22T01:34:20.877161+0000","last_peered":"2024-03-22T01:34:20.877161+0000","last_clean":"2024-03-22T01:34:20.877161+0000","last_became_active":"2024-03-22T01:34:14.467421+0000","last_became_peered":"2024-03-22T01:34:14.467421+0000","last_unstale":"2024-03-22T01:34:20.877161+0000","last_undegraded":"2024-03-22T01:34:20.877161+0000","last_fullsized":"2024-03-22T01:34:20.877161+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:26:41.770050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.092335+0000","last_change":"2024-03-22T01:34:15.479530+0000","last_active":"2024-03-22T01:34:21.092335+0000","last_peered":"2024-03-22T01:34:21.092335+0000","last_clean":"2024-03-22T01:34:21.092335+0000","last_became_active":"2024-03-22T01:34:15.479198+0000","last_became_peered":"2024-03-22T01:34:15.479198+0000","last_unstale":"2024-03-22T01:34:21.092335+0000","last_undegraded":"2024-03-22T01:34:21.092335+0000","last_fullsized":"2024-03-22T01:34:21.092335+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:28:51.624369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.511923+0000","last_change":"2024-03-22T01:34:19.524486+0000","last_active":"2024-03-22T01:34:20.511923+0000","last_peered":"2024-03-22T01:34:20.511923+0000","last_clean":"2024-03-22T01:34:20.511923+0000","last_became_active":"2024-03-22T01:34:19.524030+0000","last_became_peered":"2024-03-22T01:34:19.524030+0000","last_unstale":"2024-03-22T01:34:20.511923+0000","last_undegraded":"2024-03-22T01:34:20.511923+0000","last_fullsized":"2024-03-22T01:34:20.511923+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:31:10.157431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"86'1","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.091981+0000","last_change":"2024-03-22T01:34:14.469191+0000","last_active":"2024-03-22T01:34:21.091981+0000","last_peered":"2024-03-22T01:34:21.091981+0000","last_clean":"2024-03-22T01:34:21.091981+0000","last_became_active":"2024-03-22T01:34:14.468842+0000","last_became_peered":"2024-03-22T01:34:14.468842+0000","last_unstale":"2024-03-22T01:34:21.091981+0000","last_undegraded":"2024-03-22T01:34:21.091981+0000","last_fullsized":"2024-03-22T01:34:21.091981+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:29:56.566256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.876794+0000","last_change":"2024-03-22T01:34:15.483736+0000","last_active":"2024-03-22T01:34:20.876794+0000","last_peered":"2024-03-22T01:34:20.876794+0000","last_clean":"2024-03-22T01:34:20.876794+0000","last_became_active":"2024-03-22T01:34:15.483401+0000","last_became_peered":"2024-03-22T01:34:15.483401+0000","last_unstale":"2024-03-22T01:34:20.876794+0000","last_undegraded":"2024-03-22T01:34:20.876794+0000","last_fullsized":"2024-03-22T01:34:20.876794+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:26:51.974932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.511890+0000","last_change":"2024-03-22T01:34:19.534542+0000","last_active":"2024-03-22T01:34:20.511890+0000","last_peered":"2024-03-22T01:34:20.511890+0000","last_clean":"2024-03-22T01:34:20.511890+0000","last_became_active":"2024-03-22T01:34:19.534039+0000","last_became_peered":"2024-03-22T01:34:19.534039+0000","last_unstale":"2024-03-22T01:34:20.511890+0000","last_undegraded":"2024-03-22T01:34:20.511890+0000","last_fullsized":"2024-03-22T01:34:20.511890+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:59:37.363972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"86'1","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.191265+0000","last_change":"2024-03-22T01:34:14.468382+0000","last_active":"2024-03-22T01:34:21.191265+0000","last_peered":"2024-03-22T01:34:21.191265+0000","last_clean":"2024-03-22T01:34:21.191265+0000","last_became_active":"2024-03-22T01:34:14.467939+0000","last_became_peered":"2024-03-22T01:34:14.467939+0000","last_unstale":"2024-03-22T01:34:21.191265+0000","last_undegraded":"2024-03-22T01:34:21.191265+0000","last_fullsized":"2024-03-22T01:34:21.191265+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:57:17.031893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.092181+0000","last_change":"2024-03-22T01:34:15.484361+0000","last_active":"2024-03-22T01:34:21.092181+0000","last_peered":"2024-03-22T01:34:21.092181+0000","last_clean":"2024-03-22T01:34:21.092181+0000","last_became_active":"2024-03-22T01:34:15.483894+0000","last_became_peered":"2024-03-22T01:34:15.483894+0000","last_unstale":"2024-03-22T01:34:21.092181+0000","last_undegraded":"2024-03-22T01:34:21.092181+0000","last_fullsized":"2024-03-22T01:34:21.092181+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:18:53.527145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.191786+0000","last_change":"2024-03-22T01:34:19.531532+0000","last_active":"2024-03-22T01:34:21.191786+0000","last_peered":"2024-03-22T01:34:21.191786+0000","last_clean":"2024-03-22T01:34:21.191786+0000","last_became_active":"2024-03-22T01:34:19.531202+0000","last_became_peered":"2024-03-22T01:34:19.531202+0000","last_unstale":"2024-03-22T01:34:21.191786+0000","last_undegraded":"2024-03-22T01:34:21.191786+0000","last_fullsized":"2024-03-22T01:34:21.191786+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:52:12.823806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.989912+0000","last_change":"2024-03-22T01:34:14.464485+0000","last_active":"2024-03-22T01:34:20.989912+0000","last_peered":"2024-03-22T01:34:20.989912+0000","last_clean":"2024-03-22T01:34:20.989912+0000","last_became_active":"2024-03-22T01:34:14.463973+0000","last_became_peered":"2024-03-22T01:34:14.463973+0000","last_unstale":"2024-03-22T01:34:20.989912+0000","last_undegraded":"2024-03-22T01:34:20.989912+0000","last_fullsized":"2024-03-22T01:34:20.989912+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:11:13.265346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.191338+0000","last_change":"2024-03-22T01:34:15.483893+0000","last_active":"2024-03-22T01:34:21.191338+0000","last_peered":"2024-03-22T01:34:21.191338+0000","last_clean":"2024-03-22T01:34:21.191338+0000","last_became_active":"2024-03-22T01:34:15.482295+0000","last_became_peered":"2024-03-22T01:34:15.482295+0000","last_unstale":"2024-03-22T01:34:21.191338+0000","last_undegraded":"2024-03-22T01:34:21.191338+0000","last_fullsized":"2024-03-22T01:34:21.191338+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:17:33.274900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.192160+0000","last_change":"2024-03-22T01:34:19.522567+0000","last_active":"2024-03-22T01:34:21.192160+0000","last_peered":"2024-03-22T01:34:21.192160+0000","last_clean":"2024-03-22T01:34:21.192160+0000","last_became_active":"2024-03-22T01:34:19.522165+0000","last_became_peered":"2024-03-22T01:34:19.522165+0000","last_unstale":"2024-03-22T01:34:21.192160+0000","last_undegraded":"2024-03-22T01:34:21.192160+0000","last_fullsized":"2024-03-22T01:34:21.192160+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:41:37.829622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.091814+0000","last_change":"2024-03-22T01:34:14.463824+0000","last_active":"2024-03-22T01:34:21.091814+0000","last_peered":"2024-03-22T01:34:21.091814+0000","last_clean":"2024-03-22T01:34:21.091814+0000","last_became_active":"2024-03-22T01:34:14.463484+0000","last_became_peered":"2024-03-22T01:34:14.463484+0000","last_unstale":"2024-03-22T01:34:21.091814+0000","last_undegraded":"2024-03-22T01:34:21.091814+0000","last_fullsized":"2024-03-22T01:34:21.091814+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:19:31.568853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.510383+0000","last_change":"2024-03-22T01:34:15.488883+0000","last_active":"2024-03-22T01:34:20.510383+0000","last_peered":"2024-03-22T01:34:20.510383+0000","last_clean":"2024-03-22T01:34:20.510383+0000","last_became_active":"2024-03-22T01:34:15.488573+0000","last_became_peered":"2024-03-22T01:34:15.488573+0000","last_unstale":"2024-03-22T01:34:20.510383+0000","last_undegraded":"2024-03-22T01:34:20.510383+0000","last_fullsized":"2024-03-22T01:34:20.510383+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:28:46.388930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.192007+0000","last_change":"2024-03-22T01:34:19.527321+0000","last_active":"2024-03-22T01:34:21.192007+0000","last_peered":"2024-03-22T01:34:21.192007+0000","last_clean":"2024-03-22T01:34:21.192007+0000","last_became_active":"2024-03-22T01:34:19.526905+0000","last_became_peered":"2024-03-22T01:34:19.526905+0000","last_unstale":"2024-03-22T01:34:21.192007+0000","last_undegraded":"2024-03-22T01:34:21.192007+0000","last_fullsized":"2024-03-22T01:34:21.192007+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:49:31.023938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.191570+0000","last_change":"2024-03-22T01:34:14.471415+0000","last_active":"2024-03-22T01:34:21.191570+0000","last_peered":"2024-03-22T01:34:21.191570+0000","last_clean":"2024-03-22T01:34:21.191570+0000","last_became_active":"2024-03-22T01:34:14.470928+0000","last_became_peered":"2024-03-22T01:34:14.470928+0000","last_unstale":"2024-03-22T01:34:21.191570+0000","last_undegraded":"2024-03-22T01:34:21.191570+0000","last_fullsized":"2024-03-22T01:34:21.191570+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:09:10.286549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.600275+0000","last_change":"2024-03-22T01:34:15.489656+0000","last_active":"2024-03-22T01:34:20.600275+0000","last_peered":"2024-03-22T01:34:20.600275+0000","last_clean":"2024-03-22T01:34:20.600275+0000","last_became_active":"2024-03-22T01:34:15.488933+0000","last_became_peered":"2024-03-22T01:34:15.488933+0000","last_unstale":"2024-03-22T01:34:20.600275+0000","last_undegraded":"2024-03-22T01:34:20.600275+0000","last_fullsized":"2024-03-22T01:34:20.600275+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:17:38.789414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.510408+0000","last_change":"2024-03-22T01:34:19.534174+0000","last_active":"2024-03-22T01:34:20.510408+0000","last_peered":"2024-03-22T01:34:20.510408+0000","last_clean":"2024-03-22T01:34:20.510408+0000","last_became_active":"2024-03-22T01:34:19.533759+0000","last_became_peered":"2024-03-22T01:34:19.533759+0000","last_unstale":"2024-03-22T01:34:20.510408+0000","last_undegraded":"2024-03-22T01:34:20.510408+0000","last_fullsized":"2024-03-22T01:34:20.510408+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:24:50.363984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.084219+0000","last_change":"2024-03-22T01:34:14.465755+0000","last_active":"2024-03-22T01:34:21.084219+0000","last_peered":"2024-03-22T01:34:21.084219+0000","last_clean":"2024-03-22T01:34:21.084219+0000","last_became_active":"2024-03-22T01:34:14.465230+0000","last_became_peered":"2024-03-22T01:34:14.465230+0000","last_unstale":"2024-03-22T01:34:21.084219+0000","last_undegraded":"2024-03-22T01:34:21.084219+0000","last_fullsized":"2024-03-22T01:34:21.084219+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:56:29.286551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.191316+0000","last_change":"2024-03-22T01:34:15.476563+0000","last_active":"2024-03-22T01:34:21.191316+0000","last_peered":"2024-03-22T01:34:21.191316+0000","last_clean":"2024-03-22T01:34:21.191316+0000","last_became_active":"2024-03-22T01:34:15.476250+0000","last_became_peered":"2024-03-22T01:34:15.476250+0000","last_unstale":"2024-03-22T01:34:21.191316+0000","last_undegraded":"2024-03-22T01:34:21.191316+0000","last_fullsized":"2024-03-22T01:34:21.191316+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:45:59.605528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.511303+0000","last_change":"2024-03-22T01:34:19.527218+0000","last_active":"2024-03-22T01:34:20.511303+0000","last_peered":"2024-03-22T01:34:20.511303+0000","last_clean":"2024-03-22T01:34:20.511303+0000","last_became_active":"2024-03-22T01:34:19.526856+0000","last_became_peered":"2024-03-22T01:34:19.526856+0000","last_unstale":"2024-03-22T01:34:20.511303+0000","last_undegraded":"2024-03-22T01:34:20.511303+0000","last_fullsized":"2024-03-22T01:34:20.511303+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:25:33.878015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"86'3","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.092241+0000","last_change":"2024-03-22T01:34:14.463365+0000","last_active":"2024-03-22T01:34:21.092241+0000","last_peered":"2024-03-22T01:34:21.092241+0000","last_clean":"2024-03-22T01:34:21.092241+0000","last_became_active":"2024-03-22T01:34:14.462984+0000","last_became_peered":"2024-03-22T01:34:14.462984+0000","last_unstale":"2024-03-22T01:34:21.092241+0000","last_undegraded":"2024-03-22T01:34:21.092241+0000","last_fullsized":"2024-03-22T01:34:21.092241+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:55:27.318854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.600003+0000","last_change":"2024-03-22T01:34:15.485309+0000","last_active":"2024-03-22T01:34:20.600003+0000","last_peered":"2024-03-22T01:34:20.600003+0000","last_clean":"2024-03-22T01:34:20.600003+0000","last_became_active":"2024-03-22T01:34:15.484964+0000","last_became_peered":"2024-03-22T01:34:15.484964+0000","last_unstale":"2024-03-22T01:34:20.600003+0000","last_undegraded":"2024-03-22T01:34:20.600003+0000","last_fullsized":"2024-03-22T01:34:20.600003+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:25:57.911623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.510773+0000","last_change":"2024-03-22T01:34:19.529346+0000","last_active":"2024-03-22T01:34:20.510773+0000","last_peered":"2024-03-22T01:34:20.510773+0000","last_clean":"2024-03-22T01:34:20.510773+0000","last_became_active":"2024-03-22T01:34:19.525999+0000","last_became_peered":"2024-03-22T01:34:19.525999+0000","last_unstale":"2024-03-22T01:34:20.510773+0000","last_undegraded":"2024-03-22T01:34:20.510773+0000","last_fullsized":"2024-03-22T01:34:20.510773+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:00:21.868812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'10","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:39.513848+0000","last_change":"2024-03-22T01:34:14.469378+0000","last_active":"2024-03-22T01:34:39.513848+0000","last_peered":"2024-03-22T01:34:39.513848+0000","last_clean":"2024-03-22T01:34:39.513848+0000","last_became_active":"2024-03-22T01:34:14.468956+0000","last_became_peered":"2024-03-22T01:34:14.468956+0000","last_unstale":"2024-03-22T01:34:39.513848+0000","last_undegraded":"2024-03-22T01:34:39.513848+0000","last_fullsized":"2024-03-22T01:34:39.513848+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:40:49.048793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":6,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.877123+0000","last_change":"2024-03-22T01:34:15.488497+0000","last_active":"2024-03-22T01:34:20.877123+0000","last_peered":"2024-03-22T01:34:20.877123+0000","last_clean":"2024-03-22T01:34:20.877123+0000","last_became_active":"2024-03-22T01:34:15.488134+0000","last_became_peered":"2024-03-22T01:34:15.488134+0000","last_unstale":"2024-03-22T01:34:20.877123+0000","last_undegraded":"2024-03-22T01:34:20.877123+0000","last_fullsized":"2024-03-22T01:34:20.877123+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:50:41.206135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.092612+0000","last_change":"2024-03-22T01:34:19.528744+0000","last_active":"2024-03-22T01:34:21.092612+0000","last_peered":"2024-03-22T01:34:21.092612+0000","last_clean":"2024-03-22T01:34:21.092612+0000","last_became_active":"2024-03-22T01:34:19.528375+0000","last_became_peered":"2024-03-22T01:34:19.528375+0000","last_unstale":"2024-03-22T01:34:21.092612+0000","last_undegraded":"2024-03-22T01:34:21.092612+0000","last_fullsized":"2024-03-22T01:34:21.092612+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:47:53.753923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"86'2","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.511645+0000","last_change":"2024-03-22T01:34:14.463289+0000","last_active":"2024-03-22T01:34:20.511645+0000","last_peered":"2024-03-22T01:34:20.511645+0000","last_clean":"2024-03-22T01:34:20.511645+0000","last_became_active":"2024-03-22T01:34:14.462883+0000","last_became_peered":"2024-03-22T01:34:14.462883+0000","last_unstale":"2024-03-22T01:34:20.511645+0000","last_undegraded":"2024-03-22T01:34:20.511645+0000","last_fullsized":"2024-03-22T01:34:20.511645+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:06:14.452674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.990049+0000","last_change":"2024-03-22T01:34:15.484785+0000","last_active":"2024-03-22T01:34:20.990049+0000","last_peered":"2024-03-22T01:34:20.990049+0000","last_clean":"2024-03-22T01:34:20.990049+0000","last_became_active":"2024-03-22T01:34:15.484474+0000","last_became_peered":"2024-03-22T01:34:15.484474+0000","last_unstale":"2024-03-22T01:34:20.990049+0000","last_undegraded":"2024-03-22T01:34:20.990049+0000","last_fullsized":"2024-03-22T01:34:20.990049+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:11:13.338836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.191504+0000","last_change":"2024-03-22T01:34:19.527379+0000","last_active":"2024-03-22T01:34:21.191504+0000","last_peered":"2024-03-22T01:34:21.191504+0000","last_clean":"2024-03-22T01:34:21.191504+0000","last_became_active":"2024-03-22T01:34:19.526957+0000","last_became_peered":"2024-03-22T01:34:19.526957+0000","last_unstale":"2024-03-22T01:34:21.191504+0000","last_undegraded":"2024-03-22T01:34:21.191504+0000","last_fullsized":"2024-03-22T01:34:21.191504+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:03:09.538334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.091981+0000","last_change":"2024-03-22T01:34:14.467372+0000","last_active":"2024-03-22T01:34:21.091981+0000","last_peered":"2024-03-22T01:34:21.091981+0000","last_clean":"2024-03-22T01:34:21.091981+0000","last_became_active":"2024-03-22T01:34:14.466124+0000","last_became_peered":"2024-03-22T01:34:14.466124+0000","last_unstale":"2024-03-22T01:34:21.091981+0000","last_undegraded":"2024-03-22T01:34:21.091981+0000","last_fullsized":"2024-03-22T01:34:21.091981+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:15:41.804735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.084102+0000","last_change":"2024-03-22T01:34:15.491533+0000","last_active":"2024-03-22T01:34:21.084102+0000","last_peered":"2024-03-22T01:34:21.084102+0000","last_clean":"2024-03-22T01:34:21.084102+0000","last_became_active":"2024-03-22T01:34:15.491208+0000","last_became_peered":"2024-03-22T01:34:15.491208+0000","last_unstale":"2024-03-22T01:34:21.084102+0000","last_undegraded":"2024-03-22T01:34:21.084102+0000","last_fullsized":"2024-03-22T01:34:21.084102+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:03:07.800607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.511063+0000","last_change":"2024-03-22T01:34:19.533419+0000","last_active":"2024-03-22T01:34:20.511063+0000","last_peered":"2024-03-22T01:34:20.511063+0000","last_clean":"2024-03-22T01:34:20.511063+0000","last_became_active":"2024-03-22T01:34:19.530042+0000","last_became_peered":"2024-03-22T01:34:19.530042+0000","last_unstale":"2024-03-22T01:34:20.511063+0000","last_undegraded":"2024-03-22T01:34:20.511063+0000","last_fullsized":"2024-03-22T01:34:20.511063+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:46:59.859742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.510719+0000","last_change":"2024-03-22T01:34:14.467033+0000","last_active":"2024-03-22T01:34:20.510719+0000","last_peered":"2024-03-22T01:34:20.510719+0000","last_clean":"2024-03-22T01:34:20.510719+0000","last_became_active":"2024-03-22T01:34:14.466649+0000","last_became_peered":"2024-03-22T01:34:14.466649+0000","last_unstale":"2024-03-22T01:34:20.510719+0000","last_undegraded":"2024-03-22T01:34:20.510719+0000","last_fullsized":"2024-03-22T01:34:20.510719+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:38:44.890899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.083839+0000","last_change":"2024-03-22T01:34:15.502422+0000","last_active":"2024-03-22T01:34:21.083839+0000","last_peered":"2024-03-22T01:34:21.083839+0000","last_clean":"2024-03-22T01:34:21.083839+0000","last_became_active":"2024-03-22T01:34:15.501782+0000","last_became_peered":"2024-03-22T01:34:15.501782+0000","last_unstale":"2024-03-22T01:34:21.083839+0000","last_undegraded":"2024-03-22T01:34:21.083839+0000","last_fullsized":"2024-03-22T01:34:21.083839+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:09:06.408318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.990486+0000","last_change":"2024-03-22T01:34:19.531417+0000","last_active":"2024-03-22T01:34:20.990486+0000","last_peered":"2024-03-22T01:34:20.990486+0000","last_clean":"2024-03-22T01:34:20.990486+0000","last_became_active":"2024-03-22T01:34:19.531003+0000","last_became_peered":"2024-03-22T01:34:19.531003+0000","last_unstale":"2024-03-22T01:34:20.990486+0000","last_undegraded":"2024-03-22T01:34:20.990486+0000","last_fullsized":"2024-03-22T01:34:20.990486+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:33:52.865110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"86'8","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.876692+0000","last_change":"2024-03-22T01:34:14.472904+0000","last_active":"2024-03-22T01:34:20.876692+0000","last_peered":"2024-03-22T01:34:20.876692+0000","last_clean":"2024-03-22T01:34:20.876692+0000","last_became_active":"2024-03-22T01:34:14.472442+0000","last_became_peered":"2024-03-22T01:34:14.472442+0000","last_unstale":"2024-03-22T01:34:20.876692+0000","last_undegraded":"2024-03-22T01:34:20.876692+0000","last_fullsized":"2024-03-22T01:34:20.876692+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:57:45.478316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.091885+0000","last_change":"2024-03-22T01:34:15.477411+0000","last_active":"2024-03-22T01:34:21.091885+0000","last_peered":"2024-03-22T01:34:21.091885+0000","last_clean":"2024-03-22T01:34:21.091885+0000","last_became_active":"2024-03-22T01:34:15.477035+0000","last_became_peered":"2024-03-22T01:34:15.477035+0000","last_unstale":"2024-03-22T01:34:21.091885+0000","last_undegraded":"2024-03-22T01:34:21.091885+0000","last_fullsized":"2024-03-22T01:34:21.091885+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:33:13.410872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.091888+0000","last_change":"2024-03-22T01:34:19.528258+0000","last_active":"2024-03-22T01:34:21.091888+0000","last_peered":"2024-03-22T01:34:21.091888+0000","last_clean":"2024-03-22T01:34:21.091888+0000","last_became_active":"2024-03-22T01:34:19.527668+0000","last_became_peered":"2024-03-22T01:34:19.527668+0000","last_unstale":"2024-03-22T01:34:21.091888+0000","last_undegraded":"2024-03-22T01:34:21.091888+0000","last_fullsized":"2024-03-22T01:34:21.091888+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:34:19.162609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'1","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.877273+0000","last_change":"2024-03-22T01:34:14.464735+0000","last_active":"2024-03-22T01:34:20.877273+0000","last_peered":"2024-03-22T01:34:20.877273+0000","last_clean":"2024-03-22T01:34:20.877273+0000","last_became_active":"2024-03-22T01:34:14.464359+0000","last_became_peered":"2024-03-22T01:34:14.464359+0000","last_unstale":"2024-03-22T01:34:20.877273+0000","last_undegraded":"2024-03-22T01:34:20.877273+0000","last_fullsized":"2024-03-22T01:34:20.877273+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:21:18.354922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.191611+0000","last_change":"2024-03-22T01:34:15.493904+0000","last_active":"2024-03-22T01:34:21.191611+0000","last_peered":"2024-03-22T01:34:21.191611+0000","last_clean":"2024-03-22T01:34:21.191611+0000","last_became_active":"2024-03-22T01:34:15.493557+0000","last_became_peered":"2024-03-22T01:34:15.493557+0000","last_unstale":"2024-03-22T01:34:21.191611+0000","last_undegraded":"2024-03-22T01:34:21.191611+0000","last_fullsized":"2024-03-22T01:34:21.191611+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:52:56.295047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.876991+0000","last_change":"2024-03-22T01:34:19.668136+0000","last_active":"2024-03-22T01:34:20.876991+0000","last_peered":"2024-03-22T01:34:20.876991+0000","last_clean":"2024-03-22T01:34:20.876991+0000","last_became_active":"2024-03-22T01:34:19.667633+0000","last_became_peered":"2024-03-22T01:34:19.667633+0000","last_unstale":"2024-03-22T01:34:20.876991+0000","last_undegraded":"2024-03-22T01:34:20.876991+0000","last_fullsized":"2024-03-22T01:34:20.876991+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:07:26.455373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.510615+0000","last_change":"2024-03-22T01:34:14.471959+0000","last_active":"2024-03-22T01:34:20.510615+0000","last_peered":"2024-03-22T01:34:20.510615+0000","last_clean":"2024-03-22T01:34:20.510615+0000","last_became_active":"2024-03-22T01:34:14.471618+0000","last_became_peered":"2024-03-22T01:34:14.471618+0000","last_unstale":"2024-03-22T01:34:20.510615+0000","last_undegraded":"2024-03-22T01:34:20.510615+0000","last_fullsized":"2024-03-22T01:34:20.510615+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:45:03.537013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.191540+0000","last_change":"2024-03-22T01:34:15.495770+0000","last_active":"2024-03-22T01:34:21.191540+0000","last_peered":"2024-03-22T01:34:21.191540+0000","last_clean":"2024-03-22T01:34:21.191540+0000","last_became_active":"2024-03-22T01:34:15.495403+0000","last_became_peered":"2024-03-22T01:34:15.495403+0000","last_unstale":"2024-03-22T01:34:21.191540+0000","last_undegraded":"2024-03-22T01:34:21.191540+0000","last_fullsized":"2024-03-22T01:34:21.191540+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:48:05.184305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.990326+0000","last_change":"2024-03-22T01:34:19.520204+0000","last_active":"2024-03-22T01:34:20.990326+0000","last_peered":"2024-03-22T01:34:20.990326+0000","last_clean":"2024-03-22T01:34:20.990326+0000","last_became_active":"2024-03-22T01:34:19.519847+0000","last_became_peered":"2024-03-22T01:34:19.519847+0000","last_unstale":"2024-03-22T01:34:20.990326+0000","last_undegraded":"2024-03-22T01:34:20.990326+0000","last_fullsized":"2024-03-22T01:34:20.990326+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:06:21.389636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.084029+0000","last_change":"2024-03-22T01:34:14.464162+0000","last_active":"2024-03-22T01:34:21.084029+0000","last_peered":"2024-03-22T01:34:21.084029+0000","last_clean":"2024-03-22T01:34:21.084029+0000","last_became_active":"2024-03-22T01:34:14.462700+0000","last_became_peered":"2024-03-22T01:34:14.462700+0000","last_unstale":"2024-03-22T01:34:21.084029+0000","last_undegraded":"2024-03-22T01:34:21.084029+0000","last_fullsized":"2024-03-22T01:34:21.084029+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:45:32.262466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.191342+0000","last_change":"2024-03-22T01:34:15.488256+0000","last_active":"2024-03-22T01:34:21.191342+0000","last_peered":"2024-03-22T01:34:21.191342+0000","last_clean":"2024-03-22T01:34:21.191342+0000","last_became_active":"2024-03-22T01:34:15.487939+0000","last_became_peered":"2024-03-22T01:34:15.487939+0000","last_unstale":"2024-03-22T01:34:21.191342+0000","last_undegraded":"2024-03-22T01:34:21.191342+0000","last_fullsized":"2024-03-22T01:34:21.191342+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:30:49.323531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.990386+0000","last_change":"2024-03-22T01:34:19.667633+0000","last_active":"2024-03-22T01:34:20.990386+0000","last_peered":"2024-03-22T01:34:20.990386+0000","last_clean":"2024-03-22T01:34:20.990386+0000","last_became_active":"2024-03-22T01:34:19.667199+0000","last_became_peered":"2024-03-22T01:34:19.667199+0000","last_unstale":"2024-03-22T01:34:20.990386+0000","last_undegraded":"2024-03-22T01:34:20.990386+0000","last_fullsized":"2024-03-22T01:34:20.990386+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T01:58:13.459475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.091992+0000","last_change":"2024-03-22T01:34:15.489468+0000","last_active":"2024-03-22T01:34:21.091992+0000","last_peered":"2024-03-22T01:34:21.091992+0000","last_clean":"2024-03-22T01:34:21.091992+0000","last_became_active":"2024-03-22T01:34:15.489029+0000","last_became_peered":"2024-03-22T01:34:15.489029+0000","last_unstale":"2024-03-22T01:34:21.091992+0000","last_undegraded":"2024-03-22T01:34:21.091992+0000","last_fullsized":"2024-03-22T01:34:21.091992+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:00:01.370161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.989799+0000","last_change":"2024-03-22T01:34:14.464552+0000","last_active":"2024-03-22T01:34:20.989799+0000","last_peered":"2024-03-22T01:34:20.989799+0000","last_clean":"2024-03-22T01:34:20.989799+0000","last_became_active":"2024-03-22T01:34:14.463554+0000","last_became_peered":"2024-03-22T01:34:14.463554+0000","last_unstale":"2024-03-22T01:34:20.989799+0000","last_undegraded":"2024-03-22T01:34:20.989799+0000","last_fullsized":"2024-03-22T01:34:20.989799+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:51:16.709278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"91'1","reported_seq":15,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:22.470116+0000","last_change":"2024-03-22T01:34:19.526066+0000","last_active":"2024-03-22T01:34:22.470116+0000","last_peered":"2024-03-22T01:34:22.470116+0000","last_clean":"2024-03-22T01:34:22.470116+0000","last_became_active":"2024-03-22T01:34:19.525659+0000","last_became_peered":"2024-03-22T01:34:19.525659+0000","last_unstale":"2024-03-22T01:34:22.470116+0000","last_undegraded":"2024-03-22T01:34:22.470116+0000","last_fullsized":"2024-03-22T01:34:22.470116+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:06:26.287971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.191440+0000","last_change":"2024-03-22T01:34:15.477018+0000","last_active":"2024-03-22T01:34:21.191440+0000","last_peered":"2024-03-22T01:34:21.191440+0000","last_clean":"2024-03-22T01:34:21.191440+0000","last_became_active":"2024-03-22T01:34:15.476613+0000","last_became_peered":"2024-03-22T01:34:15.476613+0000","last_unstale":"2024-03-22T01:34:21.191440+0000","last_undegraded":"2024-03-22T01:34:21.191440+0000","last_fullsized":"2024-03-22T01:34:21.191440+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:10:07.980123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.510790+0000","last_change":"2024-03-22T01:34:14.470922+0000","last_active":"2024-03-22T01:34:20.510790+0000","last_peered":"2024-03-22T01:34:20.510790+0000","last_clean":"2024-03-22T01:34:20.510790+0000","last_became_active":"2024-03-22T01:34:14.470515+0000","last_became_peered":"2024-03-22T01:34:14.470515+0000","last_unstale":"2024-03-22T01:34:20.510790+0000","last_undegraded":"2024-03-22T01:34:20.510790+0000","last_fullsized":"2024-03-22T01:34:20.510790+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:52:50.203631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":354,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.083835+0000","last_change":"2024-03-22T01:33:03.615234+0000","last_active":"2024-03-22T01:34:21.083835+0000","last_peered":"2024-03-22T01:34:21.083835+0000","last_clean":"2024-03-22T01:34:21.083835+0000","last_became_active":"2024-03-22T01:33:03.614710+0000","last_became_peered":"2024-03-22T01:33:03.614710+0000","last_unstale":"2024-03-22T01:34:21.083835+0000","last_undegraded":"2024-03-22T01:34:21.083835+0000","last_fullsized":"2024-03-22T01:34:21.083835+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:29:38.312292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:29:38.312292+0000","last_clean_scrub_stamp":"2024-03-22T01:29:38.312292+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:56:42.348119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.092499+0000","last_change":"2024-03-22T01:34:19.668565+0000","last_active":"2024-03-22T01:34:21.092499+0000","last_peered":"2024-03-22T01:34:21.092499+0000","last_clean":"2024-03-22T01:34:21.092499+0000","last_became_active":"2024-03-22T01:34:19.667959+0000","last_became_peered":"2024-03-22T01:34:19.667959+0000","last_unstale":"2024-03-22T01:34:21.092499+0000","last_undegraded":"2024-03-22T01:34:21.092499+0000","last_fullsized":"2024-03-22T01:34:21.092499+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:13:04.447486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.989797+0000","last_change":"2024-03-22T01:34:15.489513+0000","last_active":"2024-03-22T01:34:20.989797+0000","last_peered":"2024-03-22T01:34:20.989797+0000","last_clean":"2024-03-22T01:34:20.989797+0000","last_became_active":"2024-03-22T01:34:15.489188+0000","last_became_peered":"2024-03-22T01:34:15.489188+0000","last_unstale":"2024-03-22T01:34:20.989797+0000","last_undegraded":"2024-03-22T01:34:20.989797+0000","last_fullsized":"2024-03-22T01:34:20.989797+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:08:24.859350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.083994+0000","last_change":"2024-03-22T01:34:14.466305+0000","last_active":"2024-03-22T01:34:21.083994+0000","last_peered":"2024-03-22T01:34:21.083994+0000","last_clean":"2024-03-22T01:34:21.083994+0000","last_became_active":"2024-03-22T01:34:14.465495+0000","last_became_peered":"2024-03-22T01:34:14.465495+0000","last_unstale":"2024-03-22T01:34:21.083994+0000","last_undegraded":"2024-03-22T01:34:21.083994+0000","last_fullsized":"2024-03-22T01:34:21.083994+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:45:13.481187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.191481+0000","last_change":"2024-03-22T01:34:19.526970+0000","last_active":"2024-03-22T01:34:21.191481+0000","last_peered":"2024-03-22T01:34:21.191481+0000","last_clean":"2024-03-22T01:34:21.191481+0000","last_became_active":"2024-03-22T01:34:19.526462+0000","last_became_peered":"2024-03-22T01:34:19.526462+0000","last_unstale":"2024-03-22T01:34:21.191481+0000","last_undegraded":"2024-03-22T01:34:21.191481+0000","last_fullsized":"2024-03-22T01:34:21.191481+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:04:18.417358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.877214+0000","last_change":"2024-03-22T01:34:14.473771+0000","last_active":"2024-03-22T01:34:20.877214+0000","last_peered":"2024-03-22T01:34:20.877214+0000","last_clean":"2024-03-22T01:34:20.877214+0000","last_became_active":"2024-03-22T01:34:14.473188+0000","last_became_peered":"2024-03-22T01:34:14.473188+0000","last_unstale":"2024-03-22T01:34:20.877214+0000","last_undegraded":"2024-03-22T01:34:20.877214+0000","last_fullsized":"2024-03-22T01:34:20.877214+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:25:19.541443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.990444+0000","last_change":"2024-03-22T01:34:15.490405+0000","last_active":"2024-03-22T01:34:20.990444+0000","last_peered":"2024-03-22T01:34:20.990444+0000","last_clean":"2024-03-22T01:34:20.990444+0000","last_became_active":"2024-03-22T01:34:15.489921+0000","last_became_peered":"2024-03-22T01:34:15.489921+0000","last_unstale":"2024-03-22T01:34:20.990444+0000","last_undegraded":"2024-03-22T01:34:20.990444+0000","last_fullsized":"2024-03-22T01:34:20.990444+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:27:28.804649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.191289+0000","last_change":"2024-03-22T01:34:19.667651+0000","last_active":"2024-03-22T01:34:21.191289+0000","last_peered":"2024-03-22T01:34:21.191289+0000","last_clean":"2024-03-22T01:34:21.191289+0000","last_became_active":"2024-03-22T01:34:19.667197+0000","last_became_peered":"2024-03-22T01:34:19.667197+0000","last_unstale":"2024-03-22T01:34:21.191289+0000","last_undegraded":"2024-03-22T01:34:21.191289+0000","last_fullsized":"2024-03-22T01:34:21.191289+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:32:55.874098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'3","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:41.489616+0000","last_change":"2024-03-22T01:34:14.474988+0000","last_active":"2024-03-22T01:34:41.489616+0000","last_peered":"2024-03-22T01:34:41.489616+0000","last_clean":"2024-03-22T01:34:41.489616+0000","last_became_active":"2024-03-22T01:34:14.474651+0000","last_became_peered":"2024-03-22T01:34:14.474651+0000","last_unstale":"2024-03-22T01:34:41.489616+0000","last_undegraded":"2024-03-22T01:34:41.489616+0000","last_fullsized":"2024-03-22T01:34:41.489616+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:14:59.846727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.191516+0000","last_change":"2024-03-22T01:34:15.485209+0000","last_active":"2024-03-22T01:34:21.191516+0000","last_peered":"2024-03-22T01:34:21.191516+0000","last_clean":"2024-03-22T01:34:21.191516+0000","last_became_active":"2024-03-22T01:34:15.484898+0000","last_became_peered":"2024-03-22T01:34:15.484898+0000","last_unstale":"2024-03-22T01:34:21.191516+0000","last_undegraded":"2024-03-22T01:34:21.191516+0000","last_fullsized":"2024-03-22T01:34:21.191516+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:40:01.362735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.511340+0000","last_change":"2024-03-22T01:34:19.523959+0000","last_active":"2024-03-22T01:34:20.511340+0000","last_peered":"2024-03-22T01:34:20.511340+0000","last_clean":"2024-03-22T01:34:20.511340+0000","last_became_active":"2024-03-22T01:34:19.523556+0000","last_became_peered":"2024-03-22T01:34:19.523556+0000","last_unstale":"2024-03-22T01:34:20.511340+0000","last_undegraded":"2024-03-22T01:34:20.511340+0000","last_fullsized":"2024-03-22T01:34:20.511340+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:22:21.623042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"86'2","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.511115+0000","last_change":"2024-03-22T01:34:14.463434+0000","last_active":"2024-03-22T01:34:20.511115+0000","last_peered":"2024-03-22T01:34:20.511115+0000","last_clean":"2024-03-22T01:34:20.511115+0000","last_became_active":"2024-03-22T01:34:14.463075+0000","last_became_peered":"2024-03-22T01:34:14.463075+0000","last_unstale":"2024-03-22T01:34:20.511115+0000","last_undegraded":"2024-03-22T01:34:20.511115+0000","last_fullsized":"2024-03-22T01:34:20.511115+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:43:18.223766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.192191+0000","last_change":"2024-03-22T01:34:15.482695+0000","last_active":"2024-03-22T01:34:21.192191+0000","last_peered":"2024-03-22T01:34:21.192191+0000","last_clean":"2024-03-22T01:34:21.192191+0000","last_became_active":"2024-03-22T01:34:15.481881+0000","last_became_peered":"2024-03-22T01:34:15.481881+0000","last_unstale":"2024-03-22T01:34:21.192191+0000","last_undegraded":"2024-03-22T01:34:21.192191+0000","last_fullsized":"2024-03-22T01:34:21.192191+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:44:26.290714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.511053+0000","last_change":"2024-03-22T01:34:19.668252+0000","last_active":"2024-03-22T01:34:20.511053+0000","last_peered":"2024-03-22T01:34:20.511053+0000","last_clean":"2024-03-22T01:34:20.511053+0000","last_became_active":"2024-03-22T01:34:19.667830+0000","last_became_peered":"2024-03-22T01:34:19.667830+0000","last_unstale":"2024-03-22T01:34:20.511053+0000","last_undegraded":"2024-03-22T01:34:20.511053+0000","last_fullsized":"2024-03-22T01:34:20.511053+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:08:29.602899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.599766+0000","last_change":"2024-03-22T01:34:14.473020+0000","last_active":"2024-03-22T01:34:20.599766+0000","last_peered":"2024-03-22T01:34:20.599766+0000","last_clean":"2024-03-22T01:34:20.599766+0000","last_became_active":"2024-03-22T01:34:14.472689+0000","last_became_peered":"2024-03-22T01:34:14.472689+0000","last_unstale":"2024-03-22T01:34:20.599766+0000","last_undegraded":"2024-03-22T01:34:20.599766+0000","last_fullsized":"2024-03-22T01:34:20.599766+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:04:19.724685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.876686+0000","last_change":"2024-03-22T01:34:15.491613+0000","last_active":"2024-03-22T01:34:20.876686+0000","last_peered":"2024-03-22T01:34:20.876686+0000","last_clean":"2024-03-22T01:34:20.876686+0000","last_became_active":"2024-03-22T01:34:15.490922+0000","last_became_peered":"2024-03-22T01:34:15.490922+0000","last_unstale":"2024-03-22T01:34:20.876686+0000","last_undegraded":"2024-03-22T01:34:20.876686+0000","last_fullsized":"2024-03-22T01:34:20.876686+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:09:33.177646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.511039+0000","last_change":"2024-03-22T01:34:19.534537+0000","last_active":"2024-03-22T01:34:20.511039+0000","last_peered":"2024-03-22T01:34:20.511039+0000","last_clean":"2024-03-22T01:34:20.511039+0000","last_became_active":"2024-03-22T01:34:19.534038+0000","last_became_peered":"2024-03-22T01:34:19.534038+0000","last_unstale":"2024-03-22T01:34:20.511039+0000","last_undegraded":"2024-03-22T01:34:20.511039+0000","last_fullsized":"2024-03-22T01:34:20.511039+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:59:26.020000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.191607+0000","last_change":"2024-03-22T01:34:14.464724+0000","last_active":"2024-03-22T01:34:21.191607+0000","last_peered":"2024-03-22T01:34:21.191607+0000","last_clean":"2024-03-22T01:34:21.191607+0000","last_became_active":"2024-03-22T01:34:14.464056+0000","last_became_peered":"2024-03-22T01:34:14.464056+0000","last_unstale":"2024-03-22T01:34:21.191607+0000","last_undegraded":"2024-03-22T01:34:21.191607+0000","last_fullsized":"2024-03-22T01:34:21.191607+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:49:39.098126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.191340+0000","last_change":"2024-03-22T01:34:15.488934+0000","last_active":"2024-03-22T01:34:21.191340+0000","last_peered":"2024-03-22T01:34:21.191340+0000","last_clean":"2024-03-22T01:34:21.191340+0000","last_became_active":"2024-03-22T01:34:15.488585+0000","last_became_peered":"2024-03-22T01:34:15.488585+0000","last_unstale":"2024-03-22T01:34:21.191340+0000","last_undegraded":"2024-03-22T01:34:21.191340+0000","last_fullsized":"2024-03-22T01:34:21.191340+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:59:04.554614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.511140+0000","last_change":"2024-03-22T01:34:19.534096+0000","last_active":"2024-03-22T01:34:20.511140+0000","last_peered":"2024-03-22T01:34:20.511140+0000","last_clean":"2024-03-22T01:34:20.511140+0000","last_became_active":"2024-03-22T01:34:19.533645+0000","last_became_peered":"2024-03-22T01:34:19.533645+0000","last_unstale":"2024-03-22T01:34:20.511140+0000","last_undegraded":"2024-03-22T01:34:20.511140+0000","last_fullsized":"2024-03-22T01:34:20.511140+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:50:21.259146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.511279+0000","last_change":"2024-03-22T01:34:14.469529+0000","last_active":"2024-03-22T01:34:20.511279+0000","last_peered":"2024-03-22T01:34:20.511279+0000","last_clean":"2024-03-22T01:34:20.511279+0000","last_became_active":"2024-03-22T01:34:14.469204+0000","last_became_peered":"2024-03-22T01:34:14.469204+0000","last_unstale":"2024-03-22T01:34:20.511279+0000","last_undegraded":"2024-03-22T01:34:20.511279+0000","last_fullsized":"2024-03-22T01:34:20.511279+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:39:57.048537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.191758+0000","last_change":"2024-03-22T01:34:15.501324+0000","last_active":"2024-03-22T01:34:21.191758+0000","last_peered":"2024-03-22T01:34:21.191758+0000","last_clean":"2024-03-22T01:34:21.191758+0000","last_became_active":"2024-03-22T01:34:15.501015+0000","last_became_peered":"2024-03-22T01:34:15.501015+0000","last_unstale":"2024-03-22T01:34:21.191758+0000","last_undegraded":"2024-03-22T01:34:21.191758+0000","last_fullsized":"2024-03-22T01:34:21.191758+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:35:27.673019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.511064+0000","last_change":"2024-03-22T01:34:19.667806+0000","last_active":"2024-03-22T01:34:20.511064+0000","last_peered":"2024-03-22T01:34:20.511064+0000","last_clean":"2024-03-22T01:34:20.511064+0000","last_became_active":"2024-03-22T01:34:19.667249+0000","last_became_peered":"2024-03-22T01:34:19.667249+0000","last_unstale":"2024-03-22T01:34:20.511064+0000","last_undegraded":"2024-03-22T01:34:20.511064+0000","last_fullsized":"2024-03-22T01:34:20.511064+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:44:22.476882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"86'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.876873+0000","last_change":"2024-03-22T01:34:14.471800+0000","last_active":"2024-03-22T01:34:20.876873+0000","last_peered":"2024-03-22T01:34:20.876873+0000","last_clean":"2024-03-22T01:34:20.876873+0000","last_became_active":"2024-03-22T01:34:14.471422+0000","last_became_peered":"2024-03-22T01:34:14.471422+0000","last_unstale":"2024-03-22T01:34:20.876873+0000","last_undegraded":"2024-03-22T01:34:20.876873+0000","last_fullsized":"2024-03-22T01:34:20.876873+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:25:58.546331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.083985+0000","last_change":"2024-03-22T01:34:15.487348+0000","last_active":"2024-03-22T01:34:21.083985+0000","last_peered":"2024-03-22T01:34:21.083985+0000","last_clean":"2024-03-22T01:34:21.083985+0000","last_became_active":"2024-03-22T01:34:15.486788+0000","last_became_peered":"2024-03-22T01:34:15.486788+0000","last_unstale":"2024-03-22T01:34:21.083985+0000","last_undegraded":"2024-03-22T01:34:21.083985+0000","last_fullsized":"2024-03-22T01:34:21.083985+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:10:31.489021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.599883+0000","last_change":"2024-03-22T01:34:19.521490+0000","last_active":"2024-03-22T01:34:20.599883+0000","last_peered":"2024-03-22T01:34:20.599883+0000","last_clean":"2024-03-22T01:34:20.599883+0000","last_became_active":"2024-03-22T01:34:19.521047+0000","last_became_peered":"2024-03-22T01:34:19.521047+0000","last_unstale":"2024-03-22T01:34:20.599883+0000","last_undegraded":"2024-03-22T01:34:20.599883+0000","last_fullsized":"2024-03-22T01:34:20.599883+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:36:50.176756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.511888+0000","last_change":"2024-03-22T01:34:14.457729+0000","last_active":"2024-03-22T01:34:20.511888+0000","last_peered":"2024-03-22T01:34:20.511888+0000","last_clean":"2024-03-22T01:34:20.511888+0000","last_became_active":"2024-03-22T01:34:14.457336+0000","last_became_peered":"2024-03-22T01:34:14.457336+0000","last_unstale":"2024-03-22T01:34:20.511888+0000","last_undegraded":"2024-03-22T01:34:20.511888+0000","last_fullsized":"2024-03-22T01:34:20.511888+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:11:51.184034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.084390+0000","last_change":"2024-03-22T01:34:15.491683+0000","last_active":"2024-03-22T01:34:21.084390+0000","last_peered":"2024-03-22T01:34:21.084390+0000","last_clean":"2024-03-22T01:34:21.084390+0000","last_became_active":"2024-03-22T01:34:15.491030+0000","last_became_peered":"2024-03-22T01:34:15.491030+0000","last_unstale":"2024-03-22T01:34:21.084390+0000","last_undegraded":"2024-03-22T01:34:21.084390+0000","last_fullsized":"2024-03-22T01:34:21.084390+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:02:10.115971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.190882+0000","last_change":"2024-03-22T01:34:19.528608+0000","last_active":"2024-03-22T01:34:21.190882+0000","last_peered":"2024-03-22T01:34:21.190882+0000","last_clean":"2024-03-22T01:34:21.190882+0000","last_became_active":"2024-03-22T01:34:19.528173+0000","last_became_peered":"2024-03-22T01:34:19.528173+0000","last_unstale":"2024-03-22T01:34:21.190882+0000","last_undegraded":"2024-03-22T01:34:21.190882+0000","last_fullsized":"2024-03-22T01:34:21.190882+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:36:02.193187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"86'1","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.092410+0000","last_change":"2024-03-22T01:34:14.467755+0000","last_active":"2024-03-22T01:34:21.092410+0000","last_peered":"2024-03-22T01:34:21.092410+0000","last_clean":"2024-03-22T01:34:21.092410+0000","last_became_active":"2024-03-22T01:34:14.467426+0000","last_became_peered":"2024-03-22T01:34:14.467426+0000","last_unstale":"2024-03-22T01:34:21.092410+0000","last_undegraded":"2024-03-22T01:34:21.092410+0000","last_fullsized":"2024-03-22T01:34:21.092410+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:16:35.436872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.084257+0000","last_change":"2024-03-22T01:34:15.492437+0000","last_active":"2024-03-22T01:34:21.084257+0000","last_peered":"2024-03-22T01:34:21.084257+0000","last_clean":"2024-03-22T01:34:21.084257+0000","last_became_active":"2024-03-22T01:34:15.492110+0000","last_became_peered":"2024-03-22T01:34:15.492110+0000","last_unstale":"2024-03-22T01:34:21.084257+0000","last_undegraded":"2024-03-22T01:34:21.084257+0000","last_fullsized":"2024-03-22T01:34:21.084257+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:58:57.863065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.091668+0000","last_change":"2024-03-22T01:34:19.523631+0000","last_active":"2024-03-22T01:34:21.091668+0000","last_peered":"2024-03-22T01:34:21.091668+0000","last_clean":"2024-03-22T01:34:21.091668+0000","last_became_active":"2024-03-22T01:34:19.522968+0000","last_became_peered":"2024-03-22T01:34:19.522968+0000","last_unstale":"2024-03-22T01:34:21.091668+0000","last_undegraded":"2024-03-22T01:34:21.091668+0000","last_fullsized":"2024-03-22T01:34:21.091668+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:27:24.182886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.990402+0000","last_change":"2024-03-22T01:34:14.464113+0000","last_active":"2024-03-22T01:34:20.990402+0000","last_peered":"2024-03-22T01:34:20.990402+0000","last_clean":"2024-03-22T01:34:20.990402+0000","last_became_active":"2024-03-22T01:34:14.463389+0000","last_became_peered":"2024-03-22T01:34:14.463389+0000","last_unstale":"2024-03-22T01:34:20.990402+0000","last_undegraded":"2024-03-22T01:34:20.990402+0000","last_fullsized":"2024-03-22T01:34:20.990402+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:08:42.376753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.084319+0000","last_change":"2024-03-22T01:34:15.492941+0000","last_active":"2024-03-22T01:34:21.084319+0000","last_peered":"2024-03-22T01:34:21.084319+0000","last_clean":"2024-03-22T01:34:21.084319+0000","last_became_active":"2024-03-22T01:34:15.492598+0000","last_became_peered":"2024-03-22T01:34:15.492598+0000","last_unstale":"2024-03-22T01:34:21.084319+0000","last_undegraded":"2024-03-22T01:34:21.084319+0000","last_fullsized":"2024-03-22T01:34:21.084319+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:22:35.502404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.191449+0000","last_change":"2024-03-22T01:34:15.494548+0000","last_active":"2024-03-22T01:34:21.191449+0000","last_peered":"2024-03-22T01:34:21.191449+0000","last_clean":"2024-03-22T01:34:21.191449+0000","last_became_active":"2024-03-22T01:34:15.494228+0000","last_became_peered":"2024-03-22T01:34:15.494228+0000","last_unstale":"2024-03-22T01:34:21.191449+0000","last_undegraded":"2024-03-22T01:34:21.191449+0000","last_fullsized":"2024-03-22T01:34:21.191449+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:43:26.011030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.511375+0000","last_change":"2024-03-22T01:34:15.490843+0000","last_active":"2024-03-22T01:34:20.511375+0000","last_peered":"2024-03-22T01:34:20.511375+0000","last_clean":"2024-03-22T01:34:20.511375+0000","last_became_active":"2024-03-22T01:34:15.490474+0000","last_became_peered":"2024-03-22T01:34:15.490474+0000","last_unstale":"2024-03-22T01:34:20.511375+0000","last_undegraded":"2024-03-22T01:34:20.511375+0000","last_fullsized":"2024-03-22T01:34:20.511375+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:11:31.488132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.511255+0000","last_change":"2024-03-22T01:34:15.502213+0000","last_active":"2024-03-22T01:34:20.511255+0000","last_peered":"2024-03-22T01:34:20.511255+0000","last_clean":"2024-03-22T01:34:20.511255+0000","last_became_active":"2024-03-22T01:34:15.501464+0000","last_became_peered":"2024-03-22T01:34:15.501464+0000","last_unstale":"2024-03-22T01:34:20.511255+0000","last_undegraded":"2024-03-22T01:34:20.511255+0000","last_fullsized":"2024-03-22T01:34:20.511255+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:38:02.090365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.191071+0000","last_change":"2024-03-22T01:34:15.481282+0000","last_active":"2024-03-22T01:34:21.191071+0000","last_peered":"2024-03-22T01:34:21.191071+0000","last_clean":"2024-03-22T01:34:21.191071+0000","last_became_active":"2024-03-22T01:34:15.480465+0000","last_became_peered":"2024-03-22T01:34:15.480465+0000","last_unstale":"2024-03-22T01:34:21.191071+0000","last_undegraded":"2024-03-22T01:34:21.191071+0000","last_fullsized":"2024-03-22T01:34:21.191071+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:55:59.398543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.990394+0000","last_change":"2024-03-22T01:34:15.480910+0000","last_active":"2024-03-22T01:34:20.990394+0000","last_peered":"2024-03-22T01:34:20.990394+0000","last_clean":"2024-03-22T01:34:20.990394+0000","last_became_active":"2024-03-22T01:34:15.480601+0000","last_became_peered":"2024-03-22T01:34:15.480601+0000","last_unstale":"2024-03-22T01:34:20.990394+0000","last_undegraded":"2024-03-22T01:34:20.990394+0000","last_fullsized":"2024-03-22T01:34:20.990394+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:19:51.563598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.191743+0000","last_change":"2024-03-22T01:34:15.490859+0000","last_active":"2024-03-22T01:34:21.191743+0000","last_peered":"2024-03-22T01:34:21.191743+0000","last_clean":"2024-03-22T01:34:21.191743+0000","last_became_active":"2024-03-22T01:34:15.490337+0000","last_became_peered":"2024-03-22T01:34:15.490337+0000","last_unstale":"2024-03-22T01:34:21.191743+0000","last_undegraded":"2024-03-22T01:34:21.191743+0000","last_fullsized":"2024-03-22T01:34:21.191743+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:46:07.072015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.512014+0000","last_change":"2024-03-22T01:34:15.495531+0000","last_active":"2024-03-22T01:34:20.512014+0000","last_peered":"2024-03-22T01:34:20.512014+0000","last_clean":"2024-03-22T01:34:20.512014+0000","last_became_active":"2024-03-22T01:34:15.495216+0000","last_became_peered":"2024-03-22T01:34:15.495216+0000","last_unstale":"2024-03-22T01:34:20.512014+0000","last_undegraded":"2024-03-22T01:34:20.512014+0000","last_fullsized":"2024-03-22T01:34:20.512014+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:39:22.575926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.191691+0000","last_change":"2024-03-22T01:34:15.499430+0000","last_active":"2024-03-22T01:34:21.191691+0000","last_peered":"2024-03-22T01:34:21.191691+0000","last_clean":"2024-03-22T01:34:21.191691+0000","last_became_active":"2024-03-22T01:34:15.499114+0000","last_became_peered":"2024-03-22T01:34:15.499114+0000","last_unstale":"2024-03-22T01:34:21.191691+0000","last_undegraded":"2024-03-22T01:34:21.191691+0000","last_fullsized":"2024-03-22T01:34:21.191691+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:23:01.781963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.091940+0000","last_change":"2024-03-22T01:34:15.485449+0000","last_active":"2024-03-22T01:34:21.091940+0000","last_peered":"2024-03-22T01:34:21.091940+0000","last_clean":"2024-03-22T01:34:21.091940+0000","last_became_active":"2024-03-22T01:34:15.485084+0000","last_became_peered":"2024-03-22T01:34:15.485084+0000","last_unstale":"2024-03-22T01:34:21.091940+0000","last_undegraded":"2024-03-22T01:34:21.091940+0000","last_fullsized":"2024-03-22T01:34:21.091940+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:53:25.632719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.191699+0000","last_change":"2024-03-22T01:34:15.495521+0000","last_active":"2024-03-22T01:34:21.191699+0000","last_peered":"2024-03-22T01:34:21.191699+0000","last_clean":"2024-03-22T01:34:21.191699+0000","last_became_active":"2024-03-22T01:34:15.495218+0000","last_became_peered":"2024-03-22T01:34:15.495218+0000","last_unstale":"2024-03-22T01:34:21.191699+0000","last_undegraded":"2024-03-22T01:34:21.191699+0000","last_fullsized":"2024-03-22T01:34:21.191699+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:04:32.569934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.599930+0000","last_change":"2024-03-22T01:34:15.489213+0000","last_active":"2024-03-22T01:34:20.599930+0000","last_peered":"2024-03-22T01:34:20.599930+0000","last_clean":"2024-03-22T01:34:20.599930+0000","last_became_active":"2024-03-22T01:34:15.488887+0000","last_became_peered":"2024-03-22T01:34:15.488887+0000","last_unstale":"2024-03-22T01:34:20.599930+0000","last_undegraded":"2024-03-22T01:34:20.599930+0000","last_fullsized":"2024-03-22T01:34:20.599930+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:53:39.983144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:20.990185+0000","last_change":"2024-03-22T01:34:15.485632+0000","last_active":"2024-03-22T01:34:20.990185+0000","last_peered":"2024-03-22T01:34:20.990185+0000","last_clean":"2024-03-22T01:34:20.990185+0000","last_became_active":"2024-03-22T01:34:15.485341+0000","last_became_peered":"2024-03-22T01:34:15.485341+0000","last_unstale":"2024-03-22T01:34:20.990185+0000","last_undegraded":"2024-03-22T01:34:20.990185+0000","last_fullsized":"2024-03-22T01:34:20.990185+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:33:00.248555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-22T01:34:21.191597+0000","last_change":"2024-03-22T01:34:15.483593+0000","last_active":"2024-03-22T01:34:21.191597+0000","last_peered":"2024-03-22T01:34:21.191597+0000","last_clean":"2024-03-22T01:34:21.191597+0000","last_became_active":"2024-03-22T01:34:15.483201+0000","last_became_peered":"2024-03-22T01:34:15.483201+0000","last_unstale":"2024-03-22T01:34:21.191597+0000","last_undegraded":"2024-03-22T01:34:21.191597+0000","last_fullsized":"2024-03-22T01:34:21.191597+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:03:10.955120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":10,"num_read_kb":12,"num_write":53,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":50549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45,"ondisk_log_size":45,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318290,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1003520,"data_stored":486921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579927,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36920,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708232,"statfs":{"total":95995035648,"available":95957229568,"internally_reserved":0,"allocated":1622016,"data_stored":1081325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776157,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":495695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037793,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1011712,"data_stored":487525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299431,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28584,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716568,"statfs":{"total":95995035648,"available":95965765632,"internally_reserved":0,"allocated":1605632,"data_stored":1081967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462955,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1003520,"data_stored":486921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574915,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":490763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738440,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1007616,"data_stored":487011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411391,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1056768,"data_stored":527426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149754,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1011712,"data_stored":487025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888118,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1007616,"data_stored":486943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724592,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36920,"kb_used_data":1572,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708232,"statfs":{"total":95995035648,"available":95957229568,"internally_reserved":0,"allocated":1609728,"data_stored":1077471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":40505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-22T01:34:43.565 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-22T01:34:43.566 INFO:tasks.cephadm:Setting up client nodes... 2024-03-22T01:34:43.566 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-22T01:34:43.844 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:34:43.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:43 smithi089 ceph-mon[35770]: from='client.14901 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:34:43.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:43 smithi089 ceph-mon[35770]: from='client.14907 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:34:43.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:43 smithi089 ceph-mon[35770]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-22T01:34:43.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:43 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/1244924754' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:34:43.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:43 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/693387139' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:34:43.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:43 smithi072 ceph-mon[32414]: from='client.14901 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:34:43.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:43 smithi072 ceph-mon[32414]: from='client.14907 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:34:43.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:43 smithi072 ceph-mon[32414]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-22T01:34:43.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:43 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/1244924754' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:34:43.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:43 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/693387139' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:34:43.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:43 smithi098 ceph-mon[36468]: from='client.14901 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:34:43.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:43 smithi098 ceph-mon[36468]: from='client.14907 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:34:43.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:43 smithi098 ceph-mon[36468]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-22T01:34:43.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:43 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/1244924754' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:34:43.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:43 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/693387139' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:34:44.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:44 smithi072 ceph-mon[32414]: from='client.14925 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-22T01:34:44.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:44 smithi072 ceph-mon[32414]: pgmap v305: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 426 B/s wr, 1 op/s 2024-03-22T01:34:44.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:44 smithi072 ceph-mon[32414]: from='client.14931 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-22T01:34:44.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:44 smithi089 ceph-mon[35770]: from='client.14925 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-22T01:34:44.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:44 smithi089 ceph-mon[35770]: pgmap v305: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 426 B/s wr, 1 op/s 2024-03-22T01:34:44.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:44 smithi089 ceph-mon[35770]: from='client.14931 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-22T01:34:44.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:44 smithi098 ceph-mon[36468]: from='client.14925 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-22T01:34:44.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:44 smithi098 ceph-mon[36468]: pgmap v305: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 426 B/s wr, 1 op/s 2024-03-22T01:34:44.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:44 smithi098 ceph-mon[36468]: from='client.14931 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-22T01:34:45.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:45.614+0000 7f44ea068700 1 -- 172.21.15.72:0/1140400781 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44e41015c0 msgr2=0x7f44e41019e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:45.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:45.614+0000 7f44ea068700 1 --2- 172.21.15.72:0/1140400781 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44e41015c0 0x7f44e41019e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f44d400b0d0 tx=0x7f44d400b3e0 comp rx=0 tx=0).stop 2024-03-22T01:34:45.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:45.614+0000 7f44ea068700 1 -- 172.21.15.72:0/1140400781 shutdown_connections 2024-03-22T01:34:45.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:45.614+0000 7f44ea068700 1 --2- 172.21.15.72:0/1140400781 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44e4103160 0x7f44e4109a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:45.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:45.614+0000 7f44ea068700 1 --2- 172.21.15.72:0/1140400781 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44e41015c0 0x7f44e41019e0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:45.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:45.614+0000 7f44ea068700 1 --2- 172.21.15.72:0/1140400781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44e41027c0 0x7f44e4102c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:45.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:45.614+0000 7f44ea068700 1 -- 172.21.15.72:0/1140400781 >> 172.21.15.72:0/1140400781 conn(0x7f44e40fcb60 msgr2=0x7f44e40fefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:45.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:45.614+0000 7f44ea068700 1 -- 172.21.15.72:0/1140400781 shutdown_connections 2024-03-22T01:34:45.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:45.614+0000 7f44ea068700 1 -- 172.21.15.72:0/1140400781 wait complete. 2024-03-22T01:34:45.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:45.614+0000 7f44ea068700 1 Processor -- start 2024-03-22T01:34:45.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:45.615+0000 7f44ea068700 1 -- start start 2024-03-22T01:34:45.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:45.615+0000 7f44ea068700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44e41015c0 0x7f44e4197760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:45.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:45.615+0000 7f44ea068700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44e41027c0 0x7f44e4197ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:45.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:45.615+0000 7f44ea068700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44e4103160 0x7f44e419ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:45.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:45.615+0000 7f44ea068700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f44e410d200 con 0x7f44e4103160 2024-03-22T01:34:45.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:45.615+0000 7f44ea068700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f44e410d080 con 0x7f44e41027c0 2024-03-22T01:34:45.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:45.615+0000 7f44ea068700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f44e410d380 con 0x7f44e41015c0 2024-03-22T01:34:45.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:45.615+0000 7f44e3fff700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44e4103160 0x7f44e419ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:45.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:45.615+0000 7f44e3fff700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44e4103160 0x7f44e419ed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37506/0 (socket says 172.21.15.72:37506) 2024-03-22T01:34:45.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:45.615+0000 7f44e3fff700 1 -- 172.21.15.72:0/482953384 learned_addr learned my addr 172.21.15.72:0/482953384 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:34:45.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:45.615+0000 7f44e2ffd700 1 --2- 172.21.15.72:0/482953384 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44e41027c0 0x7f44e4197ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:45.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:45.615+0000 7f44e37fe700 1 --2- 172.21.15.72:0/482953384 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44e41015c0 0x7f44e4197760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:45.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:45.615+0000 7f44e3fff700 1 -- 172.21.15.72:0/482953384 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44e41015c0 msgr2=0x7f44e4197760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:45.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:45.615+0000 7f44e3fff700 1 --2- 172.21.15.72:0/482953384 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44e41015c0 0x7f44e4197760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:45.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:45.615+0000 7f44e3fff700 1 -- 172.21.15.72:0/482953384 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44e41027c0 msgr2=0x7f44e4197ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:45.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:45.615+0000 7f44e3fff700 1 --2- 172.21.15.72:0/482953384 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44e41027c0 0x7f44e4197ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:45.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:45.615+0000 7f44e3fff700 1 -- 172.21.15.72:0/482953384 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44e419f3c0 con 0x7f44e4103160 2024-03-22T01:34:45.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:45.615+0000 7f44e2ffd700 1 --2- 172.21.15.72:0/482953384 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44e41027c0 0x7f44e4197ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:34:45.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:45.616+0000 7f44e3fff700 1 --2- 172.21.15.72:0/482953384 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44e4103160 0x7f44e419ed20 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f44dc00e970 tx=0x7f44dc00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:45.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:45.616+0000 7f44e0ff9700 1 -- 172.21.15.72:0/482953384 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44dc00cbd0 con 0x7f44e4103160 2024-03-22T01:34:45.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:45.616+0000 7f44ea068700 1 -- 172.21.15.72:0/482953384 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44e419f6b0 con 0x7f44e4103160 2024-03-22T01:34:45.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:45.616+0000 7f44e0ff9700 1 -- 172.21.15.72:0/482953384 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44dc00cd70 con 0x7f44e4103160 2024-03-22T01:34:45.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:45.616+0000 7f44e0ff9700 1 -- 172.21.15.72:0/482953384 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44dc005690 con 0x7f44e4103160 2024-03-22T01:34:45.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:45.616+0000 7f44ea068700 1 -- 172.21.15.72:0/482953384 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44e419fbf0 con 0x7f44e4103160 2024-03-22T01:34:45.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:45.617+0000 7f44ea068700 1 -- 172.21.15.72:0/482953384 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44e4050b90 con 0x7f44e4103160 2024-03-22T01:34:45.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:45.618+0000 7f44e0ff9700 1 -- 172.21.15.72:0/482953384 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7f44dc004750 con 0x7f44e4103160 2024-03-22T01:34:45.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:45.618+0000 7f44e0ff9700 1 --2- 172.21.15.72:0/482953384 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f44cc0a30f0 0x7f44cc0a55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:45.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:45.619+0000 7f44e37fe700 1 --2- 172.21.15.72:0/482953384 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f44cc0a30f0 0x7f44cc0a55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:45.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:45.619+0000 7f44e0ff9700 1 -- 172.21.15.72:0/482953384 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f44dc0cfa40 con 0x7f44e4103160 2024-03-22T01:34:45.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:45.619+0000 7f44e37fe700 1 --2- 172.21.15.72:0/482953384 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f44cc0a30f0 0x7f44cc0a55b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f44d4000f80 tx=0x7f44d400b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:45.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:45.623+0000 7f44e0ff9700 1 -- 172.21.15.72:0/482953384 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f44dc09c4d0 con 0x7f44e4103160 2024-03-22T01:34:45.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:45.901+0000 7f44ea068700 1 -- 172.21.15.72:0/482953384 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f44e402dd70 con 0x7f44e4103160 2024-03-22T01:34:45.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:45.903+0000 7f44e0ff9700 1 -- 172.21.15.72:0/482953384 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f44dc09bdd0 con 0x7f44e4103160 2024-03-22T01:34:45.903 INFO:teuthology.orchestra.run.smithi072.stdout:[client.0] 2024-03-22T01:34:45.903 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQA14PxlndjaNRAAuRNXq228pUODx58NImSdNA== 2024-03-22T01:34:45.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:45.905+0000 7f44ea068700 1 -- 172.21.15.72:0/482953384 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f44cc0a30f0 msgr2=0x7f44cc0a55b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:45.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:45.905+0000 7f44ea068700 1 --2- 172.21.15.72:0/482953384 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f44cc0a30f0 0x7f44cc0a55b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f44d4000f80 tx=0x7f44d400b040 comp rx=0 tx=0).stop 2024-03-22T01:34:45.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:45.905+0000 7f44ea068700 1 -- 172.21.15.72:0/482953384 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44e4103160 msgr2=0x7f44e419ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:45.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:45.905+0000 7f44ea068700 1 --2- 172.21.15.72:0/482953384 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44e4103160 0x7f44e419ed20 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f44dc00e970 tx=0x7f44dc00ec80 comp rx=0 tx=0).stop 2024-03-22T01:34:45.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:45.905+0000 7f44ea068700 1 -- 172.21.15.72:0/482953384 shutdown_connections 2024-03-22T01:34:45.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:45.905+0000 7f44ea068700 1 --2- 172.21.15.72:0/482953384 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f44cc0a30f0 0x7f44cc0a55b0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:45.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:45.905+0000 7f44ea068700 1 --2- 172.21.15.72:0/482953384 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44e4103160 0x7f44e419ed20 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:45.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:45.905+0000 7f44ea068700 1 --2- 172.21.15.72:0/482953384 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44e41015c0 0x7f44e4197760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:45.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:45.905+0000 7f44ea068700 1 --2- 172.21.15.72:0/482953384 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44e41027c0 0x7f44e4197ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:45.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:45.905+0000 7f44ea068700 1 -- 172.21.15.72:0/482953384 >> 172.21.15.72:0/482953384 conn(0x7f44e40fcb60 msgr2=0x7f44e4105a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:45.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:45.905+0000 7f44ea068700 1 -- 172.21.15.72:0/482953384 shutdown_connections 2024-03-22T01:34:45.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:45.906+0000 7f44ea068700 1 -- 172.21.15.72:0/482953384 wait complete. 2024-03-22T01:34:46.704 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-22T01:34:46.704 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-22T01:34:46.704 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-22T01:34:46.751 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-22T01:34:46.751 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-22T01:34:46.751 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph mgr dump --format=json 2024-03-22T01:34:46.816 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:46 smithi072 ceph-mon[32414]: pgmap v306: 129 pgs: 129 active+clean; 622 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 4.2 KiB/s wr, 3 op/s 2024-03-22T01:34:46.816 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:46 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/482953384' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-22T01:34:46.816 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:46 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/482953384' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-22T01:34:46.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:46 smithi089 ceph-mon[35770]: pgmap v306: 129 pgs: 129 active+clean; 622 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 4.2 KiB/s wr, 3 op/s 2024-03-22T01:34:46.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:46 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/482953384' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-22T01:34:46.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:46 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/482953384' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-22T01:34:46.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:46 smithi098 ceph-mon[36468]: pgmap v306: 129 pgs: 129 active+clean; 622 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 4.2 KiB/s wr, 3 op/s 2024-03-22T01:34:46.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:46 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/482953384' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-22T01:34:46.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:46 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/482953384' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-22T01:34:47.037 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:34:48.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:48.770+0000 7f46a9f13700 1 -- 172.21.15.72:0/2008281581 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f46a4101f20 msgr2=0x7f46a410e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:48.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:48.770+0000 7f46a9f13700 1 --2- 172.21.15.72:0/2008281581 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f46a4101f20 0x7f46a410e410 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f46940099c0 tx=0x7f4694009cd0 comp rx=0 tx=0).stop 2024-03-22T01:34:48.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:48.770+0000 7f46a9f13700 1 -- 172.21.15.72:0/2008281581 shutdown_connections 2024-03-22T01:34:48.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:48.770+0000 7f46a9f13700 1 --2- 172.21.15.72:0/2008281581 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f46a4101f20 0x7f46a410e410 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:48.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:48.770+0000 7f46a9f13700 1 --2- 172.21.15.72:0/2008281581 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46a4106510 0x7f46a41068f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:48.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:48.770+0000 7f46a9f13700 1 --2- 172.21.15.72:0/2008281581 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f46a4101560 0x7f46a41019e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:48.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:48.770+0000 7f46a9f13700 1 -- 172.21.15.72:0/2008281581 >> 172.21.15.72:0/2008281581 conn(0x7f46a4078590 msgr2=0x7f46a40789a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:48.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:48.770+0000 7f46a9f13700 1 -- 172.21.15.72:0/2008281581 shutdown_connections 2024-03-22T01:34:48.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:48.770+0000 7f46a9f13700 1 -- 172.21.15.72:0/2008281581 wait complete. 2024-03-22T01:34:48.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:48.771+0000 7f46a9f13700 1 Processor -- start 2024-03-22T01:34:48.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:48.771+0000 7f46a9f13700 1 -- start start 2024-03-22T01:34:48.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:48.771+0000 7f46a9f13700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f46a4101560 0x7f46a41031a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:48.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:48.771+0000 7f46a9f13700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46a4101f20 0x7f46a41036e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:48.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:48.771+0000 7f46a9f13700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f46a4106510 0x7f46a4103c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:48.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:48.771+0000 7f46a9f13700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f46a41118e0 con 0x7f46a4101560 2024-03-22T01:34:48.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:48.771+0000 7f46a9f13700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f46a4111760 con 0x7f46a4106510 2024-03-22T01:34:48.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:48.771+0000 7f46a9f13700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f46a4111a60 con 0x7f46a4101f20 2024-03-22T01:34:48.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:48.771+0000 7f46a37fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f46a4101560 0x7f46a41031a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:48.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:48.772+0000 7f46a37fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f46a4101560 0x7f46a41031a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37518/0 (socket says 172.21.15.72:37518) 2024-03-22T01:34:48.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:48.772+0000 7f46a37fe700 1 -- 172.21.15.72:0/3727550342 learned_addr learned my addr 172.21.15.72:0/3727550342 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:34:48.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:48.772+0000 7f46a3fff700 1 --2- 172.21.15.72:0/3727550342 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f46a4106510 0x7f46a4103c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:48.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:48.772+0000 7f46a2ffd700 1 --2- 172.21.15.72:0/3727550342 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46a4101f20 0x7f46a41036e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:48.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:48.772+0000 7f46a3fff700 1 -- 172.21.15.72:0/3727550342 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46a4101f20 msgr2=0x7f46a41036e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:48.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:48.772+0000 7f46a3fff700 1 --2- 172.21.15.72:0/3727550342 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46a4101f20 0x7f46a41036e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:48.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:48.772+0000 7f46a3fff700 1 -- 172.21.15.72:0/3727550342 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f46a4101560 msgr2=0x7f46a41031a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:48.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:48.772+0000 7f46a3fff700 1 --2- 172.21.15.72:0/3727550342 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f46a4101560 0x7f46a41031a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:48.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:48.772+0000 7f46a3fff700 1 -- 172.21.15.72:0/3727550342 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46a41a73b0 con 0x7f46a4106510 2024-03-22T01:34:48.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:48.772+0000 7f46a37fe700 1 --2- 172.21.15.72:0/3727550342 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f46a4101560 0x7f46a41031a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:34:48.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:48.772+0000 7f46a3fff700 1 --2- 172.21.15.72:0/3727550342 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f46a4106510 0x7f46a4103c20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f46940098e0 tx=0x7f469400fa00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:48.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:48.772+0000 7f46a0ff9700 1 -- 172.21.15.72:0/3727550342 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4694027070 con 0x7f46a4106510 2024-03-22T01:34:48.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:48.772+0000 7f46a0ff9700 1 -- 172.21.15.72:0/3727550342 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4694003680 con 0x7f46a4106510 2024-03-22T01:34:48.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:48.772+0000 7f46a9f13700 1 -- 172.21.15.72:0/3727550342 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46a41a7640 con 0x7f46a4106510 2024-03-22T01:34:48.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:48.772+0000 7f46a0ff9700 1 -- 172.21.15.72:0/3727550342 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46940173f0 con 0x7f46a4106510 2024-03-22T01:34:48.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:48.772+0000 7f46a9f13700 1 -- 172.21.15.72:0/3727550342 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46a41a7b20 con 0x7f46a4106510 2024-03-22T01:34:48.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:48.773+0000 7f46a9f13700 1 -- 172.21.15.72:0/3727550342 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46a4069340 con 0x7f46a4106510 2024-03-22T01:34:48.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:48.774+0000 7f46a0ff9700 1 -- 172.21.15.72:0/3727550342 <== mon.2 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7f469402a070 con 0x7f46a4106510 2024-03-22T01:34:48.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:48.775+0000 7f46a0ff9700 1 --2- 172.21.15.72:0/3727550342 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f46900a3040 0x7f46900a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:48.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:48.775+0000 7f46a37fe700 1 --2- 172.21.15.72:0/3727550342 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f46900a3040 0x7f46900a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:48.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:48.775+0000 7f46a0ff9700 1 -- 172.21.15.72:0/3727550342 <== mon.2 v2:172.21.15.89:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f46940d03e0 con 0x7f46a4106510 2024-03-22T01:34:48.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:48.775+0000 7f46a37fe700 1 --2- 172.21.15.72:0/3727550342 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f46900a3040 0x7f46900a5500 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f469c0060b0 tx=0x7f469c00c040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:48.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:48.779+0000 7f46a0ff9700 1 -- 172.21.15.72:0/3727550342 <== mon.2 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f469405e740 con 0x7f46a4106510 2024-03-22T01:34:48.779 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:48 smithi072 ceph-mon[32414]: pgmap v307: 129 pgs: 129 active+clean; 622 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 4.2 KiB/s wr, 3 op/s 2024-03-22T01:34:48.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:48 smithi089 ceph-mon[35770]: pgmap v307: 129 pgs: 129 active+clean; 622 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 4.2 KiB/s wr, 3 op/s 2024-03-22T01:34:48.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:48 smithi098 ceph-mon[36468]: pgmap v307: 129 pgs: 129 active+clean; 622 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 4.2 KiB/s wr, 3 op/s 2024-03-22T01:34:49.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:49.030+0000 7f46a9f13700 1 -- 172.21.15.72:0/3727550342 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f46a402dd70 con 0x7f46a4106510 2024-03-22T01:34:49.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:49.035+0000 7f46a0ff9700 1 -- 172.21.15.72:0/3727550342 <== mon.2 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+263548 (secure 0 0 0) 0x7f4694013070 con 0x7f46a4106510 2024-03-22T01:34:49.038 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:34:49.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:49.038+0000 7f46a9f13700 1 -- 172.21.15.72:0/3727550342 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f46900a3040 msgr2=0x7f46900a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:49.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:49.038+0000 7f46a9f13700 1 --2- 172.21.15.72:0/3727550342 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f46900a3040 0x7f46900a5500 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f469c0060b0 tx=0x7f469c00c040 comp rx=0 tx=0).stop 2024-03-22T01:34:49.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:49.038+0000 7f46a9f13700 1 -- 172.21.15.72:0/3727550342 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f46a4106510 msgr2=0x7f46a4103c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:49.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:49.038+0000 7f46a9f13700 1 --2- 172.21.15.72:0/3727550342 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f46a4106510 0x7f46a4103c20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f46940098e0 tx=0x7f469400fa00 comp rx=0 tx=0).stop 2024-03-22T01:34:49.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:49.038+0000 7f46a9f13700 1 -- 172.21.15.72:0/3727550342 shutdown_connections 2024-03-22T01:34:49.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:49.038+0000 7f46a9f13700 1 --2- 172.21.15.72:0/3727550342 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f46900a3040 0x7f46900a5500 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:49.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:49.038+0000 7f46a9f13700 1 --2- 172.21.15.72:0/3727550342 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f46a4106510 0x7f46a4103c20 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:49.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:49.038+0000 7f46a9f13700 1 --2- 172.21.15.72:0/3727550342 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f46a4101560 0x7f46a41031a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:49.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:49.038+0000 7f46a9f13700 1 --2- 172.21.15.72:0/3727550342 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46a4101f20 0x7f46a41036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:49.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:49.038+0000 7f46a9f13700 1 -- 172.21.15.72:0/3727550342 >> 172.21.15.72:0/3727550342 conn(0x7f46a4078590 msgr2=0x7f46a406b180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:49.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:49.038+0000 7f46a9f13700 1 -- 172.21.15.72:0/3727550342 shutdown_connections 2024-03-22T01:34:49.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:49.038+0000 7f46a9f13700 1 -- 172.21.15.72:0/3727550342 wait complete. 2024-03-22T01:34:49.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:49 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/3727550342' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-22T01:34:49.793 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":401403200},{"type":"v1","addr":"172.21.15.72:6801","nonce":401403200}]},"active_addr":"172.21.15.72:6801/401403200","active_change":"2024-03-22T01:27:15.339933+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24106,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24107,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.72:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.72:0","nonce":2032770149}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.72:0","nonce":4223447038}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.72:0","nonce":608566205}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.72:0","nonce":1161486989}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.72:0","nonce":3252141203}]}]} 2024-03-22T01:34:49.799 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-22T01:34:49.799 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-22T01:34:49.799 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-22T01:34:49.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:49 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/3727550342' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-22T01:34:49.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:49 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/3727550342' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-22T01:34:50.041 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:34:50.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:50 smithi089 ceph-mon[35770]: pgmap v308: 129 pgs: 129 active+clean; 622 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 4.2 KiB/s wr, 3 op/s 2024-03-22T01:34:50.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:50 smithi072 ceph-mon[32414]: pgmap v308: 129 pgs: 129 active+clean; 622 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 4.2 KiB/s wr, 3 op/s 2024-03-22T01:34:50.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:50 smithi098 ceph-mon[36468]: pgmap v308: 129 pgs: 129 active+clean; 622 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 4.2 KiB/s wr, 3 op/s 2024-03-22T01:34:51.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:51.797+0000 7f71139d9700 1 -- 172.21.15.72:0/1582808275 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f710c10cd70 msgr2=0x7f710c10f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:51.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:51.797+0000 7f71139d9700 1 --2- 172.21.15.72:0/1582808275 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f710c10cd70 0x7f710c10f1a0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f710400b0a0 tx=0x7f710400b3b0 comp rx=0 tx=0).stop 2024-03-22T01:34:51.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:51.797+0000 7f71139d9700 1 -- 172.21.15.72:0/1582808275 shutdown_connections 2024-03-22T01:34:51.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:51.797+0000 7f71139d9700 1 --2- 172.21.15.72:0/1582808275 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f710c10cd70 0x7f710c10f1a0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:51.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:51.797+0000 7f71139d9700 1 --2- 172.21.15.72:0/1582808275 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f710c103990 0x7f710c103d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:51.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:51.797+0000 7f71139d9700 1 --2- 172.21.15.72:0/1582808275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f710c1042b0 0x7f710c10c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:51.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:51.797+0000 7f71139d9700 1 -- 172.21.15.72:0/1582808275 >> 172.21.15.72:0/1582808275 conn(0x7f710c0786b0 msgr2=0x7f710c078ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:51.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:51.798+0000 7f71139d9700 1 -- 172.21.15.72:0/1582808275 shutdown_connections 2024-03-22T01:34:51.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:51.798+0000 7f71139d9700 1 -- 172.21.15.72:0/1582808275 wait complete. 2024-03-22T01:34:51.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:51.798+0000 7f71139d9700 1 Processor -- start 2024-03-22T01:34:51.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:51.798+0000 7f71139d9700 1 -- start start 2024-03-22T01:34:51.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:51.798+0000 7f71139d9700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f710c103990 0x7f710c198e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:51.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:51.798+0000 7f71139d9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f710c1042b0 0x7f710c1993a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:51.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:51.798+0000 7f71139d9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f710c10cd70 0x7f710c19d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:51.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:51.798+0000 7f71139d9700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f710c112a50 con 0x7f710c103990 2024-03-22T01:34:51.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:51.798+0000 7f71139d9700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f710c1128d0 con 0x7f710c1042b0 2024-03-22T01:34:51.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:51.798+0000 7f71139d9700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f710c112bd0 con 0x7f710c10cd70 2024-03-22T01:34:51.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:51.798+0000 7f7111775700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f710c103990 0x7f710c198e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:51.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:51.798+0000 7f7111775700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f710c103990 0x7f710c198e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:53778/0 (socket says 172.21.15.72:53778) 2024-03-22T01:34:51.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:51.798+0000 7f7111775700 1 -- 172.21.15.72:0/4246338048 learned_addr learned my addr 172.21.15.72:0/4246338048 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:34:51.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:51.799+0000 7f7110f74700 1 --2- 172.21.15.72:0/4246338048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f710c1042b0 0x7f710c1993a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:51.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:51.799+0000 7f7111f76700 1 --2- 172.21.15.72:0/4246338048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f710c10cd70 0x7f710c19d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:51.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:51.799+0000 7f7111775700 1 -- 172.21.15.72:0/4246338048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f710c10cd70 msgr2=0x7f710c19d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:51.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:51.799+0000 7f7111775700 1 --2- 172.21.15.72:0/4246338048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f710c10cd70 0x7f710c19d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:51.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:51.799+0000 7f7111775700 1 -- 172.21.15.72:0/4246338048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f710c1042b0 msgr2=0x7f710c1993a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:51.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:51.799+0000 7f7111775700 1 --2- 172.21.15.72:0/4246338048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f710c1042b0 0x7f710c1993a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:51.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:51.799+0000 7f7111775700 1 -- 172.21.15.72:0/4246338048 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f710c19de20 con 0x7f710c103990 2024-03-22T01:34:51.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:51.799+0000 7f7111775700 1 --2- 172.21.15.72:0/4246338048 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f710c103990 0x7f710c198e60 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f710800e9a0 tx=0x7f710800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:51.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:51.799+0000 7f71027fc700 1 -- 172.21.15.72:0/4246338048 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f710800cbd0 con 0x7f710c103990 2024-03-22T01:34:51.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:51.799+0000 7f71139d9700 1 -- 172.21.15.72:0/4246338048 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f710c19e110 con 0x7f710c103990 2024-03-22T01:34:51.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:51.799+0000 7f71027fc700 1 -- 172.21.15.72:0/4246338048 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f710800cd70 con 0x7f710c103990 2024-03-22T01:34:51.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:51.799+0000 7f71027fc700 1 -- 172.21.15.72:0/4246338048 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7108018750 con 0x7f710c103990 2024-03-22T01:34:51.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:51.799+0000 7f71139d9700 1 -- 172.21.15.72:0/4246338048 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f710c1a5990 con 0x7f710c103990 2024-03-22T01:34:51.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:51.800+0000 7f71139d9700 1 -- 172.21.15.72:0/4246338048 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f710c069340 con 0x7f710c103990 2024-03-22T01:34:51.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:51.801+0000 7f71027fc700 1 -- 172.21.15.72:0/4246338048 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7f7108010430 con 0x7f710c103990 2024-03-22T01:34:51.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:51.802+0000 7f71027fc700 1 --2- 172.21.15.72:0/4246338048 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f70f80a3040 0x7f70f80a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:51.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:51.802+0000 7f7110f74700 1 --2- 172.21.15.72:0/4246338048 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f70f80a3040 0x7f70f80a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:51.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:51.802+0000 7f71027fc700 1 -- 172.21.15.72:0/4246338048 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f71080cff30 con 0x7f710c103990 2024-03-22T01:34:51.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:51.802+0000 7f7110f74700 1 --2- 172.21.15.72:0/4246338048 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f70f80a3040 0x7f70f80a5500 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f710c10e850 tx=0x7f70fc00b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:51.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:51.805+0000 7f71027fc700 1 -- 172.21.15.72:0/4246338048 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f710809c9c0 con 0x7f710c103990 2024-03-22T01:34:52.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:52.003+0000 7f71139d9700 1 -- 172.21.15.72:0/4246338048 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f710c1a5c20 con 0x7f710c103990 2024-03-22T01:34:52.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:52.004+0000 7f71027fc700 1 -- 172.21.15.72:0/4246338048 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24045 (secure 0 0 0) 0x7f710809c2c0 con 0x7f710c103990 2024-03-22T01:34:52.005 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:34:52.005 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":91,"fsid":"022a99bc-e7eb-11ee-95cd-87774f69a715","created":"2024-03-22T01:26:23.893672+0000","modified":"2024-03-22T01:34:19.866367+0000","last_up_change":"2024-03-22T01:33:25.223684+0000","last_in_change":"2024-03-22T01:33:05.010331+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T01:29:37.438503+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-22T01:34:12.610422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-22T01:34:13.761686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-22T01:34:17.507039+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"16c15bb0-db92-4191-9063-38b4890e81e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6803","nonce":661906791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6805","nonce":661906791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6809","nonce":661906791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6807","nonce":661906791}]},"public_addr":"172.21.15.72:6803/661906791","cluster_addr":"172.21.15.72:6805/661906791","heartbeat_back_addr":"172.21.15.72:6809/661906791","heartbeat_front_addr":"172.21.15.72:6807/661906791","state":["exists","up"]},{"osd":1,"uuid":"981290ba-97dc-496b-aa22-d51704184b10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6801","nonce":2870590550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6803","nonce":2870590550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6807","nonce":2870590550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6805","nonce":2870590550}]},"public_addr":"172.21.15.89:6801/2870590550","cluster_addr":"172.21.15.89:6803/2870590550","heartbeat_back_addr":"172.21.15.89:6807/2870590550","heartbeat_front_addr":"172.21.15.89:6805/2870590550","state":["exists","up"]},{"osd":2,"uuid":"f07ed658-f545-46a1-8edd-8b1d5d49885c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6801","nonce":3865778118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6803","nonce":3865778118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6807","nonce":3865778118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6805","nonce":3865778118}]},"public_addr":"172.21.15.98:6801/3865778118","cluster_addr":"172.21.15.98:6803/3865778118","heartbeat_back_addr":"172.21.15.98:6807/3865778118","heartbeat_front_addr":"172.21.15.98:6805/3865778118","state":["exists","up"]},{"osd":3,"uuid":"ccc4bc40-2c48-46d2-8548-0c05b44f9f1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6811","nonce":1865951395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6813","nonce":1865951395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6817","nonce":1865951395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6815","nonce":1865951395}]},"public_addr":"172.21.15.72:6811/1865951395","cluster_addr":"172.21.15.72:6813/1865951395","heartbeat_back_addr":"172.21.15.72:6817/1865951395","heartbeat_front_addr":"172.21.15.72:6815/1865951395","state":["exists","up"]},{"osd":4,"uuid":"1812439f-810a-4ed2-9395-c00c9a085a13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6809","nonce":2683760795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6811","nonce":2683760795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6815","nonce":2683760795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6813","nonce":2683760795}]},"public_addr":"172.21.15.89:6809/2683760795","cluster_addr":"172.21.15.89:6811/2683760795","heartbeat_back_addr":"172.21.15.89:6815/2683760795","heartbeat_front_addr":"172.21.15.89:6813/2683760795","state":["exists","up"]},{"osd":5,"uuid":"e78954d8-f4e3-4d58-8536-d2d59c843386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6809","nonce":2635838572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6811","nonce":2635838572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6815","nonce":2635838572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6813","nonce":2635838572}]},"public_addr":"172.21.15.98:6809/2635838572","cluster_addr":"172.21.15.98:6811/2635838572","heartbeat_back_addr":"172.21.15.98:6815/2635838572","heartbeat_front_addr":"172.21.15.98:6813/2635838572","state":["exists","up"]},{"osd":6,"uuid":"dc6c8be2-b64c-4959-8015-7b0ea6d25599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6819","nonce":3158982661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6821","nonce":3158982661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6825","nonce":3158982661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6823","nonce":3158982661}]},"public_addr":"172.21.15.72:6819/3158982661","cluster_addr":"172.21.15.72:6821/3158982661","heartbeat_back_addr":"172.21.15.72:6825/3158982661","heartbeat_front_addr":"172.21.15.72:6823/3158982661","state":["exists","up"]},{"osd":7,"uuid":"7a73469f-cf3a-4306-ae93-ea46e21ebbbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6817","nonce":3075913234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6819","nonce":3075913234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6823","nonce":3075913234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6821","nonce":3075913234}]},"public_addr":"172.21.15.89:6817/3075913234","cluster_addr":"172.21.15.89:6819/3075913234","heartbeat_back_addr":"172.21.15.89:6823/3075913234","heartbeat_front_addr":"172.21.15.89:6821/3075913234","state":["exists","up"]},{"osd":8,"uuid":"c4b13bee-ea4d-4344-95fb-2622ecbb8196","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6817","nonce":1813255577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6819","nonce":1813255577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6823","nonce":1813255577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6821","nonce":1813255577}]},"public_addr":"172.21.15.98:6817/1813255577","cluster_addr":"172.21.15.98:6819/1813255577","heartbeat_back_addr":"172.21.15.98:6823/1813255577","heartbeat_front_addr":"172.21.15.98:6821/1813255577","state":["exists","up"]},{"osd":9,"uuid":"eefb0a44-4a36-4e55-9d04-0501bf718db6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6827","nonce":3377749206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6829","nonce":3377749206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6832","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6833","nonce":3377749206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6831","nonce":3377749206}]},"public_addr":"172.21.15.72:6827/3377749206","cluster_addr":"172.21.15.72:6829/3377749206","heartbeat_back_addr":"172.21.15.72:6833/3377749206","heartbeat_front_addr":"172.21.15.72:6831/3377749206","state":["exists","up"]},{"osd":10,"uuid":"8741d25f-40dd-40d7-b040-5cc5e411eb98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6825","nonce":885058786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6827","nonce":885058786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6831","nonce":885058786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6829","nonce":885058786}]},"public_addr":"172.21.15.89:6825/885058786","cluster_addr":"172.21.15.89:6827/885058786","heartbeat_back_addr":"172.21.15.89:6831/885058786","heartbeat_front_addr":"172.21.15.89:6829/885058786","state":["exists","up"]},{"osd":11,"uuid":"8d02e957-eda4-4184-b5c6-5eb3b8b38060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6825","nonce":1517095838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6827","nonce":1517095838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6831","nonce":1517095838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6829","nonce":1517095838}]},"public_addr":"172.21.15.98:6825/1517095838","cluster_addr":"172.21.15.98:6827/1517095838","heartbeat_back_addr":"172.21.15.98:6831/1517095838","heartbeat_front_addr":"172.21.15.98:6829/1517095838","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:28:47.307652+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:11.037135+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:34.619188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:59.350738+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:30:23.787337+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:30:48.994154+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:31:13.690350+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:31:39.628116+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:04.723912+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:30.689741+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:57.229508+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:33:23.154537+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:0/121087196":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/2451857761":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/1228477995":"2024-03-23T01:27:15.339809+0000","172.21.15.72:6801/1513232913":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/3558958264":"2024-03-23T01:26:50.074871+0000","172.21.15.72:0/1510187198":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6800/3518012916":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6801/3518012916":"2024-03-23T01:26:50.074871+0000","172.21.15.72:0/1372801041":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6800/1513232913":"2024-03-23T01:27:15.339809+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T01:34:52.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:52.006+0000 7f71139d9700 1 -- 172.21.15.72:0/4246338048 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f70f80a3040 msgr2=0x7f70f80a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:52.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:52.007+0000 7f71139d9700 1 --2- 172.21.15.72:0/4246338048 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f70f80a3040 0x7f70f80a5500 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f710c10e850 tx=0x7f70fc00b3c0 comp rx=0 tx=0).stop 2024-03-22T01:34:52.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:52.007+0000 7f71139d9700 1 -- 172.21.15.72:0/4246338048 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f710c103990 msgr2=0x7f710c198e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:52.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:52.007+0000 7f71139d9700 1 --2- 172.21.15.72:0/4246338048 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f710c103990 0x7f710c198e60 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f710800e9a0 tx=0x7f710800ecb0 comp rx=0 tx=0).stop 2024-03-22T01:34:52.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:52.007+0000 7f71139d9700 1 -- 172.21.15.72:0/4246338048 shutdown_connections 2024-03-22T01:34:52.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:52.007+0000 7f71139d9700 1 --2- 172.21.15.72:0/4246338048 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f70f80a3040 0x7f70f80a5500 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:52.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:52.007+0000 7f71139d9700 1 --2- 172.21.15.72:0/4246338048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f710c10cd70 0x7f710c19d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:52.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:52.007+0000 7f71139d9700 1 --2- 172.21.15.72:0/4246338048 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f710c103990 0x7f710c198e60 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:52.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:52.007+0000 7f71139d9700 1 --2- 172.21.15.72:0/4246338048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f710c1042b0 0x7f710c1993a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:52.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:52.007+0000 7f71139d9700 1 -- 172.21.15.72:0/4246338048 >> 172.21.15.72:0/4246338048 conn(0x7f710c0786b0 msgr2=0x7f710c0fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:52.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:52.007+0000 7f71139d9700 1 -- 172.21.15.72:0/4246338048 shutdown_connections 2024-03-22T01:34:52.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:52.007+0000 7f71139d9700 1 -- 172.21.15.72:0/4246338048 wait complete. 2024-03-22T01:34:52.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:52 smithi072 ceph-mon[32414]: pgmap v309: 129 pgs: 129 active+clean; 638 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 5.9 KiB/s wr, 5 op/s 2024-03-22T01:34:52.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:52 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/4246338048' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:34:52.729 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-22T01:34:52.729 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-22T01:34:52.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:52 smithi089 ceph-mon[35770]: pgmap v309: 129 pgs: 129 active+clean; 638 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 5.9 KiB/s wr, 5 op/s 2024-03-22T01:34:52.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:52 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/4246338048' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:34:52.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:52 smithi098 ceph-mon[36468]: pgmap v309: 129 pgs: 129 active+clean; 638 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 5.9 KiB/s wr, 5 op/s 2024-03-22T01:34:52.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:52 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/4246338048' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:34:52.971 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:34:54.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:54.795+0000 7ff89ae55700 1 -- 172.21.15.72:0/1715370297 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff8941075f0 msgr2=0x7ff8941079d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:54.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:54.795+0000 7ff89ae55700 1 --2- 172.21.15.72:0/1715370297 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff8941075f0 0x7ff8941079d0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7ff88800b0d0 tx=0x7ff88800b3e0 comp rx=0 tx=0).stop 2024-03-22T01:34:54.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:54.795+0000 7ff89ae55700 1 -- 172.21.15.72:0/1715370297 shutdown_connections 2024-03-22T01:34:54.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:54.795+0000 7ff89ae55700 1 --2- 172.21.15.72:0/1715370297 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff894101f70 0x7ff89410e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:54.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:54.795+0000 7ff89ae55700 1 --2- 172.21.15.72:0/1715370297 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff8941075f0 0x7ff8941079d0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:54.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:54.795+0000 7ff89ae55700 1 --2- 172.21.15.72:0/1715370297 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff8941015b0 0x7ff894101a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:54.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:54.795+0000 7ff89ae55700 1 -- 172.21.15.72:0/1715370297 >> 172.21.15.72:0/1715370297 conn(0x7ff8940775d0 msgr2=0x7ff8940779e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:54.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:54.795+0000 7ff89ae55700 1 -- 172.21.15.72:0/1715370297 shutdown_connections 2024-03-22T01:34:54.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:54.795+0000 7ff89ae55700 1 -- 172.21.15.72:0/1715370297 wait complete. 2024-03-22T01:34:54.796 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:54 smithi072 ceph-mon[32414]: pgmap v310: 129 pgs: 129 active+clean; 638 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 5.9 KiB/s wr, 4 op/s 2024-03-22T01:34:54.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:54.796+0000 7ff89ae55700 1 Processor -- start 2024-03-22T01:34:54.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:54.796+0000 7ff89ae55700 1 -- start start 2024-03-22T01:34:54.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:54.796+0000 7ff89ae55700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff8941015b0 0x7ff8941a3f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:54.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:54.796+0000 7ff89ae55700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff894101f70 0x7ff8941a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:54.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:54.796+0000 7ff89ae55700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8941075f0 0x7ff89419f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:54.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:54.796+0000 7ff89ae55700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7ff894111860 con 0x7ff894101f70 2024-03-22T01:34:54.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:54.796+0000 7ff89ae55700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff8941116e0 con 0x7ff8941015b0 2024-03-22T01:34:54.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:54.796+0000 7ff89ae55700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff8941119e0 con 0x7ff8941075f0 2024-03-22T01:34:54.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:54.796+0000 7ff893fff700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff894101f70 0x7ff8941a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:54.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:54.796+0000 7ff898bf1700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff8941015b0 0x7ff8941a3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:54.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:54.797+0000 7ff893fff700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff894101f70 0x7ff8941a44d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:53798/0 (socket says 172.21.15.72:53798) 2024-03-22T01:34:54.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:54.797+0000 7ff898bf1700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff8941015b0 0x7ff8941a3f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.72:41468/0 (socket says 172.21.15.72:41468) 2024-03-22T01:34:54.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:54.797+0000 7ff893fff700 1 -- 172.21.15.72:0/939017714 learned_addr learned my addr 172.21.15.72:0/939017714 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:34:54.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:54.797+0000 7ff8993f2700 1 --2- 172.21.15.72:0/939017714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8941075f0 0x7ff89419f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:54.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:54.797+0000 7ff898bf1700 1 -- 172.21.15.72:0/939017714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8941075f0 msgr2=0x7ff89419f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:54.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:54.797+0000 7ff898bf1700 1 --2- 172.21.15.72:0/939017714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8941075f0 0x7ff89419f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:54.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:54.797+0000 7ff898bf1700 1 -- 172.21.15.72:0/939017714 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff894101f70 msgr2=0x7ff8941a44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:54.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:54.797+0000 7ff898bf1700 1 --2- 172.21.15.72:0/939017714 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff894101f70 0x7ff8941a44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:54.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:54.797+0000 7ff898bf1700 1 -- 172.21.15.72:0/939017714 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff89419fce0 con 0x7ff8941015b0 2024-03-22T01:34:54.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:54.797+0000 7ff8993f2700 1 --2- 172.21.15.72:0/939017714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8941075f0 0x7ff89419f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T01:34:54.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:54.797+0000 7ff898bf1700 1 --2- 172.21.15.72:0/939017714 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff8941015b0 0x7ff8941a3f90 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff88800d010 tx=0x7ff888007a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:54.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:54.797+0000 7ff891ffb700 1 -- 172.21.15.72:0/939017714 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff888003c00 con 0x7ff8941015b0 2024-03-22T01:34:54.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:54.797+0000 7ff89ae55700 1 -- 172.21.15.72:0/939017714 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff89419ff70 con 0x7ff8941015b0 2024-03-22T01:34:54.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:54.797+0000 7ff891ffb700 1 -- 172.21.15.72:0/939017714 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff888003da0 con 0x7ff8941015b0 2024-03-22T01:34:54.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:54.797+0000 7ff891ffb700 1 -- 172.21.15.72:0/939017714 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff88801b930 con 0x7ff8941015b0 2024-03-22T01:34:54.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:54.797+0000 7ff89ae55700 1 -- 172.21.15.72:0/939017714 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8941abec0 con 0x7ff8941015b0 2024-03-22T01:34:54.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:54.798+0000 7ff89ae55700 1 -- 172.21.15.72:0/939017714 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff894069340 con 0x7ff8941015b0 2024-03-22T01:34:54.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:54.800+0000 7ff891ffb700 1 -- 172.21.15.72:0/939017714 <== mon.2 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7ff888005670 con 0x7ff8941015b0 2024-03-22T01:34:54.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:54.800+0000 7ff891ffb700 1 --2- 172.21.15.72:0/939017714 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7ff8840a3040 0x7ff8840a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:34:54.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:54.800+0000 7ff893fff700 1 --2- 172.21.15.72:0/939017714 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7ff8840a3040 0x7ff8840a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:34:54.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:54.800+0000 7ff891ffb700 1 -- 172.21.15.72:0/939017714 <== mon.2 v2:172.21.15.89:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7ff88800f020 con 0x7ff8941015b0 2024-03-22T01:34:54.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:54.801+0000 7ff893fff700 1 --2- 172.21.15.72:0/939017714 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7ff8840a3040 0x7ff8840a5500 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff880009680 tx=0x7ff880009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:34:54.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:54.809+0000 7ff891ffb700 1 -- 172.21.15.72:0/939017714 <== mon.2 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff88809d440 con 0x7ff8941015b0 2024-03-22T01:34:54.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:54 smithi089 ceph-mon[35770]: pgmap v310: 129 pgs: 129 active+clean; 638 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 5.9 KiB/s wr, 4 op/s 2024-03-22T01:34:54.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:54 smithi098 ceph-mon[36468]: pgmap v310: 129 pgs: 129 active+clean; 638 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 5.9 KiB/s wr, 4 op/s 2024-03-22T01:34:55.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:55.005+0000 7ff89ae55700 1 -- 172.21.15.72:0/939017714 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff8941ac090 con 0x7ff8941015b0 2024-03-22T01:34:55.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:55.006+0000 7ff891ffb700 1 -- 172.21.15.72:0/939017714 <== mon.2 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24045 (secure 0 0 0) 0x7ff88809cd40 con 0x7ff8941015b0 2024-03-22T01:34:55.006 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:34:55.006 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":91,"fsid":"022a99bc-e7eb-11ee-95cd-87774f69a715","created":"2024-03-22T01:26:23.893672+0000","modified":"2024-03-22T01:34:19.866367+0000","last_up_change":"2024-03-22T01:33:25.223684+0000","last_in_change":"2024-03-22T01:33:05.010331+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T01:29:37.438503+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-22T01:34:12.610422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-22T01:34:13.761686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-22T01:34:17.507039+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"16c15bb0-db92-4191-9063-38b4890e81e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6803","nonce":661906791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6805","nonce":661906791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6809","nonce":661906791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6807","nonce":661906791}]},"public_addr":"172.21.15.72:6803/661906791","cluster_addr":"172.21.15.72:6805/661906791","heartbeat_back_addr":"172.21.15.72:6809/661906791","heartbeat_front_addr":"172.21.15.72:6807/661906791","state":["exists","up"]},{"osd":1,"uuid":"981290ba-97dc-496b-aa22-d51704184b10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6801","nonce":2870590550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6803","nonce":2870590550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6807","nonce":2870590550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6805","nonce":2870590550}]},"public_addr":"172.21.15.89:6801/2870590550","cluster_addr":"172.21.15.89:6803/2870590550","heartbeat_back_addr":"172.21.15.89:6807/2870590550","heartbeat_front_addr":"172.21.15.89:6805/2870590550","state":["exists","up"]},{"osd":2,"uuid":"f07ed658-f545-46a1-8edd-8b1d5d49885c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6801","nonce":3865778118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6803","nonce":3865778118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6807","nonce":3865778118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6805","nonce":3865778118}]},"public_addr":"172.21.15.98:6801/3865778118","cluster_addr":"172.21.15.98:6803/3865778118","heartbeat_back_addr":"172.21.15.98:6807/3865778118","heartbeat_front_addr":"172.21.15.98:6805/3865778118","state":["exists","up"]},{"osd":3,"uuid":"ccc4bc40-2c48-46d2-8548-0c05b44f9f1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6811","nonce":1865951395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6813","nonce":1865951395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6817","nonce":1865951395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6815","nonce":1865951395}]},"public_addr":"172.21.15.72:6811/1865951395","cluster_addr":"172.21.15.72:6813/1865951395","heartbeat_back_addr":"172.21.15.72:6817/1865951395","heartbeat_front_addr":"172.21.15.72:6815/1865951395","state":["exists","up"]},{"osd":4,"uuid":"1812439f-810a-4ed2-9395-c00c9a085a13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6809","nonce":2683760795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6811","nonce":2683760795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6815","nonce":2683760795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6813","nonce":2683760795}]},"public_addr":"172.21.15.89:6809/2683760795","cluster_addr":"172.21.15.89:6811/2683760795","heartbeat_back_addr":"172.21.15.89:6815/2683760795","heartbeat_front_addr":"172.21.15.89:6813/2683760795","state":["exists","up"]},{"osd":5,"uuid":"e78954d8-f4e3-4d58-8536-d2d59c843386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6809","nonce":2635838572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6811","nonce":2635838572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6815","nonce":2635838572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6813","nonce":2635838572}]},"public_addr":"172.21.15.98:6809/2635838572","cluster_addr":"172.21.15.98:6811/2635838572","heartbeat_back_addr":"172.21.15.98:6815/2635838572","heartbeat_front_addr":"172.21.15.98:6813/2635838572","state":["exists","up"]},{"osd":6,"uuid":"dc6c8be2-b64c-4959-8015-7b0ea6d25599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6819","nonce":3158982661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6821","nonce":3158982661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6825","nonce":3158982661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6823","nonce":3158982661}]},"public_addr":"172.21.15.72:6819/3158982661","cluster_addr":"172.21.15.72:6821/3158982661","heartbeat_back_addr":"172.21.15.72:6825/3158982661","heartbeat_front_addr":"172.21.15.72:6823/3158982661","state":["exists","up"]},{"osd":7,"uuid":"7a73469f-cf3a-4306-ae93-ea46e21ebbbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6817","nonce":3075913234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6819","nonce":3075913234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6823","nonce":3075913234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6821","nonce":3075913234}]},"public_addr":"172.21.15.89:6817/3075913234","cluster_addr":"172.21.15.89:6819/3075913234","heartbeat_back_addr":"172.21.15.89:6823/3075913234","heartbeat_front_addr":"172.21.15.89:6821/3075913234","state":["exists","up"]},{"osd":8,"uuid":"c4b13bee-ea4d-4344-95fb-2622ecbb8196","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6817","nonce":1813255577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6819","nonce":1813255577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6823","nonce":1813255577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6821","nonce":1813255577}]},"public_addr":"172.21.15.98:6817/1813255577","cluster_addr":"172.21.15.98:6819/1813255577","heartbeat_back_addr":"172.21.15.98:6823/1813255577","heartbeat_front_addr":"172.21.15.98:6821/1813255577","state":["exists","up"]},{"osd":9,"uuid":"eefb0a44-4a36-4e55-9d04-0501bf718db6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6827","nonce":3377749206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6829","nonce":3377749206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6832","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6833","nonce":3377749206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6831","nonce":3377749206}]},"public_addr":"172.21.15.72:6827/3377749206","cluster_addr":"172.21.15.72:6829/3377749206","heartbeat_back_addr":"172.21.15.72:6833/3377749206","heartbeat_front_addr":"172.21.15.72:6831/3377749206","state":["exists","up"]},{"osd":10,"uuid":"8741d25f-40dd-40d7-b040-5cc5e411eb98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6825","nonce":885058786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6827","nonce":885058786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6831","nonce":885058786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6829","nonce":885058786}]},"public_addr":"172.21.15.89:6825/885058786","cluster_addr":"172.21.15.89:6827/885058786","heartbeat_back_addr":"172.21.15.89:6831/885058786","heartbeat_front_addr":"172.21.15.89:6829/885058786","state":["exists","up"]},{"osd":11,"uuid":"8d02e957-eda4-4184-b5c6-5eb3b8b38060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6825","nonce":1517095838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6827","nonce":1517095838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6831","nonce":1517095838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6829","nonce":1517095838}]},"public_addr":"172.21.15.98:6825/1517095838","cluster_addr":"172.21.15.98:6827/1517095838","heartbeat_back_addr":"172.21.15.98:6831/1517095838","heartbeat_front_addr":"172.21.15.98:6829/1517095838","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:28:47.307652+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:11.037135+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:34.619188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:59.350738+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:30:23.787337+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:30:48.994154+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:31:13.690350+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:31:39.628116+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:04.723912+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:30.689741+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:57.229508+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:33:23.154537+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:0/121087196":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/2451857761":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/1228477995":"2024-03-23T01:27:15.339809+0000","172.21.15.72:6801/1513232913":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/3558958264":"2024-03-23T01:26:50.074871+0000","172.21.15.72:0/1510187198":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6800/3518012916":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6801/3518012916":"2024-03-23T01:26:50.074871+0000","172.21.15.72:0/1372801041":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6800/1513232913":"2024-03-23T01:27:15.339809+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T01:34:55.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:55.009+0000 7ff89ae55700 1 -- 172.21.15.72:0/939017714 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7ff8840a3040 msgr2=0x7ff8840a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:55.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:55.009+0000 7ff89ae55700 1 --2- 172.21.15.72:0/939017714 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7ff8840a3040 0x7ff8840a5500 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff880009680 tx=0x7ff880009260 comp rx=0 tx=0).stop 2024-03-22T01:34:55.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:55.009+0000 7ff89ae55700 1 -- 172.21.15.72:0/939017714 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff8941015b0 msgr2=0x7ff8941a3f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:34:55.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:55.009+0000 7ff89ae55700 1 --2- 172.21.15.72:0/939017714 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff8941015b0 0x7ff8941a3f90 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff88800d010 tx=0x7ff888007a20 comp rx=0 tx=0).stop 2024-03-22T01:34:55.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:55.009+0000 7ff89ae55700 1 -- 172.21.15.72:0/939017714 shutdown_connections 2024-03-22T01:34:55.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:55.009+0000 7ff89ae55700 1 --2- 172.21.15.72:0/939017714 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7ff8840a3040 0x7ff8840a5500 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:55.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:55.009+0000 7ff89ae55700 1 --2- 172.21.15.72:0/939017714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8941075f0 0x7ff89419f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:55.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:55.009+0000 7ff89ae55700 1 --2- 172.21.15.72:0/939017714 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff8941015b0 0x7ff8941a3f90 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:55.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:55.009+0000 7ff89ae55700 1 --2- 172.21.15.72:0/939017714 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff894101f70 0x7ff8941a44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:34:55.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:55.009+0000 7ff89ae55700 1 -- 172.21.15.72:0/939017714 >> 172.21.15.72:0/939017714 conn(0x7ff8940775d0 msgr2=0x7ff8941006b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:34:55.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:55.009+0000 7ff89ae55700 1 -- 172.21.15.72:0/939017714 shutdown_connections 2024-03-22T01:34:55.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:34:55.009+0000 7ff89ae55700 1 -- 172.21.15.72:0/939017714 wait complete. 2024-03-22T01:34:55.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:55 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/939017714' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:34:55.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:55 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/939017714' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:34:55.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:55 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/939017714' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:34:55.956 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-22T01:34:55.957 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-22T01:34:55.957 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-22T01:34:55.958 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-22T01:34:55.958 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-22T01:34:55.958 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-22T01:34:55.959 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-22T01:34:55.959 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-22T01:34:55.960 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph tell osd.8 flush_pg_stats 2024-03-22T01:34:55.960 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph tell osd.9 flush_pg_stats 2024-03-22T01:34:55.960 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph tell osd.10 flush_pg_stats 2024-03-22T01:34:55.961 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph tell osd.11 flush_pg_stats 2024-03-22T01:34:56.583 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:34:56.700 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:34:56.700 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:34:56.700 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:34:56.701 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:34:56.711 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:34:56.717 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:34:56.717 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:34:56.723 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:34:56.736 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:34:56.740 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:34:56.837 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:56 smithi072 ceph-mon[32414]: pgmap v311: 129 pgs: 129 active+clean; 638 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 5.7 KiB/s wr, 4 op/s 2024-03-22T01:34:56.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:56 smithi089 ceph-mon[35770]: pgmap v311: 129 pgs: 129 active+clean; 638 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 5.7 KiB/s wr, 4 op/s 2024-03-22T01:34:56.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:56 smithi098 ceph-mon[36468]: pgmap v311: 129 pgs: 129 active+clean; 638 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 5.7 KiB/s wr, 4 op/s 2024-03-22T01:34:58.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:34:58 smithi072 ceph-mon[32414]: pgmap v312: 129 pgs: 129 active+clean; 638 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.8 KiB/s wr, 2 op/s 2024-03-22T01:34:58.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:34:58 smithi089 ceph-mon[35770]: pgmap v312: 129 pgs: 129 active+clean; 638 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.8 KiB/s wr, 2 op/s 2024-03-22T01:34:58.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:34:58 smithi098 ceph-mon[36468]: pgmap v312: 129 pgs: 129 active+clean; 638 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.8 KiB/s wr, 2 op/s 2024-03-22T01:35:00.750 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:00 smithi072 ceph-mon[32414]: pgmap v313: 129 pgs: 129 active+clean; 638 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-22T01:35:00.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:00 smithi089 ceph-mon[35770]: pgmap v313: 129 pgs: 129 active+clean; 638 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-22T01:35:00.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:00 smithi098 ceph-mon[36468]: pgmap v313: 129 pgs: 129 active+clean; 638 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-22T01:35:02.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:02 smithi089 ceph-mon[35770]: pgmap v314: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.8 KiB/s wr, 3 op/s 2024-03-22T01:35:02.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:02 smithi072 ceph-mon[32414]: pgmap v314: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.8 KiB/s wr, 3 op/s 2024-03-22T01:35:02.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:02 smithi098 ceph-mon[36468]: pgmap v314: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.8 KiB/s wr, 3 op/s 2024-03-22T01:35:04.836 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:04 smithi072 ceph-mon[32414]: pgmap v315: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2024-03-22T01:35:04.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:04 smithi089 ceph-mon[35770]: pgmap v315: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2024-03-22T01:35:04.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:04 smithi098 ceph-mon[36468]: pgmap v315: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2024-03-22T01:35:04.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:04.949+0000 7f4431d46700 1 -- 172.21.15.72:0/777485352 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f442c1086c0 msgr2=0x7f442c108aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:04.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:04.949+0000 7f4431d46700 1 --2- 172.21.15.72:0/777485352 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f442c1086c0 0x7f442c108aa0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f441c009990 tx=0x7f441c009ca0 comp rx=0 tx=0).stop 2024-03-22T01:35:04.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:04.949+0000 7f4431d46700 1 -- 172.21.15.72:0/777485352 shutdown_connections 2024-03-22T01:35:04.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:04.949+0000 7f4431d46700 1 --2- 172.21.15.72:0/777485352 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f442c101590 0x7f442c10f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:04.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:04.949+0000 7f4431d46700 1 --2- 172.21.15.72:0/777485352 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f442c1086c0 0x7f442c108aa0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:04.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:04.949+0000 7f4431d46700 1 --2- 172.21.15.72:0/777485352 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f442c100bd0 0x7f442c101050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:04.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:04.949+0000 7f4431d46700 1 -- 172.21.15.72:0/777485352 >> 172.21.15.72:0/777485352 conn(0x7f442c077b50 msgr2=0x7f442c0fef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:35:04.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:04.949+0000 7f4431d46700 1 -- 172.21.15.72:0/777485352 shutdown_connections 2024-03-22T01:35:04.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:04.949+0000 7f4431d46700 1 -- 172.21.15.72:0/777485352 wait complete. 2024-03-22T01:35:04.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:04.950+0000 7f4431d46700 1 Processor -- start 2024-03-22T01:35:04.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:04.950+0000 7f4431d46700 1 -- start start 2024-03-22T01:35:04.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:04.950+0000 7f4431d46700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f442c100bd0 0x7f442c198d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:04.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:04.950+0000 7f4431d46700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f442c101590 0x7f442c1992b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:04.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:04.950+0000 7f4431d46700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f442c1086c0 0x7f442c19d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:04.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:04.950+0000 7f4431d46700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f442c112980 con 0x7f442c101590 2024-03-22T01:35:04.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:04.950+0000 7f4431d46700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f442c112800 con 0x7f442c100bd0 2024-03-22T01:35:04.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:04.950+0000 7f4431d46700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f442c112b00 con 0x7f442c1086c0 2024-03-22T01:35:04.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:04.950+0000 7f4422ffd700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f442c101590 0x7f442c1992b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:04.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:04.950+0000 7f442b7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f442c100bd0 0x7f442c198d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:04.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:04.950+0000 7f4422ffd700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f442c101590 0x7f442c1992b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33074/0 (socket says 172.21.15.72:33074) 2024-03-22T01:35:04.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:04.950+0000 7f4422ffd700 1 -- 172.21.15.72:0/3410677080 learned_addr learned my addr 172.21.15.72:0/3410677080 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:35:04.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:04.950+0000 7f442bfff700 1 --2- 172.21.15.72:0/3410677080 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f442c1086c0 0x7f442c19d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:04.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:04.951+0000 7f442b7fe700 1 -- 172.21.15.72:0/3410677080 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f442c1086c0 msgr2=0x7f442c19d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:04.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:04.951+0000 7f442b7fe700 1 --2- 172.21.15.72:0/3410677080 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f442c1086c0 0x7f442c19d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:04.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:04.951+0000 7f442b7fe700 1 -- 172.21.15.72:0/3410677080 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f442c101590 msgr2=0x7f442c1992b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:04.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:04.951+0000 7f442b7fe700 1 --2- 172.21.15.72:0/3410677080 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f442c101590 0x7f442c1992b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:04.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:04.951+0000 7f442b7fe700 1 -- 172.21.15.72:0/3410677080 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f442c19dd30 con 0x7f442c100bd0 2024-03-22T01:35:04.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:04.951+0000 7f4422ffd700 1 --2- 172.21.15.72:0/3410677080 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f442c101590 0x7f442c1992b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:35:04.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:04.951+0000 7f442b7fe700 1 --2- 172.21.15.72:0/3410677080 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f442c100bd0 0x7f442c198d70 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f441c0059d0 tx=0x7f441c00bab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:04.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:04.951+0000 7f44297fa700 1 -- 172.21.15.72:0/3410677080 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f441c027070 con 0x7f442c100bd0 2024-03-22T01:35:04.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:04.951+0000 7f44297fa700 1 -- 172.21.15.72:0/3410677080 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f441c003d70 con 0x7f442c100bd0 2024-03-22T01:35:04.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:04.951+0000 7f4431d46700 1 -- 172.21.15.72:0/3410677080 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f442c19dfc0 con 0x7f442c100bd0 2024-03-22T01:35:04.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:04.951+0000 7f44297fa700 1 -- 172.21.15.72:0/3410677080 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f441c00f460 con 0x7f442c100bd0 2024-03-22T01:35:04.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:04.951+0000 7f4431d46700 1 -- 172.21.15.72:0/3410677080 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f442c1a58a0 con 0x7f442c100bd0 2024-03-22T01:35:04.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:04.952+0000 7f4431d46700 1 -- 172.21.15.72:0/3410677080 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f442c062e10 con 0x7f442c100bd0 2024-03-22T01:35:04.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:04.953+0000 7f44297fa700 1 -- 172.21.15.72:0/3410677080 <== mon.2 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7f441c02a070 con 0x7f442c100bd0 2024-03-22T01:35:04.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:04.954+0000 7f44297fa700 1 --2- 172.21.15.72:0/3410677080 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f44180a30c0 0x7f44180a5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:04.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:04.954+0000 7f4422ffd700 1 --2- 172.21.15.72:0/3410677080 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f44180a30c0 0x7f44180a5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:04.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:04.954+0000 7f44297fa700 1 -- 172.21.15.72:0/3410677080 <== mon.2 v2:172.21.15.89:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f441c0d10c0 con 0x7f442c100bd0 2024-03-22T01:35:04.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:04.954+0000 7f44297fa700 1 --2- 172.21.15.72:0/3410677080 >> [v2:172.21.15.89:6816/3075913234,v1:172.21.15.89:6817/3075913234] conn(0x7f44180add80 0x7f44180b01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:04.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:04.954+0000 7f44297fa700 1 -- 172.21.15.72:0/3410677080 --> [v2:172.21.15.89:6816/3075913234,v1:172.21.15.89:6817/3075913234] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f441c0043d0 con 0x7f44180add80 2024-03-22T01:35:04.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:04.954+0000 7f44297fa700 1 -- 172.21.15.72:0/3410677080 <== mon.2 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f441c0d1470 con 0x7f442c100bd0 2024-03-22T01:35:04.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:04.954+0000 7f442bfff700 1 --2- 172.21.15.72:0/3410677080 >> [v2:172.21.15.89:6816/3075913234,v1:172.21.15.89:6817/3075913234] conn(0x7f44180add80 0x7f44180b01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:04.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:04.954+0000 7f4422ffd700 1 --2- 172.21.15.72:0/3410677080 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f44180a30c0 0x7f44180a5580 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f442c0fe760 tx=0x7f4414009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:04.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:04.955+0000 7f442bfff700 1 --2- 172.21.15.72:0/3410677080 >> [v2:172.21.15.89:6816/3075913234,v1:172.21.15.89:6817/3075913234] conn(0x7f44180add80 0x7f44180b01c0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:04.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:04.955+0000 7f44297fa700 1 -- 172.21.15.72:0/3410677080 <== osd.7 v2:172.21.15.89:6816/3075913234 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f441c0043d0 con 0x7f44180add80 2024-03-22T01:35:04.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:04.972+0000 7f4431d46700 1 -- 172.21.15.72:0/3410677080 --> [v2:172.21.15.89:6816/3075913234,v1:172.21.15.89:6817/3075913234] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f442c063210 con 0x7f44180add80 2024-03-22T01:35:04.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:04.973+0000 7f44297fa700 1 -- 172.21.15.72:0/3410677080 <== osd.7 v2:172.21.15.89:6816/3075913234 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f442c063210 con 0x7f44180add80 2024-03-22T01:35:04.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:04.973+0000 7f4431d46700 1 -- 172.21.15.72:0/3410677080 >> [v2:172.21.15.89:6816/3075913234,v1:172.21.15.89:6817/3075913234] conn(0x7f44180add80 msgr2=0x7f44180b01c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:04.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:04.973+0000 7f4431d46700 1 --2- 172.21.15.72:0/3410677080 >> [v2:172.21.15.89:6816/3075913234,v1:172.21.15.89:6817/3075913234] conn(0x7f44180add80 0x7f44180b01c0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:04.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:04.973+0000 7f4431d46700 1 -- 172.21.15.72:0/3410677080 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f44180a30c0 msgr2=0x7f44180a5580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:04.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:04.973+0000 7f4431d46700 1 --2- 172.21.15.72:0/3410677080 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f44180a30c0 0x7f44180a5580 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f442c0fe760 tx=0x7f4414009260 comp rx=0 tx=0).stop 2024-03-22T01:35:04.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:04.973+0000 7f4431d46700 1 -- 172.21.15.72:0/3410677080 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f442c100bd0 msgr2=0x7f442c198d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:04.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:04.973+0000 7f4431d46700 1 --2- 172.21.15.72:0/3410677080 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f442c100bd0 0x7f442c198d70 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f441c0059d0 tx=0x7f441c00bab0 comp rx=0 tx=0).stop 2024-03-22T01:35:04.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:04.973+0000 7f442bfff700 1 -- 172.21.15.72:0/3410677080 reap_dead start 2024-03-22T01:35:04.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:04.973+0000 7f4431d46700 1 -- 172.21.15.72:0/3410677080 shutdown_connections 2024-03-22T01:35:04.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:04.973+0000 7f4431d46700 1 -- 172.21.15.72:0/3410677080 >> 172.21.15.72:0/3410677080 conn(0x7f442c077b50 msgr2=0x7f442c0ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:35:04.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:04.973+0000 7f4431d46700 1 -- 172.21.15.72:0/3410677080 shutdown_connections 2024-03-22T01:35:04.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:04.973+0000 7f4431d46700 1 -- 172.21.15.72:0/3410677080 wait complete. 2024-03-22T01:35:05.808 INFO:teuthology.orchestra.run.smithi072.stdout:223338299436 2024-03-22T01:35:05.808 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-22T01:35:06.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:06 smithi072 ceph-mon[32414]: pgmap v316: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 170 B/s wr, 2 op/s 2024-03-22T01:35:06.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:06 smithi089 ceph-mon[35770]: pgmap v316: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 170 B/s wr, 2 op/s 2024-03-22T01:35:06.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:06 smithi098 ceph-mon[36468]: pgmap v316: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 170 B/s wr, 2 op/s 2024-03-22T01:35:08.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:08 smithi089 ceph-mon[35770]: pgmap v317: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2024-03-22T01:35:08.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:08 smithi072 ceph-mon[32414]: pgmap v317: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2024-03-22T01:35:08.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:08 smithi098 ceph-mon[36468]: pgmap v317: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2024-03-22T01:35:09.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:07.528+0000 7f8a49c13700 1 -- 172.21.15.72:0/2628864033 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a441026a0 msgr2=0x7f8a44102b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:09.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:07.528+0000 7f8a49c13700 1 --2- 172.21.15.72:0/2628864033 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a441026a0 0x7f8a44102b20 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8a380098f0 tx=0x7f8a38009c00 comp rx=0 tx=0).stop 2024-03-22T01:35:09.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:07.528+0000 7f8a49c13700 1 -- 172.21.15.72:0/2628864033 shutdown_connections 2024-03-22T01:35:09.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:07.528+0000 7f8a49c13700 1 --2- 172.21.15.72:0/2628864033 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8a44103060 0x7f8a4410f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:09.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:07.528+0000 7f8a49c13700 1 --2- 172.21.15.72:0/2628864033 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a441086a0 0x7f8a44108a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:09.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:07.528+0000 7f8a49c13700 1 --2- 172.21.15.72:0/2628864033 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a441026a0 0x7f8a44102b20 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:09.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:07.528+0000 7f8a49c13700 1 -- 172.21.15.72:0/2628864033 >> 172.21.15.72:0/2628864033 conn(0x7f8a440fe150 msgr2=0x7f8a44100570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:35:09.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:07.528+0000 7f8a49c13700 1 -- 172.21.15.72:0/2628864033 shutdown_connections 2024-03-22T01:35:09.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:07.528+0000 7f8a49c13700 1 -- 172.21.15.72:0/2628864033 wait complete. 2024-03-22T01:35:09.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:07.529+0000 7f8a49c13700 1 Processor -- start 2024-03-22T01:35:09.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:07.529+0000 7f8a49c13700 1 -- start start 2024-03-22T01:35:09.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:07.529+0000 7f8a49c13700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8a441026a0 0x7f8a4410f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:09.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:07.529+0000 7f8a49c13700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a44103060 0x7f8a4410c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:09.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:07.529+0000 7f8a49c13700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a441086a0 0x7f8a4410c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:09.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:07.529+0000 7f8a49c13700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f8a440761b0 con 0x7f8a441026a0 2024-03-22T01:35:09.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:07.529+0000 7f8a49c13700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8a44076030 con 0x7f8a44103060 2024-03-22T01:35:09.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:07.529+0000 7f8a49c13700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8a44076330 con 0x7f8a441086a0 2024-03-22T01:35:09.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:07.529+0000 7f8a437fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8a441026a0 0x7f8a4410f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:09.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:07.529+0000 7f8a437fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8a441026a0 0x7f8a4410f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33092/0 (socket says 172.21.15.72:33092) 2024-03-22T01:35:09.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:07.530+0000 7f8a437fe700 1 -- 172.21.15.72:0/2851470948 learned_addr learned my addr 172.21.15.72:0/2851470948 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:35:09.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:07.530+0000 7f8a42ffd700 1 --2- 172.21.15.72:0/2851470948 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a44103060 0x7f8a4410c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:09.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:07.530+0000 7f8a43fff700 1 --2- 172.21.15.72:0/2851470948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a441086a0 0x7f8a4410c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:09.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:07.530+0000 7f8a437fe700 1 -- 172.21.15.72:0/2851470948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a441086a0 msgr2=0x7f8a4410c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:09.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:07.530+0000 7f8a437fe700 1 --2- 172.21.15.72:0/2851470948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a441086a0 0x7f8a4410c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:09.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:07.530+0000 7f8a437fe700 1 -- 172.21.15.72:0/2851470948 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a44103060 msgr2=0x7f8a4410c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:09.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:07.530+0000 7f8a437fe700 1 --2- 172.21.15.72:0/2851470948 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a44103060 0x7f8a4410c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:09.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:07.530+0000 7f8a437fe700 1 -- 172.21.15.72:0/2851470948 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a4410cfa0 con 0x7f8a441026a0 2024-03-22T01:35:09.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:07.530+0000 7f8a43fff700 1 --2- 172.21.15.72:0/2851470948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a441086a0 0x7f8a4410c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T01:35:09.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:07.530+0000 7f8a42ffd700 1 --2- 172.21.15.72:0/2851470948 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a44103060 0x7f8a4410c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:35:09.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:07.530+0000 7f8a437fe700 1 --2- 172.21.15.72:0/2851470948 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8a441026a0 0x7f8a4410f240 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f8a3400e9f0 tx=0x7f8a3400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:09.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:07.530+0000 7f8a40ff9700 1 -- 172.21.15.72:0/2851470948 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a34004a30 con 0x7f8a441026a0 2024-03-22T01:35:09.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:07.530+0000 7f8a40ff9700 1 -- 172.21.15.72:0/2851470948 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a3400cca0 con 0x7f8a441026a0 2024-03-22T01:35:09.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:07.530+0000 7f8a49c13700 1 -- 172.21.15.72:0/2851470948 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a441abb10 con 0x7f8a441026a0 2024-03-22T01:35:09.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:07.530+0000 7f8a40ff9700 1 -- 172.21.15.72:0/2851470948 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a340056d0 con 0x7f8a441026a0 2024-03-22T01:35:09.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:07.531+0000 7f8a49c13700 1 -- 172.21.15.72:0/2851470948 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a441abf70 con 0x7f8a441026a0 2024-03-22T01:35:09.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:07.531+0000 7f8a49c13700 1 -- 172.21.15.72:0/2851470948 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8a440fef10 con 0x7f8a441026a0 2024-03-22T01:35:09.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:07.532+0000 7f8a40ff9700 1 -- 172.21.15.72:0/2851470948 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7f8a3400c7c0 con 0x7f8a441026a0 2024-03-22T01:35:09.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:07.533+0000 7f8a40ff9700 1 --2- 172.21.15.72:0/2851470948 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f8a2c0a31e0 0x7f8a2c0a56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:09.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:07.533+0000 7f8a40ff9700 1 -- 172.21.15.72:0/2851470948 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f8a340cfb00 con 0x7f8a441026a0 2024-03-22T01:35:09.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:07.533+0000 7f8a42ffd700 1 --2- 172.21.15.72:0/2851470948 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f8a2c0a31e0 0x7f8a2c0a56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:09.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:07.533+0000 7f8a40ff9700 1 --2- 172.21.15.72:0/2851470948 >> [v2:172.21.15.98:6808/2635838572,v1:172.21.15.98:6809/2635838572] conn(0x7f8a2c0adea0 0x7f8a2c0b02e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:09.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:07.533+0000 7f8a40ff9700 1 -- 172.21.15.72:0/2851470948 --> [v2:172.21.15.98:6808/2635838572,v1:172.21.15.98:6809/2635838572] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8a340056d0 con 0x7f8a2c0adea0 2024-03-22T01:35:09.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:07.533+0000 7f8a40ff9700 1 -- 172.21.15.72:0/2851470948 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f8a340d42c0 con 0x7f8a441026a0 2024-03-22T01:35:09.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:07.534+0000 7f8a43fff700 1 --2- 172.21.15.72:0/2851470948 >> [v2:172.21.15.98:6808/2635838572,v1:172.21.15.98:6809/2635838572] conn(0x7f8a2c0adea0 0x7f8a2c0b02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:09.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:07.534+0000 7f8a42ffd700 1 --2- 172.21.15.72:0/2851470948 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f8a2c0a31e0 0x7f8a2c0a56a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8a38009810 tx=0x7f8a38019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:09.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:07.534+0000 7f8a43fff700 1 --2- 172.21.15.72:0/2851470948 >> [v2:172.21.15.98:6808/2635838572,v1:172.21.15.98:6809/2635838572] conn(0x7f8a2c0adea0 0x7f8a2c0b02e0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:09.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:07.535+0000 7f8a40ff9700 1 -- 172.21.15.72:0/2851470948 <== osd.5 v2:172.21.15.98:6808/2635838572 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8a340056d0 con 0x7f8a2c0adea0 2024-03-22T01:35:09.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:07.551+0000 7f8a49c13700 1 -- 172.21.15.72:0/2851470948 --> [v2:172.21.15.98:6808/2635838572,v1:172.21.15.98:6809/2635838572] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8a44103de0 con 0x7f8a2c0adea0 2024-03-22T01:35:09.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:07.552+0000 7f8a40ff9700 1 -- 172.21.15.72:0/2851470948 <== osd.5 v2:172.21.15.98:6808/2635838572 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f8a44103de0 con 0x7f8a2c0adea0 2024-03-22T01:35:09.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:07.552+0000 7f8a49c13700 1 -- 172.21.15.72:0/2851470948 >> [v2:172.21.15.98:6808/2635838572,v1:172.21.15.98:6809/2635838572] conn(0x7f8a2c0adea0 msgr2=0x7f8a2c0b02e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:09.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:07.552+0000 7f8a49c13700 1 --2- 172.21.15.72:0/2851470948 >> [v2:172.21.15.98:6808/2635838572,v1:172.21.15.98:6809/2635838572] conn(0x7f8a2c0adea0 0x7f8a2c0b02e0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:09.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:07.552+0000 7f8a49c13700 1 -- 172.21.15.72:0/2851470948 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f8a2c0a31e0 msgr2=0x7f8a2c0a56a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:09.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:07.552+0000 7f8a49c13700 1 --2- 172.21.15.72:0/2851470948 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f8a2c0a31e0 0x7f8a2c0a56a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8a38009810 tx=0x7f8a38019040 comp rx=0 tx=0).stop 2024-03-22T01:35:09.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:07.552+0000 7f8a49c13700 1 -- 172.21.15.72:0/2851470948 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8a441026a0 msgr2=0x7f8a4410f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:09.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:07.552+0000 7f8a49c13700 1 --2- 172.21.15.72:0/2851470948 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8a441026a0 0x7f8a4410f240 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f8a3400e9f0 tx=0x7f8a3400ed00 comp rx=0 tx=0).stop 2024-03-22T01:35:09.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:07.552+0000 7f8a43fff700 1 -- 172.21.15.72:0/2851470948 reap_dead start 2024-03-22T01:35:09.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:07.552+0000 7f8a49c13700 1 -- 172.21.15.72:0/2851470948 shutdown_connections 2024-03-22T01:35:09.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:07.552+0000 7f8a49c13700 1 -- 172.21.15.72:0/2851470948 >> 172.21.15.72:0/2851470948 conn(0x7f8a440fe150 msgr2=0x7f8a440fe530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:35:09.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:07.552+0000 7f8a49c13700 1 -- 172.21.15.72:0/2851470948 shutdown_connections 2024-03-22T01:35:09.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:07.552+0000 7f8a49c13700 1 -- 172.21.15.72:0/2851470948 wait complete. 2024-03-22T01:35:10.776 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:10 smithi072 ceph-mon[32414]: pgmap v318: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2024-03-22T01:35:10.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:10 smithi089 ceph-mon[35770]: pgmap v318: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2024-03-22T01:35:10.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:10 smithi098 ceph-mon[36468]: pgmap v318: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2024-03-22T01:35:10.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:10.982+0000 7f3fa482e700 1 -- 172.21.15.72:0/2299983325 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3fa01026d0 msgr2=0x7f3fa0102b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:10.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:10.982+0000 7f3fa482e700 1 --2- 172.21.15.72:0/2299983325 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3fa01026d0 0x7f3fa0102b50 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f3f940099c0 tx=0x7f3f94009cd0 comp rx=0 tx=0).stop 2024-03-22T01:35:10.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:10.982+0000 7f3fa482e700 1 -- 172.21.15.72:0/2299983325 shutdown_connections 2024-03-22T01:35:10.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:10.982+0000 7f3fa482e700 1 --2- 172.21.15.72:0/2299983325 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3fa0103090 0x7f3fa010f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:10.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:10.982+0000 7f3fa482e700 1 --2- 172.21.15.72:0/2299983325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fa01074f0 0x7f3fa01078d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:10.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:10.982+0000 7f3fa482e700 1 --2- 172.21.15.72:0/2299983325 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3fa01026d0 0x7f3fa0102b50 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:10.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:10.982+0000 7f3fa482e700 1 -- 172.21.15.72:0/2299983325 >> 172.21.15.72:0/2299983325 conn(0x7f3fa00797b0 msgr2=0x7f3fa0079bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:35:10.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:10.982+0000 7f3fa482e700 1 -- 172.21.15.72:0/2299983325 shutdown_connections 2024-03-22T01:35:10.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:10.982+0000 7f3fa482e700 1 -- 172.21.15.72:0/2299983325 wait complete. 2024-03-22T01:35:10.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:10.982+0000 7f3fa482e700 1 Processor -- start 2024-03-22T01:35:10.988 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:10.983+0000 7f3fa482e700 1 -- start start 2024-03-22T01:35:10.988 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:10.983+0000 7f3fa482e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fa01026d0 0x7f3fa0198de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:10.988 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:10.983+0000 7f3fa482e700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3fa0103090 0x7f3fa0199320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:10.988 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:10.983+0000 7f3fa482e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3fa01074f0 0x7f3fa019d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:10.988 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:10.983+0000 7f3fa482e700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f3fa0112930 con 0x7f3fa0103090 2024-03-22T01:35:10.988 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:10.983+0000 7f3fa482e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3fa01127b0 con 0x7f3fa01074f0 2024-03-22T01:35:10.988 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:10.983+0000 7f3fa482e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3fa0112ab0 con 0x7f3fa01026d0 2024-03-22T01:35:10.988 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:10.983+0000 7f3f9dd9b700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3fa0103090 0x7f3fa0199320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:10.988 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:10.983+0000 7f3f9dd9b700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3fa0103090 0x7f3fa0199320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36584/0 (socket says 172.21.15.72:36584) 2024-03-22T01:35:10.988 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:10.983+0000 7f3f9dd9b700 1 -- 172.21.15.72:0/2282802734 learned_addr learned my addr 172.21.15.72:0/2282802734 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:35:10.988 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:10.983+0000 7f3f9ed9d700 1 --2- 172.21.15.72:0/2282802734 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3fa01074f0 0x7f3fa019d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:10.988 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:10.983+0000 7f3f9e59c700 1 --2- 172.21.15.72:0/2282802734 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fa01026d0 0x7f3fa0198de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:10.988 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:10.983+0000 7f3f9ed9d700 1 -- 172.21.15.72:0/2282802734 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fa01026d0 msgr2=0x7f3fa0198de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:10.989 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:10.983+0000 7f3f9ed9d700 1 --2- 172.21.15.72:0/2282802734 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fa01026d0 0x7f3fa0198de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:10.989 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:10.983+0000 7f3f9ed9d700 1 -- 172.21.15.72:0/2282802734 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3fa0103090 msgr2=0x7f3fa0199320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:10.989 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:10.983+0000 7f3f9ed9d700 1 --2- 172.21.15.72:0/2282802734 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3fa0103090 0x7f3fa0199320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:10.989 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:10.983+0000 7f3f9ed9d700 1 -- 172.21.15.72:0/2282802734 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fa019ddf0 con 0x7f3fa01074f0 2024-03-22T01:35:10.989 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:10.983+0000 7f3f9ed9d700 1 --2- 172.21.15.72:0/2282802734 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3fa01074f0 0x7f3fa019d6b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3f9800efc0 tx=0x7f3f9800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:10.989 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:10.983+0000 7f3f8f7fe700 1 -- 172.21.15.72:0/2282802734 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f980092e0 con 0x7f3fa01074f0 2024-03-22T01:35:10.989 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:10.984+0000 7f3fa482e700 1 -- 172.21.15.72:0/2282802734 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fa019e080 con 0x7f3fa01074f0 2024-03-22T01:35:10.989 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:10.984+0000 7f3f8f7fe700 1 -- 172.21.15.72:0/2282802734 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f98004bb0 con 0x7f3fa01074f0 2024-03-22T01:35:10.989 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:10.984+0000 7f3f8f7fe700 1 -- 172.21.15.72:0/2282802734 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f98004280 con 0x7f3fa01074f0 2024-03-22T01:35:10.989 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:10.984+0000 7f3fa482e700 1 -- 172.21.15.72:0/2282802734 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fa01a5960 con 0x7f3fa01074f0 2024-03-22T01:35:10.989 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:10.984+0000 7f3fa482e700 1 -- 172.21.15.72:0/2282802734 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3fa0062e10 con 0x7f3fa01074f0 2024-03-22T01:35:10.989 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:10.985+0000 7f3f8f7fe700 1 -- 172.21.15.72:0/2282802734 <== mon.2 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7f3f9801b030 con 0x7f3fa01074f0 2024-03-22T01:35:10.989 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:10.986+0000 7f3f8f7fe700 1 --2- 172.21.15.72:0/2282802734 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f3f880a3040 0x7f3f880a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:10.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:10.986+0000 7f3f9e59c700 1 --2- 172.21.15.72:0/2282802734 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f3f880a3040 0x7f3f880a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:10.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:10.986+0000 7f3f8f7fe700 1 -- 172.21.15.72:0/2282802734 <== mon.2 v2:172.21.15.89:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f3f980cfca0 con 0x7f3fa01074f0 2024-03-22T01:35:10.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:10.986+0000 7f3f8f7fe700 1 --2- 172.21.15.72:0/2282802734 >> [v2:172.21.15.89:6800/2870590550,v1:172.21.15.89:6801/2870590550] conn(0x7f3f880add00 0x7f3f880b0140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:10.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:10.986+0000 7f3f8f7fe700 1 -- 172.21.15.72:0/2282802734 --> [v2:172.21.15.89:6800/2870590550,v1:172.21.15.89:6801/2870590550] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3f980048a0 con 0x7f3f880add00 2024-03-22T01:35:10.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:10.986+0000 7f3f8f7fe700 1 -- 172.21.15.72:0/2282802734 <== mon.2 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f3f980d0050 con 0x7f3fa01074f0 2024-03-22T01:35:10.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:10.986+0000 7f3f9e59c700 1 --2- 172.21.15.72:0/2282802734 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f3f880a3040 0x7f3f880a5500 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3f90007b60 tx=0x7f3f900073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:10.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:10.986+0000 7f3f9dd9b700 1 --2- 172.21.15.72:0/2282802734 >> [v2:172.21.15.89:6800/2870590550,v1:172.21.15.89:6801/2870590550] conn(0x7f3f880add00 0x7f3f880b0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:10.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:10.987+0000 7f3f9dd9b700 1 --2- 172.21.15.72:0/2282802734 >> [v2:172.21.15.89:6800/2870590550,v1:172.21.15.89:6801/2870590550] conn(0x7f3f880add00 0x7f3f880b0140 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:10.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:10.988+0000 7f3f8f7fe700 1 -- 172.21.15.72:0/2282802734 <== osd.1 v2:172.21.15.89:6800/2870590550 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3f980048a0 con 0x7f3f880add00 2024-03-22T01:35:11.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.004+0000 7f3fa482e700 1 -- 172.21.15.72:0/2282802734 --> [v2:172.21.15.89:6800/2870590550,v1:172.21.15.89:6801/2870590550] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3fa0063210 con 0x7f3f880add00 2024-03-22T01:35:11.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.004+0000 7f3f8f7fe700 1 -- 172.21.15.72:0/2282802734 <== osd.1 v2:172.21.15.89:6800/2870590550 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f3fa0063210 con 0x7f3f880add00 2024-03-22T01:35:11.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.004+0000 7f3fa482e700 1 -- 172.21.15.72:0/2282802734 >> [v2:172.21.15.89:6800/2870590550,v1:172.21.15.89:6801/2870590550] conn(0x7f3f880add00 msgr2=0x7f3f880b0140 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:11.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.004+0000 7f3fa482e700 1 --2- 172.21.15.72:0/2282802734 >> [v2:172.21.15.89:6800/2870590550,v1:172.21.15.89:6801/2870590550] conn(0x7f3f880add00 0x7f3f880b0140 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:11.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.004+0000 7f3fa482e700 1 -- 172.21.15.72:0/2282802734 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f3f880a3040 msgr2=0x7f3f880a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:11.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.004+0000 7f3fa482e700 1 --2- 172.21.15.72:0/2282802734 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f3f880a3040 0x7f3f880a5500 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3f90007b60 tx=0x7f3f900073d0 comp rx=0 tx=0).stop 2024-03-22T01:35:11.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.004+0000 7f3fa482e700 1 -- 172.21.15.72:0/2282802734 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3fa01074f0 msgr2=0x7f3fa019d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:11.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.004+0000 7f3fa482e700 1 --2- 172.21.15.72:0/2282802734 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3fa01074f0 0x7f3fa019d6b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3f9800efc0 tx=0x7f3f9800c3a0 comp rx=0 tx=0).stop 2024-03-22T01:35:11.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.005+0000 7f3f9ed9d700 1 -- 172.21.15.72:0/2282802734 reap_dead start 2024-03-22T01:35:11.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.005+0000 7f3fa482e700 1 -- 172.21.15.72:0/2282802734 shutdown_connections 2024-03-22T01:35:11.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.005+0000 7f3fa482e700 1 -- 172.21.15.72:0/2282802734 >> 172.21.15.72:0/2282802734 conn(0x7f3fa00797b0 msgr2=0x7f3fa0100820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:35:11.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.005+0000 7f3fa482e700 1 -- 172.21.15.72:0/2282802734 shutdown_connections 2024-03-22T01:35:11.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.005+0000 7f3fa482e700 1 -- 172.21.15.72:0/2282802734 wait complete. 2024-03-22T01:35:11.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.218+0000 7f4287c21700 1 -- 172.21.15.72:0/2734996628 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f42801075a0 msgr2=0x7f4280107980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:11.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.218+0000 7f4287c21700 1 --2- 172.21.15.72:0/2734996628 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f42801075a0 0x7f4280107980 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f4270009990 tx=0x7f4270009ca0 comp rx=0 tx=0).stop 2024-03-22T01:35:11.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.218+0000 7f4287c21700 1 -- 172.21.15.72:0/2734996628 shutdown_connections 2024-03-22T01:35:11.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.218+0000 7f4287c21700 1 --2- 172.21.15.72:0/2734996628 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4280103090 0x7f428010f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:11.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.218+0000 7f4287c21700 1 --2- 172.21.15.72:0/2734996628 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f42801075a0 0x7f4280107980 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:11.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.218+0000 7f4287c21700 1 --2- 172.21.15.72:0/2734996628 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42801026d0 0x7f4280102b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:11.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.218+0000 7f4287c21700 1 -- 172.21.15.72:0/2734996628 >> 172.21.15.72:0/2734996628 conn(0x7f42800797b0 msgr2=0x7f4280079bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:35:11.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.218+0000 7f4287c21700 1 -- 172.21.15.72:0/2734996628 shutdown_connections 2024-03-22T01:35:11.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.218+0000 7f4287c21700 1 -- 172.21.15.72:0/2734996628 wait complete. 2024-03-22T01:35:11.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.219+0000 7f4287c21700 1 Processor -- start 2024-03-22T01:35:11.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.219+0000 7f4287c21700 1 -- start start 2024-03-22T01:35:11.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.219+0000 7f4287c21700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42801026d0 0x7f4280198f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:11.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.219+0000 7f4287c21700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4280103090 0x7f4280199470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:11.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.219+0000 7f4287c21700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42801075a0 0x7f428019d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:11.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.219+0000 7f4287c21700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f4280112a20 con 0x7f4280103090 2024-03-22T01:35:11.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.219+0000 7f4287c21700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f42801128a0 con 0x7f42801026d0 2024-03-22T01:35:11.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.219+0000 7f4287c21700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4280112ba0 con 0x7f42801075a0 2024-03-22T01:35:11.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.219+0000 7f42851bc700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4280103090 0x7f4280199470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:11.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.219+0000 7f42851bc700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4280103090 0x7f4280199470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36606/0 (socket says 172.21.15.72:36606) 2024-03-22T01:35:11.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.219+0000 7f42851bc700 1 -- 172.21.15.72:0/3791648231 learned_addr learned my addr 172.21.15.72:0/3791648231 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:35:11.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.219+0000 7f42859bd700 1 --2- 172.21.15.72:0/3791648231 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42801026d0 0x7f4280198f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:11.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.219+0000 7f42861be700 1 --2- 172.21.15.72:0/3791648231 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42801075a0 0x7f428019d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:11.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.220+0000 7f42851bc700 1 -- 172.21.15.72:0/3791648231 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42801075a0 msgr2=0x7f428019d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:11.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.220+0000 7f42851bc700 1 --2- 172.21.15.72:0/3791648231 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42801075a0 0x7f428019d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:11.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.220+0000 7f42851bc700 1 -- 172.21.15.72:0/3791648231 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42801026d0 msgr2=0x7f4280198f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:11.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.220+0000 7f42851bc700 1 --2- 172.21.15.72:0/3791648231 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42801026d0 0x7f4280198f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:11.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.220+0000 7f42851bc700 1 -- 172.21.15.72:0/3791648231 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f428019df40 con 0x7f4280103090 2024-03-22T01:35:11.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.220+0000 7f42859bd700 1 --2- 172.21.15.72:0/3791648231 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42801026d0 0x7f4280198f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:35:11.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.220+0000 7f42861be700 1 --2- 172.21.15.72:0/3791648231 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42801075a0 0x7f428019d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T01:35:11.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.220+0000 7f42851bc700 1 --2- 172.21.15.72:0/3791648231 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4280103090 0x7f4280199470 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f427c00c970 tx=0x7f427c00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:11.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.220+0000 7f4276ffd700 1 -- 172.21.15.72:0/3791648231 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f427c007aa0 con 0x7f4280103090 2024-03-22T01:35:11.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.220+0000 7f4276ffd700 1 -- 172.21.15.72:0/3791648231 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f427c004d10 con 0x7f4280103090 2024-03-22T01:35:11.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.220+0000 7f4287c21700 1 -- 172.21.15.72:0/3791648231 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f428019e230 con 0x7f4280103090 2024-03-22T01:35:11.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.220+0000 7f4276ffd700 1 -- 172.21.15.72:0/3791648231 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f427c00f450 con 0x7f4280103090 2024-03-22T01:35:11.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.220+0000 7f4287c21700 1 -- 172.21.15.72:0/3791648231 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42801a5ab0 con 0x7f4280103090 2024-03-22T01:35:11.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.221+0000 7f4287c21700 1 -- 172.21.15.72:0/3791648231 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4280062d90 con 0x7f4280103090 2024-03-22T01:35:11.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.222+0000 7f4276ffd700 1 -- 172.21.15.72:0/3791648231 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7f427c004750 con 0x7f4280103090 2024-03-22T01:35:11.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.223+0000 7f4276ffd700 1 --2- 172.21.15.72:0/3791648231 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f426c0a3040 0x7f426c0a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:11.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.223+0000 7f42859bd700 1 --2- 172.21.15.72:0/3791648231 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f426c0a3040 0x7f426c0a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:11.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.223+0000 7f4276ffd700 1 -- 172.21.15.72:0/3791648231 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f427c0cf9e0 con 0x7f4280103090 2024-03-22T01:35:11.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.223+0000 7f4276ffd700 1 --2- 172.21.15.72:0/3791648231 >> [v2:172.21.15.98:6816/1813255577,v1:172.21.15.98:6817/1813255577] conn(0x7f426c0add00 0x7f426c0b0140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:11.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.223+0000 7f4276ffd700 1 -- 172.21.15.72:0/3791648231 --> [v2:172.21.15.98:6816/1813255577,v1:172.21.15.98:6817/1813255577] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f427c016ec0 con 0x7f426c0add00 2024-03-22T01:35:11.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.223+0000 7f4276ffd700 1 -- 172.21.15.72:0/3791648231 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f427c0cfe30 con 0x7f4280103090 2024-03-22T01:35:11.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.223+0000 7f42859bd700 1 --2- 172.21.15.72:0/3791648231 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f426c0a3040 0x7f426c0a5500 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f427000b2e0 tx=0x7f4270005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:11.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.223+0000 7f42861be700 1 --2- 172.21.15.72:0/3791648231 >> [v2:172.21.15.98:6816/1813255577,v1:172.21.15.98:6817/1813255577] conn(0x7f426c0add00 0x7f426c0b0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:11.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.224+0000 7f42861be700 1 --2- 172.21.15.72:0/3791648231 >> [v2:172.21.15.98:6816/1813255577,v1:172.21.15.98:6817/1813255577] conn(0x7f426c0add00 0x7f426c0b0140 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:11.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.225+0000 7f4276ffd700 1 -- 172.21.15.72:0/3791648231 <== osd.8 v2:172.21.15.98:6816/1813255577 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f427c016ec0 con 0x7f426c0add00 2024-03-22T01:35:11.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.241+0000 7f4287c21700 1 -- 172.21.15.72:0/3791648231 --> [v2:172.21.15.98:6816/1813255577,v1:172.21.15.98:6817/1813255577] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4280063210 con 0x7f426c0add00 2024-03-22T01:35:11.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.241+0000 7f4276ffd700 1 -- 172.21.15.72:0/3791648231 <== osd.8 v2:172.21.15.98:6816/1813255577 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f4280063210 con 0x7f426c0add00 2024-03-22T01:35:11.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.241+0000 7f4287c21700 1 -- 172.21.15.72:0/3791648231 >> [v2:172.21.15.98:6816/1813255577,v1:172.21.15.98:6817/1813255577] conn(0x7f426c0add00 msgr2=0x7f426c0b0140 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:11.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.241+0000 7f4287c21700 1 --2- 172.21.15.72:0/3791648231 >> [v2:172.21.15.98:6816/1813255577,v1:172.21.15.98:6817/1813255577] conn(0x7f426c0add00 0x7f426c0b0140 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:11.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.241+0000 7f4287c21700 1 -- 172.21.15.72:0/3791648231 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f426c0a3040 msgr2=0x7f426c0a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:11.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.241+0000 7f4287c21700 1 --2- 172.21.15.72:0/3791648231 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f426c0a3040 0x7f426c0a5500 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f427000b2e0 tx=0x7f4270005fd0 comp rx=0 tx=0).stop 2024-03-22T01:35:11.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.241+0000 7f4287c21700 1 -- 172.21.15.72:0/3791648231 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4280103090 msgr2=0x7f4280199470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:11.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.241+0000 7f4287c21700 1 --2- 172.21.15.72:0/3791648231 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4280103090 0x7f4280199470 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f427c00c970 tx=0x7f427c00cc80 comp rx=0 tx=0).stop 2024-03-22T01:35:11.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.241+0000 7f42861be700 1 -- 172.21.15.72:0/3791648231 reap_dead start 2024-03-22T01:35:11.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.242+0000 7f4287c21700 1 -- 172.21.15.72:0/3791648231 shutdown_connections 2024-03-22T01:35:11.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.242+0000 7f4287c21700 1 -- 172.21.15.72:0/3791648231 >> 172.21.15.72:0/3791648231 conn(0x7f42800797b0 msgr2=0x7f42801007c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:35:11.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.242+0000 7f4287c21700 1 -- 172.21.15.72:0/3791648231 shutdown_connections 2024-03-22T01:35:11.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:11.242+0000 7f4287c21700 1 -- 172.21.15.72:0/3791648231 wait complete. 2024-03-22T01:35:12.086 INFO:teuthology.orchestra.run.smithi072.stdout:257698037799 2024-03-22T01:35:12.086 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.8 2024-03-22T01:35:12.505 INFO:teuthology.orchestra.run.smithi072.stdout:167503724598 2024-03-22T01:35:12.506 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-22T01:35:12.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:12 smithi089 ceph-mon[35770]: pgmap v319: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 170 B/s wr, 2 op/s 2024-03-22T01:35:12.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:12 smithi072 ceph-mon[32414]: pgmap v319: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 170 B/s wr, 2 op/s 2024-03-22T01:35:12.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:12 smithi098 ceph-mon[36468]: pgmap v319: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 170 B/s wr, 2 op/s 2024-03-22T01:35:13.820 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:13 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:35:13.846 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:13 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:35:13.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:13 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:35:14.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.047+0000 7f8b8abfe700 1 -- 172.21.15.72:0/3188492666 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8b841075a0 msgr2=0x7f8b84107980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:14.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.047+0000 7f8b8abfe700 1 --2- 172.21.15.72:0/3188492666 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8b841075a0 0x7f8b84107980 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f8b78009990 tx=0x7f8b78009ca0 comp rx=0 tx=0).stop 2024-03-22T01:35:14.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.047+0000 7f8b8abfe700 1 -- 172.21.15.72:0/3188492666 shutdown_connections 2024-03-22T01:35:14.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.047+0000 7f8b8abfe700 1 --2- 172.21.15.72:0/3188492666 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b84103090 0x7f8b8410f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:14.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.047+0000 7f8b8abfe700 1 --2- 172.21.15.72:0/3188492666 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8b841075a0 0x7f8b84107980 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:14.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.047+0000 7f8b8abfe700 1 --2- 172.21.15.72:0/3188492666 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b841026d0 0x7f8b84102b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:14.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.047+0000 7f8b8abfe700 1 -- 172.21.15.72:0/3188492666 >> 172.21.15.72:0/3188492666 conn(0x7f8b840797b0 msgr2=0x7f8b84079bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:35:14.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.047+0000 7f8b8abfe700 1 -- 172.21.15.72:0/3188492666 shutdown_connections 2024-03-22T01:35:14.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.047+0000 7f8b8abfe700 1 -- 172.21.15.72:0/3188492666 wait complete. 2024-03-22T01:35:14.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.048+0000 7f8b8abfe700 1 Processor -- start 2024-03-22T01:35:14.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.048+0000 7f8b8abfe700 1 -- start start 2024-03-22T01:35:14.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.048+0000 7f8b8abfe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b841026d0 0x7f8b84074980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:14.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.048+0000 7f8b8abfe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8b84103090 0x7f8b84074ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:14.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.048+0000 7f8b8abfe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b841075a0 0x7f8b84075400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:14.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.048+0000 7f8b8abfe700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f8b841129b0 con 0x7f8b84103090 2024-03-22T01:35:14.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.048+0000 7f8b8abfe700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8b84112830 con 0x7f8b841075a0 2024-03-22T01:35:14.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.048+0000 7f8b8abfe700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8b84112b30 con 0x7f8b841026d0 2024-03-22T01:35:14.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.048+0000 7f8b83fff700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8b84103090 0x7f8b84074ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:14.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.048+0000 7f8b83fff700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8b84103090 0x7f8b84074ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36642/0 (socket says 172.21.15.72:36642) 2024-03-22T01:35:14.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.048+0000 7f8b83fff700 1 -- 172.21.15.72:0/1879801174 learned_addr learned my addr 172.21.15.72:0/1879801174 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:35:14.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.048+0000 7f8b8919b700 1 --2- 172.21.15.72:0/1879801174 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b841075a0 0x7f8b84075400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:14.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.048+0000 7f8b8899a700 1 --2- 172.21.15.72:0/1879801174 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b841026d0 0x7f8b84074980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:14.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.048+0000 7f8b83fff700 1 -- 172.21.15.72:0/1879801174 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b841026d0 msgr2=0x7f8b84074980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:14.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.048+0000 7f8b83fff700 1 --2- 172.21.15.72:0/1879801174 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b841026d0 0x7f8b84074980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:14.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.048+0000 7f8b83fff700 1 -- 172.21.15.72:0/1879801174 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b841075a0 msgr2=0x7f8b84075400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:14.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.048+0000 7f8b83fff700 1 --2- 172.21.15.72:0/1879801174 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b841075a0 0x7f8b84075400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:14.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.048+0000 7f8b83fff700 1 -- 172.21.15.72:0/1879801174 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b841a74b0 con 0x7f8b84103090 2024-03-22T01:35:14.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.048+0000 7f8b83fff700 1 --2- 172.21.15.72:0/1879801174 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8b84103090 0x7f8b84074ec0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f8b7000e9a0 tx=0x7f8b7000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:14.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.049+0000 7f8b81ffb700 1 -- 172.21.15.72:0/1879801174 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b7000ca10 con 0x7f8b84103090 2024-03-22T01:35:14.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.049+0000 7f8b81ffb700 1 -- 172.21.15.72:0/1879801174 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b7000cbb0 con 0x7f8b84103090 2024-03-22T01:35:14.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.049+0000 7f8b81ffb700 1 -- 172.21.15.72:0/1879801174 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b70018680 con 0x7f8b84103090 2024-03-22T01:35:14.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.049+0000 7f8b8abfe700 1 -- 172.21.15.72:0/1879801174 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b841a77a0 con 0x7f8b84103090 2024-03-22T01:35:14.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.049+0000 7f8b8abfe700 1 -- 172.21.15.72:0/1879801174 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b841a7c90 con 0x7f8b84103090 2024-03-22T01:35:14.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.050+0000 7f8b8abfe700 1 -- 172.21.15.72:0/1879801174 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8b84062d90 con 0x7f8b84103090 2024-03-22T01:35:14.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.051+0000 7f8b81ffb700 1 -- 172.21.15.72:0/1879801174 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7f8b70010430 con 0x7f8b84103090 2024-03-22T01:35:14.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.051+0000 7f8b81ffb700 1 --2- 172.21.15.72:0/1879801174 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f8b740a30c0 0x7f8b740a5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:14.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.052+0000 7f8b81ffb700 1 -- 172.21.15.72:0/1879801174 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f8b700d01d0 con 0x7f8b84103090 2024-03-22T01:35:14.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.052+0000 7f8b8899a700 1 --2- 172.21.15.72:0/1879801174 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f8b740a30c0 0x7f8b740a5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:14.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.052+0000 7f8b81ffb700 1 --2- 172.21.15.72:0/1879801174 >> [v2:172.21.15.72:6802/661906791,v1:172.21.15.72:6803/661906791] conn(0x7f8b740add80 0x7f8b740b01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:14.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.052+0000 7f8b81ffb700 1 -- 172.21.15.72:0/1879801174 --> [v2:172.21.15.72:6802/661906791,v1:172.21.15.72:6803/661906791] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8b7001bec0 con 0x7f8b740add80 2024-03-22T01:35:14.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.052+0000 7f8b81ffb700 1 -- 172.21.15.72:0/1879801174 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f8b700d0620 con 0x7f8b84103090 2024-03-22T01:35:14.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.052+0000 7f8b8919b700 1 --2- 172.21.15.72:0/1879801174 >> [v2:172.21.15.72:6802/661906791,v1:172.21.15.72:6803/661906791] conn(0x7f8b740add80 0x7f8b740b01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:14.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.052+0000 7f8b8899a700 1 --2- 172.21.15.72:0/1879801174 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f8b740a30c0 0x7f8b740a5580 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8b780098e0 tx=0x7f8b78005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:14.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.052+0000 7f8b8919b700 1 --2- 172.21.15.72:0/1879801174 >> [v2:172.21.15.72:6802/661906791,v1:172.21.15.72:6803/661906791] conn(0x7f8b740add80 0x7f8b740b01c0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:14.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.053+0000 7f8b81ffb700 1 -- 172.21.15.72:0/1879801174 <== osd.0 v2:172.21.15.72:6802/661906791 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8b7001bec0 con 0x7f8b740add80 2024-03-22T01:35:14.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.069+0000 7f8b8abfe700 1 -- 172.21.15.72:0/1879801174 --> [v2:172.21.15.72:6802/661906791,v1:172.21.15.72:6803/661906791] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8b840631e0 con 0x7f8b740add80 2024-03-22T01:35:14.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.069+0000 7f8b81ffb700 1 -- 172.21.15.72:0/1879801174 <== osd.0 v2:172.21.15.72:6802/661906791 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f8b840631e0 con 0x7f8b740add80 2024-03-22T01:35:14.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.069+0000 7f8b8abfe700 1 -- 172.21.15.72:0/1879801174 >> [v2:172.21.15.72:6802/661906791,v1:172.21.15.72:6803/661906791] conn(0x7f8b740add80 msgr2=0x7f8b740b01c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:14.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.069+0000 7f8b8abfe700 1 --2- 172.21.15.72:0/1879801174 >> [v2:172.21.15.72:6802/661906791,v1:172.21.15.72:6803/661906791] conn(0x7f8b740add80 0x7f8b740b01c0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:14.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.070+0000 7f8b8abfe700 1 -- 172.21.15.72:0/1879801174 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f8b740a30c0 msgr2=0x7f8b740a5580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:14.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.070+0000 7f8b8abfe700 1 --2- 172.21.15.72:0/1879801174 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f8b740a30c0 0x7f8b740a5580 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8b780098e0 tx=0x7f8b78005960 comp rx=0 tx=0).stop 2024-03-22T01:35:14.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.070+0000 7f8b8abfe700 1 -- 172.21.15.72:0/1879801174 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8b84103090 msgr2=0x7f8b84074ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:14.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.070+0000 7f8b8abfe700 1 --2- 172.21.15.72:0/1879801174 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8b84103090 0x7f8b84074ec0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f8b7000e9a0 tx=0x7f8b7000ecb0 comp rx=0 tx=0).stop 2024-03-22T01:35:14.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.070+0000 7f8b8919b700 1 -- 172.21.15.72:0/1879801174 reap_dead start 2024-03-22T01:35:14.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.070+0000 7f8b8abfe700 1 -- 172.21.15.72:0/1879801174 shutdown_connections 2024-03-22T01:35:14.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.070+0000 7f8b8abfe700 1 -- 172.21.15.72:0/1879801174 >> 172.21.15.72:0/1879801174 conn(0x7f8b840797b0 msgr2=0x7f8b84100880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:35:14.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.070+0000 7f8b8abfe700 1 -- 172.21.15.72:0/1879801174 shutdown_connections 2024-03-22T01:35:14.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.070+0000 7f8b8abfe700 1 -- 172.21.15.72:0/1879801174 wait complete. 2024-03-22T01:35:14.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.297+0000 7f2245427700 1 -- 172.21.15.72:0/3176335689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22401015b0 msgr2=0x7f2240101a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:14.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.297+0000 7f2245427700 1 --2- 172.21.15.72:0/3176335689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22401015b0 0x7f2240101a30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f22340098f0 tx=0x7f2234009c00 comp rx=0 tx=0).stop 2024-03-22T01:35:14.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.297+0000 7f2245427700 1 -- 172.21.15.72:0/3176335689 shutdown_connections 2024-03-22T01:35:14.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.297+0000 7f2245427700 1 --2- 172.21.15.72:0/3176335689 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2240101f70 0x7f224010e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:14.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.297+0000 7f2245427700 1 --2- 172.21.15.72:0/3176335689 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22401075f0 0x7f22401079d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:14.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.297+0000 7f2245427700 1 --2- 172.21.15.72:0/3176335689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22401015b0 0x7f2240101a30 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:14.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.297+0000 7f2245427700 1 -- 172.21.15.72:0/3176335689 >> 172.21.15.72:0/3176335689 conn(0x7f22400775d0 msgr2=0x7f22400779e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:35:14.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.297+0000 7f2245427700 1 -- 172.21.15.72:0/3176335689 shutdown_connections 2024-03-22T01:35:14.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.297+0000 7f2245427700 1 -- 172.21.15.72:0/3176335689 wait complete. 2024-03-22T01:35:14.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.297+0000 7f2245427700 1 Processor -- start 2024-03-22T01:35:14.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.298+0000 7f2245427700 1 -- start start 2024-03-22T01:35:14.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.298+0000 7f2245427700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22401015b0 0x7f2240199ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:14.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.298+0000 7f2245427700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2240101f70 0x7f224019a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:14.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.298+0000 7f2245427700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f22401075f0 0x7f22401a4d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:14.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.298+0000 7f2245427700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f22401118a0 con 0x7f22401075f0 2024-03-22T01:35:14.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.298+0000 7f2245427700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2240111720 con 0x7f22401015b0 2024-03-22T01:35:14.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.298+0000 7f2245427700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2240111a20 con 0x7f2240101f70 2024-03-22T01:35:14.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.298+0000 7f223effd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22401015b0 0x7f2240199ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:14.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.298+0000 7f223f7fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f22401075f0 0x7f22401a4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:14.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.298+0000 7f223effd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22401015b0 0x7f2240199ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.72:39200/0 (socket says 172.21.15.72:39200) 2024-03-22T01:35:14.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.298+0000 7f223e7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2240101f70 0x7f224019a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:14.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.298+0000 7f223effd700 1 -- 172.21.15.72:0/279977404 learned_addr learned my addr 172.21.15.72:0/279977404 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:35:14.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.298+0000 7f223effd700 1 -- 172.21.15.72:0/279977404 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2240101f70 msgr2=0x7f224019a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:14.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.298+0000 7f223effd700 1 --2- 172.21.15.72:0/279977404 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2240101f70 0x7f224019a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:14.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.298+0000 7f223effd700 1 -- 172.21.15.72:0/279977404 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f22401075f0 msgr2=0x7f22401a4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:14.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.298+0000 7f223effd700 1 --2- 172.21.15.72:0/279977404 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f22401075f0 0x7f22401a4d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:14.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.298+0000 7f223effd700 1 -- 172.21.15.72:0/279977404 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22401a5260 con 0x7f22401015b0 2024-03-22T01:35:14.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.298+0000 7f223e7fc700 1 --2- 172.21.15.72:0/279977404 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2240101f70 0x7f224019a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:35:14.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.298+0000 7f223effd700 1 --2- 172.21.15.72:0/279977404 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22401015b0 0x7f2240199ca0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f223000e9c0 tx=0x7f223000ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:14.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.299+0000 7f2227fff700 1 -- 172.21.15.72:0/279977404 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f223000cbd0 con 0x7f22401015b0 2024-03-22T01:35:14.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.299+0000 7f2227fff700 1 -- 172.21.15.72:0/279977404 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f223000ee60 con 0x7f22401015b0 2024-03-22T01:35:14.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.299+0000 7f2245427700 1 -- 172.21.15.72:0/279977404 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22401a5430 con 0x7f22401015b0 2024-03-22T01:35:14.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.299+0000 7f2227fff700 1 -- 172.21.15.72:0/279977404 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2230005630 con 0x7f22401015b0 2024-03-22T01:35:14.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.299+0000 7f2245427700 1 -- 172.21.15.72:0/279977404 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22401a58f0 con 0x7f22401015b0 2024-03-22T01:35:14.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.299+0000 7f2245427700 1 -- 172.21.15.72:0/279977404 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2240062e10 con 0x7f22401015b0 2024-03-22T01:35:14.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.302+0000 7f2227fff700 1 -- 172.21.15.72:0/279977404 <== mon.2 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7f2230004750 con 0x7f22401015b0 2024-03-22T01:35:14.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.302+0000 7f2227fff700 1 --2- 172.21.15.72:0/279977404 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f22280a3040 0x7f22280a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:14.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.302+0000 7f223e7fc700 1 --2- 172.21.15.72:0/279977404 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f22280a3040 0x7f22280a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:14.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.302+0000 7f2227fff700 1 -- 172.21.15.72:0/279977404 <== mon.2 v2:172.21.15.89:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f22300cfdf0 con 0x7f22401015b0 2024-03-22T01:35:14.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.302+0000 7f2227fff700 1 --2- 172.21.15.72:0/279977404 >> [v2:172.21.15.72:6818/3158982661,v1:172.21.15.72:6819/3158982661] conn(0x7f22280add00 0x7f22280b0140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:14.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.302+0000 7f2227fff700 1 -- 172.21.15.72:0/279977404 --> [v2:172.21.15.72:6818/3158982661,v1:172.21.15.72:6819/3158982661] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f22280b0810 con 0x7f22280add00 2024-03-22T01:35:14.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.303+0000 7f2227fff700 1 -- 172.21.15.72:0/279977404 <== mon.2 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f22300d0240 con 0x7f22401015b0 2024-03-22T01:35:14.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.303+0000 7f223f7fe700 1 --2- 172.21.15.72:0/279977404 >> [v2:172.21.15.72:6818/3158982661,v1:172.21.15.72:6819/3158982661] conn(0x7f22280add00 0x7f22280b0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:14.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.303+0000 7f223e7fc700 1 --2- 172.21.15.72:0/279977404 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f22280a3040 0x7f22280a5500 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f22340098c0 tx=0x7f223401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:14.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.303+0000 7f223f7fe700 1 --2- 172.21.15.72:0/279977404 >> [v2:172.21.15.72:6818/3158982661,v1:172.21.15.72:6819/3158982661] conn(0x7f22280add00 0x7f22280b0140 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:14.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.304+0000 7f2227fff700 1 -- 172.21.15.72:0/279977404 <== osd.6 v2:172.21.15.72:6818/3158982661 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f22280b0810 con 0x7f22280add00 2024-03-22T01:35:14.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.320+0000 7f2245427700 1 -- 172.21.15.72:0/279977404 --> [v2:172.21.15.72:6818/3158982661,v1:172.21.15.72:6819/3158982661] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2240063290 con 0x7f22280add00 2024-03-22T01:35:14.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.320+0000 7f2227fff700 1 -- 172.21.15.72:0/279977404 <== osd.6 v2:172.21.15.72:6818/3158982661 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f2240063290 con 0x7f22280add00 2024-03-22T01:35:14.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.320+0000 7f2245427700 1 -- 172.21.15.72:0/279977404 >> [v2:172.21.15.72:6818/3158982661,v1:172.21.15.72:6819/3158982661] conn(0x7f22280add00 msgr2=0x7f22280b0140 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:14.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.320+0000 7f2245427700 1 --2- 172.21.15.72:0/279977404 >> [v2:172.21.15.72:6818/3158982661,v1:172.21.15.72:6819/3158982661] conn(0x7f22280add00 0x7f22280b0140 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:14.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.320+0000 7f2245427700 1 -- 172.21.15.72:0/279977404 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f22280a3040 msgr2=0x7f22280a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:14.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.320+0000 7f2245427700 1 --2- 172.21.15.72:0/279977404 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f22280a3040 0x7f22280a5500 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f22340098c0 tx=0x7f223401a040 comp rx=0 tx=0).stop 2024-03-22T01:35:14.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.320+0000 7f2245427700 1 -- 172.21.15.72:0/279977404 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22401015b0 msgr2=0x7f2240199ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:14.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.320+0000 7f2245427700 1 --2- 172.21.15.72:0/279977404 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22401015b0 0x7f2240199ca0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f223000e9c0 tx=0x7f223000ecd0 comp rx=0 tx=0).stop 2024-03-22T01:35:14.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.320+0000 7f223f7fe700 1 -- 172.21.15.72:0/279977404 reap_dead start 2024-03-22T01:35:14.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.321+0000 7f2245427700 1 -- 172.21.15.72:0/279977404 shutdown_connections 2024-03-22T01:35:14.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.321+0000 7f2245427700 1 -- 172.21.15.72:0/279977404 >> 172.21.15.72:0/279977404 conn(0x7f22400775d0 msgr2=0x7f2240100290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:35:14.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.321+0000 7f2245427700 1 -- 172.21.15.72:0/279977404 shutdown_connections 2024-03-22T01:35:14.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:14.321+0000 7f2245427700 1 -- 172.21.15.72:0/279977404 wait complete. 2024-03-22T01:35:14.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:14 smithi072 ceph-mon[32414]: pgmap v320: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-22T01:35:14.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:14 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:35:14.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:14 smithi089 ceph-mon[35770]: pgmap v320: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-22T01:35:14.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:14 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:35:14.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:14 smithi098 ceph-mon[36468]: pgmap v320: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-22T01:35:14.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:14 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:35:15.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.323+0000 7f992a806700 1 -- 172.21.15.72:0/3926487610 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9924107a00 msgr2=0x7f992410f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:15.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.323+0000 7f992a806700 1 --2- 172.21.15.72:0/3926487610 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9924107a00 0x7f992410f610 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f991c00b0d0 tx=0x7f991c00b3e0 comp rx=0 tx=0).stop 2024-03-22T01:35:15.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.323+0000 7f992a806700 1 -- 172.21.15.72:0/3926487610 shutdown_connections 2024-03-22T01:35:15.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.323+0000 7f992a806700 1 --2- 172.21.15.72:0/3926487610 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9924107a00 0x7f992410f610 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:15.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.323+0000 7f992a806700 1 --2- 172.21.15.72:0/3926487610 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9924100ed0 0x7f99241012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:15.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.323+0000 7f992a806700 1 --2- 172.21.15.72:0/3926487610 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9924101880 0x7f99241072d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:15.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.323+0000 7f992a806700 1 -- 172.21.15.72:0/3926487610 >> 172.21.15.72:0/3926487610 conn(0x7f9924077b80 msgr2=0x7f99240fefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:35:15.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.323+0000 7f992a806700 1 -- 172.21.15.72:0/3926487610 shutdown_connections 2024-03-22T01:35:15.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.323+0000 7f992a806700 1 -- 172.21.15.72:0/3926487610 wait complete. 2024-03-22T01:35:15.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.323+0000 7f992a806700 1 Processor -- start 2024-03-22T01:35:15.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.323+0000 7f992a806700 1 -- start start 2024-03-22T01:35:15.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.324+0000 7f992a806700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9924100ed0 0x7f9924101e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:15.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.324+0000 7f992a806700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9924101880 0x7f9924102390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:15.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.324+0000 7f992a806700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9924107a00 0x7f99241028d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:15.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.324+0000 7f992a806700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f9924112a00 con 0x7f9924100ed0 2024-03-22T01:35:15.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.324+0000 7f992a806700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9924112880 con 0x7f9924107a00 2024-03-22T01:35:15.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.324+0000 7f992a806700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9924112b80 con 0x7f9924101880 2024-03-22T01:35:15.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.324+0000 7f9923fff700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9924100ed0 0x7f9924101e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:15.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.324+0000 7f9928da3700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9924107a00 0x7f99241028d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:15.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.324+0000 7f9928da3700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9924107a00 0x7f99241028d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.72:39246/0 (socket says 172.21.15.72:39246) 2024-03-22T01:35:15.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.324+0000 7f9923fff700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9924100ed0 0x7f9924101e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36696/0 (socket says 172.21.15.72:36696) 2024-03-22T01:35:15.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.324+0000 7f9928da3700 1 -- 172.21.15.72:0/2393390279 learned_addr learned my addr 172.21.15.72:0/2393390279 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:35:15.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.324+0000 7f99237fe700 1 --2- 172.21.15.72:0/2393390279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9924101880 0x7f9924102390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:15.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.324+0000 7f9928da3700 1 -- 172.21.15.72:0/2393390279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9924101880 msgr2=0x7f9924102390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:15.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.324+0000 7f9928da3700 1 --2- 172.21.15.72:0/2393390279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9924101880 0x7f9924102390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:15.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.324+0000 7f9928da3700 1 -- 172.21.15.72:0/2393390279 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9924100ed0 msgr2=0x7f9924101e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:15.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.324+0000 7f9928da3700 1 --2- 172.21.15.72:0/2393390279 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9924100ed0 0x7f9924101e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:15.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.324+0000 7f9928da3700 1 -- 172.21.15.72:0/2393390279 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99241a72d0 con 0x7f9924107a00 2024-03-22T01:35:15.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.325+0000 7f9928da3700 1 --2- 172.21.15.72:0/2393390279 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9924107a00 0x7f99241028d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f991c006070 tx=0x7f991c007a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:15.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.325+0000 7f99217fa700 1 -- 172.21.15.72:0/2393390279 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f991c003c00 con 0x7f9924107a00 2024-03-22T01:35:15.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.325+0000 7f992a806700 1 -- 172.21.15.72:0/2393390279 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99241a7560 con 0x7f9924107a00 2024-03-22T01:35:15.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.325+0000 7f99217fa700 1 -- 172.21.15.72:0/2393390279 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f991c003da0 con 0x7f9924107a00 2024-03-22T01:35:15.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.325+0000 7f99217fa700 1 -- 172.21.15.72:0/2393390279 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f991c01b880 con 0x7f9924107a00 2024-03-22T01:35:15.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.325+0000 7f992a806700 1 -- 172.21.15.72:0/2393390279 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99241a7a10 con 0x7f9924107a00 2024-03-22T01:35:15.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.326+0000 7f992a806700 1 -- 172.21.15.72:0/2393390279 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9910001030 con 0x7f9924107a00 2024-03-22T01:35:15.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.327+0000 7f99217fa700 1 -- 172.21.15.72:0/2393390279 <== mon.2 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7f991c026450 con 0x7f9924107a00 2024-03-22T01:35:15.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.327+0000 7f99217fa700 1 --2- 172.21.15.72:0/2393390279 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f990c0a30c0 0x7f990c0a5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:15.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.327+0000 7f9923fff700 1 --2- 172.21.15.72:0/2393390279 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f990c0a30c0 0x7f990c0a5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:15.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.327+0000 7f99217fa700 1 -- 172.21.15.72:0/2393390279 <== mon.2 v2:172.21.15.89:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f991c00f020 con 0x7f9924107a00 2024-03-22T01:35:15.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.328+0000 7f99217fa700 1 --2- 172.21.15.72:0/2393390279 >> [v2:172.21.15.98:6824/1517095838,v1:172.21.15.98:6825/1517095838] conn(0x7f990c0add80 0x7f990c0b01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:15.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.328+0000 7f99217fa700 1 -- 172.21.15.72:0/2393390279 --> [v2:172.21.15.98:6824/1517095838,v1:172.21.15.98:6825/1517095838] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f991c023ec0 con 0x7f990c0add80 2024-03-22T01:35:15.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.328+0000 7f99217fa700 1 -- 172.21.15.72:0/2393390279 <== mon.2 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f991c0d13f0 con 0x7f9924107a00 2024-03-22T01:35:15.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.328+0000 7f9923fff700 1 --2- 172.21.15.72:0/2393390279 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f990c0a30c0 0x7f990c0a5580 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f9914007be0 tx=0x7f99140073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:15.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.328+0000 7f99237fe700 1 --2- 172.21.15.72:0/2393390279 >> [v2:172.21.15.98:6824/1517095838,v1:172.21.15.98:6825/1517095838] conn(0x7f990c0add80 0x7f990c0b01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:15.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.328+0000 7f99237fe700 1 --2- 172.21.15.72:0/2393390279 >> [v2:172.21.15.98:6824/1517095838,v1:172.21.15.98:6825/1517095838] conn(0x7f990c0add80 0x7f990c0b01c0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:15.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.329+0000 7f99217fa700 1 -- 172.21.15.72:0/2393390279 <== osd.11 v2:172.21.15.98:6824/1517095838 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f991c023ec0 con 0x7f990c0add80 2024-03-22T01:35:15.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.361+0000 7f992a806700 1 -- 172.21.15.72:0/2393390279 --> [v2:172.21.15.98:6824/1517095838,v1:172.21.15.98:6825/1517095838] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9910002e60 con 0x7f990c0add80 2024-03-22T01:35:15.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.362+0000 7f99217fa700 1 -- 172.21.15.72:0/2393390279 <== osd.11 v2:172.21.15.98:6824/1517095838 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f9910002e60 con 0x7f990c0add80 2024-03-22T01:35:15.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.362+0000 7f992a806700 1 -- 172.21.15.72:0/2393390279 >> [v2:172.21.15.98:6824/1517095838,v1:172.21.15.98:6825/1517095838] conn(0x7f990c0add80 msgr2=0x7f990c0b01c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:15.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.362+0000 7f992a806700 1 --2- 172.21.15.72:0/2393390279 >> [v2:172.21.15.98:6824/1517095838,v1:172.21.15.98:6825/1517095838] conn(0x7f990c0add80 0x7f990c0b01c0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:15.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.363+0000 7f992a806700 1 -- 172.21.15.72:0/2393390279 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f990c0a30c0 msgr2=0x7f990c0a5580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:15.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.363+0000 7f992a806700 1 --2- 172.21.15.72:0/2393390279 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f990c0a30c0 0x7f990c0a5580 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f9914007be0 tx=0x7f99140073d0 comp rx=0 tx=0).stop 2024-03-22T01:35:15.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.363+0000 7f992a806700 1 -- 172.21.15.72:0/2393390279 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9924107a00 msgr2=0x7f99241028d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:15.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.363+0000 7f992a806700 1 --2- 172.21.15.72:0/2393390279 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9924107a00 0x7f99241028d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f991c006070 tx=0x7f991c007a20 comp rx=0 tx=0).stop 2024-03-22T01:35:15.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.363+0000 7f9928da3700 1 -- 172.21.15.72:0/2393390279 reap_dead start 2024-03-22T01:35:15.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.363+0000 7f992a806700 1 -- 172.21.15.72:0/2393390279 shutdown_connections 2024-03-22T01:35:15.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.363+0000 7f992a806700 1 -- 172.21.15.72:0/2393390279 >> 172.21.15.72:0/2393390279 conn(0x7f9924077b80 msgr2=0x7f99240fefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:35:15.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.363+0000 7f992a806700 1 -- 172.21.15.72:0/2393390279 shutdown_connections 2024-03-22T01:35:15.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.363+0000 7f992a806700 1 -- 172.21.15.72:0/2393390279 wait complete. 2024-03-22T01:35:15.670 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:15 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:35:15.670 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:15 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:35:15.670 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:15 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:35:15.670 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:15 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-03-22T01:35:15.670 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:15 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 1]}]: dispatch 2024-03-22T01:35:15.670 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:15 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]: dispatch 2024-03-22T01:35:15.670 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:15 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]: dispatch 2024-03-22T01:35:15.670 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:15 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [3, 9]}]: dispatch 2024-03-22T01:35:15.670 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:15 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-03-22T01:35:15.670 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:15 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8]}]: dispatch 2024-03-22T01:35:15.670 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:15 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2024-03-22T01:35:15.671 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:15 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6]}]: dispatch 2024-03-22T01:35:15.671 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:15 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 7]}]: dispatch 2024-03-22T01:35:15.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:15 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:35:15.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:15 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:35:15.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:15 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:35:15.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:15 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-03-22T01:35:15.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:15 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 1]}]: dispatch 2024-03-22T01:35:15.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:15 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]: dispatch 2024-03-22T01:35:15.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:15 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]: dispatch 2024-03-22T01:35:15.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:15 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [3, 9]}]: dispatch 2024-03-22T01:35:15.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:15 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-03-22T01:35:15.859 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:15 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8]}]: dispatch 2024-03-22T01:35:15.859 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:15 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2024-03-22T01:35:15.859 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:15 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6]}]: dispatch 2024-03-22T01:35:15.859 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:15 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 7]}]: dispatch 2024-03-22T01:35:15.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:15 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:35:15.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:15 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:35:15.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:15 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:35:15.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:15 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-03-22T01:35:15.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:15 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 1]}]: dispatch 2024-03-22T01:35:15.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:15 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]: dispatch 2024-03-22T01:35:15.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:15 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]: dispatch 2024-03-22T01:35:15.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:15 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [3, 9]}]: dispatch 2024-03-22T01:35:15.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:15 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-03-22T01:35:15.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:15 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8]}]: dispatch 2024-03-22T01:35:15.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:15 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2024-03-22T01:35:15.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:15 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6]}]: dispatch 2024-03-22T01:35:15.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:15 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 7]}]: dispatch 2024-03-22T01:35:16.182 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:35:16.185 INFO:teuthology.orchestra.run.smithi072.stdout:201863462962 2024-03-22T01:35:16.186 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-22T01:35:16.227 INFO:teuthology.orchestra.run.smithi072.stdout:55834574922 2024-03-22T01:35:16.227 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-22T01:35:16.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:13.661+0000 7fb7311fa700 1 -- 172.21.15.72:0/1385290752 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb72c108630 msgr2=0x7fb72c108a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:16.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:13.661+0000 7fb7311fa700 1 --2- 172.21.15.72:0/1385290752 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb72c108630 0x7fb72c108a10 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fb71c0099c0 tx=0x7fb71c009cd0 comp rx=0 tx=0).stop 2024-03-22T01:35:16.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:13.661+0000 7fb7311fa700 1 -- 172.21.15.72:0/1385290752 shutdown_connections 2024-03-22T01:35:16.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:13.661+0000 7fb7311fa700 1 --2- 172.21.15.72:0/1385290752 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb72c06b580 0x7fb72c10f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:16.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:13.661+0000 7fb7311fa700 1 --2- 172.21.15.72:0/1385290752 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb72c108630 0x7fb72c108a10 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:16.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:13.661+0000 7fb7311fa700 1 --2- 172.21.15.72:0/1385290752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb72c06abc0 0x7fb72c06b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:16.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:13.661+0000 7fb7311fa700 1 -- 172.21.15.72:0/1385290752 >> 172.21.15.72:0/1385290752 conn(0x7fb72c0775d0 msgr2=0x7fb72c0779e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:35:16.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:13.661+0000 7fb7311fa700 1 -- 172.21.15.72:0/1385290752 shutdown_connections 2024-03-22T01:35:16.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:13.661+0000 7fb7311fa700 1 -- 172.21.15.72:0/1385290752 wait complete. 2024-03-22T01:35:16.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:13.662+0000 7fb7311fa700 1 Processor -- start 2024-03-22T01:35:16.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:13.662+0000 7fb7311fa700 1 -- start start 2024-03-22T01:35:16.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:13.662+0000 7fb7311fa700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb72c06abc0 0x7fb72c198e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:16.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:13.662+0000 7fb7311fa700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb72c06b580 0x7fb72c1993d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:16.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:13.662+0000 7fb7311fa700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb72c108630 0x7fb72c19d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:16.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:13.662+0000 7fb7311fa700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fb72c1129c0 con 0x7fb72c06b580 2024-03-22T01:35:16.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:13.662+0000 7fb7311fa700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb72c112840 con 0x7fb72c06abc0 2024-03-22T01:35:16.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:13.662+0000 7fb7311fa700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb72c112b40 con 0x7fb72c108630 2024-03-22T01:35:16.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:13.662+0000 7fb72a59c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb72c06b580 0x7fb72c1993d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:16.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:13.662+0000 7fb72ad9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb72c06abc0 0x7fb72c198e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:16.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:13.662+0000 7fb72a59c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb72c06b580 0x7fb72c1993d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36622/0 (socket says 172.21.15.72:36622) 2024-03-22T01:35:16.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:13.662+0000 7fb72a59c700 1 -- 172.21.15.72:0/1523688222 learned_addr learned my addr 172.21.15.72:0/1523688222 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:35:16.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:13.662+0000 7fb72ad9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb72c06abc0 0x7fb72c198e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.72:39182/0 (socket says 172.21.15.72:39182) 2024-03-22T01:35:16.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:13.662+0000 7fb72b59e700 1 --2- 172.21.15.72:0/1523688222 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb72c108630 0x7fb72c19d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:16.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:13.662+0000 7fb72a59c700 1 -- 172.21.15.72:0/1523688222 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb72c108630 msgr2=0x7fb72c19d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:16.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:13.662+0000 7fb72a59c700 1 --2- 172.21.15.72:0/1523688222 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb72c108630 0x7fb72c19d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:16.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:13.662+0000 7fb72a59c700 1 -- 172.21.15.72:0/1523688222 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb72c06abc0 msgr2=0x7fb72c198e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:16.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:13.662+0000 7fb72a59c700 1 --2- 172.21.15.72:0/1523688222 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb72c06abc0 0x7fb72c198e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:16.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:13.662+0000 7fb72a59c700 1 -- 172.21.15.72:0/1523688222 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb72c19de50 con 0x7fb72c06b580 2024-03-22T01:35:16.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:13.662+0000 7fb72ad9d700 1 --2- 172.21.15.72:0/1523688222 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb72c06abc0 0x7fb72c198e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:35:16.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:13.663+0000 7fb72a59c700 1 --2- 172.21.15.72:0/1523688222 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb72c06b580 0x7fb72c1993d0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fb72000b530 tx=0x7fb72000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:16.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:13.663+0000 7fb713fff700 1 -- 172.21.15.72:0/1523688222 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb720015020 con 0x7fb72c06b580 2024-03-22T01:35:16.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:13.663+0000 7fb713fff700 1 -- 172.21.15.72:0/1523688222 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb720011d10 con 0x7fb72c06b580 2024-03-22T01:35:16.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:13.663+0000 7fb7311fa700 1 -- 172.21.15.72:0/1523688222 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb72c19e140 con 0x7fb72c06b580 2024-03-22T01:35:16.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:13.663+0000 7fb713fff700 1 -- 172.21.15.72:0/1523688222 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb72000f2e0 con 0x7fb72c06b580 2024-03-22T01:35:16.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:13.663+0000 7fb7311fa700 1 -- 172.21.15.72:0/1523688222 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb72c1015b0 con 0x7fb72c06b580 2024-03-22T01:35:16.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:13.664+0000 7fb7311fa700 1 -- 172.21.15.72:0/1523688222 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb72c062e10 con 0x7fb72c06b580 2024-03-22T01:35:16.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:13.665+0000 7fb713fff700 1 -- 172.21.15.72:0/1523688222 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7fb720018020 con 0x7fb72c06b580 2024-03-22T01:35:16.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:13.665+0000 7fb713fff700 1 --2- 172.21.15.72:0/1523688222 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fb7140a3040 0x7fb7140a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:16.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:13.666+0000 7fb72ad9d700 1 --2- 172.21.15.72:0/1523688222 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fb7140a3040 0x7fb7140a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:16.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:13.666+0000 7fb713fff700 1 -- 172.21.15.72:0/1523688222 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fb7200cf340 con 0x7fb72c06b580 2024-03-22T01:35:16.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:13.666+0000 7fb713fff700 1 --2- 172.21.15.72:0/1523688222 >> [v2:172.21.15.89:6808/2683760795,v1:172.21.15.89:6809/2683760795] conn(0x7fb7140add00 0x7fb7140b0140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:16.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:13.666+0000 7fb713fff700 1 -- 172.21.15.72:0/1523688222 --> [v2:172.21.15.89:6808/2683760795,v1:172.21.15.89:6809/2683760795] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb72001deb0 con 0x7fb7140add00 2024-03-22T01:35:16.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:13.666+0000 7fb713fff700 1 -- 172.21.15.72:0/1523688222 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fb7200cf790 con 0x7fb72c06b580 2024-03-22T01:35:16.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:13.666+0000 7fb72b59e700 1 --2- 172.21.15.72:0/1523688222 >> [v2:172.21.15.89:6808/2683760795,v1:172.21.15.89:6809/2683760795] conn(0x7fb7140add00 0x7fb7140b0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:16.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:13.666+0000 7fb72ad9d700 1 --2- 172.21.15.72:0/1523688222 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fb7140a3040 0x7fb7140a5500 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb71c000c00 tx=0x7fb71c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:16.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:13.666+0000 7fb72b59e700 1 --2- 172.21.15.72:0/1523688222 >> [v2:172.21.15.89:6808/2683760795,v1:172.21.15.89:6809/2683760795] conn(0x7fb7140add00 0x7fb7140b0140 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:16.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:13.668+0000 7fb713fff700 1 -- 172.21.15.72:0/1523688222 <== osd.4 v2:172.21.15.89:6808/2683760795 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb72001deb0 con 0x7fb7140add00 2024-03-22T01:35:16.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:13.684+0000 7fb7311fa700 1 -- 172.21.15.72:0/1523688222 --> [v2:172.21.15.89:6808/2683760795,v1:172.21.15.89:6809/2683760795] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb72c063290 con 0x7fb7140add00 2024-03-22T01:35:16.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:13.684+0000 7fb713fff700 1 -- 172.21.15.72:0/1523688222 <== osd.4 v2:172.21.15.89:6808/2683760795 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fb72c063290 con 0x7fb7140add00 2024-03-22T01:35:16.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:13.685+0000 7fb7311fa700 1 -- 172.21.15.72:0/1523688222 >> [v2:172.21.15.89:6808/2683760795,v1:172.21.15.89:6809/2683760795] conn(0x7fb7140add00 msgr2=0x7fb7140b0140 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:16.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:13.685+0000 7fb7311fa700 1 --2- 172.21.15.72:0/1523688222 >> [v2:172.21.15.89:6808/2683760795,v1:172.21.15.89:6809/2683760795] conn(0x7fb7140add00 0x7fb7140b0140 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:16.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:13.685+0000 7fb7311fa700 1 -- 172.21.15.72:0/1523688222 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fb7140a3040 msgr2=0x7fb7140a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:16.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:13.685+0000 7fb7311fa700 1 --2- 172.21.15.72:0/1523688222 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fb7140a3040 0x7fb7140a5500 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb71c000c00 tx=0x7fb71c005fb0 comp rx=0 tx=0).stop 2024-03-22T01:35:16.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:13.685+0000 7fb7311fa700 1 -- 172.21.15.72:0/1523688222 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb72c06b580 msgr2=0x7fb72c1993d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:16.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:13.685+0000 7fb7311fa700 1 --2- 172.21.15.72:0/1523688222 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb72c06b580 0x7fb72c1993d0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fb72000b530 tx=0x7fb72000b840 comp rx=0 tx=0).stop 2024-03-22T01:35:16.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:13.685+0000 7fb72b59e700 1 -- 172.21.15.72:0/1523688222 reap_dead start 2024-03-22T01:35:16.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:13.685+0000 7fb7311fa700 1 -- 172.21.15.72:0/1523688222 shutdown_connections 2024-03-22T01:35:16.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:13.685+0000 7fb7311fa700 1 -- 172.21.15.72:0/1523688222 >> 172.21.15.72:0/1523688222 conn(0x7fb72c0775d0 msgr2=0x7fb72c100250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:35:16.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:13.685+0000 7fb7311fa700 1 -- 172.21.15.72:0/1523688222 shutdown_connections 2024-03-22T01:35:16.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:13.685+0000 7fb7311fa700 1 -- 172.21.15.72:0/1523688222 wait complete. 2024-03-22T01:35:16.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:16 smithi089 ceph-mon[35770]: pgmap v321: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-22T01:35:16.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:16 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-03-22T01:35:16.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:16 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 1]}]': finished 2024-03-22T01:35:16.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:16 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]': finished 2024-03-22T01:35:16.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:16 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]': finished 2024-03-22T01:35:16.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:16 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [3, 9]}]': finished 2024-03-22T01:35:16.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:16 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-03-22T01:35:16.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:16 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8]}]': finished 2024-03-22T01:35:16.859 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:16 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2024-03-22T01:35:16.859 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:16 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6]}]': finished 2024-03-22T01:35:16.859 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:16 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 7]}]': finished 2024-03-22T01:35:16.859 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:16 smithi089 ceph-mon[35770]: osdmap e92: 12 total, 12 up, 12 in 2024-03-22T01:35:16.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:16 smithi072 ceph-mon[32414]: pgmap v321: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-22T01:35:16.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:16 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-03-22T01:35:16.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:16 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 1]}]': finished 2024-03-22T01:35:16.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:16 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]': finished 2024-03-22T01:35:16.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:16 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]': finished 2024-03-22T01:35:16.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:16 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [3, 9]}]': finished 2024-03-22T01:35:16.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:16 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-03-22T01:35:16.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:16 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8]}]': finished 2024-03-22T01:35:16.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:16 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2024-03-22T01:35:16.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:16 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6]}]': finished 2024-03-22T01:35:16.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:16 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 7]}]': finished 2024-03-22T01:35:16.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:16 smithi072 ceph-mon[32414]: osdmap e92: 12 total, 12 up, 12 in 2024-03-22T01:35:16.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:16 smithi098 ceph-mon[36468]: pgmap v321: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-22T01:35:16.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:16 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-03-22T01:35:16.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:16 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 1]}]': finished 2024-03-22T01:35:16.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:16 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]': finished 2024-03-22T01:35:16.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:16 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]': finished 2024-03-22T01:35:16.923 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:16 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [3, 9]}]': finished 2024-03-22T01:35:16.923 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:16 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-03-22T01:35:16.923 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:16 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8]}]': finished 2024-03-22T01:35:16.923 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:16 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2024-03-22T01:35:16.923 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:16 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6]}]': finished 2024-03-22T01:35:16.924 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:16 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 7]}]': finished 2024-03-22T01:35:16.924 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:16 smithi098 ceph-mon[36468]: osdmap e92: 12 total, 12 up, 12 in 2024-03-22T01:35:17.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.201+0000 7f38b3266700 1 -- 172.21.15.72:0/1894659507 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f38ac108660 msgr2=0x7f38ac108a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:17.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.201+0000 7f38b3266700 1 --2- 172.21.15.72:0/1894659507 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f38ac108660 0x7f38ac108a40 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f389c009990 tx=0x7f389c009ca0 comp rx=0 tx=0).stop 2024-03-22T01:35:17.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.201+0000 7f38b3266700 1 -- 172.21.15.72:0/1894659507 shutdown_connections 2024-03-22T01:35:17.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.201+0000 7f38b3266700 1 --2- 172.21.15.72:0/1894659507 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38ac1030c0 0x7f38ac10f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:17.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.201+0000 7f38b3266700 1 --2- 172.21.15.72:0/1894659507 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f38ac108660 0x7f38ac108a40 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:17.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.201+0000 7f38b3266700 1 --2- 172.21.15.72:0/1894659507 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38ac102700 0x7f38ac102b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:17.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.201+0000 7f38b3266700 1 -- 172.21.15.72:0/1894659507 >> 172.21.15.72:0/1894659507 conn(0x7f38ac079710 msgr2=0x7f38ac079b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:35:17.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.201+0000 7f38b3266700 1 -- 172.21.15.72:0/1894659507 shutdown_connections 2024-03-22T01:35:17.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.201+0000 7f38b3266700 1 -- 172.21.15.72:0/1894659507 wait complete. 2024-03-22T01:35:17.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.202+0000 7f38b3266700 1 Processor -- start 2024-03-22T01:35:17.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.202+0000 7f38b3266700 1 -- start start 2024-03-22T01:35:17.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.202+0000 7f38b3266700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38ac102700 0x7f38ac074980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:17.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.202+0000 7f38b3266700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38ac1030c0 0x7f38ac074ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:17.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.202+0000 7f38b3266700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f38ac108660 0x7f38ac075400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:17.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.202+0000 7f38b3266700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f38ac112970 con 0x7f38ac108660 2024-03-22T01:35:17.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.202+0000 7f38b3266700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f38ac1127f0 con 0x7f38ac102700 2024-03-22T01:35:17.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.202+0000 7f38b3266700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f38ac112af0 con 0x7f38ac1030c0 2024-03-22T01:35:17.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.202+0000 7f38b1803700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f38ac108660 0x7f38ac075400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:17.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.202+0000 7f38b0801700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38ac1030c0 0x7f38ac074ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:17.371 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.202+0000 7f38b0801700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38ac1030c0 0x7f38ac074ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.72:46814/0 (socket says 172.21.15.72:46814) 2024-03-22T01:35:17.371 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.202+0000 7f38b1803700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f38ac108660 0x7f38ac075400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36672/0 (socket says 172.21.15.72:36672) 2024-03-22T01:35:17.371 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.202+0000 7f38b1002700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38ac102700 0x7f38ac074980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:17.371 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.202+0000 7f38b0801700 1 -- 172.21.15.72:0/1557800036 learned_addr learned my addr 172.21.15.72:0/1557800036 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:35:17.371 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.202+0000 7f38b0801700 1 -- 172.21.15.72:0/1557800036 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38ac102700 msgr2=0x7f38ac074980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:17.371 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.202+0000 7f38b0801700 1 --2- 172.21.15.72:0/1557800036 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38ac102700 0x7f38ac074980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:17.371 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.202+0000 7f38b0801700 1 -- 172.21.15.72:0/1557800036 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f38ac108660 msgr2=0x7f38ac075400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:17.371 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.202+0000 7f38b0801700 1 --2- 172.21.15.72:0/1557800036 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f38ac108660 0x7f38ac075400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:17.371 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.202+0000 7f38b0801700 1 -- 172.21.15.72:0/1557800036 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38ac1a7390 con 0x7f38ac1030c0 2024-03-22T01:35:17.371 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.202+0000 7f38b1002700 1 --2- 172.21.15.72:0/1557800036 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38ac102700 0x7f38ac074980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:35:17.371 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.203+0000 7f38b0801700 1 --2- 172.21.15.72:0/1557800036 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38ac1030c0 0x7f38ac074ec0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f38a800e9c0 tx=0x7f38a800ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:17.371 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.203+0000 7f38a27fc700 1 -- 172.21.15.72:0/1557800036 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a800cbd0 con 0x7f38ac1030c0 2024-03-22T01:35:17.371 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.203+0000 7f38a27fc700 1 -- 172.21.15.72:0/1557800036 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38a800ee50 con 0x7f38ac1030c0 2024-03-22T01:35:17.371 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.203+0000 7f38b3266700 1 -- 172.21.15.72:0/1557800036 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38ac1a7680 con 0x7f38ac1030c0 2024-03-22T01:35:17.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.203+0000 7f38a27fc700 1 -- 172.21.15.72:0/1557800036 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a8005680 con 0x7f38ac1030c0 2024-03-22T01:35:17.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.203+0000 7f38b3266700 1 -- 172.21.15.72:0/1557800036 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38ac1a7b70 con 0x7f38ac1030c0 2024-03-22T01:35:17.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.204+0000 7f38b3266700 1 -- 172.21.15.72:0/1557800036 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f38ac062d90 con 0x7f38ac1030c0 2024-03-22T01:35:17.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.205+0000 7f38a27fc700 1 -- 172.21.15.72:0/1557800036 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7f38a8004750 con 0x7f38ac1030c0 2024-03-22T01:35:17.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.205+0000 7f38a27fc700 1 --2- 172.21.15.72:0/1557800036 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f38980a3040 0x7f38980a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:17.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.206+0000 7f38b1002700 1 --2- 172.21.15.72:0/1557800036 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f38980a3040 0x7f38980a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:17.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.206+0000 7f38a27fc700 1 -- 172.21.15.72:0/1557800036 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f38a80cfeb0 con 0x7f38ac1030c0 2024-03-22T01:35:17.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.206+0000 7f38a27fc700 1 --2- 172.21.15.72:0/1557800036 >> [v2:172.21.15.72:6826/3377749206,v1:172.21.15.72:6827/3377749206] conn(0x7f38980add00 0x7f38980b0140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:17.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.206+0000 7f38a27fc700 1 -- 172.21.15.72:0/1557800036 --> [v2:172.21.15.72:6826/3377749206,v1:172.21.15.72:6827/3377749206] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f38a8004500 con 0x7f38980add00 2024-03-22T01:35:17.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.206+0000 7f38a27fc700 1 -- 172.21.15.72:0/1557800036 <== mon.1 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f38a80d0260 con 0x7f38ac1030c0 2024-03-22T01:35:17.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.206+0000 7f38b1803700 1 --2- 172.21.15.72:0/1557800036 >> [v2:172.21.15.72:6826/3377749206,v1:172.21.15.72:6827/3377749206] conn(0x7f38980add00 0x7f38980b0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:17.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.206+0000 7f38b1002700 1 --2- 172.21.15.72:0/1557800036 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f38980a3040 0x7f38980a5500 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f389c00b2e0 tx=0x7f389c005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:17.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.206+0000 7f38b1803700 1 --2- 172.21.15.72:0/1557800036 >> [v2:172.21.15.72:6826/3377749206,v1:172.21.15.72:6827/3377749206] conn(0x7f38980add00 0x7f38980b0140 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:17.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.207+0000 7f38a27fc700 1 -- 172.21.15.72:0/1557800036 <== osd.9 v2:172.21.15.72:6826/3377749206 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f38a8004500 con 0x7f38980add00 2024-03-22T01:35:17.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.224+0000 7f38b3266700 1 -- 172.21.15.72:0/1557800036 --> [v2:172.21.15.72:6826/3377749206,v1:172.21.15.72:6827/3377749206] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f38ac063210 con 0x7f38980add00 2024-03-22T01:35:17.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.224+0000 7f38a27fc700 1 -- 172.21.15.72:0/1557800036 <== osd.9 v2:172.21.15.72:6826/3377749206 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f38ac063210 con 0x7f38980add00 2024-03-22T01:35:17.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.225+0000 7f38b3266700 1 -- 172.21.15.72:0/1557800036 >> [v2:172.21.15.72:6826/3377749206,v1:172.21.15.72:6827/3377749206] conn(0x7f38980add00 msgr2=0x7f38980b0140 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:17.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.225+0000 7f38b3266700 1 --2- 172.21.15.72:0/1557800036 >> [v2:172.21.15.72:6826/3377749206,v1:172.21.15.72:6827/3377749206] conn(0x7f38980add00 0x7f38980b0140 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:17.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.225+0000 7f38b3266700 1 -- 172.21.15.72:0/1557800036 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f38980a3040 msgr2=0x7f38980a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:17.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.225+0000 7f38b3266700 1 --2- 172.21.15.72:0/1557800036 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f38980a3040 0x7f38980a5500 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f389c00b2e0 tx=0x7f389c005960 comp rx=0 tx=0).stop 2024-03-22T01:35:17.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.225+0000 7f38b3266700 1 -- 172.21.15.72:0/1557800036 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38ac1030c0 msgr2=0x7f38ac074ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:17.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.225+0000 7f38b3266700 1 --2- 172.21.15.72:0/1557800036 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38ac1030c0 0x7f38ac074ec0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f38a800e9c0 tx=0x7f38a800ecd0 comp rx=0 tx=0).stop 2024-03-22T01:35:17.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.225+0000 7f38b1803700 1 -- 172.21.15.72:0/1557800036 reap_dead start 2024-03-22T01:35:17.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.225+0000 7f38b3266700 1 -- 172.21.15.72:0/1557800036 shutdown_connections 2024-03-22T01:35:17.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.225+0000 7f38b3266700 1 -- 172.21.15.72:0/1557800036 >> 172.21.15.72:0/1557800036 conn(0x7f38ac079710 msgr2=0x7f38ac1014a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:35:17.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.225+0000 7f38b3266700 1 -- 172.21.15.72:0/1557800036 shutdown_connections 2024-03-22T01:35:17.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:15.225+0000 7f38b3266700 1 -- 172.21.15.72:0/1557800036 wait complete. 2024-03-22T01:35:17.954 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:17 smithi072 ceph-mon[32414]: osdmap e93: 12 total, 12 up, 12 in 2024-03-22T01:35:17.954 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:17 smithi072 ceph-mon[32414]: pgmap v324: 129 pgs: 2 peering, 127 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-22T01:35:17.985 INFO:teuthology.orchestra.run.smithi072.stdout:292057776164 2024-03-22T01:35:17.985 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.9 2024-03-22T01:35:18.107 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:17 smithi089 ceph-mon[35770]: osdmap e93: 12 total, 12 up, 12 in 2024-03-22T01:35:18.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:17 smithi089 ceph-mon[35770]: pgmap v324: 129 pgs: 2 peering, 127 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-22T01:35:18.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:17 smithi098 ceph-mon[36468]: osdmap e93: 12 total, 12 up, 12 in 2024-03-22T01:35:18.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:17 smithi098 ceph-mon[36468]: pgmap v324: 129 pgs: 2 peering, 127 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-22T01:35:18.496 INFO:teuthology.orchestra.run.smithi072.stdout:352187318297 2024-03-22T01:35:18.497 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.11 2024-03-22T01:35:19.003 INFO:teuthology.orchestra.run.smithi072.stdout:34359738448 2024-03-22T01:35:19.003 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-22T01:35:19.169 INFO:teuthology.orchestra.run.smithi072.stdout:146028888125 2024-03-22T01:35:19.169 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-22T01:35:20.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:20.055+0000 7fb072e5e700 1 -- 172.21.15.72:0/2538197618 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb06c107550 msgr2=0x7fb06c107930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:20.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:20.055+0000 7fb072e5e700 1 --2- 172.21.15.72:0/2538197618 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb06c107550 0x7fb06c107930 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fb06400b0d0 tx=0x7fb06400b3e0 comp rx=0 tx=0).stop 2024-03-22T01:35:20.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:20.055+0000 7fb072e5e700 1 -- 172.21.15.72:0/2538197618 shutdown_connections 2024-03-22T01:35:20.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:20.055+0000 7fb072e5e700 1 --2- 172.21.15.72:0/2538197618 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb06c1030c0 0x7fb06c10f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:20.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:20.055+0000 7fb072e5e700 1 --2- 172.21.15.72:0/2538197618 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb06c107550 0x7fb06c107930 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:20.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:20.055+0000 7fb072e5e700 1 --2- 172.21.15.72:0/2538197618 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb06c102700 0x7fb06c102b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:20.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:20.055+0000 7fb072e5e700 1 -- 172.21.15.72:0/2538197618 >> 172.21.15.72:0/2538197618 conn(0x7fb06c0797e0 msgr2=0x7fb06c079bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:35:20.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:20.055+0000 7fb072e5e700 1 -- 172.21.15.72:0/2538197618 shutdown_connections 2024-03-22T01:35:20.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:20.055+0000 7fb072e5e700 1 -- 172.21.15.72:0/2538197618 wait complete. 2024-03-22T01:35:20.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:20.056+0000 7fb072e5e700 1 Processor -- start 2024-03-22T01:35:20.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:20.056+0000 7fb072e5e700 1 -- start start 2024-03-22T01:35:20.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:20.056+0000 7fb072e5e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb06c102700 0x7fb06c074980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:20.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:20.056+0000 7fb072e5e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb06c1030c0 0x7fb06c074ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:20.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:20.056+0000 7fb072e5e700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb06c107550 0x7fb06c075400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:20.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:20.056+0000 7fb072e5e700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fb06c112a60 con 0x7fb06c107550 2024-03-22T01:35:20.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:20.056+0000 7fb072e5e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb06c1128e0 con 0x7fb06c102700 2024-03-22T01:35:20.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:20.056+0000 7fb072e5e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb06c112be0 con 0x7fb06c1030c0 2024-03-22T01:35:20.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:20.056+0000 7fb0713fb700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb06c107550 0x7fb06c075400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:20.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:20.056+0000 7fb0713fb700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb06c107550 0x7fb06c075400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36718/0 (socket says 172.21.15.72:36718) 2024-03-22T01:35:20.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:20.056+0000 7fb0713fb700 1 -- 172.21.15.72:0/880304951 learned_addr learned my addr 172.21.15.72:0/880304951 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:35:20.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:20.056+0000 7fb070bfa700 1 --2- 172.21.15.72:0/880304951 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb06c102700 0x7fb06c074980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:20.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:20.057+0000 7fb06bfff700 1 --2- 172.21.15.72:0/880304951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb06c1030c0 0x7fb06c074ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:20.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:20.057+0000 7fb0713fb700 1 -- 172.21.15.72:0/880304951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb06c1030c0 msgr2=0x7fb06c074ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:20.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:20.057+0000 7fb0713fb700 1 --2- 172.21.15.72:0/880304951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb06c1030c0 0x7fb06c074ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:20.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:20.057+0000 7fb0713fb700 1 -- 172.21.15.72:0/880304951 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb06c102700 msgr2=0x7fb06c074980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:20.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:20.057+0000 7fb0713fb700 1 --2- 172.21.15.72:0/880304951 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb06c102700 0x7fb06c074980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:20.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:20.057+0000 7fb0713fb700 1 -- 172.21.15.72:0/880304951 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb06c1a73e0 con 0x7fb06c107550 2024-03-22T01:35:20.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:20.057+0000 7fb0713fb700 1 --2- 172.21.15.72:0/880304951 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb06c107550 0x7fb06c075400 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fb05c00e970 tx=0x7fb05c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:20.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:20.057+0000 7fb069ffb700 1 -- 172.21.15.72:0/880304951 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb05c004a30 con 0x7fb06c107550 2024-03-22T01:35:20.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:20.057+0000 7fb069ffb700 1 -- 172.21.15.72:0/880304951 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb05c005500 con 0x7fb06c107550 2024-03-22T01:35:20.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:20.057+0000 7fb072e5e700 1 -- 172.21.15.72:0/880304951 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb06c1a76d0 con 0x7fb06c107550 2024-03-22T01:35:20.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:20.057+0000 7fb069ffb700 1 -- 172.21.15.72:0/880304951 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb05c00cda0 con 0x7fb06c107550 2024-03-22T01:35:20.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:20.057+0000 7fb072e5e700 1 -- 172.21.15.72:0/880304951 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb06c1a7bc0 con 0x7fb06c107550 2024-03-22T01:35:20.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:20.058+0000 7fb072e5e700 1 -- 172.21.15.72:0/880304951 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb06c062d90 con 0x7fb06c107550 2024-03-22T01:35:20.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:20.059+0000 7fb069ffb700 1 -- 172.21.15.72:0/880304951 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7fb05c005000 con 0x7fb06c107550 2024-03-22T01:35:20.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:20.060+0000 7fb069ffb700 1 --2- 172.21.15.72:0/880304951 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fb0580a3040 0x7fb0580a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:20.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:20.060+0000 7fb070bfa700 1 --2- 172.21.15.72:0/880304951 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fb0580a3040 0x7fb0580a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:20.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:20.060+0000 7fb069ffb700 1 -- 172.21.15.72:0/880304951 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fb05c0d0260 con 0x7fb06c107550 2024-03-22T01:35:20.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:20.060+0000 7fb069ffb700 1 --2- 172.21.15.72:0/880304951 >> [v2:172.21.15.89:6824/885058786,v1:172.21.15.89:6825/885058786] conn(0x7fb0580ae1a0 0x7fb0580b0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:20.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:20.060+0000 7fb069ffb700 1 -- 172.21.15.72:0/880304951 --> [v2:172.21.15.89:6824/885058786,v1:172.21.15.89:6825/885058786] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb0580b0d10 con 0x7fb0580ae1a0 2024-03-22T01:35:20.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:20.060+0000 7fb069ffb700 1 -- 172.21.15.72:0/880304951 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7fb05c0d06b0 con 0x7fb06c107550 2024-03-22T01:35:20.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:20.060+0000 7fb06bfff700 1 --2- 172.21.15.72:0/880304951 >> [v2:172.21.15.89:6824/885058786,v1:172.21.15.89:6825/885058786] conn(0x7fb0580ae1a0 0x7fb0580b0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:20.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:20.060+0000 7fb070bfa700 1 --2- 172.21.15.72:0/880304951 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fb0580a3040 0x7fb0580a5500 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fb064009fd0 tx=0x7fb06400b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:20.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:20.060+0000 7fb06bfff700 1 --2- 172.21.15.72:0/880304951 >> [v2:172.21.15.89:6824/885058786,v1:172.21.15.89:6825/885058786] conn(0x7fb0580ae1a0 0x7fb0580b0640 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:20.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:20.062+0000 7fb069ffb700 1 -- 172.21.15.72:0/880304951 <== osd.10 v2:172.21.15.89:6824/885058786 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb0580b0d10 con 0x7fb0580ae1a0 2024-03-22T01:35:20.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:20.078+0000 7fb072e5e700 1 -- 172.21.15.72:0/880304951 --> [v2:172.21.15.89:6824/885058786,v1:172.21.15.89:6825/885058786] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb06c063210 con 0x7fb0580ae1a0 2024-03-22T01:35:20.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:20.078+0000 7fb069ffb700 1 -- 172.21.15.72:0/880304951 <== osd.10 v2:172.21.15.89:6824/885058786 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fb06c063210 con 0x7fb0580ae1a0 2024-03-22T01:35:20.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:20.078+0000 7fb072e5e700 1 -- 172.21.15.72:0/880304951 >> [v2:172.21.15.89:6824/885058786,v1:172.21.15.89:6825/885058786] conn(0x7fb0580ae1a0 msgr2=0x7fb0580b0640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:20.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:20.078+0000 7fb072e5e700 1 --2- 172.21.15.72:0/880304951 >> [v2:172.21.15.89:6824/885058786,v1:172.21.15.89:6825/885058786] conn(0x7fb0580ae1a0 0x7fb0580b0640 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:20.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:20.079+0000 7fb072e5e700 1 -- 172.21.15.72:0/880304951 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fb0580a3040 msgr2=0x7fb0580a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:20.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:20.079+0000 7fb072e5e700 1 --2- 172.21.15.72:0/880304951 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fb0580a3040 0x7fb0580a5500 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fb064009fd0 tx=0x7fb06400b040 comp rx=0 tx=0).stop 2024-03-22T01:35:20.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:20.079+0000 7fb072e5e700 1 -- 172.21.15.72:0/880304951 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb06c107550 msgr2=0x7fb06c075400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:20.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:20.079+0000 7fb072e5e700 1 --2- 172.21.15.72:0/880304951 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb06c107550 0x7fb06c075400 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fb05c00e970 tx=0x7fb05c00ec80 comp rx=0 tx=0).stop 2024-03-22T01:35:20.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:20.079+0000 7fb0713fb700 1 -- 172.21.15.72:0/880304951 reap_dead start 2024-03-22T01:35:20.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:20.079+0000 7fb072e5e700 1 -- 172.21.15.72:0/880304951 shutdown_connections 2024-03-22T01:35:20.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:20.079+0000 7fb072e5e700 1 -- 172.21.15.72:0/880304951 >> 172.21.15.72:0/880304951 conn(0x7fb06c0797e0 msgr2=0x7fb06c100770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:35:20.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:20.079+0000 7fb072e5e700 1 -- 172.21.15.72:0/880304951 shutdown_connections 2024-03-22T01:35:20.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:20.079+0000 7fb072e5e700 1 -- 172.21.15.72:0/880304951 wait complete. 2024-03-22T01:35:20.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:20 smithi072 ceph-mon[32414]: pgmap v325: 129 pgs: 2 peering, 127 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 1 op/s 2024-03-22T01:35:20.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:20 smithi089 ceph-mon[35770]: pgmap v325: 129 pgs: 2 peering, 127 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 1 op/s 2024-03-22T01:35:20.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:20 smithi098 ceph-mon[36468]: pgmap v325: 129 pgs: 2 peering, 127 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 1 op/s 2024-03-22T01:35:21.655 INFO:teuthology.orchestra.run.smithi072.stdout:317827579935 2024-03-22T01:35:21.655 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.10 2024-03-22T01:35:21.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:21 smithi089 ceph-mon[35770]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-22T01:35:21.868 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:21 smithi072 ceph-mon[32414]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-22T01:35:21.872 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:35:21.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:21 smithi098 ceph-mon[36468]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-22T01:35:22.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:22.152+0000 7f9aeeba2700 1 -- 172.21.15.72:0/2949392829 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ae8101a40 msgr2=0x7f9ae810f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:22.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:22.152+0000 7f9aeeba2700 1 --2- 172.21.15.72:0/2949392829 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ae8101a40 0x7f9ae810f5d0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f9ae000b0d0 tx=0x7f9ae000b3e0 comp rx=0 tx=0).stop 2024-03-22T01:35:22.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:22.152+0000 7f9aeeba2700 1 -- 172.21.15.72:0/2949392829 shutdown_connections 2024-03-22T01:35:22.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:22.152+0000 7f9aeeba2700 1 --2- 172.21.15.72:0/2949392829 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ae8101a40 0x7f9ae810f5d0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:22.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:22.152+0000 7f9aeeba2700 1 --2- 172.21.15.72:0/2949392829 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ae8107750 0x7f9ae8107b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:22.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:22.152+0000 7f9aeeba2700 1 --2- 172.21.15.72:0/2949392829 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ae8101080 0x7f9ae8101500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:22.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:22.152+0000 7f9aeeba2700 1 -- 172.21.15.72:0/2949392829 >> 172.21.15.72:0/2949392829 conn(0x7f9ae80768d0 msgr2=0x7f9ae8076ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:35:22.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:22.153+0000 7f9aeeba2700 1 -- 172.21.15.72:0/2949392829 shutdown_connections 2024-03-22T01:35:22.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:22.153+0000 7f9aeeba2700 1 -- 172.21.15.72:0/2949392829 wait complete. 2024-03-22T01:35:22.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:22.153+0000 7f9aeeba2700 1 Processor -- start 2024-03-22T01:35:22.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:22.153+0000 7f9aeeba2700 1 -- start start 2024-03-22T01:35:22.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:22.153+0000 7f9aeeba2700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ae8101080 0x7f9ae8102260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:22.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:22.153+0000 7f9aeeba2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ae8101a40 0x7f9ae81027a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:22.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:22.153+0000 7f9aeeba2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ae8107750 0x7f9ae8102ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:22.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:22.153+0000 7f9aeeba2700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f9ae8112ab0 con 0x7f9ae8101080 2024-03-22T01:35:22.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:22.153+0000 7f9aeeba2700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9ae8112930 con 0x7f9ae8101a40 2024-03-22T01:35:22.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:22.153+0000 7f9aeeba2700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9ae8112c30 con 0x7f9ae8107750 2024-03-22T01:35:22.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:22.153+0000 7f9aec93e700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ae8101080 0x7f9ae8102260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:22.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:22.153+0000 7f9aec93e700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ae8101080 0x7f9ae8102260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60060/0 (socket says 172.21.15.72:60060) 2024-03-22T01:35:22.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:22.153+0000 7f9aec93e700 1 -- 172.21.15.72:0/3847051526 learned_addr learned my addr 172.21.15.72:0/3847051526 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:35:22.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:22.153+0000 7f9ae7fff700 1 --2- 172.21.15.72:0/3847051526 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ae8101a40 0x7f9ae81027a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:22.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:22.154+0000 7f9aed13f700 1 --2- 172.21.15.72:0/3847051526 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ae8107750 0x7f9ae8102ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:22.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:22.154+0000 7f9aec93e700 1 -- 172.21.15.72:0/3847051526 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ae8107750 msgr2=0x7f9ae8102ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:22.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:22.154+0000 7f9aec93e700 1 --2- 172.21.15.72:0/3847051526 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ae8107750 0x7f9ae8102ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:22.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:22.154+0000 7f9aec93e700 1 -- 172.21.15.72:0/3847051526 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ae8101a40 msgr2=0x7f9ae81027a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:22.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:22.154+0000 7f9aec93e700 1 --2- 172.21.15.72:0/3847051526 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ae8101a40 0x7f9ae81027a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:22.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:22.154+0000 7f9aec93e700 1 -- 172.21.15.72:0/3847051526 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ae81a7370 con 0x7f9ae8101080 2024-03-22T01:35:22.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:22.154+0000 7f9ae7fff700 1 --2- 172.21.15.72:0/3847051526 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ae8101a40 0x7f9ae81027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:35:22.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:22.154+0000 7f9aec93e700 1 --2- 172.21.15.72:0/3847051526 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ae8101080 0x7f9ae8102260 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f9adc00e9a0 tx=0x7f9adc00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:22.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:22.154+0000 7f9ae5ffb700 1 -- 172.21.15.72:0/3847051526 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9adc00ca10 con 0x7f9ae8101080 2024-03-22T01:35:22.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:22.154+0000 7f9ae5ffb700 1 -- 172.21.15.72:0/3847051526 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9adc00cbb0 con 0x7f9ae8101080 2024-03-22T01:35:22.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:22.154+0000 7f9aeeba2700 1 -- 172.21.15.72:0/3847051526 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ae81a7660 con 0x7f9ae8101080 2024-03-22T01:35:22.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:22.154+0000 7f9ae5ffb700 1 -- 172.21.15.72:0/3847051526 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9adc018680 con 0x7f9ae8101080 2024-03-22T01:35:22.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:22.154+0000 7f9aeeba2700 1 -- 172.21.15.72:0/3847051526 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ae81a7bf0 con 0x7f9ae8101080 2024-03-22T01:35:22.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:22.155+0000 7f9aeeba2700 1 -- 172.21.15.72:0/3847051526 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9ae80ff990 con 0x7f9ae8101080 2024-03-22T01:35:22.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:22.157+0000 7f9ae5ffb700 1 -- 172.21.15.72:0/3847051526 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7f9adc010430 con 0x7f9ae8101080 2024-03-22T01:35:22.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:22.157+0000 7f9ae5ffb700 1 --2- 172.21.15.72:0/3847051526 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f9ad80a30c0 0x7f9ad80a5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:22.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:22.157+0000 7f9ae7fff700 1 --2- 172.21.15.72:0/3847051526 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f9ad80a30c0 0x7f9ad80a5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:22.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:22.157+0000 7f9ae5ffb700 1 -- 172.21.15.72:0/3847051526 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f9adc0d02f0 con 0x7f9ae8101080 2024-03-22T01:35:22.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:22.157+0000 7f9ae5ffb700 1 --2- 172.21.15.72:0/3847051526 >> [v2:172.21.15.98:6800/3865778118,v1:172.21.15.98:6801/3865778118] conn(0x7f9ad80ae220 0x7f9ad80b06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:22.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:22.157+0000 7f9ae5ffb700 1 -- 172.21.15.72:0/3847051526 --> [v2:172.21.15.98:6800/3865778118,v1:172.21.15.98:6801/3865778118] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9adc01bec0 con 0x7f9ad80ae220 2024-03-22T01:35:22.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:22.157+0000 7f9ae5ffb700 1 -- 172.21.15.72:0/3847051526 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7f9adc0d0740 con 0x7f9ae8101080 2024-03-22T01:35:22.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:22.158+0000 7f9ae7fff700 1 --2- 172.21.15.72:0/3847051526 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f9ad80a30c0 0x7f9ad80a5580 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f9ae8076d60 tx=0x7f9ad4009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:22.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:22.158+0000 7f9aed13f700 1 --2- 172.21.15.72:0/3847051526 >> [v2:172.21.15.98:6800/3865778118,v1:172.21.15.98:6801/3865778118] conn(0x7f9ad80ae220 0x7f9ad80b06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:22.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:22.158+0000 7f9aed13f700 1 --2- 172.21.15.72:0/3847051526 >> [v2:172.21.15.98:6800/3865778118,v1:172.21.15.98:6801/3865778118] conn(0x7f9ad80ae220 0x7f9ad80b06c0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:22.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:22.159+0000 7f9ae5ffb700 1 -- 172.21.15.72:0/3847051526 <== osd.2 v2:172.21.15.98:6800/3865778118 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9adc01bec0 con 0x7f9ad80ae220 2024-03-22T01:35:22.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:22.175+0000 7f9aeeba2700 1 -- 172.21.15.72:0/3847051526 --> [v2:172.21.15.98:6800/3865778118,v1:172.21.15.98:6801/3865778118] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9ae806a800 con 0x7f9ad80ae220 2024-03-22T01:35:22.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:22.176+0000 7f9ae5ffb700 1 -- 172.21.15.72:0/3847051526 <== osd.2 v2:172.21.15.98:6800/3865778118 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f9ae806a800 con 0x7f9ad80ae220 2024-03-22T01:35:22.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:22.176+0000 7f9aeeba2700 1 -- 172.21.15.72:0/3847051526 >> [v2:172.21.15.98:6800/3865778118,v1:172.21.15.98:6801/3865778118] conn(0x7f9ad80ae220 msgr2=0x7f9ad80b06c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:22.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:22.176+0000 7f9aeeba2700 1 --2- 172.21.15.72:0/3847051526 >> [v2:172.21.15.98:6800/3865778118,v1:172.21.15.98:6801/3865778118] conn(0x7f9ad80ae220 0x7f9ad80b06c0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:22.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:22.176+0000 7f9aeeba2700 1 -- 172.21.15.72:0/3847051526 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f9ad80a30c0 msgr2=0x7f9ad80a5580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:22.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:22.176+0000 7f9aeeba2700 1 --2- 172.21.15.72:0/3847051526 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f9ad80a30c0 0x7f9ad80a5580 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f9ae8076d60 tx=0x7f9ad4009190 comp rx=0 tx=0).stop 2024-03-22T01:35:22.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:22.176+0000 7f9aeeba2700 1 -- 172.21.15.72:0/3847051526 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ae8101080 msgr2=0x7f9ae8102260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:22.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:22.176+0000 7f9aeeba2700 1 --2- 172.21.15.72:0/3847051526 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ae8101080 0x7f9ae8102260 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f9adc00e9a0 tx=0x7f9adc00ecb0 comp rx=0 tx=0).stop 2024-03-22T01:35:22.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:22.176+0000 7f9aed13f700 1 -- 172.21.15.72:0/3847051526 reap_dead start 2024-03-22T01:35:22.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:22.176+0000 7f9aeeba2700 1 -- 172.21.15.72:0/3847051526 shutdown_connections 2024-03-22T01:35:22.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:22.176+0000 7f9aeeba2700 1 -- 172.21.15.72:0/3847051526 >> 172.21.15.72:0/3847051526 conn(0x7f9ae80768d0 msgr2=0x7f9ae80fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:35:22.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:22.176+0000 7f9aeeba2700 1 -- 172.21.15.72:0/3847051526 shutdown_connections 2024-03-22T01:35:22.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:22.176+0000 7f9aeeba2700 1 -- 172.21.15.72:0/3847051526 wait complete. 2024-03-22T01:35:22.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:22 smithi089 ceph-mon[35770]: pgmap v326: 129 pgs: 1 peering, 128 active+clean; 639 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 0 B/s, 0 objects/s recovering 2024-03-22T01:35:22.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:22 smithi072 ceph-mon[32414]: pgmap v326: 129 pgs: 1 peering, 128 active+clean; 639 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 0 B/s, 0 objects/s recovering 2024-03-22T01:35:22.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:22 smithi098 ceph-mon[36468]: pgmap v326: 129 pgs: 1 peering, 128 active+clean; 639 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 0 B/s, 0 objects/s recovering 2024-03-22T01:35:23.108 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:35:23.109 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:35:23.109 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:35:23.110 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:35:23.112 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:35:23.113 INFO:teuthology.orchestra.run.smithi072.stdout:77309411399 2024-03-22T01:35:23.113 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-22T01:35:23.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:23 smithi089 ceph-mon[35770]: pgmap v327: 129 pgs: 129 active+clean; 639 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-22T01:35:23.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:23 smithi089 ceph-mon[35770]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-22T01:35:23.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:23 smithi089 ceph-mon[35770]: Cluster is now healthy 2024-03-22T01:35:23.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:23 smithi072 ceph-mon[32414]: pgmap v327: 129 pgs: 129 active+clean; 639 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-22T01:35:23.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:23 smithi072 ceph-mon[32414]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-22T01:35:23.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:23 smithi072 ceph-mon[32414]: Cluster is now healthy 2024-03-22T01:35:23.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:23 smithi098 ceph-mon[36468]: pgmap v327: 129 pgs: 129 active+clean; 639 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-22T01:35:23.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:23 smithi098 ceph-mon[36468]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-22T01:35:23.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:23 smithi098 ceph-mon[36468]: Cluster is now healthy 2024-03-22T01:35:24.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:24.050+0000 7fd9fcf3c700 1 -- 172.21.15.72:0/2658227934 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9f8107680 msgr2=0x7fd9f8107a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:24.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:24.050+0000 7fd9fcf3c700 1 --2- 172.21.15.72:0/2658227934 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9f8107680 0x7fd9f8107a60 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fd9e8009990 tx=0x7fd9e8009ca0 comp rx=0 tx=0).stop 2024-03-22T01:35:24.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:24.050+0000 7fd9fcf3c700 1 -- 172.21.15.72:0/2658227934 shutdown_connections 2024-03-22T01:35:24.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:24.051+0000 7fd9fcf3c700 1 --2- 172.21.15.72:0/2658227934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9f8101fb0 0x7fd9f810e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:24.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:24.051+0000 7fd9fcf3c700 1 --2- 172.21.15.72:0/2658227934 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9f8107680 0x7fd9f8107a60 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:24.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:24.051+0000 7fd9fcf3c700 1 --2- 172.21.15.72:0/2658227934 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9f81015f0 0x7fd9f8101a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:24.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:24.051+0000 7fd9fcf3c700 1 -- 172.21.15.72:0/2658227934 >> 172.21.15.72:0/2658227934 conn(0x7fd9f80786b0 msgr2=0x7fd9f8078ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:35:24.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:24.051+0000 7fd9fcf3c700 1 -- 172.21.15.72:0/2658227934 shutdown_connections 2024-03-22T01:35:24.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:24.051+0000 7fd9fcf3c700 1 -- 172.21.15.72:0/2658227934 wait complete. 2024-03-22T01:35:24.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:24.052+0000 7fd9fcf3c700 1 Processor -- start 2024-03-22T01:35:24.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:24.052+0000 7fd9fcf3c700 1 -- start start 2024-03-22T01:35:24.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:24.052+0000 7fd9fcf3c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9f81015f0 0x7fd9f8198dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:24.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:24.052+0000 7fd9fcf3c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9f8101fb0 0x7fd9f8199300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:24.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:24.052+0000 7fd9fcf3c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9f8107680 0x7fd9f819d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:24.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:24.052+0000 7fd9fcf3c700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f8112a30 con 0x7fd9f81015f0 2024-03-22T01:35:24.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:24.052+0000 7fd9fcf3c700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f81128b0 con 0x7fd9f8107680 2024-03-22T01:35:24.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:24.052+0000 7fd9fcf3c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f8112bb0 con 0x7fd9f8101fb0 2024-03-22T01:35:24.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:24.052+0000 7fd9f659c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9f81015f0 0x7fd9f8198dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:24.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:24.052+0000 7fd9f659c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9f81015f0 0x7fd9f8198dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60078/0 (socket says 172.21.15.72:60078) 2024-03-22T01:35:24.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:24.052+0000 7fd9f659c700 1 -- 172.21.15.72:0/485865986 learned_addr learned my addr 172.21.15.72:0/485865986 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:35:24.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:24.052+0000 7fd9f6d9d700 1 --2- 172.21.15.72:0/485865986 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9f8107680 0x7fd9f819d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:24.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:24.052+0000 7fd9f5d9b700 1 --2- 172.21.15.72:0/485865986 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9f8101fb0 0x7fd9f8199300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:24.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:24.052+0000 7fd9f659c700 1 -- 172.21.15.72:0/485865986 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9f8101fb0 msgr2=0x7fd9f8199300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:24.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:24.052+0000 7fd9f659c700 1 --2- 172.21.15.72:0/485865986 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9f8101fb0 0x7fd9f8199300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:24.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:24.052+0000 7fd9f659c700 1 -- 172.21.15.72:0/485865986 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9f8107680 msgr2=0x7fd9f819d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:24.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:24.052+0000 7fd9f659c700 1 --2- 172.21.15.72:0/485865986 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9f8107680 0x7fd9f819d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:24.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:24.052+0000 7fd9f659c700 1 -- 172.21.15.72:0/485865986 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9f819ddd0 con 0x7fd9f81015f0 2024-03-22T01:35:24.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:24.053+0000 7fd9f5d9b700 1 --2- 172.21.15.72:0/485865986 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9f8101fb0 0x7fd9f8199300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T01:35:24.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:24.053+0000 7fd9f6d9d700 1 --2- 172.21.15.72:0/485865986 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9f8107680 0x7fd9f819d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:35:24.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:24.053+0000 7fd9f659c700 1 --2- 172.21.15.72:0/485865986 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9f81015f0 0x7fd9f8198dc0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fd9e8000c00 tx=0x7fd9e800bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:24.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:24.053+0000 7fd9e77fe700 1 -- 172.21.15.72:0/485865986 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9e8027070 con 0x7fd9f81015f0 2024-03-22T01:35:24.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:24.053+0000 7fd9fcf3c700 1 -- 172.21.15.72:0/485865986 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9f819e060 con 0x7fd9f81015f0 2024-03-22T01:35:24.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:24.053+0000 7fd9e77fe700 1 -- 172.21.15.72:0/485865986 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd9e8003d70 con 0x7fd9f81015f0 2024-03-22T01:35:24.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:24.053+0000 7fd9e77fe700 1 -- 172.21.15.72:0/485865986 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9e800f6c0 con 0x7fd9f81015f0 2024-03-22T01:35:24.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:24.053+0000 7fd9fcf3c700 1 -- 172.21.15.72:0/485865986 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9f81a5940 con 0x7fd9f81015f0 2024-03-22T01:35:24.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:24.054+0000 7fd9fcf3c700 1 -- 172.21.15.72:0/485865986 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd9f8100920 con 0x7fd9f81015f0 2024-03-22T01:35:24.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:24.055+0000 7fd9e77fe700 1 -- 172.21.15.72:0/485865986 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7fd9e802a070 con 0x7fd9f81015f0 2024-03-22T01:35:24.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:24.056+0000 7fd9e77fe700 1 --2- 172.21.15.72:0/485865986 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fd9e00a3040 0x7fd9e00a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:24.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:24.056+0000 7fd9f5d9b700 1 --2- 172.21.15.72:0/485865986 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fd9e00a3040 0x7fd9e00a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:24.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:24.056+0000 7fd9e77fe700 1 -- 172.21.15.72:0/485865986 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fd9e80d1350 con 0x7fd9f81015f0 2024-03-22T01:35:24.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:24.056+0000 7fd9e77fe700 1 --2- 172.21.15.72:0/485865986 >> [v2:172.21.15.72:6810/1865951395,v1:172.21.15.72:6811/1865951395] conn(0x7fd9e00ae1a0 0x7fd9e00b0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:24.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:24.056+0000 7fd9e77fe700 1 -- 172.21.15.72:0/485865986 --> [v2:172.21.15.72:6810/1865951395,v1:172.21.15.72:6811/1865951395] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd9e8009030 con 0x7fd9e00ae1a0 2024-03-22T01:35:24.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:24.056+0000 7fd9e77fe700 1 -- 172.21.15.72:0/485865986 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7fd9e80d17a0 con 0x7fd9f81015f0 2024-03-22T01:35:24.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:24.056+0000 7fd9f5d9b700 1 --2- 172.21.15.72:0/485865986 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fd9e00a3040 0x7fd9e00a5500 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fd9f8078b60 tx=0x7fd9ec009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:24.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:24.056+0000 7fd9f6d9d700 1 --2- 172.21.15.72:0/485865986 >> [v2:172.21.15.72:6810/1865951395,v1:172.21.15.72:6811/1865951395] conn(0x7fd9e00ae1a0 0x7fd9e00b0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:24.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:24.057+0000 7fd9f6d9d700 1 --2- 172.21.15.72:0/485865986 >> [v2:172.21.15.72:6810/1865951395,v1:172.21.15.72:6811/1865951395] conn(0x7fd9e00ae1a0 0x7fd9e00b0640 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:24.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:24.058+0000 7fd9e77fe700 1 -- 172.21.15.72:0/485865986 <== osd.3 v2:172.21.15.72:6810/1865951395 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd9e8009030 con 0x7fd9e00ae1a0 2024-03-22T01:35:24.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:24.074+0000 7fd9fcf3c700 1 -- 172.21.15.72:0/485865986 --> [v2:172.21.15.72:6810/1865951395,v1:172.21.15.72:6811/1865951395] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd9f8102d30 con 0x7fd9e00ae1a0 2024-03-22T01:35:24.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:24.074+0000 7fd9e77fe700 1 -- 172.21.15.72:0/485865986 <== osd.3 v2:172.21.15.72:6810/1865951395 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fd9f8102d30 con 0x7fd9e00ae1a0 2024-03-22T01:35:24.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:24.074+0000 7fd9fcf3c700 1 -- 172.21.15.72:0/485865986 >> [v2:172.21.15.72:6810/1865951395,v1:172.21.15.72:6811/1865951395] conn(0x7fd9e00ae1a0 msgr2=0x7fd9e00b0640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:24.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:24.074+0000 7fd9fcf3c700 1 --2- 172.21.15.72:0/485865986 >> [v2:172.21.15.72:6810/1865951395,v1:172.21.15.72:6811/1865951395] conn(0x7fd9e00ae1a0 0x7fd9e00b0640 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:24.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:24.074+0000 7fd9fcf3c700 1 -- 172.21.15.72:0/485865986 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fd9e00a3040 msgr2=0x7fd9e00a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:24.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:24.074+0000 7fd9fcf3c700 1 --2- 172.21.15.72:0/485865986 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fd9e00a3040 0x7fd9e00a5500 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fd9f8078b60 tx=0x7fd9ec009260 comp rx=0 tx=0).stop 2024-03-22T01:35:24.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:24.074+0000 7fd9fcf3c700 1 -- 172.21.15.72:0/485865986 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9f81015f0 msgr2=0x7fd9f8198dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:24.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:24.074+0000 7fd9fcf3c700 1 --2- 172.21.15.72:0/485865986 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9f81015f0 0x7fd9f8198dc0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fd9e8000c00 tx=0x7fd9e800bab0 comp rx=0 tx=0).stop 2024-03-22T01:35:24.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:24.075+0000 7fd9f6d9d700 1 -- 172.21.15.72:0/485865986 reap_dead start 2024-03-22T01:35:24.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:24.075+0000 7fd9fcf3c700 1 -- 172.21.15.72:0/485865986 shutdown_connections 2024-03-22T01:35:24.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:24.075+0000 7fd9fcf3c700 1 -- 172.21.15.72:0/485865986 >> 172.21.15.72:0/485865986 conn(0x7fd9f80786b0 msgr2=0x7fd9f80fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:35:24.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:24.075+0000 7fd9fcf3c700 1 -- 172.21.15.72:0/485865986 shutdown_connections 2024-03-22T01:35:24.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:24.075+0000 7fd9fcf3c700 1 -- 172.21.15.72:0/485865986 wait complete. 2024-03-22T01:35:25.248 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:35:26.813 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:26 smithi072 ceph-mon[32414]: pgmap v328: 129 pgs: 129 active+clean; 639 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s; 0 B/s, 0 objects/s recovering 2024-03-22T01:35:26.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:26 smithi089 ceph-mon[35770]: pgmap v328: 129 pgs: 129 active+clean; 639 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s; 0 B/s, 0 objects/s recovering 2024-03-22T01:35:26.902 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:26 smithi098 ceph-mon[36468]: pgmap v328: 129 pgs: 129 active+clean; 639 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s; 0 B/s, 0 objects/s recovering 2024-03-22T01:35:27.442 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:35:27.987 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:35:27.990 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:35:28.027 INFO:teuthology.orchestra.run.smithi072.stdout:111669149763 2024-03-22T01:35:28.027 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-22T01:35:28.799 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:28 smithi072 ceph-mon[32414]: pgmap v329: 129 pgs: 129 active+clean; 639 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 957 B/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-22T01:35:28.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:28 smithi089 ceph-mon[35770]: pgmap v329: 129 pgs: 129 active+clean; 639 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 957 B/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-22T01:35:28.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:28 smithi098 ceph-mon[36468]: pgmap v329: 129 pgs: 129 active+clean; 639 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 957 B/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-22T01:35:30.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:30 smithi089 ceph-mon[35770]: pgmap v330: 129 pgs: 129 active+clean; 639 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-22T01:35:30.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:30 smithi072 ceph-mon[32414]: pgmap v330: 129 pgs: 129 active+clean; 639 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-22T01:35:30.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:30 smithi098 ceph-mon[36468]: pgmap v330: 129 pgs: 129 active+clean; 639 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-22T01:35:32.662 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:32 smithi072 ceph-mon[32414]: pgmap v331: 129 pgs: 129 active+clean; 639 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 0 B/s, 0 objects/s recovering 2024-03-22T01:35:32.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:32 smithi089 ceph-mon[35770]: pgmap v331: 129 pgs: 129 active+clean; 639 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 0 B/s, 0 objects/s recovering 2024-03-22T01:35:32.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:32 smithi098 ceph-mon[36468]: pgmap v331: 129 pgs: 129 active+clean; 639 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 0 B/s, 0 objects/s recovering 2024-03-22T01:35:34.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.202+0000 7f03e3788700 1 -- 172.21.15.72:0/883600347 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f03dc103090 msgr2=0x7f03dc10f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:34.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.202+0000 7f03e3788700 1 --2- 172.21.15.72:0/883600347 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f03dc103090 0x7f03dc10f510 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f03d400b0a0 tx=0x7f03d400b3b0 comp rx=0 tx=0).stop 2024-03-22T01:35:34.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.202+0000 7f03e3788700 1 -- 172.21.15.72:0/883600347 shutdown_connections 2024-03-22T01:35:34.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.202+0000 7f03e3788700 1 --2- 172.21.15.72:0/883600347 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f03dc103090 0x7f03dc10f510 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:34.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.202+0000 7f03e3788700 1 --2- 172.21.15.72:0/883600347 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03dc108640 0x7f03dc108a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:34.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.202+0000 7f03e3788700 1 --2- 172.21.15.72:0/883600347 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03dc1026d0 0x7f03dc102b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:34.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.202+0000 7f03e3788700 1 -- 172.21.15.72:0/883600347 >> 172.21.15.72:0/883600347 conn(0x7f03dc0797e0 msgr2=0x7f03dc079bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:35:34.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.202+0000 7f03e3788700 1 -- 172.21.15.72:0/883600347 shutdown_connections 2024-03-22T01:35:34.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.202+0000 7f03e3788700 1 -- 172.21.15.72:0/883600347 wait complete. 2024-03-22T01:35:34.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.203+0000 7f03e3788700 1 Processor -- start 2024-03-22T01:35:34.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.203+0000 7f03e3788700 1 -- start start 2024-03-22T01:35:34.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.203+0000 7f03e3788700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03dc1026d0 0x7f03dc198e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:34.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.203+0000 7f03e3788700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f03dc103090 0x7f03dc1993a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:34.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.203+0000 7f03e3788700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03dc108640 0x7f03dc19d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:34.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.203+0000 7f03e3788700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f03dc1129d0 con 0x7f03dc103090 2024-03-22T01:35:34.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.203+0000 7f03e3788700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f03dc112850 con 0x7f03dc1026d0 2024-03-22T01:35:34.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.203+0000 7f03e3788700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f03dc112b50 con 0x7f03dc108640 2024-03-22T01:35:34.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.203+0000 7f03e0d23700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f03dc103090 0x7f03dc1993a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:34.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.203+0000 7f03e0d23700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f03dc103090 0x7f03dc1993a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37660/0 (socket says 172.21.15.72:37660) 2024-03-22T01:35:34.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.203+0000 7f03e0d23700 1 -- 172.21.15.72:0/394681556 learned_addr learned my addr 172.21.15.72:0/394681556 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:35:34.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.203+0000 7f03e1524700 1 --2- 172.21.15.72:0/394681556 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03dc1026d0 0x7f03dc198e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:34.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.203+0000 7f03e1d25700 1 --2- 172.21.15.72:0/394681556 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03dc108640 0x7f03dc19d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:34.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.203+0000 7f03e0d23700 1 -- 172.21.15.72:0/394681556 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03dc108640 msgr2=0x7f03dc19d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:34.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.203+0000 7f03e0d23700 1 --2- 172.21.15.72:0/394681556 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03dc108640 0x7f03dc19d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:34.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.203+0000 7f03e0d23700 1 -- 172.21.15.72:0/394681556 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03dc1026d0 msgr2=0x7f03dc198e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:34.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.203+0000 7f03e0d23700 1 --2- 172.21.15.72:0/394681556 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03dc1026d0 0x7f03dc198e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:34.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.203+0000 7f03e0d23700 1 -- 172.21.15.72:0/394681556 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03dc19de70 con 0x7f03dc103090 2024-03-22T01:35:34.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.203+0000 7f03e1524700 1 --2- 172.21.15.72:0/394681556 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03dc1026d0 0x7f03dc198e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:35:34.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.203+0000 7f03e0d23700 1 --2- 172.21.15.72:0/394681556 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f03dc103090 0x7f03dc1993a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f03cc00b530 tx=0x7f03cc00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:34.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.204+0000 7f03d27fc700 1 -- 172.21.15.72:0/394681556 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03cc015020 con 0x7f03dc103090 2024-03-22T01:35:34.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.204+0000 7f03e3788700 1 -- 172.21.15.72:0/394681556 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03dc19e160 con 0x7f03dc103090 2024-03-22T01:35:34.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.204+0000 7f03d27fc700 1 -- 172.21.15.72:0/394681556 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03cc011d10 con 0x7f03dc103090 2024-03-22T01:35:34.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.204+0000 7f03d27fc700 1 -- 172.21.15.72:0/394681556 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03cc00f2e0 con 0x7f03dc103090 2024-03-22T01:35:34.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.204+0000 7f03e3788700 1 -- 172.21.15.72:0/394681556 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03dc1a59e0 con 0x7f03dc103090 2024-03-22T01:35:34.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.205+0000 7f03e3788700 1 -- 172.21.15.72:0/394681556 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03dc069340 con 0x7f03dc103090 2024-03-22T01:35:34.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.206+0000 7f03d27fc700 1 -- 172.21.15.72:0/394681556 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7f03cc018020 con 0x7f03dc103090 2024-03-22T01:35:34.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.206+0000 7f03d27fc700 1 --2- 172.21.15.72:0/394681556 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f03c80a3040 0x7f03c80a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:34.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.207+0000 7f03e1524700 1 --2- 172.21.15.72:0/394681556 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f03c80a3040 0x7f03c80a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:34.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.207+0000 7f03d27fc700 1 -- 172.21.15.72:0/394681556 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f03cc0cf300 con 0x7f03dc103090 2024-03-22T01:35:34.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.207+0000 7f03e1524700 1 --2- 172.21.15.72:0/394681556 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f03c80a3040 0x7f03c80a5500 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f03d8011490 tx=0x7f03d8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:34.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.210+0000 7f03d27fc700 1 -- 172.21.15.72:0/394681556 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f03cc01f020 con 0x7f03dc103090 2024-03-22T01:35:34.414 INFO:teuthology.orchestra.run.smithi072.stdout:201863462966 2024-03-22T01:35:34.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.414+0000 7f03e3788700 1 -- 172.21.15.72:0/394681556 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f03dc19a1c0 con 0x7f03dc103090 2024-03-22T01:35:34.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.414+0000 7f03d27fc700 1 -- 172.21.15.72:0/394681556 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f03cc09bd80 con 0x7f03dc103090 2024-03-22T01:35:34.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.416+0000 7f03e3788700 1 -- 172.21.15.72:0/394681556 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f03c80a3040 msgr2=0x7f03c80a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:34.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.417+0000 7f03e3788700 1 --2- 172.21.15.72:0/394681556 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f03c80a3040 0x7f03c80a5500 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f03d8011490 tx=0x7f03d8009260 comp rx=0 tx=0).stop 2024-03-22T01:35:34.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.417+0000 7f03e3788700 1 -- 172.21.15.72:0/394681556 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f03dc103090 msgr2=0x7f03dc1993a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:34.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.417+0000 7f03e3788700 1 --2- 172.21.15.72:0/394681556 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f03dc103090 0x7f03dc1993a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f03cc00b530 tx=0x7f03cc00b840 comp rx=0 tx=0).stop 2024-03-22T01:35:34.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.417+0000 7f03e3788700 1 -- 172.21.15.72:0/394681556 shutdown_connections 2024-03-22T01:35:34.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.417+0000 7f03e3788700 1 --2- 172.21.15.72:0/394681556 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f03c80a3040 0x7f03c80a5500 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:34.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.417+0000 7f03e3788700 1 --2- 172.21.15.72:0/394681556 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03dc108640 0x7f03dc19d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:34.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.417+0000 7f03e3788700 1 --2- 172.21.15.72:0/394681556 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03dc1026d0 0x7f03dc198e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:34.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.417+0000 7f03e3788700 1 --2- 172.21.15.72:0/394681556 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f03dc103090 0x7f03dc1993a0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:34.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.417+0000 7f03e3788700 1 -- 172.21.15.72:0/394681556 >> 172.21.15.72:0/394681556 conn(0x7f03dc0797e0 msgr2=0x7f03dc100780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:35:34.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.417+0000 7f03e3788700 1 -- 172.21.15.72:0/394681556 shutdown_connections 2024-03-22T01:35:34.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.417+0000 7f03e3788700 1 -- 172.21.15.72:0/394681556 wait complete. 2024-03-22T01:35:34.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:34 smithi089 ceph-mon[35770]: pgmap v332: 129 pgs: 129 active+clean; 639 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-22T01:35:34.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:34 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/394681556' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-22T01:35:34.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:34 smithi072 ceph-mon[32414]: pgmap v332: 129 pgs: 129 active+clean; 639 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-22T01:35:34.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:34 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/394681556' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-22T01:35:34.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:34 smithi098 ceph-mon[36468]: pgmap v332: 129 pgs: 129 active+clean; 639 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-22T01:35:34.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:34 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/394681556' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-22T01:35:35.588 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462962 got 201863462966 for osd.6 2024-03-22T01:35:35.588 DEBUG:teuthology.parallel:result is None 2024-03-22T01:35:35.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:35 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/3330427596' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-22T01:35:35.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:35 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/3330427596' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-22T01:35:35.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:35 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/3330427596' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-22T01:35:36.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.981+0000 7f37020e2700 1 -- 172.21.15.72:0/655156204 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f36fc107640 msgr2=0x7f36fc06abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:36.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.981+0000 7f37020e2700 1 --2- 172.21.15.72:0/655156204 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f36fc107640 0x7f36fc06abc0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f36f400b0a0 tx=0x7f36f400b3b0 comp rx=0 tx=0).stop 2024-03-22T01:35:36.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.981+0000 7f37020e2700 1 -- 172.21.15.72:0/655156204 shutdown_connections 2024-03-22T01:35:36.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.981+0000 7f37020e2700 1 --2- 172.21.15.72:0/655156204 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36fc1050b0 0x7f36fc105530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:36.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.981+0000 7f37020e2700 1 --2- 172.21.15.72:0/655156204 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f36fc107640 0x7f36fc06abc0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:36.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.981+0000 7f37020e2700 1 --2- 172.21.15.72:0/655156204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36fc06b100 0x7f36fc06b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:36.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.981+0000 7f37020e2700 1 -- 172.21.15.72:0/655156204 >> 172.21.15.72:0/655156204 conn(0x7f36fc0775d0 msgr2=0x7f36fc0779e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:35:36.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.981+0000 7f37020e2700 1 -- 172.21.15.72:0/655156204 shutdown_connections 2024-03-22T01:35:36.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.981+0000 7f37020e2700 1 -- 172.21.15.72:0/655156204 wait complete. 2024-03-22T01:35:36.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.982+0000 7f37020e2700 1 Processor -- start 2024-03-22T01:35:36.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.982+0000 7f37020e2700 1 -- start start 2024-03-22T01:35:36.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.982+0000 7f37020e2700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f36fc06b100 0x7f36fc19f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:36.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.982+0000 7f37020e2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36fc1050b0 0x7f36fc19f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:36.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.982+0000 7f37020e2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36fc107640 0x7f36fc1a3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:36.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.982+0000 7f37020e2700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f36fc118f30 con 0x7f36fc06b100 2024-03-22T01:35:36.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.982+0000 7f37020e2700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f36fc118db0 con 0x7f36fc1050b0 2024-03-22T01:35:36.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.982+0000 7f37020e2700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f36fc1190b0 con 0x7f36fc107640 2024-03-22T01:35:36.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.983+0000 7f36fb7fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f36fc06b100 0x7f36fc19f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:36.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.983+0000 7f36faffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36fc1050b0 0x7f36fc19f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:36.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.983+0000 7f36fb7fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f36fc06b100 0x7f36fc19f400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37690/0 (socket says 172.21.15.72:37690) 2024-03-22T01:35:36.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.983+0000 7f36fb7fe700 1 -- 172.21.15.72:0/3330427596 learned_addr learned my addr 172.21.15.72:0/3330427596 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:35:36.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.983+0000 7f36fbfff700 1 --2- 172.21.15.72:0/3330427596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36fc107640 0x7f36fc1a3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:36.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.983+0000 7f36fb7fe700 1 -- 172.21.15.72:0/3330427596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36fc107640 msgr2=0x7f36fc1a3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:36.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.983+0000 7f36fb7fe700 1 --2- 172.21.15.72:0/3330427596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36fc107640 0x7f36fc1a3c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:36.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.983+0000 7f36fb7fe700 1 -- 172.21.15.72:0/3330427596 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36fc1050b0 msgr2=0x7f36fc19f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:36.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.983+0000 7f36fb7fe700 1 --2- 172.21.15.72:0/3330427596 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36fc1050b0 0x7f36fc19f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:36.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.983+0000 7f36fb7fe700 1 -- 172.21.15.72:0/3330427596 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36fc1a43c0 con 0x7f36fc06b100 2024-03-22T01:35:36.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.983+0000 7f36faffd700 1 --2- 172.21.15.72:0/3330427596 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36fc1050b0 0x7f36fc19f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:35:36.120 INFO:teuthology.orchestra.run.smithi072.stdout:167503724603 2024-03-22T01:35:36.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.983+0000 7f36fb7fe700 1 --2- 172.21.15.72:0/3330427596 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f36fc06b100 0x7f36fc19f400 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f36f400b070 tx=0x7f36f4004cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:36.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.983+0000 7f36f8ff9700 1 -- 172.21.15.72:0/3330427596 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36f4021070 con 0x7f36fc06b100 2024-03-22T01:35:36.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.983+0000 7f36f8ff9700 1 -- 172.21.15.72:0/3330427596 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36f4003c00 con 0x7f36fc06b100 2024-03-22T01:35:36.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.983+0000 7f37020e2700 1 -- 172.21.15.72:0/3330427596 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36fc1a4650 con 0x7f36fc06b100 2024-03-22T01:35:36.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.983+0000 7f36f8ff9700 1 -- 172.21.15.72:0/3330427596 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36f401d450 con 0x7f36fc06b100 2024-03-22T01:35:36.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.983+0000 7f37020e2700 1 -- 172.21.15.72:0/3330427596 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36fc1abf30 con 0x7f36fc06b100 2024-03-22T01:35:36.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.984+0000 7f37020e2700 1 -- 172.21.15.72:0/3330427596 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36fc069340 con 0x7f36fc06b100 2024-03-22T01:35:36.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.985+0000 7f36f8ff9700 1 -- 172.21.15.72:0/3330427596 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7f36f4007980 con 0x7f36fc06b100 2024-03-22T01:35:36.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.986+0000 7f36f8ff9700 1 --2- 172.21.15.72:0/3330427596 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f36e40a3040 0x7f36e40a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:36.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.986+0000 7f36faffd700 1 --2- 172.21.15.72:0/3330427596 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f36e40a3040 0x7f36e40a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:36.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.986+0000 7f36f8ff9700 1 -- 172.21.15.72:0/3330427596 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f36f400f020 con 0x7f36fc06b100 2024-03-22T01:35:36.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.987+0000 7f36faffd700 1 --2- 172.21.15.72:0/3330427596 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f36e40a3040 0x7f36e40a5500 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f36fc101690 tx=0x7f36f000b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:36.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:34.990+0000 7f36f8ff9700 1 -- 172.21.15.72:0/3330427596 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f36f402a020 con 0x7f36fc06b100 2024-03-22T01:35:36.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:35.194+0000 7f37020e2700 1 -- 172.21.15.72:0/3330427596 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f36fc1ac100 con 0x7f36fc06b100 2024-03-22T01:35:36.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:35.194+0000 7f36f8ff9700 1 -- 172.21.15.72:0/3330427596 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f36f409dc00 con 0x7f36fc06b100 2024-03-22T01:35:36.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:35.196+0000 7f37020e2700 1 -- 172.21.15.72:0/3330427596 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f36e40a3040 msgr2=0x7f36e40a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:36.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:35.196+0000 7f37020e2700 1 --2- 172.21.15.72:0/3330427596 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f36e40a3040 0x7f36e40a5500 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f36fc101690 tx=0x7f36f000b3c0 comp rx=0 tx=0).stop 2024-03-22T01:35:36.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:35.196+0000 7f37020e2700 1 -- 172.21.15.72:0/3330427596 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f36fc06b100 msgr2=0x7f36fc19f400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:36.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:35.196+0000 7f37020e2700 1 --2- 172.21.15.72:0/3330427596 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f36fc06b100 0x7f36fc19f400 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f36f400b070 tx=0x7f36f4004cd0 comp rx=0 tx=0).stop 2024-03-22T01:35:36.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:35.196+0000 7f37020e2700 1 -- 172.21.15.72:0/3330427596 shutdown_connections 2024-03-22T01:35:36.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:35.196+0000 7f37020e2700 1 --2- 172.21.15.72:0/3330427596 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f36e40a3040 0x7f36e40a5500 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:36.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:35.196+0000 7f37020e2700 1 --2- 172.21.15.72:0/3330427596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36fc107640 0x7f36fc1a3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:36.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:35.196+0000 7f37020e2700 1 --2- 172.21.15.72:0/3330427596 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f36fc06b100 0x7f36fc19f400 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:36.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:35.196+0000 7f37020e2700 1 --2- 172.21.15.72:0/3330427596 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36fc1050b0 0x7f36fc19f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:36.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:35.196+0000 7f37020e2700 1 -- 172.21.15.72:0/3330427596 >> 172.21.15.72:0/3330427596 conn(0x7f36fc0775d0 msgr2=0x7f36fc1005a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:35:36.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:35.196+0000 7f37020e2700 1 -- 172.21.15.72:0/3330427596 shutdown_connections 2024-03-22T01:35:36.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:35.196+0000 7f37020e2700 1 -- 172.21.15.72:0/3330427596 wait complete. 2024-03-22T01:35:36.771 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:36 smithi072 ceph-mon[32414]: pgmap v333: 129 pgs: 129 active+clean; 639 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-22T01:35:36.799 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724598 got 167503724603 for osd.5 2024-03-22T01:35:36.799 DEBUG:teuthology.parallel:result is None 2024-03-22T01:35:36.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:36 smithi089 ceph-mon[35770]: pgmap v333: 129 pgs: 129 active+clean; 639 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-22T01:35:36.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:36 smithi098 ceph-mon[36468]: pgmap v333: 129 pgs: 129 active+clean; 639 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-22T01:35:38.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.138+0000 7fe9d4432700 1 -- 172.21.15.72:0/1972911714 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe9cc1021c0 msgr2=0x7fe9cc1061e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:38.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.138+0000 7fe9d4432700 1 --2- 172.21.15.72:0/1972911714 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe9cc1021c0 0x7fe9cc1061e0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc0099c0 tx=0x7fe9bc009cd0 comp rx=0 tx=0).stop 2024-03-22T01:35:38.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.138+0000 7fe9d4432700 1 -- 172.21.15.72:0/1972911714 shutdown_connections 2024-03-22T01:35:38.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.138+0000 7fe9d4432700 1 --2- 172.21.15.72:0/1972911714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9cc106910 0x7fe9cc10e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:38.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.138+0000 7fe9d4432700 1 --2- 172.21.15.72:0/1972911714 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9cc101810 0x7fe9cc101bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:38.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.138+0000 7fe9d4432700 1 --2- 172.21.15.72:0/1972911714 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe9cc1021c0 0x7fe9cc1061e0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:38.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.138+0000 7fe9d4432700 1 -- 172.21.15.72:0/1972911714 >> 172.21.15.72:0/1972911714 conn(0x7fe9cc0775a0 msgr2=0x7fe9cc0779b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:35:38.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.138+0000 7fe9d4432700 1 -- 172.21.15.72:0/1972911714 shutdown_connections 2024-03-22T01:35:38.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.138+0000 7fe9d4432700 1 -- 172.21.15.72:0/1972911714 wait complete. 2024-03-22T01:35:38.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.138+0000 7fe9d4432700 1 Processor -- start 2024-03-22T01:35:38.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.139+0000 7fe9d4432700 1 -- start start 2024-03-22T01:35:38.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.139+0000 7fe9d4432700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9cc101810 0x7fe9cc199c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:38.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.139+0000 7fe9d4432700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9cc1021c0 0x7fe9cc19a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:38.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.139+0000 7fe9d4432700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe9cc106910 0x7fe9cc1a4c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:38.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.139+0000 7fe9d4432700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fe9cc1118a0 con 0x7fe9cc106910 2024-03-22T01:35:38.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.139+0000 7fe9d4432700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe9cc111720 con 0x7fe9cc1021c0 2024-03-22T01:35:38.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.139+0000 7fe9d4432700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe9cc111a20 con 0x7fe9cc101810 2024-03-22T01:35:38.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.139+0000 7fe9d29cf700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe9cc106910 0x7fe9cc1a4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:38.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.139+0000 7fe9d29cf700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe9cc106910 0x7fe9cc1a4c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37706/0 (socket says 172.21.15.72:37706) 2024-03-22T01:35:38.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.139+0000 7fe9d29cf700 1 -- 172.21.15.72:0/4263282073 learned_addr learned my addr 172.21.15.72:0/4263282073 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:35:38.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.139+0000 7fe9d19cd700 1 --2- 172.21.15.72:0/4263282073 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9cc1021c0 0x7fe9cc19a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:38.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.139+0000 7fe9d21ce700 1 --2- 172.21.15.72:0/4263282073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9cc101810 0x7fe9cc199c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:38.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.139+0000 7fe9d29cf700 1 -- 172.21.15.72:0/4263282073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9cc101810 msgr2=0x7fe9cc199c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:38.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.139+0000 7fe9d29cf700 1 --2- 172.21.15.72:0/4263282073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9cc101810 0x7fe9cc199c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:38.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.139+0000 7fe9d29cf700 1 -- 172.21.15.72:0/4263282073 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9cc1021c0 msgr2=0x7fe9cc19a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:38.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.139+0000 7fe9d29cf700 1 --2- 172.21.15.72:0/4263282073 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9cc1021c0 0x7fe9cc19a140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:38.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.139+0000 7fe9d29cf700 1 -- 172.21.15.72:0/4263282073 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9cc1a51c0 con 0x7fe9cc106910 2024-03-22T01:35:38.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.139+0000 7fe9d29cf700 1 --2- 172.21.15.72:0/4263282073 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe9cc106910 0x7fe9cc1a4c80 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe9c400b970 tx=0x7fe9c400bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:38.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.139+0000 7fe9c37fe700 1 -- 172.21.15.72:0/4263282073 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9c4019070 con 0x7fe9cc106910 2024-03-22T01:35:38.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.139+0000 7fe9c37fe700 1 -- 172.21.15.72:0/4263282073 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9c400ce10 con 0x7fe9cc106910 2024-03-22T01:35:38.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.139+0000 7fe9d4432700 1 -- 172.21.15.72:0/4263282073 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9cc1a53f0 con 0x7fe9cc106910 2024-03-22T01:35:38.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.139+0000 7fe9c37fe700 1 -- 172.21.15.72:0/4263282073 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9c4007370 con 0x7fe9cc106910 2024-03-22T01:35:38.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.140+0000 7fe9d4432700 1 -- 172.21.15.72:0/4263282073 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9cc1a5900 con 0x7fe9cc106910 2024-03-22T01:35:38.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.140+0000 7fe9d4432700 1 -- 172.21.15.72:0/4263282073 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9cc102ca0 con 0x7fe9cc106910 2024-03-22T01:35:38.678 INFO:teuthology.orchestra.run.smithi072.stdout:223338299442 2024-03-22T01:35:38.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.142+0000 7fe9c37fe700 1 -- 172.21.15.72:0/4263282073 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7fe9c401c070 con 0x7fe9cc106910 2024-03-22T01:35:38.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.142+0000 7fe9c37fe700 1 --2- 172.21.15.72:0/4263282073 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe9b80a3040 0x7fe9b80a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:38.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.142+0000 7fe9d21ce700 1 --2- 172.21.15.72:0/4263282073 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe9b80a3040 0x7fe9b80a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:38.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.142+0000 7fe9c37fe700 1 -- 172.21.15.72:0/4263282073 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fe9c40cefe0 con 0x7fe9cc106910 2024-03-22T01:35:38.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.143+0000 7fe9d21ce700 1 --2- 172.21.15.72:0/4263282073 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe9b80a3040 0x7fe9b80a5500 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe9c8009b30 tx=0x7fe9c8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:38.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.146+0000 7fe9c37fe700 1 -- 172.21.15.72:0/4263282073 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe9c409ba60 con 0x7fe9cc106910 2024-03-22T01:35:38.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.358+0000 7fe9d4432700 1 -- 172.21.15.72:0/4263282073 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fe9cc02dd70 con 0x7fe9cc106910 2024-03-22T01:35:38.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.359+0000 7fe9c37fe700 1 -- 172.21.15.72:0/4263282073 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fe9c409b360 con 0x7fe9cc106910 2024-03-22T01:35:38.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.362+0000 7fe9d4432700 1 -- 172.21.15.72:0/4263282073 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe9b80a3040 msgr2=0x7fe9b80a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:38.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.362+0000 7fe9d4432700 1 --2- 172.21.15.72:0/4263282073 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe9b80a3040 0x7fe9b80a5500 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe9c8009b30 tx=0x7fe9c8009260 comp rx=0 tx=0).stop 2024-03-22T01:35:38.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.362+0000 7fe9d4432700 1 -- 172.21.15.72:0/4263282073 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe9cc106910 msgr2=0x7fe9cc1a4c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:38.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.362+0000 7fe9d4432700 1 --2- 172.21.15.72:0/4263282073 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe9cc106910 0x7fe9cc1a4c80 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe9c400b970 tx=0x7fe9c400bc80 comp rx=0 tx=0).stop 2024-03-22T01:35:38.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.362+0000 7fe9d4432700 1 -- 172.21.15.72:0/4263282073 shutdown_connections 2024-03-22T01:35:38.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.362+0000 7fe9d4432700 1 --2- 172.21.15.72:0/4263282073 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe9b80a3040 0x7fe9b80a5500 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:38.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.362+0000 7fe9d4432700 1 --2- 172.21.15.72:0/4263282073 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe9cc106910 0x7fe9cc1a4c80 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:38.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.362+0000 7fe9d4432700 1 --2- 172.21.15.72:0/4263282073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9cc101810 0x7fe9cc199c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:38.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.362+0000 7fe9d4432700 1 --2- 172.21.15.72:0/4263282073 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9cc1021c0 0x7fe9cc19a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:38.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.362+0000 7fe9d4432700 1 -- 172.21.15.72:0/4263282073 >> 172.21.15.72:0/4263282073 conn(0x7fe9cc0775a0 msgr2=0x7fe9cc100d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:35:38.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.362+0000 7fe9d4432700 1 -- 172.21.15.72:0/4263282073 shutdown_connections 2024-03-22T01:35:38.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.362+0000 7fe9d4432700 1 -- 172.21.15.72:0/4263282073 wait complete. 2024-03-22T01:35:38.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:38 smithi089 ceph-mon[35770]: pgmap v334: 129 pgs: 129 active+clean; 639 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-22T01:35:38.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:38 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/4263282073' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-22T01:35:38.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:38 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/2198471133' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-22T01:35:38.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:38 smithi072 ceph-mon[32414]: pgmap v334: 129 pgs: 129 active+clean; 639 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-22T01:35:38.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:38 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/4263282073' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-22T01:35:38.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:38 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/2198471133' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-22T01:35:38.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:38 smithi098 ceph-mon[36468]: pgmap v334: 129 pgs: 129 active+clean; 639 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-22T01:35:38.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:38 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/4263282073' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-22T01:35:38.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:38 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/2198471133' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-22T01:35:39.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.133+0000 7f42afb79700 1 -- 172.21.15.72:0/4196224378 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f42a8107a40 msgr2=0x7f42a810f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:39.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.133+0000 7f42afb79700 1 --2- 172.21.15.72:0/4196224378 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f42a8107a40 0x7f42a810f5c0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f42a000b0d0 tx=0x7f42a000b3e0 comp rx=0 tx=0).stop 2024-03-22T01:35:39.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.133+0000 7f42afb79700 1 -- 172.21.15.72:0/4196224378 shutdown_connections 2024-03-22T01:35:39.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.133+0000 7f42afb79700 1 --2- 172.21.15.72:0/4196224378 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f42a8107a40 0x7f42a810f5c0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:39.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.133+0000 7f42afb79700 1 --2- 172.21.15.72:0/4196224378 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42a8101220 0x7f42a8101600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:39.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.133+0000 7f42afb79700 1 --2- 172.21.15.72:0/4196224378 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42a8101bd0 0x7f42a8107310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:39.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.133+0000 7f42afb79700 1 -- 172.21.15.72:0/4196224378 >> 172.21.15.72:0/4196224378 conn(0x7f42a80768e0 msgr2=0x7f42a8076cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:35:39.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.133+0000 7f42afb79700 1 -- 172.21.15.72:0/4196224378 shutdown_connections 2024-03-22T01:35:39.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.133+0000 7f42afb79700 1 -- 172.21.15.72:0/4196224378 wait complete. 2024-03-22T01:35:39.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.133+0000 7f42afb79700 1 Processor -- start 2024-03-22T01:35:39.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.133+0000 7f42afb79700 1 -- start start 2024-03-22T01:35:39.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.133+0000 7f42afb79700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f42a8101220 0x7f42a8198e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:39.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.133+0000 7f42afb79700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42a8101bd0 0x7f42a8199380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:39.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.133+0000 7f42afb79700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42a8107a40 0x7f42a819d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:39.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.133+0000 7f42afb79700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f42a81129f0 con 0x7f42a8101220 2024-03-22T01:35:39.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.133+0000 7f42afb79700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f42a8112870 con 0x7f42a8101bd0 2024-03-22T01:35:39.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.133+0000 7f42afb79700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f42a8112b70 con 0x7f42a8107a40 2024-03-22T01:35:39.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.134+0000 7f42ad915700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f42a8101220 0x7f42a8198e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:39.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.134+0000 7f42ad915700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f42a8101220 0x7f42a8198e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37742/0 (socket says 172.21.15.72:37742) 2024-03-22T01:35:39.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.134+0000 7f42ad915700 1 -- 172.21.15.72:0/1878032291 learned_addr learned my addr 172.21.15.72:0/1878032291 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:35:39.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.134+0000 7f42ad114700 1 --2- 172.21.15.72:0/1878032291 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42a8101bd0 0x7f42a8199380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:39.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.134+0000 7f42ae116700 1 --2- 172.21.15.72:0/1878032291 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42a8107a40 0x7f42a819d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:39.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.134+0000 7f42ad915700 1 -- 172.21.15.72:0/1878032291 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42a8107a40 msgr2=0x7f42a819d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:39.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.134+0000 7f42ad915700 1 --2- 172.21.15.72:0/1878032291 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42a8107a40 0x7f42a819d6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:39.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.134+0000 7f42ad915700 1 -- 172.21.15.72:0/1878032291 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42a8101bd0 msgr2=0x7f42a8199380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:39.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.134+0000 7f42ad915700 1 --2- 172.21.15.72:0/1878032291 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42a8101bd0 0x7f42a8199380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:39.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.134+0000 7f42ad915700 1 -- 172.21.15.72:0/1878032291 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42a819de00 con 0x7f42a8101220 2024-03-22T01:35:39.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.134+0000 7f42ad915700 1 --2- 172.21.15.72:0/1878032291 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f42a8101220 0x7f42a8198e40 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f42a400e9a0 tx=0x7f42a400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:39.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.134+0000 7f429effd700 1 -- 172.21.15.72:0/1878032291 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42a400cbd0 con 0x7f42a8101220 2024-03-22T01:35:39.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.134+0000 7f429effd700 1 -- 172.21.15.72:0/1878032291 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f42a400cd70 con 0x7f42a8101220 2024-03-22T01:35:39.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.134+0000 7f42afb79700 1 -- 172.21.15.72:0/1878032291 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42a819e0f0 con 0x7f42a8101220 2024-03-22T01:35:39.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.134+0000 7f429effd700 1 -- 172.21.15.72:0/1878032291 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42a4018750 con 0x7f42a8101220 2024-03-22T01:35:39.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.134+0000 7f42afb79700 1 -- 172.21.15.72:0/1878032291 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42a81a5970 con 0x7f42a8101220 2024-03-22T01:35:39.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.135+0000 7f42afb79700 1 -- 172.21.15.72:0/1878032291 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42a810ac80 con 0x7f42a8101220 2024-03-22T01:35:39.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.136+0000 7f429effd700 1 -- 172.21.15.72:0/1878032291 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7f42a4010430 con 0x7f42a8101220 2024-03-22T01:35:39.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.137+0000 7f429effd700 1 --2- 172.21.15.72:0/1878032291 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f42940a3040 0x7f42940a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:39.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.137+0000 7f42ad114700 1 --2- 172.21.15.72:0/1878032291 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f42940a3040 0x7f42940a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:39.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.137+0000 7f429effd700 1 -- 172.21.15.72:0/1878032291 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f42a40d0070 con 0x7f42a8101220 2024-03-22T01:35:39.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.137+0000 7f42ad114700 1 --2- 172.21.15.72:0/1878032291 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f42940a3040 0x7f42940a5500 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f42a8108ee0 tx=0x7f429800a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:39.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.141+0000 7f429effd700 1 -- 172.21.15.72:0/1878032291 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f42a409c9e0 con 0x7f42a8101220 2024-03-22T01:35:39.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.346+0000 7f42afb79700 1 -- 172.21.15.72:0/1878032291 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f42a810a660 con 0x7f42a8101220 2024-03-22T01:35:39.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.347+0000 7f429effd700 1 -- 172.21.15.72:0/1878032291 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f42a409c2e0 con 0x7f42a8101220 2024-03-22T01:35:39.347 INFO:teuthology.orchestra.run.smithi072.stdout:257698037805 2024-03-22T01:35:39.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.349+0000 7f42afb79700 1 -- 172.21.15.72:0/1878032291 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f42940a3040 msgr2=0x7f42940a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:39.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.349+0000 7f42afb79700 1 --2- 172.21.15.72:0/1878032291 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f42940a3040 0x7f42940a5500 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f42a8108ee0 tx=0x7f429800a300 comp rx=0 tx=0).stop 2024-03-22T01:35:39.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.349+0000 7f42afb79700 1 -- 172.21.15.72:0/1878032291 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f42a8101220 msgr2=0x7f42a8198e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:39.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.349+0000 7f42afb79700 1 --2- 172.21.15.72:0/1878032291 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f42a8101220 0x7f42a8198e40 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f42a400e9a0 tx=0x7f42a400ecb0 comp rx=0 tx=0).stop 2024-03-22T01:35:39.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.349+0000 7f42afb79700 1 -- 172.21.15.72:0/1878032291 shutdown_connections 2024-03-22T01:35:39.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.349+0000 7f42afb79700 1 --2- 172.21.15.72:0/1878032291 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f42940a3040 0x7f42940a5500 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:39.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.349+0000 7f42afb79700 1 --2- 172.21.15.72:0/1878032291 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42a8107a40 0x7f42a819d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:39.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.349+0000 7f42afb79700 1 --2- 172.21.15.72:0/1878032291 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f42a8101220 0x7f42a8198e40 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:39.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.349+0000 7f42afb79700 1 --2- 172.21.15.72:0/1878032291 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42a8101bd0 0x7f42a8199380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:39.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.349+0000 7f42afb79700 1 -- 172.21.15.72:0/1878032291 >> 172.21.15.72:0/1878032291 conn(0x7f42a80768e0 msgr2=0x7f42a80ff010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:35:39.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.349+0000 7f42afb79700 1 -- 172.21.15.72:0/1878032291 shutdown_connections 2024-03-22T01:35:39.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.349+0000 7f42afb79700 1 -- 172.21.15.72:0/1878032291 wait complete. 2024-03-22T01:35:39.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.248+0000 7f950cf3f700 1 -- 172.21.15.72:0/1054613704 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9508108630 msgr2=0x7f9508108a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:39.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.248+0000 7f950cf3f700 1 --2- 172.21.15.72:0/1054613704 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9508108630 0x7f9508108a10 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f94fc0099c0 tx=0x7f94fc009cd0 comp rx=0 tx=0).stop 2024-03-22T01:35:39.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.248+0000 7f950cf3f700 1 -- 172.21.15.72:0/1054613704 shutdown_connections 2024-03-22T01:35:39.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.248+0000 7f950cf3f700 1 --2- 172.21.15.72:0/1054613704 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f950806b580 0x7f950810f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:39.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.248+0000 7f950cf3f700 1 --2- 172.21.15.72:0/1054613704 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9508108630 0x7f9508108a10 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:39.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.248+0000 7f950cf3f700 1 --2- 172.21.15.72:0/1054613704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f950806abc0 0x7f950806b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:39.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.248+0000 7f950cf3f700 1 -- 172.21.15.72:0/1054613704 >> 172.21.15.72:0/1054613704 conn(0x7f95080775d0 msgr2=0x7f95080779e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:35:39.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.248+0000 7f950cf3f700 1 -- 172.21.15.72:0/1054613704 shutdown_connections 2024-03-22T01:35:39.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.248+0000 7f950cf3f700 1 -- 172.21.15.72:0/1054613704 wait complete. 2024-03-22T01:35:39.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.249+0000 7f950cf3f700 1 Processor -- start 2024-03-22T01:35:39.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.249+0000 7f950cf3f700 1 -- start start 2024-03-22T01:35:39.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.250+0000 7f950cf3f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f950806abc0 0x7f9508198df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:39.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.250+0000 7f950cf3f700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f950806b580 0x7f9508199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:39.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.250+0000 7f950cf3f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9508108630 0x7f950819d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:39.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.250+0000 7f950cf3f700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f9508112940 con 0x7f950806b580 2024-03-22T01:35:39.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.250+0000 7f950cf3f700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f95081127c0 con 0x7f9508108630 2024-03-22T01:35:39.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.250+0000 7f950cf3f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9508112ac0 con 0x7f950806abc0 2024-03-22T01:35:39.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.250+0000 7f9505d9b700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f950806b580 0x7f9508199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:39.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.250+0000 7f9505d9b700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f950806b580 0x7f9508199330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37726/0 (socket says 172.21.15.72:37726) 2024-03-22T01:35:39.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.250+0000 7f9505d9b700 1 -- 172.21.15.72:0/2198471133 learned_addr learned my addr 172.21.15.72:0/2198471133 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:35:39.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.250+0000 7f9506d9d700 1 --2- 172.21.15.72:0/2198471133 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9508108630 0x7f950819d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:39.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.250+0000 7f950659c700 1 --2- 172.21.15.72:0/2198471133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f950806abc0 0x7f9508198df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:39.389 INFO:teuthology.orchestra.run.smithi072.stdout:352187318302 2024-03-22T01:35:39.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.250+0000 7f9505d9b700 1 -- 172.21.15.72:0/2198471133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f950806abc0 msgr2=0x7f9508198df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:39.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.250+0000 7f9505d9b700 1 --2- 172.21.15.72:0/2198471133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f950806abc0 0x7f9508198df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:39.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.250+0000 7f9505d9b700 1 -- 172.21.15.72:0/2198471133 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9508108630 msgr2=0x7f950819d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:39.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.250+0000 7f9505d9b700 1 --2- 172.21.15.72:0/2198471133 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9508108630 0x7f950819d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:39.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.250+0000 7f9505d9b700 1 -- 172.21.15.72:0/2198471133 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f950819de00 con 0x7f950806b580 2024-03-22T01:35:39.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.250+0000 7f9505d9b700 1 --2- 172.21.15.72:0/2198471133 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f950806b580 0x7f9508199330 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f94f800c920 tx=0x7f94f800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:39.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.250+0000 7f94f77fe700 1 -- 172.21.15.72:0/2198471133 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94f800ce60 con 0x7f950806b580 2024-03-22T01:35:39.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.250+0000 7f950cf3f700 1 -- 172.21.15.72:0/2198471133 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f950819e0f0 con 0x7f950806b580 2024-03-22T01:35:39.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.250+0000 7f94f77fe700 1 -- 172.21.15.72:0/2198471133 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94f8004830 con 0x7f950806b580 2024-03-22T01:35:39.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.250+0000 7f950cf3f700 1 -- 172.21.15.72:0/2198471133 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95081a5970 con 0x7f950806b580 2024-03-22T01:35:39.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.250+0000 7f94f77fe700 1 -- 172.21.15.72:0/2198471133 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94f8005580 con 0x7f950806b580 2024-03-22T01:35:39.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.251+0000 7f950cf3f700 1 -- 172.21.15.72:0/2198471133 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9508069340 con 0x7f950806b580 2024-03-22T01:35:39.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.253+0000 7f94f77fe700 1 -- 172.21.15.72:0/2198471133 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7f94f8007b00 con 0x7f950806b580 2024-03-22T01:35:39.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.253+0000 7f94f77fe700 1 --2- 172.21.15.72:0/2198471133 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f94f00a3040 0x7f94f00a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:39.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.253+0000 7f950659c700 1 --2- 172.21.15.72:0/2198471133 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f94f00a3040 0x7f94f00a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:39.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.253+0000 7f94f77fe700 1 -- 172.21.15.72:0/2198471133 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f94f80cf7f0 con 0x7f950806b580 2024-03-22T01:35:39.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.254+0000 7f950659c700 1 --2- 172.21.15.72:0/2198471133 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f94f00a3040 0x7f94f00a5500 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f94fc0059d0 tx=0x7f94fc005940 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:39.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.261+0000 7f94f77fe700 1 -- 172.21.15.72:0/2198471133 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f94f8014030 con 0x7f950806b580 2024-03-22T01:35:39.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.473+0000 7f950cf3f700 1 -- 172.21.15.72:0/2198471133 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f950802dd70 con 0x7f950806b580 2024-03-22T01:35:39.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.473+0000 7f94f77fe700 1 -- 172.21.15.72:0/2198471133 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f94f809c160 con 0x7f950806b580 2024-03-22T01:35:39.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.475+0000 7f950cf3f700 1 -- 172.21.15.72:0/2198471133 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f94f00a3040 msgr2=0x7f94f00a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:39.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.475+0000 7f950cf3f700 1 --2- 172.21.15.72:0/2198471133 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f94f00a3040 0x7f94f00a5500 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f94fc0059d0 tx=0x7f94fc005940 comp rx=0 tx=0).stop 2024-03-22T01:35:39.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.475+0000 7f950cf3f700 1 -- 172.21.15.72:0/2198471133 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f950806b580 msgr2=0x7f9508199330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:39.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.475+0000 7f950cf3f700 1 --2- 172.21.15.72:0/2198471133 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f950806b580 0x7f9508199330 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f94f800c920 tx=0x7f94f800cce0 comp rx=0 tx=0).stop 2024-03-22T01:35:39.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.476+0000 7f950cf3f700 1 -- 172.21.15.72:0/2198471133 shutdown_connections 2024-03-22T01:35:39.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.476+0000 7f950cf3f700 1 --2- 172.21.15.72:0/2198471133 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f94f00a3040 0x7f94f00a5500 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:39.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.476+0000 7f950cf3f700 1 --2- 172.21.15.72:0/2198471133 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9508108630 0x7f950819d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:39.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.476+0000 7f950cf3f700 1 --2- 172.21.15.72:0/2198471133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f950806abc0 0x7f9508198df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:39.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.476+0000 7f950cf3f700 1 --2- 172.21.15.72:0/2198471133 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f950806b580 0x7f9508199330 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:39.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.476+0000 7f950cf3f700 1 -- 172.21.15.72:0/2198471133 >> 172.21.15.72:0/2198471133 conn(0x7f95080775d0 msgr2=0x7f95081002e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:35:39.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.476+0000 7f950cf3f700 1 -- 172.21.15.72:0/2198471133 shutdown_connections 2024-03-22T01:35:39.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:38.476+0000 7f950cf3f700 1 -- 172.21.15.72:0/2198471133 wait complete. 2024-03-22T01:35:39.590 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:39 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/1878032291' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-22T01:35:39.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:39 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/1878032291' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-22T01:35:39.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:39 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/1878032291' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-22T01:35:40.366 INFO:tasks.cephadm.ceph_manager.ceph:need seq 352187318297 got 352187318302 for osd.11 2024-03-22T01:35:40.366 DEBUG:teuthology.parallel:result is None 2024-03-22T01:35:40.697 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:40 smithi072 ceph-mon[32414]: pgmap v335: 129 pgs: 129 active+clean; 639 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-22T01:35:40.697 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:40 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/3050826109' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-22T01:35:40.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:40 smithi089 ceph-mon[35770]: pgmap v335: 129 pgs: 129 active+clean; 639 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-22T01:35:40.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:40 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/3050826109' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-22T01:35:40.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:40.894+0000 7f501bd4e700 1 -- 172.21.15.72:0/3847827904 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5014101810 msgr2=0x7f5014101bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:40.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:40.894+0000 7f501bd4e700 1 --2- 172.21.15.72:0/3847827904 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5014101810 0x7f5014101bf0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f50100099c0 tx=0x7f5010009cd0 comp rx=0 tx=0).stop 2024-03-22T01:35:40.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:40.894+0000 7f501bd4e700 1 -- 172.21.15.72:0/3847827904 shutdown_connections 2024-03-22T01:35:40.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:40.894+0000 7f501bd4e700 1 --2- 172.21.15.72:0/3847827904 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5014106910 0x7f501410e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:40.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:40.894+0000 7f501bd4e700 1 --2- 172.21.15.72:0/3847827904 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5014101810 0x7f5014101bf0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:40.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:40.894+0000 7f501bd4e700 1 --2- 172.21.15.72:0/3847827904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50141021c0 0x7f50141061e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:40.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:40.894+0000 7f501bd4e700 1 -- 172.21.15.72:0/3847827904 >> 172.21.15.72:0/3847827904 conn(0x7f50140775a0 msgr2=0x7f50140779b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:35:40.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:40.894+0000 7f501bd4e700 1 -- 172.21.15.72:0/3847827904 shutdown_connections 2024-03-22T01:35:40.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:40.894+0000 7f501bd4e700 1 -- 172.21.15.72:0/3847827904 wait complete. 2024-03-22T01:35:40.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:40.894+0000 7f501bd4e700 1 Processor -- start 2024-03-22T01:35:40.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:40.895+0000 7f501bd4e700 1 -- start start 2024-03-22T01:35:40.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:40.895+0000 7f501bd4e700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5014101810 0x7f5014199bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:40.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:40.895+0000 7f501bd4e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50141021c0 0x7f501419a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:40.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:40.895+0000 7f501bd4e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5014106910 0x7f50141a4c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:40.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:40.895+0000 7f501bd4e700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f5014111840 con 0x7f5014101810 2024-03-22T01:35:40.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:40.895+0000 7f501bd4e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f50141116c0 con 0x7f5014106910 2024-03-22T01:35:40.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:40.895+0000 7f501bd4e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f50141119c0 con 0x7f50141021c0 2024-03-22T01:35:40.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:40.895+0000 7f5019aea700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5014101810 0x7f5014199bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:40.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:40.895+0000 7f501a2eb700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5014106910 0x7f50141a4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:40.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:40.895+0000 7f5019aea700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5014101810 0x7f5014199bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:49098/0 (socket says 172.21.15.72:49098) 2024-03-22T01:35:40.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:40.895+0000 7f5019aea700 1 -- 172.21.15.72:0/1758530451 learned_addr learned my addr 172.21.15.72:0/1758530451 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:35:40.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:40.895+0000 7f50192e9700 1 --2- 172.21.15.72:0/1758530451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50141021c0 0x7f501419a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:40.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:40.895+0000 7f501a2eb700 1 -- 172.21.15.72:0/1758530451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50141021c0 msgr2=0x7f501419a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:40.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:40.895+0000 7f501a2eb700 1 --2- 172.21.15.72:0/1758530451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50141021c0 0x7f501419a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:40.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:40.895+0000 7f501a2eb700 1 -- 172.21.15.72:0/1758530451 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5014101810 msgr2=0x7f5014199bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:40.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:40.895+0000 7f501a2eb700 1 --2- 172.21.15.72:0/1758530451 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5014101810 0x7f5014199bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:40.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:40.895+0000 7f501a2eb700 1 -- 172.21.15.72:0/1758530451 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50141a5180 con 0x7f5014106910 2024-03-22T01:35:40.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:40.895+0000 7f5019aea700 1 --2- 172.21.15.72:0/1758530451 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5014101810 0x7f5014199bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:35:40.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:40.895+0000 7f501a2eb700 1 --2- 172.21.15.72:0/1758530451 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5014106910 0x7f50141a4c40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f500c00efc0 tx=0x7f500c00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:40.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:40.896+0000 7f500affd700 1 -- 172.21.15.72:0/1758530451 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f500c0092e0 con 0x7f5014106910 2024-03-22T01:35:40.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:40.896+0000 7f500affd700 1 -- 172.21.15.72:0/1758530451 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f500c004bb0 con 0x7f5014106910 2024-03-22T01:35:40.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:40.896+0000 7f501bd4e700 1 -- 172.21.15.72:0/1758530451 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50141a5350 con 0x7f5014106910 2024-03-22T01:35:40.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:40.896+0000 7f500affd700 1 -- 172.21.15.72:0/1758530451 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f500c004280 con 0x7f5014106910 2024-03-22T01:35:40.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:40.896+0000 7f501bd4e700 1 -- 172.21.15.72:0/1758530451 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50141a58b0 con 0x7f5014106910 2024-03-22T01:35:40.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:40.897+0000 7f501bd4e700 1 -- 172.21.15.72:0/1758530451 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5014109b50 con 0x7f5014106910 2024-03-22T01:35:40.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:40.898+0000 7f500affd700 1 -- 172.21.15.72:0/1758530451 <== mon.2 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7f500c01b030 con 0x7f5014106910 2024-03-22T01:35:40.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:40.898+0000 7f500affd700 1 --2- 172.21.15.72:0/1758530451 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f50000a30c0 0x7f50000a5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:40.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:40.898+0000 7f5019aea700 1 --2- 172.21.15.72:0/1758530451 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f50000a30c0 0x7f50000a5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:40.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:40.898+0000 7f500affd700 1 -- 172.21.15.72:0/1758530451 <== mon.2 v2:172.21.15.89:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f500c0cf390 con 0x7f5014106910 2024-03-22T01:35:40.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:40.899+0000 7f5019aea700 1 --2- 172.21.15.72:0/1758530451 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f50000a30c0 0x7f50000a5580 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f5010009990 tx=0x7f5010005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:40.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:40.903+0000 7f500affd700 1 -- 172.21.15.72:0/1758530451 <== mon.2 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f500c022020 con 0x7f5014106910 2024-03-22T01:35:40.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:40 smithi098 ceph-mon[36468]: pgmap v335: 129 pgs: 129 active+clean; 639 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-22T01:35:40.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:40 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/3050826109' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-22T01:35:41.101 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.102+0000 7f501bd4e700 1 -- 172.21.15.72:0/1758530451 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f5014109530 con 0x7f5014106910 2024-03-22T01:35:41.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.102+0000 7f500affd700 1 -- 172.21.15.72:0/1758530451 <== mon.2 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f500c010040 con 0x7f5014106910 2024-03-22T01:35:41.102 INFO:teuthology.orchestra.run.smithi072.stdout:55834574928 2024-03-22T01:35:41.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.105+0000 7f501bd4e700 1 -- 172.21.15.72:0/1758530451 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f50000a30c0 msgr2=0x7f50000a5580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:41.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.105+0000 7f501bd4e700 1 --2- 172.21.15.72:0/1758530451 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f50000a30c0 0x7f50000a5580 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f5010009990 tx=0x7f5010005960 comp rx=0 tx=0).stop 2024-03-22T01:35:41.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.105+0000 7f501bd4e700 1 -- 172.21.15.72:0/1758530451 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5014106910 msgr2=0x7f50141a4c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:41.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.105+0000 7f501bd4e700 1 --2- 172.21.15.72:0/1758530451 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5014106910 0x7f50141a4c40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f500c00efc0 tx=0x7f500c00c3a0 comp rx=0 tx=0).stop 2024-03-22T01:35:41.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.105+0000 7f501bd4e700 1 -- 172.21.15.72:0/1758530451 shutdown_connections 2024-03-22T01:35:41.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.105+0000 7f501bd4e700 1 --2- 172.21.15.72:0/1758530451 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f50000a30c0 0x7f50000a5580 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:41.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.105+0000 7f501bd4e700 1 --2- 172.21.15.72:0/1758530451 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5014106910 0x7f50141a4c40 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:41.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.105+0000 7f501bd4e700 1 --2- 172.21.15.72:0/1758530451 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5014101810 0x7f5014199bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:41.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.105+0000 7f501bd4e700 1 --2- 172.21.15.72:0/1758530451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50141021c0 0x7f501419a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:41.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.105+0000 7f501bd4e700 1 -- 172.21.15.72:0/1758530451 >> 172.21.15.72:0/1758530451 conn(0x7f50140775a0 msgr2=0x7f5014107a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:35:41.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.105+0000 7f501bd4e700 1 -- 172.21.15.72:0/1758530451 shutdown_connections 2024-03-22T01:35:41.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.105+0000 7f501bd4e700 1 -- 172.21.15.72:0/1758530451 wait complete. 2024-03-22T01:35:41.352 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698037799 got 257698037805 for osd.8 2024-03-22T01:35:41.352 DEBUG:teuthology.parallel:result is None 2024-03-22T01:35:41.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.549+0000 7fe95e2b8700 1 -- 172.21.15.72:0/2349212934 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe9581026d0 msgr2=0x7fe958102b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:41.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.549+0000 7fe95e2b8700 1 --2- 172.21.15.72:0/2349212934 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe9581026d0 0x7fe958102b50 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fe948009990 tx=0x7fe948009ca0 comp rx=0 tx=0).stop 2024-03-22T01:35:41.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.549+0000 7fe95e2b8700 1 -- 172.21.15.72:0/2349212934 shutdown_connections 2024-03-22T01:35:41.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.549+0000 7fe95e2b8700 1 --2- 172.21.15.72:0/2349212934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe958103090 0x7fe95810f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:41.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.549+0000 7fe95e2b8700 1 --2- 172.21.15.72:0/2349212934 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9581075a0 0x7fe958107980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:41.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.549+0000 7fe95e2b8700 1 --2- 172.21.15.72:0/2349212934 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe9581026d0 0x7fe958102b50 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:41.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.549+0000 7fe95e2b8700 1 -- 172.21.15.72:0/2349212934 >> 172.21.15.72:0/2349212934 conn(0x7fe9580797b0 msgr2=0x7fe958079bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:35:41.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.549+0000 7fe95e2b8700 1 -- 172.21.15.72:0/2349212934 shutdown_connections 2024-03-22T01:35:41.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.549+0000 7fe95e2b8700 1 -- 172.21.15.72:0/2349212934 wait complete. 2024-03-22T01:35:41.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.550+0000 7fe95e2b8700 1 Processor -- start 2024-03-22T01:35:41.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.550+0000 7fe95e2b8700 1 -- start start 2024-03-22T01:35:41.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.550+0000 7fe95e2b8700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9581026d0 0x7fe958198ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:41.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.550+0000 7fe95e2b8700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe958103090 0x7fe958199420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:41.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.550+0000 7fe95e2b8700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe9581075a0 0x7fe95819d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:41.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.550+0000 7fe95e2b8700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fe958112a50 con 0x7fe9581075a0 2024-03-22T01:35:41.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.550+0000 7fe95e2b8700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe9581128d0 con 0x7fe9581026d0 2024-03-22T01:35:41.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.550+0000 7fe95e2b8700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe958112bd0 con 0x7fe958103090 2024-03-22T01:35:41.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.550+0000 7fe957fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9581026d0 0x7fe958198ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:41.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.550+0000 7fe95c855700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe9581075a0 0x7fe95819d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:41.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.550+0000 7fe957fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9581026d0 0x7fe958198ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.72:35376/0 (socket says 172.21.15.72:35376) 2024-03-22T01:35:41.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.550+0000 7fe957fff700 1 -- 172.21.15.72:0/2933843625 learned_addr learned my addr 172.21.15.72:0/2933843625 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:35:41.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.550+0000 7fe9577fe700 1 --2- 172.21.15.72:0/2933843625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe958103090 0x7fe958199420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:41.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.550+0000 7fe957fff700 1 -- 172.21.15.72:0/2933843625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe958103090 msgr2=0x7fe958199420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:41.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.550+0000 7fe957fff700 1 --2- 172.21.15.72:0/2933843625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe958103090 0x7fe958199420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:41.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.550+0000 7fe957fff700 1 -- 172.21.15.72:0/2933843625 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe9581075a0 msgr2=0x7fe95819d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:41.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.550+0000 7fe957fff700 1 --2- 172.21.15.72:0/2933843625 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe9581075a0 0x7fe95819d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:41.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.551+0000 7fe957fff700 1 -- 172.21.15.72:0/2933843625 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe95819def0 con 0x7fe9581026d0 2024-03-22T01:35:41.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.551+0000 7fe95c855700 1 --2- 172.21.15.72:0/2933843625 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe9581075a0 0x7fe95819d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:35:41.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.551+0000 7fe957fff700 1 --2- 172.21.15.72:0/2933843625 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9581026d0 0x7fe958198ee0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe94c00e9a0 tx=0x7fe94c00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:41.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.551+0000 7fe9557fa700 1 -- 172.21.15.72:0/2933843625 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe94c00ca10 con 0x7fe9581026d0 2024-03-22T01:35:41.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.551+0000 7fe9557fa700 1 -- 172.21.15.72:0/2933843625 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe94c00cbb0 con 0x7fe9581026d0 2024-03-22T01:35:41.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.551+0000 7fe9557fa700 1 -- 172.21.15.72:0/2933843625 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe94c018680 con 0x7fe9581026d0 2024-03-22T01:35:41.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.551+0000 7fe95e2b8700 1 -- 172.21.15.72:0/2933843625 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe95819e1e0 con 0x7fe9581026d0 2024-03-22T01:35:41.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.551+0000 7fe95e2b8700 1 -- 172.21.15.72:0/2933843625 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9581a5a60 con 0x7fe9581026d0 2024-03-22T01:35:41.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.552+0000 7fe95e2b8700 1 -- 172.21.15.72:0/2933843625 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe958069340 con 0x7fe9581026d0 2024-03-22T01:35:41.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.553+0000 7fe9557fa700 1 -- 172.21.15.72:0/2933843625 <== mon.2 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7fe94c010430 con 0x7fe9581026d0 2024-03-22T01:35:41.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.554+0000 7fe9557fa700 1 --2- 172.21.15.72:0/2933843625 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe9400a30c0 0x7fe9400a5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:41.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.554+0000 7fe9577fe700 1 --2- 172.21.15.72:0/2933843625 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe9400a30c0 0x7fe9400a5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:41.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.554+0000 7fe9557fa700 1 -- 172.21.15.72:0/2933843625 <== mon.2 v2:172.21.15.89:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fe94c0cfa90 con 0x7fe9581026d0 2024-03-22T01:35:41.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.554+0000 7fe9577fe700 1 --2- 172.21.15.72:0/2933843625 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe9400a30c0 0x7fe9400a5580 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fe9480098e0 tx=0x7fe948005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:41.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.561+0000 7fe9557fa700 1 -- 172.21.15.72:0/2933843625 <== mon.2 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe94c09c400 con 0x7fe9581026d0 2024-03-22T01:35:41.703 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:41 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/1758530451' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-22T01:35:41.770 INFO:teuthology.orchestra.run.smithi072.stdout:317827579940 2024-03-22T01:35:41.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.771+0000 7fe95e2b8700 1 -- 172.21.15.72:0/2933843625 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7fe95802dd70 con 0x7fe9581026d0 2024-03-22T01:35:41.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.771+0000 7fe9557fa700 1 -- 172.21.15.72:0/2933843625 <== mon.2 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fe94c09bd00 con 0x7fe9581026d0 2024-03-22T01:35:41.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.773+0000 7fe95e2b8700 1 -- 172.21.15.72:0/2933843625 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe9400a30c0 msgr2=0x7fe9400a5580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:41.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.773+0000 7fe95e2b8700 1 --2- 172.21.15.72:0/2933843625 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe9400a30c0 0x7fe9400a5580 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fe9480098e0 tx=0x7fe948005960 comp rx=0 tx=0).stop 2024-03-22T01:35:41.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.773+0000 7fe95e2b8700 1 -- 172.21.15.72:0/2933843625 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9581026d0 msgr2=0x7fe958198ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:41.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.773+0000 7fe95e2b8700 1 --2- 172.21.15.72:0/2933843625 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9581026d0 0x7fe958198ee0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe94c00e9a0 tx=0x7fe94c00ecb0 comp rx=0 tx=0).stop 2024-03-22T01:35:41.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.774+0000 7fe95e2b8700 1 -- 172.21.15.72:0/2933843625 shutdown_connections 2024-03-22T01:35:41.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.774+0000 7fe95e2b8700 1 --2- 172.21.15.72:0/2933843625 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe9400a30c0 0x7fe9400a5580 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:41.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.774+0000 7fe95e2b8700 1 --2- 172.21.15.72:0/2933843625 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe9581075a0 0x7fe95819d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:41.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.774+0000 7fe95e2b8700 1 --2- 172.21.15.72:0/2933843625 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9581026d0 0x7fe958198ee0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:41.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.774+0000 7fe95e2b8700 1 --2- 172.21.15.72:0/2933843625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe958103090 0x7fe958199420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:41.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.774+0000 7fe95e2b8700 1 -- 172.21.15.72:0/2933843625 >> 172.21.15.72:0/2933843625 conn(0x7fe9580797b0 msgr2=0x7fe958100780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:35:41.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.774+0000 7fe95e2b8700 1 -- 172.21.15.72:0/2933843625 shutdown_connections 2024-03-22T01:35:41.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:41.774+0000 7fe95e2b8700 1 -- 172.21.15.72:0/2933843625 wait complete. 2024-03-22T01:35:41.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:41 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/1758530451' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-22T01:35:41.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:41 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/1758530451' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-22T01:35:42.693 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:42 smithi072 ceph-mon[32414]: pgmap v336: 129 pgs: 129 active+clean; 639 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-22T01:35:42.718 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:42 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/2933843625' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-22T01:35:42.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:42 smithi089 ceph-mon[35770]: pgmap v336: 129 pgs: 129 active+clean; 639 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-22T01:35:42.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:42 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/2933843625' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-22T01:35:42.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:42 smithi098 ceph-mon[36468]: pgmap v336: 129 pgs: 129 active+clean; 639 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-22T01:35:42.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:42 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/2933843625' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-22T01:35:43.119 INFO:tasks.cephadm.ceph_manager.ceph:need seq 223338299436 got 223338299442 for osd.7 2024-03-22T01:35:43.120 DEBUG:teuthology.parallel:result is None 2024-03-22T01:35:43.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:43.306+0000 7f6b87458700 1 -- 172.21.15.72:0/3934446046 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b80100870 msgr2=0x7f6b80100cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:43.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:43.306+0000 7f6b87458700 1 --2- 172.21.15.72:0/3934446046 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b80100870 0x7f6b80100cf0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f6b70009990 tx=0x7f6b70009ca0 comp rx=0 tx=0).stop 2024-03-22T01:35:43.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:43.306+0000 7f6b87458700 1 -- 172.21.15.72:0/3934446046 shutdown_connections 2024-03-22T01:35:43.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:43.306+0000 7f6b87458700 1 --2- 172.21.15.72:0/3934446046 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b80101230 0x7f6b8010d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:43.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:43.306+0000 7f6b87458700 1 --2- 172.21.15.72:0/3934446046 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b80106870 0x7f6b80106c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:43.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:43.306+0000 7f6b87458700 1 --2- 172.21.15.72:0/3934446046 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b80100870 0x7f6b80100cf0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:43.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:43.306+0000 7f6b87458700 1 -- 172.21.15.72:0/3934446046 >> 172.21.15.72:0/3934446046 conn(0x7f6b80077750 msgr2=0x7f6b800fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:35:43.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:43.306+0000 7f6b87458700 1 -- 172.21.15.72:0/3934446046 shutdown_connections 2024-03-22T01:35:43.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:43.306+0000 7f6b87458700 1 -- 172.21.15.72:0/3934446046 wait complete. 2024-03-22T01:35:43.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:43.306+0000 7f6b87458700 1 Processor -- start 2024-03-22T01:35:43.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:43.306+0000 7f6b87458700 1 -- start start 2024-03-22T01:35:43.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:43.306+0000 7f6b87458700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b80100870 0x7f6b80198df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:43.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:43.306+0000 7f6b87458700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b80101230 0x7f6b80199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:43.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:43.306+0000 7f6b87458700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b80106870 0x7f6b8019d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:43.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:43.306+0000 7f6b87458700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f6b80111880 con 0x7f6b80100870 2024-03-22T01:35:43.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:43.306+0000 7f6b87458700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f6b80111700 con 0x7f6b80106870 2024-03-22T01:35:43.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:43.306+0000 7f6b87458700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6b80111a00 con 0x7f6b80101230 2024-03-22T01:35:43.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:43.307+0000 7f6b851f4700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b80100870 0x7f6b80198df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:43.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:43.307+0000 7f6b851f4700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b80100870 0x7f6b80198df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:49136/0 (socket says 172.21.15.72:49136) 2024-03-22T01:35:43.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:43.307+0000 7f6b851f4700 1 -- 172.21.15.72:0/481859329 learned_addr learned my addr 172.21.15.72:0/481859329 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:35:43.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:43.307+0000 7f6b859f5700 1 --2- 172.21.15.72:0/481859329 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b80106870 0x7f6b8019d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:43.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:43.307+0000 7f6b849f3700 1 --2- 172.21.15.72:0/481859329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b80101230 0x7f6b80199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:43.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:43.307+0000 7f6b851f4700 1 -- 172.21.15.72:0/481859329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b80101230 msgr2=0x7f6b80199330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:43.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:43.307+0000 7f6b851f4700 1 --2- 172.21.15.72:0/481859329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b80101230 0x7f6b80199330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:43.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:43.307+0000 7f6b851f4700 1 -- 172.21.15.72:0/481859329 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b80106870 msgr2=0x7f6b8019d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:43.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:43.307+0000 7f6b851f4700 1 --2- 172.21.15.72:0/481859329 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b80106870 0x7f6b8019d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:43.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:43.307+0000 7f6b851f4700 1 -- 172.21.15.72:0/481859329 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b8019de00 con 0x7f6b80100870 2024-03-22T01:35:43.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:43.307+0000 7f6b851f4700 1 --2- 172.21.15.72:0/481859329 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b80100870 0x7f6b80198df0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c00e9a0 tx=0x7f6b7c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:43.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:43.307+0000 7f6b767fc700 1 -- 172.21.15.72:0/481859329 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b7c00cbd0 con 0x7f6b80100870 2024-03-22T01:35:43.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:43.307+0000 7f6b767fc700 1 -- 172.21.15.72:0/481859329 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b7c00cd70 con 0x7f6b80100870 2024-03-22T01:35:43.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:43.307+0000 7f6b767fc700 1 -- 172.21.15.72:0/481859329 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b7c018750 con 0x7f6b80100870 2024-03-22T01:35:43.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:43.307+0000 7f6b87458700 1 -- 172.21.15.72:0/481859329 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b8019e0f0 con 0x7f6b80100870 2024-03-22T01:35:43.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:43.307+0000 7f6b87458700 1 -- 172.21.15.72:0/481859329 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b801a5970 con 0x7f6b80100870 2024-03-22T01:35:43.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:43.308+0000 7f6b87458700 1 -- 172.21.15.72:0/481859329 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b80069340 con 0x7f6b80100870 2024-03-22T01:35:43.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:43.309+0000 7f6b767fc700 1 -- 172.21.15.72:0/481859329 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7f6b7c010430 con 0x7f6b80100870 2024-03-22T01:35:43.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:43.310+0000 7f6b767fc700 1 --2- 172.21.15.72:0/481859329 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f6b6c0a30c0 0x7f6b6c0a5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:43.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:43.310+0000 7f6b849f3700 1 --2- 172.21.15.72:0/481859329 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f6b6c0a30c0 0x7f6b6c0a5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:43.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:43.310+0000 7f6b767fc700 1 -- 172.21.15.72:0/481859329 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f6b7c0cfac0 con 0x7f6b80100870 2024-03-22T01:35:43.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:43.311+0000 7f6b849f3700 1 --2- 172.21.15.72:0/481859329 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f6b6c0a30c0 0x7f6b6c0a5580 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6b70006010 tx=0x7f6b7001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:43.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:43.315+0000 7f6b767fc700 1 -- 172.21.15.72:0/481859329 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6b7c09c430 con 0x7f6b80100870 2024-03-22T01:35:43.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:43.518+0000 7f6b87458700 1 -- 172.21.15.72:0/481859329 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f6b801a5be0 con 0x7f6b80100870 2024-03-22T01:35:43.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:43.518+0000 7f6b767fc700 1 -- 172.21.15.72:0/481859329 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f6b7c09bd30 con 0x7f6b80100870 2024-03-22T01:35:43.517 INFO:teuthology.orchestra.run.smithi072.stdout:292057776169 2024-03-22T01:35:43.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:43.520+0000 7f6b87458700 1 -- 172.21.15.72:0/481859329 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f6b6c0a30c0 msgr2=0x7f6b6c0a5580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:43.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:43.520+0000 7f6b87458700 1 --2- 172.21.15.72:0/481859329 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f6b6c0a30c0 0x7f6b6c0a5580 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6b70006010 tx=0x7f6b7001a040 comp rx=0 tx=0).stop 2024-03-22T01:35:43.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:43.520+0000 7f6b87458700 1 -- 172.21.15.72:0/481859329 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b80100870 msgr2=0x7f6b80198df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:43.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:43.520+0000 7f6b87458700 1 --2- 172.21.15.72:0/481859329 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b80100870 0x7f6b80198df0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c00e9a0 tx=0x7f6b7c00ecb0 comp rx=0 tx=0).stop 2024-03-22T01:35:43.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:43.521+0000 7f6b87458700 1 -- 172.21.15.72:0/481859329 shutdown_connections 2024-03-22T01:35:43.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:43.521+0000 7f6b87458700 1 --2- 172.21.15.72:0/481859329 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f6b6c0a30c0 0x7f6b6c0a5580 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:43.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:43.521+0000 7f6b87458700 1 --2- 172.21.15.72:0/481859329 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b80106870 0x7f6b8019d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:43.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:43.521+0000 7f6b87458700 1 --2- 172.21.15.72:0/481859329 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b80100870 0x7f6b80198df0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:43.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:43.521+0000 7f6b87458700 1 --2- 172.21.15.72:0/481859329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b80101230 0x7f6b80199330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:43.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:43.521+0000 7f6b87458700 1 -- 172.21.15.72:0/481859329 >> 172.21.15.72:0/481859329 conn(0x7f6b80077750 msgr2=0x7f6b800fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:35:43.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:43.521+0000 7f6b87458700 1 -- 172.21.15.72:0/481859329 shutdown_connections 2024-03-22T01:35:43.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:43.521+0000 7f6b87458700 1 -- 172.21.15.72:0/481859329 wait complete. 2024-03-22T01:35:43.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:43 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/481859329' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-22T01:35:43.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:43 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/481859329' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-22T01:35:43.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:43 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/481859329' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-22T01:35:44.213 INFO:tasks.cephadm.ceph_manager.ceph:need seq 317827579935 got 317827579940 for osd.10 2024-03-22T01:35:44.213 DEBUG:teuthology.parallel:result is None 2024-03-22T01:35:44.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.590+0000 7f7f9337e700 1 -- 172.21.15.72:0/452354765 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7f8c106570 msgr2=0x7f7f8c106950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:44.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.590+0000 7f7f9337e700 1 --2- 172.21.15.72:0/452354765 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7f8c106570 0x7f7f8c106950 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f7f88009990 tx=0x7f7f88009ca0 comp rx=0 tx=0).stop 2024-03-22T01:35:44.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.590+0000 7f7f9337e700 1 -- 172.21.15.72:0/452354765 shutdown_connections 2024-03-22T01:35:44.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.590+0000 7f7f9337e700 1 --2- 172.21.15.72:0/452354765 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f8c101f20 0x7f7f8c10e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:44.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.590+0000 7f7f9337e700 1 --2- 172.21.15.72:0/452354765 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7f8c106570 0x7f7f8c106950 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:44.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.590+0000 7f7f9337e700 1 --2- 172.21.15.72:0/452354765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f8c101560 0x7f7f8c1019e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:44.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.590+0000 7f7f9337e700 1 -- 172.21.15.72:0/452354765 >> 172.21.15.72:0/452354765 conn(0x7f7f8c078680 msgr2=0x7f7f8c078a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:35:44.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.591+0000 7f7f9337e700 1 -- 172.21.15.72:0/452354765 shutdown_connections 2024-03-22T01:35:44.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.591+0000 7f7f9337e700 1 -- 172.21.15.72:0/452354765 wait complete. 2024-03-22T01:35:44.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.591+0000 7f7f9337e700 1 Processor -- start 2024-03-22T01:35:44.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.591+0000 7f7f9337e700 1 -- start start 2024-03-22T01:35:44.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.591+0000 7f7f9337e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f8c101560 0x7f7f8c198db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:44.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.591+0000 7f7f9337e700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7f8c101f20 0x7f7f8c1992f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:44.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.591+0000 7f7f9337e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f8c106570 0x7f7f8c19d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:44.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.591+0000 7f7f9337e700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f7f8c1118b0 con 0x7f7f8c101f20 2024-03-22T01:35:44.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.591+0000 7f7f9337e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f7f8c111730 con 0x7f7f8c101560 2024-03-22T01:35:44.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.591+0000 7f7f9337e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7f8c111a30 con 0x7f7f8c106570 2024-03-22T01:35:44.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.592+0000 7f7f9111a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f8c101560 0x7f7f8c198db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:44.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.592+0000 7f7f90919700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7f8c101f20 0x7f7f8c1992f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:44.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.592+0000 7f7f9111a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f8c101560 0x7f7f8c198db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.72:35004/0 (socket says 172.21.15.72:35004) 2024-03-22T01:35:44.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.592+0000 7f7f9111a700 1 -- 172.21.15.72:0/3050826109 learned_addr learned my addr 172.21.15.72:0/3050826109 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:35:44.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.592+0000 7f7f9191b700 1 --2- 172.21.15.72:0/3050826109 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f8c106570 0x7f7f8c19d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:44.465 INFO:teuthology.orchestra.run.smithi072.stdout:146028888130 2024-03-22T01:35:44.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.592+0000 7f7f9111a700 1 -- 172.21.15.72:0/3050826109 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f8c106570 msgr2=0x7f7f8c19d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:44.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.592+0000 7f7f9111a700 1 --2- 172.21.15.72:0/3050826109 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f8c106570 0x7f7f8c19d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:44.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.592+0000 7f7f9111a700 1 -- 172.21.15.72:0/3050826109 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7f8c101f20 msgr2=0x7f7f8c1992f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:44.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.592+0000 7f7f9111a700 1 --2- 172.21.15.72:0/3050826109 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7f8c101f20 0x7f7f8c1992f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:44.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.592+0000 7f7f9111a700 1 -- 172.21.15.72:0/3050826109 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f8c19ddc0 con 0x7f7f8c101560 2024-03-22T01:35:44.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.592+0000 7f7f9191b700 1 --2- 172.21.15.72:0/3050826109 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f8c106570 0x7f7f8c19d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:35:44.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.592+0000 7f7f9111a700 1 --2- 172.21.15.72:0/3050826109 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f8c101560 0x7f7f8c198db0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7f88000c00 tx=0x7f7f8800bab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:44.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.592+0000 7f7f827fc700 1 -- 172.21.15.72:0/3050826109 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f88027070 con 0x7f7f8c101560 2024-03-22T01:35:44.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.592+0000 7f7f9337e700 1 -- 172.21.15.72:0/3050826109 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f8c19e050 con 0x7f7f8c101560 2024-03-22T01:35:44.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.592+0000 7f7f827fc700 1 -- 172.21.15.72:0/3050826109 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f88003d70 con 0x7f7f8c101560 2024-03-22T01:35:44.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.592+0000 7f7f827fc700 1 -- 172.21.15.72:0/3050826109 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f8800f6c0 con 0x7f7f8c101560 2024-03-22T01:35:44.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.592+0000 7f7f9337e700 1 -- 172.21.15.72:0/3050826109 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f8c1a5930 con 0x7f7f8c101560 2024-03-22T01:35:44.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.593+0000 7f7f9337e700 1 -- 172.21.15.72:0/3050826109 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f8c102c70 con 0x7f7f8c101560 2024-03-22T01:35:44.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.594+0000 7f7f827fc700 1 -- 172.21.15.72:0/3050826109 <== mon.2 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7f7f8802a070 con 0x7f7f8c101560 2024-03-22T01:35:44.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.595+0000 7f7f827fc700 1 --2- 172.21.15.72:0/3050826109 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f7f780a3040 0x7f7f780a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:44.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.595+0000 7f7f90919700 1 --2- 172.21.15.72:0/3050826109 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f7f780a3040 0x7f7f780a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:44.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.595+0000 7f7f827fc700 1 -- 172.21.15.72:0/3050826109 <== mon.2 v2:172.21.15.89:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f7f880d0920 con 0x7f7f8c101560 2024-03-22T01:35:44.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.595+0000 7f7f90919700 1 --2- 172.21.15.72:0/3050826109 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f7f780a3040 0x7f7f780a5500 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c078c80 tx=0x7f7f7c00b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:44.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.604+0000 7f7f827fc700 1 -- 172.21.15.72:0/3050826109 <== mon.2 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7f8805e740 con 0x7f7f8c101560 2024-03-22T01:35:44.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.805+0000 7f7f9337e700 1 -- 172.21.15.72:0/3050826109 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f7f8c02dd70 con 0x7f7f8c101560 2024-03-22T01:35:44.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.806+0000 7f7f827fc700 1 -- 172.21.15.72:0/3050826109 <== mon.2 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f7f88013070 con 0x7f7f8c101560 2024-03-22T01:35:44.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.808+0000 7f7f9337e700 1 -- 172.21.15.72:0/3050826109 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f7f780a3040 msgr2=0x7f7f780a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:44.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.808+0000 7f7f9337e700 1 --2- 172.21.15.72:0/3050826109 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f7f780a3040 0x7f7f780a5500 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c078c80 tx=0x7f7f7c00b3c0 comp rx=0 tx=0).stop 2024-03-22T01:35:44.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.808+0000 7f7f9337e700 1 -- 172.21.15.72:0/3050826109 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f8c101560 msgr2=0x7f7f8c198db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:44.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.808+0000 7f7f9337e700 1 --2- 172.21.15.72:0/3050826109 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f8c101560 0x7f7f8c198db0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7f88000c00 tx=0x7f7f8800bab0 comp rx=0 tx=0).stop 2024-03-22T01:35:44.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.808+0000 7f7f9337e700 1 -- 172.21.15.72:0/3050826109 shutdown_connections 2024-03-22T01:35:44.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.808+0000 7f7f9337e700 1 --2- 172.21.15.72:0/3050826109 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f7f780a3040 0x7f7f780a5500 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:44.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.808+0000 7f7f9337e700 1 --2- 172.21.15.72:0/3050826109 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f8c106570 0x7f7f8c19d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:44.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.808+0000 7f7f9337e700 1 --2- 172.21.15.72:0/3050826109 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f8c101560 0x7f7f8c198db0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:44.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.808+0000 7f7f9337e700 1 --2- 172.21.15.72:0/3050826109 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7f8c101f20 0x7f7f8c1992f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:44.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.808+0000 7f7f9337e700 1 -- 172.21.15.72:0/3050826109 >> 172.21.15.72:0/3050826109 conn(0x7f7f8c078680 msgr2=0x7f7f8c06afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:35:44.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.808+0000 7f7f9337e700 1 -- 172.21.15.72:0/3050826109 shutdown_connections 2024-03-22T01:35:44.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:39.808+0000 7f7f9337e700 1 -- 172.21.15.72:0/3050826109 wait complete. 2024-03-22T01:35:44.771 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:44 smithi072 ceph-mon[32414]: pgmap v337: 129 pgs: 129 active+clean; 639 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-22T01:35:44.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:44 smithi089 ceph-mon[35770]: pgmap v337: 129 pgs: 129 active+clean; 639 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-22T01:35:44.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:44 smithi098 ceph-mon[36468]: pgmap v337: 129 pgs: 129 active+clean; 639 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-22T01:35:45.106 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888125 got 146028888130 for osd.4 2024-03-22T01:35:45.106 DEBUG:teuthology.parallel:result is None 2024-03-22T01:35:45.321 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:35:45.775 INFO:tasks.cephadm.ceph_manager.ceph:need seq 292057776164 got 292057776169 for osd.9 2024-03-22T01:35:45.775 DEBUG:teuthology.parallel:result is None 2024-03-22T01:35:45.974 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:35:46.402 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574922 got 55834574928 for osd.1 2024-03-22T01:35:46.402 DEBUG:teuthology.parallel:result is None 2024-03-22T01:35:46.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:46.605+0000 7f8e8fc84700 1 -- 172.21.15.72:0/2126494820 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e88101210 msgr2=0x7f8e881015f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:46.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:46.605+0000 7f8e8fc84700 1 --2- 172.21.15.72:0/2126494820 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e88101210 0x7f8e881015f0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f8e840099c0 tx=0x7f8e84009cd0 comp rx=0 tx=0).stop 2024-03-22T01:35:46.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:46.605+0000 7f8e8fc84700 1 -- 172.21.15.72:0/2126494820 shutdown_connections 2024-03-22T01:35:46.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:46.605+0000 7f8e8fc84700 1 --2- 172.21.15.72:0/2126494820 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8e88107a30 0x7f8e8810f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:46.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:46.605+0000 7f8e8fc84700 1 --2- 172.21.15.72:0/2126494820 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e88101210 0x7f8e881015f0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:46.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:46.605+0000 7f8e8fc84700 1 --2- 172.21.15.72:0/2126494820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e88101bc0 0x7f8e88107300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:46.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:46.605+0000 7f8e8fc84700 1 -- 172.21.15.72:0/2126494820 >> 172.21.15.72:0/2126494820 conn(0x7f8e880768b0 msgr2=0x7f8e88076cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:35:46.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:46.605+0000 7f8e8fc84700 1 -- 172.21.15.72:0/2126494820 shutdown_connections 2024-03-22T01:35:46.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:46.605+0000 7f8e8fc84700 1 -- 172.21.15.72:0/2126494820 wait complete. 2024-03-22T01:35:46.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:46.606+0000 7f8e8fc84700 1 Processor -- start 2024-03-22T01:35:46.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:46.606+0000 7f8e8fc84700 1 -- start start 2024-03-22T01:35:46.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:46.606+0000 7f8e8fc84700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e88101210 0x7f8e88198ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:46.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:46.606+0000 7f8e8fc84700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8e88101bc0 0x7f8e881993e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:46.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:46.606+0000 7f8e8fc84700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e88107a30 0x7f8e8819d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:46.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:46.606+0000 7f8e8fc84700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f8e881129f0 con 0x7f8e88107a30 2024-03-22T01:35:46.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:46.606+0000 7f8e8fc84700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8e88112870 con 0x7f8e88101bc0 2024-03-22T01:35:46.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:46.606+0000 7f8e8fc84700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8e88112b70 con 0x7f8e88101210 2024-03-22T01:35:46.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:46.606+0000 7f8e8e221700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e88107a30 0x7f8e8819d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:46.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:46.606+0000 7f8e8e221700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e88107a30 0x7f8e8819d770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:49162/0 (socket says 172.21.15.72:49162) 2024-03-22T01:35:46.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:46.606+0000 7f8e8e221700 1 -- 172.21.15.72:0/2781025002 learned_addr learned my addr 172.21.15.72:0/2781025002 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:35:46.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:46.606+0000 7f8e8d21f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8e88101bc0 0x7f8e881993e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:46.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:46.606+0000 7f8e8da20700 1 --2- 172.21.15.72:0/2781025002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e88101210 0x7f8e88198ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:46.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:46.606+0000 7f8e8e221700 1 -- 172.21.15.72:0/2781025002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e88101210 msgr2=0x7f8e88198ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:46.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:46.606+0000 7f8e8e221700 1 --2- 172.21.15.72:0/2781025002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e88101210 0x7f8e88198ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:46.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:46.606+0000 7f8e8e221700 1 -- 172.21.15.72:0/2781025002 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8e88101bc0 msgr2=0x7f8e881993e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:46.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:46.606+0000 7f8e8e221700 1 --2- 172.21.15.72:0/2781025002 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8e88101bc0 0x7f8e881993e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:46.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:46.606+0000 7f8e8e221700 1 -- 172.21.15.72:0/2781025002 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e8819deb0 con 0x7f8e88107a30 2024-03-22T01:35:46.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:46.606+0000 7f8e8d21f700 1 --2- 172.21.15.72:0/2781025002 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8e88101bc0 0x7f8e881993e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:35:46.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:46.607+0000 7f8e8e221700 1 --2- 172.21.15.72:0/2781025002 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e88107a30 0x7f8e8819d770 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f8e8000ef30 tx=0x7f8e8000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:46.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:46.607+0000 7f8e7effd700 1 -- 172.21.15.72:0/2781025002 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e800092e0 con 0x7f8e88107a30 2024-03-22T01:35:46.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:46.607+0000 7f8e8fc84700 1 -- 172.21.15.72:0/2781025002 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e8819e140 con 0x7f8e88107a30 2024-03-22T01:35:46.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:46.607+0000 7f8e7effd700 1 -- 172.21.15.72:0/2781025002 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e80004bb0 con 0x7f8e88107a30 2024-03-22T01:35:46.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:46.607+0000 7f8e7effd700 1 -- 172.21.15.72:0/2781025002 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e80004280 con 0x7f8e88107a30 2024-03-22T01:35:46.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:46.607+0000 7f8e8fc84700 1 -- 172.21.15.72:0/2781025002 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e881020c0 con 0x7f8e88107a30 2024-03-22T01:35:46.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:46.608+0000 7f8e8fc84700 1 -- 172.21.15.72:0/2781025002 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e88069340 con 0x7f8e88107a30 2024-03-22T01:35:46.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:46.610+0000 7f8e7effd700 1 -- 172.21.15.72:0/2781025002 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7f8e8001b030 con 0x7f8e88107a30 2024-03-22T01:35:46.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:46.610+0000 7f8e7effd700 1 --2- 172.21.15.72:0/2781025002 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f8e740a2fb0 0x7f8e740a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:46.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:46.610+0000 7f8e7effd700 1 -- 172.21.15.72:0/2781025002 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f8e800cfb60 con 0x7f8e88107a30 2024-03-22T01:35:46.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:46.610+0000 7f8e8da20700 1 --2- 172.21.15.72:0/2781025002 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f8e740a2fb0 0x7f8e740a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:46.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:46.611+0000 7f8e8da20700 1 --2- 172.21.15.72:0/2781025002 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f8e740a2fb0 0x7f8e740a5470 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f8e84000c00 tx=0x7f8e84005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:46.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:46.614+0000 7f8e7effd700 1 -- 172.21.15.72:0/2781025002 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8e80022020 con 0x7f8e88107a30 2024-03-22T01:35:46.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:46 smithi072 ceph-mon[32414]: pgmap v338: 129 pgs: 129 active+clean; 639 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-22T01:35:46.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:46.817+0000 7f8e8fc84700 1 -- 172.21.15.72:0/2781025002 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f8e8802dd70 con 0x7f8e88107a30 2024-03-22T01:35:46.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:46.817+0000 7f8e7effd700 1 -- 172.21.15.72:0/2781025002 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f8e80010040 con 0x7f8e88107a30 2024-03-22T01:35:46.816 INFO:teuthology.orchestra.run.smithi072.stdout:34359738454 2024-03-22T01:35:46.818 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:46.819+0000 7f8e8fc84700 1 -- 172.21.15.72:0/2781025002 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f8e740a2fb0 msgr2=0x7f8e740a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:46.818 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:46.819+0000 7f8e8fc84700 1 --2- 172.21.15.72:0/2781025002 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f8e740a2fb0 0x7f8e740a5470 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f8e84000c00 tx=0x7f8e84005fb0 comp rx=0 tx=0).stop 2024-03-22T01:35:46.818 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:46.819+0000 7f8e8fc84700 1 -- 172.21.15.72:0/2781025002 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e88107a30 msgr2=0x7f8e8819d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:46.818 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:46.819+0000 7f8e8fc84700 1 --2- 172.21.15.72:0/2781025002 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e88107a30 0x7f8e8819d770 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f8e8000ef30 tx=0x7f8e8000c3a0 comp rx=0 tx=0).stop 2024-03-22T01:35:46.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:46.819+0000 7f8e8fc84700 1 -- 172.21.15.72:0/2781025002 shutdown_connections 2024-03-22T01:35:46.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:46.819+0000 7f8e8fc84700 1 --2- 172.21.15.72:0/2781025002 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f8e740a2fb0 0x7f8e740a5470 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:46.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:46.819+0000 7f8e8fc84700 1 --2- 172.21.15.72:0/2781025002 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e88107a30 0x7f8e8819d770 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:46.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:46.819+0000 7f8e8fc84700 1 --2- 172.21.15.72:0/2781025002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e88101210 0x7f8e88198ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:46.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:46.819+0000 7f8e8fc84700 1 --2- 172.21.15.72:0/2781025002 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8e88101bc0 0x7f8e881993e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:46.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:46.819+0000 7f8e8fc84700 1 -- 172.21.15.72:0/2781025002 >> 172.21.15.72:0/2781025002 conn(0x7f8e880768b0 msgr2=0x7f8e880ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:35:46.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:46.819+0000 7f8e8fc84700 1 -- 172.21.15.72:0/2781025002 shutdown_connections 2024-03-22T01:35:46.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:46.819+0000 7f8e8fc84700 1 -- 172.21.15.72:0/2781025002 wait complete. 2024-03-22T01:35:46.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:46 smithi089 ceph-mon[35770]: pgmap v338: 129 pgs: 129 active+clean; 639 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-22T01:35:46.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:46 smithi098 ceph-mon[36468]: pgmap v338: 129 pgs: 129 active+clean; 639 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-22T01:35:47.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:47 smithi089 ceph-mon[35770]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-22T01:35:47.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:47 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/2781025002' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-22T01:35:47.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:47 smithi072 ceph-mon[32414]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-22T01:35:47.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:47 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/2781025002' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-22T01:35:47.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:47 smithi098 ceph-mon[36468]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-22T01:35:47.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:47 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/2781025002' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-22T01:35:47.929 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738448 got 34359738454 for osd.0 2024-03-22T01:35:47.929 DEBUG:teuthology.parallel:result is None 2024-03-22T01:35:48.852 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:48 smithi072 ceph-mon[32414]: pgmap v339: 129 pgs: 129 active+clean; 639 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-22T01:35:48.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:48 smithi089 ceph-mon[35770]: pgmap v339: 129 pgs: 129 active+clean; 639 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-22T01:35:48.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:48 smithi098 ceph-mon[36468]: pgmap v339: 129 pgs: 129 active+clean; 639 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-22T01:35:50.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.486+0000 7fb3b471f700 1 -- 172.21.15.72:0/2957474855 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3ac107590 msgr2=0x7fb3ac107970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:50.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.486+0000 7fb3b471f700 1 --2- 172.21.15.72:0/2957474855 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3ac107590 0x7fb3ac107970 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fb3a8009990 tx=0x7fb3a8009ca0 comp rx=0 tx=0).stop 2024-03-22T01:35:50.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.486+0000 7fb3b471f700 1 -- 172.21.15.72:0/2957474855 shutdown_connections 2024-03-22T01:35:50.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.486+0000 7fb3b471f700 1 --2- 172.21.15.72:0/2957474855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3ac101f40 0x7fb3ac10e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:50.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.486+0000 7fb3b471f700 1 --2- 172.21.15.72:0/2957474855 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3ac107590 0x7fb3ac107970 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:50.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.486+0000 7fb3b471f700 1 --2- 172.21.15.72:0/2957474855 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3ac101580 0x7fb3ac101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:50.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.486+0000 7fb3b471f700 1 -- 172.21.15.72:0/2957474855 >> 172.21.15.72:0/2957474855 conn(0x7fb3ac0784b0 msgr2=0x7fb3ac0788c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:35:50.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.486+0000 7fb3b471f700 1 -- 172.21.15.72:0/2957474855 shutdown_connections 2024-03-22T01:35:50.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.486+0000 7fb3b471f700 1 -- 172.21.15.72:0/2957474855 wait complete. 2024-03-22T01:35:50.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.487+0000 7fb3b471f700 1 Processor -- start 2024-03-22T01:35:50.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.487+0000 7fb3b471f700 1 -- start start 2024-03-22T01:35:50.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.487+0000 7fb3b471f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3ac101580 0x7fb3ac19da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:50.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.487+0000 7fb3b471f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3ac101f40 0x7fb3ac19dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:50.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.487+0000 7fb3b471f700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3ac107590 0x7fb3ac197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:50.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.487+0000 7fb3b471f700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fb3ac111890 con 0x7fb3ac107590 2024-03-22T01:35:50.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.487+0000 7fb3b471f700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb3ac111710 con 0x7fb3ac101580 2024-03-22T01:35:50.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.487+0000 7fb3b471f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb3ac111a10 con 0x7fb3ac101f40 2024-03-22T01:35:50.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.487+0000 7fb3b2cbc700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3ac107590 0x7fb3ac197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:50.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.487+0000 7fb3b24bb700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3ac101580 0x7fb3ac19da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:50.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.487+0000 7fb3b24bb700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3ac101580 0x7fb3ac19da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.72:35428/0 (socket says 172.21.15.72:35428) 2024-03-22T01:35:50.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.487+0000 7fb3b24bb700 1 -- 172.21.15.72:0/3069115368 learned_addr learned my addr 172.21.15.72:0/3069115368 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:35:50.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.487+0000 7fb3b1cba700 1 --2- 172.21.15.72:0/3069115368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3ac101f40 0x7fb3ac19dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:50.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.488+0000 7fb3b24bb700 1 -- 172.21.15.72:0/3069115368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3ac101f40 msgr2=0x7fb3ac19dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:50.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.488+0000 7fb3b24bb700 1 --2- 172.21.15.72:0/3069115368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3ac101f40 0x7fb3ac19dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:50.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.488+0000 7fb3b24bb700 1 -- 172.21.15.72:0/3069115368 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3ac107590 msgr2=0x7fb3ac197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:50.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.488+0000 7fb3b24bb700 1 --2- 172.21.15.72:0/3069115368 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3ac107590 0x7fb3ac197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:50.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.488+0000 7fb3b24bb700 1 -- 172.21.15.72:0/3069115368 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3ac198410 con 0x7fb3ac101580 2024-03-22T01:35:50.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.488+0000 7fb3b1cba700 1 --2- 172.21.15.72:0/3069115368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3ac101f40 0x7fb3ac19dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T01:35:50.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.488+0000 7fb3b2cbc700 1 --2- 172.21.15.72:0/3069115368 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3ac107590 0x7fb3ac197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:35:50.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.488+0000 7fb3b24bb700 1 --2- 172.21.15.72:0/3069115368 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3ac101580 0x7fb3ac19da60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb3a8000c00 tx=0x7fb3a800bab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:50.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.488+0000 7fb3a37fe700 1 -- 172.21.15.72:0/3069115368 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a8027070 con 0x7fb3ac101580 2024-03-22T01:35:50.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.488+0000 7fb3b471f700 1 -- 172.21.15.72:0/3069115368 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3ac1986a0 con 0x7fb3ac101580 2024-03-22T01:35:50.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.488+0000 7fb3a37fe700 1 -- 172.21.15.72:0/3069115368 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3a8003d70 con 0x7fb3ac101580 2024-03-22T01:35:50.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.488+0000 7fb3a37fe700 1 -- 172.21.15.72:0/3069115368 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a800f6c0 con 0x7fb3ac101580 2024-03-22T01:35:50.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.488+0000 7fb3b471f700 1 -- 172.21.15.72:0/3069115368 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3ac1a4810 con 0x7fb3ac101580 2024-03-22T01:35:50.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.489+0000 7fb3b471f700 1 -- 172.21.15.72:0/3069115368 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3ac102c70 con 0x7fb3ac101580 2024-03-22T01:35:50.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.490+0000 7fb3a37fe700 1 -- 172.21.15.72:0/3069115368 <== mon.2 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7fb3a802a070 con 0x7fb3ac101580 2024-03-22T01:35:50.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.491+0000 7fb3a37fe700 1 --2- 172.21.15.72:0/3069115368 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fb3980a2fb0 0x7fb3980a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:50.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.491+0000 7fb3b1cba700 1 --2- 172.21.15.72:0/3069115368 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fb3980a2fb0 0x7fb3980a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:50.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.491+0000 7fb3a37fe700 1 -- 172.21.15.72:0/3069115368 <== mon.2 v2:172.21.15.89:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fb3a80d0cd0 con 0x7fb3ac101580 2024-03-22T01:35:50.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.491+0000 7fb3b1cba700 1 --2- 172.21.15.72:0/3069115368 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fb3980a2fb0 0x7fb3980a5470 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb3ac199280 tx=0x7fb39c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:50.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.494+0000 7fb3a37fe700 1 -- 172.21.15.72:0/3069115368 <== mon.2 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb3a8013070 con 0x7fb3ac101580 2024-03-22T01:35:50.649 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:50 smithi072 ceph-mon[32414]: pgmap v340: 129 pgs: 129 active+clean; 639 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 0 B/s wr, 1 op/s 2024-03-22T01:35:50.649 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:50 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:35:50.649 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:50 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:35:50.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.706+0000 7fb3b471f700 1 -- 172.21.15.72:0/3069115368 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fb3ac02dd70 con 0x7fb3ac101580 2024-03-22T01:35:50.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.706+0000 7fb3a37fe700 1 -- 172.21.15.72:0/3069115368 <== mon.2 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb3a809d8f0 con 0x7fb3ac101580 2024-03-22T01:35:50.705 INFO:teuthology.orchestra.run.smithi072.stdout:77309411405 2024-03-22T01:35:50.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.708+0000 7fb3b471f700 1 -- 172.21.15.72:0/3069115368 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fb3980a2fb0 msgr2=0x7fb3980a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:50.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.708+0000 7fb3b471f700 1 --2- 172.21.15.72:0/3069115368 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fb3980a2fb0 0x7fb3980a5470 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb3ac199280 tx=0x7fb39c009260 comp rx=0 tx=0).stop 2024-03-22T01:35:50.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.708+0000 7fb3b471f700 1 -- 172.21.15.72:0/3069115368 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3ac101580 msgr2=0x7fb3ac19da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:50.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.708+0000 7fb3b471f700 1 --2- 172.21.15.72:0/3069115368 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3ac101580 0x7fb3ac19da60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb3a8000c00 tx=0x7fb3a800bab0 comp rx=0 tx=0).stop 2024-03-22T01:35:50.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.709+0000 7fb3b471f700 1 -- 172.21.15.72:0/3069115368 shutdown_connections 2024-03-22T01:35:50.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.709+0000 7fb3b471f700 1 --2- 172.21.15.72:0/3069115368 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fb3980a2fb0 0x7fb3980a5470 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:50.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.709+0000 7fb3b471f700 1 --2- 172.21.15.72:0/3069115368 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3ac107590 0x7fb3ac197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:50.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.709+0000 7fb3b471f700 1 --2- 172.21.15.72:0/3069115368 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3ac101580 0x7fb3ac19da60 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:50.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.709+0000 7fb3b471f700 1 --2- 172.21.15.72:0/3069115368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3ac101f40 0x7fb3ac19dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:50.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.709+0000 7fb3b471f700 1 -- 172.21.15.72:0/3069115368 >> 172.21.15.72:0/3069115368 conn(0x7fb3ac0784b0 msgr2=0x7fb3ac100280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:35:50.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.709+0000 7fb3b471f700 1 -- 172.21.15.72:0/3069115368 shutdown_connections 2024-03-22T01:35:50.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.709+0000 7fb3b471f700 1 -- 172.21.15.72:0/3069115368 wait complete. 2024-03-22T01:35:50.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.772+0000 7f37d3bcb700 1 -- 172.21.15.72:0/1580307510 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37cc101580 msgr2=0x7f37cc101a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:50.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.772+0000 7f37d3bcb700 1 --2- 172.21.15.72:0/1580307510 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37cc101580 0x7f37cc101a00 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f37bc009990 tx=0x7f37bc009ca0 comp rx=0 tx=0).stop 2024-03-22T01:35:50.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.772+0000 7f37d3bcb700 1 -- 172.21.15.72:0/1580307510 shutdown_connections 2024-03-22T01:35:50.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.772+0000 7f37d3bcb700 1 --2- 172.21.15.72:0/1580307510 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f37cc101f40 0x7f37cc10e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:50.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.772+0000 7f37d3bcb700 1 --2- 172.21.15.72:0/1580307510 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37cc107590 0x7f37cc107970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:50.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.772+0000 7f37d3bcb700 1 --2- 172.21.15.72:0/1580307510 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37cc101580 0x7f37cc101a00 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:50.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.772+0000 7f37d3bcb700 1 -- 172.21.15.72:0/1580307510 >> 172.21.15.72:0/1580307510 conn(0x7f37cc0784b0 msgr2=0x7f37cc0788c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:35:50.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.772+0000 7f37d3bcb700 1 -- 172.21.15.72:0/1580307510 shutdown_connections 2024-03-22T01:35:50.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.772+0000 7f37d3bcb700 1 -- 172.21.15.72:0/1580307510 wait complete. 2024-03-22T01:35:50.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.773+0000 7f37d3bcb700 1 Processor -- start 2024-03-22T01:35:50.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.773+0000 7f37d3bcb700 1 -- start start 2024-03-22T01:35:50.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.773+0000 7f37d3bcb700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37cc101580 0x7f37cc198db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:50.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.773+0000 7f37d3bcb700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f37cc101f40 0x7f37cc1992f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:50.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.773+0000 7f37d3bcb700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37cc107590 0x7f37cc19d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:50.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.773+0000 7f37d3bcb700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f37cc111890 con 0x7f37cc107590 2024-03-22T01:35:50.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.773+0000 7f37d3bcb700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f37cc111710 con 0x7f37cc101f40 2024-03-22T01:35:50.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.773+0000 7f37d3bcb700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f37cc111a10 con 0x7f37cc101580 2024-03-22T01:35:50.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.773+0000 7f37d1166700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f37cc101f40 0x7f37cc1992f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:50.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.773+0000 7f37d1166700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f37cc101f40 0x7f37cc1992f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.72:55294/0 (socket says 172.21.15.72:55294) 2024-03-22T01:35:50.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.773+0000 7f37d1166700 1 -- 172.21.15.72:0/4185707954 learned_addr learned my addr 172.21.15.72:0/4185707954 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:35:50.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.773+0000 7f37d1967700 1 --2- 172.21.15.72:0/4185707954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37cc101580 0x7f37cc198db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:50.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.773+0000 7f37d2168700 1 --2- 172.21.15.72:0/4185707954 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37cc107590 0x7f37cc19d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:50.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.773+0000 7f37d1166700 1 -- 172.21.15.72:0/4185707954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37cc101580 msgr2=0x7f37cc198db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:50.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.773+0000 7f37d1166700 1 --2- 172.21.15.72:0/4185707954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37cc101580 0x7f37cc198db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:50.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.773+0000 7f37d1166700 1 -- 172.21.15.72:0/4185707954 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37cc107590 msgr2=0x7f37cc19d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:50.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.773+0000 7f37d1166700 1 --2- 172.21.15.72:0/4185707954 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37cc107590 0x7f37cc19d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:50.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.773+0000 7f37d1166700 1 -- 172.21.15.72:0/4185707954 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37cc19ddc0 con 0x7f37cc101f40 2024-03-22T01:35:50.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.774+0000 7f37d1166700 1 --2- 172.21.15.72:0/4185707954 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f37cc101f40 0x7f37cc1992f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f37bc00bae0 tx=0x7f37bc00bb10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:50.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.774+0000 7f37c2ffd700 1 -- 172.21.15.72:0/4185707954 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37bc026070 con 0x7f37cc101f40 2024-03-22T01:35:50.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.774+0000 7f37d3bcb700 1 -- 172.21.15.72:0/4185707954 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37cc19e050 con 0x7f37cc101f40 2024-03-22T01:35:50.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.774+0000 7f37d3bcb700 1 -- 172.21.15.72:0/4185707954 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37cc1030f0 con 0x7f37cc101f40 2024-03-22T01:35:50.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.774+0000 7f37c2ffd700 1 -- 172.21.15.72:0/4185707954 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37bc003d70 con 0x7f37cc101f40 2024-03-22T01:35:50.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.774+0000 7f37c2ffd700 1 -- 172.21.15.72:0/4185707954 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37bc021410 con 0x7f37cc101f40 2024-03-22T01:35:50.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.775+0000 7f37d3bcb700 1 -- 172.21.15.72:0/4185707954 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37cc069340 con 0x7f37cc101f40 2024-03-22T01:35:50.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.776+0000 7f37c2ffd700 1 -- 172.21.15.72:0/4185707954 <== mon.2 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7f37bc029070 con 0x7f37cc101f40 2024-03-22T01:35:50.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.776+0000 7f37c2ffd700 1 --2- 172.21.15.72:0/4185707954 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f37b80a3030 0x7f37b80a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:50.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.777+0000 7f37d1967700 1 --2- 172.21.15.72:0/4185707954 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f37b80a3030 0x7f37b80a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:50.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.777+0000 7f37c2ffd700 1 -- 172.21.15.72:0/4185707954 <== mon.2 v2:172.21.15.89:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f37bc0d0990 con 0x7f37cc101f40 2024-03-22T01:35:50.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.777+0000 7f37d1967700 1 --2- 172.21.15.72:0/4185707954 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f37b80a3030 0x7f37b80a54f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f37c8009a10 tx=0x7f37c8009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:50.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.784+0000 7f37c2ffd700 1 -- 172.21.15.72:0/4185707954 <== mon.2 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f37bc030080 con 0x7f37cc101f40 2024-03-22T01:35:50.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:50 smithi089 ceph-mon[35770]: pgmap v340: 129 pgs: 129 active+clean; 639 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 0 B/s wr, 1 op/s 2024-03-22T01:35:50.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:50 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:35:50.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:50 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:35:50.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:50 smithi098 ceph-mon[36468]: pgmap v340: 129 pgs: 129 active+clean; 639 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 0 B/s wr, 1 op/s 2024-03-22T01:35:50.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:50 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:35:50.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:50 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:35:50.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.993+0000 7f37d3bcb700 1 -- 172.21.15.72:0/4185707954 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f37cc063290 con 0x7f37cc101f40 2024-03-22T01:35:50.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.994+0000 7f37c2ffd700 1 -- 172.21.15.72:0/4185707954 <== mon.2 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f37bc013070 con 0x7f37cc101f40 2024-03-22T01:35:50.993 INFO:teuthology.orchestra.run.smithi072.stdout:111669149768 2024-03-22T01:35:50.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.996+0000 7f37d3bcb700 1 -- 172.21.15.72:0/4185707954 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f37b80a3030 msgr2=0x7f37b80a54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:50.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.996+0000 7f37d3bcb700 1 --2- 172.21.15.72:0/4185707954 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f37b80a3030 0x7f37b80a54f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f37c8009a10 tx=0x7f37c8009190 comp rx=0 tx=0).stop 2024-03-22T01:35:50.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.996+0000 7f37d3bcb700 1 -- 172.21.15.72:0/4185707954 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f37cc101f40 msgr2=0x7f37cc1992f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:50.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.996+0000 7f37d3bcb700 1 --2- 172.21.15.72:0/4185707954 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f37cc101f40 0x7f37cc1992f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f37bc00bae0 tx=0x7f37bc00bb10 comp rx=0 tx=0).stop 2024-03-22T01:35:50.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.996+0000 7f37d3bcb700 1 -- 172.21.15.72:0/4185707954 shutdown_connections 2024-03-22T01:35:50.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.996+0000 7f37d3bcb700 1 --2- 172.21.15.72:0/4185707954 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f37b80a3030 0x7f37b80a54f0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:50.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.996+0000 7f37d3bcb700 1 --2- 172.21.15.72:0/4185707954 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37cc107590 0x7f37cc19d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:50.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.996+0000 7f37d3bcb700 1 --2- 172.21.15.72:0/4185707954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37cc101580 0x7f37cc198db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:50.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.996+0000 7f37d3bcb700 1 --2- 172.21.15.72:0/4185707954 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f37cc101f40 0x7f37cc1992f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:50.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.996+0000 7f37d3bcb700 1 -- 172.21.15.72:0/4185707954 >> 172.21.15.72:0/4185707954 conn(0x7f37cc0784b0 msgr2=0x7f37cc1008b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:35:50.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.996+0000 7f37d3bcb700 1 -- 172.21.15.72:0/4185707954 shutdown_connections 2024-03-22T01:35:50.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:50.996+0000 7f37d3bcb700 1 -- 172.21.15.72:0/4185707954 wait complete. 2024-03-22T01:35:51.524 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411399 got 77309411405 for osd.2 2024-03-22T01:35:51.524 DEBUG:teuthology.parallel:result is None 2024-03-22T01:35:51.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:51 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/3069115368' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-22T01:35:51.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:51 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:35:51.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:51 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:35:51.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:51 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:35:51.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:51 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/4185707954' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-22T01:35:51.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:51 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/3069115368' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-22T01:35:51.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:51 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:35:51.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:51 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:35:51.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:51 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:35:51.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:51 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/4185707954' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-22T01:35:51.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:51 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/3069115368' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-22T01:35:51.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:51 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:35:51.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:51 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:35:51.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:51 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' 2024-03-22T01:35:51.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:51 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/4185707954' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-22T01:35:52.243 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149763 got 111669149768 for osd.3 2024-03-22T01:35:52.243 DEBUG:teuthology.parallel:result is None 2024-03-22T01:35:52.243 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-22T01:35:52.243 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-22T01:35:52.491 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:35:52.850 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:52 smithi072 ceph-mon[32414]: pgmap v341: 129 pgs: 129 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-22T01:35:52.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:52 smithi089 ceph-mon[35770]: pgmap v341: 129 pgs: 129 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-22T01:35:52.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:52 smithi098 ceph-mon[36468]: pgmap v341: 129 pgs: 129 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-22T01:35:54.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:54.322+0000 7f7f006e5700 1 -- 172.21.15.72:0/2725585386 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7ef8103020 msgr2=0x7f7ef810f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:54.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:54.322+0000 7f7f006e5700 1 --2- 172.21.15.72:0/2725585386 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7ef8103020 0x7f7ef810f530 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f7ef0009990 tx=0x7f7ef0009ca0 comp rx=0 tx=0).stop 2024-03-22T01:35:54.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:54.322+0000 7f7f006e5700 1 -- 172.21.15.72:0/2725585386 shutdown_connections 2024-03-22T01:35:54.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:54.322+0000 7f7f006e5700 1 --2- 172.21.15.72:0/2725585386 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7ef8103020 0x7f7ef810f530 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:54.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:54.322+0000 7f7f006e5700 1 --2- 172.21.15.72:0/2725585386 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ef8108660 0x7f7ef8108a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:54.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:54.322+0000 7f7f006e5700 1 --2- 172.21.15.72:0/2725585386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ef8102660 0x7f7ef8102ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:54.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:54.322+0000 7f7f006e5700 1 -- 172.21.15.72:0/2725585386 >> 172.21.15.72:0/2725585386 conn(0x7f7ef80fe130 msgr2=0x7f7ef8100550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:35:54.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:54.322+0000 7f7f006e5700 1 -- 172.21.15.72:0/2725585386 shutdown_connections 2024-03-22T01:35:54.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:54.322+0000 7f7f006e5700 1 -- 172.21.15.72:0/2725585386 wait complete. 2024-03-22T01:35:54.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:54.323+0000 7f7f006e5700 1 Processor -- start 2024-03-22T01:35:54.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:54.323+0000 7f7f006e5700 1 -- start start 2024-03-22T01:35:54.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:54.323+0000 7f7f006e5700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7ef8102660 0x7f7ef8198d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:54.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:54.323+0000 7f7f006e5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ef8103020 0x7f7ef81992c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:54.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:54.323+0000 7f7f006e5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ef8108660 0x7f7ef819d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:54.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:54.323+0000 7f7f006e5700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef8112990 con 0x7f7ef8102660 2024-03-22T01:35:54.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:54.323+0000 7f7f006e5700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef8112810 con 0x7f7ef8108660 2024-03-22T01:35:54.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:54.323+0000 7f7f006e5700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef8112b10 con 0x7f7ef8103020 2024-03-22T01:35:54.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:54.323+0000 7f7efe481700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7ef8102660 0x7f7ef8198d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:54.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:54.323+0000 7f7efe481700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7ef8102660 0x7f7ef8198d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57576/0 (socket says 172.21.15.72:57576) 2024-03-22T01:35:54.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:54.323+0000 7f7efe481700 1 -- 172.21.15.72:0/642380953 learned_addr learned my addr 172.21.15.72:0/642380953 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:35:54.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:54.323+0000 7f7efec82700 1 --2- 172.21.15.72:0/642380953 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ef8108660 0x7f7ef819d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:54.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:54.323+0000 7f7efdc80700 1 --2- 172.21.15.72:0/642380953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ef8103020 0x7f7ef81992c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:54.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:54.323+0000 7f7efe481700 1 -- 172.21.15.72:0/642380953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ef8103020 msgr2=0x7f7ef81992c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:54.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:54.323+0000 7f7efe481700 1 --2- 172.21.15.72:0/642380953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ef8103020 0x7f7ef81992c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:54.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:54.323+0000 7f7efe481700 1 -- 172.21.15.72:0/642380953 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ef8108660 msgr2=0x7f7ef819d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:54.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:54.323+0000 7f7efe481700 1 --2- 172.21.15.72:0/642380953 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ef8108660 0x7f7ef819d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:54.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:54.324+0000 7f7efe481700 1 -- 172.21.15.72:0/642380953 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ef819dd90 con 0x7f7ef8102660 2024-03-22T01:35:54.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:54.324+0000 7f7efec82700 1 --2- 172.21.15.72:0/642380953 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ef8108660 0x7f7ef819d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:35:54.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:54.324+0000 7f7efe481700 1 --2- 172.21.15.72:0/642380953 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7ef8102660 0x7f7ef8198d80 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f7ef400ef10 tx=0x7f7ef400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:54.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:54.324+0000 7f7eef7fe700 1 -- 172.21.15.72:0/642380953 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef400ca70 con 0x7f7ef8102660 2024-03-22T01:35:54.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:54.324+0000 7f7f006e5700 1 -- 172.21.15.72:0/642380953 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ef819e020 con 0x7f7ef8102660 2024-03-22T01:35:54.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:54.324+0000 7f7eef7fe700 1 -- 172.21.15.72:0/642380953 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ef4019070 con 0x7f7ef8102660 2024-03-22T01:35:54.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:54.324+0000 7f7eef7fe700 1 -- 172.21.15.72:0/642380953 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef40146e0 con 0x7f7ef8102660 2024-03-22T01:35:54.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:54.324+0000 7f7f006e5700 1 -- 172.21.15.72:0/642380953 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ef81a5900 con 0x7f7ef8102660 2024-03-22T01:35:54.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:54.325+0000 7f7f006e5700 1 -- 172.21.15.72:0/642380953 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ef8069340 con 0x7f7ef8102660 2024-03-22T01:35:54.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:54.326+0000 7f7eef7fe700 1 -- 172.21.15.72:0/642380953 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7f7ef4012050 con 0x7f7ef8102660 2024-03-22T01:35:54.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:54.326+0000 7f7eef7fe700 1 --2- 172.21.15.72:0/642380953 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f7ee40a2fb0 0x7f7ee40a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:54.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:54.327+0000 7f7efdc80700 1 --2- 172.21.15.72:0/642380953 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f7ee40a2fb0 0x7f7ee40a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:54.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:54.327+0000 7f7eef7fe700 1 -- 172.21.15.72:0/642380953 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f7ef40cf780 con 0x7f7ef8102660 2024-03-22T01:35:54.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:54.327+0000 7f7efdc80700 1 --2- 172.21.15.72:0/642380953 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f7ee40a2fb0 0x7f7ee40a5470 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f7ee8005950 tx=0x7f7ee80058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:54.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:54.331+0000 7f7eef7fe700 1 -- 172.21.15.72:0/642380953 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7ef4010040 con 0x7f7ef8102660 2024-03-22T01:35:54.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:54.522+0000 7f7f006e5700 1 -- 172.21.15.72:0/642380953 --> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7ef819a0e0 con 0x7f7ee40a2fb0 2024-03-22T01:35:54.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:54.532+0000 7f7eef7fe700 1 -- 172.21.15.72:0/642380953 <== mgr.14152 v2:172.21.15.72:6800/401403200 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340446 (secure 0 0 0) 0x7f7ef819a0e0 con 0x7f7ee40a2fb0 2024-03-22T01:35:54.536 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:35:54.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:54.535+0000 7f7f006e5700 1 -- 172.21.15.72:0/642380953 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f7ee40a2fb0 msgr2=0x7f7ee40a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:54.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:54.535+0000 7f7f006e5700 1 --2- 172.21.15.72:0/642380953 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f7ee40a2fb0 0x7f7ee40a5470 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f7ee8005950 tx=0x7f7ee80058e0 comp rx=0 tx=0).stop 2024-03-22T01:35:54.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:54.535+0000 7f7f006e5700 1 -- 172.21.15.72:0/642380953 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7ef8102660 msgr2=0x7f7ef8198d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:54.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:54.535+0000 7f7f006e5700 1 --2- 172.21.15.72:0/642380953 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7ef8102660 0x7f7ef8198d80 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f7ef400ef10 tx=0x7f7ef400c3a0 comp rx=0 tx=0).stop 2024-03-22T01:35:54.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:54.535+0000 7f7f006e5700 1 -- 172.21.15.72:0/642380953 shutdown_connections 2024-03-22T01:35:54.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:54.535+0000 7f7f006e5700 1 --2- 172.21.15.72:0/642380953 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f7ee40a2fb0 0x7f7ee40a5470 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:54.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:54.535+0000 7f7f006e5700 1 --2- 172.21.15.72:0/642380953 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ef8108660 0x7f7ef819d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:54.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:54.535+0000 7f7f006e5700 1 --2- 172.21.15.72:0/642380953 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7ef8102660 0x7f7ef8198d80 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:54.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:54.535+0000 7f7f006e5700 1 --2- 172.21.15.72:0/642380953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ef8103020 0x7f7ef81992c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:54.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:54.535+0000 7f7f006e5700 1 -- 172.21.15.72:0/642380953 >> 172.21.15.72:0/642380953 conn(0x7f7ef80fe130 msgr2=0x7f7ef80fe650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:35:54.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:54.535+0000 7f7f006e5700 1 -- 172.21.15.72:0/642380953 shutdown_connections 2024-03-22T01:35:54.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:54.535+0000 7f7f006e5700 1 -- 172.21.15.72:0/642380953 wait complete. 2024-03-22T01:35:54.541 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-03-22T01:35:54.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:54 smithi072 ceph-mon[32414]: pgmap v342: 129 pgs: 129 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-22T01:35:54.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:54 smithi089 ceph-mon[35770]: pgmap v342: 129 pgs: 129 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-22T01:35:54.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:54 smithi098 ceph-mon[36468]: pgmap v342: 129 pgs: 129 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-22T01:35:55.430 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":342,"stamp":"2024-03-22T01:35:53.448048+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":655005,"num_objects":26,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":292,"num_read_kb":262,"num_write":267,"num_write_kb":3033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":711,"ondisk_log_size":711,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":371664,"kb_used_data":14192,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124570160,"statfs":{"total":1151940427776,"available":1151559843840,"internally_reserved":0,"allocated":14532608,"data_stored":8067293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":365933882},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":249,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":11,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003250"},"pg_stats":[{"pgid":"3.3f","version":"93'8","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.465184+0000","last_change":"2024-03-22T01:34:15.487496+0000","last_active":"2024-03-22T01:35:45.465184+0000","last_peered":"2024-03-22T01:35:45.465184+0000","last_clean":"2024-03-22T01:35:45.465184+0000","last_became_active":"2024-03-22T01:34:15.486700+0000","last_became_peered":"2024-03-22T01:34:15.486700+0000","last_unstale":"2024-03-22T01:35:45.465184+0000","last_undegraded":"2024-03-22T01:35:45.465184+0000","last_fullsized":"2024-03-22T01:35:45.465184+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:03:19.146811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"93'4","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.455660+0000","last_change":"2024-03-22T01:34:15.489791+0000","last_active":"2024-03-22T01:35:45.455660+0000","last_peered":"2024-03-22T01:35:45.455660+0000","last_clean":"2024-03-22T01:35:45.455660+0000","last_became_active":"2024-03-22T01:34:15.489493+0000","last_became_peered":"2024-03-22T01:34:15.489493+0000","last_unstale":"2024-03-22T01:35:45.455660+0000","last_undegraded":"2024-03-22T01:35:45.455660+0000","last_fullsized":"2024-03-22T01:35:45.455660+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:45:05.060454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"93'7","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.458273+0000","last_change":"2024-03-22T01:34:15.491567+0000","last_active":"2024-03-22T01:35:45.458273+0000","last_peered":"2024-03-22T01:35:45.458273+0000","last_clean":"2024-03-22T01:35:45.458273+0000","last_became_active":"2024-03-22T01:34:15.491275+0000","last_became_peered":"2024-03-22T01:34:15.491275+0000","last_unstale":"2024-03-22T01:35:45.458273+0000","last_undegraded":"2024-03-22T01:35:45.458273+0000","last_fullsized":"2024-03-22T01:35:45.458273+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:26:36.630811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"93'8","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.449730+0000","last_change":"2024-03-22T01:34:15.488579+0000","last_active":"2024-03-22T01:35:45.449730+0000","last_peered":"2024-03-22T01:35:45.449730+0000","last_clean":"2024-03-22T01:35:45.449730+0000","last_became_active":"2024-03-22T01:34:15.488262+0000","last_became_peered":"2024-03-22T01:34:15.488262+0000","last_unstale":"2024-03-22T01:35:45.449730+0000","last_undegraded":"2024-03-22T01:35:45.449730+0000","last_fullsized":"2024-03-22T01:35:45.449730+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:58:21.788420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"93'4","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.432886+0000","last_change":"2024-03-22T01:34:15.485744+0000","last_active":"2024-03-22T01:35:45.432886+0000","last_peered":"2024-03-22T01:35:45.432886+0000","last_clean":"2024-03-22T01:35:45.432886+0000","last_became_active":"2024-03-22T01:34:15.485378+0000","last_became_peered":"2024-03-22T01:34:15.485378+0000","last_unstale":"2024-03-22T01:35:45.432886+0000","last_undegraded":"2024-03-22T01:35:45.432886+0000","last_fullsized":"2024-03-22T01:35:45.432886+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:06:40.979475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"93'6","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.460217+0000","last_change":"2024-03-22T01:34:15.495051+0000","last_active":"2024-03-22T01:35:45.460217+0000","last_peered":"2024-03-22T01:35:45.460217+0000","last_clean":"2024-03-22T01:35:45.460217+0000","last_became_active":"2024-03-22T01:34:15.494691+0000","last_became_peered":"2024-03-22T01:34:15.494691+0000","last_unstale":"2024-03-22T01:35:45.460217+0000","last_undegraded":"2024-03-22T01:35:45.460217+0000","last_fullsized":"2024-03-22T01:35:45.460217+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T01:44:28.336545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"93'5","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.453971+0000","last_change":"2024-03-22T01:34:15.493911+0000","last_active":"2024-03-22T01:35:45.453971+0000","last_peered":"2024-03-22T01:35:45.453971+0000","last_clean":"2024-03-22T01:35:45.453971+0000","last_became_active":"2024-03-22T01:34:15.493586+0000","last_became_peered":"2024-03-22T01:34:15.493586+0000","last_unstale":"2024-03-22T01:35:45.453971+0000","last_undegraded":"2024-03-22T01:35:45.453971+0000","last_fullsized":"2024-03-22T01:35:45.453971+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:22:41.391855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"93'9","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.455124+0000","last_change":"2024-03-22T01:34:15.481410+0000","last_active":"2024-03-22T01:35:45.455124+0000","last_peered":"2024-03-22T01:35:45.455124+0000","last_clean":"2024-03-22T01:35:45.455124+0000","last_became_active":"2024-03-22T01:34:15.481109+0000","last_became_peered":"2024-03-22T01:34:15.481109+0000","last_unstale":"2024-03-22T01:35:45.455124+0000","last_undegraded":"2024-03-22T01:35:45.455124+0000","last_fullsized":"2024-03-22T01:35:45.455124+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:27:10.004584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"93'11","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.463206+0000","last_change":"2024-03-22T01:34:15.485173+0000","last_active":"2024-03-22T01:35:45.463206+0000","last_peered":"2024-03-22T01:35:45.463206+0000","last_clean":"2024-03-22T01:35:45.463206+0000","last_became_active":"2024-03-22T01:34:15.484812+0000","last_became_peered":"2024-03-22T01:34:15.484812+0000","last_unstale":"2024-03-22T01:35:45.463206+0000","last_undegraded":"2024-03-22T01:35:45.463206+0000","last_fullsized":"2024-03-22T01:35:45.463206+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:44:06.383568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"93'16","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.450126+0000","last_change":"2024-03-22T01:34:15.497964+0000","last_active":"2024-03-22T01:35:45.450126+0000","last_peered":"2024-03-22T01:35:45.450126+0000","last_clean":"2024-03-22T01:35:45.450126+0000","last_became_active":"2024-03-22T01:34:15.496887+0000","last_became_peered":"2024-03-22T01:34:15.496887+0000","last_unstale":"2024-03-22T01:35:45.450126+0000","last_undegraded":"2024-03-22T01:35:45.450126+0000","last_fullsized":"2024-03-22T01:35:45.450126+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:13:40.420667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"93'6","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.454755+0000","last_change":"2024-03-22T01:34:15.496310+0000","last_active":"2024-03-22T01:35:45.454755+0000","last_peered":"2024-03-22T01:35:45.454755+0000","last_clean":"2024-03-22T01:35:45.454755+0000","last_became_active":"2024-03-22T01:34:15.494196+0000","last_became_peered":"2024-03-22T01:34:15.494196+0000","last_unstale":"2024-03-22T01:35:45.454755+0000","last_undegraded":"2024-03-22T01:35:45.454755+0000","last_fullsized":"2024-03-22T01:35:45.454755+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:33:02.798310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"93'7","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.440664+0000","last_change":"2024-03-22T01:34:15.495380+0000","last_active":"2024-03-22T01:35:45.440664+0000","last_peered":"2024-03-22T01:35:45.440664+0000","last_clean":"2024-03-22T01:35:45.440664+0000","last_became_active":"2024-03-22T01:34:15.495063+0000","last_became_peered":"2024-03-22T01:34:15.495063+0000","last_unstale":"2024-03-22T01:35:45.440664+0000","last_undegraded":"2024-03-22T01:35:45.440664+0000","last_fullsized":"2024-03-22T01:35:45.440664+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:46:29.497329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"93'10","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.454403+0000","last_change":"2024-03-22T01:34:15.497533+0000","last_active":"2024-03-22T01:35:45.454403+0000","last_peered":"2024-03-22T01:35:45.454403+0000","last_clean":"2024-03-22T01:35:45.454403+0000","last_became_active":"2024-03-22T01:34:15.497223+0000","last_became_peered":"2024-03-22T01:34:15.497223+0000","last_unstale":"2024-03-22T01:35:45.454403+0000","last_undegraded":"2024-03-22T01:35:45.454403+0000","last_fullsized":"2024-03-22T01:35:45.454403+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:17:49.202842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"93'7","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.455929+0000","last_change":"2024-03-22T01:34:15.491317+0000","last_active":"2024-03-22T01:35:45.455929+0000","last_peered":"2024-03-22T01:35:45.455929+0000","last_clean":"2024-03-22T01:35:45.455929+0000","last_became_active":"2024-03-22T01:34:15.490982+0000","last_became_peered":"2024-03-22T01:34:15.490982+0000","last_unstale":"2024-03-22T01:35:45.455929+0000","last_undegraded":"2024-03-22T01:35:45.455929+0000","last_fullsized":"2024-03-22T01:35:45.455929+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:05:39.889964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"93'7","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.455378+0000","last_change":"2024-03-22T01:34:15.488502+0000","last_active":"2024-03-22T01:35:45.455378+0000","last_peered":"2024-03-22T01:35:45.455378+0000","last_clean":"2024-03-22T01:35:45.455378+0000","last_became_active":"2024-03-22T01:34:15.488200+0000","last_became_peered":"2024-03-22T01:34:15.488200+0000","last_unstale":"2024-03-22T01:35:45.455378+0000","last_undegraded":"2024-03-22T01:35:45.455378+0000","last_fullsized":"2024-03-22T01:35:45.455378+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:52:59.991591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"93'9","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.458741+0000","last_change":"2024-03-22T01:34:15.489991+0000","last_active":"2024-03-22T01:35:45.458741+0000","last_peered":"2024-03-22T01:35:45.458741+0000","last_clean":"2024-03-22T01:35:45.458741+0000","last_became_active":"2024-03-22T01:34:15.489696+0000","last_became_peered":"2024-03-22T01:34:15.489696+0000","last_unstale":"2024-03-22T01:35:45.458741+0000","last_undegraded":"2024-03-22T01:35:45.458741+0000","last_fullsized":"2024-03-22T01:35:45.458741+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:10:45.316422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"93'8","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.465277+0000","last_change":"2024-03-22T01:34:15.488011+0000","last_active":"2024-03-22T01:35:45.465277+0000","last_peered":"2024-03-22T01:35:45.465277+0000","last_clean":"2024-03-22T01:35:45.465277+0000","last_became_active":"2024-03-22T01:34:15.487282+0000","last_became_peered":"2024-03-22T01:34:15.487282+0000","last_unstale":"2024-03-22T01:35:45.465277+0000","last_undegraded":"2024-03-22T01:35:45.465277+0000","last_fullsized":"2024-03-22T01:35:45.465277+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:16:21.396273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"93'10","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.463394+0000","last_change":"2024-03-22T01:34:15.488046+0000","last_active":"2024-03-22T01:35:45.463394+0000","last_peered":"2024-03-22T01:35:45.463394+0000","last_clean":"2024-03-22T01:35:45.463394+0000","last_became_active":"2024-03-22T01:34:15.487731+0000","last_became_peered":"2024-03-22T01:34:15.487731+0000","last_unstale":"2024-03-22T01:35:45.463394+0000","last_undegraded":"2024-03-22T01:35:45.463394+0000","last_fullsized":"2024-03-22T01:35:45.463394+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:44:16.931544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"93'7","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.442396+0000","last_change":"2024-03-22T01:34:15.503737+0000","last_active":"2024-03-22T01:35:45.442396+0000","last_peered":"2024-03-22T01:35:45.442396+0000","last_clean":"2024-03-22T01:35:45.442396+0000","last_became_active":"2024-03-22T01:34:15.502169+0000","last_became_peered":"2024-03-22T01:34:15.502169+0000","last_unstale":"2024-03-22T01:35:45.442396+0000","last_undegraded":"2024-03-22T01:35:45.442396+0000","last_fullsized":"2024-03-22T01:35:45.442396+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:05:53.027318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.752699+0000","last_change":"2024-03-22T01:34:19.527964+0000","last_active":"2024-03-22T01:35:16.752699+0000","last_peered":"2024-03-22T01:35:16.752699+0000","last_clean":"2024-03-22T01:35:16.752699+0000","last_became_active":"2024-03-22T01:34:19.527526+0000","last_became_peered":"2024-03-22T01:34:19.527526+0000","last_unstale":"2024-03-22T01:35:16.752699+0000","last_undegraded":"2024-03-22T01:35:16.752699+0000","last_fullsized":"2024-03-22T01:35:16.752699+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:35:39.404439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'4","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.748093+0000","last_change":"2024-03-22T01:34:14.473065+0000","last_active":"2024-03-22T01:35:16.748093+0000","last_peered":"2024-03-22T01:35:16.748093+0000","last_clean":"2024-03-22T01:35:16.748093+0000","last_became_active":"2024-03-22T01:34:14.472653+0000","last_became_peered":"2024-03-22T01:34:14.472653+0000","last_unstale":"2024-03-22T01:35:16.748093+0000","last_undegraded":"2024-03-22T01:35:16.748093+0000","last_fullsized":"2024-03-22T01:35:16.748093+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:11:09.525438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"93'5","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.453870+0000","last_change":"2024-03-22T01:34:15.497818+0000","last_active":"2024-03-22T01:35:45.453870+0000","last_peered":"2024-03-22T01:35:45.453870+0000","last_clean":"2024-03-22T01:35:45.453870+0000","last_became_active":"2024-03-22T01:34:15.497488+0000","last_became_peered":"2024-03-22T01:34:15.497488+0000","last_unstale":"2024-03-22T01:35:45.453870+0000","last_undegraded":"2024-03-22T01:35:45.453870+0000","last_fullsized":"2024-03-22T01:35:45.453870+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:21:09.340557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.743623+0000","last_change":"2024-03-22T01:34:19.528435+0000","last_active":"2024-03-22T01:35:16.743623+0000","last_peered":"2024-03-22T01:35:16.743623+0000","last_clean":"2024-03-22T01:35:16.743623+0000","last_became_active":"2024-03-22T01:34:19.528044+0000","last_became_peered":"2024-03-22T01:34:19.528044+0000","last_unstale":"2024-03-22T01:35:16.743623+0000","last_undegraded":"2024-03-22T01:35:16.743623+0000","last_fullsized":"2024-03-22T01:35:16.743623+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:52:05.682173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"86'1","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:17.204842+0000","last_change":"2024-03-22T01:34:14.465817+0000","last_active":"2024-03-22T01:35:17.204842+0000","last_peered":"2024-03-22T01:35:17.204842+0000","last_clean":"2024-03-22T01:35:17.204842+0000","last_became_active":"2024-03-22T01:34:14.464338+0000","last_became_peered":"2024-03-22T01:34:14.464338+0000","last_unstale":"2024-03-22T01:35:17.204842+0000","last_undegraded":"2024-03-22T01:35:17.204842+0000","last_fullsized":"2024-03-22T01:35:17.204842+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:04:47.457092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"93'6","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.460168+0000","last_change":"2024-03-22T01:34:15.481988+0000","last_active":"2024-03-22T01:35:45.460168+0000","last_peered":"2024-03-22T01:35:45.460168+0000","last_clean":"2024-03-22T01:35:45.460168+0000","last_became_active":"2024-03-22T01:34:15.481676+0000","last_became_peered":"2024-03-22T01:34:15.481676+0000","last_unstale":"2024-03-22T01:35:45.460168+0000","last_undegraded":"2024-03-22T01:35:45.460168+0000","last_fullsized":"2024-03-22T01:35:45.460168+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:37:54.609731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.743694+0000","last_change":"2024-03-22T01:34:19.528349+0000","last_active":"2024-03-22T01:35:16.743694+0000","last_peered":"2024-03-22T01:35:16.743694+0000","last_clean":"2024-03-22T01:35:16.743694+0000","last_became_active":"2024-03-22T01:34:19.527937+0000","last_became_peered":"2024-03-22T01:34:19.527937+0000","last_unstale":"2024-03-22T01:35:16.743694+0000","last_undegraded":"2024-03-22T01:35:16.743694+0000","last_fullsized":"2024-03-22T01:35:16.743694+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:48:34.242104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"86'2","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.756010+0000","last_change":"2024-03-22T01:35:16.756010+0000","last_active":"2024-03-22T01:35:16.756010+0000","last_peered":"2024-03-22T01:35:16.756010+0000","last_clean":"2024-03-22T01:35:16.756010+0000","last_became_active":"2024-03-22T01:35:16.751845+0000","last_became_peered":"2024-03-22T01:35:16.751845+0000","last_unstale":"2024-03-22T01:35:16.756010+0000","last_undegraded":"2024-03-22T01:35:16.756010+0000","last_fullsized":"2024-03-22T01:35:16.756010+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:23:18.305330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"93'9","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.464125+0000","last_change":"2024-03-22T01:34:15.491345+0000","last_active":"2024-03-22T01:35:45.464125+0000","last_peered":"2024-03-22T01:35:45.464125+0000","last_clean":"2024-03-22T01:35:45.464125+0000","last_became_active":"2024-03-22T01:34:15.490833+0000","last_became_peered":"2024-03-22T01:34:15.490833+0000","last_unstale":"2024-03-22T01:35:45.464125+0000","last_undegraded":"2024-03-22T01:35:45.464125+0000","last_fullsized":"2024-03-22T01:35:45.464125+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:30:43.950679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.743633+0000","last_change":"2024-03-22T01:34:19.528351+0000","last_active":"2024-03-22T01:35:16.743633+0000","last_peered":"2024-03-22T01:35:16.743633+0000","last_clean":"2024-03-22T01:35:16.743633+0000","last_became_active":"2024-03-22T01:34:19.527939+0000","last_became_peered":"2024-03-22T01:34:19.527939+0000","last_unstale":"2024-03-22T01:35:16.743633+0000","last_undegraded":"2024-03-22T01:35:16.743633+0000","last_fullsized":"2024-03-22T01:35:16.743633+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:30:21.055848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.756388+0000","last_change":"2024-03-22T01:35:16.755726+0000","last_active":"2024-03-22T01:35:16.756388+0000","last_peered":"2024-03-22T01:35:16.756388+0000","last_clean":"2024-03-22T01:35:16.756388+0000","last_became_active":"2024-03-22T01:35:16.749107+0000","last_became_peered":"2024-03-22T01:35:16.749107+0000","last_unstale":"2024-03-22T01:35:16.756388+0000","last_undegraded":"2024-03-22T01:35:16.756388+0000","last_fullsized":"2024-03-22T01:35:16.756388+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:49:37.500003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"93'15","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.464009+0000","last_change":"2024-03-22T01:34:15.479530+0000","last_active":"2024-03-22T01:35:45.464009+0000","last_peered":"2024-03-22T01:35:45.464009+0000","last_clean":"2024-03-22T01:35:45.464009+0000","last_became_active":"2024-03-22T01:34:15.479198+0000","last_became_peered":"2024-03-22T01:34:15.479198+0000","last_unstale":"2024-03-22T01:35:45.464009+0000","last_undegraded":"2024-03-22T01:35:45.464009+0000","last_fullsized":"2024-03-22T01:35:45.464009+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:28:51.624369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.753040+0000","last_change":"2024-03-22T01:34:19.524486+0000","last_active":"2024-03-22T01:35:16.753040+0000","last_peered":"2024-03-22T01:35:16.753040+0000","last_clean":"2024-03-22T01:35:16.753040+0000","last_became_active":"2024-03-22T01:34:19.524030+0000","last_became_peered":"2024-03-22T01:34:19.524030+0000","last_unstale":"2024-03-22T01:35:16.753040+0000","last_undegraded":"2024-03-22T01:35:16.753040+0000","last_fullsized":"2024-03-22T01:35:16.753040+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:31:10.157431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"86'1","reported_seq":26,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.745175+0000","last_change":"2024-03-22T01:34:14.469191+0000","last_active":"2024-03-22T01:35:16.745175+0000","last_peered":"2024-03-22T01:35:16.745175+0000","last_clean":"2024-03-22T01:35:16.745175+0000","last_became_active":"2024-03-22T01:34:14.468842+0000","last_became_peered":"2024-03-22T01:34:14.468842+0000","last_unstale":"2024-03-22T01:35:16.745175+0000","last_undegraded":"2024-03-22T01:35:16.745175+0000","last_fullsized":"2024-03-22T01:35:16.745175+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:29:56.566256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"93'7","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.443178+0000","last_change":"2024-03-22T01:34:15.483736+0000","last_active":"2024-03-22T01:35:45.443178+0000","last_peered":"2024-03-22T01:35:45.443178+0000","last_clean":"2024-03-22T01:35:45.443178+0000","last_became_active":"2024-03-22T01:34:15.483401+0000","last_became_peered":"2024-03-22T01:34:15.483401+0000","last_unstale":"2024-03-22T01:35:45.443178+0000","last_undegraded":"2024-03-22T01:35:45.443178+0000","last_fullsized":"2024-03-22T01:35:45.443178+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:26:51.974932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.753323+0000","last_change":"2024-03-22T01:34:19.534542+0000","last_active":"2024-03-22T01:35:16.753323+0000","last_peered":"2024-03-22T01:35:16.753323+0000","last_clean":"2024-03-22T01:35:16.753323+0000","last_became_active":"2024-03-22T01:34:19.534039+0000","last_became_peered":"2024-03-22T01:34:19.534039+0000","last_unstale":"2024-03-22T01:35:16.753323+0000","last_undegraded":"2024-03-22T01:35:16.753323+0000","last_fullsized":"2024-03-22T01:35:16.753323+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:59:37.363972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"86'1","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.814330+0000","last_change":"2024-03-22T01:34:14.468382+0000","last_active":"2024-03-22T01:35:16.814330+0000","last_peered":"2024-03-22T01:35:16.814330+0000","last_clean":"2024-03-22T01:35:16.814330+0000","last_became_active":"2024-03-22T01:34:14.467939+0000","last_became_peered":"2024-03-22T01:34:14.467939+0000","last_unstale":"2024-03-22T01:35:16.814330+0000","last_undegraded":"2024-03-22T01:35:16.814330+0000","last_fullsized":"2024-03-22T01:35:16.814330+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:57:17.031893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"93'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.457175+0000","last_change":"2024-03-22T01:35:17.206178+0000","last_active":"2024-03-22T01:35:45.457175+0000","last_peered":"2024-03-22T01:35:45.457175+0000","last_clean":"2024-03-22T01:35:45.457175+0000","last_became_active":"2024-03-22T01:35:17.205717+0000","last_became_peered":"2024-03-22T01:35:17.205717+0000","last_unstale":"2024-03-22T01:35:45.457175+0000","last_undegraded":"2024-03-22T01:35:45.457175+0000","last_fullsized":"2024-03-22T01:35:45.457175+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:14:15.920771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.746553+0000","last_change":"2024-03-22T01:34:19.531532+0000","last_active":"2024-03-22T01:35:16.746553+0000","last_peered":"2024-03-22T01:35:16.746553+0000","last_clean":"2024-03-22T01:35:16.746553+0000","last_became_active":"2024-03-22T01:34:19.531202+0000","last_became_peered":"2024-03-22T01:34:19.531202+0000","last_unstale":"2024-03-22T01:35:16.746553+0000","last_undegraded":"2024-03-22T01:35:16.746553+0000","last_fullsized":"2024-03-22T01:35:16.746553+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:52:12.823806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":26,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.743316+0000","last_change":"2024-03-22T01:34:14.464485+0000","last_active":"2024-03-22T01:35:16.743316+0000","last_peered":"2024-03-22T01:35:16.743316+0000","last_clean":"2024-03-22T01:35:16.743316+0000","last_became_active":"2024-03-22T01:34:14.463973+0000","last_became_peered":"2024-03-22T01:34:14.463973+0000","last_unstale":"2024-03-22T01:35:16.743316+0000","last_undegraded":"2024-03-22T01:35:16.743316+0000","last_fullsized":"2024-03-22T01:35:16.743316+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:11:13.265346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"93'8","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.449431+0000","last_change":"2024-03-22T01:34:15.483893+0000","last_active":"2024-03-22T01:35:45.449431+0000","last_peered":"2024-03-22T01:35:45.449431+0000","last_clean":"2024-03-22T01:35:45.449431+0000","last_became_active":"2024-03-22T01:34:15.482295+0000","last_became_peered":"2024-03-22T01:34:15.482295+0000","last_unstale":"2024-03-22T01:35:45.449431+0000","last_undegraded":"2024-03-22T01:35:45.449431+0000","last_fullsized":"2024-03-22T01:35:45.449431+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:17:33.274900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.746059+0000","last_change":"2024-03-22T01:34:19.522567+0000","last_active":"2024-03-22T01:35:16.746059+0000","last_peered":"2024-03-22T01:35:16.746059+0000","last_clean":"2024-03-22T01:35:16.746059+0000","last_became_active":"2024-03-22T01:34:19.522165+0000","last_became_peered":"2024-03-22T01:34:19.522165+0000","last_unstale":"2024-03-22T01:35:16.746059+0000","last_undegraded":"2024-03-22T01:35:16.746059+0000","last_fullsized":"2024-03-22T01:35:16.746059+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:41:37.829622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.744359+0000","last_change":"2024-03-22T01:34:14.463824+0000","last_active":"2024-03-22T01:35:16.744359+0000","last_peered":"2024-03-22T01:35:16.744359+0000","last_clean":"2024-03-22T01:35:16.744359+0000","last_became_active":"2024-03-22T01:34:14.463484+0000","last_became_peered":"2024-03-22T01:34:14.463484+0000","last_unstale":"2024-03-22T01:35:16.744359+0000","last_undegraded":"2024-03-22T01:35:16.744359+0000","last_fullsized":"2024-03-22T01:35:16.744359+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:19:31.568853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"93'11","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.459393+0000","last_change":"2024-03-22T01:34:15.488883+0000","last_active":"2024-03-22T01:35:45.459393+0000","last_peered":"2024-03-22T01:35:45.459393+0000","last_clean":"2024-03-22T01:35:45.459393+0000","last_became_active":"2024-03-22T01:34:15.488573+0000","last_became_peered":"2024-03-22T01:34:15.488573+0000","last_unstale":"2024-03-22T01:35:45.459393+0000","last_undegraded":"2024-03-22T01:35:45.459393+0000","last_fullsized":"2024-03-22T01:35:45.459393+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:28:46.388930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:17.204329+0000","last_change":"2024-03-22T01:34:19.527321+0000","last_active":"2024-03-22T01:35:17.204329+0000","last_peered":"2024-03-22T01:35:17.204329+0000","last_clean":"2024-03-22T01:35:17.204329+0000","last_became_active":"2024-03-22T01:34:19.526905+0000","last_became_peered":"2024-03-22T01:34:19.526905+0000","last_unstale":"2024-03-22T01:35:17.204329+0000","last_undegraded":"2024-03-22T01:35:17.204329+0000","last_fullsized":"2024-03-22T01:35:17.204329+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:49:31.023938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.814616+0000","last_change":"2024-03-22T01:34:14.471415+0000","last_active":"2024-03-22T01:35:16.814616+0000","last_peered":"2024-03-22T01:35:16.814616+0000","last_clean":"2024-03-22T01:35:16.814616+0000","last_became_active":"2024-03-22T01:34:14.470928+0000","last_became_peered":"2024-03-22T01:34:14.470928+0000","last_unstale":"2024-03-22T01:35:16.814616+0000","last_undegraded":"2024-03-22T01:35:16.814616+0000","last_fullsized":"2024-03-22T01:35:16.814616+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:09:10.286549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"93'10","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.464353+0000","last_change":"2024-03-22T01:34:15.489656+0000","last_active":"2024-03-22T01:35:45.464353+0000","last_peered":"2024-03-22T01:35:45.464353+0000","last_clean":"2024-03-22T01:35:45.464353+0000","last_became_active":"2024-03-22T01:34:15.488933+0000","last_became_peered":"2024-03-22T01:34:15.488933+0000","last_unstale":"2024-03-22T01:35:45.464353+0000","last_undegraded":"2024-03-22T01:35:45.464353+0000","last_fullsized":"2024-03-22T01:35:45.464353+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:17:38.789414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.746845+0000","last_change":"2024-03-22T01:34:19.534174+0000","last_active":"2024-03-22T01:35:16.746845+0000","last_peered":"2024-03-22T01:35:16.746845+0000","last_clean":"2024-03-22T01:35:16.746845+0000","last_became_active":"2024-03-22T01:34:19.533759+0000","last_became_peered":"2024-03-22T01:34:19.533759+0000","last_unstale":"2024-03-22T01:35:16.746845+0000","last_undegraded":"2024-03-22T01:35:16.746845+0000","last_fullsized":"2024-03-22T01:35:16.746845+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:24:50.363984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":26,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.746461+0000","last_change":"2024-03-22T01:34:14.465755+0000","last_active":"2024-03-22T01:35:16.746461+0000","last_peered":"2024-03-22T01:35:16.746461+0000","last_clean":"2024-03-22T01:35:16.746461+0000","last_became_active":"2024-03-22T01:34:14.465230+0000","last_became_peered":"2024-03-22T01:34:14.465230+0000","last_unstale":"2024-03-22T01:35:16.746461+0000","last_undegraded":"2024-03-22T01:35:16.746461+0000","last_fullsized":"2024-03-22T01:35:16.746461+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:56:29.286551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"93'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.458831+0000","last_change":"2024-03-22T01:35:16.748981+0000","last_active":"2024-03-22T01:35:45.458831+0000","last_peered":"2024-03-22T01:35:45.458831+0000","last_clean":"2024-03-22T01:35:45.458831+0000","last_became_active":"2024-03-22T01:35:16.748524+0000","last_became_peered":"2024-03-22T01:35:16.748524+0000","last_unstale":"2024-03-22T01:35:45.458831+0000","last_undegraded":"2024-03-22T01:35:45.458831+0000","last_fullsized":"2024-03-22T01:35:45.458831+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:23:37.763347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.743817+0000","last_change":"2024-03-22T01:34:19.527218+0000","last_active":"2024-03-22T01:35:16.743817+0000","last_peered":"2024-03-22T01:35:16.743817+0000","last_clean":"2024-03-22T01:35:16.743817+0000","last_became_active":"2024-03-22T01:34:19.526856+0000","last_became_peered":"2024-03-22T01:34:19.526856+0000","last_unstale":"2024-03-22T01:35:16.743817+0000","last_undegraded":"2024-03-22T01:35:16.743817+0000","last_fullsized":"2024-03-22T01:35:16.743817+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:25:33.878015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"86'3","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.744957+0000","last_change":"2024-03-22T01:34:14.463365+0000","last_active":"2024-03-22T01:35:16.744957+0000","last_peered":"2024-03-22T01:35:16.744957+0000","last_clean":"2024-03-22T01:35:16.744957+0000","last_became_active":"2024-03-22T01:34:14.462984+0000","last_became_peered":"2024-03-22T01:34:14.462984+0000","last_unstale":"2024-03-22T01:35:16.744957+0000","last_undegraded":"2024-03-22T01:35:16.744957+0000","last_fullsized":"2024-03-22T01:35:16.744957+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:55:27.318854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"93'11","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.463709+0000","last_change":"2024-03-22T01:34:15.485309+0000","last_active":"2024-03-22T01:35:45.463709+0000","last_peered":"2024-03-22T01:35:45.463709+0000","last_clean":"2024-03-22T01:35:45.463709+0000","last_became_active":"2024-03-22T01:34:15.484964+0000","last_became_peered":"2024-03-22T01:34:15.484964+0000","last_unstale":"2024-03-22T01:35:45.463709+0000","last_undegraded":"2024-03-22T01:35:45.463709+0000","last_fullsized":"2024-03-22T01:35:45.463709+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:25:57.911623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.747797+0000","last_change":"2024-03-22T01:34:19.529346+0000","last_active":"2024-03-22T01:35:16.747797+0000","last_peered":"2024-03-22T01:35:16.747797+0000","last_clean":"2024-03-22T01:35:16.747797+0000","last_became_active":"2024-03-22T01:34:19.525999+0000","last_became_peered":"2024-03-22T01:34:19.525999+0000","last_unstale":"2024-03-22T01:35:16.747797+0000","last_undegraded":"2024-03-22T01:35:16.747797+0000","last_fullsized":"2024-03-22T01:35:16.747797+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:00:21.868812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"93'33","reported_seq":210,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:49.617281+0000","last_change":"2024-03-22T01:34:14.469378+0000","last_active":"2024-03-22T01:35:49.617281+0000","last_peered":"2024-03-22T01:35:49.617281+0000","last_clean":"2024-03-22T01:35:49.617281+0000","last_became_active":"2024-03-22T01:34:14.468956+0000","last_became_peered":"2024-03-22T01:34:14.468956+0000","last_unstale":"2024-03-22T01:35:49.617281+0000","last_undegraded":"2024-03-22T01:35:49.617281+0000","last_fullsized":"2024-03-22T01:35:49.617281+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:40:49.048793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63605,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":150,"num_read_kb":137,"num_write":33,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"93'6","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.453026+0000","last_change":"2024-03-22T01:34:15.488497+0000","last_active":"2024-03-22T01:35:45.453026+0000","last_peered":"2024-03-22T01:35:45.453026+0000","last_clean":"2024-03-22T01:35:45.453026+0000","last_became_active":"2024-03-22T01:34:15.488134+0000","last_became_peered":"2024-03-22T01:34:15.488134+0000","last_unstale":"2024-03-22T01:35:45.453026+0000","last_undegraded":"2024-03-22T01:35:45.453026+0000","last_fullsized":"2024-03-22T01:35:45.453026+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:50:41.206135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.745269+0000","last_change":"2024-03-22T01:34:19.528744+0000","last_active":"2024-03-22T01:35:16.745269+0000","last_peered":"2024-03-22T01:35:16.745269+0000","last_clean":"2024-03-22T01:35:16.745269+0000","last_became_active":"2024-03-22T01:34:19.528375+0000","last_became_peered":"2024-03-22T01:34:19.528375+0000","last_unstale":"2024-03-22T01:35:16.745269+0000","last_undegraded":"2024-03-22T01:35:16.745269+0000","last_fullsized":"2024-03-22T01:35:16.745269+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:47:53.753923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"86'2","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.752872+0000","last_change":"2024-03-22T01:34:14.463289+0000","last_active":"2024-03-22T01:35:16.752872+0000","last_peered":"2024-03-22T01:35:16.752872+0000","last_clean":"2024-03-22T01:35:16.752872+0000","last_became_active":"2024-03-22T01:34:14.462883+0000","last_became_peered":"2024-03-22T01:34:14.462883+0000","last_unstale":"2024-03-22T01:35:16.752872+0000","last_undegraded":"2024-03-22T01:35:16.752872+0000","last_fullsized":"2024-03-22T01:35:16.752872+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:06:14.452674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"93'9","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.454086+0000","last_change":"2024-03-22T01:34:15.484785+0000","last_active":"2024-03-22T01:35:45.454086+0000","last_peered":"2024-03-22T01:35:45.454086+0000","last_clean":"2024-03-22T01:35:45.454086+0000","last_became_active":"2024-03-22T01:34:15.484474+0000","last_became_peered":"2024-03-22T01:34:15.484474+0000","last_unstale":"2024-03-22T01:35:45.454086+0000","last_undegraded":"2024-03-22T01:35:45.454086+0000","last_fullsized":"2024-03-22T01:35:45.454086+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:11:13.338836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.746861+0000","last_change":"2024-03-22T01:34:19.527379+0000","last_active":"2024-03-22T01:35:16.746861+0000","last_peered":"2024-03-22T01:35:16.746861+0000","last_clean":"2024-03-22T01:35:16.746861+0000","last_became_active":"2024-03-22T01:34:19.526957+0000","last_became_peered":"2024-03-22T01:34:19.526957+0000","last_unstale":"2024-03-22T01:35:16.746861+0000","last_undegraded":"2024-03-22T01:35:16.746861+0000","last_fullsized":"2024-03-22T01:35:16.746861+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:03:09.538334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.744463+0000","last_change":"2024-03-22T01:34:14.467372+0000","last_active":"2024-03-22T01:35:16.744463+0000","last_peered":"2024-03-22T01:35:16.744463+0000","last_clean":"2024-03-22T01:35:16.744463+0000","last_became_active":"2024-03-22T01:34:14.466124+0000","last_became_peered":"2024-03-22T01:34:14.466124+0000","last_unstale":"2024-03-22T01:35:16.744463+0000","last_undegraded":"2024-03-22T01:35:16.744463+0000","last_fullsized":"2024-03-22T01:35:16.744463+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:15:41.804735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"93'15","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.456546+0000","last_change":"2024-03-22T01:34:15.491533+0000","last_active":"2024-03-22T01:35:45.456546+0000","last_peered":"2024-03-22T01:35:45.456546+0000","last_clean":"2024-03-22T01:35:45.456546+0000","last_became_active":"2024-03-22T01:34:15.491208+0000","last_became_peered":"2024-03-22T01:34:15.491208+0000","last_unstale":"2024-03-22T01:35:45.456546+0000","last_undegraded":"2024-03-22T01:35:45.456546+0000","last_fullsized":"2024-03-22T01:35:45.456546+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:03:07.800607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.748346+0000","last_change":"2024-03-22T01:34:19.533419+0000","last_active":"2024-03-22T01:35:16.748346+0000","last_peered":"2024-03-22T01:35:16.748346+0000","last_clean":"2024-03-22T01:35:16.748346+0000","last_became_active":"2024-03-22T01:34:19.530042+0000","last_became_peered":"2024-03-22T01:34:19.530042+0000","last_unstale":"2024-03-22T01:35:16.748346+0000","last_undegraded":"2024-03-22T01:35:16.748346+0000","last_fullsized":"2024-03-22T01:35:16.748346+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:46:59.859742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.743765+0000","last_change":"2024-03-22T01:34:14.467033+0000","last_active":"2024-03-22T01:35:16.743765+0000","last_peered":"2024-03-22T01:35:16.743765+0000","last_clean":"2024-03-22T01:35:16.743765+0000","last_became_active":"2024-03-22T01:34:14.466649+0000","last_became_peered":"2024-03-22T01:34:14.466649+0000","last_unstale":"2024-03-22T01:35:16.743765+0000","last_undegraded":"2024-03-22T01:35:16.743765+0000","last_fullsized":"2024-03-22T01:35:16.743765+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:38:44.890899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"93'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.442847+0000","last_change":"2024-03-22T01:35:16.815302+0000","last_active":"2024-03-22T01:35:45.442847+0000","last_peered":"2024-03-22T01:35:45.442847+0000","last_clean":"2024-03-22T01:35:45.442847+0000","last_became_active":"2024-03-22T01:35:16.814988+0000","last_became_peered":"2024-03-22T01:35:16.814988+0000","last_unstale":"2024-03-22T01:35:45.442847+0000","last_undegraded":"2024-03-22T01:35:45.442847+0000","last_fullsized":"2024-03-22T01:35:45.442847+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:54:47.793437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.743568+0000","last_change":"2024-03-22T01:34:19.531417+0000","last_active":"2024-03-22T01:35:16.743568+0000","last_peered":"2024-03-22T01:35:16.743568+0000","last_clean":"2024-03-22T01:35:16.743568+0000","last_became_active":"2024-03-22T01:34:19.531003+0000","last_became_peered":"2024-03-22T01:34:19.531003+0000","last_unstale":"2024-03-22T01:35:16.743568+0000","last_undegraded":"2024-03-22T01:35:16.743568+0000","last_fullsized":"2024-03-22T01:35:16.743568+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:33:52.865110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"86'8","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.748313+0000","last_change":"2024-03-22T01:34:14.472904+0000","last_active":"2024-03-22T01:35:16.748313+0000","last_peered":"2024-03-22T01:35:16.748313+0000","last_clean":"2024-03-22T01:35:16.748313+0000","last_became_active":"2024-03-22T01:34:14.472442+0000","last_became_peered":"2024-03-22T01:34:14.472442+0000","last_unstale":"2024-03-22T01:35:16.748313+0000","last_undegraded":"2024-03-22T01:35:16.748313+0000","last_fullsized":"2024-03-22T01:35:16.748313+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:57:45.478316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"93'6","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.463203+0000","last_change":"2024-03-22T01:34:15.477411+0000","last_active":"2024-03-22T01:35:45.463203+0000","last_peered":"2024-03-22T01:35:45.463203+0000","last_clean":"2024-03-22T01:35:45.463203+0000","last_became_active":"2024-03-22T01:34:15.477035+0000","last_became_peered":"2024-03-22T01:34:15.477035+0000","last_unstale":"2024-03-22T01:35:45.463203+0000","last_undegraded":"2024-03-22T01:35:45.463203+0000","last_fullsized":"2024-03-22T01:35:45.463203+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:33:13.410872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.744275+0000","last_change":"2024-03-22T01:34:19.528258+0000","last_active":"2024-03-22T01:35:16.744275+0000","last_peered":"2024-03-22T01:35:16.744275+0000","last_clean":"2024-03-22T01:35:16.744275+0000","last_became_active":"2024-03-22T01:34:19.527668+0000","last_became_peered":"2024-03-22T01:34:19.527668+0000","last_unstale":"2024-03-22T01:35:16.744275+0000","last_undegraded":"2024-03-22T01:35:16.744275+0000","last_fullsized":"2024-03-22T01:35:16.744275+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:34:19.162609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'1","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.748746+0000","last_change":"2024-03-22T01:34:14.464735+0000","last_active":"2024-03-22T01:35:16.748746+0000","last_peered":"2024-03-22T01:35:16.748746+0000","last_clean":"2024-03-22T01:35:16.748746+0000","last_became_active":"2024-03-22T01:34:14.464359+0000","last_became_peered":"2024-03-22T01:34:14.464359+0000","last_unstale":"2024-03-22T01:35:16.748746+0000","last_undegraded":"2024-03-22T01:35:16.748746+0000","last_fullsized":"2024-03-22T01:35:16.748746+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:21:18.354922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"93'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.451818+0000","last_change":"2024-03-22T01:35:16.752884+0000","last_active":"2024-03-22T01:35:45.451818+0000","last_peered":"2024-03-22T01:35:45.451818+0000","last_clean":"2024-03-22T01:35:45.451818+0000","last_became_active":"2024-03-22T01:35:16.751355+0000","last_became_peered":"2024-03-22T01:35:16.751355+0000","last_unstale":"2024-03-22T01:35:45.451818+0000","last_undegraded":"2024-03-22T01:35:45.451818+0000","last_fullsized":"2024-03-22T01:35:45.451818+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:49:33.261481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.748621+0000","last_change":"2024-03-22T01:34:19.668136+0000","last_active":"2024-03-22T01:35:16.748621+0000","last_peered":"2024-03-22T01:35:16.748621+0000","last_clean":"2024-03-22T01:35:16.748621+0000","last_became_active":"2024-03-22T01:34:19.667633+0000","last_became_peered":"2024-03-22T01:34:19.667633+0000","last_unstale":"2024-03-22T01:35:16.748621+0000","last_undegraded":"2024-03-22T01:35:16.748621+0000","last_fullsized":"2024-03-22T01:35:16.748621+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:07:26.455373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.747658+0000","last_change":"2024-03-22T01:34:14.471959+0000","last_active":"2024-03-22T01:35:16.747658+0000","last_peered":"2024-03-22T01:35:16.747658+0000","last_clean":"2024-03-22T01:35:16.747658+0000","last_became_active":"2024-03-22T01:34:14.471618+0000","last_became_peered":"2024-03-22T01:34:14.471618+0000","last_unstale":"2024-03-22T01:35:16.747658+0000","last_undegraded":"2024-03-22T01:35:16.747658+0000","last_fullsized":"2024-03-22T01:35:16.747658+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:45:03.537013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"93'4","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.450651+0000","last_change":"2024-03-22T01:34:15.495770+0000","last_active":"2024-03-22T01:35:45.450651+0000","last_peered":"2024-03-22T01:35:45.450651+0000","last_clean":"2024-03-22T01:35:45.450651+0000","last_became_active":"2024-03-22T01:34:15.495403+0000","last_became_peered":"2024-03-22T01:34:15.495403+0000","last_unstale":"2024-03-22T01:35:45.450651+0000","last_undegraded":"2024-03-22T01:35:45.450651+0000","last_fullsized":"2024-03-22T01:35:45.450651+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:48:05.184305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.743667+0000","last_change":"2024-03-22T01:34:19.520204+0000","last_active":"2024-03-22T01:35:16.743667+0000","last_peered":"2024-03-22T01:35:16.743667+0000","last_clean":"2024-03-22T01:35:16.743667+0000","last_became_active":"2024-03-22T01:34:19.519847+0000","last_became_peered":"2024-03-22T01:34:19.519847+0000","last_unstale":"2024-03-22T01:35:16.743667+0000","last_undegraded":"2024-03-22T01:35:16.743667+0000","last_fullsized":"2024-03-22T01:35:16.743667+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:06:21.389636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":26,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.746100+0000","last_change":"2024-03-22T01:34:14.464162+0000","last_active":"2024-03-22T01:35:16.746100+0000","last_peered":"2024-03-22T01:35:16.746100+0000","last_clean":"2024-03-22T01:35:16.746100+0000","last_became_active":"2024-03-22T01:34:14.462700+0000","last_became_peered":"2024-03-22T01:34:14.462700+0000","last_unstale":"2024-03-22T01:35:16.746100+0000","last_undegraded":"2024-03-22T01:35:16.746100+0000","last_fullsized":"2024-03-22T01:35:16.746100+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:45:32.262466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"93'7","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.457078+0000","last_change":"2024-03-22T01:34:15.488256+0000","last_active":"2024-03-22T01:35:45.457078+0000","last_peered":"2024-03-22T01:35:45.457078+0000","last_clean":"2024-03-22T01:35:45.457078+0000","last_became_active":"2024-03-22T01:34:15.487939+0000","last_became_peered":"2024-03-22T01:34:15.487939+0000","last_unstale":"2024-03-22T01:35:45.457078+0000","last_undegraded":"2024-03-22T01:35:45.457078+0000","last_fullsized":"2024-03-22T01:35:45.457078+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:30:49.323531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.743552+0000","last_change":"2024-03-22T01:34:19.667633+0000","last_active":"2024-03-22T01:35:16.743552+0000","last_peered":"2024-03-22T01:35:16.743552+0000","last_clean":"2024-03-22T01:35:16.743552+0000","last_became_active":"2024-03-22T01:34:19.667199+0000","last_became_peered":"2024-03-22T01:34:19.667199+0000","last_unstale":"2024-03-22T01:35:16.743552+0000","last_undegraded":"2024-03-22T01:35:16.743552+0000","last_fullsized":"2024-03-22T01:35:16.743552+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T01:58:13.459475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"93'8","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.453646+0000","last_change":"2024-03-22T01:34:15.489468+0000","last_active":"2024-03-22T01:35:45.453646+0000","last_peered":"2024-03-22T01:35:45.453646+0000","last_clean":"2024-03-22T01:35:45.453646+0000","last_became_active":"2024-03-22T01:34:15.489029+0000","last_became_peered":"2024-03-22T01:34:15.489029+0000","last_unstale":"2024-03-22T01:35:45.453646+0000","last_undegraded":"2024-03-22T01:35:45.453646+0000","last_fullsized":"2024-03-22T01:35:45.453646+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:00:01.370161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":26,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.743202+0000","last_change":"2024-03-22T01:34:14.464552+0000","last_active":"2024-03-22T01:35:16.743202+0000","last_peered":"2024-03-22T01:35:16.743202+0000","last_clean":"2024-03-22T01:35:16.743202+0000","last_became_active":"2024-03-22T01:34:14.463554+0000","last_became_peered":"2024-03-22T01:34:14.463554+0000","last_unstale":"2024-03-22T01:35:16.743202+0000","last_undegraded":"2024-03-22T01:35:16.743202+0000","last_fullsized":"2024-03-22T01:35:16.743202+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:51:16.709278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"91'1","reported_seq":19,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.746441+0000","last_change":"2024-03-22T01:34:19.526066+0000","last_active":"2024-03-22T01:35:16.746441+0000","last_peered":"2024-03-22T01:35:16.746441+0000","last_clean":"2024-03-22T01:35:16.746441+0000","last_became_active":"2024-03-22T01:34:19.525659+0000","last_became_peered":"2024-03-22T01:34:19.525659+0000","last_unstale":"2024-03-22T01:35:16.746441+0000","last_undegraded":"2024-03-22T01:35:16.746441+0000","last_fullsized":"2024-03-22T01:35:16.746441+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:06:26.287971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"93'8","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.454417+0000","last_change":"2024-03-22T01:34:15.477018+0000","last_active":"2024-03-22T01:35:45.454417+0000","last_peered":"2024-03-22T01:35:45.454417+0000","last_clean":"2024-03-22T01:35:45.454417+0000","last_became_active":"2024-03-22T01:34:15.476613+0000","last_became_peered":"2024-03-22T01:34:15.476613+0000","last_unstale":"2024-03-22T01:35:45.454417+0000","last_undegraded":"2024-03-22T01:35:45.454417+0000","last_fullsized":"2024-03-22T01:35:45.454417+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:10:07.980123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.743756+0000","last_change":"2024-03-22T01:34:14.470922+0000","last_active":"2024-03-22T01:35:16.743756+0000","last_peered":"2024-03-22T01:35:16.743756+0000","last_clean":"2024-03-22T01:35:16.743756+0000","last_became_active":"2024-03-22T01:34:14.470515+0000","last_became_peered":"2024-03-22T01:34:14.470515+0000","last_unstale":"2024-03-22T01:35:16.743756+0000","last_undegraded":"2024-03-22T01:35:16.743756+0000","last_fullsized":"2024-03-22T01:35:16.743756+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:52:50.203631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":359,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.745989+0000","last_change":"2024-03-22T01:33:03.615234+0000","last_active":"2024-03-22T01:35:16.745989+0000","last_peered":"2024-03-22T01:35:16.745989+0000","last_clean":"2024-03-22T01:35:16.745989+0000","last_became_active":"2024-03-22T01:33:03.614710+0000","last_became_peered":"2024-03-22T01:33:03.614710+0000","last_unstale":"2024-03-22T01:35:16.745989+0000","last_undegraded":"2024-03-22T01:35:16.745989+0000","last_fullsized":"2024-03-22T01:35:16.745989+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:29:38.312292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:29:38.312292+0000","last_clean_scrub_stamp":"2024-03-22T01:29:38.312292+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:56:42.348119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.745330+0000","last_change":"2024-03-22T01:34:19.668565+0000","last_active":"2024-03-22T01:35:16.745330+0000","last_peered":"2024-03-22T01:35:16.745330+0000","last_clean":"2024-03-22T01:35:16.745330+0000","last_became_active":"2024-03-22T01:34:19.667959+0000","last_became_peered":"2024-03-22T01:34:19.667959+0000","last_unstale":"2024-03-22T01:35:16.745330+0000","last_undegraded":"2024-03-22T01:35:16.745330+0000","last_fullsized":"2024-03-22T01:35:16.745330+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:13:04.447486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"93'5","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.454337+0000","last_change":"2024-03-22T01:34:15.489513+0000","last_active":"2024-03-22T01:35:45.454337+0000","last_peered":"2024-03-22T01:35:45.454337+0000","last_clean":"2024-03-22T01:35:45.454337+0000","last_became_active":"2024-03-22T01:34:15.489188+0000","last_became_peered":"2024-03-22T01:34:15.489188+0000","last_unstale":"2024-03-22T01:35:45.454337+0000","last_undegraded":"2024-03-22T01:35:45.454337+0000","last_fullsized":"2024-03-22T01:35:45.454337+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:08:24.859350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":26,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.746166+0000","last_change":"2024-03-22T01:34:14.466305+0000","last_active":"2024-03-22T01:35:16.746166+0000","last_peered":"2024-03-22T01:35:16.746166+0000","last_clean":"2024-03-22T01:35:16.746166+0000","last_became_active":"2024-03-22T01:34:14.465495+0000","last_became_peered":"2024-03-22T01:34:14.465495+0000","last_unstale":"2024-03-22T01:35:16.746166+0000","last_undegraded":"2024-03-22T01:35:16.746166+0000","last_fullsized":"2024-03-22T01:35:16.746166+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:45:13.481187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.746455+0000","last_change":"2024-03-22T01:34:19.526970+0000","last_active":"2024-03-22T01:35:16.746455+0000","last_peered":"2024-03-22T01:35:16.746455+0000","last_clean":"2024-03-22T01:35:16.746455+0000","last_became_active":"2024-03-22T01:34:19.526462+0000","last_became_peered":"2024-03-22T01:34:19.526462+0000","last_unstale":"2024-03-22T01:35:16.746455+0000","last_undegraded":"2024-03-22T01:35:16.746455+0000","last_fullsized":"2024-03-22T01:35:16.746455+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:04:18.417358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.752065+0000","last_change":"2024-03-22T01:35:16.750746+0000","last_active":"2024-03-22T01:35:16.752065+0000","last_peered":"2024-03-22T01:35:16.752065+0000","last_clean":"2024-03-22T01:35:16.752065+0000","last_became_active":"2024-03-22T01:35:16.750271+0000","last_became_peered":"2024-03-22T01:35:16.750271+0000","last_unstale":"2024-03-22T01:35:16.752065+0000","last_undegraded":"2024-03-22T01:35:16.752065+0000","last_fullsized":"2024-03-22T01:35:16.752065+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:18:10.839351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"93'9","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.459554+0000","last_change":"2024-03-22T01:34:15.490405+0000","last_active":"2024-03-22T01:35:45.459554+0000","last_peered":"2024-03-22T01:35:45.459554+0000","last_clean":"2024-03-22T01:35:45.459554+0000","last_became_active":"2024-03-22T01:34:15.489921+0000","last_became_peered":"2024-03-22T01:34:15.489921+0000","last_unstale":"2024-03-22T01:35:45.459554+0000","last_undegraded":"2024-03-22T01:35:45.459554+0000","last_fullsized":"2024-03-22T01:35:45.459554+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:27:28.804649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:17.204305+0000","last_change":"2024-03-22T01:34:19.667651+0000","last_active":"2024-03-22T01:35:17.204305+0000","last_peered":"2024-03-22T01:35:17.204305+0000","last_clean":"2024-03-22T01:35:17.204305+0000","last_became_active":"2024-03-22T01:34:19.667197+0000","last_became_peered":"2024-03-22T01:34:19.667197+0000","last_unstale":"2024-03-22T01:35:17.204305+0000","last_undegraded":"2024-03-22T01:35:17.204305+0000","last_fullsized":"2024-03-22T01:35:17.204305+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:32:55.874098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"93'7","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.748086+0000","last_change":"2024-03-22T01:34:14.474988+0000","last_active":"2024-03-22T01:35:16.748086+0000","last_peered":"2024-03-22T01:35:16.748086+0000","last_clean":"2024-03-22T01:35:16.748086+0000","last_became_active":"2024-03-22T01:34:14.474651+0000","last_became_peered":"2024-03-22T01:34:14.474651+0000","last_unstale":"2024-03-22T01:35:16.748086+0000","last_undegraded":"2024-03-22T01:35:16.748086+0000","last_fullsized":"2024-03-22T01:35:16.748086+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:14:59.846727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"93'5","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.450371+0000","last_change":"2024-03-22T01:35:16.749643+0000","last_active":"2024-03-22T01:35:45.450371+0000","last_peered":"2024-03-22T01:35:45.450371+0000","last_clean":"2024-03-22T01:35:45.450371+0000","last_became_active":"2024-03-22T01:35:16.749311+0000","last_became_peered":"2024-03-22T01:35:16.749311+0000","last_unstale":"2024-03-22T01:35:45.450371+0000","last_undegraded":"2024-03-22T01:35:45.450371+0000","last_fullsized":"2024-03-22T01:35:45.450371+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:23:18.666007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,7],"acting":[9,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.752355+0000","last_change":"2024-03-22T01:34:19.523959+0000","last_active":"2024-03-22T01:35:16.752355+0000","last_peered":"2024-03-22T01:35:16.752355+0000","last_clean":"2024-03-22T01:35:16.752355+0000","last_became_active":"2024-03-22T01:34:19.523556+0000","last_became_peered":"2024-03-22T01:34:19.523556+0000","last_unstale":"2024-03-22T01:35:16.752355+0000","last_undegraded":"2024-03-22T01:35:16.752355+0000","last_fullsized":"2024-03-22T01:35:16.752355+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:22:21.623042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"86'2","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.744215+0000","last_change":"2024-03-22T01:34:14.463434+0000","last_active":"2024-03-22T01:35:16.744215+0000","last_peered":"2024-03-22T01:35:16.744215+0000","last_clean":"2024-03-22T01:35:16.744215+0000","last_became_active":"2024-03-22T01:34:14.463075+0000","last_became_peered":"2024-03-22T01:34:14.463075+0000","last_unstale":"2024-03-22T01:35:16.744215+0000","last_undegraded":"2024-03-22T01:35:16.744215+0000","last_fullsized":"2024-03-22T01:35:16.744215+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:43:18.223766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"93'7","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.452320+0000","last_change":"2024-03-22T01:34:15.482695+0000","last_active":"2024-03-22T01:35:45.452320+0000","last_peered":"2024-03-22T01:35:45.452320+0000","last_clean":"2024-03-22T01:35:45.452320+0000","last_became_active":"2024-03-22T01:34:15.481881+0000","last_became_peered":"2024-03-22T01:34:15.481881+0000","last_unstale":"2024-03-22T01:35:45.452320+0000","last_undegraded":"2024-03-22T01:35:45.452320+0000","last_fullsized":"2024-03-22T01:35:45.452320+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:44:26.290714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.744060+0000","last_change":"2024-03-22T01:34:19.668252+0000","last_active":"2024-03-22T01:35:16.744060+0000","last_peered":"2024-03-22T01:35:16.744060+0000","last_clean":"2024-03-22T01:35:16.744060+0000","last_became_active":"2024-03-22T01:34:19.667830+0000","last_became_peered":"2024-03-22T01:34:19.667830+0000","last_unstale":"2024-03-22T01:35:16.744060+0000","last_undegraded":"2024-03-22T01:35:16.744060+0000","last_fullsized":"2024-03-22T01:35:16.744060+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:08:29.602899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.746747+0000","last_change":"2024-03-22T01:34:14.473020+0000","last_active":"2024-03-22T01:35:16.746747+0000","last_peered":"2024-03-22T01:35:16.746747+0000","last_clean":"2024-03-22T01:35:16.746747+0000","last_became_active":"2024-03-22T01:34:14.472689+0000","last_became_peered":"2024-03-22T01:34:14.472689+0000","last_unstale":"2024-03-22T01:35:16.746747+0000","last_undegraded":"2024-03-22T01:35:16.746747+0000","last_fullsized":"2024-03-22T01:35:16.746747+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:04:19.724685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"93'2","reported_seq":26,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.416669+0000","last_change":"2024-03-22T01:34:15.491613+0000","last_active":"2024-03-22T01:35:45.416669+0000","last_peered":"2024-03-22T01:35:45.416669+0000","last_clean":"2024-03-22T01:35:45.416669+0000","last_became_active":"2024-03-22T01:34:15.490922+0000","last_became_peered":"2024-03-22T01:34:15.490922+0000","last_unstale":"2024-03-22T01:35:45.416669+0000","last_undegraded":"2024-03-22T01:35:45.416669+0000","last_fullsized":"2024-03-22T01:35:45.416669+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:09:33.177646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.748082+0000","last_change":"2024-03-22T01:34:19.534537+0000","last_active":"2024-03-22T01:35:16.748082+0000","last_peered":"2024-03-22T01:35:16.748082+0000","last_clean":"2024-03-22T01:35:16.748082+0000","last_became_active":"2024-03-22T01:34:19.534038+0000","last_became_peered":"2024-03-22T01:34:19.534038+0000","last_unstale":"2024-03-22T01:35:16.748082+0000","last_undegraded":"2024-03-22T01:35:16.748082+0000","last_fullsized":"2024-03-22T01:35:16.748082+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:59:26.020000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":26,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:17.204413+0000","last_change":"2024-03-22T01:34:14.464724+0000","last_active":"2024-03-22T01:35:17.204413+0000","last_peered":"2024-03-22T01:35:17.204413+0000","last_clean":"2024-03-22T01:35:17.204413+0000","last_became_active":"2024-03-22T01:34:14.464056+0000","last_became_peered":"2024-03-22T01:34:14.464056+0000","last_unstale":"2024-03-22T01:35:17.204413+0000","last_undegraded":"2024-03-22T01:35:17.204413+0000","last_fullsized":"2024-03-22T01:35:17.204413+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:49:39.098126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"93'9","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.457460+0000","last_change":"2024-03-22T01:34:15.488934+0000","last_active":"2024-03-22T01:35:45.457460+0000","last_peered":"2024-03-22T01:35:45.457460+0000","last_clean":"2024-03-22T01:35:45.457460+0000","last_became_active":"2024-03-22T01:34:15.488585+0000","last_became_peered":"2024-03-22T01:34:15.488585+0000","last_unstale":"2024-03-22T01:35:45.457460+0000","last_undegraded":"2024-03-22T01:35:45.457460+0000","last_fullsized":"2024-03-22T01:35:45.457460+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:59:04.554614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"91'1","reported_seq":19,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.743985+0000","last_change":"2024-03-22T01:34:19.534096+0000","last_active":"2024-03-22T01:35:16.743985+0000","last_peered":"2024-03-22T01:35:16.743985+0000","last_clean":"2024-03-22T01:35:16.743985+0000","last_became_active":"2024-03-22T01:34:19.533645+0000","last_became_peered":"2024-03-22T01:34:19.533645+0000","last_unstale":"2024-03-22T01:35:16.743985+0000","last_undegraded":"2024-03-22T01:35:16.743985+0000","last_fullsized":"2024-03-22T01:35:16.743985+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:50:21.259146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.752618+0000","last_change":"2024-03-22T01:34:14.469529+0000","last_active":"2024-03-22T01:35:16.752618+0000","last_peered":"2024-03-22T01:35:16.752618+0000","last_clean":"2024-03-22T01:35:16.752618+0000","last_became_active":"2024-03-22T01:34:14.469204+0000","last_became_peered":"2024-03-22T01:34:14.469204+0000","last_unstale":"2024-03-22T01:35:16.752618+0000","last_undegraded":"2024-03-22T01:35:16.752618+0000","last_fullsized":"2024-03-22T01:35:16.752618+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:39:57.048537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"93'8","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.454022+0000","last_change":"2024-03-22T01:34:15.501324+0000","last_active":"2024-03-22T01:35:45.454022+0000","last_peered":"2024-03-22T01:35:45.454022+0000","last_clean":"2024-03-22T01:35:45.454022+0000","last_became_active":"2024-03-22T01:34:15.501015+0000","last_became_peered":"2024-03-22T01:34:15.501015+0000","last_unstale":"2024-03-22T01:35:45.454022+0000","last_undegraded":"2024-03-22T01:35:45.454022+0000","last_fullsized":"2024-03-22T01:35:45.454022+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:35:27.673019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.743977+0000","last_change":"2024-03-22T01:34:19.667806+0000","last_active":"2024-03-22T01:35:16.743977+0000","last_peered":"2024-03-22T01:35:16.743977+0000","last_clean":"2024-03-22T01:35:16.743977+0000","last_became_active":"2024-03-22T01:34:19.667249+0000","last_became_peered":"2024-03-22T01:34:19.667249+0000","last_unstale":"2024-03-22T01:35:16.743977+0000","last_undegraded":"2024-03-22T01:35:16.743977+0000","last_fullsized":"2024-03-22T01:35:16.743977+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:44:22.476882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"86'2","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.748466+0000","last_change":"2024-03-22T01:34:14.471800+0000","last_active":"2024-03-22T01:35:16.748466+0000","last_peered":"2024-03-22T01:35:16.748466+0000","last_clean":"2024-03-22T01:35:16.748466+0000","last_became_active":"2024-03-22T01:34:14.471422+0000","last_became_peered":"2024-03-22T01:34:14.471422+0000","last_unstale":"2024-03-22T01:35:16.748466+0000","last_undegraded":"2024-03-22T01:35:16.748466+0000","last_fullsized":"2024-03-22T01:35:16.748466+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:25:58.546331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"93'5","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.452063+0000","last_change":"2024-03-22T01:34:15.487348+0000","last_active":"2024-03-22T01:35:45.452063+0000","last_peered":"2024-03-22T01:35:45.452063+0000","last_clean":"2024-03-22T01:35:45.452063+0000","last_became_active":"2024-03-22T01:34:15.486788+0000","last_became_peered":"2024-03-22T01:34:15.486788+0000","last_unstale":"2024-03-22T01:35:45.452063+0000","last_undegraded":"2024-03-22T01:35:45.452063+0000","last_fullsized":"2024-03-22T01:35:45.452063+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:10:31.489021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.746815+0000","last_change":"2024-03-22T01:34:19.521490+0000","last_active":"2024-03-22T01:35:16.746815+0000","last_peered":"2024-03-22T01:35:16.746815+0000","last_clean":"2024-03-22T01:35:16.746815+0000","last_became_active":"2024-03-22T01:34:19.521047+0000","last_became_peered":"2024-03-22T01:34:19.521047+0000","last_unstale":"2024-03-22T01:35:16.746815+0000","last_undegraded":"2024-03-22T01:35:16.746815+0000","last_fullsized":"2024-03-22T01:35:16.746815+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:36:50.176756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":11,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.750181+0000","last_change":"2024-03-22T01:35:16.749731+0000","last_active":"2024-03-22T01:35:16.750181+0000","last_peered":"2024-03-22T01:35:16.750181+0000","last_clean":"2024-03-22T01:35:16.750181+0000","last_became_active":"2024-03-22T01:35:16.749329+0000","last_became_peered":"2024-03-22T01:35:16.749329+0000","last_unstale":"2024-03-22T01:35:16.750181+0000","last_undegraded":"2024-03-22T01:35:16.750181+0000","last_fullsized":"2024-03-22T01:35:16.750181+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:23:27.386061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"93'7","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.451333+0000","last_change":"2024-03-22T01:34:15.491683+0000","last_active":"2024-03-22T01:35:45.451333+0000","last_peered":"2024-03-22T01:35:45.451333+0000","last_clean":"2024-03-22T01:35:45.451333+0000","last_became_active":"2024-03-22T01:34:15.491030+0000","last_became_peered":"2024-03-22T01:34:15.491030+0000","last_unstale":"2024-03-22T01:35:45.451333+0000","last_undegraded":"2024-03-22T01:35:45.451333+0000","last_fullsized":"2024-03-22T01:35:45.451333+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:02:10.115971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.745806+0000","last_change":"2024-03-22T01:34:19.528608+0000","last_active":"2024-03-22T01:35:16.745806+0000","last_peered":"2024-03-22T01:35:16.745806+0000","last_clean":"2024-03-22T01:35:16.745806+0000","last_became_active":"2024-03-22T01:34:19.528173+0000","last_became_peered":"2024-03-22T01:34:19.528173+0000","last_unstale":"2024-03-22T01:35:16.745806+0000","last_undegraded":"2024-03-22T01:35:16.745806+0000","last_fullsized":"2024-03-22T01:35:16.745806+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:36:02.193187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"86'1","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.744982+0000","last_change":"2024-03-22T01:34:14.467755+0000","last_active":"2024-03-22T01:35:16.744982+0000","last_peered":"2024-03-22T01:35:16.744982+0000","last_clean":"2024-03-22T01:35:16.744982+0000","last_became_active":"2024-03-22T01:34:14.467426+0000","last_became_peered":"2024-03-22T01:34:14.467426+0000","last_unstale":"2024-03-22T01:35:16.744982+0000","last_undegraded":"2024-03-22T01:35:16.744982+0000","last_fullsized":"2024-03-22T01:35:16.744982+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:16:35.436872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"93'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.454313+0000","last_change":"2024-03-22T01:35:16.748667+0000","last_active":"2024-03-22T01:35:45.454313+0000","last_peered":"2024-03-22T01:35:45.454313+0000","last_clean":"2024-03-22T01:35:45.454313+0000","last_became_active":"2024-03-22T01:35:16.748303+0000","last_became_peered":"2024-03-22T01:35:16.748303+0000","last_unstale":"2024-03-22T01:35:45.454313+0000","last_undegraded":"2024-03-22T01:35:45.454313+0000","last_fullsized":"2024-03-22T01:35:45.454313+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:40:55.207127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.744131+0000","last_change":"2024-03-22T01:34:19.523631+0000","last_active":"2024-03-22T01:35:16.744131+0000","last_peered":"2024-03-22T01:35:16.744131+0000","last_clean":"2024-03-22T01:35:16.744131+0000","last_became_active":"2024-03-22T01:34:19.522968+0000","last_became_peered":"2024-03-22T01:34:19.522968+0000","last_unstale":"2024-03-22T01:35:16.744131+0000","last_undegraded":"2024-03-22T01:35:16.744131+0000","last_fullsized":"2024-03-22T01:35:16.744131+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:27:24.182886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.743726+0000","last_change":"2024-03-22T01:34:14.464113+0000","last_active":"2024-03-22T01:35:16.743726+0000","last_peered":"2024-03-22T01:35:16.743726+0000","last_clean":"2024-03-22T01:35:16.743726+0000","last_became_active":"2024-03-22T01:34:14.463389+0000","last_became_peered":"2024-03-22T01:34:14.463389+0000","last_unstale":"2024-03-22T01:35:16.743726+0000","last_undegraded":"2024-03-22T01:35:16.743726+0000","last_fullsized":"2024-03-22T01:35:16.743726+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:08:42.376753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"93'9","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.454913+0000","last_change":"2024-03-22T01:34:15.492941+0000","last_active":"2024-03-22T01:35:45.454913+0000","last_peered":"2024-03-22T01:35:45.454913+0000","last_clean":"2024-03-22T01:35:45.454913+0000","last_became_active":"2024-03-22T01:34:15.492598+0000","last_became_peered":"2024-03-22T01:34:15.492598+0000","last_unstale":"2024-03-22T01:35:45.454913+0000","last_undegraded":"2024-03-22T01:35:45.454913+0000","last_fullsized":"2024-03-22T01:35:45.454913+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:22:35.502404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"93'9","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.454431+0000","last_change":"2024-03-22T01:34:15.494548+0000","last_active":"2024-03-22T01:35:45.454431+0000","last_peered":"2024-03-22T01:35:45.454431+0000","last_clean":"2024-03-22T01:35:45.454431+0000","last_became_active":"2024-03-22T01:34:15.494228+0000","last_became_peered":"2024-03-22T01:34:15.494228+0000","last_unstale":"2024-03-22T01:35:45.454431+0000","last_undegraded":"2024-03-22T01:35:45.454431+0000","last_fullsized":"2024-03-22T01:35:45.454431+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:43:26.011030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"93'12","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.452055+0000","last_change":"2024-03-22T01:34:15.490843+0000","last_active":"2024-03-22T01:35:45.452055+0000","last_peered":"2024-03-22T01:35:45.452055+0000","last_clean":"2024-03-22T01:35:45.452055+0000","last_became_active":"2024-03-22T01:34:15.490474+0000","last_became_peered":"2024-03-22T01:34:15.490474+0000","last_unstale":"2024-03-22T01:35:45.452055+0000","last_undegraded":"2024-03-22T01:35:45.452055+0000","last_fullsized":"2024-03-22T01:35:45.452055+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:11:31.488132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"93'8","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.447966+0000","last_change":"2024-03-22T01:34:15.502213+0000","last_active":"2024-03-22T01:35:45.447966+0000","last_peered":"2024-03-22T01:35:45.447966+0000","last_clean":"2024-03-22T01:35:45.447966+0000","last_became_active":"2024-03-22T01:34:15.501464+0000","last_became_peered":"2024-03-22T01:34:15.501464+0000","last_unstale":"2024-03-22T01:35:45.447966+0000","last_undegraded":"2024-03-22T01:35:45.447966+0000","last_fullsized":"2024-03-22T01:35:45.447966+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:38:02.090365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"93'7","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.455771+0000","last_change":"2024-03-22T01:34:15.481282+0000","last_active":"2024-03-22T01:35:45.455771+0000","last_peered":"2024-03-22T01:35:45.455771+0000","last_clean":"2024-03-22T01:35:45.455771+0000","last_became_active":"2024-03-22T01:34:15.480465+0000","last_became_peered":"2024-03-22T01:34:15.480465+0000","last_unstale":"2024-03-22T01:35:45.455771+0000","last_undegraded":"2024-03-22T01:35:45.455771+0000","last_fullsized":"2024-03-22T01:35:45.455771+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:55:59.398543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"93'4","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.457095+0000","last_change":"2024-03-22T01:34:15.480910+0000","last_active":"2024-03-22T01:35:45.457095+0000","last_peered":"2024-03-22T01:35:45.457095+0000","last_clean":"2024-03-22T01:35:45.457095+0000","last_became_active":"2024-03-22T01:34:15.480601+0000","last_became_peered":"2024-03-22T01:34:15.480601+0000","last_unstale":"2024-03-22T01:35:45.457095+0000","last_undegraded":"2024-03-22T01:35:45.457095+0000","last_fullsized":"2024-03-22T01:35:45.457095+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:19:51.563598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"93'6","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.453907+0000","last_change":"2024-03-22T01:34:15.490859+0000","last_active":"2024-03-22T01:35:45.453907+0000","last_peered":"2024-03-22T01:35:45.453907+0000","last_clean":"2024-03-22T01:35:45.453907+0000","last_became_active":"2024-03-22T01:34:15.490337+0000","last_became_peered":"2024-03-22T01:34:15.490337+0000","last_unstale":"2024-03-22T01:35:45.453907+0000","last_undegraded":"2024-03-22T01:35:45.453907+0000","last_fullsized":"2024-03-22T01:35:45.453907+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:46:07.072015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"93'15","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.446635+0000","last_change":"2024-03-22T01:34:15.495531+0000","last_active":"2024-03-22T01:35:45.446635+0000","last_peered":"2024-03-22T01:35:45.446635+0000","last_clean":"2024-03-22T01:35:45.446635+0000","last_became_active":"2024-03-22T01:34:15.495216+0000","last_became_peered":"2024-03-22T01:34:15.495216+0000","last_unstale":"2024-03-22T01:35:45.446635+0000","last_undegraded":"2024-03-22T01:35:45.446635+0000","last_fullsized":"2024-03-22T01:35:45.446635+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:39:22.575926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"93'2","reported_seq":26,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.447150+0000","last_change":"2024-03-22T01:34:15.499430+0000","last_active":"2024-03-22T01:35:45.447150+0000","last_peered":"2024-03-22T01:35:45.447150+0000","last_clean":"2024-03-22T01:35:45.447150+0000","last_became_active":"2024-03-22T01:34:15.499114+0000","last_became_peered":"2024-03-22T01:34:15.499114+0000","last_unstale":"2024-03-22T01:35:45.447150+0000","last_undegraded":"2024-03-22T01:35:45.447150+0000","last_fullsized":"2024-03-22T01:35:45.447150+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:23:01.781963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"93'6","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.459467+0000","last_change":"2024-03-22T01:34:15.485449+0000","last_active":"2024-03-22T01:35:45.459467+0000","last_peered":"2024-03-22T01:35:45.459467+0000","last_clean":"2024-03-22T01:35:45.459467+0000","last_became_active":"2024-03-22T01:34:15.485084+0000","last_became_peered":"2024-03-22T01:34:15.485084+0000","last_unstale":"2024-03-22T01:35:45.459467+0000","last_undegraded":"2024-03-22T01:35:45.459467+0000","last_fullsized":"2024-03-22T01:35:45.459467+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:53:25.632719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"93'8","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.450638+0000","last_change":"2024-03-22T01:34:15.495521+0000","last_active":"2024-03-22T01:35:45.450638+0000","last_peered":"2024-03-22T01:35:45.450638+0000","last_clean":"2024-03-22T01:35:45.450638+0000","last_became_active":"2024-03-22T01:34:15.495218+0000","last_became_peered":"2024-03-22T01:34:15.495218+0000","last_unstale":"2024-03-22T01:35:45.450638+0000","last_undegraded":"2024-03-22T01:35:45.450638+0000","last_fullsized":"2024-03-22T01:35:45.450638+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:04:32.569934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"93'10","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.452812+0000","last_change":"2024-03-22T01:34:15.489213+0000","last_active":"2024-03-22T01:35:45.452812+0000","last_peered":"2024-03-22T01:35:45.452812+0000","last_clean":"2024-03-22T01:35:45.452812+0000","last_became_active":"2024-03-22T01:34:15.488887+0000","last_became_peered":"2024-03-22T01:34:15.488887+0000","last_unstale":"2024-03-22T01:35:45.452812+0000","last_undegraded":"2024-03-22T01:35:45.452812+0000","last_fullsized":"2024-03-22T01:35:45.452812+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:53:39.983144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"93'11","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.457870+0000","last_change":"2024-03-22T01:34:15.485632+0000","last_active":"2024-03-22T01:35:45.457870+0000","last_peered":"2024-03-22T01:35:45.457870+0000","last_clean":"2024-03-22T01:35:45.457870+0000","last_became_active":"2024-03-22T01:34:15.485341+0000","last_became_peered":"2024-03-22T01:34:15.485341+0000","last_unstale":"2024-03-22T01:35:45.457870+0000","last_undegraded":"2024-03-22T01:35:45.457870+0000","last_fullsized":"2024-03-22T01:35:45.457870+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:33:00.248555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"93'8","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.451910+0000","last_change":"2024-03-22T01:34:15.483593+0000","last_active":"2024-03-22T01:35:45.451910+0000","last_peered":"2024-03-22T01:35:45.451910+0000","last_clean":"2024-03-22T01:35:45.451910+0000","last_became_active":"2024-03-22T01:34:15.483201+0000","last_became_peered":"2024-03-22T01:34:15.483201+0000","last_unstale":"2024-03-22T01:35:45.451910+0000","last_undegraded":"2024-03-22T01:35:45.451910+0000","last_fullsized":"2024-03-22T01:35:45.451910+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:03:10.955120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":64403,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":156,"num_read_kb":143,"num_write":84,"num_write_kb":93,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":193209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318305,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1040384,"data_stored":509947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579942,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37012,"kb_used_data":1676,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708140,"statfs":{"total":95995035648,"available":95957135360,"internally_reserved":0,"allocated":1716224,"data_stored":1160018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776172,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1048576,"data_stored":514625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037808,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1048576,"data_stored":510551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299446,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28620,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716532,"statfs":{"total":95995035648,"available":95965728768,"internally_reserved":0,"allocated":1642496,"data_stored":1104993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462970,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":505851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574930,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1110016,"data_stored":569456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738455,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1040384,"data_stored":505941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411406,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28104,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1114112,"data_stored":569490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149769,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1044480,"data_stored":505955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888133,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1044480,"data_stored":509969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724607,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36944,"kb_used_data":1608,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708208,"statfs":{"total":95995035648,"available":95957204992,"internally_reserved":0,"allocated":1646592,"data_stored":1100497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":63605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":63639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":63799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-22T01:35:55.437 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-22T01:35:55.683 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:35:55.806 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:55 smithi072 ceph-mon[32414]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-22T01:35:55.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:55 smithi089 ceph-mon[35770]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-22T01:35:55.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:55 smithi098 ceph-mon[36468]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-22T01:35:56.816 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:56 smithi072 ceph-mon[32414]: pgmap v343: 129 pgs: 129 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-22T01:35:56.822 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:56 smithi098 ceph-mon[36468]: pgmap v343: 129 pgs: 129 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-22T01:35:56.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:56 smithi089 ceph-mon[35770]: pgmap v343: 129 pgs: 129 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-22T01:35:57.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:57.356+0000 7f1bbcc62700 1 -- 172.21.15.72:0/3997021752 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1bb8103090 msgr2=0x7f1bb810f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:57.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:57.356+0000 7f1bbcc62700 1 --2- 172.21.15.72:0/3997021752 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1bb8103090 0x7f1bb810f4d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f1bb000b0d0 tx=0x7f1bb000b3e0 comp rx=0 tx=0).stop 2024-03-22T01:35:57.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:57.356+0000 7f1bbcc62700 1 -- 172.21.15.72:0/3997021752 shutdown_connections 2024-03-22T01:35:57.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:57.356+0000 7f1bbcc62700 1 --2- 172.21.15.72:0/3997021752 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1bb8103090 0x7f1bb810f4d0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:57.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:57.356+0000 7f1bbcc62700 1 --2- 172.21.15.72:0/3997021752 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1bb81074f0 0x7f1bb81078d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:57.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:57.356+0000 7f1bbcc62700 1 --2- 172.21.15.72:0/3997021752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bb81026d0 0x7f1bb8102b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:57.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:57.356+0000 7f1bbcc62700 1 -- 172.21.15.72:0/3997021752 >> 172.21.15.72:0/3997021752 conn(0x7f1bb80797b0 msgr2=0x7f1bb8079bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:35:57.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:57.356+0000 7f1bbcc62700 1 -- 172.21.15.72:0/3997021752 shutdown_connections 2024-03-22T01:35:57.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:57.356+0000 7f1bbcc62700 1 -- 172.21.15.72:0/3997021752 wait complete. 2024-03-22T01:35:57.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:57.357+0000 7f1bbcc62700 1 Processor -- start 2024-03-22T01:35:57.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:57.357+0000 7f1bbcc62700 1 -- start start 2024-03-22T01:35:57.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:57.357+0000 7f1bbcc62700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1bb81026d0 0x7f1bb8198e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:57.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:57.357+0000 7f1bbcc62700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1bb8103090 0x7f1bb8199340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:57.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:57.357+0000 7f1bbcc62700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bb81074f0 0x7f1bb819d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:57.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:57.357+0000 7f1bbcc62700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb81129b0 con 0x7f1bb8103090 2024-03-22T01:35:57.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:57.357+0000 7f1bbcc62700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb8112830 con 0x7f1bb81026d0 2024-03-22T01:35:57.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:57.357+0000 7f1bbcc62700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb8112b30 con 0x7f1bb81074f0 2024-03-22T01:35:57.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:57.357+0000 7f1baffff700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1bb8103090 0x7f1bb8199340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:57.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:57.357+0000 7f1bb659c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1bb81026d0 0x7f1bb8198e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:57.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:57.357+0000 7f1baffff700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1bb8103090 0x7f1bb8199340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57592/0 (socket says 172.21.15.72:57592) 2024-03-22T01:35:57.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:57.357+0000 7f1baffff700 1 -- 172.21.15.72:0/3116597802 learned_addr learned my addr 172.21.15.72:0/3116597802 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:35:57.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:57.357+0000 7f1baffff700 1 -- 172.21.15.72:0/3116597802 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bb81074f0 msgr2=0x7f1bb819d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:57.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:57.357+0000 7f1baffff700 1 --2- 172.21.15.72:0/3116597802 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bb81074f0 0x7f1bb819d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:57.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:57.357+0000 7f1baffff700 1 -- 172.21.15.72:0/3116597802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1bb81026d0 msgr2=0x7f1bb8198e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:57.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:57.357+0000 7f1baffff700 1 --2- 172.21.15.72:0/3116597802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1bb81026d0 0x7f1bb8198e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:57.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:57.357+0000 7f1baffff700 1 -- 172.21.15.72:0/3116597802 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bb819de10 con 0x7f1bb8103090 2024-03-22T01:35:57.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:57.357+0000 7f1bb659c700 1 --2- 172.21.15.72:0/3116597802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1bb81026d0 0x7f1bb8198e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:35:57.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:57.358+0000 7f1baffff700 1 --2- 172.21.15.72:0/3116597802 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1bb8103090 0x7f1bb8199340 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f1ba000b530 tx=0x7f1ba000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:57.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:57.358+0000 7f1baf7fe700 1 -- 172.21.15.72:0/3116597802 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba0015020 con 0x7f1bb8103090 2024-03-22T01:35:57.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:57.358+0000 7f1bbcc62700 1 -- 172.21.15.72:0/3116597802 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bb819e100 con 0x7f1bb8103090 2024-03-22T01:35:57.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:57.358+0000 7f1baf7fe700 1 -- 172.21.15.72:0/3116597802 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ba0011d10 con 0x7f1bb8103090 2024-03-22T01:35:57.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:57.358+0000 7f1baf7fe700 1 -- 172.21.15.72:0/3116597802 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba000f2e0 con 0x7f1bb8103090 2024-03-22T01:35:57.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:57.358+0000 7f1bbcc62700 1 -- 172.21.15.72:0/3116597802 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bb81a5980 con 0x7f1bb8103090 2024-03-22T01:35:57.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:57.359+0000 7f1bbcc62700 1 -- 172.21.15.72:0/3116597802 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bb8069340 con 0x7f1bb8103090 2024-03-22T01:35:57.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:57.360+0000 7f1baf7fe700 1 -- 172.21.15.72:0/3116597802 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7f1ba0018020 con 0x7f1bb8103090 2024-03-22T01:35:57.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:57.360+0000 7f1baf7fe700 1 --2- 172.21.15.72:0/3116597802 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f1ba40a2fb0 0x7f1ba40a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:35:57.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:57.360+0000 7f1bb659c700 1 --2- 172.21.15.72:0/3116597802 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f1ba40a2fb0 0x7f1ba40a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:35:57.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:57.360+0000 7f1baf7fe700 1 -- 172.21.15.72:0/3116597802 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f1ba00ceff0 con 0x7f1bb8103090 2024-03-22T01:35:57.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:57.361+0000 7f1bb659c700 1 --2- 172.21.15.72:0/3116597802 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f1ba40a2fb0 0x7f1ba40a5470 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1ba8011490 tx=0x7f1ba8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:35:57.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:57.365+0000 7f1baf7fe700 1 -- 172.21.15.72:0/3116597802 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1ba001f020 con 0x7f1bb8103090 2024-03-22T01:35:57.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:57.555+0000 7f1bbcc62700 1 -- 172.21.15.72:0/3116597802 --> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1bb8062ac0 con 0x7f1ba40a2fb0 2024-03-22T01:35:57.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:57.565+0000 7f1baf7fe700 1 -- 172.21.15.72:0/3116597802 <== mgr.14152 v2:172.21.15.72:6800/401403200 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340394 (secure 0 0 0) 0x7f1bb8062ac0 con 0x7f1ba40a2fb0 2024-03-22T01:35:57.568 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:35:57.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:57.569+0000 7f1bbcc62700 1 -- 172.21.15.72:0/3116597802 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f1ba40a2fb0 msgr2=0x7f1ba40a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:57.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:57.569+0000 7f1bbcc62700 1 --2- 172.21.15.72:0/3116597802 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f1ba40a2fb0 0x7f1ba40a5470 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1ba8011490 tx=0x7f1ba8009260 comp rx=0 tx=0).stop 2024-03-22T01:35:57.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:57.569+0000 7f1bbcc62700 1 -- 172.21.15.72:0/3116597802 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1bb8103090 msgr2=0x7f1bb8199340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:35:57.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:57.569+0000 7f1bbcc62700 1 --2- 172.21.15.72:0/3116597802 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1bb8103090 0x7f1bb8199340 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f1ba000b530 tx=0x7f1ba000b840 comp rx=0 tx=0).stop 2024-03-22T01:35:57.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:57.569+0000 7f1bbcc62700 1 -- 172.21.15.72:0/3116597802 shutdown_connections 2024-03-22T01:35:57.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:57.569+0000 7f1bbcc62700 1 --2- 172.21.15.72:0/3116597802 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f1ba40a2fb0 0x7f1ba40a5470 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:57.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:57.569+0000 7f1bbcc62700 1 --2- 172.21.15.72:0/3116597802 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bb81074f0 0x7f1bb819d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:57.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:57.569+0000 7f1bbcc62700 1 --2- 172.21.15.72:0/3116597802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1bb81026d0 0x7f1bb8198e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:57.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:57.569+0000 7f1bbcc62700 1 --2- 172.21.15.72:0/3116597802 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1bb8103090 0x7f1bb8199340 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:35:57.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:57.569+0000 7f1bbcc62700 1 -- 172.21.15.72:0/3116597802 >> 172.21.15.72:0/3116597802 conn(0x7f1bb80797b0 msgr2=0x7f1bb8100780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:35:57.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:57.569+0000 7f1bbcc62700 1 -- 172.21.15.72:0/3116597802 shutdown_connections 2024-03-22T01:35:57.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:35:57.569+0000 7f1bbcc62700 1 -- 172.21.15.72:0/3116597802 wait complete. 2024-03-22T01:35:57.572 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-03-22T01:35:58.402 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":344,"stamp":"2024-03-22T01:35:57.449289+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":655005,"num_objects":26,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":302,"num_read_kb":267,"num_write":267,"num_write_kb":3033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":711,"ondisk_log_size":711,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":371664,"kb_used_data":14192,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124570160,"statfs":{"total":1151940427776,"available":1151559843840,"internally_reserved":0,"allocated":14532608,"data_stored":8067293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":365933882},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":249,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":11,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003498"},"pg_stats":[{"pgid":"3.3f","version":"93'8","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.465184+0000","last_change":"2024-03-22T01:34:15.487496+0000","last_active":"2024-03-22T01:35:45.465184+0000","last_peered":"2024-03-22T01:35:45.465184+0000","last_clean":"2024-03-22T01:35:45.465184+0000","last_became_active":"2024-03-22T01:34:15.486700+0000","last_became_peered":"2024-03-22T01:34:15.486700+0000","last_unstale":"2024-03-22T01:35:45.465184+0000","last_undegraded":"2024-03-22T01:35:45.465184+0000","last_fullsized":"2024-03-22T01:35:45.465184+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:03:19.146811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"93'4","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.455660+0000","last_change":"2024-03-22T01:34:15.489791+0000","last_active":"2024-03-22T01:35:45.455660+0000","last_peered":"2024-03-22T01:35:45.455660+0000","last_clean":"2024-03-22T01:35:45.455660+0000","last_became_active":"2024-03-22T01:34:15.489493+0000","last_became_peered":"2024-03-22T01:34:15.489493+0000","last_unstale":"2024-03-22T01:35:45.455660+0000","last_undegraded":"2024-03-22T01:35:45.455660+0000","last_fullsized":"2024-03-22T01:35:45.455660+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:45:05.060454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"93'7","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.458273+0000","last_change":"2024-03-22T01:34:15.491567+0000","last_active":"2024-03-22T01:35:45.458273+0000","last_peered":"2024-03-22T01:35:45.458273+0000","last_clean":"2024-03-22T01:35:45.458273+0000","last_became_active":"2024-03-22T01:34:15.491275+0000","last_became_peered":"2024-03-22T01:34:15.491275+0000","last_unstale":"2024-03-22T01:35:45.458273+0000","last_undegraded":"2024-03-22T01:35:45.458273+0000","last_fullsized":"2024-03-22T01:35:45.458273+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:26:36.630811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"93'8","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.449730+0000","last_change":"2024-03-22T01:34:15.488579+0000","last_active":"2024-03-22T01:35:45.449730+0000","last_peered":"2024-03-22T01:35:45.449730+0000","last_clean":"2024-03-22T01:35:45.449730+0000","last_became_active":"2024-03-22T01:34:15.488262+0000","last_became_peered":"2024-03-22T01:34:15.488262+0000","last_unstale":"2024-03-22T01:35:45.449730+0000","last_undegraded":"2024-03-22T01:35:45.449730+0000","last_fullsized":"2024-03-22T01:35:45.449730+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:58:21.788420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"93'4","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.432886+0000","last_change":"2024-03-22T01:34:15.485744+0000","last_active":"2024-03-22T01:35:45.432886+0000","last_peered":"2024-03-22T01:35:45.432886+0000","last_clean":"2024-03-22T01:35:45.432886+0000","last_became_active":"2024-03-22T01:34:15.485378+0000","last_became_peered":"2024-03-22T01:34:15.485378+0000","last_unstale":"2024-03-22T01:35:45.432886+0000","last_undegraded":"2024-03-22T01:35:45.432886+0000","last_fullsized":"2024-03-22T01:35:45.432886+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:06:40.979475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"93'6","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.460217+0000","last_change":"2024-03-22T01:34:15.495051+0000","last_active":"2024-03-22T01:35:45.460217+0000","last_peered":"2024-03-22T01:35:45.460217+0000","last_clean":"2024-03-22T01:35:45.460217+0000","last_became_active":"2024-03-22T01:34:15.494691+0000","last_became_peered":"2024-03-22T01:34:15.494691+0000","last_unstale":"2024-03-22T01:35:45.460217+0000","last_undegraded":"2024-03-22T01:35:45.460217+0000","last_fullsized":"2024-03-22T01:35:45.460217+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T01:44:28.336545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"93'5","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.453971+0000","last_change":"2024-03-22T01:34:15.493911+0000","last_active":"2024-03-22T01:35:45.453971+0000","last_peered":"2024-03-22T01:35:45.453971+0000","last_clean":"2024-03-22T01:35:45.453971+0000","last_became_active":"2024-03-22T01:34:15.493586+0000","last_became_peered":"2024-03-22T01:34:15.493586+0000","last_unstale":"2024-03-22T01:35:45.453971+0000","last_undegraded":"2024-03-22T01:35:45.453971+0000","last_fullsized":"2024-03-22T01:35:45.453971+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:22:41.391855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"93'9","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.455124+0000","last_change":"2024-03-22T01:34:15.481410+0000","last_active":"2024-03-22T01:35:45.455124+0000","last_peered":"2024-03-22T01:35:45.455124+0000","last_clean":"2024-03-22T01:35:45.455124+0000","last_became_active":"2024-03-22T01:34:15.481109+0000","last_became_peered":"2024-03-22T01:34:15.481109+0000","last_unstale":"2024-03-22T01:35:45.455124+0000","last_undegraded":"2024-03-22T01:35:45.455124+0000","last_fullsized":"2024-03-22T01:35:45.455124+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:27:10.004584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"93'11","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.463206+0000","last_change":"2024-03-22T01:34:15.485173+0000","last_active":"2024-03-22T01:35:45.463206+0000","last_peered":"2024-03-22T01:35:45.463206+0000","last_clean":"2024-03-22T01:35:45.463206+0000","last_became_active":"2024-03-22T01:34:15.484812+0000","last_became_peered":"2024-03-22T01:34:15.484812+0000","last_unstale":"2024-03-22T01:35:45.463206+0000","last_undegraded":"2024-03-22T01:35:45.463206+0000","last_fullsized":"2024-03-22T01:35:45.463206+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:44:06.383568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"93'16","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.450126+0000","last_change":"2024-03-22T01:34:15.497964+0000","last_active":"2024-03-22T01:35:45.450126+0000","last_peered":"2024-03-22T01:35:45.450126+0000","last_clean":"2024-03-22T01:35:45.450126+0000","last_became_active":"2024-03-22T01:34:15.496887+0000","last_became_peered":"2024-03-22T01:34:15.496887+0000","last_unstale":"2024-03-22T01:35:45.450126+0000","last_undegraded":"2024-03-22T01:35:45.450126+0000","last_fullsized":"2024-03-22T01:35:45.450126+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:13:40.420667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"93'6","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.454755+0000","last_change":"2024-03-22T01:34:15.496310+0000","last_active":"2024-03-22T01:35:45.454755+0000","last_peered":"2024-03-22T01:35:45.454755+0000","last_clean":"2024-03-22T01:35:45.454755+0000","last_became_active":"2024-03-22T01:34:15.494196+0000","last_became_peered":"2024-03-22T01:34:15.494196+0000","last_unstale":"2024-03-22T01:35:45.454755+0000","last_undegraded":"2024-03-22T01:35:45.454755+0000","last_fullsized":"2024-03-22T01:35:45.454755+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:33:02.798310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"93'7","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.440664+0000","last_change":"2024-03-22T01:34:15.495380+0000","last_active":"2024-03-22T01:35:45.440664+0000","last_peered":"2024-03-22T01:35:45.440664+0000","last_clean":"2024-03-22T01:35:45.440664+0000","last_became_active":"2024-03-22T01:34:15.495063+0000","last_became_peered":"2024-03-22T01:34:15.495063+0000","last_unstale":"2024-03-22T01:35:45.440664+0000","last_undegraded":"2024-03-22T01:35:45.440664+0000","last_fullsized":"2024-03-22T01:35:45.440664+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:46:29.497329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"93'10","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.454403+0000","last_change":"2024-03-22T01:34:15.497533+0000","last_active":"2024-03-22T01:35:45.454403+0000","last_peered":"2024-03-22T01:35:45.454403+0000","last_clean":"2024-03-22T01:35:45.454403+0000","last_became_active":"2024-03-22T01:34:15.497223+0000","last_became_peered":"2024-03-22T01:34:15.497223+0000","last_unstale":"2024-03-22T01:35:45.454403+0000","last_undegraded":"2024-03-22T01:35:45.454403+0000","last_fullsized":"2024-03-22T01:35:45.454403+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:17:49.202842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"93'7","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.455929+0000","last_change":"2024-03-22T01:34:15.491317+0000","last_active":"2024-03-22T01:35:45.455929+0000","last_peered":"2024-03-22T01:35:45.455929+0000","last_clean":"2024-03-22T01:35:45.455929+0000","last_became_active":"2024-03-22T01:34:15.490982+0000","last_became_peered":"2024-03-22T01:34:15.490982+0000","last_unstale":"2024-03-22T01:35:45.455929+0000","last_undegraded":"2024-03-22T01:35:45.455929+0000","last_fullsized":"2024-03-22T01:35:45.455929+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:05:39.889964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"93'7","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.455378+0000","last_change":"2024-03-22T01:34:15.488502+0000","last_active":"2024-03-22T01:35:45.455378+0000","last_peered":"2024-03-22T01:35:45.455378+0000","last_clean":"2024-03-22T01:35:45.455378+0000","last_became_active":"2024-03-22T01:34:15.488200+0000","last_became_peered":"2024-03-22T01:34:15.488200+0000","last_unstale":"2024-03-22T01:35:45.455378+0000","last_undegraded":"2024-03-22T01:35:45.455378+0000","last_fullsized":"2024-03-22T01:35:45.455378+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:52:59.991591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"93'9","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.458741+0000","last_change":"2024-03-22T01:34:15.489991+0000","last_active":"2024-03-22T01:35:45.458741+0000","last_peered":"2024-03-22T01:35:45.458741+0000","last_clean":"2024-03-22T01:35:45.458741+0000","last_became_active":"2024-03-22T01:34:15.489696+0000","last_became_peered":"2024-03-22T01:34:15.489696+0000","last_unstale":"2024-03-22T01:35:45.458741+0000","last_undegraded":"2024-03-22T01:35:45.458741+0000","last_fullsized":"2024-03-22T01:35:45.458741+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:10:45.316422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"93'8","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.465277+0000","last_change":"2024-03-22T01:34:15.488011+0000","last_active":"2024-03-22T01:35:45.465277+0000","last_peered":"2024-03-22T01:35:45.465277+0000","last_clean":"2024-03-22T01:35:45.465277+0000","last_became_active":"2024-03-22T01:34:15.487282+0000","last_became_peered":"2024-03-22T01:34:15.487282+0000","last_unstale":"2024-03-22T01:35:45.465277+0000","last_undegraded":"2024-03-22T01:35:45.465277+0000","last_fullsized":"2024-03-22T01:35:45.465277+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:16:21.396273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"93'10","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.463394+0000","last_change":"2024-03-22T01:34:15.488046+0000","last_active":"2024-03-22T01:35:45.463394+0000","last_peered":"2024-03-22T01:35:45.463394+0000","last_clean":"2024-03-22T01:35:45.463394+0000","last_became_active":"2024-03-22T01:34:15.487731+0000","last_became_peered":"2024-03-22T01:34:15.487731+0000","last_unstale":"2024-03-22T01:35:45.463394+0000","last_undegraded":"2024-03-22T01:35:45.463394+0000","last_fullsized":"2024-03-22T01:35:45.463394+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:44:16.931544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"93'7","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.442396+0000","last_change":"2024-03-22T01:34:15.503737+0000","last_active":"2024-03-22T01:35:45.442396+0000","last_peered":"2024-03-22T01:35:45.442396+0000","last_clean":"2024-03-22T01:35:45.442396+0000","last_became_active":"2024-03-22T01:34:15.502169+0000","last_became_peered":"2024-03-22T01:34:15.502169+0000","last_unstale":"2024-03-22T01:35:45.442396+0000","last_undegraded":"2024-03-22T01:35:45.442396+0000","last_fullsized":"2024-03-22T01:35:45.442396+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:05:53.027318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.752699+0000","last_change":"2024-03-22T01:34:19.527964+0000","last_active":"2024-03-22T01:35:16.752699+0000","last_peered":"2024-03-22T01:35:16.752699+0000","last_clean":"2024-03-22T01:35:16.752699+0000","last_became_active":"2024-03-22T01:34:19.527526+0000","last_became_peered":"2024-03-22T01:34:19.527526+0000","last_unstale":"2024-03-22T01:35:16.752699+0000","last_undegraded":"2024-03-22T01:35:16.752699+0000","last_fullsized":"2024-03-22T01:35:16.752699+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:35:39.404439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'4","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.748093+0000","last_change":"2024-03-22T01:34:14.473065+0000","last_active":"2024-03-22T01:35:16.748093+0000","last_peered":"2024-03-22T01:35:16.748093+0000","last_clean":"2024-03-22T01:35:16.748093+0000","last_became_active":"2024-03-22T01:34:14.472653+0000","last_became_peered":"2024-03-22T01:34:14.472653+0000","last_unstale":"2024-03-22T01:35:16.748093+0000","last_undegraded":"2024-03-22T01:35:16.748093+0000","last_fullsized":"2024-03-22T01:35:16.748093+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:11:09.525438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"93'5","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.453870+0000","last_change":"2024-03-22T01:34:15.497818+0000","last_active":"2024-03-22T01:35:45.453870+0000","last_peered":"2024-03-22T01:35:45.453870+0000","last_clean":"2024-03-22T01:35:45.453870+0000","last_became_active":"2024-03-22T01:34:15.497488+0000","last_became_peered":"2024-03-22T01:34:15.497488+0000","last_unstale":"2024-03-22T01:35:45.453870+0000","last_undegraded":"2024-03-22T01:35:45.453870+0000","last_fullsized":"2024-03-22T01:35:45.453870+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:21:09.340557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.743623+0000","last_change":"2024-03-22T01:34:19.528435+0000","last_active":"2024-03-22T01:35:16.743623+0000","last_peered":"2024-03-22T01:35:16.743623+0000","last_clean":"2024-03-22T01:35:16.743623+0000","last_became_active":"2024-03-22T01:34:19.528044+0000","last_became_peered":"2024-03-22T01:34:19.528044+0000","last_unstale":"2024-03-22T01:35:16.743623+0000","last_undegraded":"2024-03-22T01:35:16.743623+0000","last_fullsized":"2024-03-22T01:35:16.743623+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:52:05.682173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"86'1","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:17.204842+0000","last_change":"2024-03-22T01:34:14.465817+0000","last_active":"2024-03-22T01:35:17.204842+0000","last_peered":"2024-03-22T01:35:17.204842+0000","last_clean":"2024-03-22T01:35:17.204842+0000","last_became_active":"2024-03-22T01:34:14.464338+0000","last_became_peered":"2024-03-22T01:34:14.464338+0000","last_unstale":"2024-03-22T01:35:17.204842+0000","last_undegraded":"2024-03-22T01:35:17.204842+0000","last_fullsized":"2024-03-22T01:35:17.204842+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:04:47.457092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"93'6","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.460168+0000","last_change":"2024-03-22T01:34:15.481988+0000","last_active":"2024-03-22T01:35:45.460168+0000","last_peered":"2024-03-22T01:35:45.460168+0000","last_clean":"2024-03-22T01:35:45.460168+0000","last_became_active":"2024-03-22T01:34:15.481676+0000","last_became_peered":"2024-03-22T01:34:15.481676+0000","last_unstale":"2024-03-22T01:35:45.460168+0000","last_undegraded":"2024-03-22T01:35:45.460168+0000","last_fullsized":"2024-03-22T01:35:45.460168+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:37:54.609731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.743694+0000","last_change":"2024-03-22T01:34:19.528349+0000","last_active":"2024-03-22T01:35:16.743694+0000","last_peered":"2024-03-22T01:35:16.743694+0000","last_clean":"2024-03-22T01:35:16.743694+0000","last_became_active":"2024-03-22T01:34:19.527937+0000","last_became_peered":"2024-03-22T01:34:19.527937+0000","last_unstale":"2024-03-22T01:35:16.743694+0000","last_undegraded":"2024-03-22T01:35:16.743694+0000","last_fullsized":"2024-03-22T01:35:16.743694+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:48:34.242104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"86'2","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.756010+0000","last_change":"2024-03-22T01:35:16.756010+0000","last_active":"2024-03-22T01:35:16.756010+0000","last_peered":"2024-03-22T01:35:16.756010+0000","last_clean":"2024-03-22T01:35:16.756010+0000","last_became_active":"2024-03-22T01:35:16.751845+0000","last_became_peered":"2024-03-22T01:35:16.751845+0000","last_unstale":"2024-03-22T01:35:16.756010+0000","last_undegraded":"2024-03-22T01:35:16.756010+0000","last_fullsized":"2024-03-22T01:35:16.756010+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:23:18.305330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"93'9","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.464125+0000","last_change":"2024-03-22T01:34:15.491345+0000","last_active":"2024-03-22T01:35:45.464125+0000","last_peered":"2024-03-22T01:35:45.464125+0000","last_clean":"2024-03-22T01:35:45.464125+0000","last_became_active":"2024-03-22T01:34:15.490833+0000","last_became_peered":"2024-03-22T01:34:15.490833+0000","last_unstale":"2024-03-22T01:35:45.464125+0000","last_undegraded":"2024-03-22T01:35:45.464125+0000","last_fullsized":"2024-03-22T01:35:45.464125+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:30:43.950679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.743633+0000","last_change":"2024-03-22T01:34:19.528351+0000","last_active":"2024-03-22T01:35:16.743633+0000","last_peered":"2024-03-22T01:35:16.743633+0000","last_clean":"2024-03-22T01:35:16.743633+0000","last_became_active":"2024-03-22T01:34:19.527939+0000","last_became_peered":"2024-03-22T01:34:19.527939+0000","last_unstale":"2024-03-22T01:35:16.743633+0000","last_undegraded":"2024-03-22T01:35:16.743633+0000","last_fullsized":"2024-03-22T01:35:16.743633+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:30:21.055848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.756388+0000","last_change":"2024-03-22T01:35:16.755726+0000","last_active":"2024-03-22T01:35:16.756388+0000","last_peered":"2024-03-22T01:35:16.756388+0000","last_clean":"2024-03-22T01:35:16.756388+0000","last_became_active":"2024-03-22T01:35:16.749107+0000","last_became_peered":"2024-03-22T01:35:16.749107+0000","last_unstale":"2024-03-22T01:35:16.756388+0000","last_undegraded":"2024-03-22T01:35:16.756388+0000","last_fullsized":"2024-03-22T01:35:16.756388+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:49:37.500003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"93'15","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.464009+0000","last_change":"2024-03-22T01:34:15.479530+0000","last_active":"2024-03-22T01:35:45.464009+0000","last_peered":"2024-03-22T01:35:45.464009+0000","last_clean":"2024-03-22T01:35:45.464009+0000","last_became_active":"2024-03-22T01:34:15.479198+0000","last_became_peered":"2024-03-22T01:34:15.479198+0000","last_unstale":"2024-03-22T01:35:45.464009+0000","last_undegraded":"2024-03-22T01:35:45.464009+0000","last_fullsized":"2024-03-22T01:35:45.464009+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:28:51.624369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.753040+0000","last_change":"2024-03-22T01:34:19.524486+0000","last_active":"2024-03-22T01:35:16.753040+0000","last_peered":"2024-03-22T01:35:16.753040+0000","last_clean":"2024-03-22T01:35:16.753040+0000","last_became_active":"2024-03-22T01:34:19.524030+0000","last_became_peered":"2024-03-22T01:34:19.524030+0000","last_unstale":"2024-03-22T01:35:16.753040+0000","last_undegraded":"2024-03-22T01:35:16.753040+0000","last_fullsized":"2024-03-22T01:35:16.753040+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:31:10.157431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"86'1","reported_seq":26,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.745175+0000","last_change":"2024-03-22T01:34:14.469191+0000","last_active":"2024-03-22T01:35:16.745175+0000","last_peered":"2024-03-22T01:35:16.745175+0000","last_clean":"2024-03-22T01:35:16.745175+0000","last_became_active":"2024-03-22T01:34:14.468842+0000","last_became_peered":"2024-03-22T01:34:14.468842+0000","last_unstale":"2024-03-22T01:35:16.745175+0000","last_undegraded":"2024-03-22T01:35:16.745175+0000","last_fullsized":"2024-03-22T01:35:16.745175+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:29:56.566256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"93'7","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.443178+0000","last_change":"2024-03-22T01:34:15.483736+0000","last_active":"2024-03-22T01:35:45.443178+0000","last_peered":"2024-03-22T01:35:45.443178+0000","last_clean":"2024-03-22T01:35:45.443178+0000","last_became_active":"2024-03-22T01:34:15.483401+0000","last_became_peered":"2024-03-22T01:34:15.483401+0000","last_unstale":"2024-03-22T01:35:45.443178+0000","last_undegraded":"2024-03-22T01:35:45.443178+0000","last_fullsized":"2024-03-22T01:35:45.443178+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:26:51.974932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.753323+0000","last_change":"2024-03-22T01:34:19.534542+0000","last_active":"2024-03-22T01:35:16.753323+0000","last_peered":"2024-03-22T01:35:16.753323+0000","last_clean":"2024-03-22T01:35:16.753323+0000","last_became_active":"2024-03-22T01:34:19.534039+0000","last_became_peered":"2024-03-22T01:34:19.534039+0000","last_unstale":"2024-03-22T01:35:16.753323+0000","last_undegraded":"2024-03-22T01:35:16.753323+0000","last_fullsized":"2024-03-22T01:35:16.753323+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:59:37.363972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"86'1","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.814330+0000","last_change":"2024-03-22T01:34:14.468382+0000","last_active":"2024-03-22T01:35:16.814330+0000","last_peered":"2024-03-22T01:35:16.814330+0000","last_clean":"2024-03-22T01:35:16.814330+0000","last_became_active":"2024-03-22T01:34:14.467939+0000","last_became_peered":"2024-03-22T01:34:14.467939+0000","last_unstale":"2024-03-22T01:35:16.814330+0000","last_undegraded":"2024-03-22T01:35:16.814330+0000","last_fullsized":"2024-03-22T01:35:16.814330+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:57:17.031893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"93'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.457175+0000","last_change":"2024-03-22T01:35:17.206178+0000","last_active":"2024-03-22T01:35:45.457175+0000","last_peered":"2024-03-22T01:35:45.457175+0000","last_clean":"2024-03-22T01:35:45.457175+0000","last_became_active":"2024-03-22T01:35:17.205717+0000","last_became_peered":"2024-03-22T01:35:17.205717+0000","last_unstale":"2024-03-22T01:35:45.457175+0000","last_undegraded":"2024-03-22T01:35:45.457175+0000","last_fullsized":"2024-03-22T01:35:45.457175+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:14:15.920771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.746553+0000","last_change":"2024-03-22T01:34:19.531532+0000","last_active":"2024-03-22T01:35:16.746553+0000","last_peered":"2024-03-22T01:35:16.746553+0000","last_clean":"2024-03-22T01:35:16.746553+0000","last_became_active":"2024-03-22T01:34:19.531202+0000","last_became_peered":"2024-03-22T01:34:19.531202+0000","last_unstale":"2024-03-22T01:35:16.746553+0000","last_undegraded":"2024-03-22T01:35:16.746553+0000","last_fullsized":"2024-03-22T01:35:16.746553+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:52:12.823806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":26,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.743316+0000","last_change":"2024-03-22T01:34:14.464485+0000","last_active":"2024-03-22T01:35:16.743316+0000","last_peered":"2024-03-22T01:35:16.743316+0000","last_clean":"2024-03-22T01:35:16.743316+0000","last_became_active":"2024-03-22T01:34:14.463973+0000","last_became_peered":"2024-03-22T01:34:14.463973+0000","last_unstale":"2024-03-22T01:35:16.743316+0000","last_undegraded":"2024-03-22T01:35:16.743316+0000","last_fullsized":"2024-03-22T01:35:16.743316+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:11:13.265346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"93'8","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.449431+0000","last_change":"2024-03-22T01:34:15.483893+0000","last_active":"2024-03-22T01:35:45.449431+0000","last_peered":"2024-03-22T01:35:45.449431+0000","last_clean":"2024-03-22T01:35:45.449431+0000","last_became_active":"2024-03-22T01:34:15.482295+0000","last_became_peered":"2024-03-22T01:34:15.482295+0000","last_unstale":"2024-03-22T01:35:45.449431+0000","last_undegraded":"2024-03-22T01:35:45.449431+0000","last_fullsized":"2024-03-22T01:35:45.449431+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:17:33.274900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.746059+0000","last_change":"2024-03-22T01:34:19.522567+0000","last_active":"2024-03-22T01:35:16.746059+0000","last_peered":"2024-03-22T01:35:16.746059+0000","last_clean":"2024-03-22T01:35:16.746059+0000","last_became_active":"2024-03-22T01:34:19.522165+0000","last_became_peered":"2024-03-22T01:34:19.522165+0000","last_unstale":"2024-03-22T01:35:16.746059+0000","last_undegraded":"2024-03-22T01:35:16.746059+0000","last_fullsized":"2024-03-22T01:35:16.746059+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:41:37.829622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.744359+0000","last_change":"2024-03-22T01:34:14.463824+0000","last_active":"2024-03-22T01:35:16.744359+0000","last_peered":"2024-03-22T01:35:16.744359+0000","last_clean":"2024-03-22T01:35:16.744359+0000","last_became_active":"2024-03-22T01:34:14.463484+0000","last_became_peered":"2024-03-22T01:34:14.463484+0000","last_unstale":"2024-03-22T01:35:16.744359+0000","last_undegraded":"2024-03-22T01:35:16.744359+0000","last_fullsized":"2024-03-22T01:35:16.744359+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:19:31.568853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"93'11","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.459393+0000","last_change":"2024-03-22T01:34:15.488883+0000","last_active":"2024-03-22T01:35:45.459393+0000","last_peered":"2024-03-22T01:35:45.459393+0000","last_clean":"2024-03-22T01:35:45.459393+0000","last_became_active":"2024-03-22T01:34:15.488573+0000","last_became_peered":"2024-03-22T01:34:15.488573+0000","last_unstale":"2024-03-22T01:35:45.459393+0000","last_undegraded":"2024-03-22T01:35:45.459393+0000","last_fullsized":"2024-03-22T01:35:45.459393+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:28:46.388930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:17.204329+0000","last_change":"2024-03-22T01:34:19.527321+0000","last_active":"2024-03-22T01:35:17.204329+0000","last_peered":"2024-03-22T01:35:17.204329+0000","last_clean":"2024-03-22T01:35:17.204329+0000","last_became_active":"2024-03-22T01:34:19.526905+0000","last_became_peered":"2024-03-22T01:34:19.526905+0000","last_unstale":"2024-03-22T01:35:17.204329+0000","last_undegraded":"2024-03-22T01:35:17.204329+0000","last_fullsized":"2024-03-22T01:35:17.204329+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:49:31.023938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.814616+0000","last_change":"2024-03-22T01:34:14.471415+0000","last_active":"2024-03-22T01:35:16.814616+0000","last_peered":"2024-03-22T01:35:16.814616+0000","last_clean":"2024-03-22T01:35:16.814616+0000","last_became_active":"2024-03-22T01:34:14.470928+0000","last_became_peered":"2024-03-22T01:34:14.470928+0000","last_unstale":"2024-03-22T01:35:16.814616+0000","last_undegraded":"2024-03-22T01:35:16.814616+0000","last_fullsized":"2024-03-22T01:35:16.814616+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:09:10.286549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"93'10","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.464353+0000","last_change":"2024-03-22T01:34:15.489656+0000","last_active":"2024-03-22T01:35:45.464353+0000","last_peered":"2024-03-22T01:35:45.464353+0000","last_clean":"2024-03-22T01:35:45.464353+0000","last_became_active":"2024-03-22T01:34:15.488933+0000","last_became_peered":"2024-03-22T01:34:15.488933+0000","last_unstale":"2024-03-22T01:35:45.464353+0000","last_undegraded":"2024-03-22T01:35:45.464353+0000","last_fullsized":"2024-03-22T01:35:45.464353+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:17:38.789414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.746845+0000","last_change":"2024-03-22T01:34:19.534174+0000","last_active":"2024-03-22T01:35:16.746845+0000","last_peered":"2024-03-22T01:35:16.746845+0000","last_clean":"2024-03-22T01:35:16.746845+0000","last_became_active":"2024-03-22T01:34:19.533759+0000","last_became_peered":"2024-03-22T01:34:19.533759+0000","last_unstale":"2024-03-22T01:35:16.746845+0000","last_undegraded":"2024-03-22T01:35:16.746845+0000","last_fullsized":"2024-03-22T01:35:16.746845+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:24:50.363984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":26,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.746461+0000","last_change":"2024-03-22T01:34:14.465755+0000","last_active":"2024-03-22T01:35:16.746461+0000","last_peered":"2024-03-22T01:35:16.746461+0000","last_clean":"2024-03-22T01:35:16.746461+0000","last_became_active":"2024-03-22T01:34:14.465230+0000","last_became_peered":"2024-03-22T01:34:14.465230+0000","last_unstale":"2024-03-22T01:35:16.746461+0000","last_undegraded":"2024-03-22T01:35:16.746461+0000","last_fullsized":"2024-03-22T01:35:16.746461+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:56:29.286551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"93'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.458831+0000","last_change":"2024-03-22T01:35:16.748981+0000","last_active":"2024-03-22T01:35:45.458831+0000","last_peered":"2024-03-22T01:35:45.458831+0000","last_clean":"2024-03-22T01:35:45.458831+0000","last_became_active":"2024-03-22T01:35:16.748524+0000","last_became_peered":"2024-03-22T01:35:16.748524+0000","last_unstale":"2024-03-22T01:35:45.458831+0000","last_undegraded":"2024-03-22T01:35:45.458831+0000","last_fullsized":"2024-03-22T01:35:45.458831+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:23:37.763347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.743817+0000","last_change":"2024-03-22T01:34:19.527218+0000","last_active":"2024-03-22T01:35:16.743817+0000","last_peered":"2024-03-22T01:35:16.743817+0000","last_clean":"2024-03-22T01:35:16.743817+0000","last_became_active":"2024-03-22T01:34:19.526856+0000","last_became_peered":"2024-03-22T01:34:19.526856+0000","last_unstale":"2024-03-22T01:35:16.743817+0000","last_undegraded":"2024-03-22T01:35:16.743817+0000","last_fullsized":"2024-03-22T01:35:16.743817+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:25:33.878015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"86'3","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.744957+0000","last_change":"2024-03-22T01:34:14.463365+0000","last_active":"2024-03-22T01:35:16.744957+0000","last_peered":"2024-03-22T01:35:16.744957+0000","last_clean":"2024-03-22T01:35:16.744957+0000","last_became_active":"2024-03-22T01:34:14.462984+0000","last_became_peered":"2024-03-22T01:34:14.462984+0000","last_unstale":"2024-03-22T01:35:16.744957+0000","last_undegraded":"2024-03-22T01:35:16.744957+0000","last_fullsized":"2024-03-22T01:35:16.744957+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:55:27.318854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"93'11","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.463709+0000","last_change":"2024-03-22T01:34:15.485309+0000","last_active":"2024-03-22T01:35:45.463709+0000","last_peered":"2024-03-22T01:35:45.463709+0000","last_clean":"2024-03-22T01:35:45.463709+0000","last_became_active":"2024-03-22T01:34:15.484964+0000","last_became_peered":"2024-03-22T01:34:15.484964+0000","last_unstale":"2024-03-22T01:35:45.463709+0000","last_undegraded":"2024-03-22T01:35:45.463709+0000","last_fullsized":"2024-03-22T01:35:45.463709+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:25:57.911623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.747797+0000","last_change":"2024-03-22T01:34:19.529346+0000","last_active":"2024-03-22T01:35:16.747797+0000","last_peered":"2024-03-22T01:35:16.747797+0000","last_clean":"2024-03-22T01:35:16.747797+0000","last_became_active":"2024-03-22T01:34:19.525999+0000","last_became_peered":"2024-03-22T01:34:19.525999+0000","last_unstale":"2024-03-22T01:35:16.747797+0000","last_undegraded":"2024-03-22T01:35:16.747797+0000","last_fullsized":"2024-03-22T01:35:16.747797+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:00:21.868812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"93'33","reported_seq":220,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:54.624463+0000","last_change":"2024-03-22T01:34:14.469378+0000","last_active":"2024-03-22T01:35:54.624463+0000","last_peered":"2024-03-22T01:35:54.624463+0000","last_clean":"2024-03-22T01:35:54.624463+0000","last_became_active":"2024-03-22T01:34:14.468956+0000","last_became_peered":"2024-03-22T01:34:14.468956+0000","last_unstale":"2024-03-22T01:35:54.624463+0000","last_undegraded":"2024-03-22T01:35:54.624463+0000","last_fullsized":"2024-03-22T01:35:54.624463+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:40:49.048793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63605,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":160,"num_read_kb":142,"num_write":33,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"93'6","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.453026+0000","last_change":"2024-03-22T01:34:15.488497+0000","last_active":"2024-03-22T01:35:45.453026+0000","last_peered":"2024-03-22T01:35:45.453026+0000","last_clean":"2024-03-22T01:35:45.453026+0000","last_became_active":"2024-03-22T01:34:15.488134+0000","last_became_peered":"2024-03-22T01:34:15.488134+0000","last_unstale":"2024-03-22T01:35:45.453026+0000","last_undegraded":"2024-03-22T01:35:45.453026+0000","last_fullsized":"2024-03-22T01:35:45.453026+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:50:41.206135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.745269+0000","last_change":"2024-03-22T01:34:19.528744+0000","last_active":"2024-03-22T01:35:16.745269+0000","last_peered":"2024-03-22T01:35:16.745269+0000","last_clean":"2024-03-22T01:35:16.745269+0000","last_became_active":"2024-03-22T01:34:19.528375+0000","last_became_peered":"2024-03-22T01:34:19.528375+0000","last_unstale":"2024-03-22T01:35:16.745269+0000","last_undegraded":"2024-03-22T01:35:16.745269+0000","last_fullsized":"2024-03-22T01:35:16.745269+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:47:53.753923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"86'2","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.752872+0000","last_change":"2024-03-22T01:34:14.463289+0000","last_active":"2024-03-22T01:35:16.752872+0000","last_peered":"2024-03-22T01:35:16.752872+0000","last_clean":"2024-03-22T01:35:16.752872+0000","last_became_active":"2024-03-22T01:34:14.462883+0000","last_became_peered":"2024-03-22T01:34:14.462883+0000","last_unstale":"2024-03-22T01:35:16.752872+0000","last_undegraded":"2024-03-22T01:35:16.752872+0000","last_fullsized":"2024-03-22T01:35:16.752872+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:06:14.452674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"93'9","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.454086+0000","last_change":"2024-03-22T01:34:15.484785+0000","last_active":"2024-03-22T01:35:45.454086+0000","last_peered":"2024-03-22T01:35:45.454086+0000","last_clean":"2024-03-22T01:35:45.454086+0000","last_became_active":"2024-03-22T01:34:15.484474+0000","last_became_peered":"2024-03-22T01:34:15.484474+0000","last_unstale":"2024-03-22T01:35:45.454086+0000","last_undegraded":"2024-03-22T01:35:45.454086+0000","last_fullsized":"2024-03-22T01:35:45.454086+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:11:13.338836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.746861+0000","last_change":"2024-03-22T01:34:19.527379+0000","last_active":"2024-03-22T01:35:16.746861+0000","last_peered":"2024-03-22T01:35:16.746861+0000","last_clean":"2024-03-22T01:35:16.746861+0000","last_became_active":"2024-03-22T01:34:19.526957+0000","last_became_peered":"2024-03-22T01:34:19.526957+0000","last_unstale":"2024-03-22T01:35:16.746861+0000","last_undegraded":"2024-03-22T01:35:16.746861+0000","last_fullsized":"2024-03-22T01:35:16.746861+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:03:09.538334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.744463+0000","last_change":"2024-03-22T01:34:14.467372+0000","last_active":"2024-03-22T01:35:16.744463+0000","last_peered":"2024-03-22T01:35:16.744463+0000","last_clean":"2024-03-22T01:35:16.744463+0000","last_became_active":"2024-03-22T01:34:14.466124+0000","last_became_peered":"2024-03-22T01:34:14.466124+0000","last_unstale":"2024-03-22T01:35:16.744463+0000","last_undegraded":"2024-03-22T01:35:16.744463+0000","last_fullsized":"2024-03-22T01:35:16.744463+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:15:41.804735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"93'15","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.456546+0000","last_change":"2024-03-22T01:34:15.491533+0000","last_active":"2024-03-22T01:35:45.456546+0000","last_peered":"2024-03-22T01:35:45.456546+0000","last_clean":"2024-03-22T01:35:45.456546+0000","last_became_active":"2024-03-22T01:34:15.491208+0000","last_became_peered":"2024-03-22T01:34:15.491208+0000","last_unstale":"2024-03-22T01:35:45.456546+0000","last_undegraded":"2024-03-22T01:35:45.456546+0000","last_fullsized":"2024-03-22T01:35:45.456546+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:03:07.800607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.748346+0000","last_change":"2024-03-22T01:34:19.533419+0000","last_active":"2024-03-22T01:35:16.748346+0000","last_peered":"2024-03-22T01:35:16.748346+0000","last_clean":"2024-03-22T01:35:16.748346+0000","last_became_active":"2024-03-22T01:34:19.530042+0000","last_became_peered":"2024-03-22T01:34:19.530042+0000","last_unstale":"2024-03-22T01:35:16.748346+0000","last_undegraded":"2024-03-22T01:35:16.748346+0000","last_fullsized":"2024-03-22T01:35:16.748346+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:46:59.859742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.743765+0000","last_change":"2024-03-22T01:34:14.467033+0000","last_active":"2024-03-22T01:35:16.743765+0000","last_peered":"2024-03-22T01:35:16.743765+0000","last_clean":"2024-03-22T01:35:16.743765+0000","last_became_active":"2024-03-22T01:34:14.466649+0000","last_became_peered":"2024-03-22T01:34:14.466649+0000","last_unstale":"2024-03-22T01:35:16.743765+0000","last_undegraded":"2024-03-22T01:35:16.743765+0000","last_fullsized":"2024-03-22T01:35:16.743765+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:38:44.890899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"93'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.442847+0000","last_change":"2024-03-22T01:35:16.815302+0000","last_active":"2024-03-22T01:35:45.442847+0000","last_peered":"2024-03-22T01:35:45.442847+0000","last_clean":"2024-03-22T01:35:45.442847+0000","last_became_active":"2024-03-22T01:35:16.814988+0000","last_became_peered":"2024-03-22T01:35:16.814988+0000","last_unstale":"2024-03-22T01:35:45.442847+0000","last_undegraded":"2024-03-22T01:35:45.442847+0000","last_fullsized":"2024-03-22T01:35:45.442847+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:54:47.793437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.743568+0000","last_change":"2024-03-22T01:34:19.531417+0000","last_active":"2024-03-22T01:35:16.743568+0000","last_peered":"2024-03-22T01:35:16.743568+0000","last_clean":"2024-03-22T01:35:16.743568+0000","last_became_active":"2024-03-22T01:34:19.531003+0000","last_became_peered":"2024-03-22T01:34:19.531003+0000","last_unstale":"2024-03-22T01:35:16.743568+0000","last_undegraded":"2024-03-22T01:35:16.743568+0000","last_fullsized":"2024-03-22T01:35:16.743568+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:33:52.865110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"86'8","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.748313+0000","last_change":"2024-03-22T01:34:14.472904+0000","last_active":"2024-03-22T01:35:16.748313+0000","last_peered":"2024-03-22T01:35:16.748313+0000","last_clean":"2024-03-22T01:35:16.748313+0000","last_became_active":"2024-03-22T01:34:14.472442+0000","last_became_peered":"2024-03-22T01:34:14.472442+0000","last_unstale":"2024-03-22T01:35:16.748313+0000","last_undegraded":"2024-03-22T01:35:16.748313+0000","last_fullsized":"2024-03-22T01:35:16.748313+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:57:45.478316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"93'6","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.463203+0000","last_change":"2024-03-22T01:34:15.477411+0000","last_active":"2024-03-22T01:35:45.463203+0000","last_peered":"2024-03-22T01:35:45.463203+0000","last_clean":"2024-03-22T01:35:45.463203+0000","last_became_active":"2024-03-22T01:34:15.477035+0000","last_became_peered":"2024-03-22T01:34:15.477035+0000","last_unstale":"2024-03-22T01:35:45.463203+0000","last_undegraded":"2024-03-22T01:35:45.463203+0000","last_fullsized":"2024-03-22T01:35:45.463203+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:33:13.410872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.744275+0000","last_change":"2024-03-22T01:34:19.528258+0000","last_active":"2024-03-22T01:35:16.744275+0000","last_peered":"2024-03-22T01:35:16.744275+0000","last_clean":"2024-03-22T01:35:16.744275+0000","last_became_active":"2024-03-22T01:34:19.527668+0000","last_became_peered":"2024-03-22T01:34:19.527668+0000","last_unstale":"2024-03-22T01:35:16.744275+0000","last_undegraded":"2024-03-22T01:35:16.744275+0000","last_fullsized":"2024-03-22T01:35:16.744275+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:34:19.162609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'1","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.748746+0000","last_change":"2024-03-22T01:34:14.464735+0000","last_active":"2024-03-22T01:35:16.748746+0000","last_peered":"2024-03-22T01:35:16.748746+0000","last_clean":"2024-03-22T01:35:16.748746+0000","last_became_active":"2024-03-22T01:34:14.464359+0000","last_became_peered":"2024-03-22T01:34:14.464359+0000","last_unstale":"2024-03-22T01:35:16.748746+0000","last_undegraded":"2024-03-22T01:35:16.748746+0000","last_fullsized":"2024-03-22T01:35:16.748746+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:21:18.354922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"93'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.451818+0000","last_change":"2024-03-22T01:35:16.752884+0000","last_active":"2024-03-22T01:35:45.451818+0000","last_peered":"2024-03-22T01:35:45.451818+0000","last_clean":"2024-03-22T01:35:45.451818+0000","last_became_active":"2024-03-22T01:35:16.751355+0000","last_became_peered":"2024-03-22T01:35:16.751355+0000","last_unstale":"2024-03-22T01:35:45.451818+0000","last_undegraded":"2024-03-22T01:35:45.451818+0000","last_fullsized":"2024-03-22T01:35:45.451818+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:49:33.261481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.748621+0000","last_change":"2024-03-22T01:34:19.668136+0000","last_active":"2024-03-22T01:35:16.748621+0000","last_peered":"2024-03-22T01:35:16.748621+0000","last_clean":"2024-03-22T01:35:16.748621+0000","last_became_active":"2024-03-22T01:34:19.667633+0000","last_became_peered":"2024-03-22T01:34:19.667633+0000","last_unstale":"2024-03-22T01:35:16.748621+0000","last_undegraded":"2024-03-22T01:35:16.748621+0000","last_fullsized":"2024-03-22T01:35:16.748621+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:07:26.455373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.747658+0000","last_change":"2024-03-22T01:34:14.471959+0000","last_active":"2024-03-22T01:35:16.747658+0000","last_peered":"2024-03-22T01:35:16.747658+0000","last_clean":"2024-03-22T01:35:16.747658+0000","last_became_active":"2024-03-22T01:34:14.471618+0000","last_became_peered":"2024-03-22T01:34:14.471618+0000","last_unstale":"2024-03-22T01:35:16.747658+0000","last_undegraded":"2024-03-22T01:35:16.747658+0000","last_fullsized":"2024-03-22T01:35:16.747658+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:45:03.537013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"93'4","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.450651+0000","last_change":"2024-03-22T01:34:15.495770+0000","last_active":"2024-03-22T01:35:45.450651+0000","last_peered":"2024-03-22T01:35:45.450651+0000","last_clean":"2024-03-22T01:35:45.450651+0000","last_became_active":"2024-03-22T01:34:15.495403+0000","last_became_peered":"2024-03-22T01:34:15.495403+0000","last_unstale":"2024-03-22T01:35:45.450651+0000","last_undegraded":"2024-03-22T01:35:45.450651+0000","last_fullsized":"2024-03-22T01:35:45.450651+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:48:05.184305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.743667+0000","last_change":"2024-03-22T01:34:19.520204+0000","last_active":"2024-03-22T01:35:16.743667+0000","last_peered":"2024-03-22T01:35:16.743667+0000","last_clean":"2024-03-22T01:35:16.743667+0000","last_became_active":"2024-03-22T01:34:19.519847+0000","last_became_peered":"2024-03-22T01:34:19.519847+0000","last_unstale":"2024-03-22T01:35:16.743667+0000","last_undegraded":"2024-03-22T01:35:16.743667+0000","last_fullsized":"2024-03-22T01:35:16.743667+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:06:21.389636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":26,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.746100+0000","last_change":"2024-03-22T01:34:14.464162+0000","last_active":"2024-03-22T01:35:16.746100+0000","last_peered":"2024-03-22T01:35:16.746100+0000","last_clean":"2024-03-22T01:35:16.746100+0000","last_became_active":"2024-03-22T01:34:14.462700+0000","last_became_peered":"2024-03-22T01:34:14.462700+0000","last_unstale":"2024-03-22T01:35:16.746100+0000","last_undegraded":"2024-03-22T01:35:16.746100+0000","last_fullsized":"2024-03-22T01:35:16.746100+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:45:32.262466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"93'7","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.457078+0000","last_change":"2024-03-22T01:34:15.488256+0000","last_active":"2024-03-22T01:35:45.457078+0000","last_peered":"2024-03-22T01:35:45.457078+0000","last_clean":"2024-03-22T01:35:45.457078+0000","last_became_active":"2024-03-22T01:34:15.487939+0000","last_became_peered":"2024-03-22T01:34:15.487939+0000","last_unstale":"2024-03-22T01:35:45.457078+0000","last_undegraded":"2024-03-22T01:35:45.457078+0000","last_fullsized":"2024-03-22T01:35:45.457078+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:30:49.323531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.743552+0000","last_change":"2024-03-22T01:34:19.667633+0000","last_active":"2024-03-22T01:35:16.743552+0000","last_peered":"2024-03-22T01:35:16.743552+0000","last_clean":"2024-03-22T01:35:16.743552+0000","last_became_active":"2024-03-22T01:34:19.667199+0000","last_became_peered":"2024-03-22T01:34:19.667199+0000","last_unstale":"2024-03-22T01:35:16.743552+0000","last_undegraded":"2024-03-22T01:35:16.743552+0000","last_fullsized":"2024-03-22T01:35:16.743552+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T01:58:13.459475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"93'8","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.453646+0000","last_change":"2024-03-22T01:34:15.489468+0000","last_active":"2024-03-22T01:35:45.453646+0000","last_peered":"2024-03-22T01:35:45.453646+0000","last_clean":"2024-03-22T01:35:45.453646+0000","last_became_active":"2024-03-22T01:34:15.489029+0000","last_became_peered":"2024-03-22T01:34:15.489029+0000","last_unstale":"2024-03-22T01:35:45.453646+0000","last_undegraded":"2024-03-22T01:35:45.453646+0000","last_fullsized":"2024-03-22T01:35:45.453646+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:00:01.370161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":26,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.743202+0000","last_change":"2024-03-22T01:34:14.464552+0000","last_active":"2024-03-22T01:35:16.743202+0000","last_peered":"2024-03-22T01:35:16.743202+0000","last_clean":"2024-03-22T01:35:16.743202+0000","last_became_active":"2024-03-22T01:34:14.463554+0000","last_became_peered":"2024-03-22T01:34:14.463554+0000","last_unstale":"2024-03-22T01:35:16.743202+0000","last_undegraded":"2024-03-22T01:35:16.743202+0000","last_fullsized":"2024-03-22T01:35:16.743202+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:51:16.709278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"91'1","reported_seq":19,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.746441+0000","last_change":"2024-03-22T01:34:19.526066+0000","last_active":"2024-03-22T01:35:16.746441+0000","last_peered":"2024-03-22T01:35:16.746441+0000","last_clean":"2024-03-22T01:35:16.746441+0000","last_became_active":"2024-03-22T01:34:19.525659+0000","last_became_peered":"2024-03-22T01:34:19.525659+0000","last_unstale":"2024-03-22T01:35:16.746441+0000","last_undegraded":"2024-03-22T01:35:16.746441+0000","last_fullsized":"2024-03-22T01:35:16.746441+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:06:26.287971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"93'8","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.454417+0000","last_change":"2024-03-22T01:34:15.477018+0000","last_active":"2024-03-22T01:35:45.454417+0000","last_peered":"2024-03-22T01:35:45.454417+0000","last_clean":"2024-03-22T01:35:45.454417+0000","last_became_active":"2024-03-22T01:34:15.476613+0000","last_became_peered":"2024-03-22T01:34:15.476613+0000","last_unstale":"2024-03-22T01:35:45.454417+0000","last_undegraded":"2024-03-22T01:35:45.454417+0000","last_fullsized":"2024-03-22T01:35:45.454417+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:10:07.980123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.743756+0000","last_change":"2024-03-22T01:34:14.470922+0000","last_active":"2024-03-22T01:35:16.743756+0000","last_peered":"2024-03-22T01:35:16.743756+0000","last_clean":"2024-03-22T01:35:16.743756+0000","last_became_active":"2024-03-22T01:34:14.470515+0000","last_became_peered":"2024-03-22T01:34:14.470515+0000","last_unstale":"2024-03-22T01:35:16.743756+0000","last_undegraded":"2024-03-22T01:35:16.743756+0000","last_fullsized":"2024-03-22T01:35:16.743756+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:52:50.203631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":359,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.745989+0000","last_change":"2024-03-22T01:33:03.615234+0000","last_active":"2024-03-22T01:35:16.745989+0000","last_peered":"2024-03-22T01:35:16.745989+0000","last_clean":"2024-03-22T01:35:16.745989+0000","last_became_active":"2024-03-22T01:33:03.614710+0000","last_became_peered":"2024-03-22T01:33:03.614710+0000","last_unstale":"2024-03-22T01:35:16.745989+0000","last_undegraded":"2024-03-22T01:35:16.745989+0000","last_fullsized":"2024-03-22T01:35:16.745989+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:29:38.312292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:29:38.312292+0000","last_clean_scrub_stamp":"2024-03-22T01:29:38.312292+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:56:42.348119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.745330+0000","last_change":"2024-03-22T01:34:19.668565+0000","last_active":"2024-03-22T01:35:16.745330+0000","last_peered":"2024-03-22T01:35:16.745330+0000","last_clean":"2024-03-22T01:35:16.745330+0000","last_became_active":"2024-03-22T01:34:19.667959+0000","last_became_peered":"2024-03-22T01:34:19.667959+0000","last_unstale":"2024-03-22T01:35:16.745330+0000","last_undegraded":"2024-03-22T01:35:16.745330+0000","last_fullsized":"2024-03-22T01:35:16.745330+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:13:04.447486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"93'5","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.454337+0000","last_change":"2024-03-22T01:34:15.489513+0000","last_active":"2024-03-22T01:35:45.454337+0000","last_peered":"2024-03-22T01:35:45.454337+0000","last_clean":"2024-03-22T01:35:45.454337+0000","last_became_active":"2024-03-22T01:34:15.489188+0000","last_became_peered":"2024-03-22T01:34:15.489188+0000","last_unstale":"2024-03-22T01:35:45.454337+0000","last_undegraded":"2024-03-22T01:35:45.454337+0000","last_fullsized":"2024-03-22T01:35:45.454337+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:08:24.859350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":26,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.746166+0000","last_change":"2024-03-22T01:34:14.466305+0000","last_active":"2024-03-22T01:35:16.746166+0000","last_peered":"2024-03-22T01:35:16.746166+0000","last_clean":"2024-03-22T01:35:16.746166+0000","last_became_active":"2024-03-22T01:34:14.465495+0000","last_became_peered":"2024-03-22T01:34:14.465495+0000","last_unstale":"2024-03-22T01:35:16.746166+0000","last_undegraded":"2024-03-22T01:35:16.746166+0000","last_fullsized":"2024-03-22T01:35:16.746166+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:45:13.481187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.746455+0000","last_change":"2024-03-22T01:34:19.526970+0000","last_active":"2024-03-22T01:35:16.746455+0000","last_peered":"2024-03-22T01:35:16.746455+0000","last_clean":"2024-03-22T01:35:16.746455+0000","last_became_active":"2024-03-22T01:34:19.526462+0000","last_became_peered":"2024-03-22T01:34:19.526462+0000","last_unstale":"2024-03-22T01:35:16.746455+0000","last_undegraded":"2024-03-22T01:35:16.746455+0000","last_fullsized":"2024-03-22T01:35:16.746455+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:04:18.417358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.752065+0000","last_change":"2024-03-22T01:35:16.750746+0000","last_active":"2024-03-22T01:35:16.752065+0000","last_peered":"2024-03-22T01:35:16.752065+0000","last_clean":"2024-03-22T01:35:16.752065+0000","last_became_active":"2024-03-22T01:35:16.750271+0000","last_became_peered":"2024-03-22T01:35:16.750271+0000","last_unstale":"2024-03-22T01:35:16.752065+0000","last_undegraded":"2024-03-22T01:35:16.752065+0000","last_fullsized":"2024-03-22T01:35:16.752065+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:18:10.839351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"93'9","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.459554+0000","last_change":"2024-03-22T01:34:15.490405+0000","last_active":"2024-03-22T01:35:45.459554+0000","last_peered":"2024-03-22T01:35:45.459554+0000","last_clean":"2024-03-22T01:35:45.459554+0000","last_became_active":"2024-03-22T01:34:15.489921+0000","last_became_peered":"2024-03-22T01:34:15.489921+0000","last_unstale":"2024-03-22T01:35:45.459554+0000","last_undegraded":"2024-03-22T01:35:45.459554+0000","last_fullsized":"2024-03-22T01:35:45.459554+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:27:28.804649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:17.204305+0000","last_change":"2024-03-22T01:34:19.667651+0000","last_active":"2024-03-22T01:35:17.204305+0000","last_peered":"2024-03-22T01:35:17.204305+0000","last_clean":"2024-03-22T01:35:17.204305+0000","last_became_active":"2024-03-22T01:34:19.667197+0000","last_became_peered":"2024-03-22T01:34:19.667197+0000","last_unstale":"2024-03-22T01:35:17.204305+0000","last_undegraded":"2024-03-22T01:35:17.204305+0000","last_fullsized":"2024-03-22T01:35:17.204305+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:32:55.874098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"93'7","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.748086+0000","last_change":"2024-03-22T01:34:14.474988+0000","last_active":"2024-03-22T01:35:16.748086+0000","last_peered":"2024-03-22T01:35:16.748086+0000","last_clean":"2024-03-22T01:35:16.748086+0000","last_became_active":"2024-03-22T01:34:14.474651+0000","last_became_peered":"2024-03-22T01:34:14.474651+0000","last_unstale":"2024-03-22T01:35:16.748086+0000","last_undegraded":"2024-03-22T01:35:16.748086+0000","last_fullsized":"2024-03-22T01:35:16.748086+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:14:59.846727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"93'5","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.450371+0000","last_change":"2024-03-22T01:35:16.749643+0000","last_active":"2024-03-22T01:35:45.450371+0000","last_peered":"2024-03-22T01:35:45.450371+0000","last_clean":"2024-03-22T01:35:45.450371+0000","last_became_active":"2024-03-22T01:35:16.749311+0000","last_became_peered":"2024-03-22T01:35:16.749311+0000","last_unstale":"2024-03-22T01:35:45.450371+0000","last_undegraded":"2024-03-22T01:35:45.450371+0000","last_fullsized":"2024-03-22T01:35:45.450371+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:23:18.666007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,7],"acting":[9,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.752355+0000","last_change":"2024-03-22T01:34:19.523959+0000","last_active":"2024-03-22T01:35:16.752355+0000","last_peered":"2024-03-22T01:35:16.752355+0000","last_clean":"2024-03-22T01:35:16.752355+0000","last_became_active":"2024-03-22T01:34:19.523556+0000","last_became_peered":"2024-03-22T01:34:19.523556+0000","last_unstale":"2024-03-22T01:35:16.752355+0000","last_undegraded":"2024-03-22T01:35:16.752355+0000","last_fullsized":"2024-03-22T01:35:16.752355+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:22:21.623042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"86'2","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.744215+0000","last_change":"2024-03-22T01:34:14.463434+0000","last_active":"2024-03-22T01:35:16.744215+0000","last_peered":"2024-03-22T01:35:16.744215+0000","last_clean":"2024-03-22T01:35:16.744215+0000","last_became_active":"2024-03-22T01:34:14.463075+0000","last_became_peered":"2024-03-22T01:34:14.463075+0000","last_unstale":"2024-03-22T01:35:16.744215+0000","last_undegraded":"2024-03-22T01:35:16.744215+0000","last_fullsized":"2024-03-22T01:35:16.744215+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:43:18.223766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"93'7","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.452320+0000","last_change":"2024-03-22T01:34:15.482695+0000","last_active":"2024-03-22T01:35:45.452320+0000","last_peered":"2024-03-22T01:35:45.452320+0000","last_clean":"2024-03-22T01:35:45.452320+0000","last_became_active":"2024-03-22T01:34:15.481881+0000","last_became_peered":"2024-03-22T01:34:15.481881+0000","last_unstale":"2024-03-22T01:35:45.452320+0000","last_undegraded":"2024-03-22T01:35:45.452320+0000","last_fullsized":"2024-03-22T01:35:45.452320+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:44:26.290714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.744060+0000","last_change":"2024-03-22T01:34:19.668252+0000","last_active":"2024-03-22T01:35:16.744060+0000","last_peered":"2024-03-22T01:35:16.744060+0000","last_clean":"2024-03-22T01:35:16.744060+0000","last_became_active":"2024-03-22T01:34:19.667830+0000","last_became_peered":"2024-03-22T01:34:19.667830+0000","last_unstale":"2024-03-22T01:35:16.744060+0000","last_undegraded":"2024-03-22T01:35:16.744060+0000","last_fullsized":"2024-03-22T01:35:16.744060+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:08:29.602899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.746747+0000","last_change":"2024-03-22T01:34:14.473020+0000","last_active":"2024-03-22T01:35:16.746747+0000","last_peered":"2024-03-22T01:35:16.746747+0000","last_clean":"2024-03-22T01:35:16.746747+0000","last_became_active":"2024-03-22T01:34:14.472689+0000","last_became_peered":"2024-03-22T01:34:14.472689+0000","last_unstale":"2024-03-22T01:35:16.746747+0000","last_undegraded":"2024-03-22T01:35:16.746747+0000","last_fullsized":"2024-03-22T01:35:16.746747+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:04:19.724685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"93'2","reported_seq":26,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.416669+0000","last_change":"2024-03-22T01:34:15.491613+0000","last_active":"2024-03-22T01:35:45.416669+0000","last_peered":"2024-03-22T01:35:45.416669+0000","last_clean":"2024-03-22T01:35:45.416669+0000","last_became_active":"2024-03-22T01:34:15.490922+0000","last_became_peered":"2024-03-22T01:34:15.490922+0000","last_unstale":"2024-03-22T01:35:45.416669+0000","last_undegraded":"2024-03-22T01:35:45.416669+0000","last_fullsized":"2024-03-22T01:35:45.416669+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:09:33.177646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.748082+0000","last_change":"2024-03-22T01:34:19.534537+0000","last_active":"2024-03-22T01:35:16.748082+0000","last_peered":"2024-03-22T01:35:16.748082+0000","last_clean":"2024-03-22T01:35:16.748082+0000","last_became_active":"2024-03-22T01:34:19.534038+0000","last_became_peered":"2024-03-22T01:34:19.534038+0000","last_unstale":"2024-03-22T01:35:16.748082+0000","last_undegraded":"2024-03-22T01:35:16.748082+0000","last_fullsized":"2024-03-22T01:35:16.748082+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:59:26.020000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":26,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:17.204413+0000","last_change":"2024-03-22T01:34:14.464724+0000","last_active":"2024-03-22T01:35:17.204413+0000","last_peered":"2024-03-22T01:35:17.204413+0000","last_clean":"2024-03-22T01:35:17.204413+0000","last_became_active":"2024-03-22T01:34:14.464056+0000","last_became_peered":"2024-03-22T01:34:14.464056+0000","last_unstale":"2024-03-22T01:35:17.204413+0000","last_undegraded":"2024-03-22T01:35:17.204413+0000","last_fullsized":"2024-03-22T01:35:17.204413+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:49:39.098126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"93'9","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.457460+0000","last_change":"2024-03-22T01:34:15.488934+0000","last_active":"2024-03-22T01:35:45.457460+0000","last_peered":"2024-03-22T01:35:45.457460+0000","last_clean":"2024-03-22T01:35:45.457460+0000","last_became_active":"2024-03-22T01:34:15.488585+0000","last_became_peered":"2024-03-22T01:34:15.488585+0000","last_unstale":"2024-03-22T01:35:45.457460+0000","last_undegraded":"2024-03-22T01:35:45.457460+0000","last_fullsized":"2024-03-22T01:35:45.457460+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:59:04.554614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"91'1","reported_seq":19,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.743985+0000","last_change":"2024-03-22T01:34:19.534096+0000","last_active":"2024-03-22T01:35:16.743985+0000","last_peered":"2024-03-22T01:35:16.743985+0000","last_clean":"2024-03-22T01:35:16.743985+0000","last_became_active":"2024-03-22T01:34:19.533645+0000","last_became_peered":"2024-03-22T01:34:19.533645+0000","last_unstale":"2024-03-22T01:35:16.743985+0000","last_undegraded":"2024-03-22T01:35:16.743985+0000","last_fullsized":"2024-03-22T01:35:16.743985+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:50:21.259146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.752618+0000","last_change":"2024-03-22T01:34:14.469529+0000","last_active":"2024-03-22T01:35:16.752618+0000","last_peered":"2024-03-22T01:35:16.752618+0000","last_clean":"2024-03-22T01:35:16.752618+0000","last_became_active":"2024-03-22T01:34:14.469204+0000","last_became_peered":"2024-03-22T01:34:14.469204+0000","last_unstale":"2024-03-22T01:35:16.752618+0000","last_undegraded":"2024-03-22T01:35:16.752618+0000","last_fullsized":"2024-03-22T01:35:16.752618+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:39:57.048537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"93'8","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.454022+0000","last_change":"2024-03-22T01:34:15.501324+0000","last_active":"2024-03-22T01:35:45.454022+0000","last_peered":"2024-03-22T01:35:45.454022+0000","last_clean":"2024-03-22T01:35:45.454022+0000","last_became_active":"2024-03-22T01:34:15.501015+0000","last_became_peered":"2024-03-22T01:34:15.501015+0000","last_unstale":"2024-03-22T01:35:45.454022+0000","last_undegraded":"2024-03-22T01:35:45.454022+0000","last_fullsized":"2024-03-22T01:35:45.454022+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:35:27.673019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.743977+0000","last_change":"2024-03-22T01:34:19.667806+0000","last_active":"2024-03-22T01:35:16.743977+0000","last_peered":"2024-03-22T01:35:16.743977+0000","last_clean":"2024-03-22T01:35:16.743977+0000","last_became_active":"2024-03-22T01:34:19.667249+0000","last_became_peered":"2024-03-22T01:34:19.667249+0000","last_unstale":"2024-03-22T01:35:16.743977+0000","last_undegraded":"2024-03-22T01:35:16.743977+0000","last_fullsized":"2024-03-22T01:35:16.743977+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:44:22.476882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"86'2","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.748466+0000","last_change":"2024-03-22T01:34:14.471800+0000","last_active":"2024-03-22T01:35:16.748466+0000","last_peered":"2024-03-22T01:35:16.748466+0000","last_clean":"2024-03-22T01:35:16.748466+0000","last_became_active":"2024-03-22T01:34:14.471422+0000","last_became_peered":"2024-03-22T01:34:14.471422+0000","last_unstale":"2024-03-22T01:35:16.748466+0000","last_undegraded":"2024-03-22T01:35:16.748466+0000","last_fullsized":"2024-03-22T01:35:16.748466+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:25:58.546331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"93'5","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.452063+0000","last_change":"2024-03-22T01:34:15.487348+0000","last_active":"2024-03-22T01:35:45.452063+0000","last_peered":"2024-03-22T01:35:45.452063+0000","last_clean":"2024-03-22T01:35:45.452063+0000","last_became_active":"2024-03-22T01:34:15.486788+0000","last_became_peered":"2024-03-22T01:34:15.486788+0000","last_unstale":"2024-03-22T01:35:45.452063+0000","last_undegraded":"2024-03-22T01:35:45.452063+0000","last_fullsized":"2024-03-22T01:35:45.452063+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:10:31.489021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.746815+0000","last_change":"2024-03-22T01:34:19.521490+0000","last_active":"2024-03-22T01:35:16.746815+0000","last_peered":"2024-03-22T01:35:16.746815+0000","last_clean":"2024-03-22T01:35:16.746815+0000","last_became_active":"2024-03-22T01:34:19.521047+0000","last_became_peered":"2024-03-22T01:34:19.521047+0000","last_unstale":"2024-03-22T01:35:16.746815+0000","last_undegraded":"2024-03-22T01:35:16.746815+0000","last_fullsized":"2024-03-22T01:35:16.746815+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:36:50.176756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":11,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.750181+0000","last_change":"2024-03-22T01:35:16.749731+0000","last_active":"2024-03-22T01:35:16.750181+0000","last_peered":"2024-03-22T01:35:16.750181+0000","last_clean":"2024-03-22T01:35:16.750181+0000","last_became_active":"2024-03-22T01:35:16.749329+0000","last_became_peered":"2024-03-22T01:35:16.749329+0000","last_unstale":"2024-03-22T01:35:16.750181+0000","last_undegraded":"2024-03-22T01:35:16.750181+0000","last_fullsized":"2024-03-22T01:35:16.750181+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:23:27.386061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"93'7","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.451333+0000","last_change":"2024-03-22T01:34:15.491683+0000","last_active":"2024-03-22T01:35:45.451333+0000","last_peered":"2024-03-22T01:35:45.451333+0000","last_clean":"2024-03-22T01:35:45.451333+0000","last_became_active":"2024-03-22T01:34:15.491030+0000","last_became_peered":"2024-03-22T01:34:15.491030+0000","last_unstale":"2024-03-22T01:35:45.451333+0000","last_undegraded":"2024-03-22T01:35:45.451333+0000","last_fullsized":"2024-03-22T01:35:45.451333+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:02:10.115971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.745806+0000","last_change":"2024-03-22T01:34:19.528608+0000","last_active":"2024-03-22T01:35:16.745806+0000","last_peered":"2024-03-22T01:35:16.745806+0000","last_clean":"2024-03-22T01:35:16.745806+0000","last_became_active":"2024-03-22T01:34:19.528173+0000","last_became_peered":"2024-03-22T01:34:19.528173+0000","last_unstale":"2024-03-22T01:35:16.745806+0000","last_undegraded":"2024-03-22T01:35:16.745806+0000","last_fullsized":"2024-03-22T01:35:16.745806+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:36:02.193187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"86'1","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.744982+0000","last_change":"2024-03-22T01:34:14.467755+0000","last_active":"2024-03-22T01:35:16.744982+0000","last_peered":"2024-03-22T01:35:16.744982+0000","last_clean":"2024-03-22T01:35:16.744982+0000","last_became_active":"2024-03-22T01:34:14.467426+0000","last_became_peered":"2024-03-22T01:34:14.467426+0000","last_unstale":"2024-03-22T01:35:16.744982+0000","last_undegraded":"2024-03-22T01:35:16.744982+0000","last_fullsized":"2024-03-22T01:35:16.744982+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:16:35.436872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"93'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.454313+0000","last_change":"2024-03-22T01:35:16.748667+0000","last_active":"2024-03-22T01:35:45.454313+0000","last_peered":"2024-03-22T01:35:45.454313+0000","last_clean":"2024-03-22T01:35:45.454313+0000","last_became_active":"2024-03-22T01:35:16.748303+0000","last_became_peered":"2024-03-22T01:35:16.748303+0000","last_unstale":"2024-03-22T01:35:45.454313+0000","last_undegraded":"2024-03-22T01:35:45.454313+0000","last_fullsized":"2024-03-22T01:35:45.454313+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:40:55.207127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.744131+0000","last_change":"2024-03-22T01:34:19.523631+0000","last_active":"2024-03-22T01:35:16.744131+0000","last_peered":"2024-03-22T01:35:16.744131+0000","last_clean":"2024-03-22T01:35:16.744131+0000","last_became_active":"2024-03-22T01:34:19.522968+0000","last_became_peered":"2024-03-22T01:34:19.522968+0000","last_unstale":"2024-03-22T01:35:16.744131+0000","last_undegraded":"2024-03-22T01:35:16.744131+0000","last_fullsized":"2024-03-22T01:35:16.744131+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:27:24.182886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:16.743726+0000","last_change":"2024-03-22T01:34:14.464113+0000","last_active":"2024-03-22T01:35:16.743726+0000","last_peered":"2024-03-22T01:35:16.743726+0000","last_clean":"2024-03-22T01:35:16.743726+0000","last_became_active":"2024-03-22T01:34:14.463389+0000","last_became_peered":"2024-03-22T01:34:14.463389+0000","last_unstale":"2024-03-22T01:35:16.743726+0000","last_undegraded":"2024-03-22T01:35:16.743726+0000","last_fullsized":"2024-03-22T01:35:16.743726+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:08:42.376753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"93'9","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.454913+0000","last_change":"2024-03-22T01:34:15.492941+0000","last_active":"2024-03-22T01:35:45.454913+0000","last_peered":"2024-03-22T01:35:45.454913+0000","last_clean":"2024-03-22T01:35:45.454913+0000","last_became_active":"2024-03-22T01:34:15.492598+0000","last_became_peered":"2024-03-22T01:34:15.492598+0000","last_unstale":"2024-03-22T01:35:45.454913+0000","last_undegraded":"2024-03-22T01:35:45.454913+0000","last_fullsized":"2024-03-22T01:35:45.454913+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:22:35.502404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"93'9","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.454431+0000","last_change":"2024-03-22T01:34:15.494548+0000","last_active":"2024-03-22T01:35:45.454431+0000","last_peered":"2024-03-22T01:35:45.454431+0000","last_clean":"2024-03-22T01:35:45.454431+0000","last_became_active":"2024-03-22T01:34:15.494228+0000","last_became_peered":"2024-03-22T01:34:15.494228+0000","last_unstale":"2024-03-22T01:35:45.454431+0000","last_undegraded":"2024-03-22T01:35:45.454431+0000","last_fullsized":"2024-03-22T01:35:45.454431+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:43:26.011030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"93'12","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.452055+0000","last_change":"2024-03-22T01:34:15.490843+0000","last_active":"2024-03-22T01:35:45.452055+0000","last_peered":"2024-03-22T01:35:45.452055+0000","last_clean":"2024-03-22T01:35:45.452055+0000","last_became_active":"2024-03-22T01:34:15.490474+0000","last_became_peered":"2024-03-22T01:34:15.490474+0000","last_unstale":"2024-03-22T01:35:45.452055+0000","last_undegraded":"2024-03-22T01:35:45.452055+0000","last_fullsized":"2024-03-22T01:35:45.452055+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:11:31.488132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"93'8","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.447966+0000","last_change":"2024-03-22T01:34:15.502213+0000","last_active":"2024-03-22T01:35:45.447966+0000","last_peered":"2024-03-22T01:35:45.447966+0000","last_clean":"2024-03-22T01:35:45.447966+0000","last_became_active":"2024-03-22T01:34:15.501464+0000","last_became_peered":"2024-03-22T01:34:15.501464+0000","last_unstale":"2024-03-22T01:35:45.447966+0000","last_undegraded":"2024-03-22T01:35:45.447966+0000","last_fullsized":"2024-03-22T01:35:45.447966+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:38:02.090365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"93'7","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.455771+0000","last_change":"2024-03-22T01:34:15.481282+0000","last_active":"2024-03-22T01:35:45.455771+0000","last_peered":"2024-03-22T01:35:45.455771+0000","last_clean":"2024-03-22T01:35:45.455771+0000","last_became_active":"2024-03-22T01:34:15.480465+0000","last_became_peered":"2024-03-22T01:34:15.480465+0000","last_unstale":"2024-03-22T01:35:45.455771+0000","last_undegraded":"2024-03-22T01:35:45.455771+0000","last_fullsized":"2024-03-22T01:35:45.455771+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:55:59.398543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"93'4","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.457095+0000","last_change":"2024-03-22T01:34:15.480910+0000","last_active":"2024-03-22T01:35:45.457095+0000","last_peered":"2024-03-22T01:35:45.457095+0000","last_clean":"2024-03-22T01:35:45.457095+0000","last_became_active":"2024-03-22T01:34:15.480601+0000","last_became_peered":"2024-03-22T01:34:15.480601+0000","last_unstale":"2024-03-22T01:35:45.457095+0000","last_undegraded":"2024-03-22T01:35:45.457095+0000","last_fullsized":"2024-03-22T01:35:45.457095+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:19:51.563598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"93'6","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.453907+0000","last_change":"2024-03-22T01:34:15.490859+0000","last_active":"2024-03-22T01:35:45.453907+0000","last_peered":"2024-03-22T01:35:45.453907+0000","last_clean":"2024-03-22T01:35:45.453907+0000","last_became_active":"2024-03-22T01:34:15.490337+0000","last_became_peered":"2024-03-22T01:34:15.490337+0000","last_unstale":"2024-03-22T01:35:45.453907+0000","last_undegraded":"2024-03-22T01:35:45.453907+0000","last_fullsized":"2024-03-22T01:35:45.453907+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:46:07.072015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"93'15","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.446635+0000","last_change":"2024-03-22T01:34:15.495531+0000","last_active":"2024-03-22T01:35:45.446635+0000","last_peered":"2024-03-22T01:35:45.446635+0000","last_clean":"2024-03-22T01:35:45.446635+0000","last_became_active":"2024-03-22T01:34:15.495216+0000","last_became_peered":"2024-03-22T01:34:15.495216+0000","last_unstale":"2024-03-22T01:35:45.446635+0000","last_undegraded":"2024-03-22T01:35:45.446635+0000","last_fullsized":"2024-03-22T01:35:45.446635+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:39:22.575926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"93'2","reported_seq":26,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.447150+0000","last_change":"2024-03-22T01:34:15.499430+0000","last_active":"2024-03-22T01:35:45.447150+0000","last_peered":"2024-03-22T01:35:45.447150+0000","last_clean":"2024-03-22T01:35:45.447150+0000","last_became_active":"2024-03-22T01:34:15.499114+0000","last_became_peered":"2024-03-22T01:34:15.499114+0000","last_unstale":"2024-03-22T01:35:45.447150+0000","last_undegraded":"2024-03-22T01:35:45.447150+0000","last_fullsized":"2024-03-22T01:35:45.447150+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:23:01.781963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"93'6","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.459467+0000","last_change":"2024-03-22T01:34:15.485449+0000","last_active":"2024-03-22T01:35:45.459467+0000","last_peered":"2024-03-22T01:35:45.459467+0000","last_clean":"2024-03-22T01:35:45.459467+0000","last_became_active":"2024-03-22T01:34:15.485084+0000","last_became_peered":"2024-03-22T01:34:15.485084+0000","last_unstale":"2024-03-22T01:35:45.459467+0000","last_undegraded":"2024-03-22T01:35:45.459467+0000","last_fullsized":"2024-03-22T01:35:45.459467+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:53:25.632719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"93'8","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.450638+0000","last_change":"2024-03-22T01:34:15.495521+0000","last_active":"2024-03-22T01:35:45.450638+0000","last_peered":"2024-03-22T01:35:45.450638+0000","last_clean":"2024-03-22T01:35:45.450638+0000","last_became_active":"2024-03-22T01:34:15.495218+0000","last_became_peered":"2024-03-22T01:34:15.495218+0000","last_unstale":"2024-03-22T01:35:45.450638+0000","last_undegraded":"2024-03-22T01:35:45.450638+0000","last_fullsized":"2024-03-22T01:35:45.450638+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:04:32.569934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"93'10","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.452812+0000","last_change":"2024-03-22T01:34:15.489213+0000","last_active":"2024-03-22T01:35:45.452812+0000","last_peered":"2024-03-22T01:35:45.452812+0000","last_clean":"2024-03-22T01:35:45.452812+0000","last_became_active":"2024-03-22T01:34:15.488887+0000","last_became_peered":"2024-03-22T01:34:15.488887+0000","last_unstale":"2024-03-22T01:35:45.452812+0000","last_undegraded":"2024-03-22T01:35:45.452812+0000","last_fullsized":"2024-03-22T01:35:45.452812+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:53:39.983144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"93'11","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.457870+0000","last_change":"2024-03-22T01:34:15.485632+0000","last_active":"2024-03-22T01:35:45.457870+0000","last_peered":"2024-03-22T01:35:45.457870+0000","last_clean":"2024-03-22T01:35:45.457870+0000","last_became_active":"2024-03-22T01:34:15.485341+0000","last_became_peered":"2024-03-22T01:34:15.485341+0000","last_unstale":"2024-03-22T01:35:45.457870+0000","last_undegraded":"2024-03-22T01:35:45.457870+0000","last_fullsized":"2024-03-22T01:35:45.457870+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:33:00.248555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"93'8","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-22T01:35:45.451910+0000","last_change":"2024-03-22T01:34:15.483593+0000","last_active":"2024-03-22T01:35:45.451910+0000","last_peered":"2024-03-22T01:35:45.451910+0000","last_clean":"2024-03-22T01:35:45.451910+0000","last_became_active":"2024-03-22T01:34:15.483201+0000","last_became_peered":"2024-03-22T01:34:15.483201+0000","last_unstale":"2024-03-22T01:35:45.451910+0000","last_undegraded":"2024-03-22T01:35:45.451910+0000","last_fullsized":"2024-03-22T01:35:45.451910+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:03:10.955120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":64403,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":166,"num_read_kb":148,"num_write":84,"num_write_kb":93,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":193209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318306,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1040384,"data_stored":509947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579943,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37012,"kb_used_data":1676,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708140,"statfs":{"total":95995035648,"available":95957135360,"internally_reserved":0,"allocated":1716224,"data_stored":1160018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776172,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1048576,"data_stored":514625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037809,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1048576,"data_stored":510551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299446,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28620,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716532,"statfs":{"total":95995035648,"available":95965728768,"internally_reserved":0,"allocated":1642496,"data_stored":1104993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462971,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":505851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574931,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1110016,"data_stored":569456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738456,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1040384,"data_stored":505941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411407,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28104,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1114112,"data_stored":569490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149770,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1044480,"data_stored":505955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888134,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1044480,"data_stored":509969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724608,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36944,"kb_used_data":1608,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708208,"statfs":{"total":95995035648,"available":95957204992,"internally_reserved":0,"allocated":1646592,"data_stored":1100497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":63605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":63639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":63799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-22T01:35:58.407 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-22T01:35:58.407 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-22T01:35:58.408 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-22T01:35:58.408 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph health --format=json 2024-03-22T01:35:58.651 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:35:58.774 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:58 smithi072 ceph-mon[32414]: pgmap v344: 129 pgs: 129 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-22T01:35:58.774 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:35:58 smithi072 ceph-mon[32414]: from='client.15111 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-22T01:35:58.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:58 smithi089 ceph-mon[35770]: pgmap v344: 129 pgs: 129 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-22T01:35:58.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:35:58 smithi089 ceph-mon[35770]: from='client.15111 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-22T01:35:58.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:58 smithi098 ceph-mon[36468]: pgmap v344: 129 pgs: 129 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-22T01:35:58.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:35:58 smithi098 ceph-mon[36468]: from='client.15111 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-22T01:36:00.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:00.489+0000 7f51200c9700 1 -- 172.21.15.72:0/2964938012 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5118101f40 msgr2=0x7f511810e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:00.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:00.489+0000 7f51200c9700 1 --2- 172.21.15.72:0/2964938012 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5118101f40 0x7f511810e3d0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f5110009990 tx=0x7f5110009ca0 comp rx=0 tx=0).stop 2024-03-22T01:36:00.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:00.489+0000 7f51200c9700 1 -- 172.21.15.72:0/2964938012 shutdown_connections 2024-03-22T01:36:00.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:00.489+0000 7f51200c9700 1 --2- 172.21.15.72:0/2964938012 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5118101f40 0x7f511810e3d0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:00.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:00.489+0000 7f51200c9700 1 --2- 172.21.15.72:0/2964938012 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5118107590 0x7f5118107970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:00.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:00.489+0000 7f51200c9700 1 --2- 172.21.15.72:0/2964938012 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5118101580 0x7f5118101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:00.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:00.489+0000 7f51200c9700 1 -- 172.21.15.72:0/2964938012 >> 172.21.15.72:0/2964938012 conn(0x7f51180784b0 msgr2=0x7f51180788c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:00.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:00.489+0000 7f51200c9700 1 -- 172.21.15.72:0/2964938012 shutdown_connections 2024-03-22T01:36:00.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:00.489+0000 7f51200c9700 1 -- 172.21.15.72:0/2964938012 wait complete. 2024-03-22T01:36:00.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:00.490+0000 7f51200c9700 1 Processor -- start 2024-03-22T01:36:00.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:00.490+0000 7f51200c9700 1 -- start start 2024-03-22T01:36:00.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:00.490+0000 7f51200c9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5118101580 0x7f511819d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:00.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:00.490+0000 7f51200c9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5118101f40 0x7f511819ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:00.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:00.490+0000 7f51200c9700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5118107590 0x7f5118197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:00.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:00.490+0000 7f51200c9700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f5118111830 con 0x7f5118107590 2024-03-22T01:36:00.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:00.490+0000 7f51200c9700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f51181116b0 con 0x7f5118101580 2024-03-22T01:36:00.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:00.490+0000 7f51200c9700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f51181119b0 con 0x7f5118101f40 2024-03-22T01:36:00.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:00.490+0000 7f511e666700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5118107590 0x7f5118197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:00.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:00.490+0000 7f511e666700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5118107590 0x7f5118197a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57612/0 (socket says 172.21.15.72:57612) 2024-03-22T01:36:00.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:00.490+0000 7f511e666700 1 -- 172.21.15.72:0/3390329782 learned_addr learned my addr 172.21.15.72:0/3390329782 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:00.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:00.490+0000 7f511de65700 1 --2- 172.21.15.72:0/3390329782 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5118101580 0x7f511819d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:00.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:00.491+0000 7f511d664700 1 --2- 172.21.15.72:0/3390329782 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5118101f40 0x7f511819ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:00.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:00.491+0000 7f511e666700 1 -- 172.21.15.72:0/3390329782 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5118101f40 msgr2=0x7f511819ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:00.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:00.491+0000 7f511e666700 1 --2- 172.21.15.72:0/3390329782 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5118101f40 0x7f511819ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:00.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:00.491+0000 7f511e666700 1 -- 172.21.15.72:0/3390329782 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5118101580 msgr2=0x7f511819d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:00.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:00.491+0000 7f511e666700 1 --2- 172.21.15.72:0/3390329782 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5118101580 0x7f511819d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:00.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:00.491+0000 7f511e666700 1 -- 172.21.15.72:0/3390329782 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5118198190 con 0x7f5118107590 2024-03-22T01:36:00.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:00.491+0000 7f511de65700 1 --2- 172.21.15.72:0/3390329782 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5118101580 0x7f511819d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:36:00.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:00.491+0000 7f511e666700 1 --2- 172.21.15.72:0/3390329782 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5118107590 0x7f5118197a30 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f51100098e0 tx=0x7f511000bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:00.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:00.491+0000 7f5116ffd700 1 -- 172.21.15.72:0/3390329782 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5110029070 con 0x7f5118107590 2024-03-22T01:36:00.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:00.491+0000 7f51200c9700 1 -- 172.21.15.72:0/3390329782 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5118198420 con 0x7f5118107590 2024-03-22T01:36:00.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:00.491+0000 7f5116ffd700 1 -- 172.21.15.72:0/3390329782 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5110003d70 con 0x7f5118107590 2024-03-22T01:36:00.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:00.491+0000 7f5116ffd700 1 -- 172.21.15.72:0/3390329782 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51100246d0 con 0x7f5118107590 2024-03-22T01:36:00.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:00.491+0000 7f51200c9700 1 -- 172.21.15.72:0/3390329782 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f511806a330 con 0x7f5118107590 2024-03-22T01:36:00.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:00.492+0000 7f51200c9700 1 -- 172.21.15.72:0/3390329782 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5118069340 con 0x7f5118107590 2024-03-22T01:36:00.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:00.493+0000 7f5116ffd700 1 -- 172.21.15.72:0/3390329782 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7f511002c070 con 0x7f5118107590 2024-03-22T01:36:00.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:00.494+0000 7f5116ffd700 1 --2- 172.21.15.72:0/3390329782 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f51000a2fb0 0x7f51000a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:00.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:00.494+0000 7f511de65700 1 --2- 172.21.15.72:0/3390329782 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f51000a2fb0 0x7f51000a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:00.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:00.494+0000 7f5116ffd700 1 -- 172.21.15.72:0/3390329782 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f51100d2880 con 0x7f5118107590 2024-03-22T01:36:00.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:00.494+0000 7f511de65700 1 --2- 172.21.15.72:0/3390329782 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f51000a2fb0 0x7f51000a5470 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f5118198b70 tx=0x7f51080073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:00.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:00.498+0000 7f5116ffd700 1 -- 172.21.15.72:0/3390329782 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5110060740 con 0x7f5118107590 2024-03-22T01:36:00.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:00.737+0000 7f51200c9700 1 -- 172.21.15.72:0/3390329782 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f511806adc0 con 0x7f5118107590 2024-03-22T01:36:00.736 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:00 smithi072 ceph-mon[32414]: pgmap v345: 129 pgs: 129 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-22T01:36:00.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:00.737+0000 7f5116ffd700 1 -- 172.21.15.72:0/3390329782 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f5110013070 con 0x7f5118107590 2024-03-22T01:36:00.737 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:36:00.737 INFO:teuthology.orchestra.run.smithi072.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-22T01:36:00.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:00.739+0000 7f51200c9700 1 -- 172.21.15.72:0/3390329782 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f51000a2fb0 msgr2=0x7f51000a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:00.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:00.739+0000 7f51200c9700 1 --2- 172.21.15.72:0/3390329782 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f51000a2fb0 0x7f51000a5470 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f5118198b70 tx=0x7f51080073d0 comp rx=0 tx=0).stop 2024-03-22T01:36:00.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:00.739+0000 7f51200c9700 1 -- 172.21.15.72:0/3390329782 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5118107590 msgr2=0x7f5118197a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:00.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:00.739+0000 7f51200c9700 1 --2- 172.21.15.72:0/3390329782 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5118107590 0x7f5118197a30 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f51100098e0 tx=0x7f511000bab0 comp rx=0 tx=0).stop 2024-03-22T01:36:00.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:00.739+0000 7f51200c9700 1 -- 172.21.15.72:0/3390329782 shutdown_connections 2024-03-22T01:36:00.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:00.739+0000 7f51200c9700 1 --2- 172.21.15.72:0/3390329782 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f51000a2fb0 0x7f51000a5470 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:00.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:00.739+0000 7f51200c9700 1 --2- 172.21.15.72:0/3390329782 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5118107590 0x7f5118197a30 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:00.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:00.739+0000 7f51200c9700 1 --2- 172.21.15.72:0/3390329782 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5118101580 0x7f511819d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:00.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:00.739+0000 7f51200c9700 1 --2- 172.21.15.72:0/3390329782 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5118101f40 0x7f511819ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:00.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:00.739+0000 7f51200c9700 1 -- 172.21.15.72:0/3390329782 >> 172.21.15.72:0/3390329782 conn(0x7f51180784b0 msgr2=0x7f5118100150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:00.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:00.739+0000 7f51200c9700 1 -- 172.21.15.72:0/3390329782 shutdown_connections 2024-03-22T01:36:00.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:00.739+0000 7f51200c9700 1 -- 172.21.15.72:0/3390329782 wait complete. 2024-03-22T01:36:00.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:00 smithi089 ceph-mon[35770]: pgmap v345: 129 pgs: 129 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-22T01:36:00.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:00 smithi098 ceph-mon[36468]: pgmap v345: 129 pgs: 129 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-22T01:36:01.480 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-22T01:36:01.480 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T01:36:01.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.587+0000 7fe691c4e640 1 -- 172.21.15.72:0/3361411697 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe68c104f70 msgr2=0x7fe68c107360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:01.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.587+0000 7fe691c4e640 1 --2- 172.21.15.72:0/3361411697 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe68c104f70 0x7fe68c107360 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fe674009900 tx=0x7fe67402bc30 comp rx=0 tx=0).stop 2024-03-22T01:36:01.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.587+0000 7fe691c4e640 1 -- 172.21.15.72:0/3361411697 shutdown_connections 2024-03-22T01:36:01.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.587+0000 7fe691c4e640 1 --2- 172.21.15.72:0/3361411697 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe68c1078a0 0x7fe68c109cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:01.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.587+0000 7fe691c4e640 1 --2- 172.21.15.72:0/3361411697 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe68c104f70 0x7fe68c107360 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:01.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.587+0000 7fe691c4e640 1 --2- 172.21.15.72:0/3361411697 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe68c102640 0x7fe68c104a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:01.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.587+0000 7fe691c4e640 1 -- 172.21.15.72:0/3361411697 >> 172.21.15.72:0/3361411697 conn(0x7fe68c0fc030 msgr2=0x7fe68c0fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:01.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.587+0000 7fe691c4e640 1 -- 172.21.15.72:0/3361411697 shutdown_connections 2024-03-22T01:36:01.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.587+0000 7fe691c4e640 1 -- 172.21.15.72:0/3361411697 wait complete. 2024-03-22T01:36:01.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.587+0000 7fe691c4e640 1 Processor -- start 2024-03-22T01:36:01.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.588+0000 7fe691c4e640 1 -- start start 2024-03-22T01:36:01.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.588+0000 7fe691c4e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe68c102640 0x7fe68c197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:01.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.588+0000 7fe691c4e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe68c104f70 0x7fe68c198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:01.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.588+0000 7fe691c4e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe68c1078a0 0x7fe68c19f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:01.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.588+0000 7fe691c4e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fe68c10d850 con 0x7fe68c104f70 2024-03-22T01:36:01.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.588+0000 7fe691c4e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe68c10d6d0 con 0x7fe68c1078a0 2024-03-22T01:36:01.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.588+0000 7fe691c4e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe68c10d9d0 con 0x7fe68c102640 2024-03-22T01:36:01.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.588+0000 7fe68affd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe68c104f70 0x7fe68c198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:01.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.588+0000 7fe68affd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe68c104f70 0x7fe68c198460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:35488/0 (socket says 172.21.15.72:35488) 2024-03-22T01:36:01.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.588+0000 7fe68affd640 1 -- 172.21.15.72:0/1190076164 learned_addr learned my addr 172.21.15.72:0/1190076164 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:01.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.588+0000 7fe68bfff640 1 --2- 172.21.15.72:0/1190076164 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe68c1078a0 0x7fe68c19f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:01.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.588+0000 7fe68b7fe640 1 --2- 172.21.15.72:0/1190076164 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe68c102640 0x7fe68c197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:01.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.588+0000 7fe68affd640 1 -- 172.21.15.72:0/1190076164 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe68c102640 msgr2=0x7fe68c197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:01.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.588+0000 7fe68affd640 1 --2- 172.21.15.72:0/1190076164 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe68c102640 0x7fe68c197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:01.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.588+0000 7fe68affd640 1 -- 172.21.15.72:0/1190076164 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe68c1078a0 msgr2=0x7fe68c19f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:01.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.588+0000 7fe68affd640 1 --2- 172.21.15.72:0/1190076164 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe68c1078a0 0x7fe68c19f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:01.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.588+0000 7fe68affd640 1 -- 172.21.15.72:0/1190076164 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe68c19fb40 con 0x7fe68c104f70 2024-03-22T01:36:01.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.588+0000 7fe68affd640 1 --2- 172.21.15.72:0/1190076164 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe68c104f70 0x7fe68c198460 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fe674000c00 tx=0x7fe674005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:01.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.588+0000 7fe688ff9640 1 -- 172.21.15.72:0/1190076164 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe674041070 con 0x7fe68c104f70 2024-03-22T01:36:01.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.588+0000 7fe688ff9640 1 -- 172.21.15.72:0/1190076164 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe6740044d0 con 0x7fe68c104f70 2024-03-22T01:36:01.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.588+0000 7fe688ff9640 1 -- 172.21.15.72:0/1190076164 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe674034440 con 0x7fe68c104f70 2024-03-22T01:36:01.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.589+0000 7fe691c4e640 1 -- 172.21.15.72:0/1190076164 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe68c19fdd0 con 0x7fe68c104f70 2024-03-22T01:36:01.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.589+0000 7fe691c4e640 1 -- 172.21.15.72:0/1190076164 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe68c1a0210 con 0x7fe68c104f70 2024-03-22T01:36:01.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.590+0000 7fe691c4e640 1 -- 172.21.15.72:0/1190076164 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe68c0fdb20 con 0x7fe68c104f70 2024-03-22T01:36:01.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.590+0000 7fe688ff9640 1 -- 172.21.15.72:0/1190076164 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7fe674044070 con 0x7fe68c104f70 2024-03-22T01:36:01.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.591+0000 7fe688ff9640 1 --2- 172.21.15.72:0/1190076164 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe6780a2f20 0x7fe6780a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:01.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.591+0000 7fe68b7fe640 1 --2- 172.21.15.72:0/1190076164 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe6780a2f20 0x7fe6780a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:01.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.591+0000 7fe688ff9640 1 -- 172.21.15.72:0/1190076164 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fe6740f12f0 con 0x7fe68c104f70 2024-03-22T01:36:01.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.591+0000 7fe68b7fe640 1 --2- 172.21.15.72:0/1190076164 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe6780a2f20 0x7fe6780a53e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe67c005fd0 tx=0x7fe67c005e20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:01.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.595+0000 7fe688ff9640 1 -- 172.21.15.72:0/1190076164 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe6740bddc0 con 0x7fe68c104f70 2024-03-22T01:36:01.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.771+0000 7fe691c4e640 1 -- 172.21.15.72:0/1190076164 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe68c02d5d0 con 0x7fe68c104f70 2024-03-22T01:36:01.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.772+0000 7fe688ff9640 1 -- 172.21.15.72:0/1190076164 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24570 (secure 0 0 0) 0x7fe6740bd6c0 con 0x7fe68c104f70 2024-03-22T01:36:01.771 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:36:01.771 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":93,"fsid":"022a99bc-e7eb-11ee-95cd-87774f69a715","created":"2024-03-22T01:26:23.893672+0000","modified":"2024-03-22T01:35:16.738619+0000","last_up_change":"2024-03-22T01:33:25.223684+0000","last_in_change":"2024-03-22T01:33:05.010331+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T01:29:37.438503+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-22T01:34:12.610422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-22T01:34:13.761686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-22T01:34:17.507039+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"16c15bb0-db92-4191-9063-38b4890e81e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6803","nonce":661906791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6805","nonce":661906791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6809","nonce":661906791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6807","nonce":661906791}]},"public_addr":"172.21.15.72:6803/661906791","cluster_addr":"172.21.15.72:6805/661906791","heartbeat_back_addr":"172.21.15.72:6809/661906791","heartbeat_front_addr":"172.21.15.72:6807/661906791","state":["exists","up"]},{"osd":1,"uuid":"981290ba-97dc-496b-aa22-d51704184b10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6801","nonce":2870590550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6803","nonce":2870590550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6807","nonce":2870590550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6805","nonce":2870590550}]},"public_addr":"172.21.15.89:6801/2870590550","cluster_addr":"172.21.15.89:6803/2870590550","heartbeat_back_addr":"172.21.15.89:6807/2870590550","heartbeat_front_addr":"172.21.15.89:6805/2870590550","state":["exists","up"]},{"osd":2,"uuid":"f07ed658-f545-46a1-8edd-8b1d5d49885c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6801","nonce":3865778118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6803","nonce":3865778118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6807","nonce":3865778118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6805","nonce":3865778118}]},"public_addr":"172.21.15.98:6801/3865778118","cluster_addr":"172.21.15.98:6803/3865778118","heartbeat_back_addr":"172.21.15.98:6807/3865778118","heartbeat_front_addr":"172.21.15.98:6805/3865778118","state":["exists","up"]},{"osd":3,"uuid":"ccc4bc40-2c48-46d2-8548-0c05b44f9f1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6811","nonce":1865951395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6813","nonce":1865951395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6817","nonce":1865951395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6815","nonce":1865951395}]},"public_addr":"172.21.15.72:6811/1865951395","cluster_addr":"172.21.15.72:6813/1865951395","heartbeat_back_addr":"172.21.15.72:6817/1865951395","heartbeat_front_addr":"172.21.15.72:6815/1865951395","state":["exists","up"]},{"osd":4,"uuid":"1812439f-810a-4ed2-9395-c00c9a085a13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6809","nonce":2683760795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6811","nonce":2683760795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6815","nonce":2683760795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6813","nonce":2683760795}]},"public_addr":"172.21.15.89:6809/2683760795","cluster_addr":"172.21.15.89:6811/2683760795","heartbeat_back_addr":"172.21.15.89:6815/2683760795","heartbeat_front_addr":"172.21.15.89:6813/2683760795","state":["exists","up"]},{"osd":5,"uuid":"e78954d8-f4e3-4d58-8536-d2d59c843386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6809","nonce":2635838572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6811","nonce":2635838572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6815","nonce":2635838572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6813","nonce":2635838572}]},"public_addr":"172.21.15.98:6809/2635838572","cluster_addr":"172.21.15.98:6811/2635838572","heartbeat_back_addr":"172.21.15.98:6815/2635838572","heartbeat_front_addr":"172.21.15.98:6813/2635838572","state":["exists","up"]},{"osd":6,"uuid":"dc6c8be2-b64c-4959-8015-7b0ea6d25599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6819","nonce":3158982661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6821","nonce":3158982661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6825","nonce":3158982661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6823","nonce":3158982661}]},"public_addr":"172.21.15.72:6819/3158982661","cluster_addr":"172.21.15.72:6821/3158982661","heartbeat_back_addr":"172.21.15.72:6825/3158982661","heartbeat_front_addr":"172.21.15.72:6823/3158982661","state":["exists","up"]},{"osd":7,"uuid":"7a73469f-cf3a-4306-ae93-ea46e21ebbbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6817","nonce":3075913234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6819","nonce":3075913234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6823","nonce":3075913234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6821","nonce":3075913234}]},"public_addr":"172.21.15.89:6817/3075913234","cluster_addr":"172.21.15.89:6819/3075913234","heartbeat_back_addr":"172.21.15.89:6823/3075913234","heartbeat_front_addr":"172.21.15.89:6821/3075913234","state":["exists","up"]},{"osd":8,"uuid":"c4b13bee-ea4d-4344-95fb-2622ecbb8196","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6817","nonce":1813255577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6819","nonce":1813255577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6823","nonce":1813255577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6821","nonce":1813255577}]},"public_addr":"172.21.15.98:6817/1813255577","cluster_addr":"172.21.15.98:6819/1813255577","heartbeat_back_addr":"172.21.15.98:6823/1813255577","heartbeat_front_addr":"172.21.15.98:6821/1813255577","state":["exists","up"]},{"osd":9,"uuid":"eefb0a44-4a36-4e55-9d04-0501bf718db6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6827","nonce":3377749206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6829","nonce":3377749206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6832","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6833","nonce":3377749206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6831","nonce":3377749206}]},"public_addr":"172.21.15.72:6827/3377749206","cluster_addr":"172.21.15.72:6829/3377749206","heartbeat_back_addr":"172.21.15.72:6833/3377749206","heartbeat_front_addr":"172.21.15.72:6831/3377749206","state":["exists","up"]},{"osd":10,"uuid":"8741d25f-40dd-40d7-b040-5cc5e411eb98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6825","nonce":885058786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6827","nonce":885058786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6831","nonce":885058786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6829","nonce":885058786}]},"public_addr":"172.21.15.89:6825/885058786","cluster_addr":"172.21.15.89:6827/885058786","heartbeat_back_addr":"172.21.15.89:6831/885058786","heartbeat_front_addr":"172.21.15.89:6829/885058786","state":["exists","up"]},{"osd":11,"uuid":"8d02e957-eda4-4184-b5c6-5eb3b8b38060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6825","nonce":1517095838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6827","nonce":1517095838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6831","nonce":1517095838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6829","nonce":1517095838}]},"public_addr":"172.21.15.98:6825/1517095838","cluster_addr":"172.21.15.98:6827/1517095838","heartbeat_back_addr":"172.21.15.98:6831/1517095838","heartbeat_front_addr":"172.21.15.98:6829/1517095838","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:28:47.307652+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:11.037135+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:34.619188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:59.350738+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:30:23.787337+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:30:48.994154+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:31:13.690350+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:31:39.628116+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:04.723912+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:30.689741+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:57.229508+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:33:23.154537+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:0/121087196":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/2451857761":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/1228477995":"2024-03-23T01:27:15.339809+0000","172.21.15.72:6801/1513232913":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/3558958264":"2024-03-23T01:26:50.074871+0000","172.21.15.72:0/1510187198":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6800/3518012916":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6801/3518012916":"2024-03-23T01:26:50.074871+0000","172.21.15.72:0/1372801041":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6800/1513232913":"2024-03-23T01:27:15.339809+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T01:36:01.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.774+0000 7fe691c4e640 1 -- 172.21.15.72:0/1190076164 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe6780a2f20 msgr2=0x7fe6780a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:01.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.774+0000 7fe691c4e640 1 --2- 172.21.15.72:0/1190076164 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe6780a2f20 0x7fe6780a53e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe67c005fd0 tx=0x7fe67c005e20 comp rx=0 tx=0).stop 2024-03-22T01:36:01.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.774+0000 7fe691c4e640 1 -- 172.21.15.72:0/1190076164 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe68c104f70 msgr2=0x7fe68c198460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:01.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.774+0000 7fe691c4e640 1 --2- 172.21.15.72:0/1190076164 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe68c104f70 0x7fe68c198460 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fe674000c00 tx=0x7fe674005d50 comp rx=0 tx=0).stop 2024-03-22T01:36:01.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.774+0000 7fe691c4e640 1 -- 172.21.15.72:0/1190076164 shutdown_connections 2024-03-22T01:36:01.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.774+0000 7fe691c4e640 1 --2- 172.21.15.72:0/1190076164 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe6780a2f20 0x7fe6780a53e0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:01.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.774+0000 7fe691c4e640 1 --2- 172.21.15.72:0/1190076164 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe68c1078a0 0x7fe68c19f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:01.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.774+0000 7fe691c4e640 1 --2- 172.21.15.72:0/1190076164 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe68c104f70 0x7fe68c198460 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:01.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.774+0000 7fe691c4e640 1 --2- 172.21.15.72:0/1190076164 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe68c102640 0x7fe68c197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:01.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.774+0000 7fe691c4e640 1 -- 172.21.15.72:0/1190076164 >> 172.21.15.72:0/1190076164 conn(0x7fe68c0fc030 msgr2=0x7fe68c103060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:01.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.774+0000 7fe691c4e640 1 -- 172.21.15.72:0/1190076164 shutdown_connections 2024-03-22T01:36:01.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.774+0000 7fe691c4e640 1 -- 172.21.15.72:0/1190076164 wait complete. 2024-03-22T01:36:01.783 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:01 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/3390329782' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-22T01:36:01.785 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-22T01:29:37.438503+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-22T01:34:12.610422+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-22T01:34:13.761686+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-22T01:34:17.507039+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-22T01:36:01.786 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-22T01:36:01.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:01 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/3390329782' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-22T01:36:01.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.891+0000 7f98f0d16640 1 -- 172.21.15.72:0/2345259651 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f98ec104f60 msgr2=0x7f98ec107350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:01.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.891+0000 7f98f0d16640 1 --2- 172.21.15.72:0/2345259651 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f98ec104f60 0x7f98ec107350 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f98d4009900 tx=0x7f98d402bc30 comp rx=0 tx=0).stop 2024-03-22T01:36:01.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.891+0000 7f98f0d16640 1 -- 172.21.15.72:0/2345259651 shutdown_connections 2024-03-22T01:36:01.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.891+0000 7f98f0d16640 1 --2- 172.21.15.72:0/2345259651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98ec107890 0x7f98ec109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:01.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.891+0000 7f98f0d16640 1 --2- 172.21.15.72:0/2345259651 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f98ec104f60 0x7f98ec107350 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:01.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.891+0000 7f98f0d16640 1 --2- 172.21.15.72:0/2345259651 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f98ec102630 0x7f98ec104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:01.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.891+0000 7f98f0d16640 1 -- 172.21.15.72:0/2345259651 >> 172.21.15.72:0/2345259651 conn(0x7f98ec0fc020 msgr2=0x7f98ec0fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:01.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.891+0000 7f98f0d16640 1 -- 172.21.15.72:0/2345259651 shutdown_connections 2024-03-22T01:36:01.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.891+0000 7f98f0d16640 1 -- 172.21.15.72:0/2345259651 wait complete. 2024-03-22T01:36:01.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.892+0000 7f98f0d16640 1 Processor -- start 2024-03-22T01:36:01.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.892+0000 7f98f0d16640 1 -- start start 2024-03-22T01:36:01.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.892+0000 7f98f0d16640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f98ec102630 0x7f98ec197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:01.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.892+0000 7f98f0d16640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f98ec104f60 0x7f98ec198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:01.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.892+0000 7f98f0d16640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98ec107890 0x7f98ec19f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:01.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.892+0000 7f98f0d16640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f98ec10d820 con 0x7f98ec104f60 2024-03-22T01:36:01.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.892+0000 7f98f0d16640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f98ec10d6a0 con 0x7f98ec102630 2024-03-22T01:36:01.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.892+0000 7f98f0d16640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f98ec10d9a0 con 0x7f98ec107890 2024-03-22T01:36:01.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.892+0000 7f98e3fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f98ec104f60 0x7f98ec198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:01.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.892+0000 7f98e3fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f98ec104f60 0x7f98ec198460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:35496/0 (socket says 172.21.15.72:35496) 2024-03-22T01:36:01.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.892+0000 7f98e3fff640 1 -- 172.21.15.72:0/1912949128 learned_addr learned my addr 172.21.15.72:0/1912949128 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:01.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.892+0000 7f98ea59c640 1 --2- 172.21.15.72:0/1912949128 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f98ec102630 0x7f98ec197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:01.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.892+0000 7f98e3fff640 1 -- 172.21.15.72:0/1912949128 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98ec107890 msgr2=0x7f98ec19f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:01.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.892+0000 7f98e3fff640 1 --2- 172.21.15.72:0/1912949128 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98ec107890 0x7f98ec19f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:01.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.892+0000 7f98e3fff640 1 -- 172.21.15.72:0/1912949128 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f98ec102630 msgr2=0x7f98ec197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:01.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.892+0000 7f98e3fff640 1 --2- 172.21.15.72:0/1912949128 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f98ec102630 0x7f98ec197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:01.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.892+0000 7f98e3fff640 1 -- 172.21.15.72:0/1912949128 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98ec19fb40 con 0x7f98ec104f60 2024-03-22T01:36:01.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.893+0000 7f98e3fff640 1 --2- 172.21.15.72:0/1912949128 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f98ec104f60 0x7f98ec198460 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f98d4000c00 tx=0x7f98d4005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:01.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.893+0000 7f98e37fe640 1 -- 172.21.15.72:0/1912949128 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98d4041070 con 0x7f98ec104f60 2024-03-22T01:36:01.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.893+0000 7f98f0d16640 1 -- 172.21.15.72:0/1912949128 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98ec19fdd0 con 0x7f98ec104f60 2024-03-22T01:36:01.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.893+0000 7f98e37fe640 1 -- 172.21.15.72:0/1912949128 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98d40044d0 con 0x7f98ec104f60 2024-03-22T01:36:01.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.893+0000 7f98e37fe640 1 -- 172.21.15.72:0/1912949128 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98d4034440 con 0x7f98ec104f60 2024-03-22T01:36:01.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.893+0000 7f98f0d16640 1 -- 172.21.15.72:0/1912949128 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98ec1a0280 con 0x7f98ec104f60 2024-03-22T01:36:01.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.894+0000 7f98f0d16640 1 -- 172.21.15.72:0/1912949128 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98ec0fdb10 con 0x7f98ec104f60 2024-03-22T01:36:01.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.895+0000 7f98e37fe640 1 -- 172.21.15.72:0/1912949128 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7f98d4044070 con 0x7f98ec104f60 2024-03-22T01:36:01.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.895+0000 7f98e37fe640 1 --2- 172.21.15.72:0/1912949128 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f98d80a2f20 0x7f98d80a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:01.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.896+0000 7f98ea59c640 1 --2- 172.21.15.72:0/1912949128 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f98d80a2f20 0x7f98d80a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:01.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.896+0000 7f98e37fe640 1 -- 172.21.15.72:0/1912949128 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f98d40f1480 con 0x7f98ec104f60 2024-03-22T01:36:01.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.896+0000 7f98ea59c640 1 --2- 172.21.15.72:0/1912949128 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f98d80a2f20 0x7f98d80a53e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f98dc0095f0 tx=0x7f98dc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:01.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:01.899+0000 7f98e37fe640 1 -- 172.21.15.72:0/1912949128 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f98d40bdf50 con 0x7f98ec104f60 2024-03-22T01:36:01.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:01 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/3390329782' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-22T01:36:02.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.075+0000 7f98f0d16640 1 -- 172.21.15.72:0/1912949128 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f98ec02d3b0 con 0x7f98ec104f60 2024-03-22T01:36:02.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.075+0000 7f98e37fe640 1 -- 172.21.15.72:0/1912949128 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7f98d40bd850 con 0x7f98ec104f60 2024-03-22T01:36:02.074 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-22T01:36:02.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.077+0000 7f98f0d16640 1 -- 172.21.15.72:0/1912949128 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f98d80a2f20 msgr2=0x7f98d80a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:02.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.077+0000 7f98f0d16640 1 --2- 172.21.15.72:0/1912949128 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f98d80a2f20 0x7f98d80a53e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f98dc0095f0 tx=0x7f98dc005f90 comp rx=0 tx=0).stop 2024-03-22T01:36:02.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.077+0000 7f98f0d16640 1 -- 172.21.15.72:0/1912949128 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f98ec104f60 msgr2=0x7f98ec198460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:02.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.077+0000 7f98f0d16640 1 --2- 172.21.15.72:0/1912949128 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f98ec104f60 0x7f98ec198460 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f98d4000c00 tx=0x7f98d4005d50 comp rx=0 tx=0).stop 2024-03-22T01:36:02.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.077+0000 7f98f0d16640 1 -- 172.21.15.72:0/1912949128 shutdown_connections 2024-03-22T01:36:02.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.077+0000 7f98f0d16640 1 --2- 172.21.15.72:0/1912949128 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f98d80a2f20 0x7f98d80a53e0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:02.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.077+0000 7f98f0d16640 1 --2- 172.21.15.72:0/1912949128 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98ec107890 0x7f98ec19f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:02.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.077+0000 7f98f0d16640 1 --2- 172.21.15.72:0/1912949128 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f98ec104f60 0x7f98ec198460 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:02.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.077+0000 7f98f0d16640 1 --2- 172.21.15.72:0/1912949128 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f98ec102630 0x7f98ec197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:02.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.077+0000 7f98f0d16640 1 -- 172.21.15.72:0/1912949128 >> 172.21.15.72:0/1912949128 conn(0x7f98ec0fc020 msgr2=0x7f98ec102f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:02.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.078+0000 7f98f0d16640 1 -- 172.21.15.72:0/1912949128 shutdown_connections 2024-03-22T01:36:02.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.078+0000 7f98f0d16640 1 -- 172.21.15.72:0/1912949128 wait complete. 2024-03-22T01:36:02.088 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-22T01:36:02.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.193+0000 7f5ea31d7640 1 -- 172.21.15.72:0/2632456098 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e9c0ffea0 msgr2=0x7f5e9c1002c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:02.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.193+0000 7f5ea31d7640 1 --2- 172.21.15.72:0/2632456098 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e9c0ffea0 0x7f5e9c1002c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f5e98009cd0 tx=0x7f5e9802c0f0 comp rx=0 tx=0).stop 2024-03-22T01:36:02.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.193+0000 7f5ea31d7640 1 -- 172.21.15.72:0/2632456098 shutdown_connections 2024-03-22T01:36:02.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.193+0000 7f5ea31d7640 1 --2- 172.21.15.72:0/2632456098 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e9c102f10 0x7f5e9c103390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:02.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.193+0000 7f5ea31d7640 1 --2- 172.21.15.72:0/2632456098 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e9c1038d0 0x7f5e9c10a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:02.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.193+0000 7f5ea31d7640 1 --2- 172.21.15.72:0/2632456098 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e9c0ffea0 0x7f5e9c1002c0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:02.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.193+0000 7f5ea31d7640 1 -- 172.21.15.72:0/2632456098 >> 172.21.15.72:0/2632456098 conn(0x7f5e9c0fb7b0 msgr2=0x7f5e9c0fdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:02.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.193+0000 7f5ea31d7640 1 -- 172.21.15.72:0/2632456098 shutdown_connections 2024-03-22T01:36:02.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.193+0000 7f5ea31d7640 1 -- 172.21.15.72:0/2632456098 wait complete. 2024-03-22T01:36:02.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.193+0000 7f5ea31d7640 1 Processor -- start 2024-03-22T01:36:02.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.193+0000 7f5ea31d7640 1 -- start start 2024-03-22T01:36:02.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.193+0000 7f5ea31d7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e9c0ffea0 0x7f5e9c197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:02.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.193+0000 7f5ea31d7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e9c102f10 0x7f5e9c198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:02.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.193+0000 7f5ea31d7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e9c1038d0 0x7f5e9c19f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:02.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.193+0000 7f5ea31d7640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f5e9c10d830 con 0x7f5e9c1038d0 2024-03-22T01:36:02.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.193+0000 7f5ea31d7640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5e9c10d6b0 con 0x7f5e9c102f10 2024-03-22T01:36:02.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.193+0000 7f5ea31d7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5e9c10d9b0 con 0x7f5e9c0ffea0 2024-03-22T01:36:02.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.193+0000 7f5ea1774640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e9c1038d0 0x7f5e9c19f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:02.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.194+0000 7f5ea1774640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e9c1038d0 0x7f5e9c19f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:35518/0 (socket says 172.21.15.72:35518) 2024-03-22T01:36:02.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.194+0000 7f5ea1774640 1 -- 172.21.15.72:0/104641279 learned_addr learned my addr 172.21.15.72:0/104641279 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:02.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.194+0000 7f5e93fff640 1 --2- 172.21.15.72:0/104641279 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e9c102f10 0x7f5e9c198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:02.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.194+0000 7f5ea0f73640 1 --2- 172.21.15.72:0/104641279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e9c0ffea0 0x7f5e9c197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:02.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.194+0000 7f5ea1774640 1 -- 172.21.15.72:0/104641279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e9c0ffea0 msgr2=0x7f5e9c197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:02.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.194+0000 7f5ea1774640 1 --2- 172.21.15.72:0/104641279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e9c0ffea0 0x7f5e9c197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:02.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.194+0000 7f5ea1774640 1 -- 172.21.15.72:0/104641279 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e9c102f10 msgr2=0x7f5e9c198480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:02.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.194+0000 7f5ea1774640 1 --2- 172.21.15.72:0/104641279 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e9c102f10 0x7f5e9c198480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:02.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.194+0000 7f5ea1774640 1 -- 172.21.15.72:0/104641279 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e9c19fb60 con 0x7f5e9c1038d0 2024-03-22T01:36:02.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.194+0000 7f5ea0f73640 1 --2- 172.21.15.72:0/104641279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e9c0ffea0 0x7f5e9c197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T01:36:02.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.194+0000 7f5ea1774640 1 --2- 172.21.15.72:0/104641279 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e9c1038d0 0x7f5e9c19f4f0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f5e9400c970 tx=0x7f5e9400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:02.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.194+0000 7f5e91ffb640 1 -- 172.21.15.72:0/104641279 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e94007bb0 con 0x7f5e9c1038d0 2024-03-22T01:36:02.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.194+0000 7f5e91ffb640 1 -- 172.21.15.72:0/104641279 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e94007d50 con 0x7f5e9c1038d0 2024-03-22T01:36:02.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.194+0000 7f5ea31d7640 1 -- 172.21.15.72:0/104641279 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e9c19fe50 con 0x7f5e9c1038d0 2024-03-22T01:36:02.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.194+0000 7f5e91ffb640 1 -- 172.21.15.72:0/104641279 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e9400f450 con 0x7f5e9c1038d0 2024-03-22T01:36:02.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.194+0000 7f5ea31d7640 1 -- 172.21.15.72:0/104641279 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e9c1a0310 con 0x7f5e9c1038d0 2024-03-22T01:36:02.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.195+0000 7f5ea31d7640 1 -- 172.21.15.72:0/104641279 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e9c1090c0 con 0x7f5e9c1038d0 2024-03-22T01:36:02.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.196+0000 7f5e91ffb640 1 -- 172.21.15.72:0/104641279 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7f5e94004f60 con 0x7f5e9c1038d0 2024-03-22T01:36:02.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.197+0000 7f5e91ffb640 1 --2- 172.21.15.72:0/104641279 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f5e840a2fa0 0x7f5e840a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:02.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.197+0000 7f5ea0f73640 1 --2- 172.21.15.72:0/104641279 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f5e840a2fa0 0x7f5e840a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:02.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.197+0000 7f5e91ffb640 1 -- 172.21.15.72:0/104641279 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f5e940cf300 con 0x7f5e9c1038d0 2024-03-22T01:36:02.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.197+0000 7f5ea0f73640 1 --2- 172.21.15.72:0/104641279 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f5e840a2fa0 0x7f5e840a5460 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5e98006010 tx=0x7f5e98009940 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:02.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.201+0000 7f5e91ffb640 1 -- 172.21.15.72:0/104641279 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5e94014030 con 0x7f5e9c1038d0 2024-03-22T01:36:02.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.374+0000 7f5ea31d7640 1 -- 172.21.15.72:0/104641279 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f5e9c02d560 con 0x7f5e9c1038d0 2024-03-22T01:36:02.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.375+0000 7f5e91ffb640 1 -- 172.21.15.72:0/104641279 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f5e9409be70 con 0x7f5e9c1038d0 2024-03-22T01:36:02.374 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-03-22T01:36:02.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.376+0000 7f5ea31d7640 1 -- 172.21.15.72:0/104641279 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f5e840a2fa0 msgr2=0x7f5e840a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:02.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.376+0000 7f5ea31d7640 1 --2- 172.21.15.72:0/104641279 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f5e840a2fa0 0x7f5e840a5460 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5e98006010 tx=0x7f5e98009940 comp rx=0 tx=0).stop 2024-03-22T01:36:02.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.376+0000 7f5ea31d7640 1 -- 172.21.15.72:0/104641279 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e9c1038d0 msgr2=0x7f5e9c19f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:02.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.376+0000 7f5ea31d7640 1 --2- 172.21.15.72:0/104641279 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e9c1038d0 0x7f5e9c19f4f0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f5e9400c970 tx=0x7f5e9400ce30 comp rx=0 tx=0).stop 2024-03-22T01:36:02.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.377+0000 7f5ea31d7640 1 -- 172.21.15.72:0/104641279 shutdown_connections 2024-03-22T01:36:02.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.377+0000 7f5ea31d7640 1 --2- 172.21.15.72:0/104641279 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f5e840a2fa0 0x7f5e840a5460 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:02.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.377+0000 7f5ea31d7640 1 --2- 172.21.15.72:0/104641279 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e9c102f10 0x7f5e9c198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:02.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.377+0000 7f5ea31d7640 1 --2- 172.21.15.72:0/104641279 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e9c1038d0 0x7f5e9c19f4f0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:02.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.377+0000 7f5ea31d7640 1 --2- 172.21.15.72:0/104641279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e9c0ffea0 0x7f5e9c197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:02.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.377+0000 7f5ea31d7640 1 -- 172.21.15.72:0/104641279 >> 172.21.15.72:0/104641279 conn(0x7f5e9c0fb7b0 msgr2=0x7f5e9c106290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:02.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.377+0000 7f5ea31d7640 1 -- 172.21.15.72:0/104641279 shutdown_connections 2024-03-22T01:36:02.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.377+0000 7f5ea31d7640 1 -- 172.21.15.72:0/104641279 wait complete. 2024-03-22T01:36:02.387 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-22T01:36:02.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.492+0000 7f0a13525640 1 -- 172.21.15.72:0/2621736406 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a0c104fe0 msgr2=0x7f0a0c1073d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:02.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.492+0000 7f0a13525640 1 --2- 172.21.15.72:0/2621736406 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a0c104fe0 0x7f0a0c1073d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0a00009900 tx=0x7f0a0002bc30 comp rx=0 tx=0).stop 2024-03-22T01:36:02.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.492+0000 7f0a13525640 1 -- 172.21.15.72:0/2621736406 shutdown_connections 2024-03-22T01:36:02.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.492+0000 7f0a13525640 1 --2- 172.21.15.72:0/2621736406 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a0c104fe0 0x7f0a0c1073d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:02.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.492+0000 7f0a13525640 1 --2- 172.21.15.72:0/2621736406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a0c107910 0x7f0a0c109d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:02.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.492+0000 7f0a13525640 1 --2- 172.21.15.72:0/2621736406 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0a0c1026b0 0x7f0a0c104aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:02.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.492+0000 7f0a13525640 1 -- 172.21.15.72:0/2621736406 >> 172.21.15.72:0/2621736406 conn(0x7f0a0c0fc020 msgr2=0x7f0a0c0fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:02.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.493+0000 7f0a13525640 1 -- 172.21.15.72:0/2621736406 shutdown_connections 2024-03-22T01:36:02.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.493+0000 7f0a13525640 1 -- 172.21.15.72:0/2621736406 wait complete. 2024-03-22T01:36:02.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.493+0000 7f0a13525640 1 Processor -- start 2024-03-22T01:36:02.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.493+0000 7f0a13525640 1 -- start start 2024-03-22T01:36:02.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.493+0000 7f0a13525640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a0c1026b0 0x7f0a0c197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:02.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.493+0000 7f0a13525640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0a0c104fe0 0x7f0a0c198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:02.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.493+0000 7f0a13525640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a0c107910 0x7f0a0c19f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:02.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.493+0000 7f0a13525640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f0a0c10d830 con 0x7f0a0c104fe0 2024-03-22T01:36:02.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.493+0000 7f0a13525640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f0a0c10d6b0 con 0x7f0a0c107910 2024-03-22T01:36:02.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.493+0000 7f0a13525640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0a0c10d9b0 con 0x7f0a0c1026b0 2024-03-22T01:36:02.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.493+0000 7f0a10ac0640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0a0c104fe0 0x7f0a0c198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:02.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.493+0000 7f0a10ac0640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0a0c104fe0 0x7f0a0c198470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:35548/0 (socket says 172.21.15.72:35548) 2024-03-22T01:36:02.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.493+0000 7f0a10ac0640 1 -- 172.21.15.72:0/402876385 learned_addr learned my addr 172.21.15.72:0/402876385 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:02.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.493+0000 7f0a11ac2640 1 --2- 172.21.15.72:0/402876385 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a0c107910 0x7f0a0c19f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:02.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.494+0000 7f0a112c1640 1 --2- 172.21.15.72:0/402876385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a0c1026b0 0x7f0a0c197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:02.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.494+0000 7f0a10ac0640 1 -- 172.21.15.72:0/402876385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a0c1026b0 msgr2=0x7f0a0c197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:02.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.494+0000 7f0a10ac0640 1 --2- 172.21.15.72:0/402876385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a0c1026b0 0x7f0a0c197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:02.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.494+0000 7f0a10ac0640 1 -- 172.21.15.72:0/402876385 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a0c107910 msgr2=0x7f0a0c19f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:02.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.494+0000 7f0a10ac0640 1 --2- 172.21.15.72:0/402876385 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a0c107910 0x7f0a0c19f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:02.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.494+0000 7f0a10ac0640 1 -- 172.21.15.72:0/402876385 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a0c19fb50 con 0x7f0a0c104fe0 2024-03-22T01:36:02.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.494+0000 7f0a112c1640 1 --2- 172.21.15.72:0/402876385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a0c1026b0 0x7f0a0c197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T01:36:02.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.494+0000 7f0a10ac0640 1 --2- 172.21.15.72:0/402876385 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0a0c104fe0 0x7f0a0c198470 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f0a00000c00 tx=0x7f0a00005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:02.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.494+0000 7f0a0a7fc640 1 -- 172.21.15.72:0/402876385 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a00041070 con 0x7f0a0c104fe0 2024-03-22T01:36:02.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.494+0000 7f0a13525640 1 -- 172.21.15.72:0/402876385 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a0c19fde0 con 0x7f0a0c104fe0 2024-03-22T01:36:02.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.494+0000 7f0a0a7fc640 1 -- 172.21.15.72:0/402876385 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a000044d0 con 0x7f0a0c104fe0 2024-03-22T01:36:02.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.494+0000 7f0a0a7fc640 1 -- 172.21.15.72:0/402876385 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a00034440 con 0x7f0a0c104fe0 2024-03-22T01:36:02.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.494+0000 7f0a13525640 1 -- 172.21.15.72:0/402876385 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a0c1a0250 con 0x7f0a0c104fe0 2024-03-22T01:36:02.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.495+0000 7f0a13525640 1 -- 172.21.15.72:0/402876385 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a0c0fdb10 con 0x7f0a0c104fe0 2024-03-22T01:36:02.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.496+0000 7f0a0a7fc640 1 -- 172.21.15.72:0/402876385 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7f0a00044070 con 0x7f0a0c104fe0 2024-03-22T01:36:02.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.497+0000 7f0a0a7fc640 1 --2- 172.21.15.72:0/402876385 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f09f40a2f20 0x7f09f40a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:02.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.497+0000 7f0a112c1640 1 --2- 172.21.15.72:0/402876385 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f09f40a2f20 0x7f09f40a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:02.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.497+0000 7f0a0a7fc640 1 -- 172.21.15.72:0/402876385 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f0a000f1480 con 0x7f0a0c104fe0 2024-03-22T01:36:02.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.497+0000 7f0a112c1640 1 --2- 172.21.15.72:0/402876385 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f09f40a2f20 0x7f09f40a53e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f09fc005fd0 tx=0x7f09fc009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:02.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.500+0000 7f0a0a7fc640 1 -- 172.21.15.72:0/402876385 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0a000bdf50 con 0x7f0a0c104fe0 2024-03-22T01:36:02.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.676+0000 7f0a13525640 1 -- 172.21.15.72:0/402876385 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f0a0c02d580 con 0x7f0a0c104fe0 2024-03-22T01:36:02.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.677+0000 7f0a0a7fc640 1 -- 172.21.15.72:0/402876385 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f0a000bd850 con 0x7f0a0c104fe0 2024-03-22T01:36:02.676 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-03-22T01:36:02.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.679+0000 7f0a13525640 1 -- 172.21.15.72:0/402876385 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f09f40a2f20 msgr2=0x7f09f40a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:02.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.679+0000 7f0a13525640 1 --2- 172.21.15.72:0/402876385 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f09f40a2f20 0x7f09f40a53e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f09fc005fd0 tx=0x7f09fc009500 comp rx=0 tx=0).stop 2024-03-22T01:36:02.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.679+0000 7f0a13525640 1 -- 172.21.15.72:0/402876385 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0a0c104fe0 msgr2=0x7f0a0c198470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:02.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.679+0000 7f0a13525640 1 --2- 172.21.15.72:0/402876385 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0a0c104fe0 0x7f0a0c198470 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f0a00000c00 tx=0x7f0a00005d50 comp rx=0 tx=0).stop 2024-03-22T01:36:02.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.679+0000 7f0a13525640 1 -- 172.21.15.72:0/402876385 shutdown_connections 2024-03-22T01:36:02.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.679+0000 7f0a13525640 1 --2- 172.21.15.72:0/402876385 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f09f40a2f20 0x7f09f40a53e0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:02.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.679+0000 7f0a13525640 1 --2- 172.21.15.72:0/402876385 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0a0c107910 0x7f0a0c19f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:02.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.679+0000 7f0a13525640 1 --2- 172.21.15.72:0/402876385 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0a0c104fe0 0x7f0a0c198470 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:02.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.679+0000 7f0a13525640 1 --2- 172.21.15.72:0/402876385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a0c1026b0 0x7f0a0c197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:02.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.679+0000 7f0a13525640 1 -- 172.21.15.72:0/402876385 >> 172.21.15.72:0/402876385 conn(0x7f0a0c0fc020 msgr2=0x7f0a0c1030d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:02.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.679+0000 7f0a13525640 1 -- 172.21.15.72:0/402876385 shutdown_connections 2024-03-22T01:36:02.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.679+0000 7f0a13525640 1 -- 172.21.15.72:0/402876385 wait complete. 2024-03-22T01:36:02.688 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:02 smithi072 ceph-mon[32414]: pgmap v346: 129 pgs: 129 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-22T01:36:02.688 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:02 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/1190076164' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:36:02.688 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:02 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/1912949128' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T01:36:02.688 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:02 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/104641279' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T01:36:02.689 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-22T01:36:02.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.795+0000 7f05a464e640 1 -- 172.21.15.72:0/396390290 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f059c102660 msgr2=0x7f059c108ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:02.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.795+0000 7f05a464e640 1 --2- 172.21.15.72:0/396390290 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f059c102660 0x7f059c108ef0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0594009ca0 tx=0x7f059402c0f0 comp rx=0 tx=0).stop 2024-03-22T01:36:02.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.795+0000 7f05a464e640 1 -- 172.21.15.72:0/396390290 shutdown_connections 2024-03-22T01:36:02.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.795+0000 7f05a464e640 1 --2- 172.21.15.72:0/396390290 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f059c102660 0x7f059c108ef0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:02.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.795+0000 7f05a464e640 1 --2- 172.21.15.72:0/396390290 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f059c101ca0 0x7f059c102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:02.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.795+0000 7f05a464e640 1 --2- 172.21.15.72:0/396390290 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f059c100ab0 0x7f059c100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:02.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.795+0000 7f05a464e640 1 -- 172.21.15.72:0/396390290 >> 172.21.15.72:0/396390290 conn(0x7f059c0fc020 msgr2=0x7f059c0fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:02.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.795+0000 7f05a464e640 1 -- 172.21.15.72:0/396390290 shutdown_connections 2024-03-22T01:36:02.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.795+0000 7f05a464e640 1 -- 172.21.15.72:0/396390290 wait complete. 2024-03-22T01:36:02.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.795+0000 7f05a464e640 1 Processor -- start 2024-03-22T01:36:02.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.795+0000 7f05a464e640 1 -- start start 2024-03-22T01:36:02.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.796+0000 7f05a464e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f059c100ab0 0x7f059c06ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:02.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.796+0000 7f05a464e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f059c101ca0 0x7f059c19cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:02.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.796+0000 7f05a464e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f059c102660 0x7f059c19f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:02.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.796+0000 7f05a464e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f059c10c730 con 0x7f059c102660 2024-03-22T01:36:02.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.796+0000 7f05a464e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f059c10c5b0 con 0x7f059c101ca0 2024-03-22T01:36:02.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.796+0000 7f05a464e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f059c10c8b0 con 0x7f059c100ab0 2024-03-22T01:36:02.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.796+0000 7f05a2beb640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f059c102660 0x7f059c19f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:02.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.796+0000 7f05a2beb640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f059c102660 0x7f059c19f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:35552/0 (socket says 172.21.15.72:35552) 2024-03-22T01:36:02.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.796+0000 7f05a2beb640 1 -- 172.21.15.72:0/1167597990 learned_addr learned my addr 172.21.15.72:0/1167597990 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:02.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.796+0000 7f05a1be9640 1 --2- 172.21.15.72:0/1167597990 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f059c101ca0 0x7f059c19cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:02.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.796+0000 7f05a23ea640 1 --2- 172.21.15.72:0/1167597990 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f059c100ab0 0x7f059c06ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:02.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.796+0000 7f05a2beb640 1 -- 172.21.15.72:0/1167597990 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f059c100ab0 msgr2=0x7f059c06ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:02.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.796+0000 7f05a2beb640 1 --2- 172.21.15.72:0/1167597990 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f059c100ab0 0x7f059c06ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:02.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.796+0000 7f05a2beb640 1 -- 172.21.15.72:0/1167597990 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f059c101ca0 msgr2=0x7f059c19cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:02.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.796+0000 7f05a2beb640 1 --2- 172.21.15.72:0/1167597990 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f059c101ca0 0x7f059c19cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:02.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.796+0000 7f05a2beb640 1 -- 172.21.15.72:0/1167597990 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f059c19fb00 con 0x7f059c102660 2024-03-22T01:36:02.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.796+0000 7f05a2beb640 1 --2- 172.21.15.72:0/1167597990 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f059c102660 0x7f059c19f1f0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f0594037040 tx=0x7f0594032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:02.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.796+0000 7f058f7fe640 1 -- 172.21.15.72:0/1167597990 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0594041020 con 0x7f059c102660 2024-03-22T01:36:02.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.796+0000 7f05a464e640 1 -- 172.21.15.72:0/1167597990 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f059c19fd30 con 0x7f059c102660 2024-03-22T01:36:02.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.796+0000 7f058f7fe640 1 -- 172.21.15.72:0/1167597990 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0594034d90 con 0x7f059c102660 2024-03-22T01:36:02.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.796+0000 7f058f7fe640 1 -- 172.21.15.72:0/1167597990 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0594039040 con 0x7f059c102660 2024-03-22T01:36:02.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.796+0000 7f05a464e640 1 -- 172.21.15.72:0/1167597990 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f059c1a0210 con 0x7f059c102660 2024-03-22T01:36:02.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.798+0000 7f05a464e640 1 -- 172.21.15.72:0/1167597990 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f059c107d70 con 0x7f059c102660 2024-03-22T01:36:02.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.799+0000 7f058f7fe640 1 -- 172.21.15.72:0/1167597990 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7f0594044020 con 0x7f059c102660 2024-03-22T01:36:02.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.799+0000 7f058f7fe640 1 --2- 172.21.15.72:0/1167597990 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f05880a2f20 0x7f05880a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:02.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.799+0000 7f05a23ea640 1 --2- 172.21.15.72:0/1167597990 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f05880a2f20 0x7f05880a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:02.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.799+0000 7f058f7fe640 1 -- 172.21.15.72:0/1167597990 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f05940f1630 con 0x7f059c102660 2024-03-22T01:36:02.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.800+0000 7f05a23ea640 1 --2- 172.21.15.72:0/1167597990 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f05880a2f20 0x7f05880a53e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f0598005fd0 tx=0x7f059800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:02.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.803+0000 7f058f7fe640 1 -- 172.21.15.72:0/1167597990 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f059404b020 con 0x7f059c102660 2024-03-22T01:36:02.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:02 smithi089 ceph-mon[35770]: pgmap v346: 129 pgs: 129 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-22T01:36:02.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:02 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/1190076164' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:36:02.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:02 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/1912949128' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T01:36:02.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:02 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/104641279' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T01:36:02.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:02 smithi098 ceph-mon[36468]: pgmap v346: 129 pgs: 129 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-22T01:36:02.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:02 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/1190076164' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:36:02.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:02 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/1912949128' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T01:36:02.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:02 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/104641279' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T01:36:02.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.977+0000 7f05a464e640 1 -- 172.21.15.72:0/1167597990 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f059c02d3b0 con 0x7f059c102660 2024-03-22T01:36:02.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.978+0000 7f058f7fe640 1 -- 172.21.15.72:0/1167597990 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7f05940be100 con 0x7f059c102660 2024-03-22T01:36:02.977 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-03-22T01:36:02.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.980+0000 7f05a464e640 1 -- 172.21.15.72:0/1167597990 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f05880a2f20 msgr2=0x7f05880a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:02.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.980+0000 7f05a464e640 1 --2- 172.21.15.72:0/1167597990 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f05880a2f20 0x7f05880a53e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f0598005fd0 tx=0x7f059800b390 comp rx=0 tx=0).stop 2024-03-22T01:36:02.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.980+0000 7f05a464e640 1 -- 172.21.15.72:0/1167597990 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f059c102660 msgr2=0x7f059c19f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:02.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.980+0000 7f05a464e640 1 --2- 172.21.15.72:0/1167597990 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f059c102660 0x7f059c19f1f0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f0594037040 tx=0x7f0594032f20 comp rx=0 tx=0).stop 2024-03-22T01:36:02.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.980+0000 7f05a464e640 1 -- 172.21.15.72:0/1167597990 shutdown_connections 2024-03-22T01:36:02.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.980+0000 7f05a464e640 1 --2- 172.21.15.72:0/1167597990 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f05880a2f20 0x7f05880a53e0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:02.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.980+0000 7f05a464e640 1 --2- 172.21.15.72:0/1167597990 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f059c101ca0 0x7f059c19cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:02.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.980+0000 7f05a464e640 1 --2- 172.21.15.72:0/1167597990 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f059c102660 0x7f059c19f1f0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:02.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.980+0000 7f05a464e640 1 --2- 172.21.15.72:0/1167597990 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f059c100ab0 0x7f059c06ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:02.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.980+0000 7f05a464e640 1 -- 172.21.15.72:0/1167597990 >> 172.21.15.72:0/1167597990 conn(0x7f059c0fc020 msgr2=0x7f059c104f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:02.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.980+0000 7f05a464e640 1 -- 172.21.15.72:0/1167597990 shutdown_connections 2024-03-22T01:36:02.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:02.980+0000 7f05a464e640 1 -- 172.21.15.72:0/1167597990 wait complete. 2024-03-22T01:36:02.990 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T01:36:03.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.093+0000 7f2d88bee640 1 -- 172.21.15.72:0/4024244155 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d841026d0 msgr2=0x7f2d84108f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:03.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.093+0000 7f2d88bee640 1 --2- 172.21.15.72:0/4024244155 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d841026d0 0x7f2d84108f60 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c009ca0 tx=0x7f2d7c02c0f0 comp rx=0 tx=0).stop 2024-03-22T01:36:03.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.093+0000 7f2d88bee640 1 -- 172.21.15.72:0/4024244155 shutdown_connections 2024-03-22T01:36:03.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.093+0000 7f2d88bee640 1 --2- 172.21.15.72:0/4024244155 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d84101d10 0x7f2d84102190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:03.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.093+0000 7f2d88bee640 1 --2- 172.21.15.72:0/4024244155 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d841026d0 0x7f2d84108f60 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:03.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.093+0000 7f2d88bee640 1 --2- 172.21.15.72:0/4024244155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d84100b20 0x7f2d84100f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:03.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.093+0000 7f2d88bee640 1 -- 172.21.15.72:0/4024244155 >> 172.21.15.72:0/4024244155 conn(0x7f2d840fc070 msgr2=0x7f2d840fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:03.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.093+0000 7f2d88bee640 1 -- 172.21.15.72:0/4024244155 shutdown_connections 2024-03-22T01:36:03.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.093+0000 7f2d88bee640 1 -- 172.21.15.72:0/4024244155 wait complete. 2024-03-22T01:36:03.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.093+0000 7f2d88bee640 1 Processor -- start 2024-03-22T01:36:03.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.094+0000 7f2d88bee640 1 -- start start 2024-03-22T01:36:03.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.094+0000 7f2d88bee640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d84100b20 0x7f2d84197e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:03.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.094+0000 7f2d88bee640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d84101d10 0x7f2d84198380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:03.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.094+0000 7f2d88bee640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d841026d0 0x7f2d8419f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:03.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.094+0000 7f2d88bee640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f2d8410d850 con 0x7f2d84101d10 2024-03-22T01:36:03.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.094+0000 7f2d88bee640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2d8410d6d0 con 0x7f2d841026d0 2024-03-22T01:36:03.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.094+0000 7f2d88bee640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2d8410d9d0 con 0x7f2d84100b20 2024-03-22T01:36:03.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.094+0000 7f2d81d9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d84101d10 0x7f2d84198380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:03.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.094+0000 7f2d81d9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d84101d10 0x7f2d84198380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:35562/0 (socket says 172.21.15.72:35562) 2024-03-22T01:36:03.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.094+0000 7f2d81d9b640 1 -- 172.21.15.72:0/3858217015 learned_addr learned my addr 172.21.15.72:0/3858217015 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:03.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.094+0000 7f2d82d9d640 1 --2- 172.21.15.72:0/3858217015 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d841026d0 0x7f2d8419f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:03.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.094+0000 7f2d8259c640 1 --2- 172.21.15.72:0/3858217015 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d84100b20 0x7f2d84197e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:03.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.094+0000 7f2d81d9b640 1 -- 172.21.15.72:0/3858217015 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d84100b20 msgr2=0x7f2d84197e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:03.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.094+0000 7f2d81d9b640 1 --2- 172.21.15.72:0/3858217015 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d84100b20 0x7f2d84197e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:03.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.094+0000 7f2d81d9b640 1 -- 172.21.15.72:0/3858217015 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d841026d0 msgr2=0x7f2d8419f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:03.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.094+0000 7f2d81d9b640 1 --2- 172.21.15.72:0/3858217015 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d841026d0 0x7f2d8419f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:03.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.094+0000 7f2d81d9b640 1 -- 172.21.15.72:0/3858217015 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d8419fa60 con 0x7f2d84101d10 2024-03-22T01:36:03.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.094+0000 7f2d81d9b640 1 --2- 172.21.15.72:0/3858217015 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d84101d10 0x7f2d84198380 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c00ea60 tx=0x7f2d6c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:03.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.094+0000 7f2d7b7fe640 1 -- 172.21.15.72:0/3858217015 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d6c00cde0 con 0x7f2d84101d10 2024-03-22T01:36:03.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.094+0000 7f2d7b7fe640 1 -- 172.21.15.72:0/3858217015 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d6c0035f0 con 0x7f2d84101d10 2024-03-22T01:36:03.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.094+0000 7f2d88bee640 1 -- 172.21.15.72:0/3858217015 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d8419fd50 con 0x7f2d84101d10 2024-03-22T01:36:03.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.094+0000 7f2d7b7fe640 1 -- 172.21.15.72:0/3858217015 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d6c010670 con 0x7f2d84101d10 2024-03-22T01:36:03.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.094+0000 7f2d88bee640 1 -- 172.21.15.72:0/3858217015 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d841a0240 con 0x7f2d84101d10 2024-03-22T01:36:03.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.095+0000 7f2d88bee640 1 -- 172.21.15.72:0/3858217015 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d84107e40 con 0x7f2d84101d10 2024-03-22T01:36:03.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.096+0000 7f2d7b7fe640 1 -- 172.21.15.72:0/3858217015 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7f2d6c0037a0 con 0x7f2d84101d10 2024-03-22T01:36:03.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.097+0000 7f2d7b7fe640 1 --2- 172.21.15.72:0/3858217015 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f2d700a2f20 0x7f2d700a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:03.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.097+0000 7f2d8259c640 1 --2- 172.21.15.72:0/3858217015 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f2d700a2f20 0x7f2d700a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:03.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.097+0000 7f2d7b7fe640 1 -- 172.21.15.72:0/3858217015 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f2d6c0cfba0 con 0x7f2d84101d10 2024-03-22T01:36:03.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.098+0000 7f2d8259c640 1 --2- 172.21.15.72:0/3858217015 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f2d700a2f20 0x7f2d700a53e0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f2d740058c0 tx=0x7f2d7400a4c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:03.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.101+0000 7f2d7b7fe640 1 -- 172.21.15.72:0/3858217015 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2d6c09c700 con 0x7f2d84101d10 2024-03-22T01:36:03.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.313+0000 7f2d88bee640 1 -- 172.21.15.72:0/3858217015 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2d8402d600 con 0x7f2d84101d10 2024-03-22T01:36:03.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.314+0000 7f2d7b7fe640 1 -- 172.21.15.72:0/3858217015 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11785 (secure 0 0 0) 0x7f2d6c09c000 con 0x7f2d84101d10 2024-03-22T01:36:03.313 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:36:03.313 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6835/3077216308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3077216308},{"type":"v1","addr":"172.21.15.72:6835","nonce":3077216308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2460254143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":2460254143},{"type":"v1","addr":"172.21.15.72:6837","nonce":2460254143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6839/2026432819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6838","nonce":2026432819},{"type":"v1","addr":"172.21.15.72:6839","nonce":2026432819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6841/400094545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6840","nonce":400094545},{"type":"v1","addr":"172.21.15.72:6841","nonce":400094545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6833/1578701171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":1578701171},{"type":"v1","addr":"172.21.15.89:6833","nonce":1578701171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/2703047213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2703047213},{"type":"v1","addr":"172.21.15.98:6833","nonce":2703047213}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2589468787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2589468787},{"type":"v1","addr":"172.21.15.89:6835","nonce":2589468787}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/92657636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":92657636},{"type":"v1","addr":"172.21.15.98:6835","nonce":92657636}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1662941782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1662941782},{"type":"v1","addr":"172.21.15.89:6837","nonce":1662941782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24490,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/781860371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":781860371},{"type":"v1","addr":"172.21.15.98:6837","nonce":781860371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-22T01:34:14.815677+0000","modified":"2024-03-22T01:34:39.842959+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24524},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3192173650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3192173650},{"type":"v1","addr":"172.21.15.98:6839","nonce":3192173650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6839/149200280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":149200280},{"type":"v1","addr":"172.21.15.89:6839","nonce":149200280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24524]},"id":1}]} 2024-03-22T01:36:03.313 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 21 2024-03-22T01:36:03.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.316+0000 7f2d88bee640 1 -- 172.21.15.72:0/3858217015 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f2d700a2f20 msgr2=0x7f2d700a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:03.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.316+0000 7f2d88bee640 1 --2- 172.21.15.72:0/3858217015 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f2d700a2f20 0x7f2d700a53e0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f2d740058c0 tx=0x7f2d7400a4c0 comp rx=0 tx=0).stop 2024-03-22T01:36:03.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.316+0000 7f2d88bee640 1 -- 172.21.15.72:0/3858217015 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d84101d10 msgr2=0x7f2d84198380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:03.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.316+0000 7f2d88bee640 1 --2- 172.21.15.72:0/3858217015 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d84101d10 0x7f2d84198380 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c00ea60 tx=0x7f2d6c00ef20 comp rx=0 tx=0).stop 2024-03-22T01:36:03.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.316+0000 7f2d88bee640 1 -- 172.21.15.72:0/3858217015 shutdown_connections 2024-03-22T01:36:03.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.316+0000 7f2d88bee640 1 --2- 172.21.15.72:0/3858217015 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f2d700a2f20 0x7f2d700a53e0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:03.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.316+0000 7f2d88bee640 1 --2- 172.21.15.72:0/3858217015 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d841026d0 0x7f2d8419f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:03.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.316+0000 7f2d88bee640 1 --2- 172.21.15.72:0/3858217015 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d84101d10 0x7f2d84198380 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:03.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.316+0000 7f2d88bee640 1 --2- 172.21.15.72:0/3858217015 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d84100b20 0x7f2d84197e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:03.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.316+0000 7f2d88bee640 1 -- 172.21.15.72:0/3858217015 >> 172.21.15.72:0/3858217015 conn(0x7f2d840fc070 msgr2=0x7f2d84105010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:03.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.316+0000 7f2d88bee640 1 -- 172.21.15.72:0/3858217015 shutdown_connections 2024-03-22T01:36:03.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.316+0000 7f2d88bee640 1 -- 172.21.15.72:0/3858217015 wait complete. 2024-03-22T01:36:03.326 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T01:36:03.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.428+0000 7f4ae4b0f640 1 -- 172.21.15.72:0/139994628 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ae01014b0 msgr2=0x7f4ae0101930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:03.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.428+0000 7f4ae4b0f640 1 --2- 172.21.15.72:0/139994628 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ae01014b0 0x7f4ae0101930 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f4ad4009ca0 tx=0x7f4ad402c0f0 comp rx=0 tx=0).stop 2024-03-22T01:36:03.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.428+0000 7f4ae4b0f640 1 -- 172.21.15.72:0/139994628 shutdown_connections 2024-03-22T01:36:03.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.428+0000 7f4ae4b0f640 1 --2- 172.21.15.72:0/139994628 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ae0101e70 0x7f4ae0108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:03.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.428+0000 7f4ae4b0f640 1 --2- 172.21.15.72:0/139994628 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ae01014b0 0x7f4ae0101930 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:03.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.428+0000 7f4ae4b0f640 1 --2- 172.21.15.72:0/139994628 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ae01002c0 0x7f4ae01006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:03.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.428+0000 7f4ae4b0f640 1 -- 172.21.15.72:0/139994628 >> 172.21.15.72:0/139994628 conn(0x7f4ae00fb810 msgr2=0x7f4ae00fdc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:03.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.428+0000 7f4ae4b0f640 1 -- 172.21.15.72:0/139994628 shutdown_connections 2024-03-22T01:36:03.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.428+0000 7f4ae4b0f640 1 -- 172.21.15.72:0/139994628 wait complete. 2024-03-22T01:36:03.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.429+0000 7f4ae4b0f640 1 Processor -- start 2024-03-22T01:36:03.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.429+0000 7f4ae4b0f640 1 -- start start 2024-03-22T01:36:03.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.429+0000 7f4ae4b0f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ae01002c0 0x7f4ae019a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:03.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.429+0000 7f4ae4b0f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ae01014b0 0x7f4ae019a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:03.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.429+0000 7f4ae4b0f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ae0101e70 0x7f4ae01a16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:03.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.429+0000 7f4ae4b0f640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f4ae010bee0 con 0x7f4ae0101e70 2024-03-22T01:36:03.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.429+0000 7f4ae4b0f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4ae010bd60 con 0x7f4ae01014b0 2024-03-22T01:36:03.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.429+0000 7f4ae4b0f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4ae010c060 con 0x7f4ae01002c0 2024-03-22T01:36:03.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.429+0000 7f4aded9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ae0101e70 0x7f4ae01a16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:03.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.429+0000 7f4aded9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ae0101e70 0x7f4ae01a16b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:35572/0 (socket says 172.21.15.72:35572) 2024-03-22T01:36:03.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.429+0000 7f4aded9d640 1 -- 172.21.15.72:0/3091218110 learned_addr learned my addr 172.21.15.72:0/3091218110 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:03.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.429+0000 7f4addd9b640 1 --2- 172.21.15.72:0/3091218110 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ae01014b0 0x7f4ae019a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:03.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.429+0000 7f4ade59c640 1 --2- 172.21.15.72:0/3091218110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ae01002c0 0x7f4ae019a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:03.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.429+0000 7f4aded9d640 1 -- 172.21.15.72:0/3091218110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ae01002c0 msgr2=0x7f4ae019a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:03.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.429+0000 7f4aded9d640 1 --2- 172.21.15.72:0/3091218110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ae01002c0 0x7f4ae019a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:03.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.429+0000 7f4aded9d640 1 -- 172.21.15.72:0/3091218110 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ae01014b0 msgr2=0x7f4ae019a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:03.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.429+0000 7f4aded9d640 1 --2- 172.21.15.72:0/3091218110 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ae01014b0 0x7f4ae019a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:03.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.429+0000 7f4aded9d640 1 -- 172.21.15.72:0/3091218110 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ae01a1d20 con 0x7f4ae0101e70 2024-03-22T01:36:03.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.429+0000 7f4ade59c640 1 --2- 172.21.15.72:0/3091218110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ae01002c0 0x7f4ae019a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:36:03.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.429+0000 7f4addd9b640 1 --2- 172.21.15.72:0/3091218110 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ae01014b0 0x7f4ae019a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:36:03.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.429+0000 7f4aded9d640 1 --2- 172.21.15.72:0/3091218110 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ae0101e70 0x7f4ae01a16b0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f4ad000e9b0 tx=0x7f4ad000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:03.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.430+0000 7f4acf7fe640 1 -- 172.21.15.72:0/3091218110 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ad000cd80 con 0x7f4ae0101e70 2024-03-22T01:36:03.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.430+0000 7f4acf7fe640 1 -- 172.21.15.72:0/3091218110 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ad00035f0 con 0x7f4ae0101e70 2024-03-22T01:36:03.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.430+0000 7f4ae4b0f640 1 -- 172.21.15.72:0/3091218110 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ae01a1fb0 con 0x7f4ae0101e70 2024-03-22T01:36:03.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.430+0000 7f4acf7fe640 1 -- 172.21.15.72:0/3091218110 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ad0010670 con 0x7f4ae0101e70 2024-03-22T01:36:03.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.430+0000 7f4ae4b0f640 1 -- 172.21.15.72:0/3091218110 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ae01a2480 con 0x7f4ae0101e70 2024-03-22T01:36:03.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.431+0000 7f4ae4b0f640 1 -- 172.21.15.72:0/3091218110 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ae0106e00 con 0x7f4ae0101e70 2024-03-22T01:36:03.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.431+0000 7f4acf7fe640 1 -- 172.21.15.72:0/3091218110 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7f4ad00037a0 con 0x7f4ae0101e70 2024-03-22T01:36:03.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.432+0000 7f4acf7fe640 1 --2- 172.21.15.72:0/3091218110 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f4ac80a2f20 0x7f4ac80a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:03.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.432+0000 7f4ade59c640 1 --2- 172.21.15.72:0/3091218110 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f4ac80a2f20 0x7f4ac80a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:03.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.432+0000 7f4acf7fe640 1 -- 172.21.15.72:0/3091218110 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f4ad00cfac0 con 0x7f4ae0101e70 2024-03-22T01:36:03.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.432+0000 7f4ade59c640 1 --2- 172.21.15.72:0/3091218110 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f4ac80a2f20 0x7f4ac80a53e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f4ad80058c0 tx=0x7f4ad8005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:03.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.436+0000 7f4acf7fe640 1 -- 172.21.15.72:0/3091218110 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4ad009c590 con 0x7f4ae0101e70 2024-03-22T01:36:03.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.615+0000 7f4ae4b0f640 1 -- 172.21.15.72:0/3091218110 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4ae0062790 con 0x7f4ae0101e70 2024-03-22T01:36:03.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.617+0000 7f4acf7fe640 1 -- 172.21.15.72:0/3091218110 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24570 (secure 0 0 0) 0x7f4ad009be90 con 0x7f4ae0101e70 2024-03-22T01:36:03.616 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:36:03.616 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":93,"fsid":"022a99bc-e7eb-11ee-95cd-87774f69a715","created":"2024-03-22T01:26:23.893672+0000","modified":"2024-03-22T01:35:16.738619+0000","last_up_change":"2024-03-22T01:33:25.223684+0000","last_in_change":"2024-03-22T01:33:05.010331+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T01:29:37.438503+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-22T01:34:12.610422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-22T01:34:13.761686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-22T01:34:17.507039+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"16c15bb0-db92-4191-9063-38b4890e81e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6803","nonce":661906791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6805","nonce":661906791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6809","nonce":661906791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6807","nonce":661906791}]},"public_addr":"172.21.15.72:6803/661906791","cluster_addr":"172.21.15.72:6805/661906791","heartbeat_back_addr":"172.21.15.72:6809/661906791","heartbeat_front_addr":"172.21.15.72:6807/661906791","state":["exists","up"]},{"osd":1,"uuid":"981290ba-97dc-496b-aa22-d51704184b10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6801","nonce":2870590550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6803","nonce":2870590550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6807","nonce":2870590550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6805","nonce":2870590550}]},"public_addr":"172.21.15.89:6801/2870590550","cluster_addr":"172.21.15.89:6803/2870590550","heartbeat_back_addr":"172.21.15.89:6807/2870590550","heartbeat_front_addr":"172.21.15.89:6805/2870590550","state":["exists","up"]},{"osd":2,"uuid":"f07ed658-f545-46a1-8edd-8b1d5d49885c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6801","nonce":3865778118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6803","nonce":3865778118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6807","nonce":3865778118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6805","nonce":3865778118}]},"public_addr":"172.21.15.98:6801/3865778118","cluster_addr":"172.21.15.98:6803/3865778118","heartbeat_back_addr":"172.21.15.98:6807/3865778118","heartbeat_front_addr":"172.21.15.98:6805/3865778118","state":["exists","up"]},{"osd":3,"uuid":"ccc4bc40-2c48-46d2-8548-0c05b44f9f1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6811","nonce":1865951395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6813","nonce":1865951395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6817","nonce":1865951395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6815","nonce":1865951395}]},"public_addr":"172.21.15.72:6811/1865951395","cluster_addr":"172.21.15.72:6813/1865951395","heartbeat_back_addr":"172.21.15.72:6817/1865951395","heartbeat_front_addr":"172.21.15.72:6815/1865951395","state":["exists","up"]},{"osd":4,"uuid":"1812439f-810a-4ed2-9395-c00c9a085a13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6809","nonce":2683760795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6811","nonce":2683760795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6815","nonce":2683760795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6813","nonce":2683760795}]},"public_addr":"172.21.15.89:6809/2683760795","cluster_addr":"172.21.15.89:6811/2683760795","heartbeat_back_addr":"172.21.15.89:6815/2683760795","heartbeat_front_addr":"172.21.15.89:6813/2683760795","state":["exists","up"]},{"osd":5,"uuid":"e78954d8-f4e3-4d58-8536-d2d59c843386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6809","nonce":2635838572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6811","nonce":2635838572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6815","nonce":2635838572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6813","nonce":2635838572}]},"public_addr":"172.21.15.98:6809/2635838572","cluster_addr":"172.21.15.98:6811/2635838572","heartbeat_back_addr":"172.21.15.98:6815/2635838572","heartbeat_front_addr":"172.21.15.98:6813/2635838572","state":["exists","up"]},{"osd":6,"uuid":"dc6c8be2-b64c-4959-8015-7b0ea6d25599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6819","nonce":3158982661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6821","nonce":3158982661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6825","nonce":3158982661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6823","nonce":3158982661}]},"public_addr":"172.21.15.72:6819/3158982661","cluster_addr":"172.21.15.72:6821/3158982661","heartbeat_back_addr":"172.21.15.72:6825/3158982661","heartbeat_front_addr":"172.21.15.72:6823/3158982661","state":["exists","up"]},{"osd":7,"uuid":"7a73469f-cf3a-4306-ae93-ea46e21ebbbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6817","nonce":3075913234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6819","nonce":3075913234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6823","nonce":3075913234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6821","nonce":3075913234}]},"public_addr":"172.21.15.89:6817/3075913234","cluster_addr":"172.21.15.89:6819/3075913234","heartbeat_back_addr":"172.21.15.89:6823/3075913234","heartbeat_front_addr":"172.21.15.89:6821/3075913234","state":["exists","up"]},{"osd":8,"uuid":"c4b13bee-ea4d-4344-95fb-2622ecbb8196","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6817","nonce":1813255577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6819","nonce":1813255577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6823","nonce":1813255577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6821","nonce":1813255577}]},"public_addr":"172.21.15.98:6817/1813255577","cluster_addr":"172.21.15.98:6819/1813255577","heartbeat_back_addr":"172.21.15.98:6823/1813255577","heartbeat_front_addr":"172.21.15.98:6821/1813255577","state":["exists","up"]},{"osd":9,"uuid":"eefb0a44-4a36-4e55-9d04-0501bf718db6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6827","nonce":3377749206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6829","nonce":3377749206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6832","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6833","nonce":3377749206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6831","nonce":3377749206}]},"public_addr":"172.21.15.72:6827/3377749206","cluster_addr":"172.21.15.72:6829/3377749206","heartbeat_back_addr":"172.21.15.72:6833/3377749206","heartbeat_front_addr":"172.21.15.72:6831/3377749206","state":["exists","up"]},{"osd":10,"uuid":"8741d25f-40dd-40d7-b040-5cc5e411eb98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6825","nonce":885058786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6827","nonce":885058786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6831","nonce":885058786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6829","nonce":885058786}]},"public_addr":"172.21.15.89:6825/885058786","cluster_addr":"172.21.15.89:6827/885058786","heartbeat_back_addr":"172.21.15.89:6831/885058786","heartbeat_front_addr":"172.21.15.89:6829/885058786","state":["exists","up"]},{"osd":11,"uuid":"8d02e957-eda4-4184-b5c6-5eb3b8b38060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6825","nonce":1517095838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6827","nonce":1517095838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6831","nonce":1517095838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6829","nonce":1517095838}]},"public_addr":"172.21.15.98:6825/1517095838","cluster_addr":"172.21.15.98:6827/1517095838","heartbeat_back_addr":"172.21.15.98:6831/1517095838","heartbeat_front_addr":"172.21.15.98:6829/1517095838","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:28:47.307652+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:11.037135+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:34.619188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:59.350738+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:30:23.787337+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:30:48.994154+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:31:13.690350+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:31:39.628116+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:04.723912+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:30.689741+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:57.229508+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:33:23.154537+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:0/121087196":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/2451857761":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/1228477995":"2024-03-23T01:27:15.339809+0000","172.21.15.72:6801/1513232913":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/3558958264":"2024-03-23T01:26:50.074871+0000","172.21.15.72:0/1510187198":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6800/3518012916":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6801/3518012916":"2024-03-23T01:26:50.074871+0000","172.21.15.72:0/1372801041":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6800/1513232913":"2024-03-23T01:27:15.339809+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T01:36:03.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.619+0000 7f4ae4b0f640 1 -- 172.21.15.72:0/3091218110 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f4ac80a2f20 msgr2=0x7f4ac80a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:03.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.619+0000 7f4ae4b0f640 1 --2- 172.21.15.72:0/3091218110 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f4ac80a2f20 0x7f4ac80a53e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f4ad80058c0 tx=0x7f4ad8005850 comp rx=0 tx=0).stop 2024-03-22T01:36:03.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.619+0000 7f4ae4b0f640 1 -- 172.21.15.72:0/3091218110 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ae0101e70 msgr2=0x7f4ae01a16b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:03.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.619+0000 7f4ae4b0f640 1 --2- 172.21.15.72:0/3091218110 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ae0101e70 0x7f4ae01a16b0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f4ad000e9b0 tx=0x7f4ad000ee70 comp rx=0 tx=0).stop 2024-03-22T01:36:03.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.619+0000 7f4ae4b0f640 1 -- 172.21.15.72:0/3091218110 shutdown_connections 2024-03-22T01:36:03.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.619+0000 7f4ae4b0f640 1 --2- 172.21.15.72:0/3091218110 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f4ac80a2f20 0x7f4ac80a53e0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:03.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.619+0000 7f4ae4b0f640 1 --2- 172.21.15.72:0/3091218110 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ae01014b0 0x7f4ae019a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:03.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.619+0000 7f4ae4b0f640 1 --2- 172.21.15.72:0/3091218110 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ae0101e70 0x7f4ae01a16b0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:03.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.619+0000 7f4ae4b0f640 1 --2- 172.21.15.72:0/3091218110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ae01002c0 0x7f4ae019a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:03.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.619+0000 7f4ae4b0f640 1 -- 172.21.15.72:0/3091218110 >> 172.21.15.72:0/3091218110 conn(0x7f4ae00fb810 msgr2=0x7f4ae00fd1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:03.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.619+0000 7f4ae4b0f640 1 -- 172.21.15.72:0/3091218110 shutdown_connections 2024-03-22T01:36:03.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.619+0000 7f4ae4b0f640 1 -- 172.21.15.72:0/3091218110 wait complete. 2024-03-22T01:36:03.629 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-22T01:29:37.438503+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-22T01:34:12.610422+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-22T01:34:13.761686+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-22T01:34:17.507039+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-22T01:36:03.630 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-22T01:36:03.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.735+0000 7f602f60f640 1 -- 172.21.15.72:0/1313405092 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6028104fc0 msgr2=0x7f60281073b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:03.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.735+0000 7f602f60f640 1 --2- 172.21.15.72:0/1313405092 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6028104fc0 0x7f60281073b0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f6018009900 tx=0x7f601802bc30 comp rx=0 tx=0).stop 2024-03-22T01:36:03.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.735+0000 7f602f60f640 1 -- 172.21.15.72:0/1313405092 shutdown_connections 2024-03-22T01:36:03.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.735+0000 7f602f60f640 1 --2- 172.21.15.72:0/1313405092 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f60281078f0 0x7f6028109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:03.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.735+0000 7f602f60f640 1 --2- 172.21.15.72:0/1313405092 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6028104fc0 0x7f60281073b0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:03.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.735+0000 7f602f60f640 1 --2- 172.21.15.72:0/1313405092 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6028102690 0x7f6028104a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:03.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.735+0000 7f602f60f640 1 -- 172.21.15.72:0/1313405092 >> 172.21.15.72:0/1313405092 conn(0x7f60280fc020 msgr2=0x7f60280fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:03.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.735+0000 7f602f60f640 1 -- 172.21.15.72:0/1313405092 shutdown_connections 2024-03-22T01:36:03.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.735+0000 7f602f60f640 1 -- 172.21.15.72:0/1313405092 wait complete. 2024-03-22T01:36:03.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.736+0000 7f602f60f640 1 Processor -- start 2024-03-22T01:36:03.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.736+0000 7f602f60f640 1 -- start start 2024-03-22T01:36:03.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.736+0000 7f602f60f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6028102690 0x7f6028197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:03.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.736+0000 7f602f60f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6028104fc0 0x7f6028198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:03.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.736+0000 7f602f60f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60281078f0 0x7f602819f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:03.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.736+0000 7f602f60f640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f602810d820 con 0x7f6028102690 2024-03-22T01:36:03.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.736+0000 7f602f60f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f602810d6a0 con 0x7f6028104fc0 2024-03-22T01:36:03.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.736+0000 7f602f60f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f602810d9a0 con 0x7f60281078f0 2024-03-22T01:36:03.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.736+0000 7f602d3ab640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6028102690 0x7f6028197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:03.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.736+0000 7f602d3ab640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6028102690 0x7f6028197ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:35590/0 (socket says 172.21.15.72:35590) 2024-03-22T01:36:03.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.736+0000 7f602d3ab640 1 -- 172.21.15.72:0/19702380 learned_addr learned my addr 172.21.15.72:0/19702380 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:03.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.736+0000 7f602cbaa640 1 --2- 172.21.15.72:0/19702380 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6028104fc0 0x7f6028198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:03.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.736+0000 7f602dbac640 1 --2- 172.21.15.72:0/19702380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60281078f0 0x7f602819f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:03.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.736+0000 7f602d3ab640 1 -- 172.21.15.72:0/19702380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60281078f0 msgr2=0x7f602819f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:03.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.736+0000 7f602d3ab640 1 --2- 172.21.15.72:0/19702380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60281078f0 0x7f602819f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:03.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.736+0000 7f602d3ab640 1 -- 172.21.15.72:0/19702380 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6028104fc0 msgr2=0x7f6028198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:03.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.736+0000 7f602d3ab640 1 --2- 172.21.15.72:0/19702380 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6028104fc0 0x7f6028198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:03.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.736+0000 7f602d3ab640 1 -- 172.21.15.72:0/19702380 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f602819fb10 con 0x7f6028102690 2024-03-22T01:36:03.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.736+0000 7f602d3ab640 1 --2- 172.21.15.72:0/19702380 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6028102690 0x7f6028197ef0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f602400c970 tx=0x7f602400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:03.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.736+0000 7f601e7fc640 1 -- 172.21.15.72:0/19702380 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6024007bb0 con 0x7f6028102690 2024-03-22T01:36:03.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.737+0000 7f601e7fc640 1 -- 172.21.15.72:0/19702380 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6024007d50 con 0x7f6028102690 2024-03-22T01:36:03.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.737+0000 7f602f60f640 1 -- 172.21.15.72:0/19702380 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f602819fe00 con 0x7f6028102690 2024-03-22T01:36:03.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.737+0000 7f601e7fc640 1 -- 172.21.15.72:0/19702380 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f602400f450 con 0x7f6028102690 2024-03-22T01:36:03.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.737+0000 7f602f60f640 1 -- 172.21.15.72:0/19702380 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60281a0210 con 0x7f6028102690 2024-03-22T01:36:03.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.738+0000 7f602f60f640 1 -- 172.21.15.72:0/19702380 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60280fdb10 con 0x7f6028102690 2024-03-22T01:36:03.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.739+0000 7f601e7fc640 1 -- 172.21.15.72:0/19702380 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7f6024005020 con 0x7f6028102690 2024-03-22T01:36:03.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.739+0000 7f601e7fc640 1 --2- 172.21.15.72:0/19702380 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f60140a2f20 0x7f60140a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:03.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.739+0000 7f602cbaa640 1 --2- 172.21.15.72:0/19702380 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f60140a2f20 0x7f60140a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:03.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.739+0000 7f601e7fc640 1 -- 172.21.15.72:0/19702380 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f60240cf500 con 0x7f6028102690 2024-03-22T01:36:03.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.740+0000 7f602cbaa640 1 --2- 172.21.15.72:0/19702380 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f60140a2f20 0x7f60140a53e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f6018000c00 tx=0x7f6018005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:03.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.744+0000 7f601e7fc640 1 -- 172.21.15.72:0/19702380 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6024014030 con 0x7f6028102690 2024-03-22T01:36:03.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:03 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/402876385' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T01:36:03.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:03 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/1167597990' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T01:36:03.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:03 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/3858217015' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:36:03.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:03 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/402876385' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T01:36:03.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:03 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/1167597990' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T01:36:03.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:03 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/3858217015' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:36:03.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.917+0000 7f602f60f640 1 -- 172.21.15.72:0/19702380 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6028062860 con 0x7f6028102690 2024-03-22T01:36:03.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.917+0000 7f601e7fc640 1 -- 172.21.15.72:0/19702380 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7f602409bfd0 con 0x7f6028102690 2024-03-22T01:36:03.916 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-22T01:36:03.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.919+0000 7f602f60f640 1 -- 172.21.15.72:0/19702380 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f60140a2f20 msgr2=0x7f60140a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:03.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.919+0000 7f602f60f640 1 --2- 172.21.15.72:0/19702380 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f60140a2f20 0x7f60140a53e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f6018000c00 tx=0x7f6018005fd0 comp rx=0 tx=0).stop 2024-03-22T01:36:03.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.919+0000 7f602f60f640 1 -- 172.21.15.72:0/19702380 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6028102690 msgr2=0x7f6028197ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:03.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.919+0000 7f602f60f640 1 --2- 172.21.15.72:0/19702380 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6028102690 0x7f6028197ef0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f602400c970 tx=0x7f602400ce30 comp rx=0 tx=0).stop 2024-03-22T01:36:03.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.919+0000 7f602f60f640 1 -- 172.21.15.72:0/19702380 shutdown_connections 2024-03-22T01:36:03.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.919+0000 7f602f60f640 1 --2- 172.21.15.72:0/19702380 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f60140a2f20 0x7f60140a53e0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:03.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.919+0000 7f602f60f640 1 --2- 172.21.15.72:0/19702380 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6028104fc0 0x7f6028198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:03.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.919+0000 7f602f60f640 1 --2- 172.21.15.72:0/19702380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60281078f0 0x7f602819f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:03.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.919+0000 7f602f60f640 1 --2- 172.21.15.72:0/19702380 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6028102690 0x7f6028197ef0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:03.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.919+0000 7f602f60f640 1 -- 172.21.15.72:0/19702380 >> 172.21.15.72:0/19702380 conn(0x7f60280fc020 msgr2=0x7f60281030b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:03.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.919+0000 7f602f60f640 1 -- 172.21.15.72:0/19702380 shutdown_connections 2024-03-22T01:36:03.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:03.919+0000 7f602f60f640 1 -- 172.21.15.72:0/19702380 wait complete. 2024-03-22T01:36:03.923 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:03 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/402876385' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T01:36:03.923 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:03 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/1167597990' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T01:36:03.923 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:03 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/3858217015' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:36:03.929 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-22T01:36:04.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.034+0000 7f67a0779640 1 -- 172.21.15.72:0/3553848399 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6798100ab0 msgr2=0x7f6798100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:04.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.034+0000 7f67a0779640 1 --2- 172.21.15.72:0/3553848399 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6798100ab0 0x7f6798100ed0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f6794009900 tx=0x7f679402bc30 comp rx=0 tx=0).stop 2024-03-22T01:36:04.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.035+0000 7f67a0779640 1 -- 172.21.15.72:0/3553848399 shutdown_connections 2024-03-22T01:36:04.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.035+0000 7f67a0779640 1 --2- 172.21.15.72:0/3553848399 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6798101ca0 0x7f6798102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:04.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.035+0000 7f67a0779640 1 --2- 172.21.15.72:0/3553848399 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6798102660 0x7f6798108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:04.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.035+0000 7f67a0779640 1 --2- 172.21.15.72:0/3553848399 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6798100ab0 0x7f6798100ed0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:04.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.035+0000 7f67a0779640 1 -- 172.21.15.72:0/3553848399 >> 172.21.15.72:0/3553848399 conn(0x7f67980fc020 msgr2=0x7f67980fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:04.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.035+0000 7f67a0779640 1 -- 172.21.15.72:0/3553848399 shutdown_connections 2024-03-22T01:36:04.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.035+0000 7f67a0779640 1 -- 172.21.15.72:0/3553848399 wait complete. 2024-03-22T01:36:04.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.035+0000 7f67a0779640 1 Processor -- start 2024-03-22T01:36:04.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.035+0000 7f67a0779640 1 -- start start 2024-03-22T01:36:04.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.035+0000 7f67a0779640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6798100ab0 0x7f6798197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:04.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.035+0000 7f67a0779640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6798101ca0 0x7f6798198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:04.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.035+0000 7f67a0779640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6798102660 0x7f679819f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:04.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.035+0000 7f67a0779640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f679810d850 con 0x7f6798102660 2024-03-22T01:36:04.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.035+0000 7f67a0779640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f679810d6d0 con 0x7f6798100ab0 2024-03-22T01:36:04.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.035+0000 7f67a0779640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f679810d9d0 con 0x7f6798101ca0 2024-03-22T01:36:04.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.035+0000 7f679ed16640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6798102660 0x7f679819f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:04.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.035+0000 7f679ed16640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6798102660 0x7f679819f4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:35618/0 (socket says 172.21.15.72:35618) 2024-03-22T01:36:04.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.035+0000 7f679ed16640 1 -- 172.21.15.72:0/626811904 learned_addr learned my addr 172.21.15.72:0/626811904 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:04.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.036+0000 7f679e515640 1 --2- 172.21.15.72:0/626811904 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6798100ab0 0x7f6798197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:04.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.036+0000 7f679ed16640 1 -- 172.21.15.72:0/626811904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6798101ca0 msgr2=0x7f6798198440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:04.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.036+0000 7f679ed16640 1 --2- 172.21.15.72:0/626811904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6798101ca0 0x7f6798198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:04.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.036+0000 7f679ed16640 1 -- 172.21.15.72:0/626811904 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6798100ab0 msgr2=0x7f6798197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:04.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.036+0000 7f679ed16640 1 --2- 172.21.15.72:0/626811904 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6798100ab0 0x7f6798197f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:04.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.036+0000 7f679ed16640 1 -- 172.21.15.72:0/626811904 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f679819fb20 con 0x7f6798102660 2024-03-22T01:36:04.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.036+0000 7f679ed16640 1 --2- 172.21.15.72:0/626811904 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6798102660 0x7f679819f4b0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f679000cad0 tx=0x7f679000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:04.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.036+0000 7f678f7fe640 1 -- 172.21.15.72:0/626811904 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f679000ae20 con 0x7f6798102660 2024-03-22T01:36:04.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.036+0000 7f678f7fe640 1 -- 172.21.15.72:0/626811904 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6790005020 con 0x7f6798102660 2024-03-22T01:36:04.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.036+0000 7f67a0779640 1 -- 172.21.15.72:0/626811904 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f679819fe10 con 0x7f6798102660 2024-03-22T01:36:04.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.036+0000 7f678f7fe640 1 -- 172.21.15.72:0/626811904 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f679000f6f0 con 0x7f6798102660 2024-03-22T01:36:04.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.036+0000 7f67a0779640 1 -- 172.21.15.72:0/626811904 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67981a0330 con 0x7f6798102660 2024-03-22T01:36:04.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.037+0000 7f67a0779640 1 -- 172.21.15.72:0/626811904 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6798107e70 con 0x7f6798102660 2024-03-22T01:36:04.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.038+0000 7f678f7fe640 1 -- 172.21.15.72:0/626811904 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7f67900051d0 con 0x7f6798102660 2024-03-22T01:36:04.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.038+0000 7f678f7fe640 1 --2- 172.21.15.72:0/626811904 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f67840a2f20 0x7f67840a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:04.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.039+0000 7f679e515640 1 --2- 172.21.15.72:0/626811904 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f67840a2f20 0x7f67840a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:04.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.039+0000 7f678f7fe640 1 -- 172.21.15.72:0/626811904 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f67900cfa90 con 0x7f6798102660 2024-03-22T01:36:04.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.039+0000 7f679e515640 1 --2- 172.21.15.72:0/626811904 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f67840a2f20 0x7f67840a53e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f6794000c00 tx=0x7f6794005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:04.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.043+0000 7f678f7fe640 1 -- 172.21.15.72:0/626811904 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6790014030 con 0x7f6798102660 2024-03-22T01:36:04.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.215+0000 7f67a0779640 1 -- 172.21.15.72:0/626811904 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f6798062830 con 0x7f6798102660 2024-03-22T01:36:04.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.215+0000 7f678f7fe640 1 -- 172.21.15.72:0/626811904 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f679009c560 con 0x7f6798102660 2024-03-22T01:36:04.215 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-03-22T01:36:04.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.217+0000 7f67a0779640 1 -- 172.21.15.72:0/626811904 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f67840a2f20 msgr2=0x7f67840a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:04.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.217+0000 7f67a0779640 1 --2- 172.21.15.72:0/626811904 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f67840a2f20 0x7f67840a53e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f6794000c00 tx=0x7f6794005fd0 comp rx=0 tx=0).stop 2024-03-22T01:36:04.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.217+0000 7f67a0779640 1 -- 172.21.15.72:0/626811904 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6798102660 msgr2=0x7f679819f4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:04.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.217+0000 7f67a0779640 1 --2- 172.21.15.72:0/626811904 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6798102660 0x7f679819f4b0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f679000cad0 tx=0x7f679000cf90 comp rx=0 tx=0).stop 2024-03-22T01:36:04.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.217+0000 7f67a0779640 1 -- 172.21.15.72:0/626811904 shutdown_connections 2024-03-22T01:36:04.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.217+0000 7f67a0779640 1 --2- 172.21.15.72:0/626811904 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f67840a2f20 0x7f67840a53e0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:04.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.217+0000 7f67a0779640 1 --2- 172.21.15.72:0/626811904 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6798102660 0x7f679819f4b0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:04.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.218+0000 7f67a0779640 1 --2- 172.21.15.72:0/626811904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6798101ca0 0x7f6798198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:04.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.218+0000 7f67a0779640 1 --2- 172.21.15.72:0/626811904 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6798100ab0 0x7f6798197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:04.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.218+0000 7f67a0779640 1 -- 172.21.15.72:0/626811904 >> 172.21.15.72:0/626811904 conn(0x7f67980fc020 msgr2=0x7f6798105040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:04.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.218+0000 7f67a0779640 1 -- 172.21.15.72:0/626811904 shutdown_connections 2024-03-22T01:36:04.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.218+0000 7f67a0779640 1 -- 172.21.15.72:0/626811904 wait complete. 2024-03-22T01:36:04.228 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-22T01:36:04.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.333+0000 7f8ddf839640 1 -- 172.21.15.72:0/121987162 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8dd8100a90 msgr2=0x7f8dd8100eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:04.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.333+0000 7f8ddf839640 1 --2- 172.21.15.72:0/121987162 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8dd8100a90 0x7f8dd8100eb0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f8dd4009900 tx=0x7f8dd402bc30 comp rx=0 tx=0).stop 2024-03-22T01:36:04.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.333+0000 7f8ddf839640 1 -- 172.21.15.72:0/121987162 shutdown_connections 2024-03-22T01:36:04.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.333+0000 7f8ddf839640 1 --2- 172.21.15.72:0/121987162 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8dd8101c80 0x7f8dd8102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:04.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.333+0000 7f8ddf839640 1 --2- 172.21.15.72:0/121987162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8dd8102640 0x7f8dd8108ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:04.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.333+0000 7f8ddf839640 1 --2- 172.21.15.72:0/121987162 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8dd8100a90 0x7f8dd8100eb0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:04.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.333+0000 7f8ddf839640 1 -- 172.21.15.72:0/121987162 >> 172.21.15.72:0/121987162 conn(0x7f8dd80fc000 msgr2=0x7f8dd80fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:04.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.333+0000 7f8ddf839640 1 -- 172.21.15.72:0/121987162 shutdown_connections 2024-03-22T01:36:04.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.333+0000 7f8ddf839640 1 -- 172.21.15.72:0/121987162 wait complete. 2024-03-22T01:36:04.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.334+0000 7f8ddf839640 1 Processor -- start 2024-03-22T01:36:04.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.334+0000 7f8ddf839640 1 -- start start 2024-03-22T01:36:04.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.334+0000 7f8ddf839640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8dd8100a90 0x7f8dd8196de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:04.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.334+0000 7f8ddf839640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8dd8101c80 0x7f8dd8197320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:04.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.334+0000 7f8ddf839640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8dd8102640 0x7f8dd819e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:04.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.334+0000 7f8ddf839640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f8dd810c710 con 0x7f8dd8100a90 2024-03-22T01:36:04.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.334+0000 7f8ddf839640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8dd810c590 con 0x7f8dd8101c80 2024-03-22T01:36:04.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.334+0000 7f8ddf839640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8dd810c890 con 0x7f8dd8102640 2024-03-22T01:36:04.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.334+0000 7f8ddd5d5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8dd8100a90 0x7f8dd8196de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:04.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.334+0000 7f8ddd5d5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8dd8100a90 0x7f8dd8196de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:35634/0 (socket says 172.21.15.72:35634) 2024-03-22T01:36:04.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.334+0000 7f8ddd5d5640 1 -- 172.21.15.72:0/147405656 learned_addr learned my addr 172.21.15.72:0/147405656 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:04.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.334+0000 7f8ddcdd4640 1 --2- 172.21.15.72:0/147405656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8dd8101c80 0x7f8dd8197320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:04.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.334+0000 7f8ddddd6640 1 --2- 172.21.15.72:0/147405656 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8dd8102640 0x7f8dd819e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:04.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.334+0000 7f8ddd5d5640 1 -- 172.21.15.72:0/147405656 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8dd8102640 msgr2=0x7f8dd819e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:04.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.334+0000 7f8ddd5d5640 1 --2- 172.21.15.72:0/147405656 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8dd8102640 0x7f8dd819e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:04.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.334+0000 7f8ddd5d5640 1 -- 172.21.15.72:0/147405656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8dd8101c80 msgr2=0x7f8dd8197320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:04.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.334+0000 7f8ddd5d5640 1 --2- 172.21.15.72:0/147405656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8dd8101c80 0x7f8dd8197320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:04.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.334+0000 7f8ddd5d5640 1 -- 172.21.15.72:0/147405656 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dd819ea00 con 0x7f8dd8100a90 2024-03-22T01:36:04.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.334+0000 7f8ddd5d5640 1 --2- 172.21.15.72:0/147405656 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8dd8100a90 0x7f8dd8196de0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f8dd4000c00 tx=0x7f8dd4005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:04.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.334+0000 7f8dce7fc640 1 -- 172.21.15.72:0/147405656 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dd4041070 con 0x7f8dd8100a90 2024-03-22T01:36:04.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.335+0000 7f8dce7fc640 1 -- 172.21.15.72:0/147405656 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8dd40044d0 con 0x7f8dd8100a90 2024-03-22T01:36:04.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.335+0000 7f8ddf839640 1 -- 172.21.15.72:0/147405656 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8dd819ec90 con 0x7f8dd8100a90 2024-03-22T01:36:04.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.335+0000 7f8dce7fc640 1 -- 172.21.15.72:0/147405656 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dd4034440 con 0x7f8dd8100a90 2024-03-22T01:36:04.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.335+0000 7f8ddf839640 1 -- 172.21.15.72:0/147405656 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8dd819f1d0 con 0x7f8dd8100a90 2024-03-22T01:36:04.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.336+0000 7f8ddf839640 1 -- 172.21.15.72:0/147405656 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8dd8107e80 con 0x7f8dd8100a90 2024-03-22T01:36:04.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.337+0000 7f8dce7fc640 1 -- 172.21.15.72:0/147405656 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7f8dd4044070 con 0x7f8dd8100a90 2024-03-22T01:36:04.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.337+0000 7f8dce7fc640 1 --2- 172.21.15.72:0/147405656 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f8dc40a2fa0 0x7f8dc40a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:04.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.337+0000 7f8ddcdd4640 1 --2- 172.21.15.72:0/147405656 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f8dc40a2fa0 0x7f8dc40a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:04.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.337+0000 7f8dce7fc640 1 -- 172.21.15.72:0/147405656 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f8dd40f1470 con 0x7f8dd8100a90 2024-03-22T01:36:04.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.338+0000 7f8ddcdd4640 1 --2- 172.21.15.72:0/147405656 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f8dc40a2fa0 0x7f8dc40a5460 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f8dd81982f0 tx=0x7f8dc8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:04.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.341+0000 7f8dce7fc640 1 -- 172.21.15.72:0/147405656 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8dd40bdf40 con 0x7f8dd8100a90 2024-03-22T01:36:04.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.515+0000 7f8ddf839640 1 -- 172.21.15.72:0/147405656 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f8dd802d540 con 0x7f8dd8100a90 2024-03-22T01:36:04.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.515+0000 7f8dce7fc640 1 -- 172.21.15.72:0/147405656 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f8dd40bd840 con 0x7f8dd8100a90 2024-03-22T01:36:04.518 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-03-22T01:36:04.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.517+0000 7f8ddf839640 1 -- 172.21.15.72:0/147405656 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f8dc40a2fa0 msgr2=0x7f8dc40a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:04.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.517+0000 7f8ddf839640 1 --2- 172.21.15.72:0/147405656 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f8dc40a2fa0 0x7f8dc40a5460 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f8dd81982f0 tx=0x7f8dc8005f90 comp rx=0 tx=0).stop 2024-03-22T01:36:04.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.517+0000 7f8ddf839640 1 -- 172.21.15.72:0/147405656 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8dd8100a90 msgr2=0x7f8dd8196de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:04.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.517+0000 7f8ddf839640 1 --2- 172.21.15.72:0/147405656 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8dd8100a90 0x7f8dd8196de0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f8dd4000c00 tx=0x7f8dd4005d50 comp rx=0 tx=0).stop 2024-03-22T01:36:04.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.517+0000 7f8ddf839640 1 -- 172.21.15.72:0/147405656 shutdown_connections 2024-03-22T01:36:04.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.517+0000 7f8ddf839640 1 --2- 172.21.15.72:0/147405656 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f8dc40a2fa0 0x7f8dc40a5460 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:04.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.517+0000 7f8ddf839640 1 --2- 172.21.15.72:0/147405656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8dd8101c80 0x7f8dd8197320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:04.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.517+0000 7f8ddf839640 1 --2- 172.21.15.72:0/147405656 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8dd8102640 0x7f8dd819e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:04.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.517+0000 7f8ddf839640 1 --2- 172.21.15.72:0/147405656 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8dd8100a90 0x7f8dd8196de0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:04.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.517+0000 7f8ddf839640 1 -- 172.21.15.72:0/147405656 >> 172.21.15.72:0/147405656 conn(0x7f8dd80fc000 msgr2=0x7f8dd8105050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:04.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.517+0000 7f8ddf839640 1 -- 172.21.15.72:0/147405656 shutdown_connections 2024-03-22T01:36:04.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.517+0000 7f8ddf839640 1 -- 172.21.15.72:0/147405656 wait complete. 2024-03-22T01:36:04.528 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-22T01:36:04.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.634+0000 7f0ef853e640 1 -- 172.21.15.72:0/1724986034 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0ef0102660 msgr2=0x7f0ef0108ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:04.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.634+0000 7f0ef853e640 1 --2- 172.21.15.72:0/1724986034 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0ef0102660 0x7f0ef0108ef0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8009ca0 tx=0x7f0ee802c0f0 comp rx=0 tx=0).stop 2024-03-22T01:36:04.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.634+0000 7f0ef853e640 1 -- 172.21.15.72:0/1724986034 shutdown_connections 2024-03-22T01:36:04.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.634+0000 7f0ef853e640 1 --2- 172.21.15.72:0/1724986034 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ef0101ca0 0x7f0ef0102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:04.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.634+0000 7f0ef853e640 1 --2- 172.21.15.72:0/1724986034 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0ef0102660 0x7f0ef0108ef0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:04.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.634+0000 7f0ef853e640 1 --2- 172.21.15.72:0/1724986034 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ef0100ab0 0x7f0ef0100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:04.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.634+0000 7f0ef853e640 1 -- 172.21.15.72:0/1724986034 >> 172.21.15.72:0/1724986034 conn(0x7f0ef00fc020 msgr2=0x7f0ef00fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:04.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.634+0000 7f0ef853e640 1 -- 172.21.15.72:0/1724986034 shutdown_connections 2024-03-22T01:36:04.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.634+0000 7f0ef853e640 1 -- 172.21.15.72:0/1724986034 wait complete. 2024-03-22T01:36:04.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.635+0000 7f0ef853e640 1 Processor -- start 2024-03-22T01:36:04.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.635+0000 7f0ef853e640 1 -- start start 2024-03-22T01:36:04.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.635+0000 7f0ef853e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ef0100ab0 0x7f0ef0196cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:04.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.635+0000 7f0ef853e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ef0101ca0 0x7f0ef0197230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:04.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.635+0000 7f0ef853e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0ef0102660 0x7f0ef019e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:04.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.635+0000 7f0ef853e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef010c710 con 0x7f0ef0102660 2024-03-22T01:36:04.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.635+0000 7f0ef853e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef010c590 con 0x7f0ef0100ab0 2024-03-22T01:36:04.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.635+0000 7f0ef853e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef010c890 con 0x7f0ef0101ca0 2024-03-22T01:36:04.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.635+0000 7f0ef6adb640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0ef0102660 0x7f0ef019e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:04.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.635+0000 7f0ef6adb640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0ef0102660 0x7f0ef019e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:35654/0 (socket says 172.21.15.72:35654) 2024-03-22T01:36:04.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.635+0000 7f0ef6adb640 1 -- 172.21.15.72:0/3727358543 learned_addr learned my addr 172.21.15.72:0/3727358543 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:04.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.635+0000 7f0ef62da640 1 --2- 172.21.15.72:0/3727358543 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ef0100ab0 0x7f0ef0196cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:04.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.636+0000 7f0ef6adb640 1 -- 172.21.15.72:0/3727358543 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ef0101ca0 msgr2=0x7f0ef0197230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:04.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.636+0000 7f0ef6adb640 1 --2- 172.21.15.72:0/3727358543 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ef0101ca0 0x7f0ef0197230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:04.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.636+0000 7f0ef6adb640 1 -- 172.21.15.72:0/3727358543 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ef0100ab0 msgr2=0x7f0ef0196cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:04.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.636+0000 7f0ef6adb640 1 --2- 172.21.15.72:0/3727358543 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ef0100ab0 0x7f0ef0196cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:04.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.636+0000 7f0ef6adb640 1 -- 172.21.15.72:0/3727358543 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ef019e900 con 0x7f0ef0102660 2024-03-22T01:36:04.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.636+0000 7f0ef6adb640 1 --2- 172.21.15.72:0/3727358543 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0ef0102660 0x7f0ef019e260 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8037040 tx=0x7f0ee8032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:04.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.636+0000 7f0ee77fe640 1 -- 172.21.15.72:0/3727358543 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee8034780 con 0x7f0ef0102660 2024-03-22T01:36:04.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.636+0000 7f0ee77fe640 1 -- 172.21.15.72:0/3727358543 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ee8032690 con 0x7f0ef0102660 2024-03-22T01:36:04.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.636+0000 7f0ef853e640 1 -- 172.21.15.72:0/3727358543 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ef0068e60 con 0x7f0ef0102660 2024-03-22T01:36:04.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.636+0000 7f0ee77fe640 1 -- 172.21.15.72:0/3727358543 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee80042a0 con 0x7f0ef0102660 2024-03-22T01:36:04.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.636+0000 7f0ef853e640 1 -- 172.21.15.72:0/3727358543 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ef0069350 con 0x7f0ef0102660 2024-03-22T01:36:04.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.637+0000 7f0ef853e640 1 -- 172.21.15.72:0/3727358543 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ef0068a10 con 0x7f0ef0102660 2024-03-22T01:36:04.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.638+0000 7f0ee77fe640 1 -- 172.21.15.72:0/3727358543 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7f0ee8032840 con 0x7f0ef0102660 2024-03-22T01:36:04.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.638+0000 7f0ee77fe640 1 --2- 172.21.15.72:0/3727358543 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f0edc0a2f20 0x7f0edc0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:04.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.639+0000 7f0ef62da640 1 --2- 172.21.15.72:0/3727358543 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f0edc0a2f20 0x7f0edc0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:04.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.639+0000 7f0ee77fe640 1 -- 172.21.15.72:0/3727358543 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f0ee80f3840 con 0x7f0ef0102660 2024-03-22T01:36:04.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.639+0000 7f0ef62da640 1 --2- 172.21.15.72:0/3727358543 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f0edc0a2f20 0x7f0edc0a53e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0ef01981c0 tx=0x7f0eec005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:04.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.642+0000 7f0ee77fe640 1 -- 172.21.15.72:0/3727358543 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0ee80c0310 con 0x7f0ef0102660 2024-03-22T01:36:04.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.818+0000 7f0ef853e640 1 -- 172.21.15.72:0/3727358543 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f0ef0107a20 con 0x7f0ef0102660 2024-03-22T01:36:04.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.818+0000 7f0ee77fe640 1 -- 172.21.15.72:0/3727358543 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7f0ee80bfc10 con 0x7f0ef0102660 2024-03-22T01:36:04.817 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-03-22T01:36:04.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.820+0000 7f0ef853e640 1 -- 172.21.15.72:0/3727358543 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f0edc0a2f20 msgr2=0x7f0edc0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:04.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.820+0000 7f0ef853e640 1 --2- 172.21.15.72:0/3727358543 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f0edc0a2f20 0x7f0edc0a53e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0ef01981c0 tx=0x7f0eec005f90 comp rx=0 tx=0).stop 2024-03-22T01:36:04.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.820+0000 7f0ef853e640 1 -- 172.21.15.72:0/3727358543 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0ef0102660 msgr2=0x7f0ef019e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:04.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.820+0000 7f0ef853e640 1 --2- 172.21.15.72:0/3727358543 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0ef0102660 0x7f0ef019e260 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8037040 tx=0x7f0ee8032f40 comp rx=0 tx=0).stop 2024-03-22T01:36:04.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.820+0000 7f0ef853e640 1 -- 172.21.15.72:0/3727358543 shutdown_connections 2024-03-22T01:36:04.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.820+0000 7f0ef853e640 1 --2- 172.21.15.72:0/3727358543 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f0edc0a2f20 0x7f0edc0a53e0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:04.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.820+0000 7f0ef853e640 1 --2- 172.21.15.72:0/3727358543 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0ef0102660 0x7f0ef019e260 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:04.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.820+0000 7f0ef853e640 1 --2- 172.21.15.72:0/3727358543 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ef0101ca0 0x7f0ef0197230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:04.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.820+0000 7f0ef853e640 1 --2- 172.21.15.72:0/3727358543 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ef0100ab0 0x7f0ef0196cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:04.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.820+0000 7f0ef853e640 1 -- 172.21.15.72:0/3727358543 >> 172.21.15.72:0/3727358543 conn(0x7f0ef00fc020 msgr2=0x7f0ef0104f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:04.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.820+0000 7f0ef853e640 1 -- 172.21.15.72:0/3727358543 shutdown_connections 2024-03-22T01:36:04.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.820+0000 7f0ef853e640 1 -- 172.21.15.72:0/3727358543 wait complete. 2024-03-22T01:36:04.829 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:04 smithi072 ceph-mon[32414]: pgmap v347: 129 pgs: 129 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-22T01:36:04.829 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:04 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/3091218110' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:36:04.829 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:04 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/19702380' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T01:36:04.829 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:04 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/626811904' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T01:36:04.830 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:04 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/147405656' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T01:36:04.831 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T01:36:04.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:04 smithi089 ceph-mon[35770]: pgmap v347: 129 pgs: 129 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-22T01:36:04.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:04 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/3091218110' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:36:04.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:04 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/19702380' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T01:36:04.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:04 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/626811904' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T01:36:04.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:04 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/147405656' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T01:36:04.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:04 smithi098 ceph-mon[36468]: pgmap v347: 129 pgs: 129 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-22T01:36:04.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:04 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/3091218110' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:36:04.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:04 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/19702380' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T01:36:04.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:04 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/626811904' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T01:36:04.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:04 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/147405656' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T01:36:04.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.933+0000 7f31a1866640 1 -- 172.21.15.72:0/4229476506 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f319c101c20 msgr2=0x7f319c102040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:04.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.933+0000 7f31a1866640 1 --2- 172.21.15.72:0/4229476506 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f319c101c20 0x7f319c102040 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f318c009900 tx=0x7f318c02bc30 comp rx=0 tx=0).stop 2024-03-22T01:36:04.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.933+0000 7f31a1866640 1 -- 172.21.15.72:0/4229476506 shutdown_connections 2024-03-22T01:36:04.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.933+0000 7f31a1866640 1 --2- 172.21.15.72:0/4229476506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f319c102e10 0x7f319c103290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:04.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.933+0000 7f31a1866640 1 --2- 172.21.15.72:0/4229476506 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f319c1037d0 0x7f319c10a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:04.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.933+0000 7f31a1866640 1 --2- 172.21.15.72:0/4229476506 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f319c101c20 0x7f319c102040 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:04.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.933+0000 7f31a1866640 1 -- 172.21.15.72:0/4229476506 >> 172.21.15.72:0/4229476506 conn(0x7f319c0fd1b0 msgr2=0x7f319c0ff5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:04.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.933+0000 7f31a1866640 1 -- 172.21.15.72:0/4229476506 shutdown_connections 2024-03-22T01:36:04.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.933+0000 7f31a1866640 1 -- 172.21.15.72:0/4229476506 wait complete. 2024-03-22T01:36:04.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.933+0000 7f31a1866640 1 Processor -- start 2024-03-22T01:36:04.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.934+0000 7f31a1866640 1 -- start start 2024-03-22T01:36:04.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.934+0000 7f31a1866640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f319c101c20 0x7f319c075940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:04.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.934+0000 7f31a1866640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f319c102e10 0x7f319c073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:04.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.934+0000 7f31a1866640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f319c1037d0 0x7f319c0744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:04.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.934+0000 7f31a1866640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f319c10d830 con 0x7f319c102e10 2024-03-22T01:36:04.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.934+0000 7f31a1866640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f319c10d6b0 con 0x7f319c101c20 2024-03-22T01:36:04.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.934+0000 7f31a1866640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f319c10d9b0 con 0x7f319c1037d0 2024-03-22T01:36:04.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.934+0000 7f319a7fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f319c102e10 0x7f319c073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:04.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.934+0000 7f319a7fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f319c102e10 0x7f319c073f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:35678/0 (socket says 172.21.15.72:35678) 2024-03-22T01:36:04.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.934+0000 7f319affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f319c101c20 0x7f319c075940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:04.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.934+0000 7f319a7fc640 1 -- 172.21.15.72:0/4220212356 learned_addr learned my addr 172.21.15.72:0/4220212356 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:04.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.934+0000 7f319b7fe640 1 --2- 172.21.15.72:0/4220212356 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f319c1037d0 0x7f319c0744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:04.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.934+0000 7f319a7fc640 1 -- 172.21.15.72:0/4220212356 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f319c1037d0 msgr2=0x7f319c0744c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:04.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.934+0000 7f319a7fc640 1 --2- 172.21.15.72:0/4220212356 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f319c1037d0 0x7f319c0744c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:04.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.934+0000 7f319a7fc640 1 -- 172.21.15.72:0/4220212356 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f319c101c20 msgr2=0x7f319c075940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:04.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.934+0000 7f319a7fc640 1 --2- 172.21.15.72:0/4220212356 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f319c101c20 0x7f319c075940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:04.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.934+0000 7f319a7fc640 1 -- 172.21.15.72:0/4220212356 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f319c074d50 con 0x7f319c102e10 2024-03-22T01:36:04.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.934+0000 7f319affd640 1 --2- 172.21.15.72:0/4220212356 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f319c101c20 0x7f319c075940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:36:04.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.934+0000 7f319a7fc640 1 --2- 172.21.15.72:0/4220212356 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f319c102e10 0x7f319c073f80 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f319000ea60 tx=0x7f319000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:04.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.935+0000 7f31a0864640 1 -- 172.21.15.72:0/4220212356 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f319000cde0 con 0x7f319c102e10 2024-03-22T01:36:04.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.935+0000 7f31a0864640 1 -- 172.21.15.72:0/4220212356 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31900035f0 con 0x7f319c102e10 2024-03-22T01:36:04.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.935+0000 7f31a1866640 1 -- 172.21.15.72:0/4220212356 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f319c1a1fc0 con 0x7f319c102e10 2024-03-22T01:36:04.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.935+0000 7f31a0864640 1 -- 172.21.15.72:0/4220212356 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3190010670 con 0x7f319c102e10 2024-03-22T01:36:04.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.935+0000 7f31a1866640 1 -- 172.21.15.72:0/4220212356 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f319c1a2420 con 0x7f319c102e10 2024-03-22T01:36:04.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.936+0000 7f31a1866640 1 -- 172.21.15.72:0/4220212356 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f319c108f40 con 0x7f319c102e10 2024-03-22T01:36:04.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.937+0000 7f31a0864640 1 -- 172.21.15.72:0/4220212356 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7f31900037a0 con 0x7f319c102e10 2024-03-22T01:36:04.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.937+0000 7f31a0864640 1 --2- 172.21.15.72:0/4220212356 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f31840a2f20 0x7f31840a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:04.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.937+0000 7f319affd640 1 --2- 172.21.15.72:0/4220212356 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f31840a2f20 0x7f31840a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:04.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.938+0000 7f31a0864640 1 -- 172.21.15.72:0/4220212356 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f31900cfad0 con 0x7f319c102e10 2024-03-22T01:36:04.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.938+0000 7f319affd640 1 --2- 172.21.15.72:0/4220212356 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f31840a2f20 0x7f31840a53e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f318c000c00 tx=0x7f318c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:04.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:04.942+0000 7f31a0864640 1 -- 172.21.15.72:0/4220212356 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f319009c5a0 con 0x7f319c102e10 2024-03-22T01:36:05.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.153+0000 7f31a1866640 1 -- 172.21.15.72:0/4220212356 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f319c062890 con 0x7f319c102e10 2024-03-22T01:36:05.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.154+0000 7f31a0864640 1 -- 172.21.15.72:0/4220212356 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11785 (secure 0 0 0) 0x7f319009bea0 con 0x7f319c102e10 2024-03-22T01:36:05.153 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:36:05.153 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6835/3077216308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3077216308},{"type":"v1","addr":"172.21.15.72:6835","nonce":3077216308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2460254143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":2460254143},{"type":"v1","addr":"172.21.15.72:6837","nonce":2460254143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6839/2026432819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6838","nonce":2026432819},{"type":"v1","addr":"172.21.15.72:6839","nonce":2026432819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6841/400094545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6840","nonce":400094545},{"type":"v1","addr":"172.21.15.72:6841","nonce":400094545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6833/1578701171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":1578701171},{"type":"v1","addr":"172.21.15.89:6833","nonce":1578701171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/2703047213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2703047213},{"type":"v1","addr":"172.21.15.98:6833","nonce":2703047213}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2589468787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2589468787},{"type":"v1","addr":"172.21.15.89:6835","nonce":2589468787}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/92657636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":92657636},{"type":"v1","addr":"172.21.15.98:6835","nonce":92657636}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1662941782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1662941782},{"type":"v1","addr":"172.21.15.89:6837","nonce":1662941782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24490,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/781860371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":781860371},{"type":"v1","addr":"172.21.15.98:6837","nonce":781860371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-22T01:34:14.815677+0000","modified":"2024-03-22T01:34:39.842959+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24524},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3192173650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3192173650},{"type":"v1","addr":"172.21.15.98:6839","nonce":3192173650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6839/149200280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":149200280},{"type":"v1","addr":"172.21.15.89:6839","nonce":149200280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24524]},"id":1}]} 2024-03-22T01:36:05.153 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 21 2024-03-22T01:36:05.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.156+0000 7f31a1866640 1 -- 172.21.15.72:0/4220212356 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f31840a2f20 msgr2=0x7f31840a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:05.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.156+0000 7f31a1866640 1 --2- 172.21.15.72:0/4220212356 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f31840a2f20 0x7f31840a53e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f318c000c00 tx=0x7f318c005fd0 comp rx=0 tx=0).stop 2024-03-22T01:36:05.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.156+0000 7f31a1866640 1 -- 172.21.15.72:0/4220212356 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f319c102e10 msgr2=0x7f319c073f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:05.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.156+0000 7f31a1866640 1 --2- 172.21.15.72:0/4220212356 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f319c102e10 0x7f319c073f80 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f319000ea60 tx=0x7f319000ef20 comp rx=0 tx=0).stop 2024-03-22T01:36:05.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.156+0000 7f31a1866640 1 -- 172.21.15.72:0/4220212356 shutdown_connections 2024-03-22T01:36:05.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.156+0000 7f31a1866640 1 --2- 172.21.15.72:0/4220212356 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f31840a2f20 0x7f31840a53e0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:05.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.156+0000 7f31a1866640 1 --2- 172.21.15.72:0/4220212356 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f319c1037d0 0x7f319c0744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:05.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.156+0000 7f31a1866640 1 --2- 172.21.15.72:0/4220212356 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f319c102e10 0x7f319c073f80 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:05.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.156+0000 7f31a1866640 1 --2- 172.21.15.72:0/4220212356 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f319c101c20 0x7f319c075940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:05.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.156+0000 7f31a1866640 1 -- 172.21.15.72:0/4220212356 >> 172.21.15.72:0/4220212356 conn(0x7f319c0fd1b0 msgr2=0x7f319c106110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:05.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.156+0000 7f31a1866640 1 -- 172.21.15.72:0/4220212356 shutdown_connections 2024-03-22T01:36:05.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.156+0000 7f31a1866640 1 -- 172.21.15.72:0/4220212356 wait complete. 2024-03-22T01:36:05.167 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T01:36:05.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.270+0000 7fe655956640 1 -- 172.21.15.72:0/2016941961 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe650101d20 msgr2=0x7fe6501021a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:05.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.270+0000 7fe655956640 1 --2- 172.21.15.72:0/2016941961 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe650101d20 0x7fe6501021a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe638009860 tx=0x7fe63802bb70 comp rx=0 tx=0).stop 2024-03-22T01:36:05.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.270+0000 7fe655956640 1 -- 172.21.15.72:0/2016941961 shutdown_connections 2024-03-22T01:36:05.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.270+0000 7fe655956640 1 --2- 172.21.15.72:0/2016941961 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe650101d20 0x7fe6501021a0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:05.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.270+0000 7fe655956640 1 --2- 172.21.15.72:0/2016941961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6501026e0 0x7fe650108f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:05.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.270+0000 7fe655956640 1 --2- 172.21.15.72:0/2016941961 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe650100b30 0x7fe650100f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:05.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.270+0000 7fe655956640 1 -- 172.21.15.72:0/2016941961 >> 172.21.15.72:0/2016941961 conn(0x7fe6500fc0c0 msgr2=0x7fe6500fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:05.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.270+0000 7fe655956640 1 -- 172.21.15.72:0/2016941961 shutdown_connections 2024-03-22T01:36:05.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.270+0000 7fe655956640 1 -- 172.21.15.72:0/2016941961 wait complete. 2024-03-22T01:36:05.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.270+0000 7fe655956640 1 Processor -- start 2024-03-22T01:36:05.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.270+0000 7fe655956640 1 -- start start 2024-03-22T01:36:05.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.270+0000 7fe655956640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe650100b30 0x7fe650197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:05.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.270+0000 7fe655956640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe650101d20 0x7fe6501983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:05.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.270+0000 7fe655956640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe6501026e0 0x7fe65019f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:05.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.270+0000 7fe655956640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fe65010d860 con 0x7fe6501026e0 2024-03-22T01:36:05.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.270+0000 7fe655956640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe65010d6e0 con 0x7fe650100b30 2024-03-22T01:36:05.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.270+0000 7fe655956640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe65010d9e0 con 0x7fe650101d20 2024-03-22T01:36:05.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.271+0000 7fe64f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe6501026e0 0x7fe65019f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:05.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.271+0000 7fe64f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe6501026e0 0x7fe65019f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:35688/0 (socket says 172.21.15.72:35688) 2024-03-22T01:36:05.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.271+0000 7fe64f7fe640 1 -- 172.21.15.72:0/3247125269 learned_addr learned my addr 172.21.15.72:0/3247125269 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:05.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.271+0000 7fe64effd640 1 --2- 172.21.15.72:0/3247125269 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe650100b30 0x7fe650197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:05.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.271+0000 7fe64e7fc640 1 --2- 172.21.15.72:0/3247125269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe650101d20 0x7fe6501983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:05.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.271+0000 7fe64f7fe640 1 -- 172.21.15.72:0/3247125269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe650101d20 msgr2=0x7fe6501983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:05.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.271+0000 7fe64f7fe640 1 --2- 172.21.15.72:0/3247125269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe650101d20 0x7fe6501983d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:05.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.271+0000 7fe64f7fe640 1 -- 172.21.15.72:0/3247125269 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe650100b30 msgr2=0x7fe650197e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:05.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.271+0000 7fe64f7fe640 1 --2- 172.21.15.72:0/3247125269 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe650100b30 0x7fe650197e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:05.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.271+0000 7fe64f7fe640 1 -- 172.21.15.72:0/3247125269 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe65019fab0 con 0x7fe6501026e0 2024-03-22T01:36:05.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.271+0000 7fe64e7fc640 1 --2- 172.21.15.72:0/3247125269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe650101d20 0x7fe6501983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:36:05.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.271+0000 7fe64f7fe640 1 --2- 172.21.15.72:0/3247125269 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe6501026e0 0x7fe65019f440 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fe64800eaa0 tx=0x7fe64800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:05.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.271+0000 7fe654954640 1 -- 172.21.15.72:0/3247125269 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe64800cdb0 con 0x7fe6501026e0 2024-03-22T01:36:05.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.271+0000 7fe654954640 1 -- 172.21.15.72:0/3247125269 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe648005020 con 0x7fe6501026e0 2024-03-22T01:36:05.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.271+0000 7fe655956640 1 -- 172.21.15.72:0/3247125269 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe65019fda0 con 0x7fe6501026e0 2024-03-22T01:36:05.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.271+0000 7fe654954640 1 -- 172.21.15.72:0/3247125269 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6480116b0 con 0x7fe6501026e0 2024-03-22T01:36:05.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.271+0000 7fe655956640 1 -- 172.21.15.72:0/3247125269 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6501a0260 con 0x7fe6501026e0 2024-03-22T01:36:05.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.272+0000 7fe655956640 1 -- 172.21.15.72:0/3247125269 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe650107e50 con 0x7fe6501026e0 2024-03-22T01:36:05.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.273+0000 7fe654954640 1 -- 172.21.15.72:0/3247125269 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7fe648005560 con 0x7fe6501026e0 2024-03-22T01:36:05.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.274+0000 7fe654954640 1 --2- 172.21.15.72:0/3247125269 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe63c0a2f20 0x7fe63c0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:05.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.274+0000 7fe64effd640 1 --2- 172.21.15.72:0/3247125269 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe63c0a2f20 0x7fe63c0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:05.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.274+0000 7fe654954640 1 -- 172.21.15.72:0/3247125269 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fe6480cfd20 con 0x7fe6501026e0 2024-03-22T01:36:05.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.274+0000 7fe64effd640 1 --2- 172.21.15.72:0/3247125269 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe63c0a2f20 0x7fe63c0a53e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe640009530 tx=0x7fe640009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:05.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.277+0000 7fe654954640 1 -- 172.21.15.72:0/3247125269 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe648016030 con 0x7fe6501026e0 2024-03-22T01:36:05.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.455+0000 7fe655956640 1 -- 172.21.15.72:0/3247125269 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe650062830 con 0x7fe6501026e0 2024-03-22T01:36:05.456 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.457+0000 7fe654954640 1 -- 172.21.15.72:0/3247125269 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24570 (secure 0 0 0) 0x7fe64809c7f0 con 0x7fe6501026e0 2024-03-22T01:36:05.456 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:36:05.457 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":93,"fsid":"022a99bc-e7eb-11ee-95cd-87774f69a715","created":"2024-03-22T01:26:23.893672+0000","modified":"2024-03-22T01:35:16.738619+0000","last_up_change":"2024-03-22T01:33:25.223684+0000","last_in_change":"2024-03-22T01:33:05.010331+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T01:29:37.438503+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-22T01:34:12.610422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-22T01:34:13.761686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-22T01:34:17.507039+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"16c15bb0-db92-4191-9063-38b4890e81e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6803","nonce":661906791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6805","nonce":661906791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6809","nonce":661906791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6807","nonce":661906791}]},"public_addr":"172.21.15.72:6803/661906791","cluster_addr":"172.21.15.72:6805/661906791","heartbeat_back_addr":"172.21.15.72:6809/661906791","heartbeat_front_addr":"172.21.15.72:6807/661906791","state":["exists","up"]},{"osd":1,"uuid":"981290ba-97dc-496b-aa22-d51704184b10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6801","nonce":2870590550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6803","nonce":2870590550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6807","nonce":2870590550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6805","nonce":2870590550}]},"public_addr":"172.21.15.89:6801/2870590550","cluster_addr":"172.21.15.89:6803/2870590550","heartbeat_back_addr":"172.21.15.89:6807/2870590550","heartbeat_front_addr":"172.21.15.89:6805/2870590550","state":["exists","up"]},{"osd":2,"uuid":"f07ed658-f545-46a1-8edd-8b1d5d49885c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6801","nonce":3865778118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6803","nonce":3865778118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6807","nonce":3865778118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6805","nonce":3865778118}]},"public_addr":"172.21.15.98:6801/3865778118","cluster_addr":"172.21.15.98:6803/3865778118","heartbeat_back_addr":"172.21.15.98:6807/3865778118","heartbeat_front_addr":"172.21.15.98:6805/3865778118","state":["exists","up"]},{"osd":3,"uuid":"ccc4bc40-2c48-46d2-8548-0c05b44f9f1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6811","nonce":1865951395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6813","nonce":1865951395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6817","nonce":1865951395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6815","nonce":1865951395}]},"public_addr":"172.21.15.72:6811/1865951395","cluster_addr":"172.21.15.72:6813/1865951395","heartbeat_back_addr":"172.21.15.72:6817/1865951395","heartbeat_front_addr":"172.21.15.72:6815/1865951395","state":["exists","up"]},{"osd":4,"uuid":"1812439f-810a-4ed2-9395-c00c9a085a13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6809","nonce":2683760795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6811","nonce":2683760795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6815","nonce":2683760795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6813","nonce":2683760795}]},"public_addr":"172.21.15.89:6809/2683760795","cluster_addr":"172.21.15.89:6811/2683760795","heartbeat_back_addr":"172.21.15.89:6815/2683760795","heartbeat_front_addr":"172.21.15.89:6813/2683760795","state":["exists","up"]},{"osd":5,"uuid":"e78954d8-f4e3-4d58-8536-d2d59c843386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6809","nonce":2635838572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6811","nonce":2635838572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6815","nonce":2635838572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6813","nonce":2635838572}]},"public_addr":"172.21.15.98:6809/2635838572","cluster_addr":"172.21.15.98:6811/2635838572","heartbeat_back_addr":"172.21.15.98:6815/2635838572","heartbeat_front_addr":"172.21.15.98:6813/2635838572","state":["exists","up"]},{"osd":6,"uuid":"dc6c8be2-b64c-4959-8015-7b0ea6d25599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6819","nonce":3158982661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6821","nonce":3158982661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6825","nonce":3158982661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6823","nonce":3158982661}]},"public_addr":"172.21.15.72:6819/3158982661","cluster_addr":"172.21.15.72:6821/3158982661","heartbeat_back_addr":"172.21.15.72:6825/3158982661","heartbeat_front_addr":"172.21.15.72:6823/3158982661","state":["exists","up"]},{"osd":7,"uuid":"7a73469f-cf3a-4306-ae93-ea46e21ebbbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6817","nonce":3075913234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6819","nonce":3075913234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6823","nonce":3075913234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6821","nonce":3075913234}]},"public_addr":"172.21.15.89:6817/3075913234","cluster_addr":"172.21.15.89:6819/3075913234","heartbeat_back_addr":"172.21.15.89:6823/3075913234","heartbeat_front_addr":"172.21.15.89:6821/3075913234","state":["exists","up"]},{"osd":8,"uuid":"c4b13bee-ea4d-4344-95fb-2622ecbb8196","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6817","nonce":1813255577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6819","nonce":1813255577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6823","nonce":1813255577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6821","nonce":1813255577}]},"public_addr":"172.21.15.98:6817/1813255577","cluster_addr":"172.21.15.98:6819/1813255577","heartbeat_back_addr":"172.21.15.98:6823/1813255577","heartbeat_front_addr":"172.21.15.98:6821/1813255577","state":["exists","up"]},{"osd":9,"uuid":"eefb0a44-4a36-4e55-9d04-0501bf718db6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6827","nonce":3377749206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6829","nonce":3377749206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6832","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6833","nonce":3377749206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6831","nonce":3377749206}]},"public_addr":"172.21.15.72:6827/3377749206","cluster_addr":"172.21.15.72:6829/3377749206","heartbeat_back_addr":"172.21.15.72:6833/3377749206","heartbeat_front_addr":"172.21.15.72:6831/3377749206","state":["exists","up"]},{"osd":10,"uuid":"8741d25f-40dd-40d7-b040-5cc5e411eb98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6825","nonce":885058786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6827","nonce":885058786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6831","nonce":885058786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6829","nonce":885058786}]},"public_addr":"172.21.15.89:6825/885058786","cluster_addr":"172.21.15.89:6827/885058786","heartbeat_back_addr":"172.21.15.89:6831/885058786","heartbeat_front_addr":"172.21.15.89:6829/885058786","state":["exists","up"]},{"osd":11,"uuid":"8d02e957-eda4-4184-b5c6-5eb3b8b38060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6825","nonce":1517095838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6827","nonce":1517095838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6831","nonce":1517095838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6829","nonce":1517095838}]},"public_addr":"172.21.15.98:6825/1517095838","cluster_addr":"172.21.15.98:6827/1517095838","heartbeat_back_addr":"172.21.15.98:6831/1517095838","heartbeat_front_addr":"172.21.15.98:6829/1517095838","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:28:47.307652+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:11.037135+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:34.619188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:59.350738+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:30:23.787337+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:30:48.994154+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:31:13.690350+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:31:39.628116+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:04.723912+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:30.689741+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:57.229508+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:33:23.154537+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:0/121087196":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/2451857761":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/1228477995":"2024-03-23T01:27:15.339809+0000","172.21.15.72:6801/1513232913":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/3558958264":"2024-03-23T01:26:50.074871+0000","172.21.15.72:0/1510187198":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6800/3518012916":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6801/3518012916":"2024-03-23T01:26:50.074871+0000","172.21.15.72:0/1372801041":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6800/1513232913":"2024-03-23T01:27:15.339809+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T01:36:05.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.459+0000 7fe655956640 1 -- 172.21.15.72:0/3247125269 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe63c0a2f20 msgr2=0x7fe63c0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:05.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.459+0000 7fe655956640 1 --2- 172.21.15.72:0/3247125269 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe63c0a2f20 0x7fe63c0a53e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe640009530 tx=0x7fe640009260 comp rx=0 tx=0).stop 2024-03-22T01:36:05.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.459+0000 7fe655956640 1 -- 172.21.15.72:0/3247125269 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe6501026e0 msgr2=0x7fe65019f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:05.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.459+0000 7fe655956640 1 --2- 172.21.15.72:0/3247125269 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe6501026e0 0x7fe65019f440 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fe64800eaa0 tx=0x7fe64800ef60 comp rx=0 tx=0).stop 2024-03-22T01:36:05.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.459+0000 7fe655956640 1 -- 172.21.15.72:0/3247125269 shutdown_connections 2024-03-22T01:36:05.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.459+0000 7fe655956640 1 --2- 172.21.15.72:0/3247125269 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fe63c0a2f20 0x7fe63c0a53e0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:05.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.459+0000 7fe655956640 1 --2- 172.21.15.72:0/3247125269 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe6501026e0 0x7fe65019f440 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:05.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.459+0000 7fe655956640 1 --2- 172.21.15.72:0/3247125269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe650101d20 0x7fe6501983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:05.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.459+0000 7fe655956640 1 --2- 172.21.15.72:0/3247125269 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe650100b30 0x7fe650197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:05.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.459+0000 7fe655956640 1 -- 172.21.15.72:0/3247125269 >> 172.21.15.72:0/3247125269 conn(0x7fe6500fc0c0 msgr2=0x7fe650105020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:05.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.459+0000 7fe655956640 1 -- 172.21.15.72:0/3247125269 shutdown_connections 2024-03-22T01:36:05.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.459+0000 7fe655956640 1 -- 172.21.15.72:0/3247125269 wait complete. 2024-03-22T01:36:05.470 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T01:36:05.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.573+0000 7faa75fac640 1 -- 172.21.15.72:0/522881669 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa70104fe0 msgr2=0x7faa701073d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:05.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.573+0000 7faa75fac640 1 --2- 172.21.15.72:0/522881669 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa70104fe0 0x7faa701073d0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7faa64009900 tx=0x7faa6402bc30 comp rx=0 tx=0).stop 2024-03-22T01:36:05.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.573+0000 7faa75fac640 1 -- 172.21.15.72:0/522881669 shutdown_connections 2024-03-22T01:36:05.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.573+0000 7faa75fac640 1 --2- 172.21.15.72:0/522881669 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa70107910 0x7faa70109d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:05.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.573+0000 7faa75fac640 1 --2- 172.21.15.72:0/522881669 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa70104fe0 0x7faa701073d0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:05.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.573+0000 7faa75fac640 1 --2- 172.21.15.72:0/522881669 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa701026b0 0x7faa70104aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:05.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.573+0000 7faa75fac640 1 -- 172.21.15.72:0/522881669 >> 172.21.15.72:0/522881669 conn(0x7faa700fc0a0 msgr2=0x7faa700fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:05.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.573+0000 7faa75fac640 1 -- 172.21.15.72:0/522881669 shutdown_connections 2024-03-22T01:36:05.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.573+0000 7faa75fac640 1 -- 172.21.15.72:0/522881669 wait complete. 2024-03-22T01:36:05.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.573+0000 7faa75fac640 1 Processor -- start 2024-03-22T01:36:05.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.574+0000 7faa75fac640 1 -- start start 2024-03-22T01:36:05.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.574+0000 7faa75fac640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa701026b0 0x7faa70197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:05.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.574+0000 7faa75fac640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa70104fe0 0x7faa701983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:05.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.574+0000 7faa75fac640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa70107910 0x7faa7019f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:05.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.574+0000 7faa75fac640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7faa7010d830 con 0x7faa70104fe0 2024-03-22T01:36:05.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.574+0000 7faa75fac640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7faa7010d6b0 con 0x7faa70107910 2024-03-22T01:36:05.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.574+0000 7faa75fac640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7faa7010d9b0 con 0x7faa701026b0 2024-03-22T01:36:05.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.574+0000 7faa6effd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa70104fe0 0x7faa701983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:05.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.574+0000 7faa6effd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa70104fe0 0x7faa701983f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:35708/0 (socket says 172.21.15.72:35708) 2024-03-22T01:36:05.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.574+0000 7faa6effd640 1 -- 172.21.15.72:0/947018352 learned_addr learned my addr 172.21.15.72:0/947018352 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:05.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.574+0000 7faa6ffff640 1 --2- 172.21.15.72:0/947018352 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa70107910 0x7faa7019f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:05.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.574+0000 7faa6f7fe640 1 --2- 172.21.15.72:0/947018352 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa701026b0 0x7faa70197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:05.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.574+0000 7faa6effd640 1 -- 172.21.15.72:0/947018352 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa701026b0 msgr2=0x7faa70197eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:05.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.574+0000 7faa6effd640 1 --2- 172.21.15.72:0/947018352 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa701026b0 0x7faa70197eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:05.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.574+0000 7faa6effd640 1 -- 172.21.15.72:0/947018352 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa70107910 msgr2=0x7faa7019f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:05.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.574+0000 7faa6effd640 1 --2- 172.21.15.72:0/947018352 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa70107910 0x7faa7019f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:05.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.574+0000 7faa6effd640 1 -- 172.21.15.72:0/947018352 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa7019fad0 con 0x7faa70104fe0 2024-03-22T01:36:05.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.574+0000 7faa6f7fe640 1 --2- 172.21.15.72:0/947018352 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa701026b0 0x7faa70197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T01:36:05.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.574+0000 7faa6ffff640 1 --2- 172.21.15.72:0/947018352 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa70107910 0x7faa7019f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:36:05.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.574+0000 7faa6effd640 1 --2- 172.21.15.72:0/947018352 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa70104fe0 0x7faa701983f0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7faa64000c00 tx=0x7faa64005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:05.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.574+0000 7faa6cff9640 1 -- 172.21.15.72:0/947018352 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa64041070 con 0x7faa70104fe0 2024-03-22T01:36:05.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.575+0000 7faa75fac640 1 -- 172.21.15.72:0/947018352 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa7019fd60 con 0x7faa70104fe0 2024-03-22T01:36:05.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.575+0000 7faa6cff9640 1 -- 172.21.15.72:0/947018352 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa640044d0 con 0x7faa70104fe0 2024-03-22T01:36:05.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.575+0000 7faa6cff9640 1 -- 172.21.15.72:0/947018352 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa64034440 con 0x7faa70104fe0 2024-03-22T01:36:05.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.575+0000 7faa75fac640 1 -- 172.21.15.72:0/947018352 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa701a01f0 con 0x7faa70104fe0 2024-03-22T01:36:05.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.575+0000 7faa75fac640 1 -- 172.21.15.72:0/947018352 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa700fdb90 con 0x7faa70104fe0 2024-03-22T01:36:05.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.577+0000 7faa6cff9640 1 -- 172.21.15.72:0/947018352 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7faa64044070 con 0x7faa70104fe0 2024-03-22T01:36:05.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.577+0000 7faa6cff9640 1 --2- 172.21.15.72:0/947018352 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7faa580a2f20 0x7faa580a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:05.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.577+0000 7faa6f7fe640 1 --2- 172.21.15.72:0/947018352 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7faa580a2f20 0x7faa580a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:05.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.577+0000 7faa6cff9640 1 -- 172.21.15.72:0/947018352 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7faa640f14d0 con 0x7faa70104fe0 2024-03-22T01:36:05.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.578+0000 7faa6f7fe640 1 --2- 172.21.15.72:0/947018352 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7faa580a2f20 0x7faa580a53e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7faa60009b50 tx=0x7faa60005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:05.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.581+0000 7faa6cff9640 1 -- 172.21.15.72:0/947018352 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7faa640bdfa0 con 0x7faa70104fe0 2024-03-22T01:36:05.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.793+0000 7faa75fac640 1 -- 172.21.15.72:0/947018352 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faa7002d570 con 0x7faa70104fe0 2024-03-22T01:36:05.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.794+0000 7faa6cff9640 1 -- 172.21.15.72:0/947018352 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11785 (secure 0 0 0) 0x7faa640bd8a0 con 0x7faa70104fe0 2024-03-22T01:36:05.793 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:36:05.793 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6835/3077216308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3077216308},{"type":"v1","addr":"172.21.15.72:6835","nonce":3077216308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2460254143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":2460254143},{"type":"v1","addr":"172.21.15.72:6837","nonce":2460254143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6839/2026432819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6838","nonce":2026432819},{"type":"v1","addr":"172.21.15.72:6839","nonce":2026432819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6841/400094545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6840","nonce":400094545},{"type":"v1","addr":"172.21.15.72:6841","nonce":400094545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6833/1578701171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":1578701171},{"type":"v1","addr":"172.21.15.89:6833","nonce":1578701171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/2703047213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2703047213},{"type":"v1","addr":"172.21.15.98:6833","nonce":2703047213}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2589468787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2589468787},{"type":"v1","addr":"172.21.15.89:6835","nonce":2589468787}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/92657636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":92657636},{"type":"v1","addr":"172.21.15.98:6835","nonce":92657636}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1662941782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1662941782},{"type":"v1","addr":"172.21.15.89:6837","nonce":1662941782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24490,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/781860371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":781860371},{"type":"v1","addr":"172.21.15.98:6837","nonce":781860371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-22T01:34:14.815677+0000","modified":"2024-03-22T01:34:39.842959+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24524},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3192173650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3192173650},{"type":"v1","addr":"172.21.15.98:6839","nonce":3192173650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6839/149200280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":149200280},{"type":"v1","addr":"172.21.15.89:6839","nonce":149200280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24524]},"id":1}]} 2024-03-22T01:36:05.794 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 21 2024-03-22T01:36:05.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.796+0000 7faa75fac640 1 -- 172.21.15.72:0/947018352 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7faa580a2f20 msgr2=0x7faa580a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:05.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.796+0000 7faa75fac640 1 --2- 172.21.15.72:0/947018352 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7faa580a2f20 0x7faa580a53e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7faa60009b50 tx=0x7faa60005f90 comp rx=0 tx=0).stop 2024-03-22T01:36:05.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.796+0000 7faa75fac640 1 -- 172.21.15.72:0/947018352 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa70104fe0 msgr2=0x7faa701983f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:05.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.796+0000 7faa75fac640 1 --2- 172.21.15.72:0/947018352 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa70104fe0 0x7faa701983f0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7faa64000c00 tx=0x7faa64005d50 comp rx=0 tx=0).stop 2024-03-22T01:36:05.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.796+0000 7faa75fac640 1 -- 172.21.15.72:0/947018352 shutdown_connections 2024-03-22T01:36:05.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.796+0000 7faa75fac640 1 --2- 172.21.15.72:0/947018352 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7faa580a2f20 0x7faa580a53e0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:05.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.796+0000 7faa75fac640 1 --2- 172.21.15.72:0/947018352 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa70107910 0x7faa7019f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:05.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.796+0000 7faa75fac640 1 --2- 172.21.15.72:0/947018352 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa70104fe0 0x7faa701983f0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:05.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.796+0000 7faa75fac640 1 --2- 172.21.15.72:0/947018352 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa701026b0 0x7faa70197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:05.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.796+0000 7faa75fac640 1 -- 172.21.15.72:0/947018352 >> 172.21.15.72:0/947018352 conn(0x7faa700fc0a0 msgr2=0x7faa70103010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:05.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.796+0000 7faa75fac640 1 -- 172.21.15.72:0/947018352 shutdown_connections 2024-03-22T01:36:05.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.796+0000 7faa75fac640 1 -- 172.21.15.72:0/947018352 wait complete. 2024-03-22T01:36:05.805 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:05 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/3727358543' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T01:36:05.806 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:05 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/4220212356' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:36:05.806 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:05 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/3247125269' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:36:05.807 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 21, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-22T01:34:14.815677+0000', 'modified': '2024-03-22T01:34:39.842959+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24524}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24517': {'gid': 24517, 'name': 'i', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.98:6839/3192173650', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6838', 'nonce': 3192173650}, {'type': 'v1', 'addr': '172.21.15.98:6839', 'nonce': 3192173650}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24524': {'gid': 24524, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.89:6839/149200280', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6838', 'nonce': 149200280}, {'type': 'v1', 'addr': '172.21.15.89:6839', 'nonce': 149200280}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24524, 'qdb_cluster': [24524]} 2024-03-22T01:36:05.807 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-22T01:36:05.808 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-22T01:36:05.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:05 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/3727358543' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T01:36:05.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:05 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/4220212356' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:36:05.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:05 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/3247125269' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:36:05.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.911+0000 7f89cf53e640 1 -- 172.21.15.72:0/481860662 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f89c8104f60 msgr2=0x7f89c8107350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:05.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.911+0000 7f89cf53e640 1 --2- 172.21.15.72:0/481860662 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f89c8104f60 0x7f89c8107350 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f89b8009900 tx=0x7f89b802bc30 comp rx=0 tx=0).stop 2024-03-22T01:36:05.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.911+0000 7f89cf53e640 1 -- 172.21.15.72:0/481860662 shutdown_connections 2024-03-22T01:36:05.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.911+0000 7f89cf53e640 1 --2- 172.21.15.72:0/481860662 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89c8107890 0x7f89c8109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:05.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.911+0000 7f89cf53e640 1 --2- 172.21.15.72:0/481860662 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f89c8104f60 0x7f89c8107350 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:05.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.911+0000 7f89cf53e640 1 --2- 172.21.15.72:0/481860662 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89c8102630 0x7f89c8104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:05.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.911+0000 7f89cf53e640 1 -- 172.21.15.72:0/481860662 >> 172.21.15.72:0/481860662 conn(0x7f89c80fc020 msgr2=0x7f89c80fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:05.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.911+0000 7f89cf53e640 1 -- 172.21.15.72:0/481860662 shutdown_connections 2024-03-22T01:36:05.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.911+0000 7f89cf53e640 1 -- 172.21.15.72:0/481860662 wait complete. 2024-03-22T01:36:05.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.912+0000 7f89cf53e640 1 Processor -- start 2024-03-22T01:36:05.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.912+0000 7f89cf53e640 1 -- start start 2024-03-22T01:36:05.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.912+0000 7f89cf53e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89c8102630 0x7f89c8101fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:05.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.912+0000 7f89cf53e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f89c8104f60 0x7f89c81005e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:05.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.912+0000 7f89cf53e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89c8107890 0x7f89c8100b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:05.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.912+0000 7f89cf53e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f89c810d890 con 0x7f89c8104f60 2024-03-22T01:36:05.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.912+0000 7f89cf53e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f89c810d710 con 0x7f89c8102630 2024-03-22T01:36:05.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.912+0000 7f89cf53e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f89c810da10 con 0x7f89c8107890 2024-03-22T01:36:05.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.912+0000 7f89ccad9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f89c8104f60 0x7f89c81005e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:05.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.912+0000 7f89ccad9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f89c8104f60 0x7f89c81005e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:35732/0 (socket says 172.21.15.72:35732) 2024-03-22T01:36:05.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.912+0000 7f89ccad9640 1 -- 172.21.15.72:0/2689532924 learned_addr learned my addr 172.21.15.72:0/2689532924 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:05.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.912+0000 7f89cd2da640 1 --2- 172.21.15.72:0/2689532924 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89c8102630 0x7f89c8101fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:05.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.912+0000 7f89cdadb640 1 --2- 172.21.15.72:0/2689532924 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89c8107890 0x7f89c8100b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:05.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.912+0000 7f89ccad9640 1 -- 172.21.15.72:0/2689532924 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89c8107890 msgr2=0x7f89c8100b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:05.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.912+0000 7f89ccad9640 1 --2- 172.21.15.72:0/2689532924 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89c8107890 0x7f89c8100b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:05.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.912+0000 7f89ccad9640 1 -- 172.21.15.72:0/2689532924 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89c8102630 msgr2=0x7f89c8101fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:05.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.912+0000 7f89ccad9640 1 --2- 172.21.15.72:0/2689532924 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89c8102630 0x7f89c8101fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:05.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.912+0000 7f89ccad9640 1 -- 172.21.15.72:0/2689532924 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89c81013b0 con 0x7f89c8104f60 2024-03-22T01:36:05.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.912+0000 7f89cd2da640 1 --2- 172.21.15.72:0/2689532924 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89c8102630 0x7f89c8101fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:36:05.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.912+0000 7f89ccad9640 1 --2- 172.21.15.72:0/2689532924 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f89c8104f60 0x7f89c81005e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f89b8000c00 tx=0x7f89b802c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:05.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.913+0000 7f89be7fc640 1 -- 172.21.15.72:0/2689532924 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b8043020 con 0x7f89c8104f60 2024-03-22T01:36:05.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.913+0000 7f89cf53e640 1 -- 172.21.15.72:0/2689532924 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89c81a2080 con 0x7f89c8104f60 2024-03-22T01:36:05.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.913+0000 7f89be7fc640 1 -- 172.21.15.72:0/2689532924 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89b80054e0 con 0x7f89c8104f60 2024-03-22T01:36:05.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.913+0000 7f89be7fc640 1 -- 172.21.15.72:0/2689532924 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b8003ce0 con 0x7f89c8104f60 2024-03-22T01:36:05.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.913+0000 7f89cf53e640 1 -- 172.21.15.72:0/2689532924 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89c81a25a0 con 0x7f89c8104f60 2024-03-22T01:36:05.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.916+0000 7f89d2202740 1 Processor -- start 2024-03-22T01:36:05.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.916+0000 7f89be7fc640 1 -- 172.21.15.72:0/2689532924 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7f89b8046020 con 0x7f89c8104f60 2024-03-22T01:36:05.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.916+0000 7f89d2202740 1 -- start start 2024-03-22T01:36:05.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.916+0000 7f89d2202740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b42ea52c80 0x55b42ea530a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:05.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.916+0000 7f89d2202740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b42ea53670 0x55b42ea58170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:05.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.916+0000 7f89d2202740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b42ea587e0 0x55b42ea5ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:05.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.916+0000 7f89d2202740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55b42e9b7210 con 0x55b42ea53670 2024-03-22T01:36:05.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.916+0000 7f89d2202740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55b42e95fc80 con 0x55b42ea52c80 2024-03-22T01:36:05.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.916+0000 7f89d2202740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b42e9b31c0 con 0x55b42ea587e0 2024-03-22T01:36:05.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.916+0000 7f89cd2da640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b42ea53670 0x55b42ea58170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:05.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.916+0000 7f89cd2da640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b42ea53670 0x55b42ea58170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:35736/0 (socket says 172.21.15.72:35736) 2024-03-22T01:36:05.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.916+0000 7f89cd2da640 1 -- 172.21.15.72:0/3462685588 learned_addr learned my addr 172.21.15.72:0/3462685588 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:05.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.916+0000 7f89cdadb640 1 --2- 172.21.15.72:0/3462685588 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b42ea52c80 0x55b42ea530a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:05.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.916+0000 7f89ccad9640 1 --2- 172.21.15.72:0/3462685588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b42ea587e0 0x55b42ea5ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:05.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.916+0000 7f89be7fc640 1 --2- 172.21.15.72:0/2689532924 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f89b40a2f20 0x7f89b40a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:05.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.916+0000 7f89cd2da640 1 -- 172.21.15.72:0/3462685588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b42ea587e0 msgr2=0x55b42ea5ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:36:05.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.916+0000 7f89cd2da640 1 --2- 172.21.15.72:0/3462685588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b42ea587e0 0x55b42ea5ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:05.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.916+0000 7f89cd2da640 1 -- 172.21.15.72:0/3462685588 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b42ea52c80 msgr2=0x55b42ea530a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:36:05.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.916+0000 7f89cd2da640 1 --2- 172.21.15.72:0/3462685588 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b42ea52c80 0x55b42ea530a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:05.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.916+0000 7f89cd2da640 1 -- 172.21.15.72:0/3462685588 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b42e6cb2d0 con 0x55b42ea53670 2024-03-22T01:36:05.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.916+0000 7f89cdadb640 1 --2- 172.21.15.72:0/2689532924 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f89b40a2f20 0x7f89b40a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:05.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.916+0000 7f89be7fc640 1 -- 172.21.15.72:0/2689532924 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f89b80f1850 con 0x7f89c8104f60 2024-03-22T01:36:05.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.916+0000 7f89cd2da640 1 --2- 172.21.15.72:0/3462685588 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b42ea53670 0x55b42ea58170 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f89c40104d0 tx=0x7f89c4010990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21c8c391001c60de server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:05.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.917+0000 7f89b37fe640 1 -- 172.21.15.72:0/3462685588 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c4011760 con 0x55b42ea53670 2024-03-22T01:36:05.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.917+0000 7f89b37fe640 1 -- 172.21.15.72:0/3462685588 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89c4011de0 con 0x55b42ea53670 2024-03-22T01:36:05.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.917+0000 7f89b37fe640 1 -- 172.21.15.72:0/3462685588 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c400e430 con 0x55b42ea53670 2024-03-22T01:36:05.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.917+0000 7f89d2202740 1 -- 172.21.15.72:0/3462685588 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b42ea53670 msgr2=0x55b42ea58170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:05.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.917+0000 7f89cdadb640 1 --2- 172.21.15.72:0/2689532924 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f89b40a2f20 0x7f89b40a53e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f89c0009fd0 tx=0x7f89c0004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:05.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.917+0000 7f89d2202740 1 --2- 172.21.15.72:0/3462685588 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b42ea53670 0x55b42ea58170 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f89c40104d0 tx=0x7f89c4010990 comp rx=0 tx=0).stop 2024-03-22T01:36:05.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.917+0000 7f89d2202740 1 -- 172.21.15.72:0/3462685588 shutdown_connections 2024-03-22T01:36:05.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.917+0000 7f89d2202740 1 --2- 172.21.15.72:0/3462685588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b42ea587e0 0x55b42ea5ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:05.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.917+0000 7f89d2202740 1 --2- 172.21.15.72:0/3462685588 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b42ea53670 0x55b42ea58170 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:05.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.917+0000 7f89d2202740 1 --2- 172.21.15.72:0/3462685588 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b42ea52c80 0x55b42ea530a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:05.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.917+0000 7f89d2202740 1 -- 172.21.15.72:0/3462685588 >> 172.21.15.72:0/3462685588 conn(0x55b42ea4e1f0 msgr2=0x55b42ea4e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:05.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.917+0000 7f89d2202740 1 -- 172.21.15.72:0/3462685588 shutdown_connections 2024-03-22T01:36:05.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.917+0000 7f89d2202740 1 -- 172.21.15.72:0/3462685588 wait complete. 2024-03-22T01:36:05.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.918+0000 7f89d2202740 1 -- start start 2024-03-22T01:36:05.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.918+0000 7f89d2202740 1 Processor -- start 2024-03-22T01:36:05.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.918+0000 7f89d2202740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b42ea52c80 0x55b42ea5cec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:05.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.918+0000 7f89d2202740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b42ea53670 0x55b42ea5d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:05.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.918+0000 7f89d2202740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b42ea587e0 0x55b42ea64440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:05.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.918+0000 7f89d2202740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55b42ea5b790 con 0x55b42ea587e0 2024-03-22T01:36:05.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.918+0000 7f89d2202740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55b42e972f80 con 0x55b42ea53670 2024-03-22T01:36:05.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.918+0000 7f89d2202740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b42ea5b910 con 0x55b42ea52c80 2024-03-22T01:36:05.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.918+0000 7f89d2202740 10 client.0 _ll_register_callbacks cb 0x55b42e9af4f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:36:05.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.918+0000 7f89cdadb640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b42ea587e0 0x55b42ea64440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:05.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.918+0000 7f89ccad9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b42ea53670 0x55b42ea5d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:05.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.918+0000 7f89cdadb640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b42ea587e0 0x55b42ea64440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:35742/0 (socket says 172.21.15.72:35742) 2024-03-22T01:36:05.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.918+0000 7f89cdadb640 1 -- 172.21.15.72:0/100412575 learned_addr learned my addr 172.21.15.72:0/100412575 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:05.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.918+0000 7f89cd2da640 1 --2- 172.21.15.72:0/100412575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b42ea52c80 0x55b42ea5cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:05.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.919+0000 7f89cdadb640 1 -- 172.21.15.72:0/100412575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b42ea52c80 msgr2=0x55b42ea5cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:36:05.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.919+0000 7f89cdadb640 1 --2- 172.21.15.72:0/100412575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b42ea52c80 0x55b42ea5cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:05.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.919+0000 7f89cdadb640 1 -- 172.21.15.72:0/100412575 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b42ea53670 msgr2=0x55b42ea5d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:36:05.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.919+0000 7f89cdadb640 1 --2- 172.21.15.72:0/100412575 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b42ea53670 0x55b42ea5d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:05.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.919+0000 7f89cdadb640 1 -- 172.21.15.72:0/100412575 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b42e7956b0 con 0x55b42ea587e0 2024-03-22T01:36:05.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.919+0000 7f89ccad9640 1 --2- 172.21.15.72:0/100412575 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b42ea53670 0x55b42ea5d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:36:05.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.919+0000 7f89cdadb640 1 --2- 172.21.15.72:0/100412575 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b42ea587e0 0x55b42ea64440 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f89c000fb90 tx=0x7f89c0004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7a1c0653a390bed server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:05.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.919+0000 7f89b0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:36:05.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.919+0000 7f89b0ff9640 1 -- 172.21.15.72:0/100412575 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c001ea00 con 0x55b42ea587e0 2024-03-22T01:36:05.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.919+0000 7f89b0ff9640 1 -- 172.21.15.72:0/100412575 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f89c001eba0 con 0x55b42ea587e0 2024-03-22T01:36:05.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.919+0000 7f89d2202740 1 -- 172.21.15.72:0/100412575 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89c4011de0 con 0x55b42ea587e0 2024-03-22T01:36:05.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.919+0000 7f89b0ff9640 1 -- 172.21.15.72:0/100412575 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f89c001eda0 con 0x55b42ea587e0 2024-03-22T01:36:05.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.919+0000 7f89d2202740 10 client.15213 fetch_fsmap learned FSMap version 21 2024-03-22T01:36:05.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.919+0000 7f89d2202740 1 -- 172.21.15.72:0/100412575 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55b42ea67820 con 0x55b42ea587e0 2024-03-22T01:36:05.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.919+0000 7f89b0ff9640 1 -- 172.21.15.72:0/100412575 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89c4011de0 con 0x55b42ea587e0 2024-03-22T01:36:05.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.919+0000 7f89b0ff9640 1 -- 172.21.15.72:0/100412575 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f89c0028440 con 0x55b42ea587e0 2024-03-22T01:36:05.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.919+0000 7f89d2202740 10 client.15213 fetch_fsmap finished waiting for FSMap version 21 2024-03-22T01:36:05.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.919+0000 7f89d2202740 10 client.15213 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-22T01:36:05.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.919+0000 7f89d2202740 20 client.15213 populate_metadata read hostname 'smithi072' 2024-03-22T01:36:05.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.919+0000 7f89d2202740 1 --2- 172.21.15.72:0/100412575 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55b42ea69f10 0x55b42ea6b380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:05.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.919+0000 7f89d2202740 4 client.15213 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T01:36:05.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.919+0000 7f89d2202740 1 -- 172.21.15.72:0/100412575 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b42e9c4750 con 0x55b42ea69f10 2024-03-22T01:36:05.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.920+0000 7f89cd2da640 1 --2- 172.21.15.72:0/100412575 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55b42ea69f10 0x55b42ea6b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:05.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.920+0000 7f89cd2da640 1 --2- 172.21.15.72:0/100412575 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55b42ea69f10 0x55b42ea6b380 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a89b840e518a102f server_cookie=f965ea5f3e687010 in_seq=0 out_seq=0 2024-03-22T01:36:05.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.920+0000 7f89b0ff9640 10 client.15213 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:36:05.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.921+0000 7f89b0ff9640 1 -- 172.21.15.72:0/100412575 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b42e9c4750 con 0x55b42ea69f10 2024-03-22T01:36:05.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.921+0000 7f89b0ff9640 10 client.15213 handle_command_reply: tid=0 2024-03-22T01:36:05.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.921+0000 7f89d2202740 1 client.15213 shutdown 2024-03-22T01:36:05.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.921+0000 7f89d2202740 1 -- 172.21.15.72:0/100412575 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b42ea587e0 msgr2=0x55b42ea64440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:05.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.921+0000 7f89d2202740 1 --2- 172.21.15.72:0/100412575 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b42ea587e0 0x55b42ea64440 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f89c000fb90 tx=0x7f89c0004ab0 comp rx=0 tx=0).stop 2024-03-22T01:36:05.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.921+0000 7f89d2202740 1 -- 172.21.15.72:0/100412575 shutdown_connections 2024-03-22T01:36:05.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.921+0000 7f89d2202740 1 --2- 172.21.15.72:0/100412575 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55b42ea69f10 0x55b42ea6b380 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:05.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.921+0000 7f89d2202740 1 --2- 172.21.15.72:0/100412575 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b42ea53670 0x55b42ea5d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:05.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.921+0000 7f89d2202740 1 --2- 172.21.15.72:0/100412575 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b42ea587e0 0x55b42ea64440 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:05.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.921+0000 7f89b0ff9640 0 client.15213 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:36:05.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.921+0000 7f89d2202740 1 --2- 172.21.15.72:0/100412575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b42ea52c80 0x55b42ea5cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:05.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.921+0000 7f89d2202740 1 -- 172.21.15.72:0/100412575 >> 172.21.15.72:0/100412575 conn(0x55b42ea4e1f0 msgr2=0x55b42ea4d4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:05.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.921+0000 7f89d2202740 1 -- 172.21.15.72:0/100412575 shutdown_connections 2024-03-22T01:36:05.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.921+0000 7f89d2202740 1 -- 172.21.15.72:0/100412575 wait complete. 2024-03-22T01:36:05.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.921+0000 7f89d2202740 20 client.15213 trim_cache size 0 max 16384 2024-03-22T01:36:05.928 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:05 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/3727358543' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T01:36:05.928 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:05 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/4220212356' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:36:05.928 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:05 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/3247125269' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:36:05.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.939+0000 7f89d2202740 1 Processor -- start 2024-03-22T01:36:05.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.939+0000 7f89d2202740 1 -- start start 2024-03-22T01:36:05.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.939+0000 7f89d2202740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b42ea52c80 0x55b42ea51b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:05.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.939+0000 7f89d2202740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b42ea53670 0x55b42ea64400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:05.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.939+0000 7f89d2202740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b42ea587e0 0x55b42ea54110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:05.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.939+0000 7f89d2202740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55b42ea696c0 con 0x55b42ea53670 2024-03-22T01:36:05.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.939+0000 7f89d2202740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55b42ea69d90 con 0x55b42ea52c80 2024-03-22T01:36:05.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.939+0000 7f89d2202740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b42ea6a460 con 0x55b42ea587e0 2024-03-22T01:36:05.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.939+0000 7f89ccad9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b42ea53670 0x55b42ea64400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:05.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.939+0000 7f89ccad9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b42ea53670 0x55b42ea64400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:35752/0 (socket says 172.21.15.72:35752) 2024-03-22T01:36:05.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.939+0000 7f89ccad9640 1 -- 172.21.15.72:0/254815640 learned_addr learned my addr 172.21.15.72:0/254815640 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:05.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.939+0000 7f89cd2da640 1 --2- 172.21.15.72:0/254815640 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b42ea52c80 0x55b42ea51b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:05.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.939+0000 7f89ccad9640 1 -- 172.21.15.72:0/254815640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b42ea587e0 msgr2=0x55b42ea54110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:36:05.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.939+0000 7f89ccad9640 1 --2- 172.21.15.72:0/254815640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b42ea587e0 0x55b42ea54110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:05.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.939+0000 7f89ccad9640 1 -- 172.21.15.72:0/254815640 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b42ea52c80 msgr2=0x55b42ea51b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:36:05.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.939+0000 7f89ccad9640 1 --2- 172.21.15.72:0/254815640 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b42ea52c80 0x55b42ea51b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:05.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.939+0000 7f89ccad9640 1 -- 172.21.15.72:0/254815640 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b42ea5b030 con 0x55b42ea53670 2024-03-22T01:36:05.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.939+0000 7f89cd2da640 1 --2- 172.21.15.72:0/254815640 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b42ea52c80 0x55b42ea51b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:36:05.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.939+0000 7f89ccad9640 1 --2- 172.21.15.72:0/254815640 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b42ea53670 0x55b42ea64400 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f89b8006010 tx=0x7f89b8004dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d3557d9bd615939 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:05.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.939+0000 7f89b2ffd640 1 -- 172.21.15.72:0/254815640 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b80bd580 con 0x55b42ea53670 2024-03-22T01:36:05.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.940+0000 7f89b2ffd640 1 -- 172.21.15.72:0/254815640 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89b80044a0 con 0x55b42ea53670 2024-03-22T01:36:05.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.940+0000 7f89b2ffd640 1 -- 172.21.15.72:0/254815640 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b80bbcc0 con 0x55b42ea53670 2024-03-22T01:36:05.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.940+0000 7f89d2202740 1 -- 172.21.15.72:0/254815640 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b42ea53670 msgr2=0x55b42ea64400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:05.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.940+0000 7f89d2202740 1 --2- 172.21.15.72:0/254815640 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b42ea53670 0x55b42ea64400 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f89b8006010 tx=0x7f89b8004dc0 comp rx=0 tx=0).stop 2024-03-22T01:36:05.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.940+0000 7f89d2202740 1 -- 172.21.15.72:0/254815640 shutdown_connections 2024-03-22T01:36:05.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.940+0000 7f89d2202740 1 --2- 172.21.15.72:0/254815640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b42ea587e0 0x55b42ea54110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:05.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.940+0000 7f89d2202740 1 --2- 172.21.15.72:0/254815640 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b42ea53670 0x55b42ea64400 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:05.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.940+0000 7f89d2202740 1 --2- 172.21.15.72:0/254815640 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b42ea52c80 0x55b42ea51b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:05.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.940+0000 7f89d2202740 1 -- 172.21.15.72:0/254815640 >> 172.21.15.72:0/254815640 conn(0x55b42ea4e1f0 msgr2=0x55b42ea51450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:05.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.940+0000 7f89d2202740 1 -- 172.21.15.72:0/254815640 shutdown_connections 2024-03-22T01:36:05.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.940+0000 7f89d2202740 1 -- 172.21.15.72:0/254815640 wait complete. 2024-03-22T01:36:05.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.940+0000 7f89d2202740 1 -- start start 2024-03-22T01:36:05.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.940+0000 7f89d2202740 1 Processor -- start 2024-03-22T01:36:05.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.941+0000 7f89d2202740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b42ea52c80 0x55b42ea5f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:05.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.941+0000 7f89d2202740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b42ea53670 0x55b42ea5ff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:05.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.941+0000 7f89d2202740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b42ea587e0 0x55b42ea62f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:05.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.941+0000 7f89d2202740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55b42ea6b200 con 0x55b42ea52c80 2024-03-22T01:36:05.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.941+0000 7f89d2202740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55b42ea6ab30 con 0x55b42ea53670 2024-03-22T01:36:05.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.941+0000 7f89d2202740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b42ea6b8d0 con 0x55b42ea587e0 2024-03-22T01:36:05.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.941+0000 7f89d2202740 10 client.0 _ll_register_callbacks cb 0x7f89c0009e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:36:05.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.941+0000 7f89cd2da640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b42ea52c80 0x55b42ea5f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:05.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.941+0000 7f89cd2da640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b42ea52c80 0x55b42ea5f9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:35762/0 (socket says 172.21.15.72:35762) 2024-03-22T01:36:05.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.941+0000 7f89cd2da640 1 -- 172.21.15.72:0/3535272965 learned_addr learned my addr 172.21.15.72:0/3535272965 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:05.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.941+0000 7f89ccad9640 1 --2- 172.21.15.72:0/3535272965 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b42ea53670 0x55b42ea5ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:05.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.941+0000 7f89cdadb640 1 --2- 172.21.15.72:0/3535272965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b42ea587e0 0x55b42ea62f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:05.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.941+0000 7f89cd2da640 1 -- 172.21.15.72:0/3535272965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b42ea587e0 msgr2=0x55b42ea62f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:36:05.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.941+0000 7f89cd2da640 1 --2- 172.21.15.72:0/3535272965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b42ea587e0 0x55b42ea62f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:05.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.941+0000 7f89cd2da640 1 -- 172.21.15.72:0/3535272965 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b42ea53670 msgr2=0x55b42ea5ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:36:05.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.941+0000 7f89cd2da640 1 --2- 172.21.15.72:0/3535272965 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b42ea53670 0x55b42ea5ff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:05.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.941+0000 7f89cd2da640 1 -- 172.21.15.72:0/3535272965 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b42ea63510 con 0x55b42ea52c80 2024-03-22T01:36:05.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.941+0000 7f89cdadb640 1 --2- 172.21.15.72:0/3535272965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b42ea587e0 0x55b42ea62f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:36:05.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.941+0000 7f89ccad9640 1 --2- 172.21.15.72:0/3535272965 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b42ea53670 0x55b42ea5ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:36:05.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.941+0000 7f89cd2da640 1 --2- 172.21.15.72:0/3535272965 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b42ea52c80 0x55b42ea5f9c0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f89c400be50 tx=0x7f89c400b5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6481dbf9811bf1f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:05.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.941+0000 7f89b0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:36:05.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.941+0000 7f89b0ff9640 1 -- 172.21.15.72:0/3535272965 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c4005510 con 0x55b42ea52c80 2024-03-22T01:36:05.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.942+0000 7f89b0ff9640 1 -- 172.21.15.72:0/3535272965 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f89c401c480 con 0x55b42ea52c80 2024-03-22T01:36:05.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.942+0000 7f89d2202740 1 -- 172.21.15.72:0/3535272965 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b42e9beaa0 con 0x55b42ea52c80 2024-03-22T01:36:05.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.942+0000 7f89b0ff9640 1 -- 172.21.15.72:0/3535272965 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f89c403a050 con 0x55b42ea52c80 2024-03-22T01:36:05.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.942+0000 7f89d2202740 10 client.15219 fetch_fsmap learned FSMap version 21 2024-03-22T01:36:05.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.942+0000 7f89d2202740 1 -- 172.21.15.72:0/3535272965 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55b42eacfc80 con 0x55b42ea52c80 2024-03-22T01:36:05.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.942+0000 7f89b0ff9640 1 -- 172.21.15.72:0/3535272965 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55b42e9beaa0 con 0x55b42ea52c80 2024-03-22T01:36:05.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.942+0000 7f89b0ff9640 1 -- 172.21.15.72:0/3535272965 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f89c40035f0 con 0x55b42ea52c80 2024-03-22T01:36:05.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.942+0000 7f89d2202740 10 client.15219 fetch_fsmap finished waiting for FSMap version 21 2024-03-22T01:36:05.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.942+0000 7f89d2202740 10 client.15219 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-22T01:36:05.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.942+0000 7f89d2202740 20 client.15219 populate_metadata read hostname 'smithi072' 2024-03-22T01:36:05.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.942+0000 7f89d2202740 1 --2- 172.21.15.72:0/3535272965 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55b42ead4560 0x55b42ead3270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:05.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.942+0000 7f89d2202740 4 client.15219 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-22T01:36:05.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.942+0000 7f89d2202740 1 -- 172.21.15.72:0/3535272965 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "status"}) -- 0x55b42e93ca00 con 0x55b42ead4560 2024-03-22T01:36:05.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.942+0000 7f89ccad9640 1 --2- 172.21.15.72:0/3535272965 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55b42ead4560 0x55b42ead3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:05.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.943+0000 7f89ccad9640 1 --2- 172.21.15.72:0/3535272965 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55b42ead4560 0x55b42ead3270 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f20ea4e1769ed81c server_cookie=b84dbe6d0eed198 in_seq=0 out_seq=0 2024-03-22T01:36:05.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.943+0000 7f89b0ff9640 10 client.15219 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:36:05.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.943+0000 7f89b0ff9640 1 -- 172.21.15.72:0/3535272965 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55b42e93ca00 con 0x55b42ead4560 2024-03-22T01:36:05.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.943+0000 7f89b0ff9640 10 client.15219 handle_command_reply: tid=0 2024-03-22T01:36:05.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.943+0000 7f89d2202740 1 client.15219 shutdown 2024-03-22T01:36:05.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.943+0000 7f89d2202740 1 -- 172.21.15.72:0/3535272965 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b42ea52c80 msgr2=0x55b42ea5f9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:05.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.943+0000 7f89d2202740 1 --2- 172.21.15.72:0/3535272965 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b42ea52c80 0x55b42ea5f9c0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f89c400be50 tx=0x7f89c400b5d0 comp rx=0 tx=0).stop 2024-03-22T01:36:05.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.943+0000 7f89d2202740 1 -- 172.21.15.72:0/3535272965 shutdown_connections 2024-03-22T01:36:05.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.943+0000 7f89d2202740 1 --2- 172.21.15.72:0/3535272965 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55b42ead4560 0x55b42ead3270 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:05.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.943+0000 7f89d2202740 1 --2- 172.21.15.72:0/3535272965 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b42ea53670 0x55b42ea5ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:05.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.943+0000 7f89d2202740 1 --2- 172.21.15.72:0/3535272965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b42ea587e0 0x55b42ea62f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:05.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.943+0000 7f89b0ff9640 0 client.15219 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:36:05.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.943+0000 7f89d2202740 1 --2- 172.21.15.72:0/3535272965 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b42ea52c80 0x55b42ea5f9c0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:05.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.943+0000 7f89d2202740 1 -- 172.21.15.72:0/3535272965 >> 172.21.15.72:0/3535272965 conn(0x55b42ea4e1f0 msgr2=0x55b42ea51450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:05.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.943+0000 7f89d2202740 1 -- 172.21.15.72:0/3535272965 shutdown_connections 2024-03-22T01:36:05.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.943+0000 7f89d2202740 1 -- 172.21.15.72:0/3535272965 wait complete. 2024-03-22T01:36:05.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.943+0000 7f89d2202740 20 client.15219 trim_cache size 0 max 16384 2024-03-22T01:36:05.953 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-22T01:36:05.953 INFO:teuthology.orchestra.run.smithi072.stdout: "cluster_fsid": "022a99bc-e7eb-11ee-95cd-87774f69a715", 2024-03-22T01:36:05.953 INFO:teuthology.orchestra.run.smithi072.stdout: "whoami": 0, 2024-03-22T01:36:05.953 INFO:teuthology.orchestra.run.smithi072.stdout: "id": 24524, 2024-03-22T01:36:05.953 INFO:teuthology.orchestra.run.smithi072.stdout: "want_state": "up:active", 2024-03-22T01:36:05.953 INFO:teuthology.orchestra.run.smithi072.stdout: "state": "up:active", 2024-03-22T01:36:05.953 INFO:teuthology.orchestra.run.smithi072.stdout: "fs_name": "cephfs", 2024-03-22T01:36:05.953 INFO:teuthology.orchestra.run.smithi072.stdout: "rank_uptime": 110.46299115799999, 2024-03-22T01:36:05.953 INFO:teuthology.orchestra.run.smithi072.stdout: "mdsmap_epoch": 21, 2024-03-22T01:36:05.953 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_epoch": 93, 2024-03-22T01:36:05.953 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_epoch_barrier": 91, 2024-03-22T01:36:05.954 INFO:teuthology.orchestra.run.smithi072.stdout: "uptime": 117.04467210200001 2024-03-22T01:36:05.954 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-22T01:36:05.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.943+0000 7f89cf53e640 1 -- 172.21.15.72:0/2689532924 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f89b40a2f20 msgr2=0x7f89b40a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:05.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.943+0000 7f89cf53e640 1 --2- 172.21.15.72:0/2689532924 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f89b40a2f20 0x7f89b40a53e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f89c0009fd0 tx=0x7f89c0004a40 comp rx=0 tx=0).stop 2024-03-22T01:36:05.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.944+0000 7f89cf53e640 1 -- 172.21.15.72:0/2689532924 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f89c8104f60 msgr2=0x7f89c81005e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:05.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.944+0000 7f89cf53e640 1 --2- 172.21.15.72:0/2689532924 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f89c8104f60 0x7f89c81005e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f89b8000c00 tx=0x7f89b802c6c0 comp rx=0 tx=0).stop 2024-03-22T01:36:05.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.944+0000 7f89cf53e640 1 -- 172.21.15.72:0/2689532924 shutdown_connections 2024-03-22T01:36:05.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.944+0000 7f89cf53e640 1 --2- 172.21.15.72:0/2689532924 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f89b40a2f20 0x7f89b40a53e0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:05.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.944+0000 7f89cf53e640 1 --2- 172.21.15.72:0/2689532924 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89c8107890 0x7f89c8100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:05.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.944+0000 7f89cf53e640 1 --2- 172.21.15.72:0/2689532924 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f89c8104f60 0x7f89c81005e0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:05.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.944+0000 7f89cf53e640 1 --2- 172.21.15.72:0/2689532924 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89c8102630 0x7f89c8101fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:05.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.944+0000 7f89cf53e640 1 -- 172.21.15.72:0/2689532924 >> 172.21.15.72:0/2689532924 conn(0x7f89c80fc020 msgr2=0x7f89c8103050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:05.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.944+0000 7f89cf53e640 1 -- 172.21.15.72:0/2689532924 shutdown_connections 2024-03-22T01:36:05.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:05.944+0000 7f89cf53e640 1 -- 172.21.15.72:0/2689532924 wait complete. 2024-03-22T01:36:05.956 INFO:tasks.cephadm:Setup complete, yielding 2024-03-22T01:36:05.957 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-22T01:36:05.967 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi072.front.sepia.ceph.com 2024-03-22T01:36:05.967 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- bash -c 'ceph orch status' 2024-03-22T01:36:06.204 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:36:06.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:06 smithi089 ceph-mon[35770]: pgmap v348: 129 pgs: 129 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-22T01:36:06.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:06 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/947018352' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:36:06.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:06 smithi072 ceph-mon[32414]: pgmap v348: 129 pgs: 129 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-22T01:36:06.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:06 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/947018352' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:36:06.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:06 smithi098 ceph-mon[36468]: pgmap v348: 129 pgs: 129 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-22T01:36:06.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:06 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/947018352' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:36:07.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:07.973+0000 7f8614a89700 1 -- 172.21.15.72:0/90450704 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f86101026b0 msgr2=0x7f8610102ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:07.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:07.973+0000 7f8614a89700 1 --2- 172.21.15.72:0/90450704 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f86101026b0 0x7f8610102ad0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f86000099c0 tx=0x7f8600009cd0 comp rx=0 tx=0).stop 2024-03-22T01:36:07.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:07.973+0000 7f8614a89700 1 -- 172.21.15.72:0/90450704 shutdown_connections 2024-03-22T01:36:07.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:07.973+0000 7f8614a89700 1 --2- 172.21.15.72:0/90450704 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8610104250 0x7f861010ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:07.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:07.973+0000 7f8614a89700 1 --2- 172.21.15.72:0/90450704 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f86101026b0 0x7f8610102ad0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:07.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:07.973+0000 7f8614a89700 1 --2- 172.21.15.72:0/90450704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86101038b0 0x7f8610103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:07.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:07.973+0000 7f8614a89700 1 -- 172.21.15.72:0/90450704 >> 172.21.15.72:0/90450704 conn(0x7f86100fdc90 msgr2=0x7f86101000b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:07.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:07.973+0000 7f8614a89700 1 -- 172.21.15.72:0/90450704 shutdown_connections 2024-03-22T01:36:07.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:07.973+0000 7f8614a89700 1 -- 172.21.15.72:0/90450704 wait complete. 2024-03-22T01:36:07.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:07.973+0000 7f8614a89700 1 Processor -- start 2024-03-22T01:36:07.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:07.974+0000 7f8614a89700 1 -- start start 2024-03-22T01:36:07.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:07.974+0000 7f8614a89700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86101026b0 0x7f8610198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:07.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:07.974+0000 7f8614a89700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f86101038b0 0x7f8610198ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:07.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:07.974+0000 7f8614a89700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8610104250 0x7f861019ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:07.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:07.974+0000 7f8614a89700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f861010e2f0 con 0x7f86101038b0 2024-03-22T01:36:07.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:07.974+0000 7f8614a89700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f861010e170 con 0x7f8610104250 2024-03-22T01:36:07.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:07.974+0000 7f8614a89700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f861010e470 con 0x7f86101026b0 2024-03-22T01:36:07.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:07.974+0000 7f8605d9b700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f86101038b0 0x7f8610198ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:07.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:07.974+0000 7f8605d9b700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f86101038b0 0x7f8610198ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:35788/0 (socket says 172.21.15.72:35788) 2024-03-22T01:36:07.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:07.974+0000 7f8605d9b700 1 -- 172.21.15.72:0/3255195216 learned_addr learned my addr 172.21.15.72:0/3255195216 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:07.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:07.974+0000 7f860ed9d700 1 --2- 172.21.15.72:0/3255195216 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8610104250 0x7f861019ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:07.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:07.974+0000 7f860e59c700 1 --2- 172.21.15.72:0/3255195216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86101026b0 0x7f8610198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:07.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:07.974+0000 7f860ed9d700 1 -- 172.21.15.72:0/3255195216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86101026b0 msgr2=0x7f8610198960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:07.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:07.974+0000 7f860ed9d700 1 --2- 172.21.15.72:0/3255195216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86101026b0 0x7f8610198960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:07.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:07.974+0000 7f860ed9d700 1 -- 172.21.15.72:0/3255195216 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f86101038b0 msgr2=0x7f8610198ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:07.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:07.974+0000 7f860ed9d700 1 --2- 172.21.15.72:0/3255195216 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f86101038b0 0x7f8610198ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:07.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:07.974+0000 7f860ed9d700 1 -- 172.21.15.72:0/3255195216 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86101a05c0 con 0x7f8610104250 2024-03-22T01:36:07.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:07.974+0000 7f860ed9d700 1 --2- 172.21.15.72:0/3255195216 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8610104250 0x7f861019ff20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f860800efc0 tx=0x7f860800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:07.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:07.975+0000 7f8607fff700 1 -- 172.21.15.72:0/3255195216 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86080092e0 con 0x7f8610104250 2024-03-22T01:36:07.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:07.975+0000 7f8607fff700 1 -- 172.21.15.72:0/3255195216 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8608004bb0 con 0x7f8610104250 2024-03-22T01:36:07.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:07.975+0000 7f8607fff700 1 -- 172.21.15.72:0/3255195216 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8608004280 con 0x7f8610104250 2024-03-22T01:36:07.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:07.975+0000 7f8614a89700 1 -- 172.21.15.72:0/3255195216 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86101a0850 con 0x7f8610104250 2024-03-22T01:36:07.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:07.975+0000 7f8614a89700 1 -- 172.21.15.72:0/3255195216 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86101a0dc0 con 0x7f8610104250 2024-03-22T01:36:07.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:07.976+0000 7f8614a89700 1 -- 172.21.15.72:0/3255195216 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8610069340 con 0x7f8610104250 2024-03-22T01:36:07.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:07.977+0000 7f8607fff700 1 -- 172.21.15.72:0/3255195216 <== mon.2 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7f860801b030 con 0x7f8610104250 2024-03-22T01:36:07.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:07.977+0000 7f8607fff700 1 --2- 172.21.15.72:0/3255195216 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f85fc0a2fb0 0x7f85fc0a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:07.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:07.978+0000 7f8607fff700 1 -- 172.21.15.72:0/3255195216 <== mon.2 v2:172.21.15.89:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f86080cf350 con 0x7f8610104250 2024-03-22T01:36:07.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:07.978+0000 7f860e59c700 1 --2- 172.21.15.72:0/3255195216 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f85fc0a2fb0 0x7f85fc0a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:07.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:07.978+0000 7f860e59c700 1 --2- 172.21.15.72:0/3255195216 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f85fc0a2fb0 0x7f85fc0a5470 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f8600003790 tx=0x7f8600005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:07.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:07.982+0000 7f8607fff700 1 -- 172.21.15.72:0/3255195216 <== mon.2 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8608022020 con 0x7f8610104250 2024-03-22T01:36:08.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:08.193+0000 7f8614a89700 1 -- 172.21.15.72:0/3255195216 --> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f8610062ac0 con 0x7f85fc0a2fb0 2024-03-22T01:36:08.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:08.194+0000 7f8607fff700 1 -- 172.21.15.72:0/3255195216 <== mgr.14152 v2:172.21.15.72:6800/401403200 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f8610062ac0 con 0x7f85fc0a2fb0 2024-03-22T01:36:08.193 INFO:teuthology.orchestra.run.smithi072.stdout:Backend: cephadm 2024-03-22T01:36:08.193 INFO:teuthology.orchestra.run.smithi072.stdout:Available: Yes 2024-03-22T01:36:08.193 INFO:teuthology.orchestra.run.smithi072.stdout:Paused: No 2024-03-22T01:36:08.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:08.196+0000 7f8614a89700 1 -- 172.21.15.72:0/3255195216 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f85fc0a2fb0 msgr2=0x7f85fc0a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:08.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:08.196+0000 7f8614a89700 1 --2- 172.21.15.72:0/3255195216 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f85fc0a2fb0 0x7f85fc0a5470 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f8600003790 tx=0x7f8600005fb0 comp rx=0 tx=0).stop 2024-03-22T01:36:08.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:08.196+0000 7f8614a89700 1 -- 172.21.15.72:0/3255195216 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8610104250 msgr2=0x7f861019ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:08.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:08.196+0000 7f8614a89700 1 --2- 172.21.15.72:0/3255195216 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8610104250 0x7f861019ff20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f860800efc0 tx=0x7f860800c3a0 comp rx=0 tx=0).stop 2024-03-22T01:36:08.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:08.196+0000 7f8614a89700 1 -- 172.21.15.72:0/3255195216 shutdown_connections 2024-03-22T01:36:08.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:08.196+0000 7f8614a89700 1 --2- 172.21.15.72:0/3255195216 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f85fc0a2fb0 0x7f85fc0a5470 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:08.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:08.196+0000 7f8614a89700 1 --2- 172.21.15.72:0/3255195216 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8610104250 0x7f861019ff20 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:08.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:08.196+0000 7f8614a89700 1 --2- 172.21.15.72:0/3255195216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86101026b0 0x7f8610198960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:08.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:08.196+0000 7f8614a89700 1 --2- 172.21.15.72:0/3255195216 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f86101038b0 0x7f8610198ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:08.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:08.196+0000 7f8614a89700 1 -- 172.21.15.72:0/3255195216 >> 172.21.15.72:0/3255195216 conn(0x7f86100fdc90 msgr2=0x7f8610106bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:08.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:08.196+0000 7f8614a89700 1 -- 172.21.15.72:0/3255195216 shutdown_connections 2024-03-22T01:36:08.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:08.196+0000 7f8614a89700 1 -- 172.21.15.72:0/3255195216 wait complete. 2024-03-22T01:36:08.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:08 smithi089 ceph-mon[35770]: pgmap v349: 129 pgs: 129 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-22T01:36:08.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:08 smithi072 ceph-mon[32414]: pgmap v349: 129 pgs: 129 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-22T01:36:08.906 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-22T01:36:08.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:08 smithi098 ceph-mon[36468]: pgmap v349: 129 pgs: 129 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-22T01:36:09.149 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:36:09.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:09 smithi089 ceph-mon[35770]: from='client.25085 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:36:09.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:09 smithi072 ceph-mon[32414]: from='client.25085 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:36:09.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:09 smithi098 ceph-mon[36468]: from='client.25085 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:36:10.699 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:10 smithi072 ceph-mon[32414]: pgmap v350: 129 pgs: 129 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-22T01:36:10.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:10 smithi089 ceph-mon[35770]: pgmap v350: 129 pgs: 129 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-22T01:36:10.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:10 smithi098 ceph-mon[36468]: pgmap v350: 129 pgs: 129 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-22T01:36:10.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:10.935+0000 7f183ab66700 1 -- 172.21.15.72:0/3669381960 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1834105a70 msgr2=0x7f1834107e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:10.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:10.935+0000 7f183ab66700 1 --2- 172.21.15.72:0/3669381960 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1834105a70 0x7f1834107e60 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f1820009990 tx=0x7f1820009ca0 comp rx=0 tx=0).stop 2024-03-22T01:36:10.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:10.935+0000 7f183ab66700 1 -- 172.21.15.72:0/3669381960 shutdown_connections 2024-03-22T01:36:10.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:10.935+0000 7f183ab66700 1 --2- 172.21.15.72:0/3669381960 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18341083a0 0x7f183410a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:10.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:10.935+0000 7f183ab66700 1 --2- 172.21.15.72:0/3669381960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1834103140 0x7f1834105530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:10.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:10.935+0000 7f183ab66700 1 --2- 172.21.15.72:0/3669381960 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1834105a70 0x7f1834107e60 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:10.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:10.935+0000 7f183ab66700 1 -- 172.21.15.72:0/3669381960 >> 172.21.15.72:0/3669381960 conn(0x7f18340fcb70 msgr2=0x7f18340fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:10.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:10.935+0000 7f183ab66700 1 -- 172.21.15.72:0/3669381960 shutdown_connections 2024-03-22T01:36:10.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:10.935+0000 7f183ab66700 1 -- 172.21.15.72:0/3669381960 wait complete. 2024-03-22T01:36:10.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:10.935+0000 7f183ab66700 1 Processor -- start 2024-03-22T01:36:10.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:10.935+0000 7f183ab66700 1 -- start start 2024-03-22T01:36:10.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:10.936+0000 7f183ab66700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1834103140 0x7f1834198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:10.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:10.936+0000 7f183ab66700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1834105a70 0x7f1834198db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:10.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:10.936+0000 7f183ab66700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18341083a0 0x7f183419fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:10.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:10.936+0000 7f183ab66700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f183410e280 con 0x7f1834103140 2024-03-22T01:36:10.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:10.936+0000 7f183ab66700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f183410e100 con 0x7f1834105a70 2024-03-22T01:36:10.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:10.936+0000 7f183ab66700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f183410e400 con 0x7f18341083a0 2024-03-22T01:36:10.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:10.936+0000 7f1838902700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1834103140 0x7f1834198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:10.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:10.936+0000 7f1838902700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1834103140 0x7f1834198870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:53218/0 (socket says 172.21.15.72:53218) 2024-03-22T01:36:10.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:10.936+0000 7f1833fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1834105a70 0x7f1834198db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:10.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:10.936+0000 7f1838902700 1 -- 172.21.15.72:0/1832244376 learned_addr learned my addr 172.21.15.72:0/1832244376 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:10.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:10.936+0000 7f1838902700 1 -- 172.21.15.72:0/1832244376 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18341083a0 msgr2=0x7f183419fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:10.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:10.936+0000 7f1838902700 1 --2- 172.21.15.72:0/1832244376 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18341083a0 0x7f183419fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:10.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:10.936+0000 7f1838902700 1 -- 172.21.15.72:0/1832244376 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1834105a70 msgr2=0x7f1834198db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:10.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:10.936+0000 7f1838902700 1 --2- 172.21.15.72:0/1832244376 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1834105a70 0x7f1834198db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:10.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:10.936+0000 7f1838902700 1 -- 172.21.15.72:0/1832244376 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18341a0480 con 0x7f1834103140 2024-03-22T01:36:10.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:10.936+0000 7f1838902700 1 --2- 172.21.15.72:0/1832244376 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1834103140 0x7f1834198870 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f182800c960 tx=0x7f182800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:10.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:10.936+0000 7f1831ffb700 1 -- 172.21.15.72:0/1832244376 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1828007ac0 con 0x7f1834103140 2024-03-22T01:36:10.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:10.936+0000 7f1831ffb700 1 -- 172.21.15.72:0/1832244376 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1828007c60 con 0x7f1834103140 2024-03-22T01:36:10.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:10.936+0000 7f183ab66700 1 -- 172.21.15.72:0/1832244376 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18341a0770 con 0x7f1834103140 2024-03-22T01:36:10.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:10.937+0000 7f1831ffb700 1 -- 172.21.15.72:0/1832244376 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1828018660 con 0x7f1834103140 2024-03-22T01:36:10.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:10.937+0000 7f183ab66700 1 -- 172.21.15.72:0/1832244376 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18341a0cb0 con 0x7f1834103140 2024-03-22T01:36:10.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:10.938+0000 7f183ab66700 1 -- 172.21.15.72:0/1832244376 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1834069340 con 0x7f1834103140 2024-03-22T01:36:10.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:10.939+0000 7f1831ffb700 1 -- 172.21.15.72:0/1832244376 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7f182800f450 con 0x7f1834103140 2024-03-22T01:36:10.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:10.939+0000 7f1831ffb700 1 --2- 172.21.15.72:0/1832244376 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f18240a2fb0 0x7f18240a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:10.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:10.940+0000 7f1833fff700 1 --2- 172.21.15.72:0/1832244376 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f18240a2fb0 0x7f18240a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:10.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:10.940+0000 7f1831ffb700 1 -- 172.21.15.72:0/1832244376 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f18280cf680 con 0x7f1834103140 2024-03-22T01:36:10.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:10.940+0000 7f1833fff700 1 --2- 172.21.15.72:0/1832244376 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f18240a2fb0 0x7f18240a5470 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f182000b2e0 tx=0x7f182000b880 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:10.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:10.944+0000 7f1831ffb700 1 -- 172.21.15.72:0/1832244376 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1828014030 con 0x7f1834103140 2024-03-22T01:36:11.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:11.150+0000 7f183ab66700 1 -- 172.21.15.72:0/1832244376 --> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f1834062ac0 con 0x7f18240a2fb0 2024-03-22T01:36:11.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:11.164+0000 7f1831ffb700 1 -- 172.21.15.72:0/1832244376 <== mgr.14152 v2:172.21.15.72:6800/401403200 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4092 (secure 0 0 0) 0x7f1834062ac0 con 0x7f18240a2fb0 2024-03-22T01:36:11.163 INFO:teuthology.orchestra.run.smithi072.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T01:36:11.163 INFO:teuthology.orchestra.run.smithi072.stdout:mds.a smithi072 running (2m) 20s ago 2m 13.0M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 88b4b3dad621 2024-03-22T01:36:11.163 INFO:teuthology.orchestra.run.smithi072.stdout:mds.b smithi089 running (2m) 56s ago 2m 28.4M - 19.0.0-2344-gc3a04c0e 9d92c22378ed a7ed85c8dcb6 2024-03-22T01:36:11.163 INFO:teuthology.orchestra.run.smithi072.stdout:mds.c smithi098 running (2m) 56s ago 2m 16.1M - 19.0.0-2344-gc3a04c0e 9d92c22378ed df75b211444a 2024-03-22T01:36:11.163 INFO:teuthology.orchestra.run.smithi072.stdout:mds.d smithi072 running (2m) 20s ago 2m 14.1M - 19.0.0-2344-gc3a04c0e 9d92c22378ed bf8e80142e1c 2024-03-22T01:36:11.164 INFO:teuthology.orchestra.run.smithi072.stdout:mds.e smithi089 running (2m) 56s ago 2m 12.9M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 1dd0843af24d 2024-03-22T01:36:11.164 INFO:teuthology.orchestra.run.smithi072.stdout:mds.f smithi098 running (2m) 56s ago 2m 15.2M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 882fad7c69f4 2024-03-22T01:36:11.164 INFO:teuthology.orchestra.run.smithi072.stdout:mds.g smithi072 running (2m) 20s ago 2m 12.9M - 19.0.0-2344-gc3a04c0e 9d92c22378ed d15911c82385 2024-03-22T01:36:11.164 INFO:teuthology.orchestra.run.smithi072.stdout:mds.h smithi089 running (2m) 56s ago 2m 16.2M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 608794ace6ce 2024-03-22T01:36:11.164 INFO:teuthology.orchestra.run.smithi072.stdout:mds.i smithi098 running (2m) 56s ago 2m 17.1M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 9ad857dd4375 2024-03-22T01:36:11.164 INFO:teuthology.orchestra.run.smithi072.stdout:mds.j smithi072 running (2m) 20s ago 2m 15.4M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 46300af5390d 2024-03-22T01:36:11.164 INFO:teuthology.orchestra.run.smithi072.stdout:mds.k smithi089 running (2m) 56s ago 2m 13.2M - 19.0.0-2344-gc3a04c0e 9d92c22378ed de4cb18e4d64 2024-03-22T01:36:11.164 INFO:teuthology.orchestra.run.smithi072.stdout:mds.l smithi098 running (2m) 56s ago 2m 13.1M - 19.0.0-2344-gc3a04c0e 9d92c22378ed f5b0a765eaa3 2024-03-22T01:36:11.164 INFO:teuthology.orchestra.run.smithi072.stdout:mgr.x smithi072 *:9283,8765 running (9m) 20s ago 9m 516M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 8322249439be 2024-03-22T01:36:11.164 INFO:teuthology.orchestra.run.smithi072.stdout:mgr.y smithi089 *:8443,8765 running (7m) 56s ago 7m 452M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 87cbf62f31f7 2024-03-22T01:36:11.164 INFO:teuthology.orchestra.run.smithi072.stdout:mgr.z smithi098 *:8443,8765 running (7m) 56s ago 7m 450M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 63922caaa500 2024-03-22T01:36:11.164 INFO:teuthology.orchestra.run.smithi072.stdout:mon.a smithi072 running (9m) 20s ago 9m 53.2M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed c91280b7adc3 2024-03-22T01:36:11.165 INFO:teuthology.orchestra.run.smithi072.stdout:mon.b smithi089 running (8m) 56s ago 8m 39.8M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed b9ecbcab6279 2024-03-22T01:36:11.165 INFO:teuthology.orchestra.run.smithi072.stdout:mon.c smithi098 running (8m) 56s ago 8m 38.5M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed b5e529ba5a42 2024-03-22T01:36:11.165 INFO:teuthology.orchestra.run.smithi072.stdout:osd.0 smithi072 running (7m) 20s ago 7m 45.0M 4272M 19.0.0-2344-gc3a04c0e 9d92c22378ed 7df82e628826 2024-03-22T01:36:11.166 INFO:teuthology.orchestra.run.smithi072.stdout:osd.1 smithi089 running (7m) 56s ago 7m 48.2M 4270M 19.0.0-2344-gc3a04c0e 9d92c22378ed 68400e4cebe9 2024-03-22T01:36:11.166 INFO:teuthology.orchestra.run.smithi072.stdout:osd.2 smithi098 running (6m) 56s ago 6m 46.9M 4270M 19.0.0-2344-gc3a04c0e 9d92c22378ed b1acd5988347 2024-03-22T01:36:11.166 INFO:teuthology.orchestra.run.smithi072.stdout:osd.3 smithi072 running (6m) 20s ago 6m 54.8M 4272M 19.0.0-2344-gc3a04c0e 9d92c22378ed 0b373b9fd134 2024-03-22T01:36:11.166 INFO:teuthology.orchestra.run.smithi072.stdout:osd.4 smithi089 running (5m) 56s ago 5m 44.5M 4270M 19.0.0-2344-gc3a04c0e 9d92c22378ed ad5594f41a0d 2024-03-22T01:36:11.166 INFO:teuthology.orchestra.run.smithi072.stdout:osd.5 smithi098 running (5m) 56s ago 5m 52.9M 4270M 19.0.0-2344-gc3a04c0e 9d92c22378ed d00090d52f75 2024-03-22T01:36:11.166 INFO:teuthology.orchestra.run.smithi072.stdout:osd.6 smithi072 running (5m) 20s ago 5m 43.1M 4272M 19.0.0-2344-gc3a04c0e 9d92c22378ed 3b7c37f0ddfb 2024-03-22T01:36:11.166 INFO:teuthology.orchestra.run.smithi072.stdout:osd.7 smithi089 running (4m) 56s ago 4m 45.2M 4270M 19.0.0-2344-gc3a04c0e 9d92c22378ed 385ef9dff082 2024-03-22T01:36:11.166 INFO:teuthology.orchestra.run.smithi072.stdout:osd.8 smithi098 running (4m) 56s ago 4m 54.1M 4270M 19.0.0-2344-gc3a04c0e 9d92c22378ed 1696af3747dc 2024-03-22T01:36:11.167 INFO:teuthology.orchestra.run.smithi072.stdout:osd.9 smithi072 running (3m) 20s ago 3m 42.7M 4272M 19.0.0-2344-gc3a04c0e 9d92c22378ed 72f019a34cdb 2024-03-22T01:36:11.167 INFO:teuthology.orchestra.run.smithi072.stdout:osd.10 smithi089 running (3m) 56s ago 3m 54.3M 4270M 19.0.0-2344-gc3a04c0e 9d92c22378ed 55ccb426eabe 2024-03-22T01:36:11.167 INFO:teuthology.orchestra.run.smithi072.stdout:osd.11 smithi098 running (2m) 56s ago 2m 47.7M 4270M 19.0.0-2344-gc3a04c0e 9d92c22378ed f324c17e1d0e 2024-03-22T01:36:11.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:11.167+0000 7f183ab66700 1 -- 172.21.15.72:0/1832244376 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f18240a2fb0 msgr2=0x7f18240a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:11.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:11.167+0000 7f183ab66700 1 --2- 172.21.15.72:0/1832244376 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f18240a2fb0 0x7f18240a5470 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f182000b2e0 tx=0x7f182000b880 comp rx=0 tx=0).stop 2024-03-22T01:36:11.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:11.167+0000 7f183ab66700 1 -- 172.21.15.72:0/1832244376 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1834103140 msgr2=0x7f1834198870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:11.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:11.167+0000 7f183ab66700 1 --2- 172.21.15.72:0/1832244376 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1834103140 0x7f1834198870 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f182800c960 tx=0x7f182800cc70 comp rx=0 tx=0).stop 2024-03-22T01:36:11.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:11.167+0000 7f183ab66700 1 -- 172.21.15.72:0/1832244376 shutdown_connections 2024-03-22T01:36:11.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:11.167+0000 7f183ab66700 1 --2- 172.21.15.72:0/1832244376 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f18240a2fb0 0x7f18240a5470 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:11.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:11.167+0000 7f183ab66700 1 --2- 172.21.15.72:0/1832244376 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18341083a0 0x7f183419fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:11.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:11.167+0000 7f183ab66700 1 --2- 172.21.15.72:0/1832244376 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1834103140 0x7f1834198870 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:11.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:11.167+0000 7f183ab66700 1 --2- 172.21.15.72:0/1832244376 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1834105a70 0x7f1834198db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:11.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:11.167+0000 7f183ab66700 1 -- 172.21.15.72:0/1832244376 >> 172.21.15.72:0/1832244376 conn(0x7f18340fcb70 msgr2=0x7f1834103a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:11.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:11.167+0000 7f183ab66700 1 -- 172.21.15.72:0/1832244376 shutdown_connections 2024-03-22T01:36:11.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:11.167+0000 7f183ab66700 1 -- 172.21.15.72:0/1832244376 wait complete. 2024-03-22T01:36:11.923 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-22T01:36:12.164 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:36:12.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:12 smithi089 ceph-mon[35770]: from='client.15231 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:36:12.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:12 smithi089 ceph-mon[35770]: pgmap v351: 129 pgs: 129 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-22T01:36:12.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:12 smithi072 ceph-mon[32414]: from='client.15231 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:36:12.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:12 smithi072 ceph-mon[32414]: pgmap v351: 129 pgs: 129 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-22T01:36:12.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:12 smithi098 ceph-mon[36468]: from='client.15231 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:36:12.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:12 smithi098 ceph-mon[36468]: pgmap v351: 129 pgs: 129 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-22T01:36:13.701 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:13 smithi072 ceph-mon[32414]: pgmap v352: 129 pgs: 129 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-22T01:36:13.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:13 smithi089 ceph-mon[35770]: pgmap v352: 129 pgs: 129 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-22T01:36:13.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:13 smithi098 ceph-mon[36468]: pgmap v352: 129 pgs: 129 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-22T01:36:13.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:13.928+0000 7fcc2266d700 1 -- 172.21.15.72:0/758949667 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcc1c103140 msgr2=0x7fcc1c1099f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:13.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:13.928+0000 7fcc2266d700 1 --2- 172.21.15.72:0/758949667 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcc1c103140 0x7fcc1c1099f0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fcc1400b0d0 tx=0x7fcc1400b3e0 comp rx=0 tx=0).stop 2024-03-22T01:36:13.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:13.928+0000 7fcc2266d700 1 -- 172.21.15.72:0/758949667 shutdown_connections 2024-03-22T01:36:13.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:13.928+0000 7fcc2266d700 1 --2- 172.21.15.72:0/758949667 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcc1c103140 0x7fcc1c1099f0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:13.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:13.928+0000 7fcc2266d700 1 --2- 172.21.15.72:0/758949667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc1c1015a0 0x7fcc1c1019c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:13.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:13.928+0000 7fcc2266d700 1 --2- 172.21.15.72:0/758949667 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcc1c1027a0 0x7fcc1c102c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:13.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:13.928+0000 7fcc2266d700 1 -- 172.21.15.72:0/758949667 >> 172.21.15.72:0/758949667 conn(0x7fcc1c0fcb40 msgr2=0x7fcc1c0fef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:13.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:13.928+0000 7fcc2266d700 1 -- 172.21.15.72:0/758949667 shutdown_connections 2024-03-22T01:36:13.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:13.928+0000 7fcc2266d700 1 -- 172.21.15.72:0/758949667 wait complete. 2024-03-22T01:36:13.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:13.929+0000 7fcc2266d700 1 Processor -- start 2024-03-22T01:36:13.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:13.929+0000 7fcc2266d700 1 -- start start 2024-03-22T01:36:13.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:13.929+0000 7fcc2266d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc1c1015a0 0x7fcc1c197740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:13.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:13.929+0000 7fcc2266d700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcc1c1027a0 0x7fcc1c197c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:13.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:13.929+0000 7fcc2266d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcc1c103140 0x7fcc1c19ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:13.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:13.929+0000 7fcc2266d700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fcc1c10d190 con 0x7fcc1c1027a0 2024-03-22T01:36:13.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:13.929+0000 7fcc2266d700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fcc1c10d010 con 0x7fcc1c103140 2024-03-22T01:36:13.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:13.929+0000 7fcc2266d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcc1c10d310 con 0x7fcc1c1015a0 2024-03-22T01:36:13.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:13.929+0000 7fcc1b7fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcc1c1027a0 0x7fcc1c197c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:13.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:13.930+0000 7fcc1b7fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcc1c1027a0 0x7fcc1c197c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:53228/0 (socket says 172.21.15.72:53228) 2024-03-22T01:36:13.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:13.930+0000 7fcc1b7fe700 1 -- 172.21.15.72:0/230414170 learned_addr learned my addr 172.21.15.72:0/230414170 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:13.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:13.930+0000 7fcc20c0a700 1 --2- 172.21.15.72:0/230414170 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcc1c103140 0x7fcc1c19ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:13.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:13.930+0000 7fcc1bfff700 1 --2- 172.21.15.72:0/230414170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc1c1015a0 0x7fcc1c197740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:13.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:13.930+0000 7fcc20c0a700 1 -- 172.21.15.72:0/230414170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc1c1015a0 msgr2=0x7fcc1c197740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:13.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:13.930+0000 7fcc20c0a700 1 --2- 172.21.15.72:0/230414170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc1c1015a0 0x7fcc1c197740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:13.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:13.930+0000 7fcc20c0a700 1 -- 172.21.15.72:0/230414170 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcc1c1027a0 msgr2=0x7fcc1c197c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:13.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:13.930+0000 7fcc20c0a700 1 --2- 172.21.15.72:0/230414170 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcc1c1027a0 0x7fcc1c197c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:13.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:13.930+0000 7fcc20c0a700 1 -- 172.21.15.72:0/230414170 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc1c06af00 con 0x7fcc1c103140 2024-03-22T01:36:13.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:13.930+0000 7fcc1b7fe700 1 --2- 172.21.15.72:0/230414170 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcc1c1027a0 0x7fcc1c197c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:36:13.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:13.930+0000 7fcc1bfff700 1 --2- 172.21.15.72:0/230414170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc1c1015a0 0x7fcc1c197740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:36:13.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:13.930+0000 7fcc20c0a700 1 --2- 172.21.15.72:0/230414170 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcc1c103140 0x7fcc1c19ed00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fcc1400d010 tx=0x7fcc14007a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:13.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:13.930+0000 7fcc197fa700 1 -- 172.21.15.72:0/230414170 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc14021070 con 0x7fcc1c103140 2024-03-22T01:36:13.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:13.930+0000 7fcc2266d700 1 -- 172.21.15.72:0/230414170 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc1c06b190 con 0x7fcc1c103140 2024-03-22T01:36:13.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:13.930+0000 7fcc197fa700 1 -- 172.21.15.72:0/230414170 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc1401dd40 con 0x7fcc1c103140 2024-03-22T01:36:13.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:13.930+0000 7fcc197fa700 1 -- 172.21.15.72:0/230414170 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc1401bc00 con 0x7fcc1c103140 2024-03-22T01:36:13.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:13.930+0000 7fcc2266d700 1 -- 172.21.15.72:0/230414170 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc1c06b6d0 con 0x7fcc1c103140 2024-03-22T01:36:13.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:13.931+0000 7fcc2266d700 1 -- 172.21.15.72:0/230414170 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc1c069340 con 0x7fcc1c103140 2024-03-22T01:36:13.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:13.932+0000 7fcc197fa700 1 -- 172.21.15.72:0/230414170 <== mon.2 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7fcc14024070 con 0x7fcc1c103140 2024-03-22T01:36:13.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:13.933+0000 7fcc197fa700 1 --2- 172.21.15.72:0/230414170 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fcc040a2fb0 0x7fcc040a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:13.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:13.933+0000 7fcc1bfff700 1 --2- 172.21.15.72:0/230414170 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fcc040a2fb0 0x7fcc040a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:13.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:13.933+0000 7fcc197fa700 1 -- 172.21.15.72:0/230414170 <== mon.2 v2:172.21.15.89:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fcc140d0900 con 0x7fcc1c103140 2024-03-22T01:36:13.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:13.933+0000 7fcc1bfff700 1 --2- 172.21.15.72:0/230414170 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fcc040a2fb0 0x7fcc040a5470 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c005950 tx=0x7fcc0c005f50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:13.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:13.937+0000 7fcc197fa700 1 -- 172.21.15.72:0/230414170 <== mon.2 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcc1409d400 con 0x7fcc1c103140 2024-03-22T01:36:14.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:14.144+0000 7fcc2266d700 1 -- 172.21.15.72:0/230414170 --> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7fcc1c062ac0 con 0x7fcc040a2fb0 2024-03-22T01:36:14.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:14.151+0000 7fcc197fa700 1 -- 172.21.15.72:0/230414170 <== mgr.14152 v2:172.21.15.72:6800/401403200 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7fcc1c062ac0 con 0x7fcc040a2fb0 2024-03-22T01:36:14.150 INFO:teuthology.orchestra.run.smithi072.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-22T01:36:14.150 INFO:teuthology.orchestra.run.smithi072.stdout:mds.all 12/12 59s ago 2m smithi072=a;smithi072=d;smithi072=g;smithi072=j;smithi089=b;smithi089=e;smithi089=h;smithi089=k;smithi098=c;smithi098=f;smithi098=i;smithi098=l;count:12 2024-03-22T01:36:14.150 INFO:teuthology.orchestra.run.smithi072.stdout:mgr 3/3 59s ago 7m smithi072=x;smithi089=y;smithi098=z;count:3 2024-03-22T01:36:14.151 INFO:teuthology.orchestra.run.smithi072.stdout:mon 3/3 59s ago 8m smithi072:172.21.15.72=a;smithi089:172.21.15.89=b;smithi098:172.21.15.98=c;count:3 2024-03-22T01:36:14.151 INFO:teuthology.orchestra.run.smithi072.stdout:osd 12 59s ago - 2024-03-22T01:36:14.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:14.153+0000 7fcc2266d700 1 -- 172.21.15.72:0/230414170 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fcc040a2fb0 msgr2=0x7fcc040a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:14.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:14.153+0000 7fcc2266d700 1 --2- 172.21.15.72:0/230414170 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fcc040a2fb0 0x7fcc040a5470 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c005950 tx=0x7fcc0c005f50 comp rx=0 tx=0).stop 2024-03-22T01:36:14.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:14.153+0000 7fcc2266d700 1 -- 172.21.15.72:0/230414170 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcc1c103140 msgr2=0x7fcc1c19ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:14.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:14.153+0000 7fcc2266d700 1 --2- 172.21.15.72:0/230414170 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcc1c103140 0x7fcc1c19ed00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fcc1400d010 tx=0x7fcc14007a20 comp rx=0 tx=0).stop 2024-03-22T01:36:14.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:14.153+0000 7fcc2266d700 1 -- 172.21.15.72:0/230414170 shutdown_connections 2024-03-22T01:36:14.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:14.153+0000 7fcc2266d700 1 --2- 172.21.15.72:0/230414170 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fcc040a2fb0 0x7fcc040a5470 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:14.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:14.153+0000 7fcc2266d700 1 --2- 172.21.15.72:0/230414170 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcc1c103140 0x7fcc1c19ed00 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:14.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:14.153+0000 7fcc2266d700 1 --2- 172.21.15.72:0/230414170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc1c1015a0 0x7fcc1c197740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:14.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:14.153+0000 7fcc2266d700 1 --2- 172.21.15.72:0/230414170 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcc1c1027a0 0x7fcc1c197c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:14.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:14.153+0000 7fcc2266d700 1 -- 172.21.15.72:0/230414170 >> 172.21.15.72:0/230414170 conn(0x7fcc1c0fcb40 msgr2=0x7fcc1c105a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:14.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:14.153+0000 7fcc2266d700 1 -- 172.21.15.72:0/230414170 shutdown_connections 2024-03-22T01:36:14.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:14.153+0000 7fcc2266d700 1 -- 172.21.15.72:0/230414170 wait complete. 2024-03-22T01:36:14.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:14 smithi072 ceph-mon[32414]: from='client.25097 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:36:14.910 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-22T01:36:14.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:14 smithi098 ceph-mon[36468]: from='client.25097 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:36:15.107 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:14 smithi089 ceph-mon[35770]: from='client.25097 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:36:15.159 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:36:15.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:15 smithi072 ceph-mon[32414]: pgmap v353: 129 pgs: 129 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-22T01:36:15.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:15 smithi098 ceph-mon[36468]: pgmap v353: 129 pgs: 129 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-22T01:36:16.107 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:15 smithi089 ceph-mon[35770]: pgmap v353: 129 pgs: 129 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-22T01:36:16.758 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:16 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]: dispatch 2024-03-22T01:36:16.758 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:16 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [5, 10]}]: dispatch 2024-03-22T01:36:16.758 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:16 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-03-22T01:36:16.758 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:16 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 10]}]: dispatch 2024-03-22T01:36:16.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:16 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]: dispatch 2024-03-22T01:36:16.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:16 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [5, 10]}]: dispatch 2024-03-22T01:36:16.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:16 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-03-22T01:36:16.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:16 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 10]}]: dispatch 2024-03-22T01:36:16.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:16.973+0000 7fa7d3292700 1 -- 172.21.15.72:0/4052577689 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa7cc1031c0 msgr2=0x7fa7cc109a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:16.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:16.973+0000 7fa7d3292700 1 --2- 172.21.15.72:0/4052577689 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa7cc1031c0 0x7fa7cc109a70 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fa7c400b0d0 tx=0x7fa7c400b3e0 comp rx=0 tx=0).stop 2024-03-22T01:36:16.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:16.973+0000 7fa7d3292700 1 -- 172.21.15.72:0/4052577689 shutdown_connections 2024-03-22T01:36:16.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:16.973+0000 7fa7d3292700 1 --2- 172.21.15.72:0/4052577689 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa7cc1031c0 0x7fa7cc109a70 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:16.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:16.973+0000 7fa7d3292700 1 --2- 172.21.15.72:0/4052577689 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7cc101620 0x7fa7cc101a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:16.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:16.973+0000 7fa7d3292700 1 --2- 172.21.15.72:0/4052577689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7cc102820 0x7fa7cc102c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:16.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:16.973+0000 7fa7d3292700 1 -- 172.21.15.72:0/4052577689 >> 172.21.15.72:0/4052577689 conn(0x7fa7cc0fcbe0 msgr2=0x7fa7cc0ff000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:16.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:16.973+0000 7fa7d3292700 1 -- 172.21.15.72:0/4052577689 shutdown_connections 2024-03-22T01:36:16.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:16.973+0000 7fa7d3292700 1 -- 172.21.15.72:0/4052577689 wait complete. 2024-03-22T01:36:16.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:16.974+0000 7fa7d3292700 1 Processor -- start 2024-03-22T01:36:16.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:16.974+0000 7fa7d3292700 1 -- start start 2024-03-22T01:36:16.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:16.974+0000 7fa7d3292700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7cc101620 0x7fa7cc198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:16.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:16.974+0000 7fa7d3292700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7cc102820 0x7fa7cc198db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:16.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:16.974+0000 7fa7d3292700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa7cc1031c0 0x7fa7cc19fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:16.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:16.974+0000 7fa7d3292700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fa7cc10d1a0 con 0x7fa7cc1031c0 2024-03-22T01:36:16.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:16.974+0000 7fa7d3292700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa7cc10d020 con 0x7fa7cc102820 2024-03-22T01:36:16.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:16.974+0000 7fa7d3292700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa7cc10d320 con 0x7fa7cc101620 2024-03-22T01:36:16.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:16.974+0000 7fa7d182f700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa7cc1031c0 0x7fa7cc19fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:16.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:16.974+0000 7fa7d182f700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa7cc1031c0 0x7fa7cc19fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:53248/0 (socket says 172.21.15.72:53248) 2024-03-22T01:36:16.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:16.974+0000 7fa7d182f700 1 -- 172.21.15.72:0/2326086292 learned_addr learned my addr 172.21.15.72:0/2326086292 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:16.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:16.974+0000 7fa7d082d700 1 --2- 172.21.15.72:0/2326086292 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7cc102820 0x7fa7cc198db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:16.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:16.974+0000 7fa7d102e700 1 --2- 172.21.15.72:0/2326086292 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7cc101620 0x7fa7cc198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:16.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:16.974+0000 7fa7d182f700 1 -- 172.21.15.72:0/2326086292 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7cc101620 msgr2=0x7fa7cc198870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:16.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:16.974+0000 7fa7d182f700 1 --2- 172.21.15.72:0/2326086292 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7cc101620 0x7fa7cc198870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:16.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:16.974+0000 7fa7d182f700 1 -- 172.21.15.72:0/2326086292 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7cc102820 msgr2=0x7fa7cc198db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:16.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:16.974+0000 7fa7d182f700 1 --2- 172.21.15.72:0/2326086292 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7cc102820 0x7fa7cc198db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:16.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:16.974+0000 7fa7d182f700 1 -- 172.21.15.72:0/2326086292 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7cc1a04d0 con 0x7fa7cc1031c0 2024-03-22T01:36:16.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:16.974+0000 7fa7d102e700 1 --2- 172.21.15.72:0/2326086292 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7cc101620 0x7fa7cc198870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:36:16.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:16.974+0000 7fa7d082d700 1 --2- 172.21.15.72:0/2326086292 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7cc102820 0x7fa7cc198db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:36:16.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:16.975+0000 7fa7d182f700 1 --2- 172.21.15.72:0/2326086292 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa7cc1031c0 0x7fa7cc19fe30 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fa7c400b770 tx=0x7fa7c4007a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:16.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:16.975+0000 7fa7c27fc700 1 -- 172.21.15.72:0/2326086292 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7c4003e10 con 0x7fa7cc1031c0 2024-03-22T01:36:16.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:16.975+0000 7fa7c27fc700 1 -- 172.21.15.72:0/2326086292 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7c4007cc0 con 0x7fa7cc1031c0 2024-03-22T01:36:16.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:16.975+0000 7fa7d3292700 1 -- 172.21.15.72:0/2326086292 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7cc1a0760 con 0x7fa7cc1031c0 2024-03-22T01:36:16.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:16.975+0000 7fa7c27fc700 1 -- 172.21.15.72:0/2326086292 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7c401ba90 con 0x7fa7cc1031c0 2024-03-22T01:36:16.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:16.975+0000 7fa7d3292700 1 -- 172.21.15.72:0/2326086292 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7cc1a0c10 con 0x7fa7cc1031c0 2024-03-22T01:36:16.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:16.976+0000 7fa7d3292700 1 -- 172.21.15.72:0/2326086292 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7cc069340 con 0x7fa7cc1031c0 2024-03-22T01:36:16.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:16.977+0000 7fa7c27fc700 1 -- 172.21.15.72:0/2326086292 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7fa7c4026450 con 0x7fa7cc1031c0 2024-03-22T01:36:16.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:16.977+0000 7fa7c27fc700 1 --2- 172.21.15.72:0/2326086292 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fa7b80a3030 0x7fa7b80a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:16.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:16.978+0000 7fa7d102e700 1 --2- 172.21.15.72:0/2326086292 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fa7b80a3030 0x7fa7b80a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:16.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:16.978+0000 7fa7c27fc700 1 -- 172.21.15.72:0/2326086292 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fa7c400f020 con 0x7fa7cc1031c0 2024-03-22T01:36:16.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:16.978+0000 7fa7d102e700 1 --2- 172.21.15.72:0/2326086292 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fa7b80a3030 0x7fa7b80a54f0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fa7cc102680 tx=0x7fa7c8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:16.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:16.982+0000 7fa7c27fc700 1 -- 172.21.15.72:0/2326086292 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa7c409d550 con 0x7fa7cc1031c0 2024-03-22T01:36:17.107 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:16 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]: dispatch 2024-03-22T01:36:17.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:16 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [5, 10]}]: dispatch 2024-03-22T01:36:17.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:16 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-03-22T01:36:17.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:16 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 10]}]: dispatch 2024-03-22T01:36:17.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:17.190+0000 7fa7d3292700 1 -- 172.21.15.72:0/2326086292 --> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7fa7cc0fed00 con 0x7fa7b80a3030 2024-03-22T01:36:17.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:17.192+0000 7fa7c27fc700 1 -- 172.21.15.72:0/2326086292 <== mgr.14152 v2:172.21.15.72:6800/401403200 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+187 (secure 0 0 0) 0x7fa7cc0fed00 con 0x7fa7b80a3030 2024-03-22T01:36:17.191 INFO:teuthology.orchestra.run.smithi072.stdout:HOST ADDR LABELS STATUS 2024-03-22T01:36:17.191 INFO:teuthology.orchestra.run.smithi072.stdout:smithi072 172.21.15.72 2024-03-22T01:36:17.191 INFO:teuthology.orchestra.run.smithi072.stdout:smithi089 172.21.15.89 2024-03-22T01:36:17.191 INFO:teuthology.orchestra.run.smithi072.stdout:smithi098 172.21.15.98 2024-03-22T01:36:17.191 INFO:teuthology.orchestra.run.smithi072.stdout:3 hosts in cluster 2024-03-22T01:36:17.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:17.194+0000 7fa7d3292700 1 -- 172.21.15.72:0/2326086292 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fa7b80a3030 msgr2=0x7fa7b80a54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:17.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:17.194+0000 7fa7d3292700 1 --2- 172.21.15.72:0/2326086292 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fa7b80a3030 0x7fa7b80a54f0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fa7cc102680 tx=0x7fa7c8009260 comp rx=0 tx=0).stop 2024-03-22T01:36:17.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:17.194+0000 7fa7d3292700 1 -- 172.21.15.72:0/2326086292 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa7cc1031c0 msgr2=0x7fa7cc19fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:17.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:17.194+0000 7fa7d3292700 1 --2- 172.21.15.72:0/2326086292 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa7cc1031c0 0x7fa7cc19fe30 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fa7c400b770 tx=0x7fa7c4007a20 comp rx=0 tx=0).stop 2024-03-22T01:36:17.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:17.194+0000 7fa7d3292700 1 -- 172.21.15.72:0/2326086292 shutdown_connections 2024-03-22T01:36:17.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:17.194+0000 7fa7d3292700 1 --2- 172.21.15.72:0/2326086292 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fa7b80a3030 0x7fa7b80a54f0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:17.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:17.194+0000 7fa7d3292700 1 --2- 172.21.15.72:0/2326086292 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa7cc1031c0 0x7fa7cc19fe30 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:17.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:17.194+0000 7fa7d3292700 1 --2- 172.21.15.72:0/2326086292 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7cc101620 0x7fa7cc198870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:17.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:17.194+0000 7fa7d3292700 1 --2- 172.21.15.72:0/2326086292 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7cc102820 0x7fa7cc198db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:17.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:17.194+0000 7fa7d3292700 1 -- 172.21.15.72:0/2326086292 >> 172.21.15.72:0/2326086292 conn(0x7fa7cc0fcbe0 msgr2=0x7fa7cc0fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:17.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:17.194+0000 7fa7d3292700 1 -- 172.21.15.72:0/2326086292 shutdown_connections 2024-03-22T01:36:17.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:17.194+0000 7fa7d3292700 1 -- 172.21.15.72:0/2326086292 wait complete. 2024-03-22T01:36:17.825 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:17 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]': finished 2024-03-22T01:36:17.825 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:17 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [5, 10]}]': finished 2024-03-22T01:36:17.825 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:17 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-03-22T01:36:17.825 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:17 smithi072 ceph-mon[32414]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 10]}]': finished 2024-03-22T01:36:17.826 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:17 smithi072 ceph-mon[32414]: osdmap e94: 12 total, 12 up, 12 in 2024-03-22T01:36:17.826 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:17 smithi072 ceph-mon[32414]: from='client.15243 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:36:17.826 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:17 smithi072 ceph-mon[32414]: pgmap v355: 129 pgs: 1 peering, 128 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-03-22T01:36:17.853 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-22T01:36:17.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:17 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]': finished 2024-03-22T01:36:17.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:17 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [5, 10]}]': finished 2024-03-22T01:36:17.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:17 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-03-22T01:36:17.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:17 smithi098 ceph-mon[36468]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 10]}]': finished 2024-03-22T01:36:17.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:17 smithi098 ceph-mon[36468]: osdmap e94: 12 total, 12 up, 12 in 2024-03-22T01:36:17.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:17 smithi098 ceph-mon[36468]: from='client.15243 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:36:17.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:17 smithi098 ceph-mon[36468]: pgmap v355: 129 pgs: 1 peering, 128 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-03-22T01:36:18.097 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:36:18.107 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:17 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]': finished 2024-03-22T01:36:18.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:17 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [5, 10]}]': finished 2024-03-22T01:36:18.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:17 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-03-22T01:36:18.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:17 smithi089 ceph-mon[35770]: from='mgr.14152 172.21.15.72:0/1462368005' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 10]}]': finished 2024-03-22T01:36:18.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:17 smithi089 ceph-mon[35770]: osdmap e94: 12 total, 12 up, 12 in 2024-03-22T01:36:18.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:17 smithi089 ceph-mon[35770]: from='client.15243 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:36:18.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:17 smithi089 ceph-mon[35770]: pgmap v355: 129 pgs: 1 peering, 128 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-03-22T01:36:18.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:18 smithi072 ceph-mon[32414]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-22T01:36:18.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:18 smithi072 ceph-mon[32414]: osdmap e95: 12 total, 12 up, 12 in 2024-03-22T01:36:18.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:18 smithi098 ceph-mon[36468]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-22T01:36:18.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:18 smithi098 ceph-mon[36468]: osdmap e95: 12 total, 12 up, 12 in 2024-03-22T01:36:19.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:18 smithi089 ceph-mon[35770]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-22T01:36:19.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:18 smithi089 ceph-mon[35770]: osdmap e95: 12 total, 12 up, 12 in 2024-03-22T01:36:19.780 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:19 smithi072 ceph-mon[32414]: pgmap v357: 129 pgs: 1 peering, 128 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 1 op/s 2024-03-22T01:36:19.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:19.879+0000 7fca1e937700 1 -- 172.21.15.72:0/1761897523 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fca181038b0 msgr2=0x7fca18103d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:19.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:19.879+0000 7fca1e937700 1 --2- 172.21.15.72:0/1761897523 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fca181038b0 0x7fca18103d10 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fca080099c0 tx=0x7fca08009cd0 comp rx=0 tx=0).stop 2024-03-22T01:36:19.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:19.879+0000 7fca1e937700 1 -- 172.21.15.72:0/1761897523 shutdown_connections 2024-03-22T01:36:19.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:19.879+0000 7fca1e937700 1 --2- 172.21.15.72:0/1761897523 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca18104250 0x7fca1810ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:19.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:19.879+0000 7fca1e937700 1 --2- 172.21.15.72:0/1761897523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca181026b0 0x7fca18102ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:19.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:19.879+0000 7fca1e937700 1 --2- 172.21.15.72:0/1761897523 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fca181038b0 0x7fca18103d10 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:19.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:19.879+0000 7fca1e937700 1 -- 172.21.15.72:0/1761897523 >> 172.21.15.72:0/1761897523 conn(0x7fca180fdc90 msgr2=0x7fca181000b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:19.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:19.879+0000 7fca1e937700 1 -- 172.21.15.72:0/1761897523 shutdown_connections 2024-03-22T01:36:19.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:19.879+0000 7fca1e937700 1 -- 172.21.15.72:0/1761897523 wait complete. 2024-03-22T01:36:19.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:19.879+0000 7fca1e937700 1 Processor -- start 2024-03-22T01:36:19.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:19.879+0000 7fca1e937700 1 -- start start 2024-03-22T01:36:19.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:19.880+0000 7fca1e937700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fca181026b0 0x7fca181988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:19.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:19.880+0000 7fca1e937700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca181038b0 0x7fca18198e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:19.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:19.880+0000 7fca1e937700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca18104250 0x7fca1819fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:19.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:19.880+0000 7fca1e937700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fca1810e280 con 0x7fca181026b0 2024-03-22T01:36:19.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:19.880+0000 7fca1e937700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fca1810e100 con 0x7fca18104250 2024-03-22T01:36:19.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:19.880+0000 7fca1e937700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fca1810e400 con 0x7fca181038b0 2024-03-22T01:36:19.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:19.880+0000 7fca17fff700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fca181026b0 0x7fca181988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:19.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:19.880+0000 7fca17fff700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fca181026b0 0x7fca181988c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:53278/0 (socket says 172.21.15.72:53278) 2024-03-22T01:36:19.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:19.880+0000 7fca17fff700 1 -- 172.21.15.72:0/2115255821 learned_addr learned my addr 172.21.15.72:0/2115255821 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:19.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:19.880+0000 7fca177fe700 1 --2- 172.21.15.72:0/2115255821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca181038b0 0x7fca18198e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:19.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:19.880+0000 7fca1ced4700 1 --2- 172.21.15.72:0/2115255821 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca18104250 0x7fca1819fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:19.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:19.880+0000 7fca17fff700 1 -- 172.21.15.72:0/2115255821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca181038b0 msgr2=0x7fca18198e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:19.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:19.880+0000 7fca17fff700 1 --2- 172.21.15.72:0/2115255821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca181038b0 0x7fca18198e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:19.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:19.880+0000 7fca17fff700 1 -- 172.21.15.72:0/2115255821 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca18104250 msgr2=0x7fca1819fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:19.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:19.880+0000 7fca17fff700 1 --2- 172.21.15.72:0/2115255821 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca18104250 0x7fca1819fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:19.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:19.880+0000 7fca17fff700 1 -- 172.21.15.72:0/2115255821 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca181a0520 con 0x7fca181026b0 2024-03-22T01:36:19.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:19.880+0000 7fca1ced4700 1 --2- 172.21.15.72:0/2115255821 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca18104250 0x7fca1819fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:36:19.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:19.880+0000 7fca17fff700 1 --2- 172.21.15.72:0/2115255821 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fca181026b0 0x7fca181988c0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fca0c00d9b0 tx=0x7fca0c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:19.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:19.880+0000 7fca157fa700 1 -- 172.21.15.72:0/2115255821 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca0c017030 con 0x7fca181026b0 2024-03-22T01:36:19.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:19.880+0000 7fca157fa700 1 -- 172.21.15.72:0/2115255821 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fca0c004500 con 0x7fca181026b0 2024-03-22T01:36:19.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:19.880+0000 7fca1e937700 1 -- 172.21.15.72:0/2115255821 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca181a0810 con 0x7fca181026b0 2024-03-22T01:36:19.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:19.880+0000 7fca157fa700 1 -- 172.21.15.72:0/2115255821 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca0c003970 con 0x7fca181026b0 2024-03-22T01:36:19.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:19.881+0000 7fca1e937700 1 -- 172.21.15.72:0/2115255821 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca181a0d50 con 0x7fca181026b0 2024-03-22T01:36:19.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:19.881+0000 7fca1e937700 1 -- 172.21.15.72:0/2115255821 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca18069340 con 0x7fca181026b0 2024-03-22T01:36:19.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:19.883+0000 7fca157fa700 1 -- 172.21.15.72:0/2115255821 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7fca0c01a070 con 0x7fca181026b0 2024-03-22T01:36:19.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:19.883+0000 7fca157fa700 1 --2- 172.21.15.72:0/2115255821 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fca000a2fb0 0x7fca000a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:19.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:19.883+0000 7fca177fe700 1 --2- 172.21.15.72:0/2115255821 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fca000a2fb0 0x7fca000a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:19.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:19.883+0000 7fca157fa700 1 -- 172.21.15.72:0/2115255821 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7fca0c0cf3b0 con 0x7fca181026b0 2024-03-22T01:36:19.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:19.884+0000 7fca177fe700 1 --2- 172.21.15.72:0/2115255821 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fca000a2fb0 0x7fca000a5470 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fca08000c00 tx=0x7fca08005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:19.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:19.887+0000 7fca157fa700 1 -- 172.21.15.72:0/2115255821 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fca0c00e350 con 0x7fca181026b0 2024-03-22T01:36:19.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:19 smithi098 ceph-mon[36468]: pgmap v357: 129 pgs: 1 peering, 128 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 1 op/s 2024-03-22T01:36:20.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:20.092+0000 7fca1e937700 1 -- 172.21.15.72:0/2115255821 --> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7fca18062ac0 con 0x7fca000a2fb0 2024-03-22T01:36:20.094 INFO:teuthology.orchestra.run.smithi072.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-22T01:36:20.095 INFO:teuthology.orchestra.run.smithi072.stdout:smithi072 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400V7400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-22T01:36:20.095 INFO:teuthology.orchestra.run.smithi072.stdout:smithi089 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT62040126400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-22T01:36:20.095 INFO:teuthology.orchestra.run.smithi072.stdout:smithi098 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT6204010H400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-03-22T01:36:20.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:20.094+0000 7fca157fa700 1 -- 172.21.15.72:0/2115255821 <== mgr.14152 v2:172.21.15.72:6800/401403200 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7fca18062ac0 con 0x7fca000a2fb0 2024-03-22T01:36:20.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:20.097+0000 7fca1e937700 1 -- 172.21.15.72:0/2115255821 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fca000a2fb0 msgr2=0x7fca000a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:20.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:20.097+0000 7fca1e937700 1 --2- 172.21.15.72:0/2115255821 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fca000a2fb0 0x7fca000a5470 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fca08000c00 tx=0x7fca08005fb0 comp rx=0 tx=0).stop 2024-03-22T01:36:20.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:20.097+0000 7fca1e937700 1 -- 172.21.15.72:0/2115255821 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fca181026b0 msgr2=0x7fca181988c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:20.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:20.097+0000 7fca1e937700 1 --2- 172.21.15.72:0/2115255821 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fca181026b0 0x7fca181988c0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fca0c00d9b0 tx=0x7fca0c00dcc0 comp rx=0 tx=0).stop 2024-03-22T01:36:20.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:20.097+0000 7fca1e937700 1 -- 172.21.15.72:0/2115255821 shutdown_connections 2024-03-22T01:36:20.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:20.097+0000 7fca1e937700 1 --2- 172.21.15.72:0/2115255821 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fca000a2fb0 0x7fca000a5470 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:20.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:20.097+0000 7fca1e937700 1 --2- 172.21.15.72:0/2115255821 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca18104250 0x7fca1819fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:20.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:20.097+0000 7fca1e937700 1 --2- 172.21.15.72:0/2115255821 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fca181026b0 0x7fca181988c0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:20.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:20.097+0000 7fca1e937700 1 --2- 172.21.15.72:0/2115255821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca181038b0 0x7fca18198e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:20.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:20.097+0000 7fca1e937700 1 -- 172.21.15.72:0/2115255821 >> 172.21.15.72:0/2115255821 conn(0x7fca180fdc90 msgr2=0x7fca18106c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:20.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:20.097+0000 7fca1e937700 1 -- 172.21.15.72:0/2115255821 shutdown_connections 2024-03-22T01:36:20.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:20.097+0000 7fca1e937700 1 -- 172.21.15.72:0/2115255821 wait complete. 2024-03-22T01:36:20.107 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:19 smithi089 ceph-mon[35770]: pgmap v357: 129 pgs: 1 peering, 128 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 1 op/s 2024-03-22T01:36:20.783 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:20 smithi072 ceph-mon[32414]: from='client.15249 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:36:20.810 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-22T01:36:20.820 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi072.front.sepia.ceph.com 2024-03-22T01:36:20.820 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-22T01:36:20.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:20 smithi098 ceph-mon[36468]: from='client.15249 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:36:21.060 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:36:21.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:20 smithi089 ceph-mon[35770]: from='client.15249 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:36:21.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:21 smithi072 ceph-mon[32414]: pgmap v358: 129 pgs: 1 peering, 128 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-22T01:36:21.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:21 smithi098 ceph-mon[36468]: pgmap v358: 129 pgs: 1 peering, 128 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-22T01:36:22.107 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:21 smithi089 ceph-mon[35770]: pgmap v358: 129 pgs: 1 peering, 128 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-22T01:36:22.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:22.729+0000 7fb52ccea700 1 -- 172.21.15.72:0/974288886 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb5281038b0 msgr2=0x7fb528103d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:22.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:22.729+0000 7fb52ccea700 1 --2- 172.21.15.72:0/974288886 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb5281038b0 0x7fb528103d10 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fb51c0099c0 tx=0x7fb51c009cd0 comp rx=0 tx=0).stop 2024-03-22T01:36:22.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:22.729+0000 7fb52ccea700 1 -- 172.21.15.72:0/974288886 shutdown_connections 2024-03-22T01:36:22.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:22.729+0000 7fb52ccea700 1 --2- 172.21.15.72:0/974288886 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb528104250 0x7fb52810ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:22.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:22.729+0000 7fb52ccea700 1 --2- 172.21.15.72:0/974288886 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5281026b0 0x7fb528102ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:22.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:22.729+0000 7fb52ccea700 1 --2- 172.21.15.72:0/974288886 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb5281038b0 0x7fb528103d10 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:22.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:22.729+0000 7fb52ccea700 1 -- 172.21.15.72:0/974288886 >> 172.21.15.72:0/974288886 conn(0x7fb5280fdc90 msgr2=0x7fb5281000b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:22.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:22.729+0000 7fb52ccea700 1 -- 172.21.15.72:0/974288886 shutdown_connections 2024-03-22T01:36:22.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:22.729+0000 7fb52ccea700 1 -- 172.21.15.72:0/974288886 wait complete. 2024-03-22T01:36:22.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:22.729+0000 7fb52ccea700 1 Processor -- start 2024-03-22T01:36:22.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:22.729+0000 7fb52ccea700 1 -- start start 2024-03-22T01:36:22.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:22.730+0000 7fb52ccea700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb5281026b0 0x7fb5281988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:22.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:22.730+0000 7fb52ccea700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5281038b0 0x7fb528198e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:22.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:22.730+0000 7fb52ccea700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb528104250 0x7fb52819fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:22.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:22.730+0000 7fb52ccea700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fb52810e280 con 0x7fb5281026b0 2024-03-22T01:36:22.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:22.730+0000 7fb52ccea700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb52810e100 con 0x7fb5281038b0 2024-03-22T01:36:22.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:22.730+0000 7fb52ccea700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb52810e400 con 0x7fb528104250 2024-03-22T01:36:22.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:22.730+0000 7fb52659c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb5281026b0 0x7fb5281988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:22.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:22.730+0000 7fb52659c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb5281026b0 0x7fb5281988c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:53488/0 (socket says 172.21.15.72:53488) 2024-03-22T01:36:22.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:22.730+0000 7fb52659c700 1 -- 172.21.15.72:0/2899648457 learned_addr learned my addr 172.21.15.72:0/2899648457 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:22.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:22.730+0000 7fb525d9b700 1 --2- 172.21.15.72:0/2899648457 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5281038b0 0x7fb528198e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:22.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:22.730+0000 7fb526d9d700 1 --2- 172.21.15.72:0/2899648457 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb528104250 0x7fb52819fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:22.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:22.730+0000 7fb52659c700 1 -- 172.21.15.72:0/2899648457 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb528104250 msgr2=0x7fb52819fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:22.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:22.730+0000 7fb52659c700 1 --2- 172.21.15.72:0/2899648457 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb528104250 0x7fb52819fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:22.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:22.730+0000 7fb52659c700 1 -- 172.21.15.72:0/2899648457 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5281038b0 msgr2=0x7fb528198e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:22.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:22.730+0000 7fb52659c700 1 --2- 172.21.15.72:0/2899648457 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5281038b0 0x7fb528198e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:22.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:22.730+0000 7fb52659c700 1 -- 172.21.15.72:0/2899648457 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5281a0520 con 0x7fb5281026b0 2024-03-22T01:36:22.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:22.730+0000 7fb52659c700 1 --2- 172.21.15.72:0/2899648457 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb5281026b0 0x7fb5281988c0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fb51800d9b0 tx=0x7fb51800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:22.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:22.730+0000 7fb5177fe700 1 -- 172.21.15.72:0/2899648457 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb518017030 con 0x7fb5281026b0 2024-03-22T01:36:22.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:22.730+0000 7fb52ccea700 1 -- 172.21.15.72:0/2899648457 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5281a0810 con 0x7fb5281026b0 2024-03-22T01:36:22.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:22.730+0000 7fb5177fe700 1 -- 172.21.15.72:0/2899648457 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb518004500 con 0x7fb5281026b0 2024-03-22T01:36:22.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:22.730+0000 7fb5177fe700 1 -- 172.21.15.72:0/2899648457 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb518003970 con 0x7fb5281026b0 2024-03-22T01:36:22.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:22.730+0000 7fb52ccea700 1 -- 172.21.15.72:0/2899648457 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5281a0d50 con 0x7fb5281026b0 2024-03-22T01:36:22.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:22.731+0000 7fb52ccea700 1 -- 172.21.15.72:0/2899648457 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb528069340 con 0x7fb5281026b0 2024-03-22T01:36:22.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:22.733+0000 7fb5177fe700 1 -- 172.21.15.72:0/2899648457 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7fb51801a070 con 0x7fb5281026b0 2024-03-22T01:36:22.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:22.734+0000 7fb5177fe700 1 --2- 172.21.15.72:0/2899648457 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fb5100a2fb0 0x7fb5100a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:22.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:22.734+0000 7fb525d9b700 1 --2- 172.21.15.72:0/2899648457 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fb5100a2fb0 0x7fb5100a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:22.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:22.734+0000 7fb5177fe700 1 -- 172.21.15.72:0/2899648457 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7fb5180cf3b0 con 0x7fb5281026b0 2024-03-22T01:36:22.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:22.734+0000 7fb525d9b700 1 --2- 172.21.15.72:0/2899648457 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fb5100a2fb0 0x7fb5100a5470 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb51c003760 tx=0x7fb51c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:22.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:22.737+0000 7fb5177fe700 1 -- 172.21.15.72:0/2899648457 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb51800e350 con 0x7fb5281026b0 2024-03-22T01:36:22.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:22.978+0000 7fb52ccea700 1 -- 172.21.15.72:0/2899648457 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7fb52802ddf0 con 0x7fb5281026b0 2024-03-22T01:36:22.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:22.979+0000 7fb5177fe700 1 -- 172.21.15.72:0/2899648457 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 21 v21) ==== 76+0+2994 (secure 0 0 0) 0x7fb51809be10 con 0x7fb5281026b0 2024-03-22T01:36:22.979 INFO:teuthology.orchestra.run.smithi072.stdout:e21 2024-03-22T01:36:22.979 INFO:teuthology.orchestra.run.smithi072.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-22T01:36:22.979 INFO:teuthology.orchestra.run.smithi072.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-22T01:36:22.980 INFO:teuthology.orchestra.run.smithi072.stdout:legacy client fscid: 1 2024-03-22T01:36:22.980 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:36:22.980 INFO:teuthology.orchestra.run.smithi072.stdout:Filesystem 'cephfs' (1) 2024-03-22T01:36:22.980 INFO:teuthology.orchestra.run.smithi072.stdout:fs_name cephfs 2024-03-22T01:36:22.980 INFO:teuthology.orchestra.run.smithi072.stdout:epoch 21 2024-03-22T01:36:22.980 INFO:teuthology.orchestra.run.smithi072.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-22T01:36:22.980 INFO:teuthology.orchestra.run.smithi072.stdout:created 2024-03-22T01:34:14.815677+0000 2024-03-22T01:36:22.980 INFO:teuthology.orchestra.run.smithi072.stdout:modified 2024-03-22T01:34:39.842959+0000 2024-03-22T01:36:22.980 INFO:teuthology.orchestra.run.smithi072.stdout:tableserver 0 2024-03-22T01:36:22.980 INFO:teuthology.orchestra.run.smithi072.stdout:root 0 2024-03-22T01:36:22.980 INFO:teuthology.orchestra.run.smithi072.stdout:session_timeout 300 2024-03-22T01:36:22.980 INFO:teuthology.orchestra.run.smithi072.stdout:session_autoclose 300 2024-03-22T01:36:22.981 INFO:teuthology.orchestra.run.smithi072.stdout:max_file_size 1099511627776 2024-03-22T01:36:22.981 INFO:teuthology.orchestra.run.smithi072.stdout:max_xattr_size 65536 2024-03-22T01:36:22.981 INFO:teuthology.orchestra.run.smithi072.stdout:required_client_features {} 2024-03-22T01:36:22.981 INFO:teuthology.orchestra.run.smithi072.stdout:last_failure 0 2024-03-22T01:36:22.981 INFO:teuthology.orchestra.run.smithi072.stdout:last_failure_osd_epoch 0 2024-03-22T01:36:22.981 INFO:teuthology.orchestra.run.smithi072.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-22T01:36:22.981 INFO:teuthology.orchestra.run.smithi072.stdout:max_mds 1 2024-03-22T01:36:22.981 INFO:teuthology.orchestra.run.smithi072.stdout:in 0 2024-03-22T01:36:22.981 INFO:teuthology.orchestra.run.smithi072.stdout:up {0=24524} 2024-03-22T01:36:22.981 INFO:teuthology.orchestra.run.smithi072.stdout:failed 2024-03-22T01:36:22.981 INFO:teuthology.orchestra.run.smithi072.stdout:damaged 2024-03-22T01:36:22.982 INFO:teuthology.orchestra.run.smithi072.stdout:stopped 2024-03-22T01:36:22.982 INFO:teuthology.orchestra.run.smithi072.stdout:data_pools [3,4] 2024-03-22T01:36:22.982 INFO:teuthology.orchestra.run.smithi072.stdout:metadata_pool 2 2024-03-22T01:36:22.982 INFO:teuthology.orchestra.run.smithi072.stdout:inline_data disabled 2024-03-22T01:36:22.982 INFO:teuthology.orchestra.run.smithi072.stdout:balancer 2024-03-22T01:36:22.982 INFO:teuthology.orchestra.run.smithi072.stdout:bal_rank_mask -1 2024-03-22T01:36:22.982 INFO:teuthology.orchestra.run.smithi072.stdout:standby_count_wanted 0 2024-03-22T01:36:22.982 INFO:teuthology.orchestra.run.smithi072.stdout:qdb_cluster leader: 24524 members: 24524 2024-03-22T01:36:22.982 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.b{0:24524} state up:active seq 3 addr [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T01:36:22.982 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.i{0:24517} state up:standby-replay seq 1 addr [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T01:36:22.982 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:36:22.982 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:36:22.982 INFO:teuthology.orchestra.run.smithi072.stdout:Standby daemons: 2024-03-22T01:36:22.983 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:36:22.983 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.j{-1:14532} state up:standby seq 1 addr [v2:172.21.15.72:6834/3077216308,v1:172.21.15.72:6835/3077216308] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T01:36:22.983 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.d{-1:14544} state up:standby seq 1 addr [v2:172.21.15.72:6836/2460254143,v1:172.21.15.72:6837/2460254143] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T01:36:22.983 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.g{-1:14613} state up:standby seq 1 addr [v2:172.21.15.72:6838/2026432819,v1:172.21.15.72:6839/2026432819] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T01:36:22.983 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.a{-1:14625} state up:standby seq 1 addr [v2:172.21.15.72:6840/400094545,v1:172.21.15.72:6841/400094545] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T01:36:22.983 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.k{-1:24425} state up:standby seq 1 addr [v2:172.21.15.89:6832/1578701171,v1:172.21.15.89:6833/1578701171] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T01:36:22.983 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.l{-1:24436} state up:standby seq 1 addr [v2:172.21.15.98:6832/2703047213,v1:172.21.15.98:6833/2703047213] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T01:36:22.983 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.e{-1:24452} state up:standby seq 1 addr [v2:172.21.15.89:6834/2589468787,v1:172.21.15.89:6835/2589468787] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T01:36:22.983 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.f{-1:24478} state up:standby seq 1 addr [v2:172.21.15.98:6834/92657636,v1:172.21.15.98:6835/92657636] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T01:36:22.983 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.h{-1:24488} state up:standby seq 1 addr [v2:172.21.15.89:6836/1662941782,v1:172.21.15.89:6837/1662941782] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T01:36:22.983 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.c{-1:24490} state up:standby seq 1 addr [v2:172.21.15.98:6836/781860371,v1:172.21.15.98:6837/781860371] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T01:36:22.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:22.981+0000 7fb52ccea700 1 -- 172.21.15.72:0/2899648457 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fb5100a2fb0 msgr2=0x7fb5100a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:22.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:22.981+0000 7fb52ccea700 1 --2- 172.21.15.72:0/2899648457 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fb5100a2fb0 0x7fb5100a5470 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb51c003760 tx=0x7fb51c005fb0 comp rx=0 tx=0).stop 2024-03-22T01:36:22.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:22.981+0000 7fb52ccea700 1 -- 172.21.15.72:0/2899648457 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb5281026b0 msgr2=0x7fb5281988c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:22.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:22.981+0000 7fb52ccea700 1 --2- 172.21.15.72:0/2899648457 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb5281026b0 0x7fb5281988c0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fb51800d9b0 tx=0x7fb51800dcc0 comp rx=0 tx=0).stop 2024-03-22T01:36:22.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:22.981+0000 7fb52ccea700 1 -- 172.21.15.72:0/2899648457 shutdown_connections 2024-03-22T01:36:22.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:22.981+0000 7fb52ccea700 1 --2- 172.21.15.72:0/2899648457 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fb5100a2fb0 0x7fb5100a5470 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:22.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:22.981+0000 7fb52ccea700 1 --2- 172.21.15.72:0/2899648457 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb528104250 0x7fb52819fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:22.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:22.981+0000 7fb52ccea700 1 --2- 172.21.15.72:0/2899648457 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb5281026b0 0x7fb5281988c0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:22.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:22.981+0000 7fb52ccea700 1 --2- 172.21.15.72:0/2899648457 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5281038b0 0x7fb528198e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:22.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:22.981+0000 7fb52ccea700 1 -- 172.21.15.72:0/2899648457 >> 172.21.15.72:0/2899648457 conn(0x7fb5280fdc90 msgr2=0x7fb528106c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:22.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:22.981+0000 7fb52ccea700 1 -- 172.21.15.72:0/2899648457 shutdown_connections 2024-03-22T01:36:22.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:22.981+0000 7fb52ccea700 1 -- 172.21.15.72:0/2899648457 wait complete. 2024-03-22T01:36:22.985 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 21 2024-03-22T01:36:23.357 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:23 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/2899648457' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T01:36:23.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:23 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/2899648457' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T01:36:23.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:23 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/2899648457' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T01:36:23.693 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- bash -c 'ceph osd dump' 2024-03-22T01:36:23.932 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:36:24.326 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:24 smithi072 ceph-mon[32414]: pgmap v359: 129 pgs: 129 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 1 op/s 2024-03-22T01:36:24.357 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:24 smithi089 ceph-mon[35770]: pgmap v359: 129 pgs: 129 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 1 op/s 2024-03-22T01:36:24.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:24 smithi098 ceph-mon[36468]: pgmap v359: 129 pgs: 129 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 1 op/s 2024-03-22T01:36:25.093 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:25 smithi072 ceph-mon[32414]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-22T01:36:25.094 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:25 smithi072 ceph-mon[32414]: Cluster is now healthy 2024-03-22T01:36:25.357 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:25 smithi089 ceph-mon[35770]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-22T01:36:25.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:25 smithi089 ceph-mon[35770]: Cluster is now healthy 2024-03-22T01:36:25.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:25 smithi098 ceph-mon[36468]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-22T01:36:25.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:25 smithi098 ceph-mon[36468]: Cluster is now healthy 2024-03-22T01:36:25.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:25.636+0000 7f2a9c043700 1 -- 172.21.15.72:0/1747450475 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a941015b0 msgr2=0x7f2a941019d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:25.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:25.636+0000 7f2a9c043700 1 --2- 172.21.15.72:0/1747450475 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a941015b0 0x7f2a941019d0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f2a900099c0 tx=0x7f2a90009cd0 comp rx=0 tx=0).stop 2024-03-22T01:36:25.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:25.636+0000 7f2a9c043700 1 -- 172.21.15.72:0/1747450475 shutdown_connections 2024-03-22T01:36:25.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:25.636+0000 7f2a9c043700 1 --2- 172.21.15.72:0/1747450475 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a94103150 0x7f2a94109a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:25.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:25.636+0000 7f2a9c043700 1 --2- 172.21.15.72:0/1747450475 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a941015b0 0x7f2a941019d0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:25.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:25.636+0000 7f2a9c043700 1 --2- 172.21.15.72:0/1747450475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a941027b0 0x7f2a94102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:25.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:25.636+0000 7f2a9c043700 1 -- 172.21.15.72:0/1747450475 >> 172.21.15.72:0/1747450475 conn(0x7f2a940fcb50 msgr2=0x7f2a940fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:25.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:25.636+0000 7f2a9c043700 1 -- 172.21.15.72:0/1747450475 shutdown_connections 2024-03-22T01:36:25.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:25.636+0000 7f2a9c043700 1 -- 172.21.15.72:0/1747450475 wait complete. 2024-03-22T01:36:25.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:25.637+0000 7f2a9c043700 1 Processor -- start 2024-03-22T01:36:25.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:25.637+0000 7f2a9c043700 1 -- start start 2024-03-22T01:36:25.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:25.637+0000 7f2a9c043700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a941015b0 0x7f2a9410d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:25.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:25.637+0000 7f2a9c043700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a941027b0 0x7f2a9410bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:25.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:25.637+0000 7f2a9c043700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a94103150 0x7f2a9410c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:25.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:25.637+0000 7f2a9c043700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f2a9410e2e0 con 0x7f2a941015b0 2024-03-22T01:36:25.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:25.637+0000 7f2a9c043700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2a9410e160 con 0x7f2a941027b0 2024-03-22T01:36:25.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:25.637+0000 7f2a9c043700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2a9410e460 con 0x7f2a94103150 2024-03-22T01:36:25.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:25.637+0000 7f2a99ddf700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a941015b0 0x7f2a9410d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:25.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:25.637+0000 7f2a99ddf700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a941015b0 0x7f2a9410d870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:53516/0 (socket says 172.21.15.72:53516) 2024-03-22T01:36:25.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:25.637+0000 7f2a99ddf700 1 -- 172.21.15.72:0/1344723410 learned_addr learned my addr 172.21.15.72:0/1344723410 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:25.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:25.637+0000 7f2a995de700 1 --2- 172.21.15.72:0/1344723410 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a941027b0 0x7f2a9410bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:25.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:25.637+0000 7f2a9a5e0700 1 --2- 172.21.15.72:0/1344723410 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a94103150 0x7f2a9410c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:25.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:25.637+0000 7f2a99ddf700 1 -- 172.21.15.72:0/1344723410 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a94103150 msgr2=0x7f2a9410c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:25.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:25.637+0000 7f2a99ddf700 1 --2- 172.21.15.72:0/1344723410 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a94103150 0x7f2a9410c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:25.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:25.637+0000 7f2a99ddf700 1 -- 172.21.15.72:0/1344723410 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a941027b0 msgr2=0x7f2a9410bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:25.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:25.637+0000 7f2a99ddf700 1 --2- 172.21.15.72:0/1344723410 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a941027b0 0x7f2a9410bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:25.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:25.637+0000 7f2a99ddf700 1 -- 172.21.15.72:0/1344723410 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a9410ccb0 con 0x7f2a941015b0 2024-03-22T01:36:25.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:25.637+0000 7f2a99ddf700 1 --2- 172.21.15.72:0/1344723410 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a941015b0 0x7f2a9410d870 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f2a9000fa80 tx=0x7f2a9000fab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:25.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:25.637+0000 7f2a8affd700 1 -- 172.21.15.72:0/1344723410 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a90027070 con 0x7f2a941015b0 2024-03-22T01:36:25.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:25.638+0000 7f2a9c043700 1 -- 172.21.15.72:0/1344723410 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a941a2a70 con 0x7f2a941015b0 2024-03-22T01:36:25.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:25.638+0000 7f2a8affd700 1 -- 172.21.15.72:0/1344723410 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a9000fdc0 con 0x7f2a941015b0 2024-03-22T01:36:25.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:25.638+0000 7f2a8affd700 1 -- 172.21.15.72:0/1344723410 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a900173f0 con 0x7f2a941015b0 2024-03-22T01:36:25.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:25.638+0000 7f2a9c043700 1 -- 172.21.15.72:0/1344723410 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a941a2f50 con 0x7f2a941015b0 2024-03-22T01:36:25.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:25.639+0000 7f2a9c043700 1 -- 172.21.15.72:0/1344723410 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a94069340 con 0x7f2a941015b0 2024-03-22T01:36:25.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:25.640+0000 7f2a8affd700 1 -- 172.21.15.72:0/1344723410 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7f2a9002a070 con 0x7f2a941015b0 2024-03-22T01:36:25.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:25.640+0000 7f2a8affd700 1 --2- 172.21.15.72:0/1344723410 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f2a800a2fb0 0x7f2a800a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:25.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:25.640+0000 7f2a995de700 1 --2- 172.21.15.72:0/1344723410 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f2a800a2fb0 0x7f2a800a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:25.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:25.640+0000 7f2a8affd700 1 -- 172.21.15.72:0/1344723410 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f2a900d0490 con 0x7f2a941015b0 2024-03-22T01:36:25.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:25.641+0000 7f2a995de700 1 --2- 172.21.15.72:0/1344723410 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f2a800a2fb0 0x7f2a800a5470 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f2a84005950 tx=0x7f2a8400a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:25.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:25.645+0000 7f2a8affd700 1 -- 172.21.15.72:0/1344723410 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2a9005e740 con 0x7f2a941015b0 2024-03-22T01:36:25.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:25.840+0000 7f2a9c043700 1 -- 172.21.15.72:0/1344723410 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f2a94063290 con 0x7f2a941015b0 2024-03-22T01:36:25.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:25.842+0000 7f2a8affd700 1 -- 172.21.15.72:0/1344723410 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v95) ==== 56+0+5678 (secure 0 0 0) 0x7f2a90013070 con 0x7f2a941015b0 2024-03-22T01:36:25.842 INFO:teuthology.orchestra.run.smithi072.stdout:epoch 95 2024-03-22T01:36:25.842 INFO:teuthology.orchestra.run.smithi072.stdout:fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 2024-03-22T01:36:25.842 INFO:teuthology.orchestra.run.smithi072.stdout:created 2024-03-22T01:26:23.893672+0000 2024-03-22T01:36:25.842 INFO:teuthology.orchestra.run.smithi072.stdout:modified 2024-03-22T01:36:17.626966+0000 2024-03-22T01:36:25.842 INFO:teuthology.orchestra.run.smithi072.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-22T01:36:25.842 INFO:teuthology.orchestra.run.smithi072.stdout:crush_version 27 2024-03-22T01:36:25.842 INFO:teuthology.orchestra.run.smithi072.stdout:full_ratio 0.95 2024-03-22T01:36:25.843 INFO:teuthology.orchestra.run.smithi072.stdout:backfillfull_ratio 0.9 2024-03-22T01:36:25.843 INFO:teuthology.orchestra.run.smithi072.stdout:nearfull_ratio 0.85 2024-03-22T01:36:25.843 INFO:teuthology.orchestra.run.smithi072.stdout:require_min_compat_client luminous 2024-03-22T01:36:25.843 INFO:teuthology.orchestra.run.smithi072.stdout:min_compat_client luminous 2024-03-22T01:36:25.843 INFO:teuthology.orchestra.run.smithi072.stdout:require_osd_release squid 2024-03-22T01:36:25.843 INFO:teuthology.orchestra.run.smithi072.stdout:stretch_mode_enabled false 2024-03-22T01:36:25.843 INFO:teuthology.orchestra.run.smithi072.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-03-22T01:36:25.843 INFO:teuthology.orchestra.run.smithi072.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 86 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-03-22T01:36:25.843 INFO:teuthology.orchestra.run.smithi072.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 87 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-03-22T01:36:25.843 INFO:teuthology.orchestra.run.smithi072.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 91 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-03-22T01:36:25.843 INFO:teuthology.orchestra.run.smithi072.stdout:max_osd 12 2024-03-22T01:36:25.843 INFO:teuthology.orchestra.run.smithi072.stdout:osd.0 up in weight 1 up_from 8 up_thru 94 down_at 0 last_clean_interval [0,0) [v2:172.21.15.72:6802/661906791,v1:172.21.15.72:6803/661906791] [v2:172.21.15.72:6804/661906791,v1:172.21.15.72:6805/661906791] exists,up 16c15bb0-db92-4191-9063-38b4890e81e3 2024-03-22T01:36:25.843 INFO:teuthology.orchestra.run.smithi072.stdout:osd.1 up in weight 1 up_from 13 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.89:6800/2870590550,v1:172.21.15.89:6801/2870590550] [v2:172.21.15.89:6802/2870590550,v1:172.21.15.89:6803/2870590550] exists,up 981290ba-97dc-496b-aa22-d51704184b10 2024-03-22T01:36:25.843 INFO:teuthology.orchestra.run.smithi072.stdout:osd.2 up in weight 1 up_from 18 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.98:6800/3865778118,v1:172.21.15.98:6801/3865778118] [v2:172.21.15.98:6802/3865778118,v1:172.21.15.98:6803/3865778118] exists,up f07ed658-f545-46a1-8edd-8b1d5d49885c 2024-03-22T01:36:25.844 INFO:teuthology.orchestra.run.smithi072.stdout:osd.3 up in weight 1 up_from 26 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.72:6810/1865951395,v1:172.21.15.72:6811/1865951395] [v2:172.21.15.72:6812/1865951395,v1:172.21.15.72:6813/1865951395] exists,up ccc4bc40-2c48-46d2-8548-0c05b44f9f1a 2024-03-22T01:36:25.844 INFO:teuthology.orchestra.run.smithi072.stdout:osd.4 up in weight 1 up_from 34 up_thru 94 down_at 0 last_clean_interval [0,0) [v2:172.21.15.89:6808/2683760795,v1:172.21.15.89:6809/2683760795] [v2:172.21.15.89:6810/2683760795,v1:172.21.15.89:6811/2683760795] exists,up 1812439f-810a-4ed2-9395-c00c9a085a13 2024-03-22T01:36:25.844 INFO:teuthology.orchestra.run.smithi072.stdout:osd.5 up in weight 1 up_from 39 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.98:6808/2635838572,v1:172.21.15.98:6809/2635838572] [v2:172.21.15.98:6810/2635838572,v1:172.21.15.98:6811/2635838572] exists,up e78954d8-f4e3-4d58-8536-d2d59c843386 2024-03-22T01:36:25.844 INFO:teuthology.orchestra.run.smithi072.stdout:osd.6 up in weight 1 up_from 47 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.72:6818/3158982661,v1:172.21.15.72:6819/3158982661] [v2:172.21.15.72:6820/3158982661,v1:172.21.15.72:6821/3158982661] exists,up dc6c8be2-b64c-4959-8015-7b0ea6d25599 2024-03-22T01:36:25.844 INFO:teuthology.orchestra.run.smithi072.stdout:osd.7 up in weight 1 up_from 52 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.89:6816/3075913234,v1:172.21.15.89:6817/3075913234] [v2:172.21.15.89:6818/3075913234,v1:172.21.15.89:6819/3075913234] exists,up 7a73469f-cf3a-4306-ae93-ea46e21ebbbe 2024-03-22T01:36:25.844 INFO:teuthology.orchestra.run.smithi072.stdout:osd.8 up in weight 1 up_from 60 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.98:6816/1813255577,v1:172.21.15.98:6817/1813255577] [v2:172.21.15.98:6818/1813255577,v1:172.21.15.98:6819/1813255577] exists,up c4b13bee-ea4d-4344-95fb-2622ecbb8196 2024-03-22T01:36:25.844 INFO:teuthology.orchestra.run.smithi072.stdout:osd.9 up in weight 1 up_from 68 up_thru 94 down_at 0 last_clean_interval [0,0) [v2:172.21.15.72:6826/3377749206,v1:172.21.15.72:6827/3377749206] [v2:172.21.15.72:6828/3377749206,v1:172.21.15.72:6829/3377749206] exists,up eefb0a44-4a36-4e55-9d04-0501bf718db6 2024-03-22T01:36:25.844 INFO:teuthology.orchestra.run.smithi072.stdout:osd.10 up in weight 1 up_from 74 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.89:6824/885058786,v1:172.21.15.89:6825/885058786] [v2:172.21.15.89:6826/885058786,v1:172.21.15.89:6827/885058786] exists,up 8741d25f-40dd-40d7-b040-5cc5e411eb98 2024-03-22T01:36:25.844 INFO:teuthology.orchestra.run.smithi072.stdout:osd.11 up in weight 1 up_from 82 up_thru 94 down_at 0 last_clean_interval [0,0) [v2:172.21.15.98:6824/1517095838,v1:172.21.15.98:6825/1517095838] [v2:172.21.15.98:6826/1517095838,v1:172.21.15.98:6827/1517095838] exists,up 8d02e957-eda4-4184-b5c6-5eb3b8b38060 2024-03-22T01:36:25.844 INFO:teuthology.orchestra.run.smithi072.stdout:pg_upmap_items 2.5 [8,11] 2024-03-22T01:36:25.844 INFO:teuthology.orchestra.run.smithi072.stdout:pg_upmap_items 2.c [2,1] 2024-03-22T01:36:25.844 INFO:teuthology.orchestra.run.smithi072.stdout:pg_upmap_items 2.1c [8,11] 2024-03-22T01:36:25.844 INFO:teuthology.orchestra.run.smithi072.stdout:pg_upmap_items 2.1d [2,4] 2024-03-22T01:36:25.844 INFO:teuthology.orchestra.run.smithi072.stdout:pg_upmap_items 3.7 [3,9] 2024-03-22T01:36:25.844 INFO:teuthology.orchestra.run.smithi072.stdout:pg_upmap_items 3.c [3,9] 2024-03-22T01:36:25.845 INFO:teuthology.orchestra.run.smithi072.stdout:pg_upmap_items 3.e [4,8] 2024-03-22T01:36:25.845 INFO:teuthology.orchestra.run.smithi072.stdout:pg_upmap_items 3.10 [5,9] 2024-03-22T01:36:25.845 INFO:teuthology.orchestra.run.smithi072.stdout:pg_upmap_items 3.17 [4,6] 2024-03-22T01:36:25.845 INFO:teuthology.orchestra.run.smithi072.stdout:pg_upmap_items 3.1b [4,7] 2024-03-22T01:36:25.845 INFO:teuthology.orchestra.run.smithi072.stdout:pg_upmap_items 4.b [9,8] 2024-03-22T01:36:25.845 INFO:teuthology.orchestra.run.smithi072.stdout:pg_upmap_items 4.12 [5,10] 2024-03-22T01:36:25.845 INFO:teuthology.orchestra.run.smithi072.stdout:pg_upmap_items 4.13 [9,8] 2024-03-22T01:36:25.845 INFO:teuthology.orchestra.run.smithi072.stdout:pg_upmap_items 4.1f [9,10] 2024-03-22T01:36:25.845 INFO:teuthology.orchestra.run.smithi072.stdout:blocklist 172.21.15.72:0/121087196 expires 2024-03-23T01:27:15.339809+0000 2024-03-22T01:36:25.845 INFO:teuthology.orchestra.run.smithi072.stdout:blocklist 172.21.15.72:0/2451857761 expires 2024-03-23T01:27:15.339809+0000 2024-03-22T01:36:25.845 INFO:teuthology.orchestra.run.smithi072.stdout:blocklist 172.21.15.72:0/1228477995 expires 2024-03-23T01:27:15.339809+0000 2024-03-22T01:36:25.845 INFO:teuthology.orchestra.run.smithi072.stdout:blocklist 172.21.15.72:6801/1513232913 expires 2024-03-23T01:27:15.339809+0000 2024-03-22T01:36:25.845 INFO:teuthology.orchestra.run.smithi072.stdout:blocklist 172.21.15.72:0/3558958264 expires 2024-03-23T01:26:50.074871+0000 2024-03-22T01:36:25.845 INFO:teuthology.orchestra.run.smithi072.stdout:blocklist 172.21.15.72:0/1510187198 expires 2024-03-23T01:26:50.074871+0000 2024-03-22T01:36:25.845 INFO:teuthology.orchestra.run.smithi072.stdout:blocklist 172.21.15.72:6800/3518012916 expires 2024-03-23T01:26:50.074871+0000 2024-03-22T01:36:25.846 INFO:teuthology.orchestra.run.smithi072.stdout:blocklist 172.21.15.72:6801/3518012916 expires 2024-03-23T01:26:50.074871+0000 2024-03-22T01:36:25.846 INFO:teuthology.orchestra.run.smithi072.stdout:blocklist 172.21.15.72:0/1372801041 expires 2024-03-23T01:26:50.074871+0000 2024-03-22T01:36:25.846 INFO:teuthology.orchestra.run.smithi072.stdout:blocklist 172.21.15.72:6800/1513232913 expires 2024-03-23T01:27:15.339809+0000 2024-03-22T01:36:25.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:25.844+0000 7f2a9c043700 1 -- 172.21.15.72:0/1344723410 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f2a800a2fb0 msgr2=0x7f2a800a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:25.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:25.844+0000 7f2a9c043700 1 --2- 172.21.15.72:0/1344723410 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f2a800a2fb0 0x7f2a800a5470 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f2a84005950 tx=0x7f2a8400a300 comp rx=0 tx=0).stop 2024-03-22T01:36:25.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:25.844+0000 7f2a9c043700 1 -- 172.21.15.72:0/1344723410 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a941015b0 msgr2=0x7f2a9410d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:25.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:25.844+0000 7f2a9c043700 1 --2- 172.21.15.72:0/1344723410 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a941015b0 0x7f2a9410d870 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f2a9000fa80 tx=0x7f2a9000fab0 comp rx=0 tx=0).stop 2024-03-22T01:36:25.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:25.844+0000 7f2a9c043700 1 -- 172.21.15.72:0/1344723410 shutdown_connections 2024-03-22T01:36:25.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:25.844+0000 7f2a9c043700 1 --2- 172.21.15.72:0/1344723410 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f2a800a2fb0 0x7f2a800a5470 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:25.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:25.844+0000 7f2a9c043700 1 --2- 172.21.15.72:0/1344723410 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a94103150 0x7f2a9410c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:25.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:25.844+0000 7f2a9c043700 1 --2- 172.21.15.72:0/1344723410 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a941015b0 0x7f2a9410d870 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:25.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:25.844+0000 7f2a9c043700 1 --2- 172.21.15.72:0/1344723410 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a941027b0 0x7f2a9410bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:25.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:25.844+0000 7f2a9c043700 1 -- 172.21.15.72:0/1344723410 >> 172.21.15.72:0/1344723410 conn(0x7f2a940fcb50 msgr2=0x7f2a94105b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:25.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:25.844+0000 7f2a9c043700 1 -- 172.21.15.72:0/1344723410 shutdown_connections 2024-03-22T01:36:25.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:25.844+0000 7f2a9c043700 1 -- 172.21.15.72:0/1344723410 wait complete. 2024-03-22T01:36:26.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:26 smithi072 ceph-mon[32414]: pgmap v360: 129 pgs: 129 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-22T01:36:26.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:26 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/1344723410' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-22T01:36:26.323 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:26 smithi098 ceph-mon[36468]: pgmap v360: 129 pgs: 129 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-22T01:36:26.323 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:26 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/1344723410' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-22T01:36:26.357 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:26 smithi089 ceph-mon[35770]: pgmap v360: 129 pgs: 129 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-22T01:36:26.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:26 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/1344723410' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-22T01:36:26.533 INFO:teuthology.run_tasks:Running task fs.ready... 2024-03-22T01:36:26.545 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T01:36:26.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.651+0000 7fcc61d8a640 1 -- 172.21.15.72:0/1200715674 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcc5c102e10 msgr2=0x7fcc5c103290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:26.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.651+0000 7fcc61d8a640 1 --2- 172.21.15.72:0/1200715674 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcc5c102e10 0x7fcc5c103290 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fcc50009900 tx=0x7fcc5002bc30 comp rx=0 tx=0).stop 2024-03-22T01:36:26.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.651+0000 7fcc61d8a640 1 -- 172.21.15.72:0/1200715674 shutdown_connections 2024-03-22T01:36:26.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.651+0000 7fcc61d8a640 1 --2- 172.21.15.72:0/1200715674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc5c1037d0 0x7fcc5c10a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:26.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.651+0000 7fcc61d8a640 1 --2- 172.21.15.72:0/1200715674 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcc5c102e10 0x7fcc5c103290 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:26.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.651+0000 7fcc61d8a640 1 --2- 172.21.15.72:0/1200715674 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcc5c101c20 0x7fcc5c102040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:26.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.651+0000 7fcc61d8a640 1 -- 172.21.15.72:0/1200715674 >> 172.21.15.72:0/1200715674 conn(0x7fcc5c0fd1b0 msgr2=0x7fcc5c0ff5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:26.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.651+0000 7fcc61d8a640 1 -- 172.21.15.72:0/1200715674 shutdown_connections 2024-03-22T01:36:26.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.651+0000 7fcc61d8a640 1 -- 172.21.15.72:0/1200715674 wait complete. 2024-03-22T01:36:26.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.652+0000 7fcc61d8a640 1 Processor -- start 2024-03-22T01:36:26.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.652+0000 7fcc61d8a640 1 -- start start 2024-03-22T01:36:26.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.652+0000 7fcc61d8a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc5c101c20 0x7fcc5c197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:26.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.652+0000 7fcc61d8a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcc5c102e10 0x7fcc5c198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:26.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.652+0000 7fcc61d8a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcc5c1037d0 0x7fcc5c19f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:26.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.652+0000 7fcc61d8a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fcc5c10d890 con 0x7fcc5c102e10 2024-03-22T01:36:26.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.652+0000 7fcc61d8a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fcc5c10d710 con 0x7fcc5c1037d0 2024-03-22T01:36:26.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.652+0000 7fcc61d8a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcc5c10da10 con 0x7fcc5c101c20 2024-03-22T01:36:26.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.652+0000 7fcc5affd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcc5c102e10 0x7fcc5c198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:26.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.652+0000 7fcc5affd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcc5c102e10 0x7fcc5c198470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:53540/0 (socket says 172.21.15.72:53540) 2024-03-22T01:36:26.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.652+0000 7fcc5affd640 1 -- 172.21.15.72:0/3018799237 learned_addr learned my addr 172.21.15.72:0/3018799237 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:26.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.652+0000 7fcc5bfff640 1 --2- 172.21.15.72:0/3018799237 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcc5c1037d0 0x7fcc5c19f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:26.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.652+0000 7fcc5b7fe640 1 --2- 172.21.15.72:0/3018799237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc5c101c20 0x7fcc5c197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:26.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.652+0000 7fcc5affd640 1 -- 172.21.15.72:0/3018799237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc5c101c20 msgr2=0x7fcc5c197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:26.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.652+0000 7fcc5affd640 1 --2- 172.21.15.72:0/3018799237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc5c101c20 0x7fcc5c197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:26.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.652+0000 7fcc5affd640 1 -- 172.21.15.72:0/3018799237 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcc5c1037d0 msgr2=0x7fcc5c19f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:26.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.652+0000 7fcc5affd640 1 --2- 172.21.15.72:0/3018799237 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcc5c1037d0 0x7fcc5c19f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:26.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.652+0000 7fcc5affd640 1 -- 172.21.15.72:0/3018799237 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc5c19fb50 con 0x7fcc5c102e10 2024-03-22T01:36:26.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.652+0000 7fcc5affd640 1 --2- 172.21.15.72:0/3018799237 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcc5c102e10 0x7fcc5c198470 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fcc50009230 tx=0x7fcc50005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:26.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.653+0000 7fcc58ff9640 1 -- 172.21.15.72:0/3018799237 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc50041070 con 0x7fcc5c102e10 2024-03-22T01:36:26.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.653+0000 7fcc58ff9640 1 -- 172.21.15.72:0/3018799237 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc500044d0 con 0x7fcc5c102e10 2024-03-22T01:36:26.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.653+0000 7fcc61d8a640 1 -- 172.21.15.72:0/3018799237 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc5c19fde0 con 0x7fcc5c102e10 2024-03-22T01:36:26.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.653+0000 7fcc58ff9640 1 -- 172.21.15.72:0/3018799237 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc50034440 con 0x7fcc5c102e10 2024-03-22T01:36:26.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.653+0000 7fcc61d8a640 1 -- 172.21.15.72:0/3018799237 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc5c1a0270 con 0x7fcc5c102e10 2024-03-22T01:36:26.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.654+0000 7fcc61d8a640 1 -- 172.21.15.72:0/3018799237 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc5c108f40 con 0x7fcc5c102e10 2024-03-22T01:36:26.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.655+0000 7fcc58ff9640 1 -- 172.21.15.72:0/3018799237 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7fcc50044070 con 0x7fcc5c102e10 2024-03-22T01:36:26.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.655+0000 7fcc58ff9640 1 --2- 172.21.15.72:0/3018799237 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fcc440a2f20 0x7fcc440a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:26.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.655+0000 7fcc5b7fe640 1 --2- 172.21.15.72:0/3018799237 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fcc440a2f20 0x7fcc440a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:26.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.655+0000 7fcc58ff9640 1 -- 172.21.15.72:0/3018799237 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7fcc500f1490 con 0x7fcc5c102e10 2024-03-22T01:36:26.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.656+0000 7fcc5b7fe640 1 --2- 172.21.15.72:0/3018799237 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fcc440a2f20 0x7fcc440a53e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c102c70 tx=0x7fcc4c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:26.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.660+0000 7fcc58ff9640 1 -- 172.21.15.72:0/3018799237 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcc500bdef0 con 0x7fcc5c102e10 2024-03-22T01:36:26.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.838+0000 7fcc61d8a640 1 -- 172.21.15.72:0/3018799237 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcc5c062830 con 0x7fcc5c102e10 2024-03-22T01:36:26.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.839+0000 7fcc58ff9640 1 -- 172.21.15.72:0/3018799237 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) ==== 74+0+24759 (secure 0 0 0) 0x7fcc500bd7f0 con 0x7fcc5c102e10 2024-03-22T01:36:26.839 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:36:26.839 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":95,"fsid":"022a99bc-e7eb-11ee-95cd-87774f69a715","created":"2024-03-22T01:26:23.893672+0000","modified":"2024-03-22T01:36:17.626966+0000","last_up_change":"2024-03-22T01:33:25.223684+0000","last_in_change":"2024-03-22T01:33:05.010331+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T01:29:37.438503+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-22T01:34:12.610422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-22T01:34:13.761686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-22T01:34:17.507039+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"16c15bb0-db92-4191-9063-38b4890e81e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6803","nonce":661906791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6805","nonce":661906791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6809","nonce":661906791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6807","nonce":661906791}]},"public_addr":"172.21.15.72:6803/661906791","cluster_addr":"172.21.15.72:6805/661906791","heartbeat_back_addr":"172.21.15.72:6809/661906791","heartbeat_front_addr":"172.21.15.72:6807/661906791","state":["exists","up"]},{"osd":1,"uuid":"981290ba-97dc-496b-aa22-d51704184b10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6801","nonce":2870590550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6803","nonce":2870590550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6807","nonce":2870590550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6805","nonce":2870590550}]},"public_addr":"172.21.15.89:6801/2870590550","cluster_addr":"172.21.15.89:6803/2870590550","heartbeat_back_addr":"172.21.15.89:6807/2870590550","heartbeat_front_addr":"172.21.15.89:6805/2870590550","state":["exists","up"]},{"osd":2,"uuid":"f07ed658-f545-46a1-8edd-8b1d5d49885c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6801","nonce":3865778118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6803","nonce":3865778118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6807","nonce":3865778118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6805","nonce":3865778118}]},"public_addr":"172.21.15.98:6801/3865778118","cluster_addr":"172.21.15.98:6803/3865778118","heartbeat_back_addr":"172.21.15.98:6807/3865778118","heartbeat_front_addr":"172.21.15.98:6805/3865778118","state":["exists","up"]},{"osd":3,"uuid":"ccc4bc40-2c48-46d2-8548-0c05b44f9f1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6811","nonce":1865951395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6813","nonce":1865951395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6817","nonce":1865951395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6815","nonce":1865951395}]},"public_addr":"172.21.15.72:6811/1865951395","cluster_addr":"172.21.15.72:6813/1865951395","heartbeat_back_addr":"172.21.15.72:6817/1865951395","heartbeat_front_addr":"172.21.15.72:6815/1865951395","state":["exists","up"]},{"osd":4,"uuid":"1812439f-810a-4ed2-9395-c00c9a085a13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6809","nonce":2683760795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6811","nonce":2683760795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6815","nonce":2683760795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6813","nonce":2683760795}]},"public_addr":"172.21.15.89:6809/2683760795","cluster_addr":"172.21.15.89:6811/2683760795","heartbeat_back_addr":"172.21.15.89:6815/2683760795","heartbeat_front_addr":"172.21.15.89:6813/2683760795","state":["exists","up"]},{"osd":5,"uuid":"e78954d8-f4e3-4d58-8536-d2d59c843386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6809","nonce":2635838572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6811","nonce":2635838572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6815","nonce":2635838572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6813","nonce":2635838572}]},"public_addr":"172.21.15.98:6809/2635838572","cluster_addr":"172.21.15.98:6811/2635838572","heartbeat_back_addr":"172.21.15.98:6815/2635838572","heartbeat_front_addr":"172.21.15.98:6813/2635838572","state":["exists","up"]},{"osd":6,"uuid":"dc6c8be2-b64c-4959-8015-7b0ea6d25599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6819","nonce":3158982661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6821","nonce":3158982661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6825","nonce":3158982661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6823","nonce":3158982661}]},"public_addr":"172.21.15.72:6819/3158982661","cluster_addr":"172.21.15.72:6821/3158982661","heartbeat_back_addr":"172.21.15.72:6825/3158982661","heartbeat_front_addr":"172.21.15.72:6823/3158982661","state":["exists","up"]},{"osd":7,"uuid":"7a73469f-cf3a-4306-ae93-ea46e21ebbbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6817","nonce":3075913234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6819","nonce":3075913234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6823","nonce":3075913234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6821","nonce":3075913234}]},"public_addr":"172.21.15.89:6817/3075913234","cluster_addr":"172.21.15.89:6819/3075913234","heartbeat_back_addr":"172.21.15.89:6823/3075913234","heartbeat_front_addr":"172.21.15.89:6821/3075913234","state":["exists","up"]},{"osd":8,"uuid":"c4b13bee-ea4d-4344-95fb-2622ecbb8196","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6817","nonce":1813255577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6819","nonce":1813255577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6823","nonce":1813255577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6821","nonce":1813255577}]},"public_addr":"172.21.15.98:6817/1813255577","cluster_addr":"172.21.15.98:6819/1813255577","heartbeat_back_addr":"172.21.15.98:6823/1813255577","heartbeat_front_addr":"172.21.15.98:6821/1813255577","state":["exists","up"]},{"osd":9,"uuid":"eefb0a44-4a36-4e55-9d04-0501bf718db6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6827","nonce":3377749206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6829","nonce":3377749206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6832","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6833","nonce":3377749206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6831","nonce":3377749206}]},"public_addr":"172.21.15.72:6827/3377749206","cluster_addr":"172.21.15.72:6829/3377749206","heartbeat_back_addr":"172.21.15.72:6833/3377749206","heartbeat_front_addr":"172.21.15.72:6831/3377749206","state":["exists","up"]},{"osd":10,"uuid":"8741d25f-40dd-40d7-b040-5cc5e411eb98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6825","nonce":885058786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6827","nonce":885058786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6831","nonce":885058786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6829","nonce":885058786}]},"public_addr":"172.21.15.89:6825/885058786","cluster_addr":"172.21.15.89:6827/885058786","heartbeat_back_addr":"172.21.15.89:6831/885058786","heartbeat_front_addr":"172.21.15.89:6829/885058786","state":["exists","up"]},{"osd":11,"uuid":"8d02e957-eda4-4184-b5c6-5eb3b8b38060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6825","nonce":1517095838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6827","nonce":1517095838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6831","nonce":1517095838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6829","nonce":1517095838}]},"public_addr":"172.21.15.98:6825/1517095838","cluster_addr":"172.21.15.98:6827/1517095838","heartbeat_back_addr":"172.21.15.98:6831/1517095838","heartbeat_front_addr":"172.21.15.98:6829/1517095838","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:28:47.307652+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:11.037135+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:34.619188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:59.350738+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:30:23.787337+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:30:48.994154+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:31:13.690350+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:31:39.628116+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:04.723912+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:30.689741+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:57.229508+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:33:23.154537+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":7}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:0/121087196":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/2451857761":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/1228477995":"2024-03-23T01:27:15.339809+0000","172.21.15.72:6801/1513232913":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/3558958264":"2024-03-23T01:26:50.074871+0000","172.21.15.72:0/1510187198":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6800/3518012916":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6801/3518012916":"2024-03-23T01:26:50.074871+0000","172.21.15.72:0/1372801041":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6800/1513232913":"2024-03-23T01:27:15.339809+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T01:36:26.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.841+0000 7fcc61d8a640 1 -- 172.21.15.72:0/3018799237 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fcc440a2f20 msgr2=0x7fcc440a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:26.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.841+0000 7fcc61d8a640 1 --2- 172.21.15.72:0/3018799237 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fcc440a2f20 0x7fcc440a53e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c102c70 tx=0x7fcc4c005f90 comp rx=0 tx=0).stop 2024-03-22T01:36:26.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.841+0000 7fcc61d8a640 1 -- 172.21.15.72:0/3018799237 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcc5c102e10 msgr2=0x7fcc5c198470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:26.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.841+0000 7fcc61d8a640 1 --2- 172.21.15.72:0/3018799237 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcc5c102e10 0x7fcc5c198470 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fcc50009230 tx=0x7fcc50005d50 comp rx=0 tx=0).stop 2024-03-22T01:36:26.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.841+0000 7fcc61d8a640 1 -- 172.21.15.72:0/3018799237 shutdown_connections 2024-03-22T01:36:26.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.841+0000 7fcc61d8a640 1 --2- 172.21.15.72:0/3018799237 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fcc440a2f20 0x7fcc440a53e0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:26.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.841+0000 7fcc61d8a640 1 --2- 172.21.15.72:0/3018799237 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcc5c1037d0 0x7fcc5c19f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:26.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.841+0000 7fcc61d8a640 1 --2- 172.21.15.72:0/3018799237 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcc5c102e10 0x7fcc5c198470 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:26.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.841+0000 7fcc61d8a640 1 --2- 172.21.15.72:0/3018799237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc5c101c20 0x7fcc5c197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:26.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.841+0000 7fcc61d8a640 1 -- 172.21.15.72:0/3018799237 >> 172.21.15.72:0/3018799237 conn(0x7fcc5c0fd1b0 msgr2=0x7fcc5c106110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:26.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.842+0000 7fcc61d8a640 1 -- 172.21.15.72:0/3018799237 shutdown_connections 2024-03-22T01:36:26.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.842+0000 7fcc61d8a640 1 -- 172.21.15.72:0/3018799237 wait complete. 2024-03-22T01:36:26.852 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-22T01:29:37.438503+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-22T01:34:12.610422+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-22T01:34:13.761686+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-22T01:34:17.507039+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-22T01:36:26.853 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-22T01:36:26.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.959+0000 7f130ad63640 1 -- 172.21.15.72:0/2147118179 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1304103740 msgr2=0x7f1304109fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:26.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.959+0000 7f130ad63640 1 --2- 172.21.15.72:0/2147118179 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1304103740 0x7f1304109fd0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f12fc009be0 tx=0x7f12fc02bef0 comp rx=0 tx=0).stop 2024-03-22T01:36:26.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.959+0000 7f130ad63640 1 -- 172.21.15.72:0/2147118179 shutdown_connections 2024-03-22T01:36:26.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.959+0000 7f130ad63640 1 --2- 172.21.15.72:0/2147118179 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1304103740 0x7f1304109fd0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:26.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.959+0000 7f130ad63640 1 --2- 172.21.15.72:0/2147118179 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1304102d80 0x7f1304103200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:26.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.959+0000 7f130ad63640 1 --2- 172.21.15.72:0/2147118179 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1304101b90 0x7f1304101fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:26.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.959+0000 7f130ad63640 1 -- 172.21.15.72:0/2147118179 >> 172.21.15.72:0/2147118179 conn(0x7f13040fd0e0 msgr2=0x7f13040ff540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:26.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.959+0000 7f130ad63640 1 -- 172.21.15.72:0/2147118179 shutdown_connections 2024-03-22T01:36:26.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.959+0000 7f130ad63640 1 -- 172.21.15.72:0/2147118179 wait complete. 2024-03-22T01:36:26.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.959+0000 7f130ad63640 1 Processor -- start 2024-03-22T01:36:26.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.959+0000 7f130ad63640 1 -- start start 2024-03-22T01:36:26.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.959+0000 7f130ad63640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1304101b90 0x7f1304197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:26.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.959+0000 7f130ad63640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1304102d80 0x7f13041983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:26.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.959+0000 7f130ad63640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1304103740 0x7f130419f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:26.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.959+0000 7f130ad63640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f130410d7f0 con 0x7f1304103740 2024-03-22T01:36:26.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.959+0000 7f130ad63640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f130410d670 con 0x7f1304102d80 2024-03-22T01:36:26.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.959+0000 7f130ad63640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f130410d970 con 0x7f1304101b90 2024-03-22T01:36:26.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.960+0000 7f1309300640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1304103740 0x7f130419f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:26.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.960+0000 7f1309300640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1304103740 0x7f130419f400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:53560/0 (socket says 172.21.15.72:53560) 2024-03-22T01:36:26.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.960+0000 7f1309300640 1 -- 172.21.15.72:0/4129390684 learned_addr learned my addr 172.21.15.72:0/4129390684 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:26.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.960+0000 7f1303fff640 1 --2- 172.21.15.72:0/4129390684 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1304102d80 0x7f13041983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:26.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.960+0000 7f1308aff640 1 --2- 172.21.15.72:0/4129390684 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1304101b90 0x7f1304197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:26.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.960+0000 7f1309300640 1 -- 172.21.15.72:0/4129390684 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1304101b90 msgr2=0x7f1304197ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:26.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.960+0000 7f1309300640 1 --2- 172.21.15.72:0/4129390684 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1304101b90 0x7f1304197ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:26.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.960+0000 7f1309300640 1 -- 172.21.15.72:0/4129390684 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1304102d80 msgr2=0x7f13041983e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:26.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.960+0000 7f1309300640 1 --2- 172.21.15.72:0/4129390684 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1304102d80 0x7f13041983e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:26.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.960+0000 7f1309300640 1 -- 172.21.15.72:0/4129390684 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f130419fa70 con 0x7f1304103740 2024-03-22T01:36:26.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.960+0000 7f1309300640 1 --2- 172.21.15.72:0/4129390684 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1304103740 0x7f130419f400 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f12fc036040 tx=0x7f12fc0337b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:26.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.960+0000 7f1301ffb640 1 -- 172.21.15.72:0/4129390684 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12fc043020 con 0x7f1304103740 2024-03-22T01:36:26.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.960+0000 7f130ad63640 1 -- 172.21.15.72:0/4129390684 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f130419fd00 con 0x7f1304103740 2024-03-22T01:36:26.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.960+0000 7f1301ffb640 1 -- 172.21.15.72:0/4129390684 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12fc003b80 con 0x7f1304103740 2024-03-22T01:36:26.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.960+0000 7f1301ffb640 1 -- 172.21.15.72:0/4129390684 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12fc0347b0 con 0x7f1304103740 2024-03-22T01:36:26.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.961+0000 7f130ad63640 1 -- 172.21.15.72:0/4129390684 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13041a01f0 con 0x7f1304103740 2024-03-22T01:36:26.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.961+0000 7f130ad63640 1 -- 172.21.15.72:0/4129390684 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1304108f30 con 0x7f1304103740 2024-03-22T01:36:26.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.962+0000 7f1301ffb640 1 -- 172.21.15.72:0/4129390684 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7f12fc046020 con 0x7f1304103740 2024-03-22T01:36:26.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.963+0000 7f1301ffb640 1 --2- 172.21.15.72:0/4129390684 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f12f40a2f20 0x7f12f40a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:26.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.963+0000 7f1308aff640 1 --2- 172.21.15.72:0/4129390684 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f12f40a2f20 0x7f12f40a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:26.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.963+0000 7f1301ffb640 1 -- 172.21.15.72:0/4129390684 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f12fc0f1f20 con 0x7f1304103740 2024-03-22T01:36:26.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.963+0000 7f1308aff640 1 --2- 172.21.15.72:0/4129390684 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f12f40a2f20 0x7f12f40a53e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f12f80096c0 tx=0x7f12f8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:26.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:26.967+0000 7f1301ffb640 1 -- 172.21.15.72:0/4129390684 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f12fc04d020 con 0x7f1304103740 2024-03-22T01:36:27.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:27 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/3018799237' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:36:27.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.144+0000 7f130ad63640 1 -- 172.21.15.72:0/4129390684 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1304062830 con 0x7f1304103740 2024-03-22T01:36:27.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.145+0000 7f1301ffb640 1 -- 172.21.15.72:0/4129390684 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v95) ==== 93+0+10 (secure 0 0 0) 0x7f12fc0be980 con 0x7f1304103740 2024-03-22T01:36:27.144 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-22T01:36:27.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.147+0000 7f130ad63640 1 -- 172.21.15.72:0/4129390684 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f12f40a2f20 msgr2=0x7f12f40a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:27.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.147+0000 7f130ad63640 1 --2- 172.21.15.72:0/4129390684 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f12f40a2f20 0x7f12f40a53e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f12f80096c0 tx=0x7f12f8009260 comp rx=0 tx=0).stop 2024-03-22T01:36:27.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.147+0000 7f130ad63640 1 -- 172.21.15.72:0/4129390684 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1304103740 msgr2=0x7f130419f400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:27.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.147+0000 7f130ad63640 1 --2- 172.21.15.72:0/4129390684 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1304103740 0x7f130419f400 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f12fc036040 tx=0x7f12fc0337b0 comp rx=0 tx=0).stop 2024-03-22T01:36:27.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.147+0000 7f130ad63640 1 -- 172.21.15.72:0/4129390684 shutdown_connections 2024-03-22T01:36:27.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.147+0000 7f130ad63640 1 --2- 172.21.15.72:0/4129390684 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f12f40a2f20 0x7f12f40a53e0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:27.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.147+0000 7f130ad63640 1 --2- 172.21.15.72:0/4129390684 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1304102d80 0x7f13041983e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:27.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.147+0000 7f130ad63640 1 --2- 172.21.15.72:0/4129390684 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1304103740 0x7f130419f400 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:27.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.147+0000 7f130ad63640 1 --2- 172.21.15.72:0/4129390684 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1304101b90 0x7f1304197ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:27.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.147+0000 7f130ad63640 1 -- 172.21.15.72:0/4129390684 >> 172.21.15.72:0/4129390684 conn(0x7f13040fd0e0 msgr2=0x7f1304106100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:27.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.147+0000 7f130ad63640 1 -- 172.21.15.72:0/4129390684 shutdown_connections 2024-03-22T01:36:27.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.147+0000 7f130ad63640 1 -- 172.21.15.72:0/4129390684 wait complete. 2024-03-22T01:36:27.157 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-22T01:36:27.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.267+0000 7feba7e0e640 1 -- 172.21.15.72:0/275096806 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feba0100230 msgr2=0x7feba0100650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:27.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.267+0000 7feba7e0e640 1 --2- 172.21.15.72:0/275096806 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feba0100230 0x7feba0100650 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7feb9c00aa10 tx=0x7feb9c02cf70 comp rx=0 tx=0).stop 2024-03-22T01:36:27.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.267+0000 7feba7e0e640 1 -- 172.21.15.72:0/275096806 shutdown_connections 2024-03-22T01:36:27.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.267+0000 7feba7e0e640 1 --2- 172.21.15.72:0/275096806 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feba0101de0 0x7feba0108670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:27.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.267+0000 7feba7e0e640 1 --2- 172.21.15.72:0/275096806 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feba0101420 0x7feba01018a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:27.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.267+0000 7feba7e0e640 1 --2- 172.21.15.72:0/275096806 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feba0100230 0x7feba0100650 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:27.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.267+0000 7feba7e0e640 1 -- 172.21.15.72:0/275096806 >> 172.21.15.72:0/275096806 conn(0x7feba00fb7a0 msgr2=0x7feba00fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:27.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.268+0000 7feba7e0e640 1 -- 172.21.15.72:0/275096806 shutdown_connections 2024-03-22T01:36:27.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.268+0000 7feba7e0e640 1 -- 172.21.15.72:0/275096806 wait complete. 2024-03-22T01:36:27.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.268+0000 7feba7e0e640 1 Processor -- start 2024-03-22T01:36:27.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.268+0000 7feba7e0e640 1 -- start start 2024-03-22T01:36:27.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.268+0000 7feba7e0e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feba0100230 0x7feba01974f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:27.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.268+0000 7feba7e0e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feba0101420 0x7feba0195b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:27.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.268+0000 7feba7e0e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feba0101de0 0x7feba0196070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:27.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.268+0000 7feba7e0e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7feba010b640 con 0x7feba0100230 2024-03-22T01:36:27.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.268+0000 7feba7e0e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7feba010b4c0 con 0x7feba0101de0 2024-03-22T01:36:27.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.268+0000 7feba7e0e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7feba010b7c0 con 0x7feba0101420 2024-03-22T01:36:27.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.268+0000 7feba5baa640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feba0100230 0x7feba01974f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:27.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.269+0000 7feba5baa640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feba0100230 0x7feba01974f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:53582/0 (socket says 172.21.15.72:53582) 2024-03-22T01:36:27.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.269+0000 7feba5baa640 1 -- 172.21.15.72:0/855503192 learned_addr learned my addr 172.21.15.72:0/855503192 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:27.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.269+0000 7feba63ab640 1 --2- 172.21.15.72:0/855503192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feba0101de0 0x7feba0196070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:27.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.269+0000 7feba53a9640 1 --2- 172.21.15.72:0/855503192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feba0101420 0x7feba0195b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:27.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.269+0000 7feba5baa640 1 -- 172.21.15.72:0/855503192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feba0101420 msgr2=0x7feba0195b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:27.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.269+0000 7feba5baa640 1 --2- 172.21.15.72:0/855503192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feba0101420 0x7feba0195b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:27.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.269+0000 7feba5baa640 1 -- 172.21.15.72:0/855503192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feba0101de0 msgr2=0x7feba0196070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:27.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.269+0000 7feba5baa640 1 --2- 172.21.15.72:0/855503192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feba0101de0 0x7feba0196070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:27.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.269+0000 7feba5baa640 1 -- 172.21.15.72:0/855503192 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feba0196900 con 0x7feba0100230 2024-03-22T01:36:27.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.269+0000 7feba5baa640 1 --2- 172.21.15.72:0/855503192 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feba0100230 0x7feba01974f0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7feb9c000c00 tx=0x7feb9c005230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:27.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.269+0000 7feb96ffd640 1 -- 172.21.15.72:0/855503192 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb9c042020 con 0x7feba0100230 2024-03-22T01:36:27.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.269+0000 7feb96ffd640 1 -- 172.21.15.72:0/855503192 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb9c004ad0 con 0x7feba0100230 2024-03-22T01:36:27.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.269+0000 7feba7e0e640 1 -- 172.21.15.72:0/855503192 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feba01a2080 con 0x7feba0100230 2024-03-22T01:36:27.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.269+0000 7feb96ffd640 1 -- 172.21.15.72:0/855503192 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb9c003850 con 0x7feba0100230 2024-03-22T01:36:27.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.269+0000 7feba7e0e640 1 -- 172.21.15.72:0/855503192 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feba01a25d0 con 0x7feba0100230 2024-03-22T01:36:27.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.270+0000 7feba7e0e640 1 -- 172.21.15.72:0/855503192 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feba01075c0 con 0x7feba0100230 2024-03-22T01:36:27.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.271+0000 7feb96ffd640 1 -- 172.21.15.72:0/855503192 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7feb9c045020 con 0x7feba0100230 2024-03-22T01:36:27.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.272+0000 7feb96ffd640 1 --2- 172.21.15.72:0/855503192 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7feb8c0a2f20 0x7feb8c0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:27.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.272+0000 7feba53a9640 1 --2- 172.21.15.72:0/855503192 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7feb8c0a2f20 0x7feb8c0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:27.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.272+0000 7feb96ffd640 1 -- 172.21.15.72:0/855503192 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7feb9c0f23a0 con 0x7feba0100230 2024-03-22T01:36:27.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.273+0000 7feba53a9640 1 --2- 172.21.15.72:0/855503192 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7feb8c0a2f20 0x7feb8c0a53e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7feba0197260 tx=0x7feb90005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:27.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.276+0000 7feb96ffd640 1 -- 172.21.15.72:0/855503192 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7feb9c04c020 con 0x7feba0100230 2024-03-22T01:36:27.357 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:27 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/3018799237' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:36:27.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:27 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/3018799237' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:36:27.451 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.452+0000 7feba7e0e640 1 -- 172.21.15.72:0/855503192 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7feba002d540 con 0x7feba0100230 2024-03-22T01:36:27.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.453+0000 7feb96ffd640 1 -- 172.21.15.72:0/855503192 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v95) ==== 104+0+11 (secure 0 0 0) 0x7feb9c0beea0 con 0x7feba0100230 2024-03-22T01:36:27.452 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-03-22T01:36:27.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.455+0000 7feba7e0e640 1 -- 172.21.15.72:0/855503192 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7feb8c0a2f20 msgr2=0x7feb8c0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:27.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.455+0000 7feba7e0e640 1 --2- 172.21.15.72:0/855503192 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7feb8c0a2f20 0x7feb8c0a53e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7feba0197260 tx=0x7feb90005f90 comp rx=0 tx=0).stop 2024-03-22T01:36:27.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.455+0000 7feba7e0e640 1 -- 172.21.15.72:0/855503192 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feba0100230 msgr2=0x7feba01974f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:27.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.455+0000 7feba7e0e640 1 --2- 172.21.15.72:0/855503192 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feba0100230 0x7feba01974f0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7feb9c000c00 tx=0x7feb9c005230 comp rx=0 tx=0).stop 2024-03-22T01:36:27.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.455+0000 7feba7e0e640 1 -- 172.21.15.72:0/855503192 shutdown_connections 2024-03-22T01:36:27.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.455+0000 7feba7e0e640 1 --2- 172.21.15.72:0/855503192 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7feb8c0a2f20 0x7feb8c0a53e0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:27.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.455+0000 7feba7e0e640 1 --2- 172.21.15.72:0/855503192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feba0101de0 0x7feba0196070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:27.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.455+0000 7feba7e0e640 1 --2- 172.21.15.72:0/855503192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feba0101420 0x7feba0195b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:27.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.455+0000 7feba7e0e640 1 --2- 172.21.15.72:0/855503192 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feba0100230 0x7feba01974f0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:27.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.455+0000 7feba7e0e640 1 -- 172.21.15.72:0/855503192 >> 172.21.15.72:0/855503192 conn(0x7feba00fb7a0 msgr2=0x7feba0104790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:27.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.455+0000 7feba7e0e640 1 -- 172.21.15.72:0/855503192 shutdown_connections 2024-03-22T01:36:27.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.455+0000 7feba7e0e640 1 -- 172.21.15.72:0/855503192 wait complete. 2024-03-22T01:36:27.466 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-22T01:36:27.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.571+0000 7fa66a4df640 1 -- 172.21.15.72:0/1811494883 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa664101bb0 msgr2=0x7fa664101fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:27.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.571+0000 7fa66a4df640 1 --2- 172.21.15.72:0/1811494883 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa664101bb0 0x7fa664101fd0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fa654009900 tx=0x7fa65402bc30 comp rx=0 tx=0).stop 2024-03-22T01:36:27.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.571+0000 7fa66a4df640 1 -- 172.21.15.72:0/1811494883 shutdown_connections 2024-03-22T01:36:27.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.571+0000 7fa66a4df640 1 --2- 172.21.15.72:0/1811494883 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa664102da0 0x7fa664103220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:27.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.571+0000 7fa66a4df640 1 --2- 172.21.15.72:0/1811494883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa664103760 0x7fa664109ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:27.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.571+0000 7fa66a4df640 1 --2- 172.21.15.72:0/1811494883 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa664101bb0 0x7fa664101fd0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:27.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.571+0000 7fa66a4df640 1 -- 172.21.15.72:0/1811494883 >> 172.21.15.72:0/1811494883 conn(0x7fa6640fd120 msgr2=0x7fa6640ff560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:27.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.571+0000 7fa66a4df640 1 -- 172.21.15.72:0/1811494883 shutdown_connections 2024-03-22T01:36:27.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.571+0000 7fa66a4df640 1 -- 172.21.15.72:0/1811494883 wait complete. 2024-03-22T01:36:27.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.571+0000 7fa66a4df640 1 Processor -- start 2024-03-22T01:36:27.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.571+0000 7fa66a4df640 1 -- start start 2024-03-22T01:36:27.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.572+0000 7fa66a4df640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa664101bb0 0x7fa664075940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:27.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.572+0000 7fa66a4df640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa664102da0 0x7fa664073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:27.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.572+0000 7fa66a4df640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa664103760 0x7fa6640744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:27.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.572+0000 7fa66a4df640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fa66410d830 con 0x7fa664103760 2024-03-22T01:36:27.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.572+0000 7fa66a4df640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa66410d6b0 con 0x7fa664102da0 2024-03-22T01:36:27.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.572+0000 7fa66a4df640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa66410d9b0 con 0x7fa664101bb0 2024-03-22T01:36:27.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.572+0000 7fa668a7c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa664103760 0x7fa6640744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:27.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.572+0000 7fa668a7c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa664103760 0x7fa6640744c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:53594/0 (socket says 172.21.15.72:53594) 2024-03-22T01:36:27.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.572+0000 7fa6637fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa664102da0 0x7fa664073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:27.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.572+0000 7fa668a7c640 1 -- 172.21.15.72:0/3873822128 learned_addr learned my addr 172.21.15.72:0/3873822128 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:27.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.572+0000 7fa663fff640 1 --2- 172.21.15.72:0/3873822128 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa664101bb0 0x7fa664075940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:27.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.572+0000 7fa668a7c640 1 -- 172.21.15.72:0/3873822128 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa664101bb0 msgr2=0x7fa664075940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:27.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.572+0000 7fa668a7c640 1 --2- 172.21.15.72:0/3873822128 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa664101bb0 0x7fa664075940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:27.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.572+0000 7fa668a7c640 1 -- 172.21.15.72:0/3873822128 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa664102da0 msgr2=0x7fa664073f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:27.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.572+0000 7fa668a7c640 1 --2- 172.21.15.72:0/3873822128 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa664102da0 0x7fa664073f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:27.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.572+0000 7fa668a7c640 1 -- 172.21.15.72:0/3873822128 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa664074d50 con 0x7fa664103760 2024-03-22T01:36:27.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.572+0000 7fa6637fe640 1 --2- 172.21.15.72:0/3873822128 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa664102da0 0x7fa664073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:36:27.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.572+0000 7fa668a7c640 1 --2- 172.21.15.72:0/3873822128 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa664103760 0x7fa6640744c0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fa65c00eaa0 tx=0x7fa65c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:27.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.572+0000 7fa6617fa640 1 -- 172.21.15.72:0/3873822128 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa65c00cdb0 con 0x7fa664103760 2024-03-22T01:36:27.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.572+0000 7fa6617fa640 1 -- 172.21.15.72:0/3873822128 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa65c005020 con 0x7fa664103760 2024-03-22T01:36:27.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.572+0000 7fa66a4df640 1 -- 172.21.15.72:0/3873822128 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6641a1fc0 con 0x7fa664103760 2024-03-22T01:36:27.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.572+0000 7fa6617fa640 1 -- 172.21.15.72:0/3873822128 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa65c0116b0 con 0x7fa664103760 2024-03-22T01:36:27.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.572+0000 7fa66a4df640 1 -- 172.21.15.72:0/3873822128 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6641a2420 con 0x7fa664103760 2024-03-22T01:36:27.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.574+0000 7fa66a4df640 1 -- 172.21.15.72:0/3873822128 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa664108ed0 con 0x7fa664103760 2024-03-22T01:36:27.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.575+0000 7fa6617fa640 1 -- 172.21.15.72:0/3873822128 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7fa65c005560 con 0x7fa664103760 2024-03-22T01:36:27.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.575+0000 7fa6617fa640 1 --2- 172.21.15.72:0/3873822128 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fa64c0a2f20 0x7fa64c0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:27.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.575+0000 7fa663fff640 1 --2- 172.21.15.72:0/3873822128 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fa64c0a2f20 0x7fa64c0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:27.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.575+0000 7fa6617fa640 1 -- 172.21.15.72:0/3873822128 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7fa65c0cf930 con 0x7fa664103760 2024-03-22T01:36:27.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.576+0000 7fa663fff640 1 --2- 172.21.15.72:0/3873822128 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fa64c0a2f20 0x7fa64c0a53e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fa654000c00 tx=0x7fa654005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:27.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.580+0000 7fa6617fa640 1 -- 172.21.15.72:0/3873822128 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa65c016030 con 0x7fa664103760 2024-03-22T01:36:27.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.753+0000 7fa66a4df640 1 -- 172.21.15.72:0/3873822128 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa664062830 con 0x7fa664103760 2024-03-22T01:36:27.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.753+0000 7fa6617fa640 1 -- 172.21.15.72:0/3873822128 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v95) ==== 100+0+11 (secure 0 0 0) 0x7fa65c09bd40 con 0x7fa664103760 2024-03-22T01:36:27.752 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-03-22T01:36:27.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.755+0000 7fa66a4df640 1 -- 172.21.15.72:0/3873822128 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fa64c0a2f20 msgr2=0x7fa64c0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:27.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.755+0000 7fa66a4df640 1 --2- 172.21.15.72:0/3873822128 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fa64c0a2f20 0x7fa64c0a53e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fa654000c00 tx=0x7fa654005fd0 comp rx=0 tx=0).stop 2024-03-22T01:36:27.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.755+0000 7fa66a4df640 1 -- 172.21.15.72:0/3873822128 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa664103760 msgr2=0x7fa6640744c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:27.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.755+0000 7fa66a4df640 1 --2- 172.21.15.72:0/3873822128 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa664103760 0x7fa6640744c0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fa65c00eaa0 tx=0x7fa65c00ef60 comp rx=0 tx=0).stop 2024-03-22T01:36:27.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.755+0000 7fa66a4df640 1 -- 172.21.15.72:0/3873822128 shutdown_connections 2024-03-22T01:36:27.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.755+0000 7fa66a4df640 1 --2- 172.21.15.72:0/3873822128 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7fa64c0a2f20 0x7fa64c0a53e0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:27.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.755+0000 7fa66a4df640 1 --2- 172.21.15.72:0/3873822128 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa664102da0 0x7fa664073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:27.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.755+0000 7fa66a4df640 1 --2- 172.21.15.72:0/3873822128 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa664103760 0x7fa6640744c0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:27.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.755+0000 7fa66a4df640 1 --2- 172.21.15.72:0/3873822128 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa664101bb0 0x7fa664075940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:27.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.755+0000 7fa66a4df640 1 -- 172.21.15.72:0/3873822128 >> 172.21.15.72:0/3873822128 conn(0x7fa6640fd120 msgr2=0x7fa6641060a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:27.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.755+0000 7fa66a4df640 1 -- 172.21.15.72:0/3873822128 shutdown_connections 2024-03-22T01:36:27.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.755+0000 7fa66a4df640 1 -- 172.21.15.72:0/3873822128 wait complete. 2024-03-22T01:36:27.765 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-22T01:36:27.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.870+0000 7f3d4bad6640 1 -- 172.21.15.72:0/3253527035 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d44101ca0 msgr2=0x7f3d44102120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:27.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.870+0000 7f3d4bad6640 1 --2- 172.21.15.72:0/3253527035 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d44101ca0 0x7f3d44102120 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f3d34009900 tx=0x7f3d3402bc30 comp rx=0 tx=0).stop 2024-03-22T01:36:27.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.870+0000 7f3d4bad6640 1 -- 172.21.15.72:0/3253527035 shutdown_connections 2024-03-22T01:36:27.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.870+0000 7f3d4bad6640 1 --2- 172.21.15.72:0/3253527035 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3d44102660 0x7f3d44108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:27.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.870+0000 7f3d4bad6640 1 --2- 172.21.15.72:0/3253527035 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d44101ca0 0x7f3d44102120 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:27.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.870+0000 7f3d4bad6640 1 --2- 172.21.15.72:0/3253527035 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d44100ab0 0x7f3d44100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:27.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.871+0000 7f3d4bad6640 1 -- 172.21.15.72:0/3253527035 >> 172.21.15.72:0/3253527035 conn(0x7f3d440fc020 msgr2=0x7f3d440fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:27.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.871+0000 7f3d4bad6640 1 -- 172.21.15.72:0/3253527035 shutdown_connections 2024-03-22T01:36:27.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.871+0000 7f3d4bad6640 1 -- 172.21.15.72:0/3253527035 wait complete. 2024-03-22T01:36:27.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.871+0000 7f3d4bad6640 1 Processor -- start 2024-03-22T01:36:27.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.871+0000 7f3d4bad6640 1 -- start start 2024-03-22T01:36:27.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.871+0000 7f3d4bad6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d44100ab0 0x7f3d44197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:27.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.871+0000 7f3d4bad6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3d44101ca0 0x7f3d44198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:27.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.871+0000 7f3d4bad6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d44102660 0x7f3d4419f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:27.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.871+0000 7f3d4bad6640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f3d4410c730 con 0x7f3d44100ab0 2024-03-22T01:36:27.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.871+0000 7f3d4bad6640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3d4410c5b0 con 0x7f3d44101ca0 2024-03-22T01:36:27.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.871+0000 7f3d4bad6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3d4410c8b0 con 0x7f3d44102660 2024-03-22T01:36:27.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.871+0000 7f3d49872640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d44100ab0 0x7f3d44197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:27.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.871+0000 7f3d49872640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d44100ab0 0x7f3d44197f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:53616/0 (socket says 172.21.15.72:53616) 2024-03-22T01:36:27.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.871+0000 7f3d49872640 1 -- 172.21.15.72:0/2772932760 learned_addr learned my addr 172.21.15.72:0/2772932760 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:27.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.871+0000 7f3d49071640 1 --2- 172.21.15.72:0/2772932760 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3d44101ca0 0x7f3d44198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:27.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.872+0000 7f3d4a073640 1 --2- 172.21.15.72:0/2772932760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d44102660 0x7f3d4419f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:27.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.872+0000 7f3d49872640 1 -- 172.21.15.72:0/2772932760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d44102660 msgr2=0x7f3d4419f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:27.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.872+0000 7f3d49872640 1 --2- 172.21.15.72:0/2772932760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d44102660 0x7f3d4419f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:27.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.872+0000 7f3d49872640 1 -- 172.21.15.72:0/2772932760 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3d44101ca0 msgr2=0x7f3d44198480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:27.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.872+0000 7f3d49872640 1 --2- 172.21.15.72:0/2772932760 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3d44101ca0 0x7f3d44198480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:27.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.872+0000 7f3d49872640 1 -- 172.21.15.72:0/2772932760 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d4419fb60 con 0x7f3d44100ab0 2024-03-22T01:36:27.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.872+0000 7f3d4a073640 1 --2- 172.21.15.72:0/2772932760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d44102660 0x7f3d4419f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:36:27.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.872+0000 7f3d49872640 1 --2- 172.21.15.72:0/2772932760 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d44100ab0 0x7f3d44197f40 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f3d4000d950 tx=0x7f3d4000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:27.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.872+0000 7f3d3affd640 1 -- 172.21.15.72:0/2772932760 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d40017070 con 0x7f3d44100ab0 2024-03-22T01:36:27.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.872+0000 7f3d3affd640 1 -- 172.21.15.72:0/2772932760 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d40009c30 con 0x7f3d44100ab0 2024-03-22T01:36:27.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.872+0000 7f3d4bad6640 1 -- 172.21.15.72:0/2772932760 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d4419fe50 con 0x7f3d44100ab0 2024-03-22T01:36:27.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.872+0000 7f3d3affd640 1 -- 172.21.15.72:0/2772932760 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d4000f600 con 0x7f3d44100ab0 2024-03-22T01:36:27.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.872+0000 7f3d4bad6640 1 -- 172.21.15.72:0/2772932760 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d441a0260 con 0x7f3d44100ab0 2024-03-22T01:36:27.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.873+0000 7f3d4bad6640 1 -- 172.21.15.72:0/2772932760 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d441a0490 con 0x7f3d44100ab0 2024-03-22T01:36:27.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.874+0000 7f3d3affd640 1 -- 172.21.15.72:0/2772932760 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7f3d4001a070 con 0x7f3d44100ab0 2024-03-22T01:36:27.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.875+0000 7f3d3affd640 1 --2- 172.21.15.72:0/2772932760 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f3d300a2f20 0x7f3d300a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:27.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.875+0000 7f3d49071640 1 --2- 172.21.15.72:0/2772932760 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f3d300a2f20 0x7f3d300a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:27.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.875+0000 7f3d3affd640 1 -- 172.21.15.72:0/2772932760 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f3d400cf220 con 0x7f3d44100ab0 2024-03-22T01:36:27.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.875+0000 7f3d49071640 1 --2- 172.21.15.72:0/2772932760 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f3d300a2f20 0x7f3d300a53e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3d3402c7b0 tx=0x7f3d34005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:27.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:27.879+0000 7f3d3affd640 1 -- 172.21.15.72:0/2772932760 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3d40009de0 con 0x7f3d44100ab0 2024-03-22T01:36:28.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.053+0000 7f3d4bad6640 1 -- 172.21.15.72:0/2772932760 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f3d441a0780 con 0x7f3d44100ab0 2024-03-22T01:36:28.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.054+0000 7f3d3affd640 1 -- 172.21.15.72:0/2772932760 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v95) ==== 103+0+11 (secure 0 0 0) 0x7f3d4009bc80 con 0x7f3d44100ab0 2024-03-22T01:36:28.053 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-03-22T01:36:28.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.056+0000 7f3d4bad6640 1 -- 172.21.15.72:0/2772932760 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f3d300a2f20 msgr2=0x7f3d300a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:28.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.056+0000 7f3d4bad6640 1 --2- 172.21.15.72:0/2772932760 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f3d300a2f20 0x7f3d300a53e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3d3402c7b0 tx=0x7f3d34005fd0 comp rx=0 tx=0).stop 2024-03-22T01:36:28.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.056+0000 7f3d4bad6640 1 -- 172.21.15.72:0/2772932760 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d44100ab0 msgr2=0x7f3d44197f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:28.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.056+0000 7f3d4bad6640 1 --2- 172.21.15.72:0/2772932760 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d44100ab0 0x7f3d44197f40 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f3d4000d950 tx=0x7f3d4000de10 comp rx=0 tx=0).stop 2024-03-22T01:36:28.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.056+0000 7f3d4bad6640 1 -- 172.21.15.72:0/2772932760 shutdown_connections 2024-03-22T01:36:28.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.056+0000 7f3d4bad6640 1 --2- 172.21.15.72:0/2772932760 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f3d300a2f20 0x7f3d300a53e0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:28.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.056+0000 7f3d4bad6640 1 --2- 172.21.15.72:0/2772932760 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3d44101ca0 0x7f3d44198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:28.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.056+0000 7f3d4bad6640 1 --2- 172.21.15.72:0/2772932760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d44102660 0x7f3d4419f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:28.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.056+0000 7f3d4bad6640 1 --2- 172.21.15.72:0/2772932760 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d44100ab0 0x7f3d44197f40 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:28.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.056+0000 7f3d4bad6640 1 -- 172.21.15.72:0/2772932760 >> 172.21.15.72:0/2772932760 conn(0x7f3d440fc020 msgr2=0x7f3d44068e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:28.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.056+0000 7f3d4bad6640 1 -- 172.21.15.72:0/2772932760 shutdown_connections 2024-03-22T01:36:28.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.056+0000 7f3d4bad6640 1 -- 172.21.15.72:0/2772932760 wait complete. 2024-03-22T01:36:28.067 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T01:36:28.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.169+0000 7efef3fb6640 1 -- 172.21.15.72:0/3740119433 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efeec100b30 msgr2=0x7efeec100f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:28.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.169+0000 7efef3fb6640 1 --2- 172.21.15.72:0/3740119433 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efeec100b30 0x7efeec100f50 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7efee4009900 tx=0x7efee402bc30 comp rx=0 tx=0).stop 2024-03-22T01:36:28.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.169+0000 7efef3fb6640 1 -- 172.21.15.72:0/3740119433 shutdown_connections 2024-03-22T01:36:28.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.169+0000 7efef3fb6640 1 --2- 172.21.15.72:0/3740119433 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efeec101d20 0x7efeec1021a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:28.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.169+0000 7efef3fb6640 1 --2- 172.21.15.72:0/3740119433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efeec1026e0 0x7efeec108f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:28.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.169+0000 7efef3fb6640 1 --2- 172.21.15.72:0/3740119433 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efeec100b30 0x7efeec100f50 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:28.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.169+0000 7efef3fb6640 1 -- 172.21.15.72:0/3740119433 >> 172.21.15.72:0/3740119433 conn(0x7efeec0fc0a0 msgr2=0x7efeec0fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:28.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.169+0000 7efef3fb6640 1 -- 172.21.15.72:0/3740119433 shutdown_connections 2024-03-22T01:36:28.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.169+0000 7efef3fb6640 1 -- 172.21.15.72:0/3740119433 wait complete. 2024-03-22T01:36:28.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.170+0000 7efef3fb6640 1 Processor -- start 2024-03-22T01:36:28.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.170+0000 7efef3fb6640 1 -- start start 2024-03-22T01:36:28.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.170+0000 7efef3fb6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efeec100b30 0x7efeec197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:28.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.170+0000 7efef3fb6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efeec101d20 0x7efeec198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:28.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.170+0000 7efef3fb6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efeec1026e0 0x7efeec19f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:28.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.170+0000 7efef3fb6640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7efeec10c740 con 0x7efeec1026e0 2024-03-22T01:36:28.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.170+0000 7efef3fb6640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7efeec10c5c0 con 0x7efeec101d20 2024-03-22T01:36:28.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.170+0000 7efef3fb6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efeec10c8c0 con 0x7efeec100b30 2024-03-22T01:36:28.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.170+0000 7efef2553640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efeec1026e0 0x7efeec19f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:28.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.170+0000 7efef2553640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efeec1026e0 0x7efeec19f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:53636/0 (socket says 172.21.15.72:53636) 2024-03-22T01:36:28.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.170+0000 7efef2553640 1 -- 172.21.15.72:0/2571501517 learned_addr learned my addr 172.21.15.72:0/2571501517 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:28.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.170+0000 7efef1551640 1 --2- 172.21.15.72:0/2571501517 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efeec101d20 0x7efeec198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:28.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.170+0000 7efef1d52640 1 --2- 172.21.15.72:0/2571501517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efeec100b30 0x7efeec197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:28.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.170+0000 7efef2553640 1 -- 172.21.15.72:0/2571501517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efeec100b30 msgr2=0x7efeec197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:28.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.170+0000 7efef2553640 1 --2- 172.21.15.72:0/2571501517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efeec100b30 0x7efeec197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:28.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.170+0000 7efef2553640 1 -- 172.21.15.72:0/2571501517 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efeec101d20 msgr2=0x7efeec198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:28.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.170+0000 7efef2553640 1 --2- 172.21.15.72:0/2571501517 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efeec101d20 0x7efeec198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:28.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.170+0000 7efef2553640 1 -- 172.21.15.72:0/2571501517 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efeec19fb00 con 0x7efeec1026e0 2024-03-22T01:36:28.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.170+0000 7efef1551640 1 --2- 172.21.15.72:0/2571501517 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efeec101d20 0x7efeec198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:36:28.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.170+0000 7efef2553640 1 --2- 172.21.15.72:0/2571501517 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efeec1026e0 0x7efeec19f490 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7efedc00eaa0 tx=0x7efedc00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:28.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.170+0000 7efeeaffd640 1 -- 172.21.15.72:0/2571501517 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efedc00cdb0 con 0x7efeec1026e0 2024-03-22T01:36:28.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.170+0000 7efeeaffd640 1 -- 172.21.15.72:0/2571501517 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efedc005020 con 0x7efeec1026e0 2024-03-22T01:36:28.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.171+0000 7efef3fb6640 1 -- 172.21.15.72:0/2571501517 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efeec19fdf0 con 0x7efeec1026e0 2024-03-22T01:36:28.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.171+0000 7efeeaffd640 1 -- 172.21.15.72:0/2571501517 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efedc0116b0 con 0x7efeec1026e0 2024-03-22T01:36:28.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.171+0000 7efef3fb6640 1 -- 172.21.15.72:0/2571501517 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efeec1a0300 con 0x7efeec1026e0 2024-03-22T01:36:28.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.172+0000 7efef3fb6640 1 -- 172.21.15.72:0/2571501517 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efeec1a05c0 con 0x7efeec1026e0 2024-03-22T01:36:28.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.172+0000 7efeeaffd640 1 -- 172.21.15.72:0/2571501517 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7efedc005560 con 0x7efeec1026e0 2024-03-22T01:36:28.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.173+0000 7efeeaffd640 1 --2- 172.21.15.72:0/2571501517 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7efed40a2fa0 0x7efed40a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:28.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.173+0000 7efef1d52640 1 --2- 172.21.15.72:0/2571501517 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7efed40a2fa0 0x7efed40a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:28.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.173+0000 7efeeaffd640 1 -- 172.21.15.72:0/2571501517 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7efedc0cf8d0 con 0x7efeec1026e0 2024-03-22T01:36:28.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.173+0000 7efef1d52640 1 --2- 172.21.15.72:0/2571501517 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7efed40a2fa0 0x7efed40a5460 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7efee4000c00 tx=0x7efee4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:28.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.177+0000 7efeeaffd640 1 -- 172.21.15.72:0/2571501517 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7efedc016030 con 0x7efeec1026e0 2024-03-22T01:36:28.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:28 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/4129390684' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T01:36:28.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:28 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/855503192' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T01:36:28.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:28 smithi089 ceph-mon[35770]: pgmap v361: 129 pgs: 129 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-03-22T01:36:28.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:28 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/3873822128' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T01:36:28.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:28 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/4129390684' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T01:36:28.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:28 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/855503192' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T01:36:28.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:28 smithi072 ceph-mon[32414]: pgmap v361: 129 pgs: 129 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-03-22T01:36:28.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:28 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/3873822128' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T01:36:28.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.390+0000 7efef3fb6640 1 -- 172.21.15.72:0/2571501517 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efeec1a0790 con 0x7efeec1026e0 2024-03-22T01:36:28.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.391+0000 7efeeaffd640 1 -- 172.21.15.72:0/2571501517 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11785 (secure 0 0 0) 0x7efedc09bce0 con 0x7efeec1026e0 2024-03-22T01:36:28.390 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:36:28.391 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6835/3077216308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3077216308},{"type":"v1","addr":"172.21.15.72:6835","nonce":3077216308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2460254143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":2460254143},{"type":"v1","addr":"172.21.15.72:6837","nonce":2460254143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6839/2026432819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6838","nonce":2026432819},{"type":"v1","addr":"172.21.15.72:6839","nonce":2026432819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6841/400094545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6840","nonce":400094545},{"type":"v1","addr":"172.21.15.72:6841","nonce":400094545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6833/1578701171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":1578701171},{"type":"v1","addr":"172.21.15.89:6833","nonce":1578701171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/2703047213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2703047213},{"type":"v1","addr":"172.21.15.98:6833","nonce":2703047213}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2589468787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2589468787},{"type":"v1","addr":"172.21.15.89:6835","nonce":2589468787}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/92657636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":92657636},{"type":"v1","addr":"172.21.15.98:6835","nonce":92657636}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1662941782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1662941782},{"type":"v1","addr":"172.21.15.89:6837","nonce":1662941782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24490,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/781860371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":781860371},{"type":"v1","addr":"172.21.15.98:6837","nonce":781860371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-22T01:34:14.815677+0000","modified":"2024-03-22T01:34:39.842959+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24524},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3192173650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3192173650},{"type":"v1","addr":"172.21.15.98:6839","nonce":3192173650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6839/149200280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":149200280},{"type":"v1","addr":"172.21.15.89:6839","nonce":149200280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24524]},"id":1}]} 2024-03-22T01:36:28.391 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 21 2024-03-22T01:36:28.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.393+0000 7efef3fb6640 1 -- 172.21.15.72:0/2571501517 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7efed40a2fa0 msgr2=0x7efed40a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:28.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.393+0000 7efef3fb6640 1 --2- 172.21.15.72:0/2571501517 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7efed40a2fa0 0x7efed40a5460 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7efee4000c00 tx=0x7efee4005fd0 comp rx=0 tx=0).stop 2024-03-22T01:36:28.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.393+0000 7efef3fb6640 1 -- 172.21.15.72:0/2571501517 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efeec1026e0 msgr2=0x7efeec19f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:28.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.393+0000 7efef3fb6640 1 --2- 172.21.15.72:0/2571501517 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efeec1026e0 0x7efeec19f490 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7efedc00eaa0 tx=0x7efedc00ef60 comp rx=0 tx=0).stop 2024-03-22T01:36:28.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.393+0000 7efef3fb6640 1 -- 172.21.15.72:0/2571501517 shutdown_connections 2024-03-22T01:36:28.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.393+0000 7efef3fb6640 1 --2- 172.21.15.72:0/2571501517 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7efed40a2fa0 0x7efed40a5460 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:28.393 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.393+0000 7efef3fb6640 1 --2- 172.21.15.72:0/2571501517 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efeec101d20 0x7efeec198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:28.393 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.393+0000 7efef3fb6640 1 --2- 172.21.15.72:0/2571501517 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efeec1026e0 0x7efeec19f490 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:28.393 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.393+0000 7efef3fb6640 1 --2- 172.21.15.72:0/2571501517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efeec100b30 0x7efeec197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:28.393 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.393+0000 7efef3fb6640 1 -- 172.21.15.72:0/2571501517 >> 172.21.15.72:0/2571501517 conn(0x7efeec0fc0a0 msgr2=0x7efeec0fdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:28.393 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.393+0000 7efef3fb6640 1 -- 172.21.15.72:0/2571501517 shutdown_connections 2024-03-22T01:36:28.393 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.393+0000 7efef3fb6640 1 -- 172.21.15.72:0/2571501517 wait complete. 2024-03-22T01:36:28.404 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T01:36:28.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:28 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/4129390684' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T01:36:28.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:28 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/855503192' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T01:36:28.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:28 smithi098 ceph-mon[36468]: pgmap v361: 129 pgs: 129 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-03-22T01:36:28.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:28 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/3873822128' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T01:36:28.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.507+0000 7f18e13ae640 1 -- 172.21.15.72:0/167103362 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18dc1037d0 msgr2=0x7f18dc10a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:28.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.507+0000 7f18e13ae640 1 --2- 172.21.15.72:0/167103362 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18dc1037d0 0x7f18dc10a060 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f18d4009be0 tx=0x7f18d402bef0 comp rx=0 tx=0).stop 2024-03-22T01:36:28.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.507+0000 7f18e13ae640 1 -- 172.21.15.72:0/167103362 shutdown_connections 2024-03-22T01:36:28.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.507+0000 7f18e13ae640 1 --2- 172.21.15.72:0/167103362 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18dc1037d0 0x7f18dc10a060 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:28.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.507+0000 7f18e13ae640 1 --2- 172.21.15.72:0/167103362 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18dc102e10 0x7f18dc103290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:28.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.507+0000 7f18e13ae640 1 --2- 172.21.15.72:0/167103362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18dc101c20 0x7f18dc102040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:28.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.507+0000 7f18e13ae640 1 -- 172.21.15.72:0/167103362 >> 172.21.15.72:0/167103362 conn(0x7f18dc0fd1b0 msgr2=0x7f18dc0ff5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:28.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.507+0000 7f18e13ae640 1 -- 172.21.15.72:0/167103362 shutdown_connections 2024-03-22T01:36:28.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.507+0000 7f18e13ae640 1 -- 172.21.15.72:0/167103362 wait complete. 2024-03-22T01:36:28.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.507+0000 7f18e13ae640 1 Processor -- start 2024-03-22T01:36:28.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.507+0000 7f18e13ae640 1 -- start start 2024-03-22T01:36:28.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.508+0000 7f18e13ae640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18dc101c20 0x7f18dc197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:28.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.508+0000 7f18e13ae640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18dc102e10 0x7f18dc1983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:28.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.508+0000 7f18e13ae640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18dc1037d0 0x7f18dc19f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:28.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.508+0000 7f18e13ae640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f18dc10d830 con 0x7f18dc1037d0 2024-03-22T01:36:28.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.508+0000 7f18e13ae640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f18dc10d6b0 con 0x7f18dc101c20 2024-03-22T01:36:28.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.508+0000 7f18e13ae640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f18dc10d9b0 con 0x7f18dc102e10 2024-03-22T01:36:28.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.508+0000 7f18daffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18dc101c20 0x7f18dc197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:28.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.508+0000 7f18daffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18dc101c20 0x7f18dc197e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.72:42544/0 (socket says 172.21.15.72:42544) 2024-03-22T01:36:28.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.508+0000 7f18daffd640 1 -- 172.21.15.72:0/2470145942 learned_addr learned my addr 172.21.15.72:0/2470145942 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:28.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.508+0000 7f18db7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18dc1037d0 0x7f18dc19f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:28.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.508+0000 7f18da7fc640 1 --2- 172.21.15.72:0/2470145942 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18dc102e10 0x7f18dc1983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:28.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.508+0000 7f18daffd640 1 -- 172.21.15.72:0/2470145942 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18dc102e10 msgr2=0x7f18dc1983b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:28.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.508+0000 7f18daffd640 1 --2- 172.21.15.72:0/2470145942 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18dc102e10 0x7f18dc1983b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:28.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.508+0000 7f18daffd640 1 -- 172.21.15.72:0/2470145942 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18dc1037d0 msgr2=0x7f18dc19f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:28.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.508+0000 7f18daffd640 1 --2- 172.21.15.72:0/2470145942 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18dc1037d0 0x7f18dc19f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:28.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.508+0000 7f18daffd640 1 -- 172.21.15.72:0/2470145942 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18dc19fa90 con 0x7f18dc101c20 2024-03-22T01:36:28.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.508+0000 7f18daffd640 1 --2- 172.21.15.72:0/2470145942 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18dc101c20 0x7f18dc197e70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f18c400c970 tx=0x7f18c400ce30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:28.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.508+0000 7f18d3fff640 1 -- 172.21.15.72:0/2470145942 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18c4004220 con 0x7f18dc101c20 2024-03-22T01:36:28.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.508+0000 7f18d3fff640 1 -- 172.21.15.72:0/2470145942 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18c40043c0 con 0x7f18dc101c20 2024-03-22T01:36:28.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.508+0000 7f18e13ae640 1 -- 172.21.15.72:0/2470145942 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18dc19fd80 con 0x7f18dc101c20 2024-03-22T01:36:28.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.508+0000 7f18d3fff640 1 -- 172.21.15.72:0/2470145942 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18c4004a00 con 0x7f18dc101c20 2024-03-22T01:36:28.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.508+0000 7f18e13ae640 1 -- 172.21.15.72:0/2470145942 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18dc1a0210 con 0x7f18dc101c20 2024-03-22T01:36:28.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.509+0000 7f18e13ae640 1 -- 172.21.15.72:0/2470145942 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18dc108f40 con 0x7f18dc101c20 2024-03-22T01:36:28.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.510+0000 7f18d3fff640 1 -- 172.21.15.72:0/2470145942 <== mon.2 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7f18c40079b0 con 0x7f18dc101c20 2024-03-22T01:36:28.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.511+0000 7f18d3fff640 1 --2- 172.21.15.72:0/2470145942 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f18c80a2fa0 0x7f18c80a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:28.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.511+0000 7f18da7fc640 1 --2- 172.21.15.72:0/2470145942 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f18c80a2fa0 0x7f18c80a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:28.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.511+0000 7f18d3fff640 1 -- 172.21.15.72:0/2470145942 <== mon.2 v2:172.21.15.89:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f18c40cf5a0 con 0x7f18dc101c20 2024-03-22T01:36:28.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.511+0000 7f18da7fc640 1 --2- 172.21.15.72:0/2470145942 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f18c80a2fa0 0x7f18c80a5460 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f18dc199380 tx=0x7f18cc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:28.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.515+0000 7f18d3fff640 1 -- 172.21.15.72:0/2470145942 <== mon.2 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f18c4014030 con 0x7f18dc101c20 2024-03-22T01:36:28.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.691+0000 7f18e13ae640 1 -- 172.21.15.72:0/2470145942 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f18dc02d550 con 0x7f18dc101c20 2024-03-22T01:36:28.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.693+0000 7f18d3fff640 1 -- 172.21.15.72:0/2470145942 <== mon.2 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) ==== 74+0+24759 (secure 0 0 0) 0x7f18c409ba30 con 0x7f18dc101c20 2024-03-22T01:36:28.692 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:36:28.692 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":95,"fsid":"022a99bc-e7eb-11ee-95cd-87774f69a715","created":"2024-03-22T01:26:23.893672+0000","modified":"2024-03-22T01:36:17.626966+0000","last_up_change":"2024-03-22T01:33:25.223684+0000","last_in_change":"2024-03-22T01:33:05.010331+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T01:29:37.438503+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-22T01:34:12.610422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-22T01:34:13.761686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-22T01:34:17.507039+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"16c15bb0-db92-4191-9063-38b4890e81e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6803","nonce":661906791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6805","nonce":661906791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6809","nonce":661906791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6807","nonce":661906791}]},"public_addr":"172.21.15.72:6803/661906791","cluster_addr":"172.21.15.72:6805/661906791","heartbeat_back_addr":"172.21.15.72:6809/661906791","heartbeat_front_addr":"172.21.15.72:6807/661906791","state":["exists","up"]},{"osd":1,"uuid":"981290ba-97dc-496b-aa22-d51704184b10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6801","nonce":2870590550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6803","nonce":2870590550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6807","nonce":2870590550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6805","nonce":2870590550}]},"public_addr":"172.21.15.89:6801/2870590550","cluster_addr":"172.21.15.89:6803/2870590550","heartbeat_back_addr":"172.21.15.89:6807/2870590550","heartbeat_front_addr":"172.21.15.89:6805/2870590550","state":["exists","up"]},{"osd":2,"uuid":"f07ed658-f545-46a1-8edd-8b1d5d49885c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6801","nonce":3865778118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6803","nonce":3865778118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6807","nonce":3865778118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6805","nonce":3865778118}]},"public_addr":"172.21.15.98:6801/3865778118","cluster_addr":"172.21.15.98:6803/3865778118","heartbeat_back_addr":"172.21.15.98:6807/3865778118","heartbeat_front_addr":"172.21.15.98:6805/3865778118","state":["exists","up"]},{"osd":3,"uuid":"ccc4bc40-2c48-46d2-8548-0c05b44f9f1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6811","nonce":1865951395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6813","nonce":1865951395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6817","nonce":1865951395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6815","nonce":1865951395}]},"public_addr":"172.21.15.72:6811/1865951395","cluster_addr":"172.21.15.72:6813/1865951395","heartbeat_back_addr":"172.21.15.72:6817/1865951395","heartbeat_front_addr":"172.21.15.72:6815/1865951395","state":["exists","up"]},{"osd":4,"uuid":"1812439f-810a-4ed2-9395-c00c9a085a13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6809","nonce":2683760795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6811","nonce":2683760795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6815","nonce":2683760795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6813","nonce":2683760795}]},"public_addr":"172.21.15.89:6809/2683760795","cluster_addr":"172.21.15.89:6811/2683760795","heartbeat_back_addr":"172.21.15.89:6815/2683760795","heartbeat_front_addr":"172.21.15.89:6813/2683760795","state":["exists","up"]},{"osd":5,"uuid":"e78954d8-f4e3-4d58-8536-d2d59c843386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6809","nonce":2635838572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6811","nonce":2635838572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6815","nonce":2635838572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6813","nonce":2635838572}]},"public_addr":"172.21.15.98:6809/2635838572","cluster_addr":"172.21.15.98:6811/2635838572","heartbeat_back_addr":"172.21.15.98:6815/2635838572","heartbeat_front_addr":"172.21.15.98:6813/2635838572","state":["exists","up"]},{"osd":6,"uuid":"dc6c8be2-b64c-4959-8015-7b0ea6d25599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6819","nonce":3158982661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6821","nonce":3158982661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6825","nonce":3158982661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6823","nonce":3158982661}]},"public_addr":"172.21.15.72:6819/3158982661","cluster_addr":"172.21.15.72:6821/3158982661","heartbeat_back_addr":"172.21.15.72:6825/3158982661","heartbeat_front_addr":"172.21.15.72:6823/3158982661","state":["exists","up"]},{"osd":7,"uuid":"7a73469f-cf3a-4306-ae93-ea46e21ebbbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6817","nonce":3075913234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6819","nonce":3075913234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6823","nonce":3075913234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6821","nonce":3075913234}]},"public_addr":"172.21.15.89:6817/3075913234","cluster_addr":"172.21.15.89:6819/3075913234","heartbeat_back_addr":"172.21.15.89:6823/3075913234","heartbeat_front_addr":"172.21.15.89:6821/3075913234","state":["exists","up"]},{"osd":8,"uuid":"c4b13bee-ea4d-4344-95fb-2622ecbb8196","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6817","nonce":1813255577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6819","nonce":1813255577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6823","nonce":1813255577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6821","nonce":1813255577}]},"public_addr":"172.21.15.98:6817/1813255577","cluster_addr":"172.21.15.98:6819/1813255577","heartbeat_back_addr":"172.21.15.98:6823/1813255577","heartbeat_front_addr":"172.21.15.98:6821/1813255577","state":["exists","up"]},{"osd":9,"uuid":"eefb0a44-4a36-4e55-9d04-0501bf718db6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6827","nonce":3377749206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6829","nonce":3377749206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6832","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6833","nonce":3377749206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6831","nonce":3377749206}]},"public_addr":"172.21.15.72:6827/3377749206","cluster_addr":"172.21.15.72:6829/3377749206","heartbeat_back_addr":"172.21.15.72:6833/3377749206","heartbeat_front_addr":"172.21.15.72:6831/3377749206","state":["exists","up"]},{"osd":10,"uuid":"8741d25f-40dd-40d7-b040-5cc5e411eb98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6825","nonce":885058786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6827","nonce":885058786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6831","nonce":885058786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6829","nonce":885058786}]},"public_addr":"172.21.15.89:6825/885058786","cluster_addr":"172.21.15.89:6827/885058786","heartbeat_back_addr":"172.21.15.89:6831/885058786","heartbeat_front_addr":"172.21.15.89:6829/885058786","state":["exists","up"]},{"osd":11,"uuid":"8d02e957-eda4-4184-b5c6-5eb3b8b38060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6825","nonce":1517095838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6827","nonce":1517095838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6831","nonce":1517095838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6829","nonce":1517095838}]},"public_addr":"172.21.15.98:6825/1517095838","cluster_addr":"172.21.15.98:6827/1517095838","heartbeat_back_addr":"172.21.15.98:6831/1517095838","heartbeat_front_addr":"172.21.15.98:6829/1517095838","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:28:47.307652+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:11.037135+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:34.619188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:59.350738+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:30:23.787337+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:30:48.994154+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:31:13.690350+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:31:39.628116+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:04.723912+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:30.689741+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:57.229508+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:33:23.154537+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":7}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:0/121087196":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/2451857761":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/1228477995":"2024-03-23T01:27:15.339809+0000","172.21.15.72:6801/1513232913":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/3558958264":"2024-03-23T01:26:50.074871+0000","172.21.15.72:0/1510187198":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6800/3518012916":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6801/3518012916":"2024-03-23T01:26:50.074871+0000","172.21.15.72:0/1372801041":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6800/1513232913":"2024-03-23T01:27:15.339809+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T01:36:28.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.695+0000 7f18e13ae640 1 -- 172.21.15.72:0/2470145942 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f18c80a2fa0 msgr2=0x7f18c80a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:28.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.695+0000 7f18e13ae640 1 --2- 172.21.15.72:0/2470145942 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f18c80a2fa0 0x7f18c80a5460 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f18dc199380 tx=0x7f18cc009260 comp rx=0 tx=0).stop 2024-03-22T01:36:28.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.695+0000 7f18e13ae640 1 -- 172.21.15.72:0/2470145942 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18dc101c20 msgr2=0x7f18dc197e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:28.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.695+0000 7f18e13ae640 1 --2- 172.21.15.72:0/2470145942 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18dc101c20 0x7f18dc197e70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f18c400c970 tx=0x7f18c400ce30 comp rx=0 tx=0).stop 2024-03-22T01:36:28.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.695+0000 7f18e13ae640 1 -- 172.21.15.72:0/2470145942 shutdown_connections 2024-03-22T01:36:28.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.695+0000 7f18e13ae640 1 --2- 172.21.15.72:0/2470145942 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f18c80a2fa0 0x7f18c80a5460 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:28.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.695+0000 7f18e13ae640 1 --2- 172.21.15.72:0/2470145942 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18dc1037d0 0x7f18dc19f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:28.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.695+0000 7f18e13ae640 1 --2- 172.21.15.72:0/2470145942 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18dc102e10 0x7f18dc1983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:28.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.695+0000 7f18e13ae640 1 --2- 172.21.15.72:0/2470145942 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18dc101c20 0x7f18dc197e70 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:28.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.695+0000 7f18e13ae640 1 -- 172.21.15.72:0/2470145942 >> 172.21.15.72:0/2470145942 conn(0x7f18dc0fd1b0 msgr2=0x7f18dc106110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:28.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.695+0000 7f18e13ae640 1 -- 172.21.15.72:0/2470145942 shutdown_connections 2024-03-22T01:36:28.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.695+0000 7f18e13ae640 1 -- 172.21.15.72:0/2470145942 wait complete. 2024-03-22T01:36:28.706 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-22T01:29:37.438503+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-22T01:34:12.610422+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-22T01:34:13.761686+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-22T01:34:17.507039+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-22T01:36:28.706 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-22T01:36:28.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.812+0000 7f4451617640 1 -- 172.21.15.72:0/135881661 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f444c107890 msgr2=0x7f444c109ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:28.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.812+0000 7f4451617640 1 --2- 172.21.15.72:0/135881661 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f444c107890 0x7f444c109ce0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f4444009ca0 tx=0x7f444402c0f0 comp rx=0 tx=0).stop 2024-03-22T01:36:28.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.812+0000 7f4451617640 1 -- 172.21.15.72:0/135881661 shutdown_connections 2024-03-22T01:36:28.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.812+0000 7f4451617640 1 --2- 172.21.15.72:0/135881661 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f444c107890 0x7f444c109ce0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:28.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.812+0000 7f4451617640 1 --2- 172.21.15.72:0/135881661 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f444c104f60 0x7f444c107350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:28.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.812+0000 7f4451617640 1 --2- 172.21.15.72:0/135881661 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f444c102630 0x7f444c104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:28.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.812+0000 7f4451617640 1 -- 172.21.15.72:0/135881661 >> 172.21.15.72:0/135881661 conn(0x7f444c0fc020 msgr2=0x7f444c0fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:28.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.813+0000 7f4451617640 1 -- 172.21.15.72:0/135881661 shutdown_connections 2024-03-22T01:36:28.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.813+0000 7f4451617640 1 -- 172.21.15.72:0/135881661 wait complete. 2024-03-22T01:36:28.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.813+0000 7f4451617640 1 Processor -- start 2024-03-22T01:36:28.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.813+0000 7f4451617640 1 -- start start 2024-03-22T01:36:28.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.813+0000 7f4451617640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f444c102630 0x7f444c101fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:28.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.813+0000 7f4451617640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f444c104f60 0x7f444c1005e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:28.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.813+0000 7f4451617640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f444c107890 0x7f444c100b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:28.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.813+0000 7f4451617640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f444c10d830 con 0x7f444c102630 2024-03-22T01:36:28.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.813+0000 7f4451617640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f444c10d6b0 con 0x7f444c107890 2024-03-22T01:36:28.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.813+0000 7f4451617640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f444c10d9b0 con 0x7f444c104f60 2024-03-22T01:36:28.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.813+0000 7f444affd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f444c102630 0x7f444c101fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:28.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.813+0000 7f444affd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f444c102630 0x7f444c101fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:53670/0 (socket says 172.21.15.72:53670) 2024-03-22T01:36:28.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.813+0000 7f444affd640 1 -- 172.21.15.72:0/992727296 learned_addr learned my addr 172.21.15.72:0/992727296 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:28.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.813+0000 7f444b7fe640 1 --2- 172.21.15.72:0/992727296 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f444c107890 0x7f444c100b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:28.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.813+0000 7f4443fff640 1 --2- 172.21.15.72:0/992727296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f444c104f60 0x7f444c1005e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:28.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.814+0000 7f444affd640 1 -- 172.21.15.72:0/992727296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f444c104f60 msgr2=0x7f444c1005e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:28.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.814+0000 7f444affd640 1 --2- 172.21.15.72:0/992727296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f444c104f60 0x7f444c1005e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:28.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.814+0000 7f444affd640 1 -- 172.21.15.72:0/992727296 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f444c107890 msgr2=0x7f444c100b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:28.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.814+0000 7f444affd640 1 --2- 172.21.15.72:0/992727296 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f444c107890 0x7f444c100b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:28.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.814+0000 7f444affd640 1 -- 172.21.15.72:0/992727296 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f444c1013b0 con 0x7f444c102630 2024-03-22T01:36:28.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.814+0000 7f4443fff640 1 --2- 172.21.15.72:0/992727296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f444c104f60 0x7f444c1005e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:36:28.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.814+0000 7f444affd640 1 --2- 172.21.15.72:0/992727296 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f444c102630 0x7f444c101fa0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f443c00ea60 tx=0x7f443c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:28.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.814+0000 7f4448ff9640 1 -- 172.21.15.72:0/992727296 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f443c00cde0 con 0x7f444c102630 2024-03-22T01:36:28.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.814+0000 7f4448ff9640 1 -- 172.21.15.72:0/992727296 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f443c0035f0 con 0x7f444c102630 2024-03-22T01:36:28.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.814+0000 7f4451617640 1 -- 172.21.15.72:0/992727296 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f444c1a1f10 con 0x7f444c102630 2024-03-22T01:36:28.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.814+0000 7f4448ff9640 1 -- 172.21.15.72:0/992727296 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f443c010670 con 0x7f444c102630 2024-03-22T01:36:28.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.814+0000 7f4451617640 1 -- 172.21.15.72:0/992727296 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f444c1a2320 con 0x7f444c102630 2024-03-22T01:36:28.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.815+0000 7f4451617640 1 -- 172.21.15.72:0/992727296 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f444c0fdb10 con 0x7f444c102630 2024-03-22T01:36:28.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.816+0000 7f4448ff9640 1 -- 172.21.15.72:0/992727296 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7f443c0037a0 con 0x7f444c102630 2024-03-22T01:36:28.818 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.816+0000 7f4448ff9640 1 --2- 172.21.15.72:0/992727296 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f44380a2f20 0x7f44380a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:28.818 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.817+0000 7f4443fff640 1 --2- 172.21.15.72:0/992727296 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f44380a2f20 0x7f44380a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:28.818 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.817+0000 7f4448ff9640 1 -- 172.21.15.72:0/992727296 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f443c0cfb20 con 0x7f444c102630 2024-03-22T01:36:28.818 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.817+0000 7f4443fff640 1 --2- 172.21.15.72:0/992727296 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f44380a2f20 0x7f44380a53e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f444c101d10 tx=0x7f443400a5c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:28.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.821+0000 7f4448ff9640 1 -- 172.21.15.72:0/992727296 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f443c09c580 con 0x7f444c102630 2024-03-22T01:36:28.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.996+0000 7f4451617640 1 -- 172.21.15.72:0/992727296 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f444c061fd0 con 0x7f444c102630 2024-03-22T01:36:28.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.997+0000 7f4448ff9640 1 -- 172.21.15.72:0/992727296 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v95) ==== 93+0+10 (secure 0 0 0) 0x7f443c09be80 con 0x7f444c102630 2024-03-22T01:36:28.996 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-22T01:36:28.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.999+0000 7f4451617640 1 -- 172.21.15.72:0/992727296 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f44380a2f20 msgr2=0x7f44380a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:28.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.999+0000 7f4451617640 1 --2- 172.21.15.72:0/992727296 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f44380a2f20 0x7f44380a53e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f444c101d10 tx=0x7f443400a5c0 comp rx=0 tx=0).stop 2024-03-22T01:36:28.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.999+0000 7f4451617640 1 -- 172.21.15.72:0/992727296 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f444c102630 msgr2=0x7f444c101fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:28.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.999+0000 7f4451617640 1 --2- 172.21.15.72:0/992727296 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f444c102630 0x7f444c101fa0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f443c00ea60 tx=0x7f443c00ef20 comp rx=0 tx=0).stop 2024-03-22T01:36:28.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.999+0000 7f4451617640 1 -- 172.21.15.72:0/992727296 shutdown_connections 2024-03-22T01:36:28.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.999+0000 7f4451617640 1 --2- 172.21.15.72:0/992727296 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f44380a2f20 0x7f44380a53e0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:28.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.999+0000 7f4451617640 1 --2- 172.21.15.72:0/992727296 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f444c107890 0x7f444c100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:28.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.999+0000 7f4451617640 1 --2- 172.21.15.72:0/992727296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f444c104f60 0x7f444c1005e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:28.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.999+0000 7f4451617640 1 --2- 172.21.15.72:0/992727296 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f444c102630 0x7f444c101fa0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:28.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.999+0000 7f4451617640 1 -- 172.21.15.72:0/992727296 >> 172.21.15.72:0/992727296 conn(0x7f444c0fc020 msgr2=0x7f444c102f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:28.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.999+0000 7f4451617640 1 -- 172.21.15.72:0/992727296 shutdown_connections 2024-03-22T01:36:28.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:28.999+0000 7f4451617640 1 -- 172.21.15.72:0/992727296 wait complete. 2024-03-22T01:36:29.010 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-22T01:36:29.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.116+0000 7f1d5ea9a640 1 -- 172.21.15.72:0/2807905404 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1d58100ab0 msgr2=0x7f1d58100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:29.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.116+0000 7f1d5ea9a640 1 --2- 172.21.15.72:0/2807905404 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1d58100ab0 0x7f1d58100ed0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f1d4c009900 tx=0x7f1d4c02bc30 comp rx=0 tx=0).stop 2024-03-22T01:36:29.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.116+0000 7f1d5ea9a640 1 -- 172.21.15.72:0/2807905404 shutdown_connections 2024-03-22T01:36:29.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.116+0000 7f1d5ea9a640 1 --2- 172.21.15.72:0/2807905404 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d58102660 0x7f1d58108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:29.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.116+0000 7f1d5ea9a640 1 --2- 172.21.15.72:0/2807905404 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d58101ca0 0x7f1d58102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:29.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.116+0000 7f1d5ea9a640 1 --2- 172.21.15.72:0/2807905404 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1d58100ab0 0x7f1d58100ed0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:29.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.116+0000 7f1d5ea9a640 1 -- 172.21.15.72:0/2807905404 >> 172.21.15.72:0/2807905404 conn(0x7f1d580fc020 msgr2=0x7f1d580fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:29.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.116+0000 7f1d5ea9a640 1 -- 172.21.15.72:0/2807905404 shutdown_connections 2024-03-22T01:36:29.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.116+0000 7f1d5ea9a640 1 -- 172.21.15.72:0/2807905404 wait complete. 2024-03-22T01:36:29.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.116+0000 7f1d5ea9a640 1 Processor -- start 2024-03-22T01:36:29.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.116+0000 7f1d5ea9a640 1 -- start start 2024-03-22T01:36:29.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.116+0000 7f1d5ea9a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d58100ab0 0x7f1d5806ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:29.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.116+0000 7f1d5ea9a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d58101ca0 0x7f1d5819cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:29.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.116+0000 7f1d5ea9a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1d58102660 0x7f1d5819f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:29.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.116+0000 7f1d5ea9a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f1d5810c700 con 0x7f1d58102660 2024-03-22T01:36:29.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.116+0000 7f1d5ea9a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1d5810c580 con 0x7f1d58100ab0 2024-03-22T01:36:29.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.116+0000 7f1d5ea9a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1d5810c880 con 0x7f1d58101ca0 2024-03-22T01:36:29.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.117+0000 7f1d5d037640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1d58102660 0x7f1d5819f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:29.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.117+0000 7f1d5d037640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1d58102660 0x7f1d5819f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:53692/0 (socket says 172.21.15.72:53692) 2024-03-22T01:36:29.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.117+0000 7f1d5c836640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d58100ab0 0x7f1d5806ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:29.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.117+0000 7f1d5d037640 1 -- 172.21.15.72:0/367083126 learned_addr learned my addr 172.21.15.72:0/367083126 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:29.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.117+0000 7f1d57fff640 1 --2- 172.21.15.72:0/367083126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d58101ca0 0x7f1d5819cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:29.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.117+0000 7f1d5d037640 1 -- 172.21.15.72:0/367083126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d58101ca0 msgr2=0x7f1d5819cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:29.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.117+0000 7f1d5d037640 1 --2- 172.21.15.72:0/367083126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d58101ca0 0x7f1d5819cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:29.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.117+0000 7f1d5d037640 1 -- 172.21.15.72:0/367083126 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d58100ab0 msgr2=0x7f1d5806ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:29.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.117+0000 7f1d5d037640 1 --2- 172.21.15.72:0/367083126 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d58100ab0 0x7f1d5806ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:29.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.117+0000 7f1d5d037640 1 -- 172.21.15.72:0/367083126 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d5819fb50 con 0x7f1d58102660 2024-03-22T01:36:29.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.117+0000 7f1d5c836640 1 --2- 172.21.15.72:0/367083126 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d58100ab0 0x7f1d5806ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:36:29.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.117+0000 7f1d5d037640 1 --2- 172.21.15.72:0/367083126 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1d58102660 0x7f1d5819f240 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f1d5000efd0 tx=0x7f1d5000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:29.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.117+0000 7f1d55ffb640 1 -- 172.21.15.72:0/367083126 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d50003b70 con 0x7f1d58102660 2024-03-22T01:36:29.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.117+0000 7f1d55ffb640 1 -- 172.21.15.72:0/367083126 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d50003d10 con 0x7f1d58102660 2024-03-22T01:36:29.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.117+0000 7f1d55ffb640 1 -- 172.21.15.72:0/367083126 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d500109f0 con 0x7f1d58102660 2024-03-22T01:36:29.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.117+0000 7f1d5ea9a640 1 -- 172.21.15.72:0/367083126 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d5819fde0 con 0x7f1d58102660 2024-03-22T01:36:29.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.117+0000 7f1d5ea9a640 1 -- 172.21.15.72:0/367083126 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d581a0250 con 0x7f1d58102660 2024-03-22T01:36:29.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.118+0000 7f1d5ea9a640 1 -- 172.21.15.72:0/367083126 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d58107dd0 con 0x7f1d58102660 2024-03-22T01:36:29.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.119+0000 7f1d55ffb640 1 -- 172.21.15.72:0/367083126 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7f1d50004020 con 0x7f1d58102660 2024-03-22T01:36:29.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.120+0000 7f1d55ffb640 1 --2- 172.21.15.72:0/367083126 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f1d480a2fa0 0x7f1d480a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:29.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.120+0000 7f1d5c836640 1 --2- 172.21.15.72:0/367083126 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f1d480a2fa0 0x7f1d480a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:29.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.120+0000 7f1d55ffb640 1 -- 172.21.15.72:0/367083126 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f1d500cfcd0 con 0x7f1d58102660 2024-03-22T01:36:29.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.120+0000 7f1d5c836640 1 --2- 172.21.15.72:0/367083126 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f1d480a2fa0 0x7f1d480a5460 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f1d4c000c00 tx=0x7f1d4c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:29.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.124+0000 7f1d55ffb640 1 -- 172.21.15.72:0/367083126 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1d5009c6d0 con 0x7f1d58102660 2024-03-22T01:36:29.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.299+0000 7f1d5ea9a640 1 -- 172.21.15.72:0/367083126 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1d58062830 con 0x7f1d58102660 2024-03-22T01:36:29.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.299+0000 7f1d55ffb640 1 -- 172.21.15.72:0/367083126 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v95) ==== 104+0+11 (secure 0 0 0) 0x7f1d5009bfd0 con 0x7f1d58102660 2024-03-22T01:36:29.299 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-03-22T01:36:29.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.301+0000 7f1d5ea9a640 1 -- 172.21.15.72:0/367083126 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f1d480a2fa0 msgr2=0x7f1d480a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:29.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.301+0000 7f1d5ea9a640 1 --2- 172.21.15.72:0/367083126 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f1d480a2fa0 0x7f1d480a5460 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f1d4c000c00 tx=0x7f1d4c005fd0 comp rx=0 tx=0).stop 2024-03-22T01:36:29.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.301+0000 7f1d5ea9a640 1 -- 172.21.15.72:0/367083126 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1d58102660 msgr2=0x7f1d5819f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:29.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.301+0000 7f1d5ea9a640 1 --2- 172.21.15.72:0/367083126 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1d58102660 0x7f1d5819f240 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f1d5000efd0 tx=0x7f1d5000c6a0 comp rx=0 tx=0).stop 2024-03-22T01:36:29.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.301+0000 7f1d5ea9a640 1 -- 172.21.15.72:0/367083126 shutdown_connections 2024-03-22T01:36:29.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.301+0000 7f1d5ea9a640 1 --2- 172.21.15.72:0/367083126 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f1d480a2fa0 0x7f1d480a5460 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:29.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.301+0000 7f1d5ea9a640 1 --2- 172.21.15.72:0/367083126 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1d58102660 0x7f1d5819f240 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:29.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.301+0000 7f1d5ea9a640 1 --2- 172.21.15.72:0/367083126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d58101ca0 0x7f1d5819cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:29.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.301+0000 7f1d5ea9a640 1 --2- 172.21.15.72:0/367083126 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d58100ab0 0x7f1d5806ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:29.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.301+0000 7f1d5ea9a640 1 -- 172.21.15.72:0/367083126 >> 172.21.15.72:0/367083126 conn(0x7f1d580fc020 msgr2=0x7f1d58104fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:29.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.302+0000 7f1d5ea9a640 1 -- 172.21.15.72:0/367083126 shutdown_connections 2024-03-22T01:36:29.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.302+0000 7f1d5ea9a640 1 -- 172.21.15.72:0/367083126 wait complete. 2024-03-22T01:36:29.311 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:29 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/2772932760' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T01:36:29.311 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:29 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/2571501517' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:36:29.311 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:29 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/2470145942' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:36:29.311 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:29 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/992727296' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T01:36:29.312 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-22T01:36:29.357 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:29 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/2772932760' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T01:36:29.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:29 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/2571501517' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:36:29.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:29 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/2470145942' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:36:29.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:29 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/992727296' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T01:36:29.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.418+0000 7f5fb9cdf640 1 -- 172.21.15.72:0/2758993598 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5fb41078f0 msgr2=0x7f5fb4109ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:29.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.418+0000 7f5fb9cdf640 1 --2- 172.21.15.72:0/2758993598 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5fb41078f0 0x7f5fb4109ce0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f5fac009ca0 tx=0x7f5fac02c0f0 comp rx=0 tx=0).stop 2024-03-22T01:36:29.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.418+0000 7f5fb9cdf640 1 -- 172.21.15.72:0/2758993598 shutdown_connections 2024-03-22T01:36:29.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.418+0000 7f5fb9cdf640 1 --2- 172.21.15.72:0/2758993598 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5fb41078f0 0x7f5fb4109ce0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:29.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.418+0000 7f5fb9cdf640 1 --2- 172.21.15.72:0/2758993598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fb4104fc0 0x7f5fb41073b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:29.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.418+0000 7f5fb9cdf640 1 --2- 172.21.15.72:0/2758993598 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fb4102690 0x7f5fb4104a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:29.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.418+0000 7f5fb9cdf640 1 -- 172.21.15.72:0/2758993598 >> 172.21.15.72:0/2758993598 conn(0x7f5fb40fc020 msgr2=0x7f5fb40fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:29.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.418+0000 7f5fb9cdf640 1 -- 172.21.15.72:0/2758993598 shutdown_connections 2024-03-22T01:36:29.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.418+0000 7f5fb9cdf640 1 -- 172.21.15.72:0/2758993598 wait complete. 2024-03-22T01:36:29.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.419+0000 7f5fb9cdf640 1 Processor -- start 2024-03-22T01:36:29.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.419+0000 7f5fb9cdf640 1 -- start start 2024-03-22T01:36:29.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.419+0000 7f5fb9cdf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fb4102690 0x7f5fb4197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:29.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.419+0000 7f5fb9cdf640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5fb4104fc0 0x7f5fb41983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:29.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.419+0000 7f5fb9cdf640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fb41078f0 0x7f5fb419f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:29.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.419+0000 7f5fb9cdf640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb410d850 con 0x7f5fb4104fc0 2024-03-22T01:36:29.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.419+0000 7f5fb9cdf640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb410d6d0 con 0x7f5fb41078f0 2024-03-22T01:36:29.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.419+0000 7f5fb9cdf640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb410d9d0 con 0x7f5fb4102690 2024-03-22T01:36:29.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.419+0000 7f5fb2ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5fb4104fc0 0x7f5fb41983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:29.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.419+0000 7f5fb2ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5fb4104fc0 0x7f5fb41983f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:53712/0 (socket says 172.21.15.72:53712) 2024-03-22T01:36:29.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.419+0000 7f5fb2ffd640 1 -- 172.21.15.72:0/2184403303 learned_addr learned my addr 172.21.15.72:0/2184403303 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:29.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.419+0000 7f5fb3fff640 1 --2- 172.21.15.72:0/2184403303 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fb41078f0 0x7f5fb419f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:29.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.419+0000 7f5fb37fe640 1 --2- 172.21.15.72:0/2184403303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fb4102690 0x7f5fb4197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:29.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.419+0000 7f5fb2ffd640 1 -- 172.21.15.72:0/2184403303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fb4102690 msgr2=0x7f5fb4197eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:29.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.419+0000 7f5fb2ffd640 1 --2- 172.21.15.72:0/2184403303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fb4102690 0x7f5fb4197eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:29.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.419+0000 7f5fb2ffd640 1 -- 172.21.15.72:0/2184403303 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fb41078f0 msgr2=0x7f5fb419f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:29.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.419+0000 7f5fb2ffd640 1 --2- 172.21.15.72:0/2184403303 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fb41078f0 0x7f5fb419f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:29.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.419+0000 7f5fb2ffd640 1 -- 172.21.15.72:0/2184403303 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fb419fad0 con 0x7f5fb4104fc0 2024-03-22T01:36:29.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.419+0000 7f5fb37fe640 1 --2- 172.21.15.72:0/2184403303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fb4102690 0x7f5fb4197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T01:36:29.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.420+0000 7f5fb2ffd640 1 --2- 172.21.15.72:0/2184403303 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5fb4104fc0 0x7f5fb41983f0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f5fa800e9f0 tx=0x7f5fa800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:29.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.420+0000 7f5fb0ff9640 1 -- 172.21.15.72:0/2184403303 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa800ccc0 con 0x7f5fb4104fc0 2024-03-22T01:36:29.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.420+0000 7f5fb0ff9640 1 -- 172.21.15.72:0/2184403303 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5fa8005500 con 0x7f5fb4104fc0 2024-03-22T01:36:29.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.420+0000 7f5fb9cdf640 1 -- 172.21.15.72:0/2184403303 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fb419fdc0 con 0x7f5fb4104fc0 2024-03-22T01:36:29.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.420+0000 7f5fb0ff9640 1 -- 172.21.15.72:0/2184403303 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa8010690 con 0x7f5fb4104fc0 2024-03-22T01:36:29.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.420+0000 7f5fb9cdf640 1 -- 172.21.15.72:0/2184403303 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fb41a0230 con 0x7f5fb4104fc0 2024-03-22T01:36:29.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.421+0000 7f5fb9cdf640 1 -- 172.21.15.72:0/2184403303 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fb40fdb10 con 0x7f5fb4104fc0 2024-03-22T01:36:29.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:29 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/2772932760' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T01:36:29.423 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:29 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/2571501517' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:36:29.423 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:29 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/2470145942' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:36:29.423 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:29 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/992727296' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T01:36:29.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.422+0000 7f5fb0ff9640 1 -- 172.21.15.72:0/2184403303 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7f5fa80046c0 con 0x7f5fb4104fc0 2024-03-22T01:36:29.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.422+0000 7f5fb0ff9640 1 --2- 172.21.15.72:0/2184403303 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f5f9c0a2f20 0x7f5f9c0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:29.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.423+0000 7f5fb37fe640 1 --2- 172.21.15.72:0/2184403303 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f5f9c0a2f20 0x7f5f9c0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:29.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.423+0000 7f5fb0ff9640 1 -- 172.21.15.72:0/2184403303 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f5fa80cfd40 con 0x7f5fb4104fc0 2024-03-22T01:36:29.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.423+0000 7f5fb37fe640 1 --2- 172.21.15.72:0/2184403303 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f5f9c0a2f20 0x7f5f9c0a53e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5fa4009b50 tx=0x7f5fa4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:29.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.426+0000 7f5fb0ff9640 1 -- 172.21.15.72:0/2184403303 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5fa8053740 con 0x7f5fb4104fc0 2024-03-22T01:36:29.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.600+0000 7f5fb9cdf640 1 -- 172.21.15.72:0/2184403303 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f5fb402d600 con 0x7f5fb4104fc0 2024-03-22T01:36:29.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.601+0000 7f5fb0ff9640 1 -- 172.21.15.72:0/2184403303 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v95) ==== 100+0+11 (secure 0 0 0) 0x7f5fa809c7a0 con 0x7f5fb4104fc0 2024-03-22T01:36:29.600 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-03-22T01:36:29.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.603+0000 7f5fb9cdf640 1 -- 172.21.15.72:0/2184403303 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f5f9c0a2f20 msgr2=0x7f5f9c0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:29.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.603+0000 7f5fb9cdf640 1 --2- 172.21.15.72:0/2184403303 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f5f9c0a2f20 0x7f5f9c0a53e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5fa4009b50 tx=0x7f5fa4005f90 comp rx=0 tx=0).stop 2024-03-22T01:36:29.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.603+0000 7f5fb9cdf640 1 -- 172.21.15.72:0/2184403303 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5fb4104fc0 msgr2=0x7f5fb41983f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:29.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.603+0000 7f5fb9cdf640 1 --2- 172.21.15.72:0/2184403303 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5fb4104fc0 0x7f5fb41983f0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f5fa800e9f0 tx=0x7f5fa800eeb0 comp rx=0 tx=0).stop 2024-03-22T01:36:29.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.603+0000 7f5fb9cdf640 1 -- 172.21.15.72:0/2184403303 shutdown_connections 2024-03-22T01:36:29.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.603+0000 7f5fb9cdf640 1 --2- 172.21.15.72:0/2184403303 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f5f9c0a2f20 0x7f5f9c0a53e0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:29.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.603+0000 7f5fb9cdf640 1 --2- 172.21.15.72:0/2184403303 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fb41078f0 0x7f5fb419f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:29.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.603+0000 7f5fb9cdf640 1 --2- 172.21.15.72:0/2184403303 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5fb4104fc0 0x7f5fb41983f0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:29.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.603+0000 7f5fb9cdf640 1 --2- 172.21.15.72:0/2184403303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fb4102690 0x7f5fb4197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:29.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.603+0000 7f5fb9cdf640 1 -- 172.21.15.72:0/2184403303 >> 172.21.15.72:0/2184403303 conn(0x7f5fb40fc020 msgr2=0x7f5fb4102f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:29.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.603+0000 7f5fb9cdf640 1 -- 172.21.15.72:0/2184403303 shutdown_connections 2024-03-22T01:36:29.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.603+0000 7f5fb9cdf640 1 -- 172.21.15.72:0/2184403303 wait complete. 2024-03-22T01:36:29.613 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-22T01:36:29.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.718+0000 7f95e9be4640 1 -- 172.21.15.72:0/992962527 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f95e4100aa0 msgr2=0x7f95e4100ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:29.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.718+0000 7f95e9be4640 1 --2- 172.21.15.72:0/992962527 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f95e4100aa0 0x7f95e4100ec0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f95d4009900 tx=0x7f95d402bc30 comp rx=0 tx=0).stop 2024-03-22T01:36:29.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.718+0000 7f95e9be4640 1 -- 172.21.15.72:0/992962527 shutdown_connections 2024-03-22T01:36:29.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.718+0000 7f95e9be4640 1 --2- 172.21.15.72:0/992962527 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95e4101c90 0x7f95e4102110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:29.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.718+0000 7f95e9be4640 1 --2- 172.21.15.72:0/992962527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95e4102650 0x7f95e4108ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:29.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.718+0000 7f95e9be4640 1 --2- 172.21.15.72:0/992962527 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f95e4100aa0 0x7f95e4100ec0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:29.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.718+0000 7f95e9be4640 1 -- 172.21.15.72:0/992962527 >> 172.21.15.72:0/992962527 conn(0x7f95e40fbff0 msgr2=0x7f95e40fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:29.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.718+0000 7f95e9be4640 1 -- 172.21.15.72:0/992962527 shutdown_connections 2024-03-22T01:36:29.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.718+0000 7f95e9be4640 1 -- 172.21.15.72:0/992962527 wait complete. 2024-03-22T01:36:29.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.718+0000 7f95e9be4640 1 Processor -- start 2024-03-22T01:36:29.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.718+0000 7f95e9be4640 1 -- start start 2024-03-22T01:36:29.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.718+0000 7f95e9be4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95e4100aa0 0x7f95e406adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:29.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.719+0000 7f95e9be4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f95e4101c90 0x7f95e419cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:29.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.719+0000 7f95e9be4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95e4102650 0x7f95e419f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:29.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.719+0000 7f95e9be4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f95e410c6f0 con 0x7f95e4101c90 2024-03-22T01:36:29.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.719+0000 7f95e9be4640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f95e410c570 con 0x7f95e4100aa0 2024-03-22T01:36:29.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.719+0000 7f95e9be4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f95e410c870 con 0x7f95e4102650 2024-03-22T01:36:29.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.719+0000 7f95e2ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f95e4101c90 0x7f95e419cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:29.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.719+0000 7f95e2ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f95e4101c90 0x7f95e419cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:53732/0 (socket says 172.21.15.72:53732) 2024-03-22T01:36:29.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.719+0000 7f95e2ffd640 1 -- 172.21.15.72:0/3977552314 learned_addr learned my addr 172.21.15.72:0/3977552314 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:29.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.719+0000 7f95e37fe640 1 --2- 172.21.15.72:0/3977552314 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95e4100aa0 0x7f95e406adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:29.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.719+0000 7f95e3fff640 1 --2- 172.21.15.72:0/3977552314 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95e4102650 0x7f95e419f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:29.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.719+0000 7f95e2ffd640 1 -- 172.21.15.72:0/3977552314 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95e4102650 msgr2=0x7f95e419f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:29.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.719+0000 7f95e2ffd640 1 --2- 172.21.15.72:0/3977552314 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95e4102650 0x7f95e419f290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:29.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.719+0000 7f95e2ffd640 1 -- 172.21.15.72:0/3977552314 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95e4100aa0 msgr2=0x7f95e406adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:29.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.719+0000 7f95e2ffd640 1 --2- 172.21.15.72:0/3977552314 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95e4100aa0 0x7f95e406adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:29.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.719+0000 7f95e2ffd640 1 -- 172.21.15.72:0/3977552314 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95e419fba0 con 0x7f95e4101c90 2024-03-22T01:36:29.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.719+0000 7f95e2ffd640 1 --2- 172.21.15.72:0/3977552314 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f95e4101c90 0x7f95e419cc70 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f95d800ea60 tx=0x7f95d800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:29.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.719+0000 7f95e0ff9640 1 -- 172.21.15.72:0/3977552314 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95d800cde0 con 0x7f95e4101c90 2024-03-22T01:36:29.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.719+0000 7f95e9be4640 1 -- 172.21.15.72:0/3977552314 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95e419fe30 con 0x7f95e4101c90 2024-03-22T01:36:29.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.719+0000 7f95e0ff9640 1 -- 172.21.15.72:0/3977552314 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f95d80035f0 con 0x7f95e4101c90 2024-03-22T01:36:29.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.719+0000 7f95e0ff9640 1 -- 172.21.15.72:0/3977552314 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95d8010670 con 0x7f95e4101c90 2024-03-22T01:36:29.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.719+0000 7f95e9be4640 1 -- 172.21.15.72:0/3977552314 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95e41a02f0 con 0x7f95e4101c90 2024-03-22T01:36:29.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.720+0000 7f95e9be4640 1 -- 172.21.15.72:0/3977552314 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95e4107e40 con 0x7f95e4101c90 2024-03-22T01:36:29.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.721+0000 7f95e0ff9640 1 -- 172.21.15.72:0/3977552314 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7f95d80037a0 con 0x7f95e4101c90 2024-03-22T01:36:29.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.722+0000 7f95e0ff9640 1 --2- 172.21.15.72:0/3977552314 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f95cc0a2f20 0x7f95cc0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:29.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.722+0000 7f95e37fe640 1 --2- 172.21.15.72:0/3977552314 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f95cc0a2f20 0x7f95cc0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:29.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.722+0000 7f95e0ff9640 1 -- 172.21.15.72:0/3977552314 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f95d80cfb20 con 0x7f95e4101c90 2024-03-22T01:36:29.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.722+0000 7f95e37fe640 1 --2- 172.21.15.72:0/3977552314 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f95cc0a2f20 0x7f95cc0a53e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f95d4000c00 tx=0x7f95d4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:29.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.726+0000 7f95e0ff9640 1 -- 172.21.15.72:0/3977552314 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f95d809c580 con 0x7f95e4101c90 2024-03-22T01:36:29.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.900+0000 7f95e9be4640 1 -- 172.21.15.72:0/3977552314 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f95e4062830 con 0x7f95e4101c90 2024-03-22T01:36:29.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.900+0000 7f95e0ff9640 1 -- 172.21.15.72:0/3977552314 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v95) ==== 103+0+11 (secure 0 0 0) 0x7f95d809be80 con 0x7f95e4101c90 2024-03-22T01:36:29.899 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-03-22T01:36:29.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.902+0000 7f95e9be4640 1 -- 172.21.15.72:0/3977552314 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f95cc0a2f20 msgr2=0x7f95cc0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:29.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.902+0000 7f95e9be4640 1 --2- 172.21.15.72:0/3977552314 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f95cc0a2f20 0x7f95cc0a53e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f95d4000c00 tx=0x7f95d4005fd0 comp rx=0 tx=0).stop 2024-03-22T01:36:29.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.902+0000 7f95e9be4640 1 -- 172.21.15.72:0/3977552314 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f95e4101c90 msgr2=0x7f95e419cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:29.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.902+0000 7f95e9be4640 1 --2- 172.21.15.72:0/3977552314 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f95e4101c90 0x7f95e419cc70 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f95d800ea60 tx=0x7f95d800ef20 comp rx=0 tx=0).stop 2024-03-22T01:36:29.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.902+0000 7f95e9be4640 1 -- 172.21.15.72:0/3977552314 shutdown_connections 2024-03-22T01:36:29.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.902+0000 7f95e9be4640 1 --2- 172.21.15.72:0/3977552314 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f95cc0a2f20 0x7f95cc0a53e0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:29.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.902+0000 7f95e9be4640 1 --2- 172.21.15.72:0/3977552314 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95e4102650 0x7f95e419f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:29.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.902+0000 7f95e9be4640 1 --2- 172.21.15.72:0/3977552314 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f95e4101c90 0x7f95e419cc70 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:29.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.902+0000 7f95e9be4640 1 --2- 172.21.15.72:0/3977552314 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95e4100aa0 0x7f95e406adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:29.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.902+0000 7f95e9be4640 1 -- 172.21.15.72:0/3977552314 >> 172.21.15.72:0/3977552314 conn(0x7f95e40fbff0 msgr2=0x7f95e4105010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:29.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.902+0000 7f95e9be4640 1 -- 172.21.15.72:0/3977552314 shutdown_connections 2024-03-22T01:36:29.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:29.902+0000 7f95e9be4640 1 -- 172.21.15.72:0/3977552314 wait complete. 2024-03-22T01:36:29.912 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T01:36:30.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.015+0000 7f03c51f7640 1 -- 172.21.15.72:0/3880306554 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f03c0100b20 msgr2=0x7f03c0100f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:30.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.015+0000 7f03c51f7640 1 --2- 172.21.15.72:0/3880306554 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f03c0100b20 0x7f03c0100f40 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f03b0009900 tx=0x7f03b002bc30 comp rx=0 tx=0).stop 2024-03-22T01:36:30.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.015+0000 7f03c51f7640 1 -- 172.21.15.72:0/3880306554 shutdown_connections 2024-03-22T01:36:30.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.015+0000 7f03c51f7640 1 --2- 172.21.15.72:0/3880306554 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03c01026d0 0x7f03c0108f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:30.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.015+0000 7f03c51f7640 1 --2- 172.21.15.72:0/3880306554 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03c0101d10 0x7f03c0102190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:30.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.015+0000 7f03c51f7640 1 --2- 172.21.15.72:0/3880306554 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f03c0100b20 0x7f03c0100f40 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:30.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.015+0000 7f03c51f7640 1 -- 172.21.15.72:0/3880306554 >> 172.21.15.72:0/3880306554 conn(0x7f03c00fc070 msgr2=0x7f03c00fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:30.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.015+0000 7f03c51f7640 1 -- 172.21.15.72:0/3880306554 shutdown_connections 2024-03-22T01:36:30.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.015+0000 7f03c51f7640 1 -- 172.21.15.72:0/3880306554 wait complete. 2024-03-22T01:36:30.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.016+0000 7f03c51f7640 1 Processor -- start 2024-03-22T01:36:30.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.016+0000 7f03c51f7640 1 -- start start 2024-03-22T01:36:30.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.016+0000 7f03c51f7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03c0100b20 0x7f03c0197e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:30.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.016+0000 7f03c51f7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f03c0101d10 0x7f03c01983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:30.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.016+0000 7f03c51f7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03c01026d0 0x7f03c019f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:30.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.016+0000 7f03c51f7640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f03c010c700 con 0x7f03c0101d10 2024-03-22T01:36:30.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.016+0000 7f03c51f7640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f03c010c580 con 0x7f03c0100b20 2024-03-22T01:36:30.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.016+0000 7f03c51f7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f03c010c880 con 0x7f03c01026d0 2024-03-22T01:36:30.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.016+0000 7f03b7fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f03c0101d10 0x7f03c01983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:30.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.016+0000 7f03b7fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f03c0101d10 0x7f03c01983a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:53750/0 (socket says 172.21.15.72:53750) 2024-03-22T01:36:30.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.016+0000 7f03b7fff640 1 -- 172.21.15.72:0/3918677331 learned_addr learned my addr 172.21.15.72:0/3918677331 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:30.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.016+0000 7f03bed9d640 1 --2- 172.21.15.72:0/3918677331 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03c0100b20 0x7f03c0197e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:30.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.016+0000 7f03bf59e640 1 --2- 172.21.15.72:0/3918677331 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03c01026d0 0x7f03c019f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:30.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.016+0000 7f03b7fff640 1 -- 172.21.15.72:0/3918677331 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03c01026d0 msgr2=0x7f03c019f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:30.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.016+0000 7f03b7fff640 1 --2- 172.21.15.72:0/3918677331 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03c01026d0 0x7f03c019f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:30.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.016+0000 7f03b7fff640 1 -- 172.21.15.72:0/3918677331 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03c0100b20 msgr2=0x7f03c0197e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:30.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.016+0000 7f03b7fff640 1 --2- 172.21.15.72:0/3918677331 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03c0100b20 0x7f03c0197e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:30.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.016+0000 7f03b7fff640 1 -- 172.21.15.72:0/3918677331 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03c019fa80 con 0x7f03c0101d10 2024-03-22T01:36:30.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.016+0000 7f03bf59e640 1 --2- 172.21.15.72:0/3918677331 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03c01026d0 0x7f03c019f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T01:36:30.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.017+0000 7f03b7fff640 1 --2- 172.21.15.72:0/3918677331 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f03c0101d10 0x7f03c01983a0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f03a800c920 tx=0x7f03a800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:30.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.017+0000 7f03bcd99640 1 -- 172.21.15.72:0/3918677331 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03a8007c70 con 0x7f03c0101d10 2024-03-22T01:36:30.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.017+0000 7f03c51f7640 1 -- 172.21.15.72:0/3918677331 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03c019fd70 con 0x7f03c0101d10 2024-03-22T01:36:30.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.017+0000 7f03bcd99640 1 -- 172.21.15.72:0/3918677331 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03a8007e10 con 0x7f03c0101d10 2024-03-22T01:36:30.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.017+0000 7f03bcd99640 1 -- 172.21.15.72:0/3918677331 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03a8018890 con 0x7f03c0101d10 2024-03-22T01:36:30.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.017+0000 7f03c51f7640 1 -- 172.21.15.72:0/3918677331 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03c01a0260 con 0x7f03c0101d10 2024-03-22T01:36:30.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.018+0000 7f03c51f7640 1 -- 172.21.15.72:0/3918677331 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03c01a04f0 con 0x7f03c0101d10 2024-03-22T01:36:30.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.019+0000 7f03bcd99640 1 -- 172.21.15.72:0/3918677331 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7f03a8018a70 con 0x7f03c0101d10 2024-03-22T01:36:30.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.019+0000 7f03bcd99640 1 --2- 172.21.15.72:0/3918677331 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f03ac0a2f20 0x7f03ac0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:30.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.019+0000 7f03bed9d640 1 --2- 172.21.15.72:0/3918677331 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f03ac0a2f20 0x7f03ac0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:30.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.019+0000 7f03bcd99640 1 -- 172.21.15.72:0/3918677331 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f03a80cf8f0 con 0x7f03c0101d10 2024-03-22T01:36:30.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.020+0000 7f03bed9d640 1 --2- 172.21.15.72:0/3918677331 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f03ac0a2f20 0x7f03ac0a53e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f03b0000c00 tx=0x7f03b0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:30.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.024+0000 7f03bcd99640 1 -- 172.21.15.72:0/3918677331 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f03a8014030 con 0x7f03c0101d10 2024-03-22T01:36:30.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.238+0000 7f03c51f7640 1 -- 172.21.15.72:0/3918677331 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f03c01a06c0 con 0x7f03c0101d10 2024-03-22T01:36:30.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.238+0000 7f03bcd99640 1 -- 172.21.15.72:0/3918677331 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11785 (secure 0 0 0) 0x7f03a809c350 con 0x7f03c0101d10 2024-03-22T01:36:30.238 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:36:30.238 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6835/3077216308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3077216308},{"type":"v1","addr":"172.21.15.72:6835","nonce":3077216308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2460254143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":2460254143},{"type":"v1","addr":"172.21.15.72:6837","nonce":2460254143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6839/2026432819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6838","nonce":2026432819},{"type":"v1","addr":"172.21.15.72:6839","nonce":2026432819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6841/400094545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6840","nonce":400094545},{"type":"v1","addr":"172.21.15.72:6841","nonce":400094545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6833/1578701171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":1578701171},{"type":"v1","addr":"172.21.15.89:6833","nonce":1578701171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/2703047213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2703047213},{"type":"v1","addr":"172.21.15.98:6833","nonce":2703047213}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2589468787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2589468787},{"type":"v1","addr":"172.21.15.89:6835","nonce":2589468787}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/92657636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":92657636},{"type":"v1","addr":"172.21.15.98:6835","nonce":92657636}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1662941782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1662941782},{"type":"v1","addr":"172.21.15.89:6837","nonce":1662941782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24490,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/781860371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":781860371},{"type":"v1","addr":"172.21.15.98:6837","nonce":781860371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-22T01:34:14.815677+0000","modified":"2024-03-22T01:34:39.842959+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24524},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3192173650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3192173650},{"type":"v1","addr":"172.21.15.98:6839","nonce":3192173650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6839/149200280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":149200280},{"type":"v1","addr":"172.21.15.89:6839","nonce":149200280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24524]},"id":1}]} 2024-03-22T01:36:30.238 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 21 2024-03-22T01:36:30.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.240+0000 7f03c51f7640 1 -- 172.21.15.72:0/3918677331 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f03ac0a2f20 msgr2=0x7f03ac0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:30.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.240+0000 7f03c51f7640 1 --2- 172.21.15.72:0/3918677331 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f03ac0a2f20 0x7f03ac0a53e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f03b0000c00 tx=0x7f03b0005fd0 comp rx=0 tx=0).stop 2024-03-22T01:36:30.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.240+0000 7f03c51f7640 1 -- 172.21.15.72:0/3918677331 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f03c0101d10 msgr2=0x7f03c01983a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:30.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.240+0000 7f03c51f7640 1 --2- 172.21.15.72:0/3918677331 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f03c0101d10 0x7f03c01983a0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f03a800c920 tx=0x7f03a800cde0 comp rx=0 tx=0).stop 2024-03-22T01:36:30.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.240+0000 7f03c51f7640 1 -- 172.21.15.72:0/3918677331 shutdown_connections 2024-03-22T01:36:30.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.240+0000 7f03c51f7640 1 --2- 172.21.15.72:0/3918677331 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f03ac0a2f20 0x7f03ac0a53e0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:30.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.240+0000 7f03c51f7640 1 --2- 172.21.15.72:0/3918677331 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03c01026d0 0x7f03c019f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:30.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.240+0000 7f03c51f7640 1 --2- 172.21.15.72:0/3918677331 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f03c0101d10 0x7f03c01983a0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:30.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.240+0000 7f03c51f7640 1 --2- 172.21.15.72:0/3918677331 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03c0100b20 0x7f03c0197e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:30.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.240+0000 7f03c51f7640 1 -- 172.21.15.72:0/3918677331 >> 172.21.15.72:0/3918677331 conn(0x7f03c00fc070 msgr2=0x7f03c00fdae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:30.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.241+0000 7f03c51f7640 1 -- 172.21.15.72:0/3918677331 shutdown_connections 2024-03-22T01:36:30.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.241+0000 7f03c51f7640 1 -- 172.21.15.72:0/3918677331 wait complete. 2024-03-22T01:36:30.250 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:30 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/367083126' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T01:36:30.250 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:30 smithi072 ceph-mon[32414]: pgmap v362: 129 pgs: 129 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 865 B/s rd, 1 op/s 2024-03-22T01:36:30.250 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:30 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/2184403303' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T01:36:30.250 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:30 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/3977552314' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T01:36:30.251 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T01:36:30.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.354+0000 7feb75bbc640 1 -- 172.21.15.72:0/220240640 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feb700ffe90 msgr2=0x7feb701002b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:30.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.354+0000 7feb75bbc640 1 --2- 172.21.15.72:0/220240640 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feb700ffe90 0x7feb701002b0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7feb60009930 tx=0x7feb6002bc30 comp rx=0 tx=0).stop 2024-03-22T01:36:30.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.354+0000 7feb75bbc640 1 -- 172.21.15.72:0/220240640 shutdown_connections 2024-03-22T01:36:30.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.354+0000 7feb75bbc640 1 --2- 172.21.15.72:0/220240640 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb70102e60 0x7feb701032e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:30.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.354+0000 7feb75bbc640 1 --2- 172.21.15.72:0/220240640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb70103820 0x7feb7010a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:30.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.354+0000 7feb75bbc640 1 --2- 172.21.15.72:0/220240640 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feb700ffe90 0x7feb701002b0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:30.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.354+0000 7feb75bbc640 1 -- 172.21.15.72:0/220240640 >> 172.21.15.72:0/220240640 conn(0x7feb700fb7c0 msgr2=0x7feb700fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:30.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.354+0000 7feb75bbc640 1 -- 172.21.15.72:0/220240640 shutdown_connections 2024-03-22T01:36:30.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.354+0000 7feb75bbc640 1 -- 172.21.15.72:0/220240640 wait complete. 2024-03-22T01:36:30.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.355+0000 7feb75bbc640 1 Processor -- start 2024-03-22T01:36:30.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.355+0000 7feb75bbc640 1 -- start start 2024-03-22T01:36:30.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.355+0000 7feb75bbc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feb700ffe90 0x7feb70197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:30.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.355+0000 7feb75bbc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb70102e60 0x7feb70198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:30.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.355+0000 7feb75bbc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb70103820 0x7feb7019f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:30.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.355+0000 7feb75bbc640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7feb7010d860 con 0x7feb700ffe90 2024-03-22T01:36:30.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.355+0000 7feb75bbc640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7feb7010d6e0 con 0x7feb70103820 2024-03-22T01:36:30.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.355+0000 7feb75bbc640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7feb7010d9e0 con 0x7feb70102e60 2024-03-22T01:36:30.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.355+0000 7feb6f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feb700ffe90 0x7feb70197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:30.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.355+0000 7feb6f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feb700ffe90 0x7feb70197f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:53758/0 (socket says 172.21.15.72:53758) 2024-03-22T01:36:30.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.355+0000 7feb6f7fe640 1 -- 172.21.15.72:0/305124512 learned_addr learned my addr 172.21.15.72:0/305124512 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:30.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.355+0000 7feb6ffff640 1 --2- 172.21.15.72:0/305124512 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb70103820 0x7feb7019f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:30.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.356+0000 7feb66ffd640 1 --2- 172.21.15.72:0/305124512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb70102e60 0x7feb70198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:30.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.356+0000 7feb6f7fe640 1 -- 172.21.15.72:0/305124512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb70102e60 msgr2=0x7feb70198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:30.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.356+0000 7feb6f7fe640 1 --2- 172.21.15.72:0/305124512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb70102e60 0x7feb70198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:30.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.356+0000 7feb6f7fe640 1 -- 172.21.15.72:0/305124512 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb70103820 msgr2=0x7feb7019f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:30.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.356+0000 7feb6f7fe640 1 --2- 172.21.15.72:0/305124512 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb70103820 0x7feb7019f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:30.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.356+0000 7feb6f7fe640 1 -- 172.21.15.72:0/305124512 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb7019fb50 con 0x7feb700ffe90 2024-03-22T01:36:30.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.356+0000 7feb66ffd640 1 --2- 172.21.15.72:0/305124512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb70102e60 0x7feb70198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:36:30.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.356+0000 7feb6f7fe640 1 --2- 172.21.15.72:0/305124512 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feb700ffe90 0x7feb70197f30 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7feb6002c7b0 tx=0x7feb6002c6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:30.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.356+0000 7feb6d7fa640 1 -- 172.21.15.72:0/305124512 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb60041070 con 0x7feb700ffe90 2024-03-22T01:36:30.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.356+0000 7feb75bbc640 1 -- 172.21.15.72:0/305124512 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb7019fde0 con 0x7feb700ffe90 2024-03-22T01:36:30.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.356+0000 7feb6d7fa640 1 -- 172.21.15.72:0/305124512 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb600054e0 con 0x7feb700ffe90 2024-03-22T01:36:30.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.356+0000 7feb6d7fa640 1 -- 172.21.15.72:0/305124512 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb60034440 con 0x7feb700ffe90 2024-03-22T01:36:30.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.356+0000 7feb75bbc640 1 -- 172.21.15.72:0/305124512 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb701a0210 con 0x7feb700ffe90 2024-03-22T01:36:30.359 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:30 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/367083126' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T01:36:30.359 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:30 smithi089 ceph-mon[35770]: pgmap v362: 129 pgs: 129 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 865 B/s rd, 1 op/s 2024-03-22T01:36:30.359 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:30 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/2184403303' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T01:36:30.359 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:30 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/3977552314' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T01:36:30.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.357+0000 7feb75bbc640 1 -- 172.21.15.72:0/305124512 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb70109010 con 0x7feb700ffe90 2024-03-22T01:36:30.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.358+0000 7feb6d7fa640 1 -- 172.21.15.72:0/305124512 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7feb60044070 con 0x7feb700ffe90 2024-03-22T01:36:30.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.359+0000 7feb6d7fa640 1 --2- 172.21.15.72:0/305124512 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7feb500a2f20 0x7feb500a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:30.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.359+0000 7feb66ffd640 1 --2- 172.21.15.72:0/305124512 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7feb500a2f20 0x7feb500a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:30.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.359+0000 7feb6d7fa640 1 -- 172.21.15.72:0/305124512 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7feb600f1330 con 0x7feb700ffe90 2024-03-22T01:36:30.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.359+0000 7feb66ffd640 1 --2- 172.21.15.72:0/305124512 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7feb500a2f20 0x7feb500a53e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7feb70199440 tx=0x7feb58005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:30.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.363+0000 7feb6d7fa640 1 -- 172.21.15.72:0/305124512 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7feb600bdd90 con 0x7feb700ffe90 2024-03-22T01:36:30.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:30 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/367083126' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T01:36:30.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:30 smithi098 ceph-mon[36468]: pgmap v362: 129 pgs: 129 active+clean; 640 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 865 B/s rd, 1 op/s 2024-03-22T01:36:30.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:30 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/2184403303' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T01:36:30.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:30 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/3977552314' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T01:36:30.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.539+0000 7feb75bbc640 1 -- 172.21.15.72:0/305124512 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7feb70062830 con 0x7feb700ffe90 2024-03-22T01:36:30.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.540+0000 7feb6d7fa640 1 -- 172.21.15.72:0/305124512 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) ==== 74+0+24759 (secure 0 0 0) 0x7feb600bd690 con 0x7feb700ffe90 2024-03-22T01:36:30.540 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:36:30.540 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":95,"fsid":"022a99bc-e7eb-11ee-95cd-87774f69a715","created":"2024-03-22T01:26:23.893672+0000","modified":"2024-03-22T01:36:17.626966+0000","last_up_change":"2024-03-22T01:33:25.223684+0000","last_in_change":"2024-03-22T01:33:05.010331+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T01:29:37.438503+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-22T01:34:12.610422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-22T01:34:13.761686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-22T01:34:17.507039+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"16c15bb0-db92-4191-9063-38b4890e81e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6803","nonce":661906791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6805","nonce":661906791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6809","nonce":661906791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6807","nonce":661906791}]},"public_addr":"172.21.15.72:6803/661906791","cluster_addr":"172.21.15.72:6805/661906791","heartbeat_back_addr":"172.21.15.72:6809/661906791","heartbeat_front_addr":"172.21.15.72:6807/661906791","state":["exists","up"]},{"osd":1,"uuid":"981290ba-97dc-496b-aa22-d51704184b10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6801","nonce":2870590550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6803","nonce":2870590550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6807","nonce":2870590550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6805","nonce":2870590550}]},"public_addr":"172.21.15.89:6801/2870590550","cluster_addr":"172.21.15.89:6803/2870590550","heartbeat_back_addr":"172.21.15.89:6807/2870590550","heartbeat_front_addr":"172.21.15.89:6805/2870590550","state":["exists","up"]},{"osd":2,"uuid":"f07ed658-f545-46a1-8edd-8b1d5d49885c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6801","nonce":3865778118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6803","nonce":3865778118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6807","nonce":3865778118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6805","nonce":3865778118}]},"public_addr":"172.21.15.98:6801/3865778118","cluster_addr":"172.21.15.98:6803/3865778118","heartbeat_back_addr":"172.21.15.98:6807/3865778118","heartbeat_front_addr":"172.21.15.98:6805/3865778118","state":["exists","up"]},{"osd":3,"uuid":"ccc4bc40-2c48-46d2-8548-0c05b44f9f1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6811","nonce":1865951395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6813","nonce":1865951395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6817","nonce":1865951395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6815","nonce":1865951395}]},"public_addr":"172.21.15.72:6811/1865951395","cluster_addr":"172.21.15.72:6813/1865951395","heartbeat_back_addr":"172.21.15.72:6817/1865951395","heartbeat_front_addr":"172.21.15.72:6815/1865951395","state":["exists","up"]},{"osd":4,"uuid":"1812439f-810a-4ed2-9395-c00c9a085a13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6809","nonce":2683760795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6811","nonce":2683760795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6815","nonce":2683760795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6813","nonce":2683760795}]},"public_addr":"172.21.15.89:6809/2683760795","cluster_addr":"172.21.15.89:6811/2683760795","heartbeat_back_addr":"172.21.15.89:6815/2683760795","heartbeat_front_addr":"172.21.15.89:6813/2683760795","state":["exists","up"]},{"osd":5,"uuid":"e78954d8-f4e3-4d58-8536-d2d59c843386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6809","nonce":2635838572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6811","nonce":2635838572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6815","nonce":2635838572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6813","nonce":2635838572}]},"public_addr":"172.21.15.98:6809/2635838572","cluster_addr":"172.21.15.98:6811/2635838572","heartbeat_back_addr":"172.21.15.98:6815/2635838572","heartbeat_front_addr":"172.21.15.98:6813/2635838572","state":["exists","up"]},{"osd":6,"uuid":"dc6c8be2-b64c-4959-8015-7b0ea6d25599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6819","nonce":3158982661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6821","nonce":3158982661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6825","nonce":3158982661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6823","nonce":3158982661}]},"public_addr":"172.21.15.72:6819/3158982661","cluster_addr":"172.21.15.72:6821/3158982661","heartbeat_back_addr":"172.21.15.72:6825/3158982661","heartbeat_front_addr":"172.21.15.72:6823/3158982661","state":["exists","up"]},{"osd":7,"uuid":"7a73469f-cf3a-4306-ae93-ea46e21ebbbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6817","nonce":3075913234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6819","nonce":3075913234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6823","nonce":3075913234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6821","nonce":3075913234}]},"public_addr":"172.21.15.89:6817/3075913234","cluster_addr":"172.21.15.89:6819/3075913234","heartbeat_back_addr":"172.21.15.89:6823/3075913234","heartbeat_front_addr":"172.21.15.89:6821/3075913234","state":["exists","up"]},{"osd":8,"uuid":"c4b13bee-ea4d-4344-95fb-2622ecbb8196","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6817","nonce":1813255577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6819","nonce":1813255577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6823","nonce":1813255577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6821","nonce":1813255577}]},"public_addr":"172.21.15.98:6817/1813255577","cluster_addr":"172.21.15.98:6819/1813255577","heartbeat_back_addr":"172.21.15.98:6823/1813255577","heartbeat_front_addr":"172.21.15.98:6821/1813255577","state":["exists","up"]},{"osd":9,"uuid":"eefb0a44-4a36-4e55-9d04-0501bf718db6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6827","nonce":3377749206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6829","nonce":3377749206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6832","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6833","nonce":3377749206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6831","nonce":3377749206}]},"public_addr":"172.21.15.72:6827/3377749206","cluster_addr":"172.21.15.72:6829/3377749206","heartbeat_back_addr":"172.21.15.72:6833/3377749206","heartbeat_front_addr":"172.21.15.72:6831/3377749206","state":["exists","up"]},{"osd":10,"uuid":"8741d25f-40dd-40d7-b040-5cc5e411eb98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6825","nonce":885058786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6827","nonce":885058786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6831","nonce":885058786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6829","nonce":885058786}]},"public_addr":"172.21.15.89:6825/885058786","cluster_addr":"172.21.15.89:6827/885058786","heartbeat_back_addr":"172.21.15.89:6831/885058786","heartbeat_front_addr":"172.21.15.89:6829/885058786","state":["exists","up"]},{"osd":11,"uuid":"8d02e957-eda4-4184-b5c6-5eb3b8b38060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6825","nonce":1517095838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6827","nonce":1517095838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6831","nonce":1517095838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6829","nonce":1517095838}]},"public_addr":"172.21.15.98:6825/1517095838","cluster_addr":"172.21.15.98:6827/1517095838","heartbeat_back_addr":"172.21.15.98:6831/1517095838","heartbeat_front_addr":"172.21.15.98:6829/1517095838","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:28:47.307652+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:11.037135+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:34.619188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:59.350738+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:30:23.787337+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:30:48.994154+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:31:13.690350+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:31:39.628116+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:04.723912+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:30.689741+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:57.229508+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:33:23.154537+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":7}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:0/121087196":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/2451857761":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/1228477995":"2024-03-23T01:27:15.339809+0000","172.21.15.72:6801/1513232913":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/3558958264":"2024-03-23T01:26:50.074871+0000","172.21.15.72:0/1510187198":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6800/3518012916":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6801/3518012916":"2024-03-23T01:26:50.074871+0000","172.21.15.72:0/1372801041":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6800/1513232913":"2024-03-23T01:27:15.339809+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T01:36:30.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.542+0000 7feb75bbc640 1 -- 172.21.15.72:0/305124512 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7feb500a2f20 msgr2=0x7feb500a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:30.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.542+0000 7feb75bbc640 1 --2- 172.21.15.72:0/305124512 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7feb500a2f20 0x7feb500a53e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7feb70199440 tx=0x7feb58005f90 comp rx=0 tx=0).stop 2024-03-22T01:36:30.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.542+0000 7feb75bbc640 1 -- 172.21.15.72:0/305124512 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feb700ffe90 msgr2=0x7feb70197f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:30.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.542+0000 7feb75bbc640 1 --2- 172.21.15.72:0/305124512 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feb700ffe90 0x7feb70197f30 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7feb6002c7b0 tx=0x7feb6002c6d0 comp rx=0 tx=0).stop 2024-03-22T01:36:30.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.542+0000 7feb75bbc640 1 -- 172.21.15.72:0/305124512 shutdown_connections 2024-03-22T01:36:30.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.542+0000 7feb75bbc640 1 --2- 172.21.15.72:0/305124512 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7feb500a2f20 0x7feb500a53e0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:30.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.542+0000 7feb75bbc640 1 --2- 172.21.15.72:0/305124512 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb70103820 0x7feb7019f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:30.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.542+0000 7feb75bbc640 1 --2- 172.21.15.72:0/305124512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb70102e60 0x7feb70198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:30.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.542+0000 7feb75bbc640 1 --2- 172.21.15.72:0/305124512 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feb700ffe90 0x7feb70197f30 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:30.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.542+0000 7feb75bbc640 1 -- 172.21.15.72:0/305124512 >> 172.21.15.72:0/305124512 conn(0x7feb700fb7c0 msgr2=0x7feb701061e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:30.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.543+0000 7feb75bbc640 1 -- 172.21.15.72:0/305124512 shutdown_connections 2024-03-22T01:36:30.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.543+0000 7feb75bbc640 1 -- 172.21.15.72:0/305124512 wait complete. 2024-03-22T01:36:30.554 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 21, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-22T01:34:14.815677+0000', 'modified': '2024-03-22T01:34:39.842959+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24524}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24517': {'gid': 24517, 'name': 'i', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.98:6839/3192173650', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6838', 'nonce': 3192173650}, {'type': 'v1', 'addr': '172.21.15.98:6839', 'nonce': 3192173650}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24524': {'gid': 24524, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.89:6839/149200280', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6838', 'nonce': 149200280}, {'type': 'v1', 'addr': '172.21.15.89:6839', 'nonce': 149200280}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24524, 'qdb_cluster': [24524]} 2024-03-22T01:36:30.554 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-22T01:36:30.554 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-22T01:36:30.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.657+0000 7eff3129d640 1 -- 172.21.15.72:0/3419636375 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff2c1037f0 msgr2=0x7eff2c10a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:30.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.657+0000 7eff3129d640 1 --2- 172.21.15.72:0/3419636375 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff2c1037f0 0x7eff2c10a080 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7eff24009cd0 tx=0x7eff2402c0f0 comp rx=0 tx=0).stop 2024-03-22T01:36:30.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.657+0000 7eff3129d640 1 -- 172.21.15.72:0/3419636375 shutdown_connections 2024-03-22T01:36:30.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.657+0000 7eff3129d640 1 --2- 172.21.15.72:0/3419636375 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff2c1037f0 0x7eff2c10a080 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:30.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.657+0000 7eff3129d640 1 --2- 172.21.15.72:0/3419636375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff2c102e30 0x7eff2c1032b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:30.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.657+0000 7eff3129d640 1 --2- 172.21.15.72:0/3419636375 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff2c0ffe60 0x7eff2c100280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:30.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.657+0000 7eff3129d640 1 -- 172.21.15.72:0/3419636375 >> 172.21.15.72:0/3419636375 conn(0x7eff2c0fb7b0 msgr2=0x7eff2c0fdbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:30.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.657+0000 7eff3129d640 1 -- 172.21.15.72:0/3419636375 shutdown_connections 2024-03-22T01:36:30.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.657+0000 7eff3129d640 1 -- 172.21.15.72:0/3419636375 wait complete. 2024-03-22T01:36:30.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.657+0000 7eff3129d640 1 Processor -- start 2024-03-22T01:36:30.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.658+0000 7eff3129d640 1 -- start start 2024-03-22T01:36:30.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.658+0000 7eff3129d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff2c0ffe60 0x7eff2c197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:30.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.658+0000 7eff3129d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff2c102e30 0x7eff2c1983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:30.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.658+0000 7eff3129d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff2c1037f0 0x7eff2c19f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:30.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.658+0000 7eff3129d640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7eff2c10d810 con 0x7eff2c0ffe60 2024-03-22T01:36:30.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.658+0000 7eff3129d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7eff2c10d690 con 0x7eff2c1037f0 2024-03-22T01:36:30.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.658+0000 7eff3129d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7eff2c10d990 con 0x7eff2c102e30 2024-03-22T01:36:30.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.658+0000 7eff2affd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff2c0ffe60 0x7eff2c197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:30.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.658+0000 7eff2affd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff2c0ffe60 0x7eff2c197eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54134/0 (socket says 172.21.15.72:54134) 2024-03-22T01:36:30.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.658+0000 7eff2affd640 1 -- 172.21.15.72:0/4030586055 learned_addr learned my addr 172.21.15.72:0/4030586055 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:30.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.658+0000 7eff2b7fe640 1 --2- 172.21.15.72:0/4030586055 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff2c1037f0 0x7eff2c19f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:30.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.658+0000 7eff2affd640 1 -- 172.21.15.72:0/4030586055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff2c102e30 msgr2=0x7eff2c1983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:30.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.658+0000 7eff227fc640 1 --2- 172.21.15.72:0/4030586055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff2c102e30 0x7eff2c1983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:30.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.658+0000 7eff2affd640 1 --2- 172.21.15.72:0/4030586055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff2c102e30 0x7eff2c1983f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:30.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.658+0000 7eff2affd640 1 -- 172.21.15.72:0/4030586055 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff2c1037f0 msgr2=0x7eff2c19f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:30.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.658+0000 7eff2affd640 1 --2- 172.21.15.72:0/4030586055 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff2c1037f0 0x7eff2c19f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:30.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.658+0000 7eff2affd640 1 -- 172.21.15.72:0/4030586055 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff2c19fad0 con 0x7eff2c0ffe60 2024-03-22T01:36:30.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.658+0000 7eff227fc640 1 --2- 172.21.15.72:0/4030586055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff2c102e30 0x7eff2c1983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T01:36:30.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.658+0000 7eff2affd640 1 --2- 172.21.15.72:0/4030586055 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff2c0ffe60 0x7eff2c197eb0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7eff1c00e9c0 tx=0x7eff1c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:30.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.659+0000 7eff28ff9640 1 -- 172.21.15.72:0/4030586055 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff1c00cde0 con 0x7eff2c0ffe60 2024-03-22T01:36:30.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.659+0000 7eff28ff9640 1 -- 172.21.15.72:0/4030586055 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff1c010910 con 0x7eff2c0ffe60 2024-03-22T01:36:30.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.659+0000 7eff3129d640 1 -- 172.21.15.72:0/4030586055 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff2c19fdc0 con 0x7eff2c0ffe60 2024-03-22T01:36:30.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.659+0000 7eff28ff9640 1 -- 172.21.15.72:0/4030586055 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff1c018a00 con 0x7eff2c0ffe60 2024-03-22T01:36:30.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.659+0000 7eff3129d640 1 -- 172.21.15.72:0/4030586055 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff2c1a0230 con 0x7eff2c0ffe60 2024-03-22T01:36:30.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.661+0000 7eff28ff9640 1 -- 172.21.15.72:0/4030586055 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7eff1c01f020 con 0x7eff2c0ffe60 2024-03-22T01:36:30.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.661+0000 7eff33f67740 1 Processor -- start 2024-03-22T01:36:30.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.661+0000 7eff33f67740 1 -- start start 2024-03-22T01:36:30.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.661+0000 7eff28ff9640 1 --2- 172.21.15.72:0/4030586055 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7eff180a2f20 0x7eff180a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:30.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.661+0000 7eff2b7fe640 1 --2- 172.21.15.72:0/4030586055 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7eff180a2f20 0x7eff180a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:30.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.661+0000 7eff28ff9640 1 -- 172.21.15.72:0/4030586055 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7eff1c0cfa40 con 0x7eff2c0ffe60 2024-03-22T01:36:30.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.661+0000 7eff33f67740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561b83484c80 0x561b834850a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:30.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.661+0000 7eff33f67740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b83485670 0x561b8348a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:30.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.661+0000 7eff33f67740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561b8348a7e0 0x561b8348cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:30.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.661+0000 7eff33f67740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x561b833e9210 con 0x561b8348a7e0 2024-03-22T01:36:30.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.661+0000 7eff33f67740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x561b83391c80 con 0x561b83485670 2024-03-22T01:36:30.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.661+0000 7eff33f67740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561b833e51c0 con 0x561b83484c80 2024-03-22T01:36:30.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.662+0000 7eff2b7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561b8348a7e0 0x561b8348cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:30.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.662+0000 7eff2b7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561b8348a7e0 0x561b8348cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54138/0 (socket says 172.21.15.72:54138) 2024-03-22T01:36:30.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.662+0000 7eff2b7fe640 1 -- 172.21.15.72:0/3186161782 learned_addr learned my addr 172.21.15.72:0/3186161782 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:30.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.662+0000 7eff227fc640 1 --2- 172.21.15.72:0/3186161782 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b83485670 0x561b8348a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:30.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.662+0000 7eff2affd640 1 --2- 172.21.15.72:0/3186161782 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561b83484c80 0x561b834850a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:30.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.662+0000 7eff2b7fe640 1 --2- 172.21.15.72:0/4030586055 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7eff180a2f20 0x7eff180a53e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7eff24006010 tx=0x7eff2402ce20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:30.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.662+0000 7eff227fc640 1 -- 172.21.15.72:0/3186161782 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561b83484c80 msgr2=0x561b834850a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:36:30.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.662+0000 7eff227fc640 1 --2- 172.21.15.72:0/3186161782 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561b83484c80 0x561b834850a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:30.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.662+0000 7eff227fc640 1 -- 172.21.15.72:0/3186161782 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561b8348a7e0 msgr2=0x561b8348cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:36:30.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.662+0000 7eff227fc640 1 --2- 172.21.15.72:0/3186161782 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561b8348a7e0 0x561b8348cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:30.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.662+0000 7eff227fc640 1 -- 172.21.15.72:0/3186161782 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b830fd2d0 con 0x561b83485670 2024-03-22T01:36:30.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.662+0000 7eff2b7fe640 1 --2- 172.21.15.72:0/3186161782 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561b8348a7e0 0x561b8348cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:36:30.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.662+0000 7eff227fc640 1 --2- 172.21.15.72:0/3186161782 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b83485670 0x561b8348a170 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7eff14009b60 tx=0x7eff1400e6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac0e160df15e47b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:30.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.662+0000 7eff217fa640 1 -- 172.21.15.72:0/3186161782 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff14016070 con 0x561b83485670 2024-03-22T01:36:30.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.662+0000 7eff217fa640 1 -- 172.21.15.72:0/3186161782 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff1400ee90 con 0x561b83485670 2024-03-22T01:36:30.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.662+0000 7eff217fa640 1 -- 172.21.15.72:0/3186161782 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff140116b0 con 0x561b83485670 2024-03-22T01:36:30.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.663+0000 7eff33f67740 1 -- 172.21.15.72:0/3186161782 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b83485670 msgr2=0x561b8348a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:30.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.663+0000 7eff33f67740 1 --2- 172.21.15.72:0/3186161782 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b83485670 0x561b8348a170 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7eff14009b60 tx=0x7eff1400e6c0 comp rx=0 tx=0).stop 2024-03-22T01:36:30.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.663+0000 7eff33f67740 1 -- 172.21.15.72:0/3186161782 shutdown_connections 2024-03-22T01:36:30.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.663+0000 7eff33f67740 1 --2- 172.21.15.72:0/3186161782 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b83485670 0x561b8348a170 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:30.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.663+0000 7eff33f67740 1 --2- 172.21.15.72:0/3186161782 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561b8348a7e0 0x561b8348cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:30.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.663+0000 7eff33f67740 1 --2- 172.21.15.72:0/3186161782 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561b83484c80 0x561b834850a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:30.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.663+0000 7eff33f67740 1 -- 172.21.15.72:0/3186161782 >> 172.21.15.72:0/3186161782 conn(0x561b834801f0 msgr2=0x561b834805f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:30.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.663+0000 7eff33f67740 1 -- 172.21.15.72:0/3186161782 shutdown_connections 2024-03-22T01:36:30.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.663+0000 7eff33f67740 1 -- 172.21.15.72:0/3186161782 wait complete. 2024-03-22T01:36:30.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.663+0000 7eff33f67740 1 -- start start 2024-03-22T01:36:30.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.663+0000 7eff33f67740 1 Processor -- start 2024-03-22T01:36:30.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.664+0000 7eff33f67740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561b83484c80 0x561b8348ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:30.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.664+0000 7eff33f67740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b83485670 0x561b8348f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:30.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.664+0000 7eff33f67740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561b8348a7e0 0x561b834963b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:30.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.664+0000 7eff33f67740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x561b8348d790 con 0x561b83484c80 2024-03-22T01:36:30.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.664+0000 7eff33f67740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x561b833a4f80 con 0x561b83485670 2024-03-22T01:36:30.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.664+0000 7eff33f67740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561b8348d910 con 0x561b8348a7e0 2024-03-22T01:36:30.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.664+0000 7eff2affd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561b83484c80 0x561b8348ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:30.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.664+0000 7eff33f67740 10 client.0 _ll_register_callbacks cb 0x561b833e14f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:36:30.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.664+0000 7eff2affd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561b83484c80 0x561b8348ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54142/0 (socket says 172.21.15.72:54142) 2024-03-22T01:36:30.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.664+0000 7eff2affd640 1 -- 172.21.15.72:0/1115685170 learned_addr learned my addr 172.21.15.72:0/1115685170 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:30.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.664+0000 7eff227fc640 1 --2- 172.21.15.72:0/1115685170 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b83485670 0x561b8348f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:30.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.664+0000 7eff2b7fe640 1 --2- 172.21.15.72:0/1115685170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561b8348a7e0 0x561b834963b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:30.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.664+0000 7eff2affd640 1 -- 172.21.15.72:0/1115685170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561b8348a7e0 msgr2=0x561b834963b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:36:30.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.664+0000 7eff2affd640 1 --2- 172.21.15.72:0/1115685170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561b8348a7e0 0x561b834963b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:30.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.664+0000 7eff2affd640 1 -- 172.21.15.72:0/1115685170 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b83485670 msgr2=0x561b8348f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:36:30.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.664+0000 7eff2affd640 1 --2- 172.21.15.72:0/1115685170 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b83485670 0x561b8348f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:30.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.664+0000 7eff2affd640 1 -- 172.21.15.72:0/1115685170 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b831c76b0 con 0x561b83484c80 2024-03-22T01:36:30.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.664+0000 7eff2affd640 1 --2- 172.21.15.72:0/1115685170 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561b83484c80 0x561b8348ee30 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7eff1c00c6a0 tx=0x7eff1c09c5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4ea176355a61826 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:30.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.664+0000 7eff127fc640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:36:30.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.664+0000 7eff127fc640 1 -- 172.21.15.72:0/1115685170 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff1c0b7020 con 0x561b83484c80 2024-03-22T01:36:30.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.664+0000 7eff127fc640 1 -- 172.21.15.72:0/1115685170 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7eff1c0ba660 con 0x561b83484c80 2024-03-22T01:36:30.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.664+0000 7eff33f67740 1 -- 172.21.15.72:0/1115685170 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7eff1400ee90 con 0x561b83484c80 2024-03-22T01:36:30.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.665+0000 7eff127fc640 1 -- 172.21.15.72:0/1115685170 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7eff1c09f070 con 0x561b83484c80 2024-03-22T01:36:30.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.665+0000 7eff33f67740 10 client.15351 fetch_fsmap learned FSMap version 21 2024-03-22T01:36:30.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.665+0000 7eff33f67740 1 -- 172.21.15.72:0/1115685170 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x561b83499840 con 0x561b83484c80 2024-03-22T01:36:30.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.665+0000 7eff127fc640 1 -- 172.21.15.72:0/1115685170 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff1400ee90 con 0x561b83484c80 2024-03-22T01:36:30.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.665+0000 7eff127fc640 1 -- 172.21.15.72:0/1115685170 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7eff1c0a4320 con 0x561b83484c80 2024-03-22T01:36:30.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.665+0000 7eff33f67740 10 client.15351 fetch_fsmap finished waiting for FSMap version 21 2024-03-22T01:36:30.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.665+0000 7eff33f67740 10 client.15351 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-22T01:36:30.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.665+0000 7eff33f67740 20 client.15351 populate_metadata read hostname 'smithi072' 2024-03-22T01:36:30.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.665+0000 7eff33f67740 1 --2- 172.21.15.72:0/1115685170 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x561b8349bf10 0x561b8349d380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:30.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.665+0000 7eff33f67740 4 client.15351 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T01:36:30.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.665+0000 7eff33f67740 1 -- 172.21.15.72:0/1115685170 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561b833f6750 con 0x561b8349bf10 2024-03-22T01:36:30.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.665+0000 7eff227fc640 1 --2- 172.21.15.72:0/1115685170 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x561b8349bf10 0x561b8349d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:30.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.666+0000 7eff227fc640 1 --2- 172.21.15.72:0/1115685170 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x561b8349bf10 0x561b8349d380 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4be4dd9f8d92030d server_cookie=6021673f617ec00f in_seq=0 out_seq=0 2024-03-22T01:36:30.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.666+0000 7eff127fc640 10 client.15351 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:36:30.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.666+0000 7eff127fc640 1 -- 172.21.15.72:0/1115685170 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561b833f6750 con 0x561b8349bf10 2024-03-22T01:36:30.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.666+0000 7eff127fc640 10 client.15351 handle_command_reply: tid=0 2024-03-22T01:36:30.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.666+0000 7eff33f67740 1 client.15351 shutdown 2024-03-22T01:36:30.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.666+0000 7eff33f67740 1 -- 172.21.15.72:0/1115685170 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561b83484c80 msgr2=0x561b8348ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:30.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.666+0000 7eff33f67740 1 --2- 172.21.15.72:0/1115685170 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561b83484c80 0x561b8348ee30 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7eff1c00c6a0 tx=0x7eff1c09c5e0 comp rx=0 tx=0).stop 2024-03-22T01:36:30.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.666+0000 7eff33f67740 1 -- 172.21.15.72:0/1115685170 shutdown_connections 2024-03-22T01:36:30.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.666+0000 7eff33f67740 1 --2- 172.21.15.72:0/1115685170 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x561b8349bf10 0x561b8349d380 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:30.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.666+0000 7eff33f67740 1 --2- 172.21.15.72:0/1115685170 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b83485670 0x561b8348f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:30.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.666+0000 7eff127fc640 0 client.15351 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:36:30.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.666+0000 7eff33f67740 1 --2- 172.21.15.72:0/1115685170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561b8348a7e0 0x561b834963b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:30.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.666+0000 7eff33f67740 1 --2- 172.21.15.72:0/1115685170 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561b83484c80 0x561b8348ee30 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:30.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.666+0000 7eff33f67740 1 -- 172.21.15.72:0/1115685170 >> 172.21.15.72:0/1115685170 conn(0x561b834801f0 msgr2=0x561b8347f4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:30.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.666+0000 7eff33f67740 1 -- 172.21.15.72:0/1115685170 shutdown_connections 2024-03-22T01:36:30.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.666+0000 7eff33f67740 1 -- 172.21.15.72:0/1115685170 wait complete. 2024-03-22T01:36:30.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.666+0000 7eff33f67740 20 client.15351 trim_cache size 0 max 16384 2024-03-22T01:36:30.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.684+0000 7eff33f67740 1 Processor -- start 2024-03-22T01:36:30.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.684+0000 7eff33f67740 1 -- start start 2024-03-22T01:36:30.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.684+0000 7eff33f67740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561b83484c80 0x561b8348ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:30.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.684+0000 7eff33f67740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561b83485670 0x561b83496340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:30.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.684+0000 7eff33f67740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b8348a7e0 0x561b8348ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:30.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.684+0000 7eff33f67740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x561b8349b6c0 con 0x561b83485670 2024-03-22T01:36:30.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.684+0000 7eff33f67740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x561b8349bd90 con 0x561b8348a7e0 2024-03-22T01:36:30.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.684+0000 7eff33f67740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561b8349c460 con 0x561b83484c80 2024-03-22T01:36:30.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.685+0000 7eff227fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561b83485670 0x561b83496340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:30.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.685+0000 7eff227fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561b83485670 0x561b83496340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54158/0 (socket says 172.21.15.72:54158) 2024-03-22T01:36:30.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.685+0000 7eff227fc640 1 -- 172.21.15.72:0/2173233250 learned_addr learned my addr 172.21.15.72:0/2173233250 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:30.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.685+0000 7eff2b7fe640 1 --2- 172.21.15.72:0/2173233250 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b8348a7e0 0x561b8348ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:30.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.685+0000 7eff2affd640 1 --2- 172.21.15.72:0/2173233250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561b83484c80 0x561b8348ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:30.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.685+0000 7eff227fc640 1 -- 172.21.15.72:0/2173233250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561b83484c80 msgr2=0x561b8348ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:36:30.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.685+0000 7eff227fc640 1 --2- 172.21.15.72:0/2173233250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561b83484c80 0x561b8348ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:30.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.685+0000 7eff227fc640 1 -- 172.21.15.72:0/2173233250 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b8348a7e0 msgr2=0x561b8348ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:36:30.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.685+0000 7eff227fc640 1 --2- 172.21.15.72:0/2173233250 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b8348a7e0 0x561b8348ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:30.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.685+0000 7eff227fc640 1 -- 172.21.15.72:0/2173233250 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b83480f60 con 0x561b83485670 2024-03-22T01:36:30.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.685+0000 7eff227fc640 1 --2- 172.21.15.72:0/2173233250 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561b83485670 0x561b83496340 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7eff14009b60 tx=0x7eff14005760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8edba602b173a24d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:30.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.685+0000 7eff20ff9640 1 -- 172.21.15.72:0/2173233250 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff14016040 con 0x561b83485670 2024-03-22T01:36:30.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.685+0000 7eff20ff9640 1 -- 172.21.15.72:0/2173233250 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff14009e50 con 0x561b83485670 2024-03-22T01:36:30.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.685+0000 7eff20ff9640 1 -- 172.21.15.72:0/2173233250 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff14005410 con 0x561b83485670 2024-03-22T01:36:30.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.686+0000 7eff33f67740 1 -- 172.21.15.72:0/2173233250 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561b83485670 msgr2=0x561b83496340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:30.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.686+0000 7eff33f67740 1 --2- 172.21.15.72:0/2173233250 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561b83485670 0x561b83496340 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7eff14009b60 tx=0x7eff14005760 comp rx=0 tx=0).stop 2024-03-22T01:36:30.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.686+0000 7eff33f67740 1 -- 172.21.15.72:0/2173233250 shutdown_connections 2024-03-22T01:36:30.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.686+0000 7eff33f67740 1 --2- 172.21.15.72:0/2173233250 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b8348a7e0 0x561b8348ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:30.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.686+0000 7eff33f67740 1 --2- 172.21.15.72:0/2173233250 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561b83485670 0x561b83496340 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:30.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.686+0000 7eff33f67740 1 --2- 172.21.15.72:0/2173233250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561b83484c80 0x561b8348ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:30.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.686+0000 7eff33f67740 1 -- 172.21.15.72:0/2173233250 >> 172.21.15.72:0/2173233250 conn(0x561b834801f0 msgr2=0x561b8349ad00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:30.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.686+0000 7eff33f67740 1 -- 172.21.15.72:0/2173233250 shutdown_connections 2024-03-22T01:36:30.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.686+0000 7eff33f67740 1 -- 172.21.15.72:0/2173233250 wait complete. 2024-03-22T01:36:30.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.686+0000 7eff33f67740 1 -- start start 2024-03-22T01:36:30.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.686+0000 7eff33f67740 1 Processor -- start 2024-03-22T01:36:30.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.686+0000 7eff33f67740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561b83484c80 0x561b83494330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:30.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.686+0000 7eff33f67740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561b83485670 0x561b834fdc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:30.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.686+0000 7eff33f67740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b8348a7e0 0x561b835001b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:30.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.686+0000 7eff33f67740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x561b8349d200 con 0x561b83485670 2024-03-22T01:36:30.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.686+0000 7eff33f67740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x561b8349cb30 con 0x561b8348a7e0 2024-03-22T01:36:30.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.686+0000 7eff33f67740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561b8349d8d0 con 0x561b83484c80 2024-03-22T01:36:30.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.686+0000 7eff33f67740 10 client.0 _ll_register_callbacks cb 0x7eff1c09b7f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:36:30.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.686+0000 7eff227fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561b83485670 0x561b834fdc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:30.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.687+0000 7eff227fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561b83485670 0x561b834fdc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54168/0 (socket says 172.21.15.72:54168) 2024-03-22T01:36:30.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.687+0000 7eff227fc640 1 -- 172.21.15.72:0/1853840054 learned_addr learned my addr 172.21.15.72:0/1853840054 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:30.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.687+0000 7eff2b7fe640 1 --2- 172.21.15.72:0/1853840054 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b8348a7e0 0x561b835001b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:30.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.687+0000 7eff2affd640 1 --2- 172.21.15.72:0/1853840054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561b83484c80 0x561b83494330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:30.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.687+0000 7eff227fc640 1 -- 172.21.15.72:0/1853840054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561b83484c80 msgr2=0x561b83494330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:36:30.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.687+0000 7eff227fc640 1 --2- 172.21.15.72:0/1853840054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561b83484c80 0x561b83494330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:30.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.687+0000 7eff227fc640 1 -- 172.21.15.72:0/1853840054 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b8348a7e0 msgr2=0x561b835001b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:36:30.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.687+0000 7eff227fc640 1 --2- 172.21.15.72:0/1853840054 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b8348a7e0 0x561b835001b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:30.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.687+0000 7eff227fc640 1 -- 172.21.15.72:0/1853840054 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b835007f0 con 0x561b83485670 2024-03-22T01:36:30.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.687+0000 7eff227fc640 1 --2- 172.21.15.72:0/1853840054 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561b83485670 0x561b834fdc60 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7eff1400ebc0 tx=0x7eff140092a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaf371b2fd903349 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:30.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.687+0000 7eff127fc640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:36:30.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.687+0000 7eff127fc640 1 -- 172.21.15.72:0/1853840054 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff14016070 con 0x561b83485670 2024-03-22T01:36:30.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.687+0000 7eff127fc640 1 -- 172.21.15.72:0/1853840054 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7eff14005390 con 0x561b83485670 2024-03-22T01:36:30.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.687+0000 7eff33f67740 1 -- 172.21.15.72:0/1853840054 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7eff14009e50 con 0x561b83485670 2024-03-22T01:36:30.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.687+0000 7eff127fc640 1 -- 172.21.15.72:0/1853840054 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7eff1400ee90 con 0x561b83485670 2024-03-22T01:36:30.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.687+0000 7eff33f67740 10 client.15357 fetch_fsmap learned FSMap version 21 2024-03-22T01:36:30.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.687+0000 7eff33f67740 1 -- 172.21.15.72:0/1853840054 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x561b83501ad0 con 0x561b83485670 2024-03-22T01:36:30.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.687+0000 7eff127fc640 1 -- 172.21.15.72:0/1853840054 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff14009e50 con 0x561b83485670 2024-03-22T01:36:30.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.687+0000 7eff127fc640 1 -- 172.21.15.72:0/1853840054 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7eff1400f040 con 0x561b83485670 2024-03-22T01:36:30.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.687+0000 7eff33f67740 10 client.15357 fetch_fsmap finished waiting for FSMap version 21 2024-03-22T01:36:30.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.688+0000 7eff33f67740 10 client.15357 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-22T01:36:30.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.688+0000 7eff33f67740 20 client.15357 populate_metadata read hostname 'smithi072' 2024-03-22T01:36:30.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.688+0000 7eff33f67740 1 --2- 172.21.15.72:0/1853840054 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x561b83506070 0x561b83504d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:30.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.688+0000 7eff33f67740 4 client.15357 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-22T01:36:30.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.688+0000 7eff33f67740 1 -- 172.21.15.72:0/1853840054 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "status"}) -- 0x561b833f0aa0 con 0x561b83506070 2024-03-22T01:36:30.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.688+0000 7eff2affd640 1 --2- 172.21.15.72:0/1853840054 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x561b83506070 0x561b83504d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:30.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.688+0000 7eff127fc640 10 client.15357 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:36:30.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.688+0000 7eff2affd640 1 --2- 172.21.15.72:0/1853840054 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x561b83506070 0x561b83504d80 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cebca54e4b2258be server_cookie=16260787dcd91daf in_seq=0 out_seq=0 2024-03-22T01:36:30.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.688+0000 7eff127fc640 1 -- 172.21.15.72:0/1853840054 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x561b833f0aa0 con 0x561b83506070 2024-03-22T01:36:30.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.688+0000 7eff127fc640 10 client.15357 handle_command_reply: tid=0 2024-03-22T01:36:30.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.688+0000 7eff33f67740 1 client.15357 shutdown 2024-03-22T01:36:30.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.688+0000 7eff33f67740 1 -- 172.21.15.72:0/1853840054 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561b83485670 msgr2=0x561b834fdc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:30.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.688+0000 7eff33f67740 1 --2- 172.21.15.72:0/1853840054 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561b83485670 0x561b834fdc60 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7eff1400ebc0 tx=0x7eff140092a0 comp rx=0 tx=0).stop 2024-03-22T01:36:30.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.688+0000 7eff33f67740 1 -- 172.21.15.72:0/1853840054 shutdown_connections 2024-03-22T01:36:30.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.688+0000 7eff33f67740 1 --2- 172.21.15.72:0/1853840054 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x561b83506070 0x561b83504d80 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:30.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.688+0000 7eff33f67740 1 --2- 172.21.15.72:0/1853840054 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b8348a7e0 0x561b835001b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:30.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.688+0000 7eff33f67740 1 --2- 172.21.15.72:0/1853840054 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561b83485670 0x561b834fdc60 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:30.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.688+0000 7eff33f67740 1 --2- 172.21.15.72:0/1853840054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561b83484c80 0x561b83494330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:30.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.688+0000 7eff127fc640 0 client.15357 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:36:30.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.688+0000 7eff33f67740 1 -- 172.21.15.72:0/1853840054 >> 172.21.15.72:0/1853840054 conn(0x561b834801f0 msgr2=0x561b8348e9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:30.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.688+0000 7eff33f67740 1 -- 172.21.15.72:0/1853840054 shutdown_connections 2024-03-22T01:36:30.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.688+0000 7eff33f67740 1 -- 172.21.15.72:0/1853840054 wait complete. 2024-03-22T01:36:30.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.688+0000 7eff33f67740 20 client.15357 trim_cache size 0 max 16384 2024-03-22T01:36:30.695 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-22T01:36:30.695 INFO:teuthology.orchestra.run.smithi072.stdout: "cluster_fsid": "022a99bc-e7eb-11ee-95cd-87774f69a715", 2024-03-22T01:36:30.695 INFO:teuthology.orchestra.run.smithi072.stdout: "whoami": 0, 2024-03-22T01:36:30.695 INFO:teuthology.orchestra.run.smithi072.stdout: "id": 24524, 2024-03-22T01:36:30.696 INFO:teuthology.orchestra.run.smithi072.stdout: "want_state": "up:active", 2024-03-22T01:36:30.696 INFO:teuthology.orchestra.run.smithi072.stdout: "state": "up:active", 2024-03-22T01:36:30.696 INFO:teuthology.orchestra.run.smithi072.stdout: "fs_name": "cephfs", 2024-03-22T01:36:30.696 INFO:teuthology.orchestra.run.smithi072.stdout: "rank_uptime": 135.20848323499999, 2024-03-22T01:36:30.696 INFO:teuthology.orchestra.run.smithi072.stdout: "mdsmap_epoch": 21, 2024-03-22T01:36:30.696 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_epoch": 95, 2024-03-22T01:36:30.696 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_epoch_barrier": 91, 2024-03-22T01:36:30.696 INFO:teuthology.orchestra.run.smithi072.stdout: "uptime": 141.790168945 2024-03-22T01:36:30.696 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-22T01:36:30.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.689+0000 7eff3129d640 1 -- 172.21.15.72:0/4030586055 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7eff180a2f20 msgr2=0x7eff180a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:30.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.689+0000 7eff3129d640 1 --2- 172.21.15.72:0/4030586055 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7eff180a2f20 0x7eff180a53e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7eff24006010 tx=0x7eff2402ce20 comp rx=0 tx=0).stop 2024-03-22T01:36:30.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.689+0000 7eff3129d640 1 -- 172.21.15.72:0/4030586055 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff2c0ffe60 msgr2=0x7eff2c197eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:30.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.689+0000 7eff3129d640 1 --2- 172.21.15.72:0/4030586055 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff2c0ffe60 0x7eff2c197eb0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7eff1c00e9c0 tx=0x7eff1c00ee80 comp rx=0 tx=0).stop 2024-03-22T01:36:30.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.689+0000 7eff3129d640 1 -- 172.21.15.72:0/4030586055 shutdown_connections 2024-03-22T01:36:30.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.689+0000 7eff3129d640 1 --2- 172.21.15.72:0/4030586055 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7eff180a2f20 0x7eff180a53e0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:30.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.689+0000 7eff3129d640 1 --2- 172.21.15.72:0/4030586055 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff2c1037f0 0x7eff2c19f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:30.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.689+0000 7eff3129d640 1 --2- 172.21.15.72:0/4030586055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff2c102e30 0x7eff2c1983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:30.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.689+0000 7eff3129d640 1 --2- 172.21.15.72:0/4030586055 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff2c0ffe60 0x7eff2c197eb0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:30.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.689+0000 7eff3129d640 1 -- 172.21.15.72:0/4030586055 >> 172.21.15.72:0/4030586055 conn(0x7eff2c0fb7b0 msgr2=0x7eff2c106180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:30.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.689+0000 7eff3129d640 1 -- 172.21.15.72:0/4030586055 shutdown_connections 2024-03-22T01:36:30.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.689+0000 7eff3129d640 1 -- 172.21.15.72:0/4030586055 wait complete. 2024-03-22T01:36:30.699 INFO:teuthology.run_tasks:Running task sequential... 2024-03-22T01:36:30.709 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-03-22T01:36:30.709 INFO:teuthology.task.sequential:In sequential, running task print... 2024-03-22T01:36:30.709 INFO:teuthology.task.print:Enabling mgr modules 2024-03-22T01:36:30.709 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-03-22T01:36:30.710 INFO:teuthology.task.exec:Executing custom commands... 2024-03-22T01:36:30.710 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi072.front.sepia.ceph.com 2024-03-22T01:36:30.710 DEBUG:teuthology.orchestra.run.smithi072:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-03-22T01:36:30.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.812+0000 7f56aaa4e640 1 -- 172.21.15.72:0/2495245578 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f56a4100e40 msgr2=0x7f56a41012a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:30.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.812+0000 7f56aaa4e640 1 --2- 172.21.15.72:0/2495245578 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f56a4100e40 0x7f56a41012a0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f5694009900 tx=0x7f569402bcd0 comp rx=0 tx=0).stop 2024-03-22T01:36:30.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.812+0000 7f56aaa4e640 1 -- 172.21.15.72:0/2495245578 shutdown_connections 2024-03-22T01:36:30.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.812+0000 7f56aaa4e640 1 --2- 172.21.15.72:0/2495245578 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56a41017e0 0x7f56a4101c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:30.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.812+0000 7f56aaa4e640 1 --2- 172.21.15.72:0/2495245578 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f56a4100e40 0x7f56a41012a0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:30.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.812+0000 7f56aaa4e640 1 --2- 172.21.15.72:0/2495245578 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56a41004e0 0x7f56a4100900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:30.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.812+0000 7f56aaa4e640 1 -- 172.21.15.72:0/2495245578 >> 172.21.15.72:0/2495245578 conn(0x7f56a40fc0a0 msgr2=0x7f56a40fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:30.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.813+0000 7f56aaa4e640 1 -- 172.21.15.72:0/2495245578 shutdown_connections 2024-03-22T01:36:30.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.813+0000 7f56aaa4e640 1 -- 172.21.15.72:0/2495245578 wait complete. 2024-03-22T01:36:30.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.813+0000 7f56aaa4e640 1 Processor -- start 2024-03-22T01:36:30.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.813+0000 7f56aaa4e640 1 -- start start 2024-03-22T01:36:30.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.813+0000 7f56aaa4e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f56a41004e0 0x7f56a419a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:30.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.813+0000 7f56aaa4e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56a4100e40 0x7f56a419a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:30.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.813+0000 7f56aaa4e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56a41017e0 0x7f56a41a1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:30.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.813+0000 7f56aaa4e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f56a410fa70 con 0x7f56a41004e0 2024-03-22T01:36:30.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.813+0000 7f56aaa4e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f56a410f8f0 con 0x7f56a41017e0 2024-03-22T01:36:30.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.813+0000 7f56aaa4e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f56a410fbf0 con 0x7f56a4100e40 2024-03-22T01:36:30.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.813+0000 7f56a3fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f56a41004e0 0x7f56a419a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:30.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.813+0000 7f56a3fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f56a41004e0 0x7f56a419a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54190/0 (socket says 172.21.15.72:54190) 2024-03-22T01:36:30.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.814+0000 7f56a3fff640 1 -- 172.21.15.72:0/1403608084 learned_addr learned my addr 172.21.15.72:0/1403608084 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:30.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.814+0000 7f56a8feb640 1 --2- 172.21.15.72:0/1403608084 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56a41017e0 0x7f56a41a1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:30.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.814+0000 7f56a37fe640 1 --2- 172.21.15.72:0/1403608084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56a4100e40 0x7f56a419a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:30.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.814+0000 7f56a3fff640 1 -- 172.21.15.72:0/1403608084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56a4100e40 msgr2=0x7f56a419a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:30.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.814+0000 7f56a3fff640 1 --2- 172.21.15.72:0/1403608084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56a4100e40 0x7f56a419a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:30.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.814+0000 7f56a3fff640 1 -- 172.21.15.72:0/1403608084 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56a41017e0 msgr2=0x7f56a41a1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:30.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.814+0000 7f56a3fff640 1 --2- 172.21.15.72:0/1403608084 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56a41017e0 0x7f56a41a1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:30.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.814+0000 7f56a3fff640 1 -- 172.21.15.72:0/1403608084 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56a41a1da0 con 0x7f56a41004e0 2024-03-22T01:36:30.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.814+0000 7f56a3fff640 1 --2- 172.21.15.72:0/1403608084 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f56a41004e0 0x7f56a419a180 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f569800ea60 tx=0x7f569800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:30.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.814+0000 7f56a17fa640 1 -- 172.21.15.72:0/1403608084 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f569800cde0 con 0x7f56a41004e0 2024-03-22T01:36:30.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.814+0000 7f56aaa4e640 1 -- 172.21.15.72:0/1403608084 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56a41a2090 con 0x7f56a41004e0 2024-03-22T01:36:30.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.814+0000 7f56a17fa640 1 -- 172.21.15.72:0/1403608084 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f56980035f0 con 0x7f56a41004e0 2024-03-22T01:36:30.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.814+0000 7f56a17fa640 1 -- 172.21.15.72:0/1403608084 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5698010670 con 0x7f56a41004e0 2024-03-22T01:36:30.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.814+0000 7f56aaa4e640 1 -- 172.21.15.72:0/1403608084 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56a41a25b0 con 0x7f56a41004e0 2024-03-22T01:36:30.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.815+0000 7f56aaa4e640 1 -- 172.21.15.72:0/1403608084 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56a410b230 con 0x7f56a41004e0 2024-03-22T01:36:30.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.816+0000 7f56a17fa640 1 -- 172.21.15.72:0/1403608084 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7f56980037a0 con 0x7f56a41004e0 2024-03-22T01:36:30.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.817+0000 7f56a17fa640 1 --2- 172.21.15.72:0/1403608084 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f568c0a2f10 0x7f568c0a53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:30.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.817+0000 7f56a37fe640 1 --2- 172.21.15.72:0/1403608084 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f568c0a2f10 0x7f568c0a53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:30.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.817+0000 7f56a17fa640 1 -- 172.21.15.72:0/1403608084 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f56980cfaa0 con 0x7f56a41004e0 2024-03-22T01:36:30.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.817+0000 7f56a37fe640 1 --2- 172.21.15.72:0/1403608084 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f568c0a2f10 0x7f568c0a53d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f569402c850 tx=0x7f5694000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:30.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:30.821+0000 7f56a17fa640 1 -- 172.21.15.72:0/1403608084 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f569809c500 con 0x7f56a41004e0 2024-03-22T01:36:31.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.041+0000 7f56aaa4e640 1 -- 172.21.15.72:0/1403608084 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f56a4062790 con 0x7f56a41004e0 2024-03-22T01:36:31.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.060+0000 7f56a17fa640 1 -- 172.21.15.72:0/1403608084 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7f569809be00 con 0x7f56a41004e0 2024-03-22T01:36:31.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.062+0000 7f56aaa4e640 1 -- 172.21.15.72:0/1403608084 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f568c0a2f10 msgr2=0x7f568c0a53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:31.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.062+0000 7f56aaa4e640 1 --2- 172.21.15.72:0/1403608084 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f568c0a2f10 0x7f568c0a53d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f569402c850 tx=0x7f5694000bc0 comp rx=0 tx=0).stop 2024-03-22T01:36:31.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.062+0000 7f56aaa4e640 1 -- 172.21.15.72:0/1403608084 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f56a41004e0 msgr2=0x7f56a419a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:31.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.062+0000 7f56aaa4e640 1 --2- 172.21.15.72:0/1403608084 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f56a41004e0 0x7f56a419a180 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f569800ea60 tx=0x7f569800ef20 comp rx=0 tx=0).stop 2024-03-22T01:36:31.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.064+0000 7f56aaa4e640 1 -- 172.21.15.72:0/1403608084 shutdown_connections 2024-03-22T01:36:31.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.064+0000 7f56aaa4e640 1 --2- 172.21.15.72:0/1403608084 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f568c0a2f10 0x7f568c0a53d0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:31.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.064+0000 7f56aaa4e640 1 --2- 172.21.15.72:0/1403608084 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56a41017e0 0x7f56a41a1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:31.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.064+0000 7f56aaa4e640 1 --2- 172.21.15.72:0/1403608084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56a4100e40 0x7f56a419a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:31.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.064+0000 7f56aaa4e640 1 --2- 172.21.15.72:0/1403608084 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f56a41004e0 0x7f56a419a180 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:31.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.064+0000 7f56aaa4e640 1 -- 172.21.15.72:0/1403608084 >> 172.21.15.72:0/1403608084 conn(0x7f56a40fc0a0 msgr2=0x7f56a40fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:31.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.065+0000 7f56aaa4e640 1 -- 172.21.15.72:0/1403608084 shutdown_connections 2024-03-22T01:36:31.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.065+0000 7f56aaa4e640 1 -- 172.21.15.72:0/1403608084 wait complete. 2024-03-22T01:36:31.079 DEBUG:teuthology.orchestra.run.smithi072:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-03-22T01:36:31.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.188+0000 7f87cde29640 1 -- 172.21.15.72:0/2477923412 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87c8107900 msgr2=0x7f87c8109cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:31.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.188+0000 7f87cde29640 1 --2- 172.21.15.72:0/2477923412 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87c8107900 0x7f87c8109cf0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f87c0009ca0 tx=0x7f87c002c0f0 comp rx=0 tx=0).stop 2024-03-22T01:36:31.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.188+0000 7f87cde29640 1 -- 172.21.15.72:0/2477923412 shutdown_connections 2024-03-22T01:36:31.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.188+0000 7f87cde29640 1 --2- 172.21.15.72:0/2477923412 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87c8107900 0x7f87c8109cf0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:31.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.188+0000 7f87cde29640 1 --2- 172.21.15.72:0/2477923412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87c8104fd0 0x7f87c81073c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:31.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.188+0000 7f87cde29640 1 --2- 172.21.15.72:0/2477923412 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87c81026a0 0x7f87c8104a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:31.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.188+0000 7f87cde29640 1 -- 172.21.15.72:0/2477923412 >> 172.21.15.72:0/2477923412 conn(0x7f87c80fc030 msgr2=0x7f87c80fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:31.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.189+0000 7f87cde29640 1 -- 172.21.15.72:0/2477923412 shutdown_connections 2024-03-22T01:36:31.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.189+0000 7f87cde29640 1 -- 172.21.15.72:0/2477923412 wait complete. 2024-03-22T01:36:31.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.189+0000 7f87cde29640 1 Processor -- start 2024-03-22T01:36:31.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.189+0000 7f87cde29640 1 -- start start 2024-03-22T01:36:31.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.189+0000 7f87cde29640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87c81026a0 0x7f87c8197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:31.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.189+0000 7f87cde29640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87c8104fd0 0x7f87c81983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:31.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.189+0000 7f87cde29640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87c8107900 0x7f87c819f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:31.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.189+0000 7f87cde29640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f87c810d840 con 0x7f87c8104fd0 2024-03-22T01:36:31.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.189+0000 7f87cde29640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f87c810d6c0 con 0x7f87c81026a0 2024-03-22T01:36:31.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.189+0000 7f87cde29640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f87c810d9c0 con 0x7f87c8107900 2024-03-22T01:36:31.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.190+0000 7f87c6ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87c8104fd0 0x7f87c81983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:31.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.190+0000 7f87c77fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87c81026a0 0x7f87c8197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:31.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.190+0000 7f87c77fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87c81026a0 0x7f87c8197e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.72:49914/0 (socket says 172.21.15.72:49914) 2024-03-22T01:36:31.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.190+0000 7f87c77fe640 1 -- 172.21.15.72:0/1165703189 learned_addr learned my addr 172.21.15.72:0/1165703189 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:31.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.190+0000 7f87c6ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87c8104fd0 0x7f87c81983d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54208/0 (socket says 172.21.15.72:54208) 2024-03-22T01:36:31.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.190+0000 7f87c7fff640 1 --2- 172.21.15.72:0/1165703189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87c8107900 0x7f87c819f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:31.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.190+0000 7f87c77fe640 1 -- 172.21.15.72:0/1165703189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87c8107900 msgr2=0x7f87c819f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:31.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.190+0000 7f87c77fe640 1 --2- 172.21.15.72:0/1165703189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87c8107900 0x7f87c819f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:31.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.190+0000 7f87c77fe640 1 -- 172.21.15.72:0/1165703189 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87c8104fd0 msgr2=0x7f87c81983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:31.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.190+0000 7f87c77fe640 1 --2- 172.21.15.72:0/1165703189 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87c8104fd0 0x7f87c81983d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:31.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.190+0000 7f87c77fe640 1 -- 172.21.15.72:0/1165703189 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87c819fab0 con 0x7f87c81026a0 2024-03-22T01:36:31.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.190+0000 7f87c6ffd640 1 --2- 172.21.15.72:0/1165703189 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87c8104fd0 0x7f87c81983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:36:31.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.190+0000 7f87c77fe640 1 --2- 172.21.15.72:0/1165703189 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87c81026a0 0x7f87c8197e90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f87b800e9f0 tx=0x7f87b800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:31.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.190+0000 7f87c4ff9640 1 -- 172.21.15.72:0/1165703189 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b800cd80 con 0x7f87c81026a0 2024-03-22T01:36:31.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.190+0000 7f87cde29640 1 -- 172.21.15.72:0/1165703189 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87c819fda0 con 0x7f87c81026a0 2024-03-22T01:36:31.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.190+0000 7f87c4ff9640 1 -- 172.21.15.72:0/1165703189 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f87b8010910 con 0x7f87c81026a0 2024-03-22T01:36:31.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.190+0000 7f87c4ff9640 1 -- 172.21.15.72:0/1165703189 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b80188f0 con 0x7f87c81026a0 2024-03-22T01:36:31.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.190+0000 7f87cde29640 1 -- 172.21.15.72:0/1165703189 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87c81a0260 con 0x7f87c81026a0 2024-03-22T01:36:31.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.191+0000 7f87cde29640 1 -- 172.21.15.72:0/1165703189 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87c80fdb20 con 0x7f87c81026a0 2024-03-22T01:36:31.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.192+0000 7f87c4ff9640 1 -- 172.21.15.72:0/1165703189 <== mon.2 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) ==== 137871+0+0 (secure 0 0 0) 0x7f87b8010ac0 con 0x7f87c81026a0 2024-03-22T01:36:31.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.193+0000 7f87c4ff9640 1 --2- 172.21.15.72:0/1165703189 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f87b40a2f60 0x7f87b40a5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:31.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.193+0000 7f87c6ffd640 1 -- 172.21.15.72:0/1165703189 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f87b40a2f60 msgr2=0x7f87b40a5420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.72:6800/401403200 2024-03-22T01:36:31.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.193+0000 7f87c4ff9640 1 -- 172.21.15.72:0/1165703189 <== mon.2 v2:172.21.15.89:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f87b80cfff0 con 0x7f87c81026a0 2024-03-22T01:36:31.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.193+0000 7f87c6ffd640 1 --2- 172.21.15.72:0/1165703189 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f87b40a2f60 0x7f87b40a5420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-22T01:36:31.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.201+0000 7f87c4ff9640 1 -- 172.21.15.72:0/1165703189 <== mon.2 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f87b809c3a0 con 0x7f87c81026a0 2024-03-22T01:36:31.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:31 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/3918677331' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:36:31.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:31 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/305124512' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:36:31.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:31 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/1403608084' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-22T01:36:31.358 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:36:31 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-y[37164]: ignoring --setuser ceph since I am not root 2024-03-22T01:36:31.358 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:36:31 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-y[37164]: ignoring --setgroup ceph since I am not root 2024-03-22T01:36:31.358 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:36:31 smithi089 ceph-mgr[37168]: -- 172.21.15.89:0/3071574032 <== mon.2 v2:172.21.15.89:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563d4bd2e5a0 con 0x563d4bd02c00 2024-03-22T01:36:31.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:31 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/3918677331' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:36:31.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:31 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/305124512' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:36:31.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:31 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/1403608084' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-22T01:36:31.372 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:36:31 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: ignoring --setuser ceph since I am not root 2024-03-22T01:36:31.372 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:36:31 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: ignoring --setgroup ceph since I am not root 2024-03-22T01:36:31.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.384+0000 7f87cde29640 1 -- 172.21.15.72:0/1165703189 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f87c802d3b0 con 0x7f87c81026a0 2024-03-22T01:36:31.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.391+0000 7f87c4ff9640 1 -- 172.21.15.72:0/1165703189 <== mon.2 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v28)=0 v28) ==== 138+0+0 (secure 0 0 0) 0x7f87b80a0440 con 0x7f87c81026a0 2024-03-22T01:36:31.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.393+0000 7f87cde29640 1 -- 172.21.15.72:0/1165703189 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f87b40a2f60 msgr2=0x7f87b40a5420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-22T01:36:31.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.393+0000 7f87cde29640 1 --2- 172.21.15.72:0/1165703189 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f87b40a2f60 0x7f87b40a5420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:31.393 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.393+0000 7f87cde29640 1 -- 172.21.15.72:0/1165703189 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87c81026a0 msgr2=0x7f87c8197e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:31.393 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.393+0000 7f87cde29640 1 --2- 172.21.15.72:0/1165703189 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87c81026a0 0x7f87c8197e90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f87b800e9f0 tx=0x7f87b800eeb0 comp rx=0 tx=0).stop 2024-03-22T01:36:31.393 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.393+0000 7f87cde29640 1 -- 172.21.15.72:0/1165703189 shutdown_connections 2024-03-22T01:36:31.393 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.393+0000 7f87cde29640 1 --2- 172.21.15.72:0/1165703189 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f87b40a2f60 0x7f87b40a5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:31.393 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.393+0000 7f87cde29640 1 --2- 172.21.15.72:0/1165703189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87c8107900 0x7f87c819f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:31.393 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.393+0000 7f87cde29640 1 --2- 172.21.15.72:0/1165703189 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87c8104fd0 0x7f87c81983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:31.393 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.393+0000 7f87cde29640 1 --2- 172.21.15.72:0/1165703189 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87c81026a0 0x7f87c8197e90 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:31.393 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.393+0000 7f87cde29640 1 -- 172.21.15.72:0/1165703189 >> 172.21.15.72:0/1165703189 conn(0x7f87c80fc030 msgr2=0x7f87c8103000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:31.393 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.393+0000 7f87cde29640 1 -- 172.21.15.72:0/1165703189 shutdown_connections 2024-03-22T01:36:31.393 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.393+0000 7f87cde29640 1 -- 172.21.15.72:0/1165703189 wait complete. 2024-03-22T01:36:31.404 DEBUG:teuthology.orchestra.run.smithi072:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-03-22T01:36:31.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:31 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/3918677331' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:36:31.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:31 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/305124512' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:36:31.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:31 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/1403608084' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-22T01:36:31.422 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:36:31 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-z[38319]: ignoring --setuser ceph since I am not root 2024-03-22T01:36:31.422 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:36:31 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-z[38319]: ignoring --setgroup ceph since I am not root 2024-03-22T01:36:31.422 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:36:31 smithi098 ceph-mgr[38323]: -- 172.21.15.98:0/3957809250 <== mon.0 v2:172.21.15.72:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5596b06965a0 con 0x5596b066a800 2024-03-22T01:36:31.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.510+0000 7f1673156640 1 -- 172.21.15.72:0/2328493039 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f166c1026a0 msgr2=0x7f166c108f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:31.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.510+0000 7f1673156640 1 --2- 172.21.15.72:0/2328493039 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f166c1026a0 0x7f166c108f30 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f1664009ca0 tx=0x7f166402c0f0 comp rx=0 tx=0).stop 2024-03-22T01:36:31.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.510+0000 7f1673156640 1 -- 172.21.15.72:0/2328493039 shutdown_connections 2024-03-22T01:36:31.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.510+0000 7f1673156640 1 --2- 172.21.15.72:0/2328493039 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f166c101ce0 0x7f166c102160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:31.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.510+0000 7f1673156640 1 --2- 172.21.15.72:0/2328493039 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f166c1026a0 0x7f166c108f30 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:31.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.510+0000 7f1673156640 1 --2- 172.21.15.72:0/2328493039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f166c100af0 0x7f166c100f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:31.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.510+0000 7f1673156640 1 -- 172.21.15.72:0/2328493039 >> 172.21.15.72:0/2328493039 conn(0x7f166c0fc060 msgr2=0x7f166c0fe4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:31.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.510+0000 7f1673156640 1 -- 172.21.15.72:0/2328493039 shutdown_connections 2024-03-22T01:36:31.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.510+0000 7f1673156640 1 -- 172.21.15.72:0/2328493039 wait complete. 2024-03-22T01:36:31.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.510+0000 7f1673156640 1 Processor -- start 2024-03-22T01:36:31.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.510+0000 7f1673156640 1 -- start start 2024-03-22T01:36:31.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.511+0000 7f1673156640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f166c100af0 0x7f166c196da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:31.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.511+0000 7f1673156640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f166c101ce0 0x7f166c1972e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:31.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.511+0000 7f1673156640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f166c1026a0 0x7f166c19e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:31.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.511+0000 7f1673156640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f166c10c770 con 0x7f166c101ce0 2024-03-22T01:36:31.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.511+0000 7f1673156640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f166c10c5f0 con 0x7f166c100af0 2024-03-22T01:36:31.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.511+0000 7f1673156640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f166c10c8f0 con 0x7f166c1026a0 2024-03-22T01:36:31.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.511+0000 7f166bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f166c101ce0 0x7f166c1972e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:31.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.511+0000 7f166bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f166c101ce0 0x7f166c1972e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54228/0 (socket says 172.21.15.72:54228) 2024-03-22T01:36:31.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.511+0000 7f166bfff640 1 -- 172.21.15.72:0/3496977923 learned_addr learned my addr 172.21.15.72:0/3496977923 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:31.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.511+0000 7f1670ef2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f166c100af0 0x7f166c196da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:31.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.511+0000 7f16716f3640 1 --2- 172.21.15.72:0/3496977923 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f166c1026a0 0x7f166c19e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:31.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.511+0000 7f166bfff640 1 -- 172.21.15.72:0/3496977923 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f166c1026a0 msgr2=0x7f166c19e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:31.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.511+0000 7f166bfff640 1 --2- 172.21.15.72:0/3496977923 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f166c1026a0 0x7f166c19e310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:31.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.511+0000 7f166bfff640 1 -- 172.21.15.72:0/3496977923 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f166c100af0 msgr2=0x7f166c196da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:31.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.511+0000 7f166bfff640 1 --2- 172.21.15.72:0/3496977923 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f166c100af0 0x7f166c196da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:31.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.511+0000 7f166bfff640 1 -- 172.21.15.72:0/3496977923 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f166c19e9b0 con 0x7f166c101ce0 2024-03-22T01:36:31.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.511+0000 7f16716f3640 1 --2- 172.21.15.72:0/3496977923 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f166c1026a0 0x7f166c19e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:36:31.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.511+0000 7f166bfff640 1 --2- 172.21.15.72:0/3496977923 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f166c101ce0 0x7f166c1972e0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f165800e9f0 tx=0x7f165800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:31.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.511+0000 7f1669ffb640 1 -- 172.21.15.72:0/3496977923 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f165800cd80 con 0x7f166c101ce0 2024-03-22T01:36:31.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.511+0000 7f1673156640 1 -- 172.21.15.72:0/3496977923 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f166c19eca0 con 0x7f166c101ce0 2024-03-22T01:36:31.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.511+0000 7f1669ffb640 1 -- 172.21.15.72:0/3496977923 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1658010910 con 0x7f166c101ce0 2024-03-22T01:36:31.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.511+0000 7f1669ffb640 1 -- 172.21.15.72:0/3496977923 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16580188f0 con 0x7f166c101ce0 2024-03-22T01:36:31.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.512+0000 7f1673156640 1 -- 172.21.15.72:0/3496977923 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f166c19f1a0 con 0x7f166c101ce0 2024-03-22T01:36:31.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.513+0000 7f1673156640 1 -- 172.21.15.72:0/3496977923 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f166c107db0 con 0x7f166c101ce0 2024-03-22T01:36:31.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.514+0000 7f1669ffb640 1 -- 172.21.15.72:0/3496977923 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 19) ==== 137871+0+0 (secure 0 0 0) 0x7f1658010ac0 con 0x7f166c101ce0 2024-03-22T01:36:31.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.515+0000 7f1669ffb640 1 --2- 172.21.15.72:0/3496977923 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f165c0a2f60 0x7f165c0a5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:31.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.515+0000 7f1670ef2640 1 -- 172.21.15.72:0/3496977923 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f165c0a2f60 msgr2=0x7f165c0a5420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.72:6800/401403200 2024-03-22T01:36:31.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.515+0000 7f1669ffb640 1 -- 172.21.15.72:0/3496977923 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f16580cfc90 con 0x7f166c101ce0 2024-03-22T01:36:31.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.515+0000 7f1670ef2640 1 --2- 172.21.15.72:0/3496977923 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f165c0a2f60 0x7f165c0a5420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-22T01:36:31.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.519+0000 7f1669ffb640 1 -- 172.21.15.72:0/3496977923 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f165809c0a0 con 0x7f166c101ce0 2024-03-22T01:36:31.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.701+0000 7f1673156640 1 -- 172.21.15.72:0/3496977923 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f166c0628a0 con 0x7f166c101ce0 2024-03-22T01:36:31.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.706+0000 7f1669ffb640 1 -- 172.21.15.72:0/3496977923 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v29)=0 v29) ==== 133+0+0 (secure 0 0 0) 0x7f16580a0140 con 0x7f166c101ce0 2024-03-22T01:36:31.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.708+0000 7f1673156640 1 -- 172.21.15.72:0/3496977923 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f165c0a2f60 msgr2=0x7f165c0a5420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-22T01:36:31.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.708+0000 7f1673156640 1 --2- 172.21.15.72:0/3496977923 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f165c0a2f60 0x7f165c0a5420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:31.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.708+0000 7f1673156640 1 -- 172.21.15.72:0/3496977923 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f166c101ce0 msgr2=0x7f166c1972e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:31.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.708+0000 7f1673156640 1 --2- 172.21.15.72:0/3496977923 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f166c101ce0 0x7f166c1972e0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f165800e9f0 tx=0x7f165800eeb0 comp rx=0 tx=0).stop 2024-03-22T01:36:31.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.708+0000 7f1673156640 1 -- 172.21.15.72:0/3496977923 shutdown_connections 2024-03-22T01:36:31.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.708+0000 7f1673156640 1 --2- 172.21.15.72:0/3496977923 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f165c0a2f60 0x7f165c0a5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:31.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.708+0000 7f1673156640 1 --2- 172.21.15.72:0/3496977923 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f166c1026a0 0x7f166c19e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:31.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.708+0000 7f1673156640 1 --2- 172.21.15.72:0/3496977923 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f166c101ce0 0x7f166c1972e0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:31.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.708+0000 7f1673156640 1 --2- 172.21.15.72:0/3496977923 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f166c100af0 0x7f166c196da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:31.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.708+0000 7f1673156640 1 -- 172.21.15.72:0/3496977923 >> 172.21.15.72:0/3496977923 conn(0x7f166c0fc060 msgr2=0x7f166c104f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:31.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.708+0000 7f1673156640 1 -- 172.21.15.72:0/3496977923 shutdown_connections 2024-03-22T01:36:31.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:31.708+0000 7f1673156640 1 -- 172.21.15.72:0/3496977923 wait complete. 2024-03-22T01:36:31.718 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:36:31 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:36:31.649+0000 7fa517128200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-22T01:36:31.720 INFO:teuthology.run_tasks:Running task kclient... 2024-03-22T01:36:31.739 INFO:tasks.kclient:Mounting kernel clients... 2024-03-22T01:36:31.739 INFO:tasks.kclient:config is {'client.0': {}} 2024-03-22T01:36:31.740 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T01:36:31.740 DEBUG:teuthology.orchestra.run.smithi072:> ip netns list 2024-03-22T01:36:31.757 INFO:teuthology.orchestra.run.smithi072.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-03-22T01:36:31.758 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T01:36:31.758 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-22T01:36:31.826 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T01:36:31.826 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link delete ceph-brx 2024-03-22T01:36:31.921 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:36:31 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-z[38319]: 2024-03-22T01:36:31.628+0000 7f30d2ea1200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-22T01:36:31.921 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:36:31 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-z[38319]: 2024-03-22T01:36:31.722+0000 7f30d2ea1200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-22T01:36:31.934 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=secure', 'nowsync'], 'syntax': 'v2', 'mount_subvol_num': 1} 2024-03-22T01:36:31.934 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-22T01:36:31.934 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-22T01:36:31.934 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph fs ls 2024-03-22T01:36:32.072 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:36:31 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:36:31.748+0000 7fa517128200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-22T01:36:32.072 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:36:31 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-y[37164]: 2024-03-22T01:36:31.633+0000 7fa3858b5200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-22T01:36:32.073 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:36:31 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-y[37164]: 2024-03-22T01:36:31.728+0000 7fa3858b5200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-22T01:36:32.180 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:36:32.357 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:32 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/1403608084' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-22T01:36:32.357 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:32 smithi098 ceph-mon[36468]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-03-22T01:36:32.357 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:32 smithi098 ceph-mon[36468]: from='client.? ' entity='client.admin' 2024-03-22T01:36:32.358 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:32 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/3496977923' entity='client.admin' 2024-03-22T01:36:32.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:32 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/1403608084' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-22T01:36:32.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:32 smithi089 ceph-mon[35770]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-03-22T01:36:32.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:32 smithi089 ceph-mon[35770]: from='client.? ' entity='client.admin' 2024-03-22T01:36:32.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:32 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/3496977923' entity='client.admin' 2024-03-22T01:36:32.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:32 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/1403608084' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-22T01:36:32.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:32 smithi072 ceph-mon[32414]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-03-22T01:36:32.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:32 smithi072 ceph-mon[32414]: from='client.? ' entity='client.admin' 2024-03-22T01:36:32.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:32 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/3496977923' entity='client.admin' 2024-03-22T01:36:32.671 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:36:32 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-z[38319]: 2024-03-22T01:36:32.358+0000 7f30d2ea1200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-22T01:36:32.671 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:36:32 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-z[38319]: 2024-03-22T01:36:32.450+0000 7f30d2ea1200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-22T01:36:32.671 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:36:32 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-z[38319]: 2024-03-22T01:36:32.590+0000 7f30d2ea1200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-22T01:36:32.696 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:36:32 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:36:32.421+0000 7fa517128200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-22T01:36:32.697 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:36:32 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:36:32.518+0000 7fa517128200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-22T01:36:32.697 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:36:32 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:36:32.661+0000 7fa517128200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-22T01:36:32.709 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:36:32 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-y[37164]: 2024-03-22T01:36:32.374+0000 7fa3858b5200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-22T01:36:32.709 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:36:32 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-y[37164]: 2024-03-22T01:36:32.468+0000 7fa3858b5200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-22T01:36:32.709 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:36:32 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-y[37164]: 2024-03-22T01:36:32.613+0000 7fa3858b5200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-22T01:36:32.709 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:36:32 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-y[37164]: 2024-03-22T01:36:32.710+0000 7fa3858b5200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-22T01:36:33.028 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:36:32 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-z[38319]: 2024-03-22T01:36:32.684+0000 7f30d2ea1200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-22T01:36:33.029 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:36:32 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-z[38319]: 2024-03-22T01:36:32.767+0000 7f30d2ea1200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-22T01:36:33.029 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:36:33 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-z[38319]: 2024-03-22T01:36:33.029+0000 7f30d2ea1200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-22T01:36:33.044 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:36:32 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:36:32.757+0000 7fa517128200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-22T01:36:33.044 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:36:32 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:36:32.840+0000 7fa517128200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-22T01:36:33.088 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:36:32 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-y[37164]: 2024-03-22T01:36:32.793+0000 7fa3858b5200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-22T01:36:33.303 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:36:33 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:36:33.106+0000 7fa517128200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-22T01:36:33.303 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:36:33 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:36:33.216+0000 7fa517128200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-22T01:36:33.317 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:36:33 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-z[38319]: 2024-03-22T01:36:33.139+0000 7f30d2ea1200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-22T01:36:33.317 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:36:33 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-z[38319]: 2024-03-22T01:36:33.225+0000 7f30d2ea1200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-22T01:36:33.358 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:36:33 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-y[37164]: 2024-03-22T01:36:33.089+0000 7fa3858b5200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-22T01:36:33.358 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:36:33 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-y[37164]: 2024-03-22T01:36:33.200+0000 7fa3858b5200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-22T01:36:33.358 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:36:33 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-y[37164]: 2024-03-22T01:36:33.287+0000 7fa3858b5200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-22T01:36:33.583 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:36:33 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:36:33.304+0000 7fa517128200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-22T01:36:33.583 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:36:33 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:36:33.403+0000 7fa517128200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-22T01:36:33.671 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:36:33 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-z[38319]: 2024-03-22T01:36:33.318+0000 7f30d2ea1200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-22T01:36:33.671 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:36:33 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-z[38319]: 2024-03-22T01:36:33.492+0000 7f30d2ea1200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-22T01:36:33.857 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:36:33 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-y[37164]: 2024-03-22T01:36:33.382+0000 7fa3858b5200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-22T01:36:33.858 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:36:33 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-y[37164]: 2024-03-22T01:36:33.561+0000 7fa3858b5200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-22T01:36:33.872 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:36:33 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:36:33.584+0000 7fa517128200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-22T01:36:33.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:33.894+0000 7f454412a700 1 -- 172.21.15.72:0/402504550 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f453c1076b0 msgr2=0x7f453c107a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:33.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:33.894+0000 7f454412a700 1 --2- 172.21.15.72:0/402504550 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f453c1076b0 0x7f453c107a90 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f45380099c0 tx=0x7f4538009cd0 comp rx=0 tx=0).stop 2024-03-22T01:36:33.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:33.894+0000 7f454412a700 1 -- 172.21.15.72:0/402504550 shutdown_connections 2024-03-22T01:36:33.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:33.894+0000 7f454412a700 1 --2- 172.21.15.72:0/402504550 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f453c102220 0x7f453c10e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:33.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:33.894+0000 7f454412a700 1 --2- 172.21.15.72:0/402504550 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f453c1076b0 0x7f453c107a90 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:33.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:33.894+0000 7f454412a700 1 --2- 172.21.15.72:0/402504550 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f453c101860 0x7f453c101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:33.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:33.894+0000 7f454412a700 1 -- 172.21.15.72:0/402504550 >> 172.21.15.72:0/402504550 conn(0x7f453c078440 msgr2=0x7f453c078850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:33.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:33.894+0000 7f454412a700 1 -- 172.21.15.72:0/402504550 shutdown_connections 2024-03-22T01:36:33.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:33.894+0000 7f454412a700 1 -- 172.21.15.72:0/402504550 wait complete. 2024-03-22T01:36:33.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:33.895+0000 7f454412a700 1 Processor -- start 2024-03-22T01:36:33.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:33.895+0000 7f454412a700 1 -- start start 2024-03-22T01:36:33.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:33.895+0000 7f454412a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f453c101860 0x7f453c19d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:33.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:33.895+0000 7f454412a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f453c102220 0x7f453c19ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:33.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:33.895+0000 7f454412a700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f453c1076b0 0x7f453c197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:33.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:33.895+0000 7f454412a700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f453c111860 con 0x7f453c1076b0 2024-03-22T01:36:33.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:33.895+0000 7f454412a700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f453c1116e0 con 0x7f453c101860 2024-03-22T01:36:33.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:33.895+0000 7f454412a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f453c1119e0 con 0x7f453c102220 2024-03-22T01:36:33.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:33.895+0000 7f4541ec6700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f453c101860 0x7f453c19d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:33.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:33.895+0000 7f4541ec6700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f453c101860 0x7f453c19d880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.72:49944/0 (socket says 172.21.15.72:49944) 2024-03-22T01:36:33.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:33.895+0000 7f4541ec6700 1 -- 172.21.15.72:0/4076953919 learned_addr learned my addr 172.21.15.72:0/4076953919 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:33.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:33.895+0000 7f45416c5700 1 --2- 172.21.15.72:0/4076953919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f453c102220 0x7f453c19ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:33.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:33.895+0000 7f45426c7700 1 --2- 172.21.15.72:0/4076953919 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f453c1076b0 0x7f453c197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:33.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:33.896+0000 7f45416c5700 1 -- 172.21.15.72:0/4076953919 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f453c101860 msgr2=0x7f453c19d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:33.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:33.896+0000 7f45416c5700 1 --2- 172.21.15.72:0/4076953919 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f453c101860 0x7f453c19d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:33.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:33.896+0000 7f45416c5700 1 -- 172.21.15.72:0/4076953919 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f453c1076b0 msgr2=0x7f453c197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:33.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:33.896+0000 7f45416c5700 1 --2- 172.21.15.72:0/4076953919 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f453c1076b0 0x7f453c197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:33.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:33.896+0000 7f45416c5700 1 -- 172.21.15.72:0/4076953919 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f453c1981a0 con 0x7f453c102220 2024-03-22T01:36:33.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:33.896+0000 7f4541ec6700 1 --2- 172.21.15.72:0/4076953919 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f453c101860 0x7f453c19d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:36:33.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:33.896+0000 7f45426c7700 1 --2- 172.21.15.72:0/4076953919 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f453c1076b0 0x7f453c197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:36:33.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:33.896+0000 7f45416c5700 1 --2- 172.21.15.72:0/4076953919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f453c102220 0x7f453c19ddc0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f452c00d750 tx=0x7f452c00da60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:33.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:33.896+0000 7f4532ffd700 1 -- 172.21.15.72:0/4076953919 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f452c014030 con 0x7f453c102220 2024-03-22T01:36:33.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:33.896+0000 7f4532ffd700 1 -- 172.21.15.72:0/4076953919 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f452c004500 con 0x7f453c102220 2024-03-22T01:36:33.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:33.896+0000 7f454412a700 1 -- 172.21.15.72:0/4076953919 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f453c198490 con 0x7f453c102220 2024-03-22T01:36:33.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:33.896+0000 7f454412a700 1 -- 172.21.15.72:0/4076953919 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f453c06ad20 con 0x7f453c102220 2024-03-22T01:36:33.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:33.896+0000 7f4532ffd700 1 -- 172.21.15.72:0/4076953919 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f452c005420 con 0x7f453c102220 2024-03-22T01:36:33.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:33.897+0000 7f454412a700 1 -- 172.21.15.72:0/4076953919 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f453c069340 con 0x7f453c102220 2024-03-22T01:36:33.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:33.899+0000 7f4532ffd700 1 -- 172.21.15.72:0/4076953919 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) ==== 137871+0+0 (secure 0 0 0) 0x7f452c004750 con 0x7f453c102220 2024-03-22T01:36:33.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:33.899+0000 7f4532ffd700 1 --2- 172.21.15.72:0/4076953919 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f45280a3000 0x7f45280a54c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:33.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:33.899+0000 7f4541ec6700 1 -- 172.21.15.72:0/4076953919 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f45280a3000 msgr2=0x7f45280a54c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.72:6800/401403200 2024-03-22T01:36:33.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:33.900+0000 7f4532ffd700 1 -- 172.21.15.72:0/4076953919 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f452c0cf670 con 0x7f453c102220 2024-03-22T01:36:33.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:33.900+0000 7f4541ec6700 1 --2- 172.21.15.72:0/4076953919 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f45280a3000 0x7f45280a54c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-22T01:36:33.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:33.903+0000 7f4532ffd700 1 -- 172.21.15.72:0/4076953919 <== mon.1 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f452c09c0d0 con 0x7f453c102220 2024-03-22T01:36:34.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:34.101+0000 7f4541ec6700 1 -- 172.21.15.72:0/4076953919 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f45280a3000 msgr2=0x7f45280a54c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.72:6800/401403200 2024-03-22T01:36:34.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:34.101+0000 7f4541ec6700 1 --2- 172.21.15.72:0/4076953919 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f45280a3000 0x7f45280a54c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-22T01:36:34.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:34.183+0000 7f454412a700 1 -- 172.21.15.72:0/4076953919 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f453c06b7a0 con 0x7f453c102220 2024-03-22T01:36:34.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:34.183+0000 7f4532ffd700 1 -- 172.21.15.72:0/4076953919 <== mon.1 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v21) ==== 53+0+88 (secure 0 0 0) 0x7f452c09b9d0 con 0x7f453c102220 2024-03-22T01:36:34.182 INFO:teuthology.orchestra.run.smithi072.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-22T01:36:34.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:34.185+0000 7f454412a700 1 -- 172.21.15.72:0/4076953919 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f45280a3000 msgr2=0x7f45280a54c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-22T01:36:34.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:34.185+0000 7f454412a700 1 --2- 172.21.15.72:0/4076953919 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f45280a3000 0x7f45280a54c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:34.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:34.185+0000 7f454412a700 1 -- 172.21.15.72:0/4076953919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f453c102220 msgr2=0x7f453c19ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:34.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:34.185+0000 7f454412a700 1 --2- 172.21.15.72:0/4076953919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f453c102220 0x7f453c19ddc0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f452c00d750 tx=0x7f452c00da60 comp rx=0 tx=0).stop 2024-03-22T01:36:34.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:34.185+0000 7f454412a700 1 -- 172.21.15.72:0/4076953919 shutdown_connections 2024-03-22T01:36:34.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:34.185+0000 7f454412a700 1 --2- 172.21.15.72:0/4076953919 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f45280a3000 0x7f45280a54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:34.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:34.185+0000 7f454412a700 1 --2- 172.21.15.72:0/4076953919 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f453c1076b0 0x7f453c197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:34.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:34.185+0000 7f454412a700 1 --2- 172.21.15.72:0/4076953919 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f453c101860 0x7f453c19d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:34.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:34.185+0000 7f454412a700 1 --2- 172.21.15.72:0/4076953919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f453c102220 0x7f453c19ddc0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:34.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:34.185+0000 7f454412a700 1 -- 172.21.15.72:0/4076953919 >> 172.21.15.72:0/4076953919 conn(0x7f453c078440 msgr2=0x7f453c100500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:34.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:34.185+0000 7f454412a700 1 -- 172.21.15.72:0/4076953919 shutdown_connections 2024-03-22T01:36:34.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:34.185+0000 7f454412a700 1 -- 172.21.15.72:0/4076953919 wait complete. 2024-03-22T01:36:34.564 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:34 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/4076953919' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-22T01:36:34.578 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:34 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/4076953919' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-22T01:36:34.578 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:36:34 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-z[38319]: 2024-03-22T01:36:34.450+0000 7f30d2ea1200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-22T01:36:34.606 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:34 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/4076953919' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-22T01:36:34.852 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:36:34 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:36:34.568+0000 7fa517128200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-22T01:36:34.852 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:36:34 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:36:34.701+0000 7fa517128200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-22T01:36:34.857 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:36:34 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-y[37164]: 2024-03-22T01:36:34.606+0000 7fa3858b5200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-22T01:36:34.857 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:36:34 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-y[37164]: 2024-03-22T01:36:34.772+0000 7fa3858b5200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-22T01:36:34.883 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-22T01:36:34.883 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-22T01:36:34.883 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi072.front.sepia.ceph.com 2024-03-22T01:36:34.883 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-22T01:36:34.883 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-22T01:36:34.883 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-22T01:36:34.883 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-22T01:36:34.883 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-22T01:36:34.883 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-22T01:36:34.883 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T01:36:34.884 DEBUG:teuthology.orchestra.run.smithi072:> ip addr 2024-03-22T01:36:34.900 INFO:teuthology.orchestra.run.smithi072.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-22T01:36:34.900 INFO:teuthology.orchestra.run.smithi072.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-22T01:36:34.900 INFO:teuthology.orchestra.run.smithi072.stdout: inet 127.0.0.1/8 scope host lo 2024-03-22T01:36:34.900 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-22T01:36:34.900 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 ::1/128 scope host 2024-03-22T01:36:34.900 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-22T01:36:34.900 INFO:teuthology.orchestra.run.smithi072.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-22T01:36:34.900 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:d8:da:f4 brd ff:ff:ff:ff:ff:ff 2024-03-22T01:36:34.900 INFO:teuthology.orchestra.run.smithi072.stdout: altname enp7s0f0 2024-03-22T01:36:34.900 INFO:teuthology.orchestra.run.smithi072.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-22T01:36:34.900 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:d8:da:f5 brd ff:ff:ff:ff:ff:ff 2024-03-22T01:36:34.900 INFO:teuthology.orchestra.run.smithi072.stdout: altname enp7s0f1 2024-03-22T01:36:34.900 INFO:teuthology.orchestra.run.smithi072.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-22T01:36:34.901 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:8f:b7:64 brd ff:ff:ff:ff:ff:ff 2024-03-22T01:36:34.901 INFO:teuthology.orchestra.run.smithi072.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-22T01:36:34.901 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:8f:b7:65 brd ff:ff:ff:ff:ff:ff 2024-03-22T01:36:34.901 INFO:teuthology.orchestra.run.smithi072.stdout: inet 172.21.15.72/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-22T01:36:34.901 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft 42068sec preferred_lft 42068sec 2024-03-22T01:36:34.901 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::ec4:7aff:fe8f:b765/64 scope link 2024-03-22T01:36:34.901 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-22T01:36:34.901 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-22T01:36:34.901 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-22T01:36:34.901 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-03-22T01:36:34.902 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link add name ceph-brx type bridge 2024-03-22T01:36:34.902 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip addr flush dev ceph-brx 2024-03-22T01:36:34.902 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link set ceph-brx up 2024-03-22T01:36:34.902 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-22T01:36:34.902 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-03-22T01:36:34.910 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:36:34 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-z[38319]: 2024-03-22T01:36:34.579+0000 7f30d2ea1200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-22T01:36:34.989 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:34 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-22T01:36:35.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-22T01:36:35.055 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T01:36:35.055 DEBUG:teuthology.orchestra.run.smithi072:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-22T01:36:35.082 INFO:teuthology.orchestra.run.smithi072.stdout:1 2024-03-22T01:36:35.083 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T01:36:35.083 DEBUG:teuthology.orchestra.run.smithi072:> route 2024-03-22T01:36:35.122 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:36:35 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:36:35.047+0000 7fa517128200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-22T01:36:35.138 INFO:teuthology.orchestra.run.smithi072.stdout:Kernel IP routing table 2024-03-22T01:36:35.138 INFO:teuthology.orchestra.run.smithi072.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-22T01:36:35.138 INFO:teuthology.orchestra.run.smithi072.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-22T01:36:35.138 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-22T01:36:35.138 INFO:teuthology.orchestra.run.smithi072.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-22T01:36:35.138 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-22T01:36:35.139 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-03-22T01:36:35.139 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-22T01:36:35.139 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-22T01:36:35.139 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-22T01:36:35.139 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-03-22T01:36:35.171 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:36:34 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-z[38319]: 2024-03-22T01:36:34.910+0000 7f30d2ea1200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-22T01:36:35.171 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:36:35 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-z[38319]: 2024-03-22T01:36:34.999+0000 7f30d2ea1200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-22T01:36:35.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:35 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-22T01:36:35.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-22T01:36:35.275 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T01:36:35.275 DEBUG:teuthology.orchestra.run.smithi072:> ip netns list 2024-03-22T01:36:35.291 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T01:36:35.291 DEBUG:teuthology.orchestra.run.smithi072:> ip netns list-id 2024-03-22T01:36:35.346 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-22T01:36:35.346 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-03-22T01:36:35.346 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-22T01:36:35.346 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-22T01:36:35.346 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-03-22T01:36:35.399 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:36:35 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-y[37164]: 2024-03-22T01:36:35.110+0000 7fa3858b5200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-22T01:36:35.399 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:36:35 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-y[37164]: 2024-03-22T01:36:35.201+0000 7fa3858b5200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-22T01:36:35.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:35 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-22T01:36:35.446 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:36:35 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:36:35.144+0000 7fa517128200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-22T01:36:35.447 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:36:35 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:36:35.353+0000 7fa517128200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-22T01:36:35.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-22T01:36:35.470 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-22T01:36:35.471 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-22T01:36:35.471 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-03-22T01:36:35.471 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-22T01:36:35.471 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-22T01:36:35.471 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-22T01:36:35.471 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-22T01:36:35.471 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-22T01:36:35.471 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-03-22T01:36:35.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:35 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-22T01:36:35.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-22T01:36:35.599 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-22T01:36:35.599 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-03-22T01:36:35.599 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link set brx.0 up 2024-03-22T01:36:35.599 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link set dev brx.0 master ceph-brx 2024-03-22T01:36:35.599 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-03-22T01:36:35.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:35 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-22T01:36:35.656 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:36:35 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-z[38319]: 2024-03-22T01:36:35.193+0000 7f30d2ea1200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-22T01:36:35.656 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:36:35 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-z[38319]: 2024-03-22T01:36:35.376+0000 7f30d2ea1200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-22T01:36:35.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-22T01:36:35.683 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-22T01:36:35.683 DEBUG:teuthology.orchestra.run.smithi072:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-22T01:36:35.705 INFO:teuthology.orchestra.run.smithi072.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-22T01:36:35.705 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-22T01:36:35.705 DEBUG:teuthology.orchestra.run.smithi072:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-22T01:36:35.760 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-03-22T01:36:35.761 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-03-22T01:36:35.761 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/qa/sv_1/63be0226-262f-43ca-bc55-6e0603eb71d2 2024-03-22T01:36:35.761 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T01:36:35.761 DEBUG:teuthology.orchestra.run.smithi072:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/qa/sv_1/63be0226-262f-43ca-bc55-6e0603eb71d2 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,nowsync,nofallback 2024-03-22T01:36:35.825 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:36:35 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:36:35.556+0000 7fa517128200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-22T01:36:35.846 INFO:teuthology.orchestra.run.smithi072.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,nowsync,nofallback 2024-03-22T01:36:35.846 INFO:teuthology.orchestra.run.smithi072.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,nowsync". 2024-03-22T01:36:35.858 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:36:35 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-y[37164]: 2024-03-22T01:36:35.399+0000 7fa3858b5200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-22T01:36:35.858 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:36:35 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-y[37164]: 2024-03-22T01:36:35.586+0000 7fa3858b5200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-22T01:36:35.915 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:36:35 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-z[38319]: 2024-03-22T01:36:35.657+0000 7f30d2ea1200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-22T01:36:35.969 INFO:teuthology.orchestra.run.smithi072.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,nowsync,nofallback 2024-03-22T01:36:35.969 INFO:teuthology.orchestra.run.smithi072.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,nowsync". 2024-03-22T01:36:35.969 INFO:teuthology.orchestra.run.smithi072.stdout:mount.ceph: resolved to: "172.21.15.72:3300,172.21.15.89:3300,172.21.15.98:3300" 2024-03-22T01:36:35.969 INFO:teuthology.orchestra.run.smithi072.stdout:mount.ceph: trying mount with new device syntax: 0@022a99bc-e7eb-11ee-95cd-87774f69a715.cephfs=/volumes/qa/sv_1/63be0226-262f-43ca-bc55-6e0603eb71d2 2024-03-22T01:36:35.969 INFO:teuthology.orchestra.run.smithi072.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,nowsync,name=0,key=0,mon_addr=172.21.15.72:3300/172.21.15.89:3300/172.21.15.98:3300" will pass to kernel 2024-03-22T01:36:35.970 INFO:teuthology.orchestra.run.smithi072.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-22T01:36:35.971 INFO:teuthology.orchestra.run.smithi072.stdout: You just mounted a file system that supports labels which does not 2024-03-22T01:36:35.971 INFO:teuthology.orchestra.run.smithi072.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-22T01:36:35.971 INFO:teuthology.orchestra.run.smithi072.stdout: applications will generate AVC messages and not be allowed access to 2024-03-22T01:36:35.971 INFO:teuthology.orchestra.run.smithi072.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-22T01:36:35.972 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-22T01:36:35.972 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T01:36:35.973 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-22T01:36:36.046 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-22T01:36:36.122 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:36:35 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:36:35.861+0000 7fa517128200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-22T01:36:36.135 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:36:35 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-y[37164]: 2024-03-22T01:36:35.873+0000 7fa3858b5200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-22T01:36:36.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-22T01:36:36.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-22T01:36:36.145 INFO:teuthology.orchestra.run.smithi072.stdout:client15402 2024-03-22T01:36:36.145 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-22T01:36:36.145 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-22T01:36:36.171 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:36:35 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-z[38319]: 2024-03-22T01:36:35.916+0000 7f30d2ea1200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-22T01:36:36.171 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:36:36 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-z[38319]: 2024-03-22T01:36:36.152+0000 7f30d2ea1200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-22T01:36:36.202 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-22T01:36:36.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-22T01:36:36.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-22T01:36:36.300 INFO:teuthology.orchestra.run.smithi072.stdout:client15402 2024-03-22T01:36:36.301 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-22T01:36:36.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-22T01:36:36.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-22T01:36:36.357 INFO:teuthology.orchestra.run.smithi072.stdout:client15402 2024-03-22T01:36:36.358 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715.client15402/status') 2024-03-22T01:36:36.388 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:36:36 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:36:36.135+0000 7fa517128200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-22T01:36:36.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-22T01:36:36.419 INFO:teuthology.orchestra.run.smithi072.stdout:instance: client.15402 (3)192.168.0.1:0/108207156 2024-03-22T01:36:36.419 INFO:teuthology.orchestra.run.smithi072.stdout:blocklisted: false 2024-03-22T01:36:36.419 INFO:teuthology.orchestra.run.smithi072.stderr:0+1 records in 2024-03-22T01:36:36.420 INFO:teuthology.orchestra.run.smithi072.stderr:0+1 records out 2024-03-22T01:36:36.420 INFO:teuthology.orchestra.run.smithi072.stderr:69 bytes copied, 2.1742e-05 s, 3.2 MB/s 2024-03-22T01:36:36.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-22T01:36:36.427 INFO:teuthology.run_tasks:Running task check-counter... 2024-03-22T01:36:36.467 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-03-22T01:36:36.481 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T01:36:36.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.587+0000 7f04e5439640 1 -- 172.21.15.72:0/4062113859 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04e0107940 msgr2=0x7f04e0109d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:36.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.587+0000 7f04e5439640 1 --2- 172.21.15.72:0/4062113859 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04e0107940 0x7f04e0109d30 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f04d8009ca0 tx=0x7f04d802c0f0 comp rx=0 tx=0).stop 2024-03-22T01:36:36.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.587+0000 7f04e5439640 1 -- 172.21.15.72:0/4062113859 shutdown_connections 2024-03-22T01:36:36.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.587+0000 7f04e5439640 1 --2- 172.21.15.72:0/4062113859 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04e0105010 0x7f04e0107400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:36.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.587+0000 7f04e5439640 1 --2- 172.21.15.72:0/4062113859 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04e0107940 0x7f04e0109d30 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:36.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.587+0000 7f04e5439640 1 --2- 172.21.15.72:0/4062113859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04e01026e0 0x7f04e0104ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:36.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.587+0000 7f04e5439640 1 -- 172.21.15.72:0/4062113859 >> 172.21.15.72:0/4062113859 conn(0x7f04e00fc030 msgr2=0x7f04e00fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:36.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.587+0000 7f04e5439640 1 -- 172.21.15.72:0/4062113859 shutdown_connections 2024-03-22T01:36:36.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.587+0000 7f04e5439640 1 -- 172.21.15.72:0/4062113859 wait complete. 2024-03-22T01:36:36.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.588+0000 7f04e5439640 1 Processor -- start 2024-03-22T01:36:36.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.588+0000 7f04e5439640 1 -- start start 2024-03-22T01:36:36.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.588+0000 7f04e5439640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04e01026e0 0x7f04e0197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:36.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.588+0000 7f04e5439640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04e0105010 0x7f04e01983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:36.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.588+0000 7f04e5439640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04e0107940 0x7f04e019f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:36.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.588+0000 7f04e5439640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f04e010d840 con 0x7f04e0105010 2024-03-22T01:36:36.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.588+0000 7f04e5439640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f04e010d6c0 con 0x7f04e01026e0 2024-03-22T01:36:36.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.588+0000 7f04e5439640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f04e010d9c0 con 0x7f04e0107940 2024-03-22T01:36:36.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.588+0000 7f04de7fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04e0105010 0x7f04e01983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:36.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.588+0000 7f04de7fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04e0105010 0x7f04e01983d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54248/0 (socket says 172.21.15.72:54248) 2024-03-22T01:36:36.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.588+0000 7f04de7fc640 1 -- 172.21.15.72:0/2432926360 learned_addr learned my addr 172.21.15.72:0/2432926360 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:36.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.588+0000 7f04deffd640 1 --2- 172.21.15.72:0/2432926360 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04e01026e0 0x7f04e0197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:36.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.588+0000 7f04df7fe640 1 --2- 172.21.15.72:0/2432926360 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04e0107940 0x7f04e019f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:36.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.589+0000 7f04de7fc640 1 -- 172.21.15.72:0/2432926360 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04e0107940 msgr2=0x7f04e019f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:36.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.589+0000 7f04de7fc640 1 --2- 172.21.15.72:0/2432926360 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04e0107940 0x7f04e019f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:36.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.589+0000 7f04de7fc640 1 -- 172.21.15.72:0/2432926360 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04e01026e0 msgr2=0x7f04e0197e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:36.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.589+0000 7f04de7fc640 1 --2- 172.21.15.72:0/2432926360 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04e01026e0 0x7f04e0197e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:36.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.589+0000 7f04de7fc640 1 -- 172.21.15.72:0/2432926360 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04e019fab0 con 0x7f04e0105010 2024-03-22T01:36:36.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.589+0000 7f04deffd640 1 --2- 172.21.15.72:0/2432926360 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04e01026e0 0x7f04e0197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:36:36.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.589+0000 7f04de7fc640 1 --2- 172.21.15.72:0/2432926360 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04e0105010 0x7f04e01983d0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f04d000e9c0 tx=0x7f04d000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:36.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.589+0000 7f04c7fff640 1 -- 172.21.15.72:0/2432926360 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04d000cd80 con 0x7f04e0105010 2024-03-22T01:36:36.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.589+0000 7f04c7fff640 1 -- 172.21.15.72:0/2432926360 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04d0005500 con 0x7f04e0105010 2024-03-22T01:36:36.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.589+0000 7f04e5439640 1 -- 172.21.15.72:0/2432926360 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04e019fda0 con 0x7f04e0105010 2024-03-22T01:36:36.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.589+0000 7f04c7fff640 1 -- 172.21.15.72:0/2432926360 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04d0010690 con 0x7f04e0105010 2024-03-22T01:36:36.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.589+0000 7f04e5439640 1 -- 172.21.15.72:0/2432926360 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04e01a0260 con 0x7f04e0105010 2024-03-22T01:36:36.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.590+0000 7f04e5439640 1 -- 172.21.15.72:0/2432926360 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04e00fdb20 con 0x7f04e0105010 2024-03-22T01:36:36.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.591+0000 7f04c7fff640 1 -- 172.21.15.72:0/2432926360 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 19) ==== 137871+0+0 (secure 0 0 0) 0x7f04d00046c0 con 0x7f04e0105010 2024-03-22T01:36:36.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.591+0000 7f04c7fff640 1 --2- 172.21.15.72:0/2432926360 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f04c80a2f70 0x7f04c80a5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:36.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.592+0000 7f04deffd640 1 -- 172.21.15.72:0/2432926360 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f04c80a2f70 msgr2=0x7f04c80a5430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.72:6800/401403200 2024-03-22T01:36:36.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.592+0000 7f04c7fff640 1 -- 172.21.15.72:0/2432926360 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f04d00cfb60 con 0x7f04e0105010 2024-03-22T01:36:36.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.592+0000 7f04deffd640 1 --2- 172.21.15.72:0/2432926360 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f04c80a2f70 0x7f04c80a5430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-22T01:36:36.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.596+0000 7f04c7fff640 1 -- 172.21.15.72:0/2432926360 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f04d009c5c0 con 0x7f04e0105010 2024-03-22T01:36:36.607 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:36:36 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-y[37164]: 2024-03-22T01:36:36.136+0000 7fa3858b5200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-22T01:36:36.607 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:36:36 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-y[37164]: 2024-03-22T01:36:36.375+0000 7fa3858b5200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-22T01:36:36.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.777+0000 7f04e5439640 1 -- 172.21.15.72:0/2432926360 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f04e0062890 con 0x7f04e0105010 2024-03-22T01:36:36.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.779+0000 7f04c7fff640 1 -- 172.21.15.72:0/2432926360 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) ==== 74+0+24759 (secure 0 0 0) 0x7f04d009bec0 con 0x7f04e0105010 2024-03-22T01:36:36.778 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:36:36.778 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":95,"fsid":"022a99bc-e7eb-11ee-95cd-87774f69a715","created":"2024-03-22T01:26:23.893672+0000","modified":"2024-03-22T01:36:17.626966+0000","last_up_change":"2024-03-22T01:33:25.223684+0000","last_in_change":"2024-03-22T01:33:05.010331+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T01:29:37.438503+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-22T01:34:12.610422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-22T01:34:13.761686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-22T01:34:17.507039+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"16c15bb0-db92-4191-9063-38b4890e81e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6803","nonce":661906791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6805","nonce":661906791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6809","nonce":661906791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6807","nonce":661906791}]},"public_addr":"172.21.15.72:6803/661906791","cluster_addr":"172.21.15.72:6805/661906791","heartbeat_back_addr":"172.21.15.72:6809/661906791","heartbeat_front_addr":"172.21.15.72:6807/661906791","state":["exists","up"]},{"osd":1,"uuid":"981290ba-97dc-496b-aa22-d51704184b10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6801","nonce":2870590550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6803","nonce":2870590550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6807","nonce":2870590550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6805","nonce":2870590550}]},"public_addr":"172.21.15.89:6801/2870590550","cluster_addr":"172.21.15.89:6803/2870590550","heartbeat_back_addr":"172.21.15.89:6807/2870590550","heartbeat_front_addr":"172.21.15.89:6805/2870590550","state":["exists","up"]},{"osd":2,"uuid":"f07ed658-f545-46a1-8edd-8b1d5d49885c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6801","nonce":3865778118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6803","nonce":3865778118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6807","nonce":3865778118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6805","nonce":3865778118}]},"public_addr":"172.21.15.98:6801/3865778118","cluster_addr":"172.21.15.98:6803/3865778118","heartbeat_back_addr":"172.21.15.98:6807/3865778118","heartbeat_front_addr":"172.21.15.98:6805/3865778118","state":["exists","up"]},{"osd":3,"uuid":"ccc4bc40-2c48-46d2-8548-0c05b44f9f1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6811","nonce":1865951395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6813","nonce":1865951395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6817","nonce":1865951395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6815","nonce":1865951395}]},"public_addr":"172.21.15.72:6811/1865951395","cluster_addr":"172.21.15.72:6813/1865951395","heartbeat_back_addr":"172.21.15.72:6817/1865951395","heartbeat_front_addr":"172.21.15.72:6815/1865951395","state":["exists","up"]},{"osd":4,"uuid":"1812439f-810a-4ed2-9395-c00c9a085a13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6809","nonce":2683760795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6811","nonce":2683760795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6815","nonce":2683760795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6813","nonce":2683760795}]},"public_addr":"172.21.15.89:6809/2683760795","cluster_addr":"172.21.15.89:6811/2683760795","heartbeat_back_addr":"172.21.15.89:6815/2683760795","heartbeat_front_addr":"172.21.15.89:6813/2683760795","state":["exists","up"]},{"osd":5,"uuid":"e78954d8-f4e3-4d58-8536-d2d59c843386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6809","nonce":2635838572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6811","nonce":2635838572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6815","nonce":2635838572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6813","nonce":2635838572}]},"public_addr":"172.21.15.98:6809/2635838572","cluster_addr":"172.21.15.98:6811/2635838572","heartbeat_back_addr":"172.21.15.98:6815/2635838572","heartbeat_front_addr":"172.21.15.98:6813/2635838572","state":["exists","up"]},{"osd":6,"uuid":"dc6c8be2-b64c-4959-8015-7b0ea6d25599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6819","nonce":3158982661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6821","nonce":3158982661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6825","nonce":3158982661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6823","nonce":3158982661}]},"public_addr":"172.21.15.72:6819/3158982661","cluster_addr":"172.21.15.72:6821/3158982661","heartbeat_back_addr":"172.21.15.72:6825/3158982661","heartbeat_front_addr":"172.21.15.72:6823/3158982661","state":["exists","up"]},{"osd":7,"uuid":"7a73469f-cf3a-4306-ae93-ea46e21ebbbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6817","nonce":3075913234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6819","nonce":3075913234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6823","nonce":3075913234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6821","nonce":3075913234}]},"public_addr":"172.21.15.89:6817/3075913234","cluster_addr":"172.21.15.89:6819/3075913234","heartbeat_back_addr":"172.21.15.89:6823/3075913234","heartbeat_front_addr":"172.21.15.89:6821/3075913234","state":["exists","up"]},{"osd":8,"uuid":"c4b13bee-ea4d-4344-95fb-2622ecbb8196","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6817","nonce":1813255577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6819","nonce":1813255577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6823","nonce":1813255577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6821","nonce":1813255577}]},"public_addr":"172.21.15.98:6817/1813255577","cluster_addr":"172.21.15.98:6819/1813255577","heartbeat_back_addr":"172.21.15.98:6823/1813255577","heartbeat_front_addr":"172.21.15.98:6821/1813255577","state":["exists","up"]},{"osd":9,"uuid":"eefb0a44-4a36-4e55-9d04-0501bf718db6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6827","nonce":3377749206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6829","nonce":3377749206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6832","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6833","nonce":3377749206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6831","nonce":3377749206}]},"public_addr":"172.21.15.72:6827/3377749206","cluster_addr":"172.21.15.72:6829/3377749206","heartbeat_back_addr":"172.21.15.72:6833/3377749206","heartbeat_front_addr":"172.21.15.72:6831/3377749206","state":["exists","up"]},{"osd":10,"uuid":"8741d25f-40dd-40d7-b040-5cc5e411eb98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6825","nonce":885058786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6827","nonce":885058786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6831","nonce":885058786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6829","nonce":885058786}]},"public_addr":"172.21.15.89:6825/885058786","cluster_addr":"172.21.15.89:6827/885058786","heartbeat_back_addr":"172.21.15.89:6831/885058786","heartbeat_front_addr":"172.21.15.89:6829/885058786","state":["exists","up"]},{"osd":11,"uuid":"8d02e957-eda4-4184-b5c6-5eb3b8b38060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6825","nonce":1517095838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6827","nonce":1517095838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6831","nonce":1517095838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6829","nonce":1517095838}]},"public_addr":"172.21.15.98:6825/1517095838","cluster_addr":"172.21.15.98:6827/1517095838","heartbeat_back_addr":"172.21.15.98:6831/1517095838","heartbeat_front_addr":"172.21.15.98:6829/1517095838","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:28:47.307652+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:11.037135+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:34.619188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:59.350738+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:30:23.787337+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:30:48.994154+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:31:13.690350+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:31:39.628116+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:04.723912+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:30.689741+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:57.229508+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:33:23.154537+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":7}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:0/121087196":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/2451857761":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/1228477995":"2024-03-23T01:27:15.339809+0000","172.21.15.72:6801/1513232913":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/3558958264":"2024-03-23T01:26:50.074871+0000","172.21.15.72:0/1510187198":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6800/3518012916":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6801/3518012916":"2024-03-23T01:26:50.074871+0000","172.21.15.72:0/1372801041":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6800/1513232913":"2024-03-23T01:27:15.339809+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T01:36:36.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.781+0000 7f04e5439640 1 -- 172.21.15.72:0/2432926360 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f04c80a2f70 msgr2=0x7f04c80a5430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-22T01:36:36.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.781+0000 7f04e5439640 1 --2- 172.21.15.72:0/2432926360 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f04c80a2f70 0x7f04c80a5430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:36.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.781+0000 7f04e5439640 1 -- 172.21.15.72:0/2432926360 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04e0105010 msgr2=0x7f04e01983d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:36.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.781+0000 7f04e5439640 1 --2- 172.21.15.72:0/2432926360 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04e0105010 0x7f04e01983d0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f04d000e9c0 tx=0x7f04d000ee80 comp rx=0 tx=0).stop 2024-03-22T01:36:36.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.781+0000 7f04e5439640 1 -- 172.21.15.72:0/2432926360 shutdown_connections 2024-03-22T01:36:36.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.781+0000 7f04e5439640 1 --2- 172.21.15.72:0/2432926360 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f04c80a2f70 0x7f04c80a5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:36.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.781+0000 7f04e5439640 1 --2- 172.21.15.72:0/2432926360 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04e0107940 0x7f04e019f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:36.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.781+0000 7f04e5439640 1 --2- 172.21.15.72:0/2432926360 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04e0105010 0x7f04e01983d0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:36.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.781+0000 7f04e5439640 1 --2- 172.21.15.72:0/2432926360 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04e01026e0 0x7f04e0197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:36.781 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.781+0000 7f04e5439640 1 -- 172.21.15.72:0/2432926360 >> 172.21.15.72:0/2432926360 conn(0x7f04e00fc030 msgr2=0x7f04e0103020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:36.781 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.781+0000 7f04e5439640 1 -- 172.21.15.72:0/2432926360 shutdown_connections 2024-03-22T01:36:36.781 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.781+0000 7f04e5439640 1 -- 172.21.15.72:0/2432926360 wait complete. 2024-03-22T01:36:36.790 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:36:36 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:36:36.389+0000 7fa517128200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-22T01:36:36.792 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-22T01:29:37.438503+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-22T01:34:12.610422+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-22T01:34:13.761686+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-22T01:34:17.507039+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-22T01:36:36.792 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-22T01:36:36.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.901+0000 7f5375847640 1 -- 172.21.15.72:0/1903963791 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5370103760 msgr2=0x7f5370109ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:36.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.901+0000 7f5375847640 1 --2- 172.21.15.72:0/1903963791 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5370103760 0x7f5370109ff0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f5368009ca0 tx=0x7f536802c0f0 comp rx=0 tx=0).stop 2024-03-22T01:36:36.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.901+0000 7f5375847640 1 -- 172.21.15.72:0/1903963791 shutdown_connections 2024-03-22T01:36:36.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.901+0000 7f5375847640 1 --2- 172.21.15.72:0/1903963791 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5370103760 0x7f5370109ff0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:36.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.901+0000 7f5375847640 1 --2- 172.21.15.72:0/1903963791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5370102da0 0x7f5370103220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:36.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.901+0000 7f5375847640 1 --2- 172.21.15.72:0/1903963791 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5370101bb0 0x7f5370101fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:36.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.901+0000 7f5375847640 1 -- 172.21.15.72:0/1903963791 >> 172.21.15.72:0/1903963791 conn(0x7f53700fd120 msgr2=0x7f53700ff560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:36.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.901+0000 7f5375847640 1 -- 172.21.15.72:0/1903963791 shutdown_connections 2024-03-22T01:36:36.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.901+0000 7f5375847640 1 -- 172.21.15.72:0/1903963791 wait complete. 2024-03-22T01:36:36.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.901+0000 7f5375847640 1 Processor -- start 2024-03-22T01:36:36.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.902+0000 7f5375847640 1 -- start start 2024-03-22T01:36:36.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.902+0000 7f5375847640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5370101bb0 0x7f5370197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:36.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.902+0000 7f5375847640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5370102da0 0x7f5370198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:36.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.902+0000 7f5375847640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5370103760 0x7f537019f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:36.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.902+0000 7f5375847640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f537010d830 con 0x7f5370103760 2024-03-22T01:36:36.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.902+0000 7f5375847640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f537010d6b0 con 0x7f5370102da0 2024-03-22T01:36:36.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.902+0000 7f5375847640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f537010d9b0 con 0x7f5370101bb0 2024-03-22T01:36:36.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.902+0000 7f536f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5370103760 0x7f537019f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:36.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.902+0000 7f536e7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5370102da0 0x7f5370198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:36.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.902+0000 7f536effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5370101bb0 0x7f5370197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:36.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.902+0000 7f536f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5370103760 0x7f537019f400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54270/0 (socket says 172.21.15.72:54270) 2024-03-22T01:36:36.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.902+0000 7f536e7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5370102da0 0x7f5370198390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.72:49964/0 (socket says 172.21.15.72:49964) 2024-03-22T01:36:36.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.902+0000 7f536f7fe640 1 -- 172.21.15.72:0/408405625 learned_addr learned my addr 172.21.15.72:0/408405625 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:36.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.902+0000 7f536effd640 1 -- 172.21.15.72:0/408405625 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5370102da0 msgr2=0x7f5370198390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:36.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.902+0000 7f536effd640 1 --2- 172.21.15.72:0/408405625 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5370102da0 0x7f5370198390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:36.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.902+0000 7f536effd640 1 -- 172.21.15.72:0/408405625 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5370103760 msgr2=0x7f537019f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:36.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.902+0000 7f536effd640 1 --2- 172.21.15.72:0/408405625 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5370103760 0x7f537019f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:36.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.902+0000 7f536effd640 1 -- 172.21.15.72:0/408405625 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f537019fa70 con 0x7f5370101bb0 2024-03-22T01:36:36.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.902+0000 7f536e7fc640 1 --2- 172.21.15.72:0/408405625 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5370102da0 0x7f5370198390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:36:36.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.902+0000 7f536f7fe640 1 --2- 172.21.15.72:0/408405625 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5370103760 0x7f537019f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:36:36.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.902+0000 7f536effd640 1 --2- 172.21.15.72:0/408405625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5370101bb0 0x7f5370197e50 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f536400e9c0 tx=0x7f536400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:36.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.903+0000 7f5374845640 1 -- 172.21.15.72:0/408405625 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f536400cd80 con 0x7f5370101bb0 2024-03-22T01:36:36.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.903+0000 7f5374845640 1 -- 172.21.15.72:0/408405625 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5364010910 con 0x7f5370101bb0 2024-03-22T01:36:36.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.903+0000 7f5375847640 1 -- 172.21.15.72:0/408405625 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f537019fd60 con 0x7f5370101bb0 2024-03-22T01:36:36.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.903+0000 7f5374845640 1 -- 172.21.15.72:0/408405625 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5364018980 con 0x7f5370101bb0 2024-03-22T01:36:36.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.903+0000 7f5375847640 1 -- 172.21.15.72:0/408405625 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53701a01f0 con 0x7f5370101bb0 2024-03-22T01:36:36.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.904+0000 7f5375847640 1 -- 172.21.15.72:0/408405625 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5370108e70 con 0x7f5370101bb0 2024-03-22T01:36:36.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.905+0000 7f5374845640 1 -- 172.21.15.72:0/408405625 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) ==== 137871+0+0 (secure 0 0 0) 0x7f536401f020 con 0x7f5370101bb0 2024-03-22T01:36:36.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.905+0000 7f5374845640 1 --2- 172.21.15.72:0/408405625 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f53580a2f70 0x7f53580a5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:36.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.905+0000 7f536e7fc640 1 -- 172.21.15.72:0/408405625 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f53580a2f70 msgr2=0x7f53580a5430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.72:6800/401403200 2024-03-22T01:36:36.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.905+0000 7f5374845640 1 -- 172.21.15.72:0/408405625 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f53640cfd80 con 0x7f5370101bb0 2024-03-22T01:36:36.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.905+0000 7f536e7fc640 1 --2- 172.21.15.72:0/408405625 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f53580a2f70 0x7f53580a5430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-22T01:36:36.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:36.909+0000 7f5374845640 1 -- 172.21.15.72:0/408405625 <== mon.1 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f536409c780 con 0x7f5370101bb0 2024-03-22T01:36:37.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.086+0000 7f5375847640 1 -- 172.21.15.72:0/408405625 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f537002d3b0 con 0x7f5370101bb0 2024-03-22T01:36:37.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.086+0000 7f5374845640 1 -- 172.21.15.72:0/408405625 <== mon.1 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v95) ==== 93+0+10 (secure 0 0 0) 0x7f536409c080 con 0x7f5370101bb0 2024-03-22T01:36:37.085 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-22T01:36:37.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.089+0000 7f5375847640 1 -- 172.21.15.72:0/408405625 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f53580a2f70 msgr2=0x7f53580a5430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-22T01:36:37.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.089+0000 7f5375847640 1 --2- 172.21.15.72:0/408405625 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f53580a2f70 0x7f53580a5430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:37.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.089+0000 7f5375847640 1 -- 172.21.15.72:0/408405625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5370101bb0 msgr2=0x7f5370197e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:37.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.089+0000 7f5375847640 1 --2- 172.21.15.72:0/408405625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5370101bb0 0x7f5370197e50 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f536400e9c0 tx=0x7f536400ee80 comp rx=0 tx=0).stop 2024-03-22T01:36:37.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.089+0000 7f5375847640 1 -- 172.21.15.72:0/408405625 shutdown_connections 2024-03-22T01:36:37.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.089+0000 7f5375847640 1 --2- 172.21.15.72:0/408405625 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f53580a2f70 0x7f53580a5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:37.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.089+0000 7f5375847640 1 --2- 172.21.15.72:0/408405625 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5370102da0 0x7f5370198390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:37.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.089+0000 7f5375847640 1 --2- 172.21.15.72:0/408405625 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5370103760 0x7f537019f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:37.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.089+0000 7f5375847640 1 --2- 172.21.15.72:0/408405625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5370101bb0 0x7f5370197e50 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:37.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.089+0000 7f5375847640 1 -- 172.21.15.72:0/408405625 >> 172.21.15.72:0/408405625 conn(0x7f53700fd120 msgr2=0x7f5370106040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:37.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.089+0000 7f5375847640 1 -- 172.21.15.72:0/408405625 shutdown_connections 2024-03-22T01:36:37.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.089+0000 7f5375847640 1 -- 172.21.15.72:0/408405625 wait complete. 2024-03-22T01:36:37.098 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:36 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/2432926360' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:36:37.099 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-22T01:36:37.107 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:36 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/2432926360' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:36:37.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:36 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/2432926360' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:36:37.171 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:36:36 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-z[38319]: 2024-03-22T01:36:36.948+0000 7f30d2ea1200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-22T01:36:37.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.207+0000 7f1aabec3640 1 -- 172.21.15.72:0/3394979553 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1aa4102660 msgr2=0x7f1aa4108ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:37.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.207+0000 7f1aabec3640 1 --2- 172.21.15.72:0/3394979553 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1aa4102660 0x7f1aa4108ef0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c009ca0 tx=0x7f1a9c02c0f0 comp rx=0 tx=0).stop 2024-03-22T01:36:37.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.207+0000 7f1aabec3640 1 -- 172.21.15.72:0/3394979553 shutdown_connections 2024-03-22T01:36:37.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.207+0000 7f1aabec3640 1 --2- 172.21.15.72:0/3394979553 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1aa4102660 0x7f1aa4108ef0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:37.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.207+0000 7f1aabec3640 1 --2- 172.21.15.72:0/3394979553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1aa4101ca0 0x7f1aa4102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:37.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.207+0000 7f1aabec3640 1 --2- 172.21.15.72:0/3394979553 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1aa4100ab0 0x7f1aa4100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:37.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.207+0000 7f1aabec3640 1 -- 172.21.15.72:0/3394979553 >> 172.21.15.72:0/3394979553 conn(0x7f1aa40fc020 msgr2=0x7f1aa40fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:37.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.207+0000 7f1aabec3640 1 -- 172.21.15.72:0/3394979553 shutdown_connections 2024-03-22T01:36:37.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.207+0000 7f1aabec3640 1 -- 172.21.15.72:0/3394979553 wait complete. 2024-03-22T01:36:37.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.207+0000 7f1aabec3640 1 Processor -- start 2024-03-22T01:36:37.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.207+0000 7f1aabec3640 1 -- start start 2024-03-22T01:36:37.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.208+0000 7f1aabec3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1aa4100ab0 0x7f1aa4196d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:37.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.208+0000 7f1aabec3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1aa4101ca0 0x7f1aa4197290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:37.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.208+0000 7f1aabec3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1aa4102660 0x7f1aa419e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:37.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.208+0000 7f1aabec3640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f1aa410c730 con 0x7f1aa4100ab0 2024-03-22T01:36:37.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.208+0000 7f1aabec3640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1aa410c5b0 con 0x7f1aa4102660 2024-03-22T01:36:37.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.208+0000 7f1aabec3640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1aa410c8b0 con 0x7f1aa4101ca0 2024-03-22T01:36:37.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.208+0000 7f1aa9c5f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1aa4100ab0 0x7f1aa4196d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:37.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.208+0000 7f1aa9c5f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1aa4100ab0 0x7f1aa4196d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54296/0 (socket says 172.21.15.72:54296) 2024-03-22T01:36:37.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.208+0000 7f1aa9c5f640 1 -- 172.21.15.72:0/2284643037 learned_addr learned my addr 172.21.15.72:0/2284643037 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:37.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.208+0000 7f1aa945e640 1 --2- 172.21.15.72:0/2284643037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1aa4101ca0 0x7f1aa4197290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:37.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.208+0000 7f1aaa460640 1 --2- 172.21.15.72:0/2284643037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1aa4102660 0x7f1aa419e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:37.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.208+0000 7f1aa9c5f640 1 -- 172.21.15.72:0/2284643037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1aa4101ca0 msgr2=0x7f1aa4197290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:37.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.208+0000 7f1aa9c5f640 1 --2- 172.21.15.72:0/2284643037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1aa4101ca0 0x7f1aa4197290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:37.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.208+0000 7f1aa9c5f640 1 -- 172.21.15.72:0/2284643037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1aa4102660 msgr2=0x7f1aa419e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:37.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.208+0000 7f1aa9c5f640 1 --2- 172.21.15.72:0/2284643037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1aa4102660 0x7f1aa419e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:37.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.208+0000 7f1aa9c5f640 1 -- 172.21.15.72:0/2284643037 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1aa419e970 con 0x7f1aa4100ab0 2024-03-22T01:36:37.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.208+0000 7f1aa9c5f640 1 --2- 172.21.15.72:0/2284643037 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1aa4100ab0 0x7f1aa4196d50 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f1aa000e9c0 tx=0x7f1aa000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:37.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.208+0000 7f1a9affd640 1 -- 172.21.15.72:0/2284643037 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa000cd80 con 0x7f1aa4100ab0 2024-03-22T01:36:37.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.208+0000 7f1a9affd640 1 -- 172.21.15.72:0/2284643037 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1aa00035f0 con 0x7f1aa4100ab0 2024-03-22T01:36:37.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.208+0000 7f1aabec3640 1 -- 172.21.15.72:0/2284643037 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1aa419ec60 con 0x7f1aa4100ab0 2024-03-22T01:36:37.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.208+0000 7f1a9affd640 1 -- 172.21.15.72:0/2284643037 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa0010690 con 0x7f1aa4100ab0 2024-03-22T01:36:37.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.208+0000 7f1aabec3640 1 -- 172.21.15.72:0/2284643037 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1aa419f1b0 con 0x7f1aa4100ab0 2024-03-22T01:36:37.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.210+0000 7f1aabec3640 1 -- 172.21.15.72:0/2284643037 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1aa4107dd0 con 0x7f1aa4100ab0 2024-03-22T01:36:37.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.211+0000 7f1a9affd640 1 -- 172.21.15.72:0/2284643037 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 19) ==== 137871+0+0 (secure 0 0 0) 0x7f1aa00037a0 con 0x7f1aa4100ab0 2024-03-22T01:36:37.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.211+0000 7f1a9affd640 1 --2- 172.21.15.72:0/2284643037 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f1a900a2f70 0x7f1a900a5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:37.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.211+0000 7f1aa945e640 1 -- 172.21.15.72:0/2284643037 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f1a900a2f70 msgr2=0x7f1a900a5430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.72:6800/401403200 2024-03-22T01:36:37.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.211+0000 7f1a9affd640 1 -- 172.21.15.72:0/2284643037 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f1aa00cfbf0 con 0x7f1aa4100ab0 2024-03-22T01:36:37.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.211+0000 7f1aa945e640 1 --2- 172.21.15.72:0/2284643037 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f1a900a2f70 0x7f1a900a5430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-22T01:36:37.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.216+0000 7f1a9affd640 1 -- 172.21.15.72:0/2284643037 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1aa009c650 con 0x7f1aa4100ab0 2024-03-22T01:36:37.372 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:36:37 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:36:37.241+0000 7fa517128200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-22T01:36:37.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.392+0000 7f1aabec3640 1 -- 172.21.15.72:0/2284643037 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1aa402d5f0 con 0x7f1aa4100ab0 2024-03-22T01:36:37.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.392+0000 7f1a9affd640 1 -- 172.21.15.72:0/2284643037 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v95) ==== 104+0+11 (secure 0 0 0) 0x7f1aa009bf50 con 0x7f1aa4100ab0 2024-03-22T01:36:37.392 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-03-22T01:36:37.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.395+0000 7f1aabec3640 1 -- 172.21.15.72:0/2284643037 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f1a900a2f70 msgr2=0x7f1a900a5430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-22T01:36:37.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.395+0000 7f1aabec3640 1 --2- 172.21.15.72:0/2284643037 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f1a900a2f70 0x7f1a900a5430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:37.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.395+0000 7f1aabec3640 1 -- 172.21.15.72:0/2284643037 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1aa4100ab0 msgr2=0x7f1aa4196d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:37.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.395+0000 7f1aabec3640 1 --2- 172.21.15.72:0/2284643037 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1aa4100ab0 0x7f1aa4196d50 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f1aa000e9c0 tx=0x7f1aa000ee80 comp rx=0 tx=0).stop 2024-03-22T01:36:37.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.395+0000 7f1aabec3640 1 -- 172.21.15.72:0/2284643037 shutdown_connections 2024-03-22T01:36:37.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.395+0000 7f1aabec3640 1 --2- 172.21.15.72:0/2284643037 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f1a900a2f70 0x7f1a900a5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:37.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.395+0000 7f1aabec3640 1 --2- 172.21.15.72:0/2284643037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1aa4102660 0x7f1aa419e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:37.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.395+0000 7f1aabec3640 1 --2- 172.21.15.72:0/2284643037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1aa4101ca0 0x7f1aa4197290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:37.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.395+0000 7f1aabec3640 1 --2- 172.21.15.72:0/2284643037 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1aa4100ab0 0x7f1aa4196d50 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:37.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.395+0000 7f1aabec3640 1 -- 172.21.15.72:0/2284643037 >> 172.21.15.72:0/2284643037 conn(0x7f1aa40fc020 msgr2=0x7f1aa4104fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:37.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.395+0000 7f1aabec3640 1 -- 172.21.15.72:0/2284643037 shutdown_connections 2024-03-22T01:36:37.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.395+0000 7f1aabec3640 1 -- 172.21.15.72:0/2284643037 wait complete. 2024-03-22T01:36:37.406 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-22T01:36:37.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.512+0000 7f10608b5640 1 -- 172.21.15.72:0/3131853130 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f105c102690 msgr2=0x7f105c104a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:37.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.512+0000 7f10608b5640 1 --2- 172.21.15.72:0/3131853130 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f105c102690 0x7f105c104a80 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f104c009900 tx=0x7f104c02bc30 comp rx=0 tx=0).stop 2024-03-22T01:36:37.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.512+0000 7f10608b5640 1 -- 172.21.15.72:0/3131853130 shutdown_connections 2024-03-22T01:36:37.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.512+0000 7f10608b5640 1 --2- 172.21.15.72:0/3131853130 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f105c104fc0 0x7f105c1073b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:37.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.512+0000 7f10608b5640 1 --2- 172.21.15.72:0/3131853130 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f105c1078f0 0x7f105c109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:37.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.512+0000 7f10608b5640 1 --2- 172.21.15.72:0/3131853130 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f105c102690 0x7f105c104a80 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:37.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.512+0000 7f10608b5640 1 -- 172.21.15.72:0/3131853130 >> 172.21.15.72:0/3131853130 conn(0x7f105c0fc020 msgr2=0x7f105c0fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:37.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.513+0000 7f10608b5640 1 -- 172.21.15.72:0/3131853130 shutdown_connections 2024-03-22T01:36:37.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.513+0000 7f10608b5640 1 -- 172.21.15.72:0/3131853130 wait complete. 2024-03-22T01:36:37.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.513+0000 7f10608b5640 1 Processor -- start 2024-03-22T01:36:37.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.513+0000 7f10608b5640 1 -- start start 2024-03-22T01:36:37.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.513+0000 7f10608b5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f105c102690 0x7f105c197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:37.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.513+0000 7f10608b5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f105c104fc0 0x7f105c198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:37.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.513+0000 7f10608b5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f105c1078f0 0x7f105c19f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:37.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.513+0000 7f10608b5640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f105c10d850 con 0x7f105c1078f0 2024-03-22T01:36:37.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.513+0000 7f10608b5640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f105c10d6d0 con 0x7f105c104fc0 2024-03-22T01:36:37.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.513+0000 7f10608b5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f105c10d9d0 con 0x7f105c102690 2024-03-22T01:36:37.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.514+0000 7f105a59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f105c102690 0x7f105c197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:37.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.514+0000 7f105a59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f105c102690 0x7f105c197ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.72:44604/0 (socket says 172.21.15.72:44604) 2024-03-22T01:36:37.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.514+0000 7f105ad9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f105c1078f0 0x7f105c19f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:37.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.514+0000 7f105a59c640 1 -- 172.21.15.72:0/2253815746 learned_addr learned my addr 172.21.15.72:0/2253815746 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:37.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.514+0000 7f1059d9b640 1 --2- 172.21.15.72:0/2253815746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f105c104fc0 0x7f105c198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:37.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.514+0000 7f105a59c640 1 -- 172.21.15.72:0/2253815746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f105c104fc0 msgr2=0x7f105c198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:37.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.514+0000 7f105a59c640 1 --2- 172.21.15.72:0/2253815746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f105c104fc0 0x7f105c198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:37.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.514+0000 7f105a59c640 1 -- 172.21.15.72:0/2253815746 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f105c1078f0 msgr2=0x7f105c19f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:37.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.514+0000 7f105a59c640 1 --2- 172.21.15.72:0/2253815746 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f105c1078f0 0x7f105c19f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:37.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.514+0000 7f105a59c640 1 -- 172.21.15.72:0/2253815746 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f105c19faf0 con 0x7f105c102690 2024-03-22T01:36:37.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.514+0000 7f105ad9d640 1 --2- 172.21.15.72:0/2253815746 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f105c1078f0 0x7f105c19f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:36:37.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.514+0000 7f105a59c640 1 --2- 172.21.15.72:0/2253815746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f105c102690 0x7f105c197ed0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f104c009230 tx=0x7f104c02c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:37.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.514+0000 7f104b7fe640 1 -- 172.21.15.72:0/2253815746 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f104c041070 con 0x7f105c102690 2024-03-22T01:36:37.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.514+0000 7f10608b5640 1 -- 172.21.15.72:0/2253815746 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f105c19fd80 con 0x7f105c102690 2024-03-22T01:36:37.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.514+0000 7f104b7fe640 1 -- 172.21.15.72:0/2253815746 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f104c004450 con 0x7f105c102690 2024-03-22T01:36:37.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.514+0000 7f104b7fe640 1 -- 172.21.15.72:0/2253815746 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f104c003e20 con 0x7f105c102690 2024-03-22T01:36:37.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.514+0000 7f10608b5640 1 -- 172.21.15.72:0/2253815746 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f105c1a0200 con 0x7f105c102690 2024-03-22T01:36:37.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.515+0000 7f10608b5640 1 -- 172.21.15.72:0/2253815746 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f105c0fdb10 con 0x7f105c102690 2024-03-22T01:36:37.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.517+0000 7f104b7fe640 1 -- 172.21.15.72:0/2253815746 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) ==== 137871+0+0 (secure 0 0 0) 0x7f104c044070 con 0x7f105c102690 2024-03-22T01:36:37.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.517+0000 7f104b7fe640 1 --2- 172.21.15.72:0/2253815746 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f10440a2f70 0x7f10440a5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:37.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.517+0000 7f1059d9b640 1 -- 172.21.15.72:0/2253815746 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f10440a2f70 msgr2=0x7f10440a5430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.72:6800/401403200 2024-03-22T01:36:37.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.517+0000 7f104b7fe640 1 -- 172.21.15.72:0/2253815746 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f104c0f16b0 con 0x7f105c102690 2024-03-22T01:36:37.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.517+0000 7f1059d9b640 1 --2- 172.21.15.72:0/2253815746 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f10440a2f70 0x7f10440a5430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-22T01:36:37.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.525+0000 7f104b7fe640 1 -- 172.21.15.72:0/2253815746 <== mon.1 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f104c0bdac0 con 0x7f105c102690 2024-03-22T01:36:37.607 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:36:37 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-y[37164]: 2024-03-22T01:36:37.191+0000 7fa3858b5200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-22T01:36:37.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.702+0000 7f10608b5640 1 -- 172.21.15.72:0/2253815746 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f105c02d540 con 0x7f105c102690 2024-03-22T01:36:37.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.703+0000 7f104b7fe640 1 -- 172.21.15.72:0/2253815746 <== mon.1 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v95) ==== 100+0+11 (secure 0 0 0) 0x7f104c0c1b60 con 0x7f105c102690 2024-03-22T01:36:37.702 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-03-22T01:36:37.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.705+0000 7f10608b5640 1 -- 172.21.15.72:0/2253815746 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f10440a2f70 msgr2=0x7f10440a5430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-22T01:36:37.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.705+0000 7f10608b5640 1 --2- 172.21.15.72:0/2253815746 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f10440a2f70 0x7f10440a5430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:37.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.705+0000 7f10608b5640 1 -- 172.21.15.72:0/2253815746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f105c102690 msgr2=0x7f105c197ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:37.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.705+0000 7f10608b5640 1 --2- 172.21.15.72:0/2253815746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f105c102690 0x7f105c197ed0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f104c009230 tx=0x7f104c02c650 comp rx=0 tx=0).stop 2024-03-22T01:36:37.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.705+0000 7f10608b5640 1 -- 172.21.15.72:0/2253815746 shutdown_connections 2024-03-22T01:36:37.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.705+0000 7f10608b5640 1 --2- 172.21.15.72:0/2253815746 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f10440a2f70 0x7f10440a5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:37.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.705+0000 7f10608b5640 1 --2- 172.21.15.72:0/2253815746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f105c104fc0 0x7f105c198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:37.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.705+0000 7f10608b5640 1 --2- 172.21.15.72:0/2253815746 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f105c1078f0 0x7f105c19f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:37.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.705+0000 7f10608b5640 1 --2- 172.21.15.72:0/2253815746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f105c102690 0x7f105c197ed0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:37.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.705+0000 7f10608b5640 1 -- 172.21.15.72:0/2253815746 >> 172.21.15.72:0/2253815746 conn(0x7f105c0fc020 msgr2=0x7f105c102f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:37.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.705+0000 7f10608b5640 1 -- 172.21.15.72:0/2253815746 shutdown_connections 2024-03-22T01:36:37.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.705+0000 7f10608b5640 1 -- 172.21.15.72:0/2253815746 wait complete. 2024-03-22T01:36:37.714 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-22T01:36:37.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.821+0000 7f75aed5c640 1 -- 172.21.15.72:0/2345909688 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f75a8100ab0 msgr2=0x7f75a8100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:37.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.821+0000 7f75aed5c640 1 --2- 172.21.15.72:0/2345909688 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f75a8100ab0 0x7f75a8100ed0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f759c009900 tx=0x7f759c02bc30 comp rx=0 tx=0).stop 2024-03-22T01:36:37.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.821+0000 7f75aed5c640 1 -- 172.21.15.72:0/2345909688 shutdown_connections 2024-03-22T01:36:37.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.821+0000 7f75aed5c640 1 --2- 172.21.15.72:0/2345909688 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75a8101ca0 0x7f75a8102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:37.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.821+0000 7f75aed5c640 1 --2- 172.21.15.72:0/2345909688 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75a8102660 0x7f75a8108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:37.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.821+0000 7f75aed5c640 1 --2- 172.21.15.72:0/2345909688 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f75a8100ab0 0x7f75a8100ed0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:37.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.821+0000 7f75aed5c640 1 -- 172.21.15.72:0/2345909688 >> 172.21.15.72:0/2345909688 conn(0x7f75a80fc020 msgr2=0x7f75a80fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:37.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.821+0000 7f75aed5c640 1 -- 172.21.15.72:0/2345909688 shutdown_connections 2024-03-22T01:36:37.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.821+0000 7f75aed5c640 1 -- 172.21.15.72:0/2345909688 wait complete. 2024-03-22T01:36:37.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.822+0000 7f75aed5c640 1 Processor -- start 2024-03-22T01:36:37.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.822+0000 7f75aed5c640 1 -- start start 2024-03-22T01:36:37.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.822+0000 7f75aed5c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75a8100ab0 0x7f75a8197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:37.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.822+0000 7f75aed5c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f75a8101ca0 0x7f75a81984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:37.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.822+0000 7f75aed5c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75a8102660 0x7f75a819f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:37.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.822+0000 7f75aed5c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f75a810c730 con 0x7f75a8101ca0 2024-03-22T01:36:37.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.822+0000 7f75aed5c640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f75a810c5b0 con 0x7f75a8102660 2024-03-22T01:36:37.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.822+0000 7f75aed5c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f75a810c8b0 con 0x7f75a8100ab0 2024-03-22T01:36:37.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.822+0000 7f75a7fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f75a8101ca0 0x7f75a81984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:37.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.822+0000 7f75a7fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f75a8101ca0 0x7f75a81984a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54328/0 (socket says 172.21.15.72:54328) 2024-03-22T01:36:37.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.822+0000 7f75a7fff640 1 -- 172.21.15.72:0/805035586 learned_addr learned my addr 172.21.15.72:0/805035586 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:37.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.822+0000 7f75ad2f9640 1 --2- 172.21.15.72:0/805035586 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75a8102660 0x7f75a819f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:37.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.822+0000 7f75acaf8640 1 --2- 172.21.15.72:0/805035586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75a8100ab0 0x7f75a8197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:37.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.822+0000 7f75a7fff640 1 -- 172.21.15.72:0/805035586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75a8100ab0 msgr2=0x7f75a8197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:37.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.822+0000 7f75a7fff640 1 --2- 172.21.15.72:0/805035586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75a8100ab0 0x7f75a8197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:37.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.822+0000 7f75a7fff640 1 -- 172.21.15.72:0/805035586 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75a8102660 msgr2=0x7f75a819f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:37.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.822+0000 7f75a7fff640 1 --2- 172.21.15.72:0/805035586 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75a8102660 0x7f75a819f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:37.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.822+0000 7f75a7fff640 1 -- 172.21.15.72:0/805035586 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75a819fb30 con 0x7f75a8101ca0 2024-03-22T01:36:37.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.822+0000 7f75ad2f9640 1 --2- 172.21.15.72:0/805035586 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75a8102660 0x7f75a819f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:36:37.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.823+0000 7f75a7fff640 1 --2- 172.21.15.72:0/805035586 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f75a8101ca0 0x7f75a81984a0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f759400e9f0 tx=0x7f759400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:37.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.823+0000 7f75a5ffb640 1 -- 172.21.15.72:0/805035586 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f759400cd80 con 0x7f75a8101ca0 2024-03-22T01:36:37.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.823+0000 7f75a5ffb640 1 -- 172.21.15.72:0/805035586 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7594010910 con 0x7f75a8101ca0 2024-03-22T01:36:37.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.823+0000 7f75a5ffb640 1 -- 172.21.15.72:0/805035586 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7594018990 con 0x7f75a8101ca0 2024-03-22T01:36:37.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.823+0000 7f75aed5c640 1 -- 172.21.15.72:0/805035586 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75a819fe20 con 0x7f75a8101ca0 2024-03-22T01:36:37.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.823+0000 7f75aed5c640 1 -- 172.21.15.72:0/805035586 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75a81a02e0 con 0x7f75a8101ca0 2024-03-22T01:36:37.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.824+0000 7f75aed5c640 1 -- 172.21.15.72:0/805035586 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75a81a0510 con 0x7f75a8101ca0 2024-03-22T01:36:37.828 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:36:37 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-z[38319]: 2024-03-22T01:36:37.512+0000 7f30d2ea1200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-22T01:36:37.828 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:36:37 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-z[38319]: 2024-03-22T01:36:37.613+0000 7f30d2ea1200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-22T01:36:37.828 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:36:37 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-z[38319]: 2024-03-22T01:36:37.828+0000 7f30d2ea1200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-22T01:36:37.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.826+0000 7f75a5ffb640 1 -- 172.21.15.72:0/805035586 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 19) ==== 137871+0+0 (secure 0 0 0) 0x7f7594010ac0 con 0x7f75a8101ca0 2024-03-22T01:36:37.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.827+0000 7f75a5ffb640 1 --2- 172.21.15.72:0/805035586 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f75980a2f70 0x7f75980a5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:37.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.827+0000 7f75acaf8640 1 -- 172.21.15.72:0/805035586 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f75980a2f70 msgr2=0x7f75980a5430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.72:6800/401403200 2024-03-22T01:36:37.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.827+0000 7f75a5ffb640 1 -- 172.21.15.72:0/805035586 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f75940cff30 con 0x7f75a8101ca0 2024-03-22T01:36:37.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.827+0000 7f75acaf8640 1 --2- 172.21.15.72:0/805035586 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f75980a2f70 0x7f75980a5430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-22T01:36:37.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:37.830+0000 7f75a5ffb640 1 -- 172.21.15.72:0/805035586 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f759409c340 con 0x7f75a8101ca0 2024-03-22T01:36:38.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.009+0000 7f75aed5c640 1 -- 172.21.15.72:0/805035586 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f75a81a06e0 con 0x7f75a8101ca0 2024-03-22T01:36:38.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.009+0000 7f75a5ffb640 1 -- 172.21.15.72:0/805035586 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v95) ==== 103+0+11 (secure 0 0 0) 0x7f75940a03e0 con 0x7f75a8101ca0 2024-03-22T01:36:38.009 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-03-22T01:36:38.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.011+0000 7f75aed5c640 1 -- 172.21.15.72:0/805035586 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f75980a2f70 msgr2=0x7f75980a5430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-22T01:36:38.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.011+0000 7f75aed5c640 1 --2- 172.21.15.72:0/805035586 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f75980a2f70 0x7f75980a5430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:38.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.011+0000 7f75aed5c640 1 -- 172.21.15.72:0/805035586 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f75a8101ca0 msgr2=0x7f75a81984a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:38.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.012+0000 7f75aed5c640 1 --2- 172.21.15.72:0/805035586 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f75a8101ca0 0x7f75a81984a0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f759400e9f0 tx=0x7f759400eeb0 comp rx=0 tx=0).stop 2024-03-22T01:36:38.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.012+0000 7f75aed5c640 1 -- 172.21.15.72:0/805035586 shutdown_connections 2024-03-22T01:36:38.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.012+0000 7f75aed5c640 1 --2- 172.21.15.72:0/805035586 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7f75980a2f70 0x7f75980a5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:38.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.012+0000 7f75aed5c640 1 --2- 172.21.15.72:0/805035586 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75a8102660 0x7f75a819f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:38.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.012+0000 7f75aed5c640 1 --2- 172.21.15.72:0/805035586 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f75a8101ca0 0x7f75a81984a0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:38.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.012+0000 7f75aed5c640 1 --2- 172.21.15.72:0/805035586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75a8100ab0 0x7f75a8197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:38.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.012+0000 7f75aed5c640 1 -- 172.21.15.72:0/805035586 >> 172.21.15.72:0/805035586 conn(0x7f75a80fc020 msgr2=0x7f75a8068e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:38.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.012+0000 7f75aed5c640 1 -- 172.21.15.72:0/805035586 shutdown_connections 2024-03-22T01:36:38.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.012+0000 7f75aed5c640 1 -- 172.21.15.72:0/805035586 wait complete. 2024-03-22T01:36:38.020 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:37 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/408405625' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T01:36:38.021 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:37 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/2284643037' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T01:36:38.021 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:37 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/2253815746' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T01:36:38.021 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:36:37 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:36:37.831+0000 7fa517128200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-22T01:36:38.021 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:36:37 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:36:37.935+0000 7fa517128200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-22T01:36:38.022 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T01:36:38.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:37 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/408405625' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T01:36:38.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:37 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/2284643037' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T01:36:38.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:37 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/2253815746' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T01:36:38.108 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:36:37 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-y[37164]: 2024-03-22T01:36:37.795+0000 7fa3858b5200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-22T01:36:38.108 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:36:37 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-y[37164]: 2024-03-22T01:36:37.913+0000 7fa3858b5200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-22T01:36:38.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.126+0000 7ff8e635e640 1 -- 172.21.15.72:0/1951632355 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff8e0102670 msgr2=0x7ff8e0108f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:38.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.126+0000 7ff8e635e640 1 --2- 172.21.15.72:0/1951632355 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff8e0102670 0x7ff8e0108f00 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7ff8d8009ca0 tx=0x7ff8d802c0f0 comp rx=0 tx=0).stop 2024-03-22T01:36:38.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.127+0000 7ff8e635e640 1 -- 172.21.15.72:0/1951632355 shutdown_connections 2024-03-22T01:36:38.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.127+0000 7ff8e635e640 1 --2- 172.21.15.72:0/1951632355 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff8e0102670 0x7ff8e0108f00 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:38.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.127+0000 7ff8e635e640 1 --2- 172.21.15.72:0/1951632355 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8e0101cb0 0x7ff8e0102130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:38.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.127+0000 7ff8e635e640 1 --2- 172.21.15.72:0/1951632355 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff8e0100ac0 0x7ff8e0100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:38.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.127+0000 7ff8e635e640 1 -- 172.21.15.72:0/1951632355 >> 172.21.15.72:0/1951632355 conn(0x7ff8e00fc030 msgr2=0x7ff8e00fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:38.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.127+0000 7ff8e635e640 1 -- 172.21.15.72:0/1951632355 shutdown_connections 2024-03-22T01:36:38.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.127+0000 7ff8e635e640 1 -- 172.21.15.72:0/1951632355 wait complete. 2024-03-22T01:36:38.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.127+0000 7ff8e635e640 1 Processor -- start 2024-03-22T01:36:38.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.127+0000 7ff8e635e640 1 -- start start 2024-03-22T01:36:38.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.127+0000 7ff8e635e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff8e0100ac0 0x7ff8e0104a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:38.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.127+0000 7ff8e635e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff8e0101cb0 0x7ff8e0103090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:38.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.128+0000 7ff8e635e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8e0102670 0x7ff8e01035d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:38.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.128+0000 7ff8e635e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7ff8e010c740 con 0x7ff8e0100ac0 2024-03-22T01:36:38.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.128+0000 7ff8e635e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff8e010c5c0 con 0x7ff8e0101cb0 2024-03-22T01:36:38.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.128+0000 7ff8e635e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff8e010c8c0 con 0x7ff8e0102670 2024-03-22T01:36:38.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.128+0000 7ff8dffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff8e0100ac0 0x7ff8e0104a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:38.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.128+0000 7ff8dffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff8e0100ac0 0x7ff8e0104a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54352/0 (socket says 172.21.15.72:54352) 2024-03-22T01:36:38.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.128+0000 7ff8dffff640 1 -- 172.21.15.72:0/154504233 learned_addr learned my addr 172.21.15.72:0/154504233 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:38.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.128+0000 7ff8df7fe640 1 --2- 172.21.15.72:0/154504233 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff8e0101cb0 0x7ff8e0103090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:38.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.128+0000 7ff8e48fb640 1 --2- 172.21.15.72:0/154504233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8e0102670 0x7ff8e01035d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:38.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.128+0000 7ff8dffff640 1 -- 172.21.15.72:0/154504233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8e0102670 msgr2=0x7ff8e01035d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:38.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.128+0000 7ff8dffff640 1 --2- 172.21.15.72:0/154504233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8e0102670 0x7ff8e01035d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:38.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.128+0000 7ff8dffff640 1 -- 172.21.15.72:0/154504233 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff8e0101cb0 msgr2=0x7ff8e0103090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:38.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.128+0000 7ff8dffff640 1 --2- 172.21.15.72:0/154504233 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff8e0101cb0 0x7ff8e0103090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:38.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.128+0000 7ff8dffff640 1 -- 172.21.15.72:0/154504233 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8e0103e60 con 0x7ff8e0100ac0 2024-03-22T01:36:38.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.128+0000 7ff8dffff640 1 --2- 172.21.15.72:0/154504233 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff8e0100ac0 0x7ff8e0104a50 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7ff8d000e9c0 tx=0x7ff8d000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:38.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.128+0000 7ff8dd7fa640 1 -- 172.21.15.72:0/154504233 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d000cd80 con 0x7ff8e0100ac0 2024-03-22T01:36:38.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.128+0000 7ff8dd7fa640 1 -- 172.21.15.72:0/154504233 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8d00035f0 con 0x7ff8e0100ac0 2024-03-22T01:36:38.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.128+0000 7ff8e635e640 1 -- 172.21.15.72:0/154504233 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8e01a1f70 con 0x7ff8e0100ac0 2024-03-22T01:36:38.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.128+0000 7ff8dd7fa640 1 -- 172.21.15.72:0/154504233 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d0010690 con 0x7ff8e0100ac0 2024-03-22T01:36:38.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.128+0000 7ff8e635e640 1 -- 172.21.15.72:0/154504233 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8e01a23d0 con 0x7ff8e0100ac0 2024-03-22T01:36:38.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.129+0000 7ff8e635e640 1 -- 172.21.15.72:0/154504233 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8e01a26c0 con 0x7ff8e0100ac0 2024-03-22T01:36:38.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.130+0000 7ff8dd7fa640 1 -- 172.21.15.72:0/154504233 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 19) ==== 137871+0+0 (secure 0 0 0) 0x7ff8d00037a0 con 0x7ff8e0100ac0 2024-03-22T01:36:38.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.131+0000 7ff8dd7fa640 1 --2- 172.21.15.72:0/154504233 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7ff8c80a2ff0 0x7ff8c80a54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:38.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.131+0000 7ff8df7fe640 1 -- 172.21.15.72:0/154504233 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7ff8c80a2ff0 msgr2=0x7ff8c80a54b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.72:6800/401403200 2024-03-22T01:36:38.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.131+0000 7ff8dd7fa640 1 -- 172.21.15.72:0/154504233 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7ff8d00cfb80 con 0x7ff8e0100ac0 2024-03-22T01:36:38.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.131+0000 7ff8df7fe640 1 --2- 172.21.15.72:0/154504233 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7ff8c80a2ff0 0x7ff8c80a54b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-22T01:36:38.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.135+0000 7ff8dd7fa640 1 -- 172.21.15.72:0/154504233 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff8d009c5e0 con 0x7ff8e0100ac0 2024-03-22T01:36:38.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:37 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/408405625' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T01:36:38.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:37 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/2284643037' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T01:36:38.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:37 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/2253815746' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T01:36:38.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.196+0000 7ff8dd7fa640 1 -- 172.21.15.72:0/154504233 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mgrmap(e 20) ==== 137539+0+0 (secure 0 0 0) 0x7ff8d0098550 con 0x7ff8e0100ac0 2024-03-22T01:36:38.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.196+0000 7ff8dd7fa640 1 -- 172.21.15.72:0/154504233 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7ff8c80a2ff0 msgr2=0x7ff8c80a54b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-22T01:36:38.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.196+0000 7ff8dd7fa640 1 --2- 172.21.15.72:0/154504233 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7ff8c80a2ff0 0x7ff8c80a54b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:38.338 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:36:38.170+0000 7fa517128200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-22T01:36:38.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.359+0000 7ff8e635e640 1 -- 172.21.15.72:0/154504233 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff8e01a2a10 con 0x7ff8e0100ac0 2024-03-22T01:36:38.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.362+0000 7ff8dd7fa640 1 -- 172.21.15.72:0/154504233 <== mon.0 v2:172.21.15.72:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) ==== 74+0+25134 (secure 0 0 0) 0x7ff8d00c7a70 con 0x7ff8e0100ac0 2024-03-22T01:36:38.361 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:36:38.361 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":96,"fsid":"022a99bc-e7eb-11ee-95cd-87774f69a715","created":"2024-03-22T01:26:23.893672+0000","modified":"2024-03-22T01:36:38.183837+0000","last_up_change":"2024-03-22T01:33:25.223684+0000","last_in_change":"2024-03-22T01:33:05.010331+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T01:29:37.438503+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-22T01:34:12.610422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-22T01:34:13.761686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-22T01:34:17.507039+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"16c15bb0-db92-4191-9063-38b4890e81e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6803","nonce":661906791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6805","nonce":661906791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6809","nonce":661906791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6807","nonce":661906791}]},"public_addr":"172.21.15.72:6803/661906791","cluster_addr":"172.21.15.72:6805/661906791","heartbeat_back_addr":"172.21.15.72:6809/661906791","heartbeat_front_addr":"172.21.15.72:6807/661906791","state":["exists","up"]},{"osd":1,"uuid":"981290ba-97dc-496b-aa22-d51704184b10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6801","nonce":2870590550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6803","nonce":2870590550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6807","nonce":2870590550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6805","nonce":2870590550}]},"public_addr":"172.21.15.89:6801/2870590550","cluster_addr":"172.21.15.89:6803/2870590550","heartbeat_back_addr":"172.21.15.89:6807/2870590550","heartbeat_front_addr":"172.21.15.89:6805/2870590550","state":["exists","up"]},{"osd":2,"uuid":"f07ed658-f545-46a1-8edd-8b1d5d49885c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6801","nonce":3865778118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6803","nonce":3865778118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6807","nonce":3865778118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6805","nonce":3865778118}]},"public_addr":"172.21.15.98:6801/3865778118","cluster_addr":"172.21.15.98:6803/3865778118","heartbeat_back_addr":"172.21.15.98:6807/3865778118","heartbeat_front_addr":"172.21.15.98:6805/3865778118","state":["exists","up"]},{"osd":3,"uuid":"ccc4bc40-2c48-46d2-8548-0c05b44f9f1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6811","nonce":1865951395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6813","nonce":1865951395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6817","nonce":1865951395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6815","nonce":1865951395}]},"public_addr":"172.21.15.72:6811/1865951395","cluster_addr":"172.21.15.72:6813/1865951395","heartbeat_back_addr":"172.21.15.72:6817/1865951395","heartbeat_front_addr":"172.21.15.72:6815/1865951395","state":["exists","up"]},{"osd":4,"uuid":"1812439f-810a-4ed2-9395-c00c9a085a13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6809","nonce":2683760795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6811","nonce":2683760795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6815","nonce":2683760795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6813","nonce":2683760795}]},"public_addr":"172.21.15.89:6809/2683760795","cluster_addr":"172.21.15.89:6811/2683760795","heartbeat_back_addr":"172.21.15.89:6815/2683760795","heartbeat_front_addr":"172.21.15.89:6813/2683760795","state":["exists","up"]},{"osd":5,"uuid":"e78954d8-f4e3-4d58-8536-d2d59c843386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6809","nonce":2635838572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6811","nonce":2635838572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6815","nonce":2635838572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6813","nonce":2635838572}]},"public_addr":"172.21.15.98:6809/2635838572","cluster_addr":"172.21.15.98:6811/2635838572","heartbeat_back_addr":"172.21.15.98:6815/2635838572","heartbeat_front_addr":"172.21.15.98:6813/2635838572","state":["exists","up"]},{"osd":6,"uuid":"dc6c8be2-b64c-4959-8015-7b0ea6d25599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6819","nonce":3158982661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6821","nonce":3158982661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6825","nonce":3158982661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6823","nonce":3158982661}]},"public_addr":"172.21.15.72:6819/3158982661","cluster_addr":"172.21.15.72:6821/3158982661","heartbeat_back_addr":"172.21.15.72:6825/3158982661","heartbeat_front_addr":"172.21.15.72:6823/3158982661","state":["exists","up"]},{"osd":7,"uuid":"7a73469f-cf3a-4306-ae93-ea46e21ebbbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6817","nonce":3075913234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6819","nonce":3075913234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6823","nonce":3075913234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6821","nonce":3075913234}]},"public_addr":"172.21.15.89:6817/3075913234","cluster_addr":"172.21.15.89:6819/3075913234","heartbeat_back_addr":"172.21.15.89:6823/3075913234","heartbeat_front_addr":"172.21.15.89:6821/3075913234","state":["exists","up"]},{"osd":8,"uuid":"c4b13bee-ea4d-4344-95fb-2622ecbb8196","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6817","nonce":1813255577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6819","nonce":1813255577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6823","nonce":1813255577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6821","nonce":1813255577}]},"public_addr":"172.21.15.98:6817/1813255577","cluster_addr":"172.21.15.98:6819/1813255577","heartbeat_back_addr":"172.21.15.98:6823/1813255577","heartbeat_front_addr":"172.21.15.98:6821/1813255577","state":["exists","up"]},{"osd":9,"uuid":"eefb0a44-4a36-4e55-9d04-0501bf718db6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6827","nonce":3377749206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6829","nonce":3377749206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6832","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6833","nonce":3377749206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6831","nonce":3377749206}]},"public_addr":"172.21.15.72:6827/3377749206","cluster_addr":"172.21.15.72:6829/3377749206","heartbeat_back_addr":"172.21.15.72:6833/3377749206","heartbeat_front_addr":"172.21.15.72:6831/3377749206","state":["exists","up"]},{"osd":10,"uuid":"8741d25f-40dd-40d7-b040-5cc5e411eb98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6825","nonce":885058786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6827","nonce":885058786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6831","nonce":885058786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6829","nonce":885058786}]},"public_addr":"172.21.15.89:6825/885058786","cluster_addr":"172.21.15.89:6827/885058786","heartbeat_back_addr":"172.21.15.89:6831/885058786","heartbeat_front_addr":"172.21.15.89:6829/885058786","state":["exists","up"]},{"osd":11,"uuid":"8d02e957-eda4-4184-b5c6-5eb3b8b38060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6825","nonce":1517095838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6827","nonce":1517095838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6831","nonce":1517095838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6829","nonce":1517095838}]},"public_addr":"172.21.15.98:6825/1517095838","cluster_addr":"172.21.15.98:6827/1517095838","heartbeat_back_addr":"172.21.15.98:6831/1517095838","heartbeat_front_addr":"172.21.15.98:6829/1517095838","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:28:47.307652+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:11.037135+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:34.619188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:59.350738+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:30:23.787337+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:30:48.994154+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:31:13.690350+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:31:39.628116+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:04.723912+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:30.689741+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:57.229508+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:33:23.154537+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":7}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:0/2032770149":"2024-03-23T01:36:38.183806+0000","172.21.15.72:6801/401403200":"2024-03-23T01:36:38.183806+0000","172.21.15.72:0/1161486989":"2024-03-23T01:36:38.183806+0000","172.21.15.72:6800/401403200":"2024-03-23T01:36:38.183806+0000","172.21.15.72:0/121087196":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/2451857761":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/1228477995":"2024-03-23T01:27:15.339809+0000","172.21.15.72:6801/1513232913":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/4223447038":"2024-03-23T01:36:38.183806+0000","172.21.15.72:0/3558958264":"2024-03-23T01:26:50.074871+0000","172.21.15.72:0/1510187198":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6800/3518012916":"2024-03-23T01:26:50.074871+0000","172.21.15.72:0/608566205":"2024-03-23T01:36:38.183806+0000","172.21.15.72:6801/3518012916":"2024-03-23T01:26:50.074871+0000","172.21.15.72:0/1372801041":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6800/1513232913":"2024-03-23T01:27:15.339809+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T01:36:38.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.364+0000 7ff8e635e640 1 -- 172.21.15.72:0/154504233 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff8e0100ac0 msgr2=0x7ff8e0104a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:38.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.364+0000 7ff8e635e640 1 --2- 172.21.15.72:0/154504233 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff8e0100ac0 0x7ff8e0104a50 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7ff8d000e9c0 tx=0x7ff8d000ee80 comp rx=0 tx=0).stop 2024-03-22T01:36:38.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.364+0000 7ff8e635e640 1 -- 172.21.15.72:0/154504233 shutdown_connections 2024-03-22T01:36:38.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.364+0000 7ff8e635e640 1 --2- 172.21.15.72:0/154504233 >> [v2:172.21.15.72:6800/401403200,v1:172.21.15.72:6801/401403200] conn(0x7ff8c80a2ff0 0x7ff8c80a54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:38.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.364+0000 7ff8e635e640 1 --2- 172.21.15.72:0/154504233 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff8e0101cb0 0x7ff8e0103090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:38.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.364+0000 7ff8e635e640 1 --2- 172.21.15.72:0/154504233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8e0102670 0x7ff8e01035d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:38.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.364+0000 7ff8e635e640 1 --2- 172.21.15.72:0/154504233 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff8e0100ac0 0x7ff8e0104a50 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:38.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.364+0000 7ff8e635e640 1 -- 172.21.15.72:0/154504233 >> 172.21.15.72:0/154504233 conn(0x7ff8e00fc030 msgr2=0x7ff8e00fda60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:38.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.364+0000 7ff8e635e640 1 -- 172.21.15.72:0/154504233 shutdown_connections 2024-03-22T01:36:38.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.364+0000 7ff8e635e640 1 -- 172.21.15.72:0/154504233 wait complete. 2024-03-22T01:36:38.378 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-22T01:29:37.438503+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-22T01:34:12.610422+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-22T01:34:13.761686+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-22T01:34:17.507039+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-22T01:36:38.379 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-22T01:36:38.443 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:36:38 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-y[37164]: 2024-03-22T01:36:38.186+0000 7fa3858b5200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-22T01:36:38.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.525+0000 7ff7f8466640 1 -- 172.21.15.72:0/1465696154 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff7f0101780 msgr2=0x7ff7f0101c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:38.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.525+0000 7ff7f8466640 1 --2- 172.21.15.72:0/1465696154 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff7f0101780 0x7ff7f0101c00 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7ff7e8009be0 tx=0x7ff7e802bef0 comp rx=0 tx=0).stop 2024-03-22T01:36:38.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.525+0000 7ff7f8466640 1 -- 172.21.15.72:0/1465696154 shutdown_connections 2024-03-22T01:36:38.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.525+0000 7ff7f8466640 1 --2- 172.21.15.72:0/1465696154 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff7f0100de0 0x7ff7f0101240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:38.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.525+0000 7ff7f8466640 1 --2- 172.21.15.72:0/1465696154 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff7f0101780 0x7ff7f0101c00 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:38.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.525+0000 7ff7f8466640 1 --2- 172.21.15.72:0/1465696154 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7f0100480 0x7ff7f01008a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:38.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.525+0000 7ff7f8466640 1 -- 172.21.15.72:0/1465696154 >> 172.21.15.72:0/1465696154 conn(0x7ff7f00fc020 msgr2=0x7ff7f00fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:38.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.525+0000 7ff7f8466640 1 -- 172.21.15.72:0/1465696154 shutdown_connections 2024-03-22T01:36:38.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.525+0000 7ff7f8466640 1 -- 172.21.15.72:0/1465696154 wait complete. 2024-03-22T01:36:38.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.525+0000 7ff7f8466640 1 Processor -- start 2024-03-22T01:36:38.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.526+0000 7ff7f8466640 1 -- start start 2024-03-22T01:36:38.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.526+0000 7ff7f8466640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff7f0100480 0x7ff7f019a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:38.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.526+0000 7ff7f8466640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7f0100de0 0x7ff7f019a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:38.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.526+0000 7ff7f8466640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff7f0101780 0x7ff7f01a1620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:38.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.526+0000 7ff7f8466640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f010fa00 con 0x7ff7f0100480 2024-03-22T01:36:38.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.526+0000 7ff7f8466640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f010f880 con 0x7ff7f0101780 2024-03-22T01:36:38.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.526+0000 7ff7f8466640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f010fb80 con 0x7ff7f0100de0 2024-03-22T01:36:38.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.526+0000 7ff7f6202640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff7f0100480 0x7ff7f019a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:38.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.526+0000 7ff7f6202640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff7f0100480 0x7ff7f019a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54476/0 (socket says 172.21.15.72:54476) 2024-03-22T01:36:38.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.526+0000 7ff7f6202640 1 -- 172.21.15.72:0/846009310 learned_addr learned my addr 172.21.15.72:0/846009310 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:38.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.526+0000 7ff7f6a03640 1 --2- 172.21.15.72:0/846009310 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff7f0101780 0x7ff7f01a1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:38.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.526+0000 7ff7f5a01640 1 --2- 172.21.15.72:0/846009310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7f0100de0 0x7ff7f019a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:38.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.526+0000 7ff7f6202640 1 -- 172.21.15.72:0/846009310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7f0100de0 msgr2=0x7ff7f019a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:38.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.526+0000 7ff7f6202640 1 --2- 172.21.15.72:0/846009310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7f0100de0 0x7ff7f019a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:38.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.526+0000 7ff7f6202640 1 -- 172.21.15.72:0/846009310 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff7f0101780 msgr2=0x7ff7f01a1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:38.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.526+0000 7ff7f6202640 1 --2- 172.21.15.72:0/846009310 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff7f0101780 0x7ff7f01a1620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:38.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.526+0000 7ff7f6202640 1 -- 172.21.15.72:0/846009310 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7f01a1c90 con 0x7ff7f0100480 2024-03-22T01:36:38.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.526+0000 7ff7f6202640 1 --2- 172.21.15.72:0/846009310 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff7f0100480 0x7ff7f019a070 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec00e9c0 tx=0x7ff7ec00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:38.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.526+0000 7ff7e77fe640 1 -- 172.21.15.72:0/846009310 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7ec00ccc0 con 0x7ff7f0100480 2024-03-22T01:36:38.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.526+0000 7ff7e77fe640 1 -- 172.21.15.72:0/846009310 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7ec005500 con 0x7ff7f0100480 2024-03-22T01:36:38.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.527+0000 7ff7f8466640 1 -- 172.21.15.72:0/846009310 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7f01a1f80 con 0x7ff7f0100480 2024-03-22T01:36:38.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.527+0000 7ff7e77fe640 1 -- 172.21.15.72:0/846009310 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7ec010430 con 0x7ff7f0100480 2024-03-22T01:36:38.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.527+0000 7ff7f8466640 1 -- 172.21.15.72:0/846009310 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7f01a2440 con 0x7ff7f0100480 2024-03-22T01:36:38.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.528+0000 7ff7f8466640 1 -- 172.21.15.72:0/846009310 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7f010b1d0 con 0x7ff7f0100480 2024-03-22T01:36:38.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.530+0000 7ff7e77fe640 1 -- 172.21.15.72:0/846009310 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 20) ==== 137539+0+0 (secure 0 0 0) 0x7ff7ec0046c0 con 0x7ff7f0100480 2024-03-22T01:36:38.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.531+0000 7ff7e77fe640 1 -- 172.21.15.72:0/846009310 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7ff7ec0cf700 con 0x7ff7f0100480 2024-03-22T01:36:38.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.534+0000 7ff7e77fe640 1 -- 172.21.15.72:0/846009310 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff7ec053740 con 0x7ff7f0100480 2024-03-22T01:36:38.622 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:36:38.453+0000 7fa479b40700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-22T01:36:38.622 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:36:38.453+0000 7fa479b40700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-22T01:36:38.623 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:36:38.453+0000 7fa479b40700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-22T01:36:38.623 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:36:38.453+0000 7fa479b40700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-22T01:36:38.623 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:36:38.453+0000 7fa479b40700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-22T01:36:38.623 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:36:38.454+0000 7fa475337700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-22T01:36:38.623 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:36:38.454+0000 7fa475337700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-22T01:36:38.623 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:36:38.454+0000 7fa475337700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-22T01:36:38.623 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:36:38.454+0000 7fa475337700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-22T01:36:38.623 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:36:38.454+0000 7fa475337700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-22T01:36:38.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.724+0000 7ff7f8466640 1 -- 172.21.15.72:0/846009310 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff7f002d3b0 con 0x7ff7f0100480 2024-03-22T01:36:38.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.725+0000 7ff7e77fe640 1 -- 172.21.15.72:0/846009310 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v96) ==== 93+0+10 (secure 0 0 0) 0x7ff7ec09c460 con 0x7ff7f0100480 2024-03-22T01:36:38.724 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-22T01:36:38.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.727+0000 7ff7f8466640 1 -- 172.21.15.72:0/846009310 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff7f0100480 msgr2=0x7ff7f019a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:38.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.727+0000 7ff7f8466640 1 --2- 172.21.15.72:0/846009310 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff7f0100480 0x7ff7f019a070 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec00e9c0 tx=0x7ff7ec00ee80 comp rx=0 tx=0).stop 2024-03-22T01:36:38.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.727+0000 7ff7f8466640 1 -- 172.21.15.72:0/846009310 shutdown_connections 2024-03-22T01:36:38.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.727+0000 7ff7f8466640 1 --2- 172.21.15.72:0/846009310 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff7f0101780 0x7ff7f01a1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:38.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.727+0000 7ff7f8466640 1 --2- 172.21.15.72:0/846009310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7f0100de0 0x7ff7f019a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:38.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.727+0000 7ff7f8466640 1 --2- 172.21.15.72:0/846009310 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff7f0100480 0x7ff7f019a070 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:38.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.727+0000 7ff7f8466640 1 -- 172.21.15.72:0/846009310 >> 172.21.15.72:0/846009310 conn(0x7ff7f00fc020 msgr2=0x7ff7f00fdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:38.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.727+0000 7ff7f8466640 1 -- 172.21.15.72:0/846009310 shutdown_connections 2024-03-22T01:36:38.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.727+0000 7ff7f8466640 1 -- 172.21.15.72:0/846009310 wait complete. 2024-03-22T01:36:38.738 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-22T01:36:38.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.846+0000 7f92a4d9a640 1 -- 172.21.15.72:0/4181471568 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f92a0104f60 msgr2=0x7f92a0107350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:38.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.846+0000 7f92a4d9a640 1 --2- 172.21.15.72:0/4181471568 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f92a0104f60 0x7f92a0107350 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f9294009900 tx=0x7f929402bc30 comp rx=0 tx=0).stop 2024-03-22T01:36:38.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.846+0000 7f92a4d9a640 1 -- 172.21.15.72:0/4181471568 shutdown_connections 2024-03-22T01:36:38.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.846+0000 7f92a4d9a640 1 --2- 172.21.15.72:0/4181471568 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92a0107890 0x7f92a0109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:38.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.846+0000 7f92a4d9a640 1 --2- 172.21.15.72:0/4181471568 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f92a0104f60 0x7f92a0107350 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:38.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.846+0000 7f92a4d9a640 1 --2- 172.21.15.72:0/4181471568 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92a0102630 0x7f92a0104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:38.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.846+0000 7f92a4d9a640 1 -- 172.21.15.72:0/4181471568 >> 172.21.15.72:0/4181471568 conn(0x7f92a00fc020 msgr2=0x7f92a00fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:38.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.846+0000 7f92a4d9a640 1 -- 172.21.15.72:0/4181471568 shutdown_connections 2024-03-22T01:36:38.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.846+0000 7f92a4d9a640 1 -- 172.21.15.72:0/4181471568 wait complete. 2024-03-22T01:36:38.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.847+0000 7f92a4d9a640 1 Processor -- start 2024-03-22T01:36:38.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.847+0000 7f92a4d9a640 1 -- start start 2024-03-22T01:36:38.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.847+0000 7f92a4d9a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92a0102630 0x7f92a0197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:38.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.847+0000 7f92a4d9a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92a0104f60 0x7f92a0198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:38.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.847+0000 7f92a4d9a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f92a0107890 0x7f92a019f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:38.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.847+0000 7f92a4d9a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f92a010d820 con 0x7f92a0107890 2024-03-22T01:36:38.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.847+0000 7f92a4d9a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f92a010d6a0 con 0x7f92a0102630 2024-03-22T01:36:38.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.847+0000 7f92a4d9a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f92a010d9a0 con 0x7f92a0104f60 2024-03-22T01:36:38.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.847+0000 7f929ed9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f92a0107890 0x7f92a019f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:38.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.847+0000 7f929ed9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f92a0107890 0x7f92a019f4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54482/0 (socket says 172.21.15.72:54482) 2024-03-22T01:36:38.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.847+0000 7f929ed9d640 1 -- 172.21.15.72:0/2788133080 learned_addr learned my addr 172.21.15.72:0/2788133080 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:38.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.847+0000 7f929dd9b640 1 --2- 172.21.15.72:0/2788133080 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92a0104f60 0x7f92a0198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:38.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.847+0000 7f929e59c640 1 --2- 172.21.15.72:0/2788133080 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92a0102630 0x7f92a0197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:38.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.848+0000 7f929e59c640 1 -- 172.21.15.72:0/2788133080 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92a0104f60 msgr2=0x7f92a0198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:38.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.848+0000 7f929e59c640 1 --2- 172.21.15.72:0/2788133080 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92a0104f60 0x7f92a0198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:38.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.848+0000 7f929e59c640 1 -- 172.21.15.72:0/2788133080 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f92a0107890 msgr2=0x7f92a019f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:38.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.848+0000 7f929e59c640 1 --2- 172.21.15.72:0/2788133080 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f92a0107890 0x7f92a019f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:38.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.848+0000 7f929e59c640 1 -- 172.21.15.72:0/2788133080 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92a019fb10 con 0x7f92a0102630 2024-03-22T01:36:38.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.848+0000 7f929e59c640 1 --2- 172.21.15.72:0/2788133080 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92a0102630 0x7f92a0197ef0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f929000e9f0 tx=0x7f929000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:38.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.848+0000 7f928f7fe640 1 -- 172.21.15.72:0/2788133080 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f929000ccc0 con 0x7f92a0102630 2024-03-22T01:36:38.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.848+0000 7f928f7fe640 1 -- 172.21.15.72:0/2788133080 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9290010910 con 0x7f92a0102630 2024-03-22T01:36:38.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.848+0000 7f92a4d9a640 1 -- 172.21.15.72:0/2788133080 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92a019fe00 con 0x7f92a0102630 2024-03-22T01:36:38.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.848+0000 7f928f7fe640 1 -- 172.21.15.72:0/2788133080 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9290018960 con 0x7f92a0102630 2024-03-22T01:36:38.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.848+0000 7f92a4d9a640 1 -- 172.21.15.72:0/2788133080 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92a01a0260 con 0x7f92a0102630 2024-03-22T01:36:38.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.849+0000 7f92a4d9a640 1 -- 172.21.15.72:0/2788133080 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92a00fdb10 con 0x7f92a0102630 2024-03-22T01:36:38.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.851+0000 7f928f7fe640 1 -- 172.21.15.72:0/2788133080 <== mon.2 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 137539+0+0 (secure 0 0 0) 0x7f9290010430 con 0x7f92a0102630 2024-03-22T01:36:38.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.851+0000 7f928f7fe640 1 -- 172.21.15.72:0/2788133080 <== mon.2 v2:172.21.15.89:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f92900cf770 con 0x7f92a0102630 2024-03-22T01:36:38.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:38.856+0000 7f928f7fe640 1 -- 172.21.15.72:0/2788133080 <== mon.2 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f929009c4d0 con 0x7f92a0102630 2024-03-22T01:36:39.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.075+0000 7f92a4d9a640 1 -- 172.21.15.72:0/2788133080 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f92a00627c0 con 0x7f92a0102630 2024-03-22T01:36:39.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.076+0000 7f928f7fe640 1 -- 172.21.15.72:0/2788133080 <== mon.2 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v96) ==== 104+0+11 (secure 0 0 0) 0x7f929009bdd0 con 0x7f92a0102630 2024-03-22T01:36:39.075 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-03-22T01:36:39.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.078+0000 7f92a4d9a640 1 -- 172.21.15.72:0/2788133080 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92a0102630 msgr2=0x7f92a0197ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:39.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.078+0000 7f92a4d9a640 1 --2- 172.21.15.72:0/2788133080 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92a0102630 0x7f92a0197ef0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f929000e9f0 tx=0x7f929000eeb0 comp rx=0 tx=0).stop 2024-03-22T01:36:39.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.078+0000 7f92a4d9a640 1 -- 172.21.15.72:0/2788133080 shutdown_connections 2024-03-22T01:36:39.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.078+0000 7f92a4d9a640 1 --2- 172.21.15.72:0/2788133080 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f92a0107890 0x7f92a019f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:39.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.078+0000 7f92a4d9a640 1 --2- 172.21.15.72:0/2788133080 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92a0104f60 0x7f92a0198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:39.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.078+0000 7f92a4d9a640 1 --2- 172.21.15.72:0/2788133080 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92a0102630 0x7f92a0197ef0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:39.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.078+0000 7f92a4d9a640 1 -- 172.21.15.72:0/2788133080 >> 172.21.15.72:0/2788133080 conn(0x7f92a00fc020 msgr2=0x7f92a01030b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:39.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.078+0000 7f92a4d9a640 1 -- 172.21.15.72:0/2788133080 shutdown_connections 2024-03-22T01:36:39.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.078+0000 7f92a4d9a640 1 -- 172.21.15.72:0/2788133080 wait complete. 2024-03-22T01:36:39.088 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-mon[32414]: Standby manager daemon z restarted 2024-03-22T01:36:39.088 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-mon[32414]: Standby manager daemon z started 2024-03-22T01:36:39.088 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-mon[32414]: from='mgr.? 172.21.15.98:0/3149048105' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-22T01:36:39.088 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-mon[32414]: from='mgr.? 172.21.15.98:0/3149048105' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-22T01:36:39.088 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-mon[32414]: from='mgr.? 172.21.15.98:0/3149048105' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-22T01:36:39.088 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-mon[32414]: from='mgr.? 172.21.15.98:0/3149048105' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-22T01:36:39.088 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/805035586' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T01:36:39.088 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-mon[32414]: Active manager daemon x restarted 2024-03-22T01:36:39.088 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-mon[32414]: Activating manager daemon x 2024-03-22T01:36:39.088 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-mon[32414]: osdmap e96: 12 total, 12 up, 12 in 2024-03-22T01:36:39.089 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-mon[32414]: mgrmap e20: x(active, starting, since 0.0135082s), standbys: y, z 2024-03-22T01:36:39.089 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T01:36:39.089 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T01:36:39.089 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T01:36:39.089 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-mon[32414]: Standby manager daemon y restarted 2024-03-22T01:36:39.089 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-mon[32414]: Standby manager daemon y started 2024-03-22T01:36:39.089 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-22T01:36:39.089 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-22T01:36:39.089 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-22T01:36:39.089 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-22T01:36:39.089 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-22T01:36:39.089 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-22T01:36:39.089 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-22T01:36:39.089 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-22T01:36:39.090 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-22T01:36:39.090 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-22T01:36:39.090 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-mon[32414]: from='mgr.? 172.21.15.89:0/2207599102' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-22T01:36:39.090 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-22T01:36:39.090 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-22T01:36:39.090 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-22T01:36:39.090 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-22T01:36:39.090 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-22T01:36:39.090 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T01:36:39.090 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T01:36:39.090 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-mon[32414]: from='mgr.? 172.21.15.89:0/2207599102' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-22T01:36:39.090 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T01:36:39.090 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T01:36:39.090 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T01:36:39.091 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T01:36:39.091 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T01:36:39.091 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T01:36:39.091 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-mon[32414]: from='mgr.? 172.21.15.89:0/2207599102' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-22T01:36:39.091 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-22T01:36:39.091 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-22T01:36:39.092 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-mon[32414]: from='mgr.? 172.21.15.89:0/2207599102' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-22T01:36:39.092 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-22T01:36:39.092 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-22T01:36:39.092 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-22T01:36:39.092 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-22T01:36:39.092 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-22T01:36:39.092 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-mon[32414]: Manager daemon x is now available 2024-03-22T01:36:39.092 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:36:39.092 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-22T01:36:39.092 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-22T01:36:39.092 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/154504233' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:36:39.092 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:38 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/846009310' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T01:36:39.093 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-22T01:36:39.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:38 smithi089 ceph-mon[35770]: Standby manager daemon z restarted 2024-03-22T01:36:39.109 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:38 smithi089 ceph-mon[35770]: Standby manager daemon z started 2024-03-22T01:36:39.109 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:38 smithi089 ceph-mon[35770]: from='mgr.? 172.21.15.98:0/3149048105' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-22T01:36:39.109 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:38 smithi089 ceph-mon[35770]: from='mgr.? 172.21.15.98:0/3149048105' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-22T01:36:39.109 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:38 smithi089 ceph-mon[35770]: from='mgr.? 172.21.15.98:0/3149048105' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-22T01:36:39.109 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:38 smithi089 ceph-mon[35770]: from='mgr.? 172.21.15.98:0/3149048105' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-22T01:36:39.109 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:38 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/805035586' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T01:36:39.110 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:38 smithi089 ceph-mon[35770]: Active manager daemon x restarted 2024-03-22T01:36:39.110 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:38 smithi089 ceph-mon[35770]: Activating manager daemon x 2024-03-22T01:36:39.110 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:38 smithi089 ceph-mon[35770]: osdmap e96: 12 total, 12 up, 12 in 2024-03-22T01:36:39.110 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:38 smithi089 ceph-mon[35770]: mgrmap e20: x(active, starting, since 0.0135082s), standbys: y, z 2024-03-22T01:36:39.110 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:38 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T01:36:39.110 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:38 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T01:36:39.110 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:38 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T01:36:39.110 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:38 smithi089 ceph-mon[35770]: Standby manager daemon y restarted 2024-03-22T01:36:39.110 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:38 smithi089 ceph-mon[35770]: Standby manager daemon y started 2024-03-22T01:36:39.110 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:38 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-22T01:36:39.110 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:38 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-22T01:36:39.110 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:38 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-22T01:36:39.110 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:38 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-22T01:36:39.110 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:38 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-22T01:36:39.111 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:38 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-22T01:36:39.111 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:38 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-22T01:36:39.111 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:38 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-22T01:36:39.111 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:38 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-22T01:36:39.111 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:38 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-22T01:36:39.111 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:38 smithi089 ceph-mon[35770]: from='mgr.? 172.21.15.89:0/2207599102' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-22T01:36:39.111 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:38 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-22T01:36:39.111 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:38 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-22T01:36:39.111 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:38 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-22T01:36:39.111 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:38 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-22T01:36:39.111 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:38 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-22T01:36:39.111 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:38 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T01:36:39.111 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:38 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T01:36:39.111 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:38 smithi089 ceph-mon[35770]: from='mgr.? 172.21.15.89:0/2207599102' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-22T01:36:39.111 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:38 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T01:36:39.112 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:38 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T01:36:39.112 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:38 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T01:36:39.112 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:38 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T01:36:39.112 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:38 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T01:36:39.112 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:38 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T01:36:39.112 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:38 smithi089 ceph-mon[35770]: from='mgr.? 172.21.15.89:0/2207599102' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-22T01:36:39.112 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:38 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-22T01:36:39.112 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:38 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-22T01:36:39.112 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:38 smithi089 ceph-mon[35770]: from='mgr.? 172.21.15.89:0/2207599102' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-22T01:36:39.112 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:38 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-22T01:36:39.112 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:38 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-22T01:36:39.113 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:38 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-22T01:36:39.113 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:38 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-22T01:36:39.113 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:38 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-22T01:36:39.113 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:38 smithi089 ceph-mon[35770]: Manager daemon x is now available 2024-03-22T01:36:39.113 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:38 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:36:39.113 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:38 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-22T01:36:39.113 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:38 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-22T01:36:39.113 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:38 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/154504233' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:36:39.113 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:38 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/846009310' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T01:36:39.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:38 smithi098 ceph-mon[36468]: Standby manager daemon z restarted 2024-03-22T01:36:39.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:38 smithi098 ceph-mon[36468]: Standby manager daemon z started 2024-03-22T01:36:39.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:38 smithi098 ceph-mon[36468]: from='mgr.? 172.21.15.98:0/3149048105' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-22T01:36:39.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:38 smithi098 ceph-mon[36468]: from='mgr.? 172.21.15.98:0/3149048105' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-22T01:36:39.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:38 smithi098 ceph-mon[36468]: from='mgr.? 172.21.15.98:0/3149048105' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-22T01:36:39.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:38 smithi098 ceph-mon[36468]: from='mgr.? 172.21.15.98:0/3149048105' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-22T01:36:39.173 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:38 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/805035586' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T01:36:39.173 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:38 smithi098 ceph-mon[36468]: Active manager daemon x restarted 2024-03-22T01:36:39.173 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:38 smithi098 ceph-mon[36468]: Activating manager daemon x 2024-03-22T01:36:39.173 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:38 smithi098 ceph-mon[36468]: osdmap e96: 12 total, 12 up, 12 in 2024-03-22T01:36:39.173 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:38 smithi098 ceph-mon[36468]: mgrmap e20: x(active, starting, since 0.0135082s), standbys: y, z 2024-03-22T01:36:39.173 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:38 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T01:36:39.173 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:38 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T01:36:39.173 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:38 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T01:36:39.173 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:38 smithi098 ceph-mon[36468]: Standby manager daemon y restarted 2024-03-22T01:36:39.173 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:38 smithi098 ceph-mon[36468]: Standby manager daemon y started 2024-03-22T01:36:39.173 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:38 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-22T01:36:39.173 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:38 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-22T01:36:39.173 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:38 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-22T01:36:39.173 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:38 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-22T01:36:39.173 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:38 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-22T01:36:39.174 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:38 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-22T01:36:39.174 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:38 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-22T01:36:39.174 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:38 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-22T01:36:39.174 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:38 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-22T01:36:39.174 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:38 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-22T01:36:39.174 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:38 smithi098 ceph-mon[36468]: from='mgr.? 172.21.15.89:0/2207599102' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-22T01:36:39.174 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:38 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-22T01:36:39.174 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:38 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-22T01:36:39.174 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:38 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-22T01:36:39.174 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:38 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-22T01:36:39.175 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:38 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-22T01:36:39.175 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:38 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T01:36:39.175 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:38 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T01:36:39.175 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:38 smithi098 ceph-mon[36468]: from='mgr.? 172.21.15.89:0/2207599102' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-22T01:36:39.175 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:38 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T01:36:39.175 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:38 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T01:36:39.175 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:38 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T01:36:39.175 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:38 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T01:36:39.175 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:38 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T01:36:39.175 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:38 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T01:36:39.175 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:38 smithi098 ceph-mon[36468]: from='mgr.? 172.21.15.89:0/2207599102' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-22T01:36:39.175 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:38 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-22T01:36:39.175 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:38 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-22T01:36:39.176 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:38 smithi098 ceph-mon[36468]: from='mgr.? 172.21.15.89:0/2207599102' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-22T01:36:39.176 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:38 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-22T01:36:39.176 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:38 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-22T01:36:39.176 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:38 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-22T01:36:39.176 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:38 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-22T01:36:39.176 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:38 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-22T01:36:39.176 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:38 smithi098 ceph-mon[36468]: Manager daemon x is now available 2024-03-22T01:36:39.176 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:38 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:36:39.176 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:38 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-22T01:36:39.176 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:38 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-22T01:36:39.176 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:38 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/154504233' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:36:39.176 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:38 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/846009310' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T01:36:39.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.216+0000 7f3ebeb19640 1 -- 172.21.15.72:0/3224496149 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3eb8101cc0 msgr2=0x7f3eb8102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:39.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.216+0000 7f3ebeb19640 1 --2- 172.21.15.72:0/3224496149 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3eb8101cc0 0x7f3eb8102140 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f3ea4009900 tx=0x7f3ea402bc50 comp rx=0 tx=0).stop 2024-03-22T01:36:39.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.216+0000 7f3ebeb19640 1 -- 172.21.15.72:0/3224496149 shutdown_connections 2024-03-22T01:36:39.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.216+0000 7f3ebeb19640 1 --2- 172.21.15.72:0/3224496149 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3eb8102680 0x7f3eb8108f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:39.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.216+0000 7f3ebeb19640 1 --2- 172.21.15.72:0/3224496149 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3eb8101cc0 0x7f3eb8102140 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:39.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.216+0000 7f3ebeb19640 1 --2- 172.21.15.72:0/3224496149 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3eb8100ad0 0x7f3eb8100ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:39.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.216+0000 7f3ebeb19640 1 -- 172.21.15.72:0/3224496149 >> 172.21.15.72:0/3224496149 conn(0x7f3eb80fc060 msgr2=0x7f3eb80fe480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:39.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.216+0000 7f3ebeb19640 1 -- 172.21.15.72:0/3224496149 shutdown_connections 2024-03-22T01:36:39.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.216+0000 7f3ebeb19640 1 -- 172.21.15.72:0/3224496149 wait complete. 2024-03-22T01:36:39.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.217+0000 7f3ebeb19640 1 Processor -- start 2024-03-22T01:36:39.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.217+0000 7f3ebeb19640 1 -- start start 2024-03-22T01:36:39.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.217+0000 7f3ebeb19640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3eb8100ad0 0x7f3eb806a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:39.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.217+0000 7f3ebeb19640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3eb8101cc0 0x7f3eb819cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:39.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.217+0000 7f3ebeb19640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3eb8102680 0x7f3eb819f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:39.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.217+0000 7f3ebeb19640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb810c730 con 0x7f3eb8101cc0 2024-03-22T01:36:39.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.217+0000 7f3ebeb19640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb810c5b0 con 0x7f3eb8100ad0 2024-03-22T01:36:39.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.217+0000 7f3ebeb19640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb810c8b0 con 0x7f3eb8102680 2024-03-22T01:36:39.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.217+0000 7f3eb7fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3eb8101cc0 0x7f3eb819cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:39.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.217+0000 7f3eb7fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3eb8101cc0 0x7f3eb819cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54496/0 (socket says 172.21.15.72:54496) 2024-03-22T01:36:39.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.217+0000 7f3eb7fff640 1 -- 172.21.15.72:0/2785936325 learned_addr learned my addr 172.21.15.72:0/2785936325 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:39.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.217+0000 7f3ebc8b5640 1 --2- 172.21.15.72:0/2785936325 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3eb8100ad0 0x7f3eb806a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:39.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.217+0000 7f3ebd0b6640 1 --2- 172.21.15.72:0/2785936325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3eb8102680 0x7f3eb819f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:39.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.217+0000 7f3eb7fff640 1 -- 172.21.15.72:0/2785936325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3eb8102680 msgr2=0x7f3eb819f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:39.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.217+0000 7f3eb7fff640 1 --2- 172.21.15.72:0/2785936325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3eb8102680 0x7f3eb819f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:39.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.217+0000 7f3eb7fff640 1 -- 172.21.15.72:0/2785936325 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3eb8100ad0 msgr2=0x7f3eb806a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:39.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.217+0000 7f3eb7fff640 1 --2- 172.21.15.72:0/2785936325 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3eb8100ad0 0x7f3eb806a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:39.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.217+0000 7f3eb7fff640 1 -- 172.21.15.72:0/2785936325 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3eb819fba0 con 0x7f3eb8101cc0 2024-03-22T01:36:39.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.218+0000 7f3eb7fff640 1 --2- 172.21.15.72:0/2785936325 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3eb8101cc0 0x7f3eb819cc70 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f3ea4009230 tx=0x7f3ea4005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:39.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.218+0000 7f3eb5ffb640 1 -- 172.21.15.72:0/2785936325 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea4042020 con 0x7f3eb8101cc0 2024-03-22T01:36:39.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.218+0000 7f3ebeb19640 1 -- 172.21.15.72:0/2785936325 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3eb819fdd0 con 0x7f3eb8101cc0 2024-03-22T01:36:39.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.218+0000 7f3eb5ffb640 1 -- 172.21.15.72:0/2785936325 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ea40054e0 con 0x7f3eb8101cc0 2024-03-22T01:36:39.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.218+0000 7f3eb5ffb640 1 -- 172.21.15.72:0/2785936325 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea403e460 con 0x7f3eb8101cc0 2024-03-22T01:36:39.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.218+0000 7f3ebeb19640 1 -- 172.21.15.72:0/2785936325 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3eb81a0290 con 0x7f3eb8101cc0 2024-03-22T01:36:39.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.219+0000 7f3ebeb19640 1 -- 172.21.15.72:0/2785936325 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3eb8107df0 con 0x7f3eb8101cc0 2024-03-22T01:36:39.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.220+0000 7f3eb5ffb640 1 -- 172.21.15.72:0/2785936325 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 21) ==== 137666+0+0 (secure 0 0 0) 0x7f3ea4045020 con 0x7f3eb8101cc0 2024-03-22T01:36:39.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.221+0000 7f3eb5ffb640 1 --2- 172.21.15.72:0/2785936325 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f3ea80a2d20 0x7f3ea80a51e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:39.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.221+0000 7f3ebc8b5640 1 --2- 172.21.15.72:0/2785936325 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f3ea80a2d20 0x7f3ea80a51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:39.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.221+0000 7f3eb5ffb640 1 -- 172.21.15.72:0/2785936325 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f3ea40f1b80 con 0x7f3eb8101cc0 2024-03-22T01:36:39.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.221+0000 7f3ebc8b5640 1 --2- 172.21.15.72:0/2785936325 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f3ea80a2d20 0x7f3ea80a51e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3eb8101b20 tx=0x7f3eac00b390 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:39.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.226+0000 7f3eb5ffb640 1 -- 172.21.15.72:0/2785936325 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3ea404c020 con 0x7f3eb8101cc0 2024-03-22T01:36:39.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.412+0000 7f3ebeb19640 1 -- 172.21.15.72:0/2785936325 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f3eb802d540 con 0x7f3eb8101cc0 2024-03-22T01:36:39.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.412+0000 7f3eb5ffb640 1 -- 172.21.15.72:0/2785936325 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v96) ==== 100+0+11 (secure 0 0 0) 0x7f3ea40be8f0 con 0x7f3eb8101cc0 2024-03-22T01:36:39.412 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-03-22T01:36:39.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.415+0000 7f3ebeb19640 1 -- 172.21.15.72:0/2785936325 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f3ea80a2d20 msgr2=0x7f3ea80a51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:39.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.415+0000 7f3ebeb19640 1 --2- 172.21.15.72:0/2785936325 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f3ea80a2d20 0x7f3ea80a51e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3eb8101b20 tx=0x7f3eac00b390 comp rx=0 tx=0).stop 2024-03-22T01:36:39.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.415+0000 7f3ebeb19640 1 -- 172.21.15.72:0/2785936325 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3eb8101cc0 msgr2=0x7f3eb819cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:39.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.415+0000 7f3ebeb19640 1 --2- 172.21.15.72:0/2785936325 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3eb8101cc0 0x7f3eb819cc70 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f3ea4009230 tx=0x7f3ea4005dd0 comp rx=0 tx=0).stop 2024-03-22T01:36:39.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.415+0000 7f3ebeb19640 1 -- 172.21.15.72:0/2785936325 shutdown_connections 2024-03-22T01:36:39.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.415+0000 7f3ebeb19640 1 --2- 172.21.15.72:0/2785936325 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f3ea80a2d20 0x7f3ea80a51e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:39.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.415+0000 7f3ebeb19640 1 --2- 172.21.15.72:0/2785936325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3eb8102680 0x7f3eb819f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:39.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.415+0000 7f3ebeb19640 1 --2- 172.21.15.72:0/2785936325 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3eb8101cc0 0x7f3eb819cc70 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:39.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.415+0000 7f3ebeb19640 1 --2- 172.21.15.72:0/2785936325 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3eb8100ad0 0x7f3eb806a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:39.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.415+0000 7f3ebeb19640 1 -- 172.21.15.72:0/2785936325 >> 172.21.15.72:0/2785936325 conn(0x7f3eb80fc060 msgr2=0x7f3eb8104fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:39.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.415+0000 7f3ebeb19640 1 -- 172.21.15.72:0/2785936325 shutdown_connections 2024-03-22T01:36:39.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.415+0000 7f3ebeb19640 1 -- 172.21.15.72:0/2785936325 wait complete. 2024-03-22T01:36:39.425 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-22T01:36:39.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.537+0000 7f9d6ba55640 1 -- 172.21.15.72:0/923869858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d64101ca0 msgr2=0x7f9d64102120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:39.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.537+0000 7f9d6ba55640 1 --2- 172.21.15.72:0/923869858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d64101ca0 0x7f9d64102120 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9d54009900 tx=0x7f9d5402bc30 comp rx=0 tx=0).stop 2024-03-22T01:36:39.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.537+0000 7f9d6ba55640 1 -- 172.21.15.72:0/923869858 shutdown_connections 2024-03-22T01:36:39.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.537+0000 7f9d6ba55640 1 --2- 172.21.15.72:0/923869858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d64101ca0 0x7f9d64102120 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:39.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.537+0000 7f9d6ba55640 1 --2- 172.21.15.72:0/923869858 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d64102660 0x7f9d64108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:39.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.537+0000 7f9d6ba55640 1 --2- 172.21.15.72:0/923869858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d64100ab0 0x7f9d64100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:39.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.537+0000 7f9d6ba55640 1 -- 172.21.15.72:0/923869858 >> 172.21.15.72:0/923869858 conn(0x7f9d640fc020 msgr2=0x7f9d640fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:39.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.537+0000 7f9d6ba55640 1 -- 172.21.15.72:0/923869858 shutdown_connections 2024-03-22T01:36:39.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.537+0000 7f9d6ba55640 1 -- 172.21.15.72:0/923869858 wait complete. 2024-03-22T01:36:39.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.538+0000 7f9d6ba55640 1 Processor -- start 2024-03-22T01:36:39.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.538+0000 7f9d6ba55640 1 -- start start 2024-03-22T01:36:39.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.538+0000 7f9d6ba55640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d64100ab0 0x7f9d64197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:39.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.538+0000 7f9d6ba55640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d64101ca0 0x7f9d64198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:39.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.538+0000 7f9d6ba55640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d64102660 0x7f9d6419f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:39.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.538+0000 7f9d6ba55640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f9d6410c730 con 0x7f9d64101ca0 2024-03-22T01:36:39.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.538+0000 7f9d6ba55640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9d6410c5b0 con 0x7f9d64100ab0 2024-03-22T01:36:39.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.538+0000 7f9d6ba55640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9d6410c8b0 con 0x7f9d64102660 2024-03-22T01:36:39.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.538+0000 7f9d697f1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d64100ab0 0x7f9d64197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:39.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.538+0000 7f9d68ff0640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d64101ca0 0x7f9d64198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:39.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.538+0000 7f9d697f1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d64100ab0 0x7f9d64197f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.72:50218/0 (socket says 172.21.15.72:50218) 2024-03-22T01:36:39.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.538+0000 7f9d697f1640 1 -- 172.21.15.72:0/2500350388 learned_addr learned my addr 172.21.15.72:0/2500350388 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:39.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.538+0000 7f9d69ff2640 1 --2- 172.21.15.72:0/2500350388 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d64102660 0x7f9d6419f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:39.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.538+0000 7f9d68ff0640 1 -- 172.21.15.72:0/2500350388 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d64102660 msgr2=0x7f9d6419f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:39.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.538+0000 7f9d68ff0640 1 --2- 172.21.15.72:0/2500350388 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d64102660 0x7f9d6419f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:39.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.538+0000 7f9d68ff0640 1 -- 172.21.15.72:0/2500350388 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d64100ab0 msgr2=0x7f9d64197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:39.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.538+0000 7f9d68ff0640 1 --2- 172.21.15.72:0/2500350388 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d64100ab0 0x7f9d64197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:39.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.538+0000 7f9d68ff0640 1 -- 172.21.15.72:0/2500350388 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d6419fb50 con 0x7f9d64101ca0 2024-03-22T01:36:39.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.538+0000 7f9d697f1640 1 --2- 172.21.15.72:0/2500350388 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d64100ab0 0x7f9d64197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:36:39.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.538+0000 7f9d68ff0640 1 --2- 172.21.15.72:0/2500350388 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d64101ca0 0x7f9d64198470 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f9d54009230 tx=0x7f9d5402c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:39.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.539+0000 7f9d5a7fc640 1 -- 172.21.15.72:0/2500350388 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d54041070 con 0x7f9d64101ca0 2024-03-22T01:36:39.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.539+0000 7f9d6ba55640 1 -- 172.21.15.72:0/2500350388 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d6419fde0 con 0x7f9d64101ca0 2024-03-22T01:36:39.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.539+0000 7f9d5a7fc640 1 -- 172.21.15.72:0/2500350388 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d54004450 con 0x7f9d64101ca0 2024-03-22T01:36:39.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.539+0000 7f9d5a7fc640 1 -- 172.21.15.72:0/2500350388 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d54003df0 con 0x7f9d64101ca0 2024-03-22T01:36:39.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.539+0000 7f9d6ba55640 1 -- 172.21.15.72:0/2500350388 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d641a0250 con 0x7f9d64101ca0 2024-03-22T01:36:39.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.540+0000 7f9d6ba55640 1 -- 172.21.15.72:0/2500350388 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d641a0540 con 0x7f9d64101ca0 2024-03-22T01:36:39.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.541+0000 7f9d5a7fc640 1 -- 172.21.15.72:0/2500350388 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 21) ==== 137666+0+0 (secure 0 0 0) 0x7f9d54044070 con 0x7f9d64101ca0 2024-03-22T01:36:39.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.541+0000 7f9d5a7fc640 1 --2- 172.21.15.72:0/2500350388 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f9d500a2d20 0x7f9d500a51e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:39.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.542+0000 7f9d697f1640 1 --2- 172.21.15.72:0/2500350388 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f9d500a2d20 0x7f9d500a51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:39.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.542+0000 7f9d5a7fc640 1 -- 172.21.15.72:0/2500350388 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f9d540f0e00 con 0x7f9d64101ca0 2024-03-22T01:36:39.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.542+0000 7f9d697f1640 1 --2- 172.21.15.72:0/2500350388 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f9d500a2d20 0x7f9d500a51e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c005f40 tx=0x7f9d5c009500 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:39.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.545+0000 7f9d5a7fc640 1 -- 172.21.15.72:0/2500350388 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9d540bd400 con 0x7f9d64101ca0 2024-03-22T01:36:39.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.727+0000 7f9d6ba55640 1 -- 172.21.15.72:0/2500350388 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f9d641a0830 con 0x7f9d64101ca0 2024-03-22T01:36:39.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.727+0000 7f9d5a7fc640 1 -- 172.21.15.72:0/2500350388 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v96) ==== 103+0+11 (secure 0 0 0) 0x7f9d540c14a0 con 0x7f9d64101ca0 2024-03-22T01:36:39.727 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-03-22T01:36:39.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.730+0000 7f9d6ba55640 1 -- 172.21.15.72:0/2500350388 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f9d500a2d20 msgr2=0x7f9d500a51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:39.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.730+0000 7f9d6ba55640 1 --2- 172.21.15.72:0/2500350388 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f9d500a2d20 0x7f9d500a51e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c005f40 tx=0x7f9d5c009500 comp rx=0 tx=0).stop 2024-03-22T01:36:39.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.730+0000 7f9d6ba55640 1 -- 172.21.15.72:0/2500350388 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d64101ca0 msgr2=0x7f9d64198470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:39.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.730+0000 7f9d6ba55640 1 --2- 172.21.15.72:0/2500350388 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d64101ca0 0x7f9d64198470 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f9d54009230 tx=0x7f9d5402c650 comp rx=0 tx=0).stop 2024-03-22T01:36:39.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.730+0000 7f9d6ba55640 1 -- 172.21.15.72:0/2500350388 shutdown_connections 2024-03-22T01:36:39.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.730+0000 7f9d6ba55640 1 --2- 172.21.15.72:0/2500350388 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f9d500a2d20 0x7f9d500a51e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:39.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.730+0000 7f9d6ba55640 1 --2- 172.21.15.72:0/2500350388 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d64102660 0x7f9d6419f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:39.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.730+0000 7f9d6ba55640 1 --2- 172.21.15.72:0/2500350388 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d64101ca0 0x7f9d64198470 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:39.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.730+0000 7f9d6ba55640 1 --2- 172.21.15.72:0/2500350388 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d64100ab0 0x7f9d64197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:39.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.730+0000 7f9d6ba55640 1 -- 172.21.15.72:0/2500350388 >> 172.21.15.72:0/2500350388 conn(0x7f9d640fc020 msgr2=0x7f9d640fdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:39.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.730+0000 7f9d6ba55640 1 -- 172.21.15.72:0/2500350388 shutdown_connections 2024-03-22T01:36:39.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.730+0000 7f9d6ba55640 1 -- 172.21.15.72:0/2500350388 wait complete. 2024-03-22T01:36:39.742 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-03-22T01:36:39.743 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T01:36:39.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.848+0000 7f9ca2b02640 1 -- 172.21.15.72:0/692172034 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9c9c100b10 msgr2=0x7f9c9c100f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:39.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.848+0000 7f9ca2b02640 1 --2- 172.21.15.72:0/692172034 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9c9c100b10 0x7f9c9c100f30 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f9c98009900 tx=0x7f9c9802bc30 comp rx=0 tx=0).stop 2024-03-22T01:36:39.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.848+0000 7f9ca2b02640 1 -- 172.21.15.72:0/692172034 shutdown_connections 2024-03-22T01:36:39.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.848+0000 7f9ca2b02640 1 --2- 172.21.15.72:0/692172034 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c9c1026c0 0x7f9c9c108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:39.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.848+0000 7f9ca2b02640 1 --2- 172.21.15.72:0/692172034 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c9c101d00 0x7f9c9c102180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:39.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.848+0000 7f9ca2b02640 1 --2- 172.21.15.72:0/692172034 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9c9c100b10 0x7f9c9c100f30 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:39.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.848+0000 7f9ca2b02640 1 -- 172.21.15.72:0/692172034 >> 172.21.15.72:0/692172034 conn(0x7f9c9c0fc0a0 msgr2=0x7f9c9c0fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:39.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.848+0000 7f9ca2b02640 1 -- 172.21.15.72:0/692172034 shutdown_connections 2024-03-22T01:36:39.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.848+0000 7f9ca2b02640 1 -- 172.21.15.72:0/692172034 wait complete. 2024-03-22T01:36:39.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.848+0000 7f9ca2b02640 1 Processor -- start 2024-03-22T01:36:39.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.848+0000 7f9ca2b02640 1 -- start start 2024-03-22T01:36:39.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.849+0000 7f9ca2b02640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9c9c100b10 0x7f9c9c197e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:39.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.849+0000 7f9ca2b02640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c9c101d00 0x7f9c9c1983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:39.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.849+0000 7f9ca2b02640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c9c1026c0 0x7f9c9c19f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:39.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.849+0000 7f9ca2b02640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f9c9c10d840 con 0x7f9c9c100b10 2024-03-22T01:36:39.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.849+0000 7f9ca2b02640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9c9c10d6c0 con 0x7f9c9c1026c0 2024-03-22T01:36:39.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.849+0000 7f9ca2b02640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9c9c10d9c0 con 0x7f9c9c101d00 2024-03-22T01:36:39.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.849+0000 7f9ca089e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9c9c100b10 0x7f9c9c197e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:39.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.849+0000 7f9ca089e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9c9c100b10 0x7f9c9c197e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54530/0 (socket says 172.21.15.72:54530) 2024-03-22T01:36:39.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.849+0000 7f9ca089e640 1 -- 172.21.15.72:0/2728439601 learned_addr learned my addr 172.21.15.72:0/2728439601 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:39.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.849+0000 7f9ca109f640 1 --2- 172.21.15.72:0/2728439601 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c9c1026c0 0x7f9c9c19f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:39.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.849+0000 7f9c93fff640 1 --2- 172.21.15.72:0/2728439601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c9c101d00 0x7f9c9c1983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:39.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.849+0000 7f9ca089e640 1 -- 172.21.15.72:0/2728439601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c9c101d00 msgr2=0x7f9c9c1983a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:39.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.849+0000 7f9ca089e640 1 --2- 172.21.15.72:0/2728439601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c9c101d00 0x7f9c9c1983a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:39.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.849+0000 7f9ca089e640 1 -- 172.21.15.72:0/2728439601 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c9c1026c0 msgr2=0x7f9c9c19f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:39.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.849+0000 7f9ca089e640 1 --2- 172.21.15.72:0/2728439601 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c9c1026c0 0x7f9c9c19f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:39.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.849+0000 7f9ca089e640 1 -- 172.21.15.72:0/2728439601 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c9c19fa80 con 0x7f9c9c100b10 2024-03-22T01:36:39.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.849+0000 7f9ca089e640 1 --2- 172.21.15.72:0/2728439601 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9c9c100b10 0x7f9c9c197e60 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f9c98000c00 tx=0x7f9c98005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:39.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.849+0000 7f9c91ffb640 1 -- 172.21.15.72:0/2728439601 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c98041070 con 0x7f9c9c100b10 2024-03-22T01:36:39.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.849+0000 7f9c91ffb640 1 -- 172.21.15.72:0/2728439601 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c980044d0 con 0x7f9c9c100b10 2024-03-22T01:36:39.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.849+0000 7f9c91ffb640 1 -- 172.21.15.72:0/2728439601 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c98034440 con 0x7f9c9c100b10 2024-03-22T01:36:39.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.849+0000 7f9ca2b02640 1 -- 172.21.15.72:0/2728439601 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c9c19fd10 con 0x7f9c9c100b10 2024-03-22T01:36:39.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.850+0000 7f9ca2b02640 1 -- 172.21.15.72:0/2728439601 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c9c1a01b0 con 0x7f9c9c100b10 2024-03-22T01:36:39.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.850+0000 7f9ca2b02640 1 -- 172.21.15.72:0/2728439601 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c9c107dd0 con 0x7f9c9c100b10 2024-03-22T01:36:39.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.852+0000 7f9c91ffb640 1 -- 172.21.15.72:0/2728439601 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 21) ==== 137666+0+0 (secure 0 0 0) 0x7f9c98044070 con 0x7f9c9c100b10 2024-03-22T01:36:39.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.852+0000 7f9c91ffb640 1 --2- 172.21.15.72:0/2728439601 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f9c840a2d20 0x7f9c840a51e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:39.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.852+0000 7f9c93fff640 1 --2- 172.21.15.72:0/2728439601 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f9c840a2d20 0x7f9c840a51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:39.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.852+0000 7f9c91ffb640 1 -- 172.21.15.72:0/2728439601 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f9c980f10f0 con 0x7f9c9c100b10 2024-03-22T01:36:39.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.853+0000 7f9c93fff640 1 --2- 172.21.15.72:0/2728439601 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f9c840a2d20 0x7f9c840a51e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9c8c005e10 tx=0x7f9c8c005da0 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:39.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:39.857+0000 7f9c91ffb640 1 -- 172.21.15.72:0/2728439601 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9c980bdd40 con 0x7f9c9c100b10 2024-03-22T01:36:40.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.084+0000 7f9ca2b02640 1 -- 172.21.15.72:0/2728439601 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9c9c062830 con 0x7f9c9c100b10 2024-03-22T01:36:40.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.085+0000 7f9c91ffb640 1 -- 172.21.15.72:0/2728439601 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11785 (secure 0 0 0) 0x7f9c980bd640 con 0x7f9c9c100b10 2024-03-22T01:36:40.084 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:36:40.084 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6835/3077216308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3077216308},{"type":"v1","addr":"172.21.15.72:6835","nonce":3077216308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2460254143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":2460254143},{"type":"v1","addr":"172.21.15.72:6837","nonce":2460254143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6839/2026432819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6838","nonce":2026432819},{"type":"v1","addr":"172.21.15.72:6839","nonce":2026432819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6841/400094545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6840","nonce":400094545},{"type":"v1","addr":"172.21.15.72:6841","nonce":400094545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6833/1578701171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":1578701171},{"type":"v1","addr":"172.21.15.89:6833","nonce":1578701171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/2703047213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2703047213},{"type":"v1","addr":"172.21.15.98:6833","nonce":2703047213}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2589468787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2589468787},{"type":"v1","addr":"172.21.15.89:6835","nonce":2589468787}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/92657636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":92657636},{"type":"v1","addr":"172.21.15.98:6835","nonce":92657636}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1662941782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1662941782},{"type":"v1","addr":"172.21.15.89:6837","nonce":1662941782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24490,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/781860371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":781860371},{"type":"v1","addr":"172.21.15.98:6837","nonce":781860371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-22T01:34:14.815677+0000","modified":"2024-03-22T01:34:39.842959+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24524},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3192173650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3192173650},{"type":"v1","addr":"172.21.15.98:6839","nonce":3192173650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6839/149200280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":149200280},{"type":"v1","addr":"172.21.15.89:6839","nonce":149200280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24524]},"id":1}]} 2024-03-22T01:36:40.084 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 21 2024-03-22T01:36:40.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.087+0000 7f9ca2b02640 1 -- 172.21.15.72:0/2728439601 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f9c840a2d20 msgr2=0x7f9c840a51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:40.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.087+0000 7f9ca2b02640 1 --2- 172.21.15.72:0/2728439601 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f9c840a2d20 0x7f9c840a51e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9c8c005e10 tx=0x7f9c8c005da0 comp rx=0 tx=0).stop 2024-03-22T01:36:40.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.087+0000 7f9ca2b02640 1 -- 172.21.15.72:0/2728439601 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9c9c100b10 msgr2=0x7f9c9c197e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:40.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.087+0000 7f9ca2b02640 1 --2- 172.21.15.72:0/2728439601 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9c9c100b10 0x7f9c9c197e60 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f9c98000c00 tx=0x7f9c98005d50 comp rx=0 tx=0).stop 2024-03-22T01:36:40.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.087+0000 7f9ca2b02640 1 -- 172.21.15.72:0/2728439601 shutdown_connections 2024-03-22T01:36:40.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.087+0000 7f9ca2b02640 1 --2- 172.21.15.72:0/2728439601 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f9c840a2d20 0x7f9c840a51e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:40.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.087+0000 7f9ca2b02640 1 --2- 172.21.15.72:0/2728439601 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c9c1026c0 0x7f9c9c19f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:40.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.087+0000 7f9ca2b02640 1 --2- 172.21.15.72:0/2728439601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c9c101d00 0x7f9c9c1983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:40.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.087+0000 7f9ca2b02640 1 --2- 172.21.15.72:0/2728439601 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9c9c100b10 0x7f9c9c197e60 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:40.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.087+0000 7f9ca2b02640 1 -- 172.21.15.72:0/2728439601 >> 172.21.15.72:0/2728439601 conn(0x7f9c9c0fc0a0 msgr2=0x7f9c9c104fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:40.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.087+0000 7f9ca2b02640 1 -- 172.21.15.72:0/2728439601 shutdown_connections 2024-03-22T01:36:40.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.087+0000 7f9ca2b02640 1 -- 172.21.15.72:0/2728439601 wait complete. 2024-03-22T01:36:40.097 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:39 smithi072 ceph-mon[32414]: [22/Mar/2024:01:36:38] ENGINE Bus STARTING 2024-03-22T01:36:40.097 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:39 smithi072 ceph-mon[32414]: [22/Mar/2024:01:36:38] ENGINE Serving on http://172.21.15.72:8765 2024-03-22T01:36:40.097 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:39 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/2788133080' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T01:36:40.097 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:39 smithi072 ceph-mon[32414]: [22/Mar/2024:01:36:39] ENGINE Serving on https://172.21.15.72:7150 2024-03-22T01:36:40.097 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:39 smithi072 ceph-mon[32414]: [22/Mar/2024:01:36:39] ENGINE Bus STARTED 2024-03-22T01:36:40.097 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:39 smithi072 ceph-mon[32414]: mgrmap e21: x(active, since 1.01866s), standbys: y, z 2024-03-22T01:36:40.097 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:39 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/2785936325' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T01:36:40.097 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:39 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/2500350388' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T01:36:40.098 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-03-22T01:36:40.099 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-03-22T01:36:40.099 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-22T01:36:40.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:39 smithi089 ceph-mon[35770]: [22/Mar/2024:01:36:38] ENGINE Bus STARTING 2024-03-22T01:36:40.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:39 smithi089 ceph-mon[35770]: [22/Mar/2024:01:36:38] ENGINE Serving on http://172.21.15.72:8765 2024-03-22T01:36:40.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:39 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/2788133080' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T01:36:40.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:39 smithi089 ceph-mon[35770]: [22/Mar/2024:01:36:39] ENGINE Serving on https://172.21.15.72:7150 2024-03-22T01:36:40.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:39 smithi089 ceph-mon[35770]: [22/Mar/2024:01:36:39] ENGINE Bus STARTED 2024-03-22T01:36:40.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:39 smithi089 ceph-mon[35770]: mgrmap e21: x(active, since 1.01866s), standbys: y, z 2024-03-22T01:36:40.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:39 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/2785936325' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T01:36:40.109 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:39 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/2500350388' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T01:36:40.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:39 smithi098 ceph-mon[36468]: [22/Mar/2024:01:36:38] ENGINE Bus STARTING 2024-03-22T01:36:40.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:39 smithi098 ceph-mon[36468]: [22/Mar/2024:01:36:38] ENGINE Serving on http://172.21.15.72:8765 2024-03-22T01:36:40.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:39 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/2788133080' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T01:36:40.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:39 smithi098 ceph-mon[36468]: [22/Mar/2024:01:36:39] ENGINE Serving on https://172.21.15.72:7150 2024-03-22T01:36:40.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:39 smithi098 ceph-mon[36468]: [22/Mar/2024:01:36:39] ENGINE Bus STARTED 2024-03-22T01:36:40.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:39 smithi098 ceph-mon[36468]: mgrmap e21: x(active, since 1.01866s), standbys: y, z 2024-03-22T01:36:40.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:39 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/2785936325' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T01:36:40.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:39 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/2500350388' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T01:36:40.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.210+0000 7f9bf566d640 1 -- 172.21.15.72:0/3625890081 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9bf0100ac0 msgr2=0x7f9bf0100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:40.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.210+0000 7f9bf566d640 1 --2- 172.21.15.72:0/3625890081 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9bf0100ac0 0x7f9bf0100ee0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f9be0009900 tx=0x7f9be002bc30 comp rx=0 tx=0).stop 2024-03-22T01:36:40.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.210+0000 7f9bf566d640 1 -- 172.21.15.72:0/3625890081 shutdown_connections 2024-03-22T01:36:40.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.210+0000 7f9bf566d640 1 --2- 172.21.15.72:0/3625890081 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9bf0101cb0 0x7f9bf0102130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:40.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.210+0000 7f9bf566d640 1 --2- 172.21.15.72:0/3625890081 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9bf0102670 0x7f9bf0108f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:40.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.210+0000 7f9bf566d640 1 --2- 172.21.15.72:0/3625890081 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9bf0100ac0 0x7f9bf0100ee0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:40.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.210+0000 7f9bf566d640 1 -- 172.21.15.72:0/3625890081 >> 172.21.15.72:0/3625890081 conn(0x7f9bf00fc030 msgr2=0x7f9bf00fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:40.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.210+0000 7f9bf566d640 1 -- 172.21.15.72:0/3625890081 shutdown_connections 2024-03-22T01:36:40.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.210+0000 7f9bf566d640 1 -- 172.21.15.72:0/3625890081 wait complete. 2024-03-22T01:36:40.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.211+0000 7f9bf566d640 1 Processor -- start 2024-03-22T01:36:40.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.211+0000 7f9bf566d640 1 -- start start 2024-03-22T01:36:40.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.211+0000 7f9bf566d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9bf0100ac0 0x7f9bf0197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:40.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.211+0000 7f9bf566d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9bf0101cb0 0x7f9bf0198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:40.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.211+0000 7f9bf566d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9bf0102670 0x7f9bf019f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:40.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.211+0000 7f9bf566d640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f9bf010c790 con 0x7f9bf0100ac0 2024-03-22T01:36:40.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.211+0000 7f9bf566d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9bf010c610 con 0x7f9bf0102670 2024-03-22T01:36:40.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.211+0000 7f9bf566d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9bf010c910 con 0x7f9bf0101cb0 2024-03-22T01:36:40.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.211+0000 7f9beeffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9bf0100ac0 0x7f9bf0197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:40.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.211+0000 7f9beeffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9bf0100ac0 0x7f9bf0197f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54544/0 (socket says 172.21.15.72:54544) 2024-03-22T01:36:40.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.211+0000 7f9beeffd640 1 -- 172.21.15.72:0/1879818794 learned_addr learned my addr 172.21.15.72:0/1879818794 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:40.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.211+0000 7f9bef7fe640 1 --2- 172.21.15.72:0/1879818794 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9bf0102670 0x7f9bf019f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:40.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.211+0000 7f9be7fff640 1 --2- 172.21.15.72:0/1879818794 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9bf0101cb0 0x7f9bf0198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:40.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.211+0000 7f9beeffd640 1 -- 172.21.15.72:0/1879818794 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9bf0101cb0 msgr2=0x7f9bf0198440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:40.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.211+0000 7f9beeffd640 1 --2- 172.21.15.72:0/1879818794 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9bf0101cb0 0x7f9bf0198440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:40.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.211+0000 7f9beeffd640 1 -- 172.21.15.72:0/1879818794 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9bf0102670 msgr2=0x7f9bf019f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:40.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.211+0000 7f9beeffd640 1 --2- 172.21.15.72:0/1879818794 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9bf0102670 0x7f9bf019f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:40.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.211+0000 7f9beeffd640 1 -- 172.21.15.72:0/1879818794 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bf019fb20 con 0x7f9bf0100ac0 2024-03-22T01:36:40.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.211+0000 7f9beeffd640 1 --2- 172.21.15.72:0/1879818794 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9bf0100ac0 0x7f9bf0197f00 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f9be0000c00 tx=0x7f9be002c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:40.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.212+0000 7f9becff9640 1 -- 172.21.15.72:0/1879818794 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9be0043020 con 0x7f9bf0100ac0 2024-03-22T01:36:40.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.212+0000 7f9bf566d640 1 -- 172.21.15.72:0/1879818794 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bf019fdb0 con 0x7f9bf0100ac0 2024-03-22T01:36:40.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.212+0000 7f9becff9640 1 -- 172.21.15.72:0/1879818794 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9be00054e0 con 0x7f9bf0100ac0 2024-03-22T01:36:40.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.212+0000 7f9becff9640 1 -- 172.21.15.72:0/1879818794 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9be0003c40 con 0x7f9bf0100ac0 2024-03-22T01:36:40.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.212+0000 7f9bf566d640 1 -- 172.21.15.72:0/1879818794 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bf01a0270 con 0x7f9bf0100ac0 2024-03-22T01:36:40.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.213+0000 7f9bf566d640 1 -- 172.21.15.72:0/1879818794 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9bf01a0560 con 0x7f9bf0100ac0 2024-03-22T01:36:40.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.214+0000 7f9becff9640 1 -- 172.21.15.72:0/1879818794 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 21) ==== 137666+0+0 (secure 0 0 0) 0x7f9be0046020 con 0x7f9bf0100ac0 2024-03-22T01:36:40.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.214+0000 7f9becff9640 1 --2- 172.21.15.72:0/1879818794 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f9bdc0a2d20 0x7f9bdc0a51e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:40.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.215+0000 7f9becff9640 1 -- 172.21.15.72:0/1879818794 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f9be00f1850 con 0x7f9bf0100ac0 2024-03-22T01:36:40.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.215+0000 7f9be7fff640 1 --2- 172.21.15.72:0/1879818794 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f9bdc0a2d20 0x7f9bdc0a51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:40.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.215+0000 7f9be7fff640 1 --2- 172.21.15.72:0/1879818794 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f9bdc0a2d20 0x7f9bdc0a51e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9bf0199410 tx=0x7f9bd8005f90 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:40.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.218+0000 7f9becff9640 1 -- 172.21.15.72:0/1879818794 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9be004d020 con 0x7f9bf0100ac0 2024-03-22T01:36:40.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.427+0000 7f9bf566d640 1 -- 172.21.15.72:0/1879818794 --> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9bf01a0730 con 0x7f9bdc0a2d20 2024-03-22T01:36:40.436 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.437+0000 7f9becff9640 1 -- 172.21.15.72:0/1879818794 <== mgr.15381 v2:172.21.15.72:6800/1379139045 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339302 (secure 0 0 0) 0x7f9bf01a0730 con 0x7f9bdc0a2d20 2024-03-22T01:36:40.441 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:36:40.443 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-03-22T01:36:40.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.441+0000 7f9bf566d640 1 -- 172.21.15.72:0/1879818794 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f9bdc0a2d20 msgr2=0x7f9bdc0a51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:40.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.441+0000 7f9bf566d640 1 --2- 172.21.15.72:0/1879818794 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f9bdc0a2d20 0x7f9bdc0a51e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9bf0199410 tx=0x7f9bd8005f90 comp rx=0 tx=0).stop 2024-03-22T01:36:40.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.441+0000 7f9bf566d640 1 -- 172.21.15.72:0/1879818794 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9bf0100ac0 msgr2=0x7f9bf0197f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:40.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.441+0000 7f9bf566d640 1 --2- 172.21.15.72:0/1879818794 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9bf0100ac0 0x7f9bf0197f00 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f9be0000c00 tx=0x7f9be002c650 comp rx=0 tx=0).stop 2024-03-22T01:36:40.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.441+0000 7f9bf566d640 1 -- 172.21.15.72:0/1879818794 shutdown_connections 2024-03-22T01:36:40.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.441+0000 7f9bf566d640 1 --2- 172.21.15.72:0/1879818794 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f9bdc0a2d20 0x7f9bdc0a51e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:40.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.441+0000 7f9bf566d640 1 --2- 172.21.15.72:0/1879818794 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9bf0102670 0x7f9bf019f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:40.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.441+0000 7f9bf566d640 1 --2- 172.21.15.72:0/1879818794 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9bf0101cb0 0x7f9bf0198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:40.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.441+0000 7f9bf566d640 1 --2- 172.21.15.72:0/1879818794 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9bf0100ac0 0x7f9bf0197f00 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:40.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.441+0000 7f9bf566d640 1 -- 172.21.15.72:0/1879818794 >> 172.21.15.72:0/1879818794 conn(0x7f9bf00fc030 msgr2=0x7f9bf00fda90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:40.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.441+0000 7f9bf566d640 1 -- 172.21.15.72:0/1879818794 shutdown_connections 2024-03-22T01:36:40.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.441+0000 7f9bf566d640 1 -- 172.21.15.72:0/1879818794 wait complete. 2024-03-22T01:36:40.451 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":4,"stamp":"2024-03-22T01:36:40.202937+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1186284,"num_objects":28,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":484,"num_read_kb":398,"num_write":383,"num_write_kb":4350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":789,"ondisk_log_size":789,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":373868,"kb_used_data":16176,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124567956,"statfs":{"total":1151940427776,"available":1151557586944,"internally_reserved":0,"allocated":16564224,"data_stored":9860107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":365933882},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.990801"},"pg_stats":[{"pgid":"3.3f","version":"93'8","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500839+0000","last_change":"2024-03-22T01:34:15.487496+0000","last_active":"2024-03-22T01:36:38.500839+0000","last_peered":"2024-03-22T01:36:38.500839+0000","last_clean":"2024-03-22T01:36:38.500839+0000","last_became_active":"2024-03-22T01:34:15.486700+0000","last_became_peered":"2024-03-22T01:34:15.486700+0000","last_unstale":"2024-03-22T01:36:38.500839+0000","last_undegraded":"2024-03-22T01:36:38.500839+0000","last_fullsized":"2024-03-22T01:36:38.500839+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:03:19.146811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"93'4","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.501995+0000","last_change":"2024-03-22T01:34:15.489791+0000","last_active":"2024-03-22T01:36:38.501995+0000","last_peered":"2024-03-22T01:36:38.501995+0000","last_clean":"2024-03-22T01:36:38.501995+0000","last_became_active":"2024-03-22T01:34:15.489493+0000","last_became_peered":"2024-03-22T01:34:15.489493+0000","last_unstale":"2024-03-22T01:36:38.501995+0000","last_undegraded":"2024-03-22T01:36:38.501995+0000","last_fullsized":"2024-03-22T01:36:38.501995+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:45:05.060454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"93'7","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.501785+0000","last_change":"2024-03-22T01:34:15.491567+0000","last_active":"2024-03-22T01:36:38.501785+0000","last_peered":"2024-03-22T01:36:38.501785+0000","last_clean":"2024-03-22T01:36:38.501785+0000","last_became_active":"2024-03-22T01:34:15.491275+0000","last_became_peered":"2024-03-22T01:34:15.491275+0000","last_unstale":"2024-03-22T01:36:38.501785+0000","last_undegraded":"2024-03-22T01:36:38.501785+0000","last_fullsized":"2024-03-22T01:36:38.501785+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:26:36.630811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"93'8","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500405+0000","last_change":"2024-03-22T01:34:15.488579+0000","last_active":"2024-03-22T01:36:38.500405+0000","last_peered":"2024-03-22T01:36:38.500405+0000","last_clean":"2024-03-22T01:36:38.500405+0000","last_became_active":"2024-03-22T01:34:15.488262+0000","last_became_peered":"2024-03-22T01:34:15.488262+0000","last_unstale":"2024-03-22T01:36:38.500405+0000","last_undegraded":"2024-03-22T01:36:38.500405+0000","last_fullsized":"2024-03-22T01:36:38.500405+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:58:21.788420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"93'4","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.307700+0000","last_change":"2024-03-22T01:34:15.485744+0000","last_active":"2024-03-22T01:36:38.307700+0000","last_peered":"2024-03-22T01:36:38.307700+0000","last_clean":"2024-03-22T01:36:38.307700+0000","last_became_active":"2024-03-22T01:34:15.485378+0000","last_became_peered":"2024-03-22T01:34:15.485378+0000","last_unstale":"2024-03-22T01:36:38.307700+0000","last_undegraded":"2024-03-22T01:36:38.307700+0000","last_fullsized":"2024-03-22T01:36:38.307700+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:06:40.979475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"93'6","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.192414+0000","last_change":"2024-03-22T01:34:15.495051+0000","last_active":"2024-03-22T01:36:38.192414+0000","last_peered":"2024-03-22T01:36:38.192414+0000","last_clean":"2024-03-22T01:36:38.192414+0000","last_became_active":"2024-03-22T01:34:15.494691+0000","last_became_peered":"2024-03-22T01:34:15.494691+0000","last_unstale":"2024-03-22T01:36:38.192414+0000","last_undegraded":"2024-03-22T01:36:38.192414+0000","last_fullsized":"2024-03-22T01:36:38.192414+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T01:44:28.336545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"93'5","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.300153+0000","last_change":"2024-03-22T01:34:15.493911+0000","last_active":"2024-03-22T01:36:38.300153+0000","last_peered":"2024-03-22T01:36:38.300153+0000","last_clean":"2024-03-22T01:36:38.300153+0000","last_became_active":"2024-03-22T01:34:15.493586+0000","last_became_peered":"2024-03-22T01:34:15.493586+0000","last_unstale":"2024-03-22T01:36:38.300153+0000","last_undegraded":"2024-03-22T01:36:38.300153+0000","last_fullsized":"2024-03-22T01:36:38.300153+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:22:41.391855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"93'9","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.307138+0000","last_change":"2024-03-22T01:34:15.481410+0000","last_active":"2024-03-22T01:36:38.307138+0000","last_peered":"2024-03-22T01:36:38.307138+0000","last_clean":"2024-03-22T01:36:38.307138+0000","last_became_active":"2024-03-22T01:34:15.481109+0000","last_became_peered":"2024-03-22T01:34:15.481109+0000","last_unstale":"2024-03-22T01:36:38.307138+0000","last_undegraded":"2024-03-22T01:36:38.307138+0000","last_fullsized":"2024-03-22T01:36:38.307138+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:27:10.004584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"93'11","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.460250+0000","last_change":"2024-03-22T01:34:15.485173+0000","last_active":"2024-03-22T01:36:38.460250+0000","last_peered":"2024-03-22T01:36:38.460250+0000","last_clean":"2024-03-22T01:36:38.460250+0000","last_became_active":"2024-03-22T01:34:15.484812+0000","last_became_peered":"2024-03-22T01:34:15.484812+0000","last_unstale":"2024-03-22T01:36:38.460250+0000","last_undegraded":"2024-03-22T01:36:38.460250+0000","last_fullsized":"2024-03-22T01:36:38.460250+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:44:06.383568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"93'16","reported_seq":47,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.192571+0000","last_change":"2024-03-22T01:34:15.497964+0000","last_active":"2024-03-22T01:36:38.192571+0000","last_peered":"2024-03-22T01:36:38.192571+0000","last_clean":"2024-03-22T01:36:38.192571+0000","last_became_active":"2024-03-22T01:34:15.496887+0000","last_became_peered":"2024-03-22T01:34:15.496887+0000","last_unstale":"2024-03-22T01:36:38.192571+0000","last_undegraded":"2024-03-22T01:36:38.192571+0000","last_fullsized":"2024-03-22T01:36:38.192571+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:13:40.420667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"93'6","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.300582+0000","last_change":"2024-03-22T01:34:15.496310+0000","last_active":"2024-03-22T01:36:38.300582+0000","last_peered":"2024-03-22T01:36:38.300582+0000","last_clean":"2024-03-22T01:36:38.300582+0000","last_became_active":"2024-03-22T01:34:15.494196+0000","last_became_peered":"2024-03-22T01:34:15.494196+0000","last_unstale":"2024-03-22T01:36:38.300582+0000","last_undegraded":"2024-03-22T01:36:38.300582+0000","last_fullsized":"2024-03-22T01:36:38.300582+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:33:02.798310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"93'7","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500458+0000","last_change":"2024-03-22T01:34:15.495380+0000","last_active":"2024-03-22T01:36:38.500458+0000","last_peered":"2024-03-22T01:36:38.500458+0000","last_clean":"2024-03-22T01:36:38.500458+0000","last_became_active":"2024-03-22T01:34:15.495063+0000","last_became_peered":"2024-03-22T01:34:15.495063+0000","last_unstale":"2024-03-22T01:36:38.500458+0000","last_undegraded":"2024-03-22T01:36:38.500458+0000","last_fullsized":"2024-03-22T01:36:38.500458+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:46:29.497329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"93'10","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.189104+0000","last_change":"2024-03-22T01:34:15.497533+0000","last_active":"2024-03-22T01:36:38.189104+0000","last_peered":"2024-03-22T01:36:38.189104+0000","last_clean":"2024-03-22T01:36:38.189104+0000","last_became_active":"2024-03-22T01:34:15.497223+0000","last_became_peered":"2024-03-22T01:34:15.497223+0000","last_unstale":"2024-03-22T01:36:38.189104+0000","last_undegraded":"2024-03-22T01:36:38.189104+0000","last_fullsized":"2024-03-22T01:36:38.189104+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:17:49.202842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"93'7","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.501331+0000","last_change":"2024-03-22T01:34:15.491317+0000","last_active":"2024-03-22T01:36:38.501331+0000","last_peered":"2024-03-22T01:36:38.501331+0000","last_clean":"2024-03-22T01:36:38.501331+0000","last_became_active":"2024-03-22T01:34:15.490982+0000","last_became_peered":"2024-03-22T01:34:15.490982+0000","last_unstale":"2024-03-22T01:36:38.501331+0000","last_undegraded":"2024-03-22T01:36:38.501331+0000","last_fullsized":"2024-03-22T01:36:38.501331+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:05:39.889964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"93'7","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.501112+0000","last_change":"2024-03-22T01:34:15.488502+0000","last_active":"2024-03-22T01:36:38.501112+0000","last_peered":"2024-03-22T01:36:38.501112+0000","last_clean":"2024-03-22T01:36:38.501112+0000","last_became_active":"2024-03-22T01:34:15.488200+0000","last_became_peered":"2024-03-22T01:34:15.488200+0000","last_unstale":"2024-03-22T01:36:38.501112+0000","last_undegraded":"2024-03-22T01:36:38.501112+0000","last_fullsized":"2024-03-22T01:36:38.501112+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:52:59.991591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"93'9","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.459533+0000","last_change":"2024-03-22T01:34:15.489991+0000","last_active":"2024-03-22T01:36:38.459533+0000","last_peered":"2024-03-22T01:36:38.459533+0000","last_clean":"2024-03-22T01:36:38.459533+0000","last_became_active":"2024-03-22T01:34:15.489696+0000","last_became_peered":"2024-03-22T01:34:15.489696+0000","last_unstale":"2024-03-22T01:36:38.459533+0000","last_undegraded":"2024-03-22T01:36:38.459533+0000","last_fullsized":"2024-03-22T01:36:38.459533+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:10:45.316422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"93'8","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500862+0000","last_change":"2024-03-22T01:34:15.488011+0000","last_active":"2024-03-22T01:36:38.500862+0000","last_peered":"2024-03-22T01:36:38.500862+0000","last_clean":"2024-03-22T01:36:38.500862+0000","last_became_active":"2024-03-22T01:34:15.487282+0000","last_became_peered":"2024-03-22T01:34:15.487282+0000","last_unstale":"2024-03-22T01:36:38.500862+0000","last_undegraded":"2024-03-22T01:36:38.500862+0000","last_fullsized":"2024-03-22T01:36:38.500862+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:16:21.396273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"93'10","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.305057+0000","last_change":"2024-03-22T01:34:15.488046+0000","last_active":"2024-03-22T01:36:38.305057+0000","last_peered":"2024-03-22T01:36:38.305057+0000","last_clean":"2024-03-22T01:36:38.305057+0000","last_became_active":"2024-03-22T01:34:15.487731+0000","last_became_peered":"2024-03-22T01:34:15.487731+0000","last_unstale":"2024-03-22T01:36:38.305057+0000","last_undegraded":"2024-03-22T01:36:38.305057+0000","last_fullsized":"2024-03-22T01:36:38.305057+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:44:16.931544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"93'7","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.191341+0000","last_change":"2024-03-22T01:34:15.503737+0000","last_active":"2024-03-22T01:36:38.191341+0000","last_peered":"2024-03-22T01:36:38.191341+0000","last_clean":"2024-03-22T01:36:38.191341+0000","last_became_active":"2024-03-22T01:34:15.502169+0000","last_became_peered":"2024-03-22T01:34:15.502169+0000","last_unstale":"2024-03-22T01:36:38.191341+0000","last_undegraded":"2024-03-22T01:36:38.191341+0000","last_fullsized":"2024-03-22T01:36:38.191341+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:05:53.027318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":24,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.191251+0000","last_change":"2024-03-22T01:34:19.527964+0000","last_active":"2024-03-22T01:36:38.191251+0000","last_peered":"2024-03-22T01:36:38.191251+0000","last_clean":"2024-03-22T01:36:38.191251+0000","last_became_active":"2024-03-22T01:34:19.527526+0000","last_became_peered":"2024-03-22T01:34:19.527526+0000","last_unstale":"2024-03-22T01:36:38.191251+0000","last_undegraded":"2024-03-22T01:36:38.191251+0000","last_fullsized":"2024-03-22T01:36:38.191251+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:35:39.404439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'4","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.305744+0000","last_change":"2024-03-22T01:34:14.473065+0000","last_active":"2024-03-22T01:36:38.305744+0000","last_peered":"2024-03-22T01:36:38.305744+0000","last_clean":"2024-03-22T01:36:38.305744+0000","last_became_active":"2024-03-22T01:34:14.472653+0000","last_became_peered":"2024-03-22T01:34:14.472653+0000","last_unstale":"2024-03-22T01:36:38.305744+0000","last_undegraded":"2024-03-22T01:36:38.305744+0000","last_fullsized":"2024-03-22T01:36:38.305744+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:11:09.525438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"93'5","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.308183+0000","last_change":"2024-03-22T01:34:15.497818+0000","last_active":"2024-03-22T01:36:38.308183+0000","last_peered":"2024-03-22T01:36:38.308183+0000","last_clean":"2024-03-22T01:36:38.308183+0000","last_became_active":"2024-03-22T01:34:15.497488+0000","last_became_peered":"2024-03-22T01:34:15.497488+0000","last_unstale":"2024-03-22T01:36:38.308183+0000","last_undegraded":"2024-03-22T01:36:38.308183+0000","last_fullsized":"2024-03-22T01:36:38.308183+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:21:09.340557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":24,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500006+0000","last_change":"2024-03-22T01:34:19.528435+0000","last_active":"2024-03-22T01:36:38.500006+0000","last_peered":"2024-03-22T01:36:38.500006+0000","last_clean":"2024-03-22T01:36:38.500006+0000","last_became_active":"2024-03-22T01:34:19.528044+0000","last_became_peered":"2024-03-22T01:34:19.528044+0000","last_unstale":"2024-03-22T01:36:38.500006+0000","last_undegraded":"2024-03-22T01:36:38.500006+0000","last_fullsized":"2024-03-22T01:36:38.500006+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:52:05.682173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"86'1","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.308372+0000","last_change":"2024-03-22T01:34:14.465817+0000","last_active":"2024-03-22T01:36:38.308372+0000","last_peered":"2024-03-22T01:36:38.308372+0000","last_clean":"2024-03-22T01:36:38.308372+0000","last_became_active":"2024-03-22T01:34:14.464338+0000","last_became_peered":"2024-03-22T01:34:14.464338+0000","last_unstale":"2024-03-22T01:36:38.308372+0000","last_undegraded":"2024-03-22T01:36:38.308372+0000","last_fullsized":"2024-03-22T01:36:38.308372+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:04:47.457092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"93'6","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.502083+0000","last_change":"2024-03-22T01:34:15.481988+0000","last_active":"2024-03-22T01:36:38.502083+0000","last_peered":"2024-03-22T01:36:38.502083+0000","last_clean":"2024-03-22T01:36:38.502083+0000","last_became_active":"2024-03-22T01:34:15.481676+0000","last_became_peered":"2024-03-22T01:34:15.481676+0000","last_unstale":"2024-03-22T01:36:38.502083+0000","last_undegraded":"2024-03-22T01:36:38.502083+0000","last_fullsized":"2024-03-22T01:36:38.502083+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:37:54.609731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":24,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500197+0000","last_change":"2024-03-22T01:34:19.528349+0000","last_active":"2024-03-22T01:36:38.500197+0000","last_peered":"2024-03-22T01:36:38.500197+0000","last_clean":"2024-03-22T01:36:38.500197+0000","last_became_active":"2024-03-22T01:34:19.527937+0000","last_became_peered":"2024-03-22T01:34:19.527937+0000","last_unstale":"2024-03-22T01:36:38.500197+0000","last_undegraded":"2024-03-22T01:36:38.500197+0000","last_fullsized":"2024-03-22T01:36:38.500197+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:48:34.242104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"86'2","reported_seq":44,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.308006+0000","last_change":"2024-03-22T01:35:16.756010+0000","last_active":"2024-03-22T01:36:38.308006+0000","last_peered":"2024-03-22T01:36:38.308006+0000","last_clean":"2024-03-22T01:36:38.308006+0000","last_became_active":"2024-03-22T01:35:16.751845+0000","last_became_peered":"2024-03-22T01:35:16.751845+0000","last_unstale":"2024-03-22T01:36:38.308006+0000","last_undegraded":"2024-03-22T01:36:38.308006+0000","last_fullsized":"2024-03-22T01:36:38.308006+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:23:18.305330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"93'9","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.459910+0000","last_change":"2024-03-22T01:34:15.491345+0000","last_active":"2024-03-22T01:36:38.459910+0000","last_peered":"2024-03-22T01:36:38.459910+0000","last_clean":"2024-03-22T01:36:38.459910+0000","last_became_active":"2024-03-22T01:34:15.490833+0000","last_became_peered":"2024-03-22T01:34:15.490833+0000","last_unstale":"2024-03-22T01:36:38.459910+0000","last_undegraded":"2024-03-22T01:36:38.459910+0000","last_fullsized":"2024-03-22T01:36:38.459910+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:30:43.950679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":24,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500078+0000","last_change":"2024-03-22T01:34:19.528351+0000","last_active":"2024-03-22T01:36:38.500078+0000","last_peered":"2024-03-22T01:36:38.500078+0000","last_clean":"2024-03-22T01:36:38.500078+0000","last_became_active":"2024-03-22T01:34:19.527939+0000","last_became_peered":"2024-03-22T01:34:19.527939+0000","last_unstale":"2024-03-22T01:36:38.500078+0000","last_undegraded":"2024-03-22T01:36:38.500078+0000","last_fullsized":"2024-03-22T01:36:38.500078+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:30:21.055848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":21,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500428+0000","last_change":"2024-03-22T01:35:16.755726+0000","last_active":"2024-03-22T01:36:38.500428+0000","last_peered":"2024-03-22T01:36:38.500428+0000","last_clean":"2024-03-22T01:36:38.500428+0000","last_became_active":"2024-03-22T01:35:16.749107+0000","last_became_peered":"2024-03-22T01:35:16.749107+0000","last_unstale":"2024-03-22T01:36:38.500428+0000","last_undegraded":"2024-03-22T01:36:38.500428+0000","last_fullsized":"2024-03-22T01:36:38.500428+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:49:37.500003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"93'15","reported_seq":44,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.314906+0000","last_change":"2024-03-22T01:34:15.479530+0000","last_active":"2024-03-22T01:36:38.314906+0000","last_peered":"2024-03-22T01:36:38.314906+0000","last_clean":"2024-03-22T01:36:38.314906+0000","last_became_active":"2024-03-22T01:34:15.479198+0000","last_became_peered":"2024-03-22T01:34:15.479198+0000","last_unstale":"2024-03-22T01:36:38.314906+0000","last_undegraded":"2024-03-22T01:36:38.314906+0000","last_fullsized":"2024-03-22T01:36:38.314906+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:28:51.624369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":24,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.191181+0000","last_change":"2024-03-22T01:34:19.524486+0000","last_active":"2024-03-22T01:36:38.191181+0000","last_peered":"2024-03-22T01:36:38.191181+0000","last_clean":"2024-03-22T01:36:38.191181+0000","last_became_active":"2024-03-22T01:34:19.524030+0000","last_became_peered":"2024-03-22T01:34:19.524030+0000","last_unstale":"2024-03-22T01:36:38.191181+0000","last_undegraded":"2024-03-22T01:36:38.191181+0000","last_fullsized":"2024-03-22T01:36:38.191181+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:31:10.157431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"86'1","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.314671+0000","last_change":"2024-03-22T01:34:14.469191+0000","last_active":"2024-03-22T01:36:38.314671+0000","last_peered":"2024-03-22T01:36:38.314671+0000","last_clean":"2024-03-22T01:36:38.314671+0000","last_became_active":"2024-03-22T01:34:14.468842+0000","last_became_peered":"2024-03-22T01:34:14.468842+0000","last_unstale":"2024-03-22T01:36:38.314671+0000","last_undegraded":"2024-03-22T01:36:38.314671+0000","last_fullsized":"2024-03-22T01:36:38.314671+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:29:56.566256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"93'7","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.307636+0000","last_change":"2024-03-22T01:34:15.483736+0000","last_active":"2024-03-22T01:36:38.307636+0000","last_peered":"2024-03-22T01:36:38.307636+0000","last_clean":"2024-03-22T01:36:38.307636+0000","last_became_active":"2024-03-22T01:34:15.483401+0000","last_became_peered":"2024-03-22T01:34:15.483401+0000","last_unstale":"2024-03-22T01:36:38.307636+0000","last_undegraded":"2024-03-22T01:36:38.307636+0000","last_fullsized":"2024-03-22T01:36:38.307636+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:26:51.974932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":24,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.191134+0000","last_change":"2024-03-22T01:34:19.534542+0000","last_active":"2024-03-22T01:36:38.191134+0000","last_peered":"2024-03-22T01:36:38.191134+0000","last_clean":"2024-03-22T01:36:38.191134+0000","last_became_active":"2024-03-22T01:34:19.534039+0000","last_became_peered":"2024-03-22T01:34:19.534039+0000","last_unstale":"2024-03-22T01:36:38.191134+0000","last_undegraded":"2024-03-22T01:36:38.191134+0000","last_fullsized":"2024-03-22T01:36:38.191134+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:59:37.363972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"86'1","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.192279+0000","last_change":"2024-03-22T01:34:14.468382+0000","last_active":"2024-03-22T01:36:38.192279+0000","last_peered":"2024-03-22T01:36:38.192279+0000","last_clean":"2024-03-22T01:36:38.192279+0000","last_became_active":"2024-03-22T01:34:14.467939+0000","last_became_peered":"2024-03-22T01:34:14.467939+0000","last_unstale":"2024-03-22T01:36:38.192279+0000","last_undegraded":"2024-03-22T01:36:38.192279+0000","last_fullsized":"2024-03-22T01:36:38.192279+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:57:17.031893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"93'8","reported_seq":43,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.314801+0000","last_change":"2024-03-22T01:35:17.206178+0000","last_active":"2024-03-22T01:36:38.314801+0000","last_peered":"2024-03-22T01:36:38.314801+0000","last_clean":"2024-03-22T01:36:38.314801+0000","last_became_active":"2024-03-22T01:35:17.205717+0000","last_became_peered":"2024-03-22T01:35:17.205717+0000","last_unstale":"2024-03-22T01:36:38.314801+0000","last_undegraded":"2024-03-22T01:36:38.314801+0000","last_fullsized":"2024-03-22T01:36:38.314801+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:14:15.920771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500689+0000","last_change":"2024-03-22T01:36:17.659141+0000","last_active":"2024-03-22T01:36:38.500689+0000","last_peered":"2024-03-22T01:36:38.500689+0000","last_clean":"2024-03-22T01:36:38.500689+0000","last_became_active":"2024-03-22T01:36:17.658611+0000","last_became_peered":"2024-03-22T01:36:17.658611+0000","last_unstale":"2024-03-22T01:36:38.500689+0000","last_undegraded":"2024-03-22T01:36:38.500689+0000","last_fullsized":"2024-03-22T01:36:38.500689+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:45:28.895847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,10,0],"acting":[11,4,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.189382+0000","last_change":"2024-03-22T01:34:14.464485+0000","last_active":"2024-03-22T01:36:38.189382+0000","last_peered":"2024-03-22T01:36:38.189382+0000","last_clean":"2024-03-22T01:36:38.189382+0000","last_became_active":"2024-03-22T01:34:14.463973+0000","last_became_peered":"2024-03-22T01:34:14.463973+0000","last_unstale":"2024-03-22T01:36:38.189382+0000","last_undegraded":"2024-03-22T01:36:38.189382+0000","last_fullsized":"2024-03-22T01:36:38.189382+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:11:13.265346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"93'8","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.307338+0000","last_change":"2024-03-22T01:34:15.483893+0000","last_active":"2024-03-22T01:36:38.307338+0000","last_peered":"2024-03-22T01:36:38.307338+0000","last_clean":"2024-03-22T01:36:38.307338+0000","last_became_active":"2024-03-22T01:34:15.482295+0000","last_became_peered":"2024-03-22T01:34:15.482295+0000","last_unstale":"2024-03-22T01:36:38.307338+0000","last_undegraded":"2024-03-22T01:36:38.307338+0000","last_fullsized":"2024-03-22T01:36:38.307338+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:17:33.274900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":25,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.501992+0000","last_change":"2024-03-22T01:34:19.522567+0000","last_active":"2024-03-22T01:36:38.501992+0000","last_peered":"2024-03-22T01:36:38.501992+0000","last_clean":"2024-03-22T01:36:38.501992+0000","last_became_active":"2024-03-22T01:34:19.522165+0000","last_became_peered":"2024-03-22T01:34:19.522165+0000","last_unstale":"2024-03-22T01:36:38.501992+0000","last_undegraded":"2024-03-22T01:36:38.501992+0000","last_fullsized":"2024-03-22T01:36:38.501992+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:41:37.829622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":31,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.314474+0000","last_change":"2024-03-22T01:34:14.463824+0000","last_active":"2024-03-22T01:36:38.314474+0000","last_peered":"2024-03-22T01:36:38.314474+0000","last_clean":"2024-03-22T01:36:38.314474+0000","last_became_active":"2024-03-22T01:34:14.463484+0000","last_became_peered":"2024-03-22T01:34:14.463484+0000","last_unstale":"2024-03-22T01:36:38.314474+0000","last_undegraded":"2024-03-22T01:36:38.314474+0000","last_fullsized":"2024-03-22T01:36:38.314474+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:19:31.568853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"93'11","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.304658+0000","last_change":"2024-03-22T01:34:15.488883+0000","last_active":"2024-03-22T01:36:38.304658+0000","last_peered":"2024-03-22T01:36:38.304658+0000","last_clean":"2024-03-22T01:36:38.304658+0000","last_became_active":"2024-03-22T01:34:15.488573+0000","last_became_peered":"2024-03-22T01:34:15.488573+0000","last_unstale":"2024-03-22T01:36:38.304658+0000","last_undegraded":"2024-03-22T01:36:38.304658+0000","last_fullsized":"2024-03-22T01:36:38.304658+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:28:46.388930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.307244+0000","last_change":"2024-03-22T01:34:19.527321+0000","last_active":"2024-03-22T01:36:38.307244+0000","last_peered":"2024-03-22T01:36:38.307244+0000","last_clean":"2024-03-22T01:36:38.307244+0000","last_became_active":"2024-03-22T01:34:19.526905+0000","last_became_peered":"2024-03-22T01:34:19.526905+0000","last_unstale":"2024-03-22T01:36:38.307244+0000","last_undegraded":"2024-03-22T01:36:38.307244+0000","last_fullsized":"2024-03-22T01:36:38.307244+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:49:31.023938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.192604+0000","last_change":"2024-03-22T01:34:14.471415+0000","last_active":"2024-03-22T01:36:38.192604+0000","last_peered":"2024-03-22T01:36:38.192604+0000","last_clean":"2024-03-22T01:36:38.192604+0000","last_became_active":"2024-03-22T01:34:14.470928+0000","last_became_peered":"2024-03-22T01:34:14.470928+0000","last_unstale":"2024-03-22T01:36:38.192604+0000","last_undegraded":"2024-03-22T01:36:38.192604+0000","last_fullsized":"2024-03-22T01:36:38.192604+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:09:10.286549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"93'10","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.460165+0000","last_change":"2024-03-22T01:34:15.489656+0000","last_active":"2024-03-22T01:36:38.460165+0000","last_peered":"2024-03-22T01:36:38.460165+0000","last_clean":"2024-03-22T01:36:38.460165+0000","last_became_active":"2024-03-22T01:34:15.488933+0000","last_became_peered":"2024-03-22T01:34:15.488933+0000","last_unstale":"2024-03-22T01:36:38.460165+0000","last_undegraded":"2024-03-22T01:36:38.460165+0000","last_fullsized":"2024-03-22T01:36:38.460165+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:17:38.789414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.304368+0000","last_change":"2024-03-22T01:34:19.534174+0000","last_active":"2024-03-22T01:36:38.304368+0000","last_peered":"2024-03-22T01:36:38.304368+0000","last_clean":"2024-03-22T01:36:38.304368+0000","last_became_active":"2024-03-22T01:34:19.533759+0000","last_became_peered":"2024-03-22T01:34:19.533759+0000","last_unstale":"2024-03-22T01:36:38.304368+0000","last_undegraded":"2024-03-22T01:36:38.304368+0000","last_fullsized":"2024-03-22T01:36:38.304368+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:24:50.363984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.300490+0000","last_change":"2024-03-22T01:34:14.465755+0000","last_active":"2024-03-22T01:36:38.300490+0000","last_peered":"2024-03-22T01:36:38.300490+0000","last_clean":"2024-03-22T01:36:38.300490+0000","last_became_active":"2024-03-22T01:34:14.465230+0000","last_became_peered":"2024-03-22T01:34:14.465230+0000","last_unstale":"2024-03-22T01:36:38.300490+0000","last_undegraded":"2024-03-22T01:36:38.300490+0000","last_fullsized":"2024-03-22T01:36:38.300490+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:56:29.286551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"93'9","reported_seq":45,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.502022+0000","last_change":"2024-03-22T01:35:16.748981+0000","last_active":"2024-03-22T01:36:38.502022+0000","last_peered":"2024-03-22T01:36:38.502022+0000","last_clean":"2024-03-22T01:36:38.502022+0000","last_became_active":"2024-03-22T01:35:16.748524+0000","last_became_peered":"2024-03-22T01:35:16.748524+0000","last_unstale":"2024-03-22T01:36:38.502022+0000","last_undegraded":"2024-03-22T01:36:38.502022+0000","last_fullsized":"2024-03-22T01:36:38.502022+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:23:37.763347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":31,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500236+0000","last_change":"2024-03-22T01:36:17.789745+0000","last_active":"2024-03-22T01:36:38.500236+0000","last_peered":"2024-03-22T01:36:38.500236+0000","last_clean":"2024-03-22T01:36:38.500236+0000","last_became_active":"2024-03-22T01:36:17.789277+0000","last_became_peered":"2024-03-22T01:36:17.789277+0000","last_unstale":"2024-03-22T01:36:38.500236+0000","last_undegraded":"2024-03-22T01:36:38.500236+0000","last_fullsized":"2024-03-22T01:36:38.500236+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:57:47.671252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"86'3","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.315060+0000","last_change":"2024-03-22T01:34:14.463365+0000","last_active":"2024-03-22T01:36:38.315060+0000","last_peered":"2024-03-22T01:36:38.315060+0000","last_clean":"2024-03-22T01:36:38.315060+0000","last_became_active":"2024-03-22T01:34:14.462984+0000","last_became_peered":"2024-03-22T01:34:14.462984+0000","last_unstale":"2024-03-22T01:36:38.315060+0000","last_undegraded":"2024-03-22T01:36:38.315060+0000","last_fullsized":"2024-03-22T01:36:38.315060+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:55:27.318854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"93'11","reported_seq":43,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.460074+0000","last_change":"2024-03-22T01:34:15.485309+0000","last_active":"2024-03-22T01:36:38.460074+0000","last_peered":"2024-03-22T01:36:38.460074+0000","last_clean":"2024-03-22T01:36:38.460074+0000","last_became_active":"2024-03-22T01:34:15.484964+0000","last_became_peered":"2024-03-22T01:34:15.484964+0000","last_unstale":"2024-03-22T01:36:38.460074+0000","last_undegraded":"2024-03-22T01:36:38.460074+0000","last_fullsized":"2024-03-22T01:36:38.460074+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:25:57.911623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":31,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.304733+0000","last_change":"2024-03-22T01:36:17.659080+0000","last_active":"2024-03-22T01:36:38.304733+0000","last_peered":"2024-03-22T01:36:38.304733+0000","last_clean":"2024-03-22T01:36:38.304733+0000","last_became_active":"2024-03-22T01:36:17.658738+0000","last_became_peered":"2024-03-22T01:36:17.658738+0000","last_unstale":"2024-03-22T01:36:38.304733+0000","last_undegraded":"2024-03-22T01:36:38.304733+0000","last_fullsized":"2024-03-22T01:36:38.304733+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:28:38.338235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,10],"acting":[9,8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"96'37","reported_seq":321,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.688979+0000","last_change":"2024-03-22T01:34:14.469378+0000","last_active":"2024-03-22T01:36:38.688979+0000","last_peered":"2024-03-22T01:36:38.688979+0000","last_clean":"2024-03-22T01:36:38.688979+0000","last_became_active":"2024-03-22T01:34:14.468956+0000","last_became_peered":"2024-03-22T01:34:14.468956+0000","last_unstale":"2024-03-22T01:36:38.688979+0000","last_undegraded":"2024-03-22T01:36:38.688979+0000","last_fullsized":"2024-03-22T01:36:38.688979+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:40:49.048793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70068,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":250,"num_read_kb":194,"num_write":37,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"93'6","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.307927+0000","last_change":"2024-03-22T01:34:15.488497+0000","last_active":"2024-03-22T01:36:38.307927+0000","last_peered":"2024-03-22T01:36:38.307927+0000","last_clean":"2024-03-22T01:36:38.307927+0000","last_became_active":"2024-03-22T01:34:15.488134+0000","last_became_peered":"2024-03-22T01:34:15.488134+0000","last_unstale":"2024-03-22T01:36:38.307927+0000","last_undegraded":"2024-03-22T01:36:38.307927+0000","last_fullsized":"2024-03-22T01:36:38.307927+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:50:41.206135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":24,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.315310+0000","last_change":"2024-03-22T01:34:19.528744+0000","last_active":"2024-03-22T01:36:38.315310+0000","last_peered":"2024-03-22T01:36:38.315310+0000","last_clean":"2024-03-22T01:36:38.315310+0000","last_became_active":"2024-03-22T01:34:19.528375+0000","last_became_peered":"2024-03-22T01:34:19.528375+0000","last_unstale":"2024-03-22T01:36:38.315310+0000","last_undegraded":"2024-03-22T01:36:38.315310+0000","last_fullsized":"2024-03-22T01:36:38.315310+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:47:53.753923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"86'2","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.191536+0000","last_change":"2024-03-22T01:34:14.463289+0000","last_active":"2024-03-22T01:36:38.191536+0000","last_peered":"2024-03-22T01:36:38.191536+0000","last_clean":"2024-03-22T01:36:38.191536+0000","last_became_active":"2024-03-22T01:34:14.462883+0000","last_became_peered":"2024-03-22T01:34:14.462883+0000","last_unstale":"2024-03-22T01:36:38.191536+0000","last_undegraded":"2024-03-22T01:36:38.191536+0000","last_fullsized":"2024-03-22T01:36:38.191536+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:06:14.452674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"93'9","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.188986+0000","last_change":"2024-03-22T01:34:15.484785+0000","last_active":"2024-03-22T01:36:38.188986+0000","last_peered":"2024-03-22T01:36:38.188986+0000","last_clean":"2024-03-22T01:36:38.188986+0000","last_became_active":"2024-03-22T01:34:15.484474+0000","last_became_peered":"2024-03-22T01:34:15.484474+0000","last_unstale":"2024-03-22T01:36:38.188986+0000","last_undegraded":"2024-03-22T01:36:38.188986+0000","last_fullsized":"2024-03-22T01:36:38.188986+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:11:13.338836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":24,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500472+0000","last_change":"2024-03-22T01:34:19.527379+0000","last_active":"2024-03-22T01:36:38.500472+0000","last_peered":"2024-03-22T01:36:38.500472+0000","last_clean":"2024-03-22T01:36:38.500472+0000","last_became_active":"2024-03-22T01:34:19.526957+0000","last_became_peered":"2024-03-22T01:34:19.526957+0000","last_unstale":"2024-03-22T01:36:38.500472+0000","last_undegraded":"2024-03-22T01:36:38.500472+0000","last_fullsized":"2024-03-22T01:36:38.500472+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:03:09.538334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.314794+0000","last_change":"2024-03-22T01:34:14.467372+0000","last_active":"2024-03-22T01:36:38.314794+0000","last_peered":"2024-03-22T01:36:38.314794+0000","last_clean":"2024-03-22T01:36:38.314794+0000","last_became_active":"2024-03-22T01:34:14.466124+0000","last_became_peered":"2024-03-22T01:34:14.466124+0000","last_unstale":"2024-03-22T01:36:38.314794+0000","last_undegraded":"2024-03-22T01:36:38.314794+0000","last_fullsized":"2024-03-22T01:36:38.314794+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:15:41.804735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"93'15","reported_seq":46,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.300521+0000","last_change":"2024-03-22T01:34:15.491533+0000","last_active":"2024-03-22T01:36:38.300521+0000","last_peered":"2024-03-22T01:36:38.300521+0000","last_clean":"2024-03-22T01:36:38.300521+0000","last_became_active":"2024-03-22T01:34:15.491208+0000","last_became_peered":"2024-03-22T01:34:15.491208+0000","last_unstale":"2024-03-22T01:36:38.300521+0000","last_undegraded":"2024-03-22T01:36:38.300521+0000","last_fullsized":"2024-03-22T01:36:38.300521+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:03:07.800607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":24,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.306930+0000","last_change":"2024-03-22T01:34:19.533419+0000","last_active":"2024-03-22T01:36:38.306930+0000","last_peered":"2024-03-22T01:36:38.306930+0000","last_clean":"2024-03-22T01:36:38.306930+0000","last_became_active":"2024-03-22T01:34:19.530042+0000","last_became_peered":"2024-03-22T01:34:19.530042+0000","last_unstale":"2024-03-22T01:36:38.306930+0000","last_undegraded":"2024-03-22T01:36:38.306930+0000","last_fullsized":"2024-03-22T01:36:38.306930+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:46:59.859742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.499888+0000","last_change":"2024-03-22T01:34:14.467033+0000","last_active":"2024-03-22T01:36:38.499888+0000","last_peered":"2024-03-22T01:36:38.499888+0000","last_clean":"2024-03-22T01:36:38.499888+0000","last_became_active":"2024-03-22T01:34:14.466649+0000","last_became_peered":"2024-03-22T01:34:14.466649+0000","last_unstale":"2024-03-22T01:36:38.499888+0000","last_undegraded":"2024-03-22T01:36:38.499888+0000","last_fullsized":"2024-03-22T01:36:38.499888+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:38:44.890899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"93'6","reported_seq":43,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.300199+0000","last_change":"2024-03-22T01:35:16.815302+0000","last_active":"2024-03-22T01:36:38.300199+0000","last_peered":"2024-03-22T01:36:38.300199+0000","last_clean":"2024-03-22T01:36:38.300199+0000","last_became_active":"2024-03-22T01:35:16.814988+0000","last_became_peered":"2024-03-22T01:35:16.814988+0000","last_unstale":"2024-03-22T01:36:38.300199+0000","last_undegraded":"2024-03-22T01:36:38.300199+0000","last_fullsized":"2024-03-22T01:36:38.300199+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:54:47.793437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":24,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.189238+0000","last_change":"2024-03-22T01:34:19.531417+0000","last_active":"2024-03-22T01:36:38.189238+0000","last_peered":"2024-03-22T01:36:38.189238+0000","last_clean":"2024-03-22T01:36:38.189238+0000","last_became_active":"2024-03-22T01:34:19.531003+0000","last_became_peered":"2024-03-22T01:34:19.531003+0000","last_unstale":"2024-03-22T01:36:38.189238+0000","last_undegraded":"2024-03-22T01:36:38.189238+0000","last_fullsized":"2024-03-22T01:36:38.189238+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:33:52.865110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"86'8","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.307580+0000","last_change":"2024-03-22T01:34:14.472904+0000","last_active":"2024-03-22T01:36:38.307580+0000","last_peered":"2024-03-22T01:36:38.307580+0000","last_clean":"2024-03-22T01:36:38.307580+0000","last_became_active":"2024-03-22T01:34:14.472442+0000","last_became_peered":"2024-03-22T01:34:14.472442+0000","last_unstale":"2024-03-22T01:36:38.307580+0000","last_undegraded":"2024-03-22T01:36:38.307580+0000","last_fullsized":"2024-03-22T01:36:38.307580+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:57:45.478316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"93'6","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.314972+0000","last_change":"2024-03-22T01:34:15.477411+0000","last_active":"2024-03-22T01:36:38.314972+0000","last_peered":"2024-03-22T01:36:38.314972+0000","last_clean":"2024-03-22T01:36:38.314972+0000","last_became_active":"2024-03-22T01:34:15.477035+0000","last_became_peered":"2024-03-22T01:34:15.477035+0000","last_unstale":"2024-03-22T01:36:38.314972+0000","last_undegraded":"2024-03-22T01:36:38.314972+0000","last_fullsized":"2024-03-22T01:36:38.314972+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:33:13.410872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.314593+0000","last_change":"2024-03-22T01:34:19.528258+0000","last_active":"2024-03-22T01:36:38.314593+0000","last_peered":"2024-03-22T01:36:38.314593+0000","last_clean":"2024-03-22T01:36:38.314593+0000","last_became_active":"2024-03-22T01:34:19.527668+0000","last_became_peered":"2024-03-22T01:34:19.527668+0000","last_unstale":"2024-03-22T01:36:38.314593+0000","last_undegraded":"2024-03-22T01:36:38.314593+0000","last_fullsized":"2024-03-22T01:36:38.314593+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:34:19.162609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'1","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.308046+0000","last_change":"2024-03-22T01:34:14.464735+0000","last_active":"2024-03-22T01:36:38.308046+0000","last_peered":"2024-03-22T01:36:38.308046+0000","last_clean":"2024-03-22T01:36:38.308046+0000","last_became_active":"2024-03-22T01:34:14.464359+0000","last_became_peered":"2024-03-22T01:34:14.464359+0000","last_unstale":"2024-03-22T01:36:38.308046+0000","last_undegraded":"2024-03-22T01:36:38.308046+0000","last_fullsized":"2024-03-22T01:36:38.308046+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:21:18.354922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"93'5","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500679+0000","last_change":"2024-03-22T01:35:16.752884+0000","last_active":"2024-03-22T01:36:38.500679+0000","last_peered":"2024-03-22T01:36:38.500679+0000","last_clean":"2024-03-22T01:36:38.500679+0000","last_became_active":"2024-03-22T01:35:16.751355+0000","last_became_peered":"2024-03-22T01:35:16.751355+0000","last_unstale":"2024-03-22T01:36:38.500679+0000","last_undegraded":"2024-03-22T01:36:38.500679+0000","last_fullsized":"2024-03-22T01:36:38.500679+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:49:33.261481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":25,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.307644+0000","last_change":"2024-03-22T01:34:19.668136+0000","last_active":"2024-03-22T01:36:38.307644+0000","last_peered":"2024-03-22T01:36:38.307644+0000","last_clean":"2024-03-22T01:36:38.307644+0000","last_became_active":"2024-03-22T01:34:19.667633+0000","last_became_peered":"2024-03-22T01:34:19.667633+0000","last_unstale":"2024-03-22T01:36:38.307644+0000","last_undegraded":"2024-03-22T01:36:38.307644+0000","last_fullsized":"2024-03-22T01:36:38.307644+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:07:26.455373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.305067+0000","last_change":"2024-03-22T01:34:14.471959+0000","last_active":"2024-03-22T01:36:38.305067+0000","last_peered":"2024-03-22T01:36:38.305067+0000","last_clean":"2024-03-22T01:36:38.305067+0000","last_became_active":"2024-03-22T01:34:14.471618+0000","last_became_peered":"2024-03-22T01:34:14.471618+0000","last_unstale":"2024-03-22T01:36:38.305067+0000","last_undegraded":"2024-03-22T01:36:38.305067+0000","last_fullsized":"2024-03-22T01:36:38.305067+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:45:03.537013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"93'4","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.192471+0000","last_change":"2024-03-22T01:34:15.495770+0000","last_active":"2024-03-22T01:36:38.192471+0000","last_peered":"2024-03-22T01:36:38.192471+0000","last_clean":"2024-03-22T01:36:38.192471+0000","last_became_active":"2024-03-22T01:34:15.495403+0000","last_became_peered":"2024-03-22T01:34:15.495403+0000","last_unstale":"2024-03-22T01:36:38.192471+0000","last_undegraded":"2024-03-22T01:36:38.192471+0000","last_fullsized":"2024-03-22T01:36:38.192471+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:48:05.184305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":25,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.189283+0000","last_change":"2024-03-22T01:34:19.520204+0000","last_active":"2024-03-22T01:36:38.189283+0000","last_peered":"2024-03-22T01:36:38.189283+0000","last_clean":"2024-03-22T01:36:38.189283+0000","last_became_active":"2024-03-22T01:34:19.519847+0000","last_became_peered":"2024-03-22T01:34:19.519847+0000","last_unstale":"2024-03-22T01:36:38.189283+0000","last_undegraded":"2024-03-22T01:36:38.189283+0000","last_fullsized":"2024-03-22T01:36:38.189283+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:06:21.389636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.300364+0000","last_change":"2024-03-22T01:34:14.464162+0000","last_active":"2024-03-22T01:36:38.300364+0000","last_peered":"2024-03-22T01:36:38.300364+0000","last_clean":"2024-03-22T01:36:38.300364+0000","last_became_active":"2024-03-22T01:34:14.462700+0000","last_became_peered":"2024-03-22T01:34:14.462700+0000","last_unstale":"2024-03-22T01:36:38.300364+0000","last_undegraded":"2024-03-22T01:36:38.300364+0000","last_fullsized":"2024-03-22T01:36:38.300364+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:45:32.262466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"93'7","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.192324+0000","last_change":"2024-03-22T01:34:15.488256+0000","last_active":"2024-03-22T01:36:38.192324+0000","last_peered":"2024-03-22T01:36:38.192324+0000","last_clean":"2024-03-22T01:36:38.192324+0000","last_became_active":"2024-03-22T01:34:15.487939+0000","last_became_peered":"2024-03-22T01:34:15.487939+0000","last_unstale":"2024-03-22T01:36:38.192324+0000","last_undegraded":"2024-03-22T01:36:38.192324+0000","last_fullsized":"2024-03-22T01:36:38.192324+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:30:49.323531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.189196+0000","last_change":"2024-03-22T01:34:19.667633+0000","last_active":"2024-03-22T01:36:38.189196+0000","last_peered":"2024-03-22T01:36:38.189196+0000","last_clean":"2024-03-22T01:36:38.189196+0000","last_became_active":"2024-03-22T01:34:19.667199+0000","last_became_peered":"2024-03-22T01:34:19.667199+0000","last_unstale":"2024-03-22T01:36:38.189196+0000","last_undegraded":"2024-03-22T01:36:38.189196+0000","last_fullsized":"2024-03-22T01:36:38.189196+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T01:58:13.459475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"93'8","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.314599+0000","last_change":"2024-03-22T01:34:15.489468+0000","last_active":"2024-03-22T01:36:38.314599+0000","last_peered":"2024-03-22T01:36:38.314599+0000","last_clean":"2024-03-22T01:36:38.314599+0000","last_became_active":"2024-03-22T01:34:15.489029+0000","last_became_peered":"2024-03-22T01:34:15.489029+0000","last_unstale":"2024-03-22T01:36:38.314599+0000","last_undegraded":"2024-03-22T01:36:38.314599+0000","last_fullsized":"2024-03-22T01:36:38.314599+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:00:01.370161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.188685+0000","last_change":"2024-03-22T01:34:14.464552+0000","last_active":"2024-03-22T01:36:38.188685+0000","last_peered":"2024-03-22T01:36:38.188685+0000","last_clean":"2024-03-22T01:36:38.188685+0000","last_became_active":"2024-03-22T01:34:14.463554+0000","last_became_peered":"2024-03-22T01:34:14.463554+0000","last_unstale":"2024-03-22T01:36:38.188685+0000","last_undegraded":"2024-03-22T01:36:38.188685+0000","last_fullsized":"2024-03-22T01:36:38.188685+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:51:16.709278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"91'1","reported_seq":26,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.300450+0000","last_change":"2024-03-22T01:34:19.526066+0000","last_active":"2024-03-22T01:36:38.300450+0000","last_peered":"2024-03-22T01:36:38.300450+0000","last_clean":"2024-03-22T01:36:38.300450+0000","last_became_active":"2024-03-22T01:34:19.525659+0000","last_became_peered":"2024-03-22T01:34:19.525659+0000","last_unstale":"2024-03-22T01:36:38.300450+0000","last_undegraded":"2024-03-22T01:36:38.300450+0000","last_fullsized":"2024-03-22T01:36:38.300450+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:06:26.287971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"93'8","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.307732+0000","last_change":"2024-03-22T01:34:15.477018+0000","last_active":"2024-03-22T01:36:38.307732+0000","last_peered":"2024-03-22T01:36:38.307732+0000","last_clean":"2024-03-22T01:36:38.307732+0000","last_became_active":"2024-03-22T01:34:15.476613+0000","last_became_peered":"2024-03-22T01:34:15.476613+0000","last_unstale":"2024-03-22T01:36:38.307732+0000","last_undegraded":"2024-03-22T01:36:38.307732+0000","last_fullsized":"2024-03-22T01:36:38.307732+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:10:07.980123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.499827+0000","last_change":"2024-03-22T01:34:14.470922+0000","last_active":"2024-03-22T01:36:38.499827+0000","last_peered":"2024-03-22T01:36:38.499827+0000","last_clean":"2024-03-22T01:36:38.499827+0000","last_became_active":"2024-03-22T01:34:14.470515+0000","last_became_peered":"2024-03-22T01:34:14.470515+0000","last_unstale":"2024-03-22T01:36:38.499827+0000","last_undegraded":"2024-03-22T01:36:38.499827+0000","last_fullsized":"2024-03-22T01:36:38.499827+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:52:50.203631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":365,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.300114+0000","last_change":"2024-03-22T01:33:03.615234+0000","last_active":"2024-03-22T01:36:38.300114+0000","last_peered":"2024-03-22T01:36:38.300114+0000","last_clean":"2024-03-22T01:36:38.300114+0000","last_became_active":"2024-03-22T01:33:03.614710+0000","last_became_peered":"2024-03-22T01:33:03.614710+0000","last_unstale":"2024-03-22T01:36:38.300114+0000","last_undegraded":"2024-03-22T01:36:38.300114+0000","last_fullsized":"2024-03-22T01:36:38.300114+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:29:38.312292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:29:38.312292+0000","last_clean_scrub_stamp":"2024-03-22T01:29:38.312292+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:56:42.348119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.314929+0000","last_change":"2024-03-22T01:34:19.668565+0000","last_active":"2024-03-22T01:36:38.314929+0000","last_peered":"2024-03-22T01:36:38.314929+0000","last_clean":"2024-03-22T01:36:38.314929+0000","last_became_active":"2024-03-22T01:34:19.667959+0000","last_became_peered":"2024-03-22T01:34:19.667959+0000","last_unstale":"2024-03-22T01:36:38.314929+0000","last_undegraded":"2024-03-22T01:36:38.314929+0000","last_fullsized":"2024-03-22T01:36:38.314929+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:13:04.447486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"93'5","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.188685+0000","last_change":"2024-03-22T01:34:15.489513+0000","last_active":"2024-03-22T01:36:38.188685+0000","last_peered":"2024-03-22T01:36:38.188685+0000","last_clean":"2024-03-22T01:36:38.188685+0000","last_became_active":"2024-03-22T01:34:15.489188+0000","last_became_peered":"2024-03-22T01:34:15.489188+0000","last_unstale":"2024-03-22T01:36:38.188685+0000","last_undegraded":"2024-03-22T01:36:38.188685+0000","last_fullsized":"2024-03-22T01:36:38.188685+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:08:24.859350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.300274+0000","last_change":"2024-03-22T01:34:14.466305+0000","last_active":"2024-03-22T01:36:38.300274+0000","last_peered":"2024-03-22T01:36:38.300274+0000","last_clean":"2024-03-22T01:36:38.300274+0000","last_became_active":"2024-03-22T01:34:14.465495+0000","last_became_peered":"2024-03-22T01:34:14.465495+0000","last_unstale":"2024-03-22T01:36:38.300274+0000","last_undegraded":"2024-03-22T01:36:38.300274+0000","last_fullsized":"2024-03-22T01:36:38.300274+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:45:13.481187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500369+0000","last_change":"2024-03-22T01:34:19.526970+0000","last_active":"2024-03-22T01:36:38.500369+0000","last_peered":"2024-03-22T01:36:38.500369+0000","last_clean":"2024-03-22T01:36:38.500369+0000","last_became_active":"2024-03-22T01:34:19.526462+0000","last_became_peered":"2024-03-22T01:34:19.526462+0000","last_unstale":"2024-03-22T01:36:38.500369+0000","last_undegraded":"2024-03-22T01:36:38.500369+0000","last_fullsized":"2024-03-22T01:36:38.500369+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:04:18.417358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500521+0000","last_change":"2024-03-22T01:35:16.750746+0000","last_active":"2024-03-22T01:36:38.500521+0000","last_peered":"2024-03-22T01:36:38.500521+0000","last_clean":"2024-03-22T01:36:38.500521+0000","last_became_active":"2024-03-22T01:35:16.750271+0000","last_became_peered":"2024-03-22T01:35:16.750271+0000","last_unstale":"2024-03-22T01:36:38.500521+0000","last_undegraded":"2024-03-22T01:36:38.500521+0000","last_fullsized":"2024-03-22T01:36:38.500521+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:18:10.839351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"93'9","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.189402+0000","last_change":"2024-03-22T01:34:15.490405+0000","last_active":"2024-03-22T01:36:38.189402+0000","last_peered":"2024-03-22T01:36:38.189402+0000","last_clean":"2024-03-22T01:36:38.189402+0000","last_became_active":"2024-03-22T01:34:15.489921+0000","last_became_peered":"2024-03-22T01:34:15.489921+0000","last_unstale":"2024-03-22T01:36:38.189402+0000","last_undegraded":"2024-03-22T01:36:38.189402+0000","last_fullsized":"2024-03-22T01:36:38.189402+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:27:28.804649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.307314+0000","last_change":"2024-03-22T01:34:19.667651+0000","last_active":"2024-03-22T01:36:38.307314+0000","last_peered":"2024-03-22T01:36:38.307314+0000","last_clean":"2024-03-22T01:36:38.307314+0000","last_became_active":"2024-03-22T01:34:19.667197+0000","last_became_peered":"2024-03-22T01:34:19.667197+0000","last_unstale":"2024-03-22T01:36:38.307314+0000","last_undegraded":"2024-03-22T01:36:38.307314+0000","last_fullsized":"2024-03-22T01:36:38.307314+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:32:55.874098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"96'43","reported_seq":100,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.424707+0000","last_change":"2024-03-22T01:34:14.474988+0000","last_active":"2024-03-22T01:36:38.424707+0000","last_peered":"2024-03-22T01:36:38.424707+0000","last_clean":"2024-03-22T01:36:38.424707+0000","last_became_active":"2024-03-22T01:34:14.474651+0000","last_became_peered":"2024-03-22T01:34:14.474651+0000","last_unstale":"2024-03-22T01:36:38.424707+0000","last_undegraded":"2024-03-22T01:36:38.424707+0000","last_fullsized":"2024-03-22T01:36:38.424707+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:14:59.846727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":77,"num_write_kb":837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"93'5","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.305472+0000","last_change":"2024-03-22T01:35:16.749643+0000","last_active":"2024-03-22T01:36:38.305472+0000","last_peered":"2024-03-22T01:36:38.305472+0000","last_clean":"2024-03-22T01:36:38.305472+0000","last_became_active":"2024-03-22T01:35:16.749311+0000","last_became_peered":"2024-03-22T01:35:16.749311+0000","last_unstale":"2024-03-22T01:36:38.305472+0000","last_undegraded":"2024-03-22T01:36:38.305472+0000","last_fullsized":"2024-03-22T01:36:38.305472+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:23:18.666007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,7],"acting":[9,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.190868+0000","last_change":"2024-03-22T01:34:19.523959+0000","last_active":"2024-03-22T01:36:38.190868+0000","last_peered":"2024-03-22T01:36:38.190868+0000","last_clean":"2024-03-22T01:36:38.190868+0000","last_became_active":"2024-03-22T01:34:19.523556+0000","last_became_peered":"2024-03-22T01:34:19.523556+0000","last_unstale":"2024-03-22T01:36:38.190868+0000","last_undegraded":"2024-03-22T01:36:38.190868+0000","last_fullsized":"2024-03-22T01:36:38.190868+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:22:21.623042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"86'2","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.499880+0000","last_change":"2024-03-22T01:34:14.463434+0000","last_active":"2024-03-22T01:36:38.499880+0000","last_peered":"2024-03-22T01:36:38.499880+0000","last_clean":"2024-03-22T01:36:38.499880+0000","last_became_active":"2024-03-22T01:34:14.463075+0000","last_became_peered":"2024-03-22T01:34:14.463075+0000","last_unstale":"2024-03-22T01:36:38.499880+0000","last_undegraded":"2024-03-22T01:36:38.499880+0000","last_fullsized":"2024-03-22T01:36:38.499880+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:43:18.223766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"93'7","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.502109+0000","last_change":"2024-03-22T01:34:15.482695+0000","last_active":"2024-03-22T01:36:38.502109+0000","last_peered":"2024-03-22T01:36:38.502109+0000","last_clean":"2024-03-22T01:36:38.502109+0000","last_became_active":"2024-03-22T01:34:15.481881+0000","last_became_peered":"2024-03-22T01:34:15.481881+0000","last_unstale":"2024-03-22T01:36:38.502109+0000","last_undegraded":"2024-03-22T01:36:38.502109+0000","last_fullsized":"2024-03-22T01:36:38.502109+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:44:26.290714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":25,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500458+0000","last_change":"2024-03-22T01:34:19.668252+0000","last_active":"2024-03-22T01:36:38.500458+0000","last_peered":"2024-03-22T01:36:38.500458+0000","last_clean":"2024-03-22T01:36:38.500458+0000","last_became_active":"2024-03-22T01:34:19.667830+0000","last_became_peered":"2024-03-22T01:34:19.667830+0000","last_unstale":"2024-03-22T01:36:38.500458+0000","last_undegraded":"2024-03-22T01:36:38.500458+0000","last_fullsized":"2024-03-22T01:36:38.500458+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:08:29.602899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.459520+0000","last_change":"2024-03-22T01:34:14.473020+0000","last_active":"2024-03-22T01:36:38.459520+0000","last_peered":"2024-03-22T01:36:38.459520+0000","last_clean":"2024-03-22T01:36:38.459520+0000","last_became_active":"2024-03-22T01:34:14.472689+0000","last_became_peered":"2024-03-22T01:34:14.472689+0000","last_unstale":"2024-03-22T01:36:38.459520+0000","last_undegraded":"2024-03-22T01:36:38.459520+0000","last_fullsized":"2024-03-22T01:36:38.459520+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:04:19.724685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"93'2","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.307503+0000","last_change":"2024-03-22T01:34:15.491613+0000","last_active":"2024-03-22T01:36:38.307503+0000","last_peered":"2024-03-22T01:36:38.307503+0000","last_clean":"2024-03-22T01:36:38.307503+0000","last_became_active":"2024-03-22T01:34:15.490922+0000","last_became_peered":"2024-03-22T01:34:15.490922+0000","last_unstale":"2024-03-22T01:36:38.307503+0000","last_undegraded":"2024-03-22T01:36:38.307503+0000","last_fullsized":"2024-03-22T01:36:38.307503+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:09:33.177646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":25,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.306867+0000","last_change":"2024-03-22T01:34:19.534537+0000","last_active":"2024-03-22T01:36:38.306867+0000","last_peered":"2024-03-22T01:36:38.306867+0000","last_clean":"2024-03-22T01:36:38.306867+0000","last_became_active":"2024-03-22T01:34:19.534038+0000","last_became_peered":"2024-03-22T01:34:19.534038+0000","last_unstale":"2024-03-22T01:36:38.306867+0000","last_undegraded":"2024-03-22T01:36:38.306867+0000","last_fullsized":"2024-03-22T01:36:38.306867+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:59:26.020000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.307492+0000","last_change":"2024-03-22T01:34:14.464724+0000","last_active":"2024-03-22T01:36:38.307492+0000","last_peered":"2024-03-22T01:36:38.307492+0000","last_clean":"2024-03-22T01:36:38.307492+0000","last_became_active":"2024-03-22T01:34:14.464056+0000","last_became_peered":"2024-03-22T01:34:14.464056+0000","last_unstale":"2024-03-22T01:36:38.307492+0000","last_undegraded":"2024-03-22T01:36:38.307492+0000","last_fullsized":"2024-03-22T01:36:38.307492+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:49:39.098126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"93'9","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500261+0000","last_change":"2024-03-22T01:34:15.488934+0000","last_active":"2024-03-22T01:36:38.500261+0000","last_peered":"2024-03-22T01:36:38.500261+0000","last_clean":"2024-03-22T01:36:38.500261+0000","last_became_active":"2024-03-22T01:34:15.488585+0000","last_became_peered":"2024-03-22T01:34:15.488585+0000","last_unstale":"2024-03-22T01:36:38.500261+0000","last_undegraded":"2024-03-22T01:36:38.500261+0000","last_fullsized":"2024-03-22T01:36:38.500261+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:59:04.554614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"91'1","reported_seq":26,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500321+0000","last_change":"2024-03-22T01:34:19.534096+0000","last_active":"2024-03-22T01:36:38.500321+0000","last_peered":"2024-03-22T01:36:38.500321+0000","last_clean":"2024-03-22T01:36:38.500321+0000","last_became_active":"2024-03-22T01:34:19.533645+0000","last_became_peered":"2024-03-22T01:34:19.533645+0000","last_unstale":"2024-03-22T01:36:38.500321+0000","last_undegraded":"2024-03-22T01:36:38.500321+0000","last_fullsized":"2024-03-22T01:36:38.500321+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:50:21.259146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.190987+0000","last_change":"2024-03-22T01:34:14.469529+0000","last_active":"2024-03-22T01:36:38.190987+0000","last_peered":"2024-03-22T01:36:38.190987+0000","last_clean":"2024-03-22T01:36:38.190987+0000","last_became_active":"2024-03-22T01:34:14.469204+0000","last_became_peered":"2024-03-22T01:34:14.469204+0000","last_unstale":"2024-03-22T01:36:38.190987+0000","last_undegraded":"2024-03-22T01:36:38.190987+0000","last_fullsized":"2024-03-22T01:36:38.190987+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:39:57.048537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"93'8","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.307968+0000","last_change":"2024-03-22T01:34:15.501324+0000","last_active":"2024-03-22T01:36:38.307968+0000","last_peered":"2024-03-22T01:36:38.307968+0000","last_clean":"2024-03-22T01:36:38.307968+0000","last_became_active":"2024-03-22T01:34:15.501015+0000","last_became_peered":"2024-03-22T01:34:15.501015+0000","last_unstale":"2024-03-22T01:36:38.307968+0000","last_undegraded":"2024-03-22T01:36:38.307968+0000","last_fullsized":"2024-03-22T01:36:38.307968+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:35:27.673019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500540+0000","last_change":"2024-03-22T01:34:19.667806+0000","last_active":"2024-03-22T01:36:38.500540+0000","last_peered":"2024-03-22T01:36:38.500540+0000","last_clean":"2024-03-22T01:36:38.500540+0000","last_became_active":"2024-03-22T01:34:19.667249+0000","last_became_peered":"2024-03-22T01:34:19.667249+0000","last_unstale":"2024-03-22T01:36:38.500540+0000","last_undegraded":"2024-03-22T01:36:38.500540+0000","last_fullsized":"2024-03-22T01:36:38.500540+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:44:22.476882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"86'2","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.307691+0000","last_change":"2024-03-22T01:34:14.471800+0000","last_active":"2024-03-22T01:36:38.307691+0000","last_peered":"2024-03-22T01:36:38.307691+0000","last_clean":"2024-03-22T01:36:38.307691+0000","last_became_active":"2024-03-22T01:34:14.471422+0000","last_became_peered":"2024-03-22T01:34:14.471422+0000","last_unstale":"2024-03-22T01:36:38.307691+0000","last_undegraded":"2024-03-22T01:36:38.307691+0000","last_fullsized":"2024-03-22T01:36:38.307691+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:25:58.546331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"93'5","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.300283+0000","last_change":"2024-03-22T01:34:15.487348+0000","last_active":"2024-03-22T01:36:38.300283+0000","last_peered":"2024-03-22T01:36:38.300283+0000","last_clean":"2024-03-22T01:36:38.300283+0000","last_became_active":"2024-03-22T01:34:15.486788+0000","last_became_peered":"2024-03-22T01:34:15.486788+0000","last_unstale":"2024-03-22T01:36:38.300283+0000","last_undegraded":"2024-03-22T01:36:38.300283+0000","last_fullsized":"2024-03-22T01:36:38.300283+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:10:31.489021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.459783+0000","last_change":"2024-03-22T01:34:19.521490+0000","last_active":"2024-03-22T01:36:38.459783+0000","last_peered":"2024-03-22T01:36:38.459783+0000","last_clean":"2024-03-22T01:36:38.459783+0000","last_became_active":"2024-03-22T01:34:19.521047+0000","last_became_peered":"2024-03-22T01:34:19.521047+0000","last_unstale":"2024-03-22T01:36:38.459783+0000","last_undegraded":"2024-03-22T01:36:38.459783+0000","last_fullsized":"2024-03-22T01:36:38.459783+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:36:50.176756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":17,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.459782+0000","last_change":"2024-03-22T01:35:16.749731+0000","last_active":"2024-03-22T01:36:38.459782+0000","last_peered":"2024-03-22T01:36:38.459782+0000","last_clean":"2024-03-22T01:36:38.459782+0000","last_became_active":"2024-03-22T01:35:16.749329+0000","last_became_peered":"2024-03-22T01:35:16.749329+0000","last_unstale":"2024-03-22T01:36:38.459782+0000","last_undegraded":"2024-03-22T01:36:38.459782+0000","last_fullsized":"2024-03-22T01:36:38.459782+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:23:27.386061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"93'7","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.300670+0000","last_change":"2024-03-22T01:34:15.491683+0000","last_active":"2024-03-22T01:36:38.300670+0000","last_peered":"2024-03-22T01:36:38.300670+0000","last_clean":"2024-03-22T01:36:38.300670+0000","last_became_active":"2024-03-22T01:34:15.491030+0000","last_became_peered":"2024-03-22T01:34:15.491030+0000","last_unstale":"2024-03-22T01:36:38.300670+0000","last_undegraded":"2024-03-22T01:36:38.300670+0000","last_fullsized":"2024-03-22T01:36:38.300670+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:02:10.115971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500471+0000","last_change":"2024-03-22T01:36:17.767646+0000","last_active":"2024-03-22T01:36:38.500471+0000","last_peered":"2024-03-22T01:36:38.500471+0000","last_clean":"2024-03-22T01:36:38.500471+0000","last_became_active":"2024-03-22T01:36:17.767257+0000","last_became_peered":"2024-03-22T01:36:17.767257+0000","last_unstale":"2024-03-22T01:36:38.500471+0000","last_undegraded":"2024-03-22T01:36:38.500471+0000","last_fullsized":"2024-03-22T01:36:38.500471+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:16:11.523941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,1,4],"acting":[0,8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"96'39","reported_seq":79,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.419218+0000","last_change":"2024-03-22T01:34:14.467755+0000","last_active":"2024-03-22T01:36:38.419218+0000","last_peered":"2024-03-22T01:36:38.419218+0000","last_clean":"2024-03-22T01:36:38.419218+0000","last_became_active":"2024-03-22T01:34:14.467426+0000","last_became_peered":"2024-03-22T01:34:14.467426+0000","last_unstale":"2024-03-22T01:36:38.419218+0000","last_undegraded":"2024-03-22T01:36:38.419218+0000","last_fullsized":"2024-03-22T01:36:38.419218+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:16:35.436872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"93'6","reported_seq":43,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.300373+0000","last_change":"2024-03-22T01:35:16.748667+0000","last_active":"2024-03-22T01:36:38.300373+0000","last_peered":"2024-03-22T01:36:38.300373+0000","last_clean":"2024-03-22T01:36:38.300373+0000","last_became_active":"2024-03-22T01:35:16.748303+0000","last_became_peered":"2024-03-22T01:35:16.748303+0000","last_unstale":"2024-03-22T01:36:38.300373+0000","last_undegraded":"2024-03-22T01:36:38.300373+0000","last_fullsized":"2024-03-22T01:36:38.300373+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:40:55.207127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.314390+0000","last_change":"2024-03-22T01:34:19.523631+0000","last_active":"2024-03-22T01:36:38.314390+0000","last_peered":"2024-03-22T01:36:38.314390+0000","last_clean":"2024-03-22T01:36:38.314390+0000","last_became_active":"2024-03-22T01:34:19.522968+0000","last_became_peered":"2024-03-22T01:34:19.522968+0000","last_unstale":"2024-03-22T01:36:38.314390+0000","last_undegraded":"2024-03-22T01:36:38.314390+0000","last_fullsized":"2024-03-22T01:36:38.314390+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:27:24.182886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.189345+0000","last_change":"2024-03-22T01:34:14.464113+0000","last_active":"2024-03-22T01:36:38.189345+0000","last_peered":"2024-03-22T01:36:38.189345+0000","last_clean":"2024-03-22T01:36:38.189345+0000","last_became_active":"2024-03-22T01:34:14.463389+0000","last_became_peered":"2024-03-22T01:34:14.463389+0000","last_unstale":"2024-03-22T01:36:38.189345+0000","last_undegraded":"2024-03-22T01:36:38.189345+0000","last_fullsized":"2024-03-22T01:36:38.189345+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:08:42.376753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"93'9","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.300652+0000","last_change":"2024-03-22T01:34:15.492941+0000","last_active":"2024-03-22T01:36:38.300652+0000","last_peered":"2024-03-22T01:36:38.300652+0000","last_clean":"2024-03-22T01:36:38.300652+0000","last_became_active":"2024-03-22T01:34:15.492598+0000","last_became_peered":"2024-03-22T01:34:15.492598+0000","last_unstale":"2024-03-22T01:36:38.300652+0000","last_undegraded":"2024-03-22T01:36:38.300652+0000","last_fullsized":"2024-03-22T01:36:38.300652+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:22:35.502404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"93'9","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500326+0000","last_change":"2024-03-22T01:34:15.494548+0000","last_active":"2024-03-22T01:36:38.500326+0000","last_peered":"2024-03-22T01:36:38.500326+0000","last_clean":"2024-03-22T01:36:38.500326+0000","last_became_active":"2024-03-22T01:34:15.494228+0000","last_became_peered":"2024-03-22T01:34:15.494228+0000","last_unstale":"2024-03-22T01:36:38.500326+0000","last_undegraded":"2024-03-22T01:36:38.500326+0000","last_fullsized":"2024-03-22T01:36:38.500326+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:43:26.011030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"93'12","reported_seq":44,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.191036+0000","last_change":"2024-03-22T01:34:15.490843+0000","last_active":"2024-03-22T01:36:38.191036+0000","last_peered":"2024-03-22T01:36:38.191036+0000","last_clean":"2024-03-22T01:36:38.191036+0000","last_became_active":"2024-03-22T01:34:15.490474+0000","last_became_peered":"2024-03-22T01:34:15.490474+0000","last_unstale":"2024-03-22T01:36:38.191036+0000","last_undegraded":"2024-03-22T01:36:38.191036+0000","last_fullsized":"2024-03-22T01:36:38.191036+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:11:31.488132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"93'8","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.190920+0000","last_change":"2024-03-22T01:34:15.502213+0000","last_active":"2024-03-22T01:36:38.190920+0000","last_peered":"2024-03-22T01:36:38.190920+0000","last_clean":"2024-03-22T01:36:38.190920+0000","last_became_active":"2024-03-22T01:34:15.501464+0000","last_became_peered":"2024-03-22T01:34:15.501464+0000","last_unstale":"2024-03-22T01:36:38.190920+0000","last_undegraded":"2024-03-22T01:36:38.190920+0000","last_fullsized":"2024-03-22T01:36:38.190920+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:38:02.090365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"93'7","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.501571+0000","last_change":"2024-03-22T01:34:15.481282+0000","last_active":"2024-03-22T01:36:38.501571+0000","last_peered":"2024-03-22T01:36:38.501571+0000","last_clean":"2024-03-22T01:36:38.501571+0000","last_became_active":"2024-03-22T01:34:15.480465+0000","last_became_peered":"2024-03-22T01:34:15.480465+0000","last_unstale":"2024-03-22T01:36:38.501571+0000","last_undegraded":"2024-03-22T01:36:38.501571+0000","last_fullsized":"2024-03-22T01:36:38.501571+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:55:59.398543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"93'4","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.189395+0000","last_change":"2024-03-22T01:34:15.480910+0000","last_active":"2024-03-22T01:36:38.189395+0000","last_peered":"2024-03-22T01:36:38.189395+0000","last_clean":"2024-03-22T01:36:38.189395+0000","last_became_active":"2024-03-22T01:34:15.480601+0000","last_became_peered":"2024-03-22T01:34:15.480601+0000","last_unstale":"2024-03-22T01:36:38.189395+0000","last_undegraded":"2024-03-22T01:36:38.189395+0000","last_fullsized":"2024-03-22T01:36:38.189395+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:19:51.563598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"93'6","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500686+0000","last_change":"2024-03-22T01:34:15.490859+0000","last_active":"2024-03-22T01:36:38.500686+0000","last_peered":"2024-03-22T01:36:38.500686+0000","last_clean":"2024-03-22T01:36:38.500686+0000","last_became_active":"2024-03-22T01:34:15.490337+0000","last_became_peered":"2024-03-22T01:34:15.490337+0000","last_unstale":"2024-03-22T01:36:38.500686+0000","last_undegraded":"2024-03-22T01:36:38.500686+0000","last_fullsized":"2024-03-22T01:36:38.500686+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:46:07.072015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"93'15","reported_seq":47,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.191416+0000","last_change":"2024-03-22T01:34:15.495531+0000","last_active":"2024-03-22T01:36:38.191416+0000","last_peered":"2024-03-22T01:36:38.191416+0000","last_clean":"2024-03-22T01:36:38.191416+0000","last_became_active":"2024-03-22T01:34:15.495216+0000","last_became_peered":"2024-03-22T01:34:15.495216+0000","last_unstale":"2024-03-22T01:36:38.191416+0000","last_undegraded":"2024-03-22T01:36:38.191416+0000","last_fullsized":"2024-03-22T01:36:38.191416+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:39:22.575926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"93'2","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.308446+0000","last_change":"2024-03-22T01:34:15.499430+0000","last_active":"2024-03-22T01:36:38.308446+0000","last_peered":"2024-03-22T01:36:38.308446+0000","last_clean":"2024-03-22T01:36:38.308446+0000","last_became_active":"2024-03-22T01:34:15.499114+0000","last_became_peered":"2024-03-22T01:34:15.499114+0000","last_unstale":"2024-03-22T01:36:38.308446+0000","last_undegraded":"2024-03-22T01:36:38.308446+0000","last_fullsized":"2024-03-22T01:36:38.308446+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:23:01.781963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"93'6","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.314538+0000","last_change":"2024-03-22T01:34:15.485449+0000","last_active":"2024-03-22T01:36:38.314538+0000","last_peered":"2024-03-22T01:36:38.314538+0000","last_clean":"2024-03-22T01:36:38.314538+0000","last_became_active":"2024-03-22T01:34:15.485084+0000","last_became_peered":"2024-03-22T01:34:15.485084+0000","last_unstale":"2024-03-22T01:36:38.314538+0000","last_undegraded":"2024-03-22T01:36:38.314538+0000","last_fullsized":"2024-03-22T01:36:38.314538+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:53:25.632719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"93'8","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.307592+0000","last_change":"2024-03-22T01:34:15.495521+0000","last_active":"2024-03-22T01:36:38.307592+0000","last_peered":"2024-03-22T01:36:38.307592+0000","last_clean":"2024-03-22T01:36:38.307592+0000","last_became_active":"2024-03-22T01:34:15.495218+0000","last_became_peered":"2024-03-22T01:34:15.495218+0000","last_unstale":"2024-03-22T01:36:38.307592+0000","last_undegraded":"2024-03-22T01:36:38.307592+0000","last_fullsized":"2024-03-22T01:36:38.307592+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:04:32.569934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"93'10","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.460006+0000","last_change":"2024-03-22T01:34:15.489213+0000","last_active":"2024-03-22T01:36:38.460006+0000","last_peered":"2024-03-22T01:36:38.460006+0000","last_clean":"2024-03-22T01:36:38.460006+0000","last_became_active":"2024-03-22T01:34:15.488887+0000","last_became_peered":"2024-03-22T01:34:15.488887+0000","last_unstale":"2024-03-22T01:36:38.460006+0000","last_undegraded":"2024-03-22T01:36:38.460006+0000","last_fullsized":"2024-03-22T01:36:38.460006+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:53:39.983144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"93'11","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.189182+0000","last_change":"2024-03-22T01:34:15.485632+0000","last_active":"2024-03-22T01:36:38.189182+0000","last_peered":"2024-03-22T01:36:38.189182+0000","last_clean":"2024-03-22T01:36:38.189182+0000","last_became_active":"2024-03-22T01:34:15.485341+0000","last_became_peered":"2024-03-22T01:34:15.485341+0000","last_unstale":"2024-03-22T01:36:38.189182+0000","last_undegraded":"2024-03-22T01:36:38.189182+0000","last_fullsized":"2024-03-22T01:36:38.189182+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:33:00.248555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"93'8","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500583+0000","last_change":"2024-03-22T01:34:15.483593+0000","last_active":"2024-03-22T01:36:38.500583+0000","last_peered":"2024-03-22T01:36:38.500583+0000","last_clean":"2024-03-22T01:36:38.500583+0000","last_became_active":"2024-03-22T01:34:15.483201+0000","last_became_peered":"2024-03-22T01:34:15.483201+0000","last_unstale":"2024-03-22T01:36:38.500583+0000","last_undegraded":"2024-03-22T01:36:38.500583+0000","last_fullsized":"2024-03-22T01:36:38.500583+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:03:10.955120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":595682,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":348,"num_read_kb":279,"num_write":200,"num_write_kb":1410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1880064,"data_stored":1787046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318314,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32244,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712908,"statfs":{"total":95995035648,"available":95962017792,"internally_reserved":0,"allocated":1089536,"data_stored":538835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579951,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37064,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708088,"statfs":{"total":95995035648,"available":95957082112,"internally_reserved":0,"allocated":1765376,"data_stored":1189214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776180,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28476,"kb_used_data":1456,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716676,"statfs":{"total":95995035648,"available":95965876224,"internally_reserved":0,"allocated":1490944,"data_stored":936729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037816,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32636,"kb_used_data":1456,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712516,"statfs":{"total":95995035648,"available":95961616384,"internally_reserved":0,"allocated":1490944,"data_stored":932655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299454,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28672,"kb_used_data":1652,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716480,"statfs":{"total":95995035648,"available":95965675520,"internally_reserved":0,"allocated":1691648,"data_stored":1133881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462978,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1085440,"data_stored":534739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574939,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28156,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716996,"statfs":{"total":95995035648,"available":95966203904,"internally_reserved":0,"allocated":1163264,"data_stored":604807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738464,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1073152,"data_stored":524905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411414,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28160,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716992,"statfs":{"total":95995035648,"available":95966199808,"internally_reserved":0,"allocated":1167360,"data_stored":604841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149777,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36920,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708232,"statfs":{"total":95995035648,"available":95957229568,"internally_reserved":0,"allocated":1622016,"data_stored":1059659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888141,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1093632,"data_stored":538857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724616,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37304,"kb_used_data":1788,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93707848,"statfs":{"total":95995035648,"available":95956836352,"internally_reserved":0,"allocated":1830912,"data_stored":1260985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":70068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":70102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":536576,"data_stored":524920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":131782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":393820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":397312,"data_stored":393798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":225280,"data_stored":201862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-22T01:36:40.457 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-22T01:36:40.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.595+0000 7f829225c640 1 -- 172.21.15.72:0/749458095 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f828c101bd0 msgr2=0x7f828c101ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:40.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.595+0000 7f829225c640 1 --2- 172.21.15.72:0/749458095 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f828c101bd0 0x7f828c101ff0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f8280009900 tx=0x7f828002bc30 comp rx=0 tx=0).stop 2024-03-22T01:36:40.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.596+0000 7f829225c640 1 -- 172.21.15.72:0/749458095 shutdown_connections 2024-03-22T01:36:40.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.596+0000 7f829225c640 1 --2- 172.21.15.72:0/749458095 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f828c102dc0 0x7f828c103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:40.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.596+0000 7f829225c640 1 --2- 172.21.15.72:0/749458095 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f828c103780 0x7f828c10a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:40.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.596+0000 7f829225c640 1 --2- 172.21.15.72:0/749458095 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f828c101bd0 0x7f828c101ff0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:40.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.596+0000 7f829225c640 1 -- 172.21.15.72:0/749458095 >> 172.21.15.72:0/749458095 conn(0x7f828c0fd120 msgr2=0x7f828c0ff580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:40.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.596+0000 7f829225c640 1 -- 172.21.15.72:0/749458095 shutdown_connections 2024-03-22T01:36:40.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.596+0000 7f829225c640 1 -- 172.21.15.72:0/749458095 wait complete. 2024-03-22T01:36:40.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.596+0000 7f829225c640 1 Processor -- start 2024-03-22T01:36:40.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.597+0000 7f829225c640 1 -- start start 2024-03-22T01:36:40.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.597+0000 7f829225c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f828c101bd0 0x7f828c197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:40.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.597+0000 7f829225c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f828c102dc0 0x7f828c1983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:40.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.597+0000 7f829225c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f828c103780 0x7f828c19f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:40.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.597+0000 7f829225c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f828c10d850 con 0x7f828c103780 2024-03-22T01:36:40.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.597+0000 7f829225c640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f828c10d6d0 con 0x7f828c102dc0 2024-03-22T01:36:40.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.597+0000 7f829225c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f828c10d9d0 con 0x7f828c101bd0 2024-03-22T01:36:40.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.597+0000 7f828bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f828c103780 0x7f828c19f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:40.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.597+0000 7f828affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f828c102dc0 0x7f828c1983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:40.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.597+0000 7f828bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f828c103780 0x7f828c19f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54574/0 (socket says 172.21.15.72:54574) 2024-03-22T01:36:40.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.597+0000 7f828bfff640 1 -- 172.21.15.72:0/2265409634 learned_addr learned my addr 172.21.15.72:0/2265409634 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:40.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.597+0000 7f828affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f828c102dc0 0x7f828c1983e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.72:50304/0 (socket says 172.21.15.72:50304) 2024-03-22T01:36:40.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.597+0000 7f828b7fe640 1 --2- 172.21.15.72:0/2265409634 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f828c101bd0 0x7f828c197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:40.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.597+0000 7f828affd640 1 -- 172.21.15.72:0/2265409634 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f828c101bd0 msgr2=0x7f828c197ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:40.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.597+0000 7f828affd640 1 --2- 172.21.15.72:0/2265409634 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f828c101bd0 0x7f828c197ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:40.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.597+0000 7f828affd640 1 -- 172.21.15.72:0/2265409634 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f828c103780 msgr2=0x7f828c19f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:40.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.597+0000 7f828affd640 1 --2- 172.21.15.72:0/2265409634 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f828c103780 0x7f828c19f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:40.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.597+0000 7f828affd640 1 -- 172.21.15.72:0/2265409634 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f828c19fac0 con 0x7f828c102dc0 2024-03-22T01:36:40.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.597+0000 7f828bfff640 1 --2- 172.21.15.72:0/2265409634 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f828c103780 0x7f828c19f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:36:40.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.597+0000 7f828affd640 1 --2- 172.21.15.72:0/2265409634 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f828c102dc0 0x7f828c1983e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f827c00e9f0 tx=0x7f827c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:40.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.598+0000 7f8288ff9640 1 -- 172.21.15.72:0/2265409634 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f827c00ccc0 con 0x7f828c102dc0 2024-03-22T01:36:40.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.598+0000 7f8288ff9640 1 -- 172.21.15.72:0/2265409634 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f827c010910 con 0x7f828c102dc0 2024-03-22T01:36:40.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.598+0000 7f829225c640 1 -- 172.21.15.72:0/2265409634 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f828c19fdb0 con 0x7f828c102dc0 2024-03-22T01:36:40.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.598+0000 7f8288ff9640 1 -- 172.21.15.72:0/2265409634 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f827c018960 con 0x7f828c102dc0 2024-03-22T01:36:40.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.598+0000 7f829225c640 1 -- 172.21.15.72:0/2265409634 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f828c1a0270 con 0x7f828c102dc0 2024-03-22T01:36:40.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.599+0000 7f829225c640 1 -- 172.21.15.72:0/2265409634 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f828c108ef0 con 0x7f828c102dc0 2024-03-22T01:36:40.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.600+0000 7f8288ff9640 1 -- 172.21.15.72:0/2265409634 <== mon.2 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 21) ==== 137666+0+0 (secure 0 0 0) 0x7f827c010ac0 con 0x7f828c102dc0 2024-03-22T01:36:40.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.600+0000 7f8288ff9640 1 --2- 172.21.15.72:0/2265409634 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f82740a2d20 0x7f82740a51e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:40.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.601+0000 7f828b7fe640 1 --2- 172.21.15.72:0/2265409634 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f82740a2d20 0x7f82740a51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:40.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.601+0000 7f8288ff9640 1 -- 172.21.15.72:0/2265409634 <== mon.2 v2:172.21.15.89:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f827c0cf870 con 0x7f828c102dc0 2024-03-22T01:36:40.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.601+0000 7f828b7fe640 1 --2- 172.21.15.72:0/2265409634 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f82740a2d20 0x7f82740a51e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8280000c00 tx=0x7f8280005fd0 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:40.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.604+0000 7f8288ff9640 1 -- 172.21.15.72:0/2265409634 <== mon.2 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f827c09be30 con 0x7f828c102dc0 2024-03-22T01:36:40.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.779+0000 7f829225c640 1 -- 172.21.15.72:0/2265409634 --> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f828c061fd0 con 0x7f82740a2d20 2024-03-22T01:36:40.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.788+0000 7f8288ff9640 1 -- 172.21.15.72:0/2265409634 <== mgr.15381 v2:172.21.15.72:6800/1379139045 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339302 (secure 0 0 0) 0x7f828c061fd0 con 0x7f82740a2d20 2024-03-22T01:36:40.791 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:36:40.793 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-03-22T01:36:40.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.792+0000 7f829225c640 1 -- 172.21.15.72:0/2265409634 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f82740a2d20 msgr2=0x7f82740a51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:40.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.792+0000 7f829225c640 1 --2- 172.21.15.72:0/2265409634 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f82740a2d20 0x7f82740a51e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8280000c00 tx=0x7f8280005fd0 comp rx=0 tx=0).stop 2024-03-22T01:36:40.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.792+0000 7f829225c640 1 -- 172.21.15.72:0/2265409634 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f828c102dc0 msgr2=0x7f828c1983e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:40.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.792+0000 7f829225c640 1 --2- 172.21.15.72:0/2265409634 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f828c102dc0 0x7f828c1983e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f827c00e9f0 tx=0x7f827c00eeb0 comp rx=0 tx=0).stop 2024-03-22T01:36:40.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.792+0000 7f829225c640 1 -- 172.21.15.72:0/2265409634 shutdown_connections 2024-03-22T01:36:40.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.792+0000 7f829225c640 1 --2- 172.21.15.72:0/2265409634 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f82740a2d20 0x7f82740a51e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:40.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.792+0000 7f829225c640 1 --2- 172.21.15.72:0/2265409634 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f828c102dc0 0x7f828c1983e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:40.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.792+0000 7f829225c640 1 --2- 172.21.15.72:0/2265409634 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f828c103780 0x7f828c19f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:40.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.792+0000 7f829225c640 1 --2- 172.21.15.72:0/2265409634 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f828c101bd0 0x7f828c197ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:40.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.792+0000 7f829225c640 1 -- 172.21.15.72:0/2265409634 >> 172.21.15.72:0/2265409634 conn(0x7f828c0fd120 msgr2=0x7f828c1060c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:40.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.792+0000 7f829225c640 1 -- 172.21.15.72:0/2265409634 shutdown_connections 2024-03-22T01:36:40.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.792+0000 7f829225c640 1 -- 172.21.15.72:0/2265409634 wait complete. 2024-03-22T01:36:40.802 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":4,"stamp":"2024-03-22T01:36:40.202937+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1186284,"num_objects":28,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":484,"num_read_kb":398,"num_write":383,"num_write_kb":4350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":789,"ondisk_log_size":789,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":373868,"kb_used_data":16176,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124567956,"statfs":{"total":1151940427776,"available":1151557586944,"internally_reserved":0,"allocated":16564224,"data_stored":9860107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":365933882},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.990801"},"pg_stats":[{"pgid":"3.3f","version":"93'8","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500839+0000","last_change":"2024-03-22T01:34:15.487496+0000","last_active":"2024-03-22T01:36:38.500839+0000","last_peered":"2024-03-22T01:36:38.500839+0000","last_clean":"2024-03-22T01:36:38.500839+0000","last_became_active":"2024-03-22T01:34:15.486700+0000","last_became_peered":"2024-03-22T01:34:15.486700+0000","last_unstale":"2024-03-22T01:36:38.500839+0000","last_undegraded":"2024-03-22T01:36:38.500839+0000","last_fullsized":"2024-03-22T01:36:38.500839+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:03:19.146811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"93'4","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.501995+0000","last_change":"2024-03-22T01:34:15.489791+0000","last_active":"2024-03-22T01:36:38.501995+0000","last_peered":"2024-03-22T01:36:38.501995+0000","last_clean":"2024-03-22T01:36:38.501995+0000","last_became_active":"2024-03-22T01:34:15.489493+0000","last_became_peered":"2024-03-22T01:34:15.489493+0000","last_unstale":"2024-03-22T01:36:38.501995+0000","last_undegraded":"2024-03-22T01:36:38.501995+0000","last_fullsized":"2024-03-22T01:36:38.501995+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:45:05.060454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"93'7","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.501785+0000","last_change":"2024-03-22T01:34:15.491567+0000","last_active":"2024-03-22T01:36:38.501785+0000","last_peered":"2024-03-22T01:36:38.501785+0000","last_clean":"2024-03-22T01:36:38.501785+0000","last_became_active":"2024-03-22T01:34:15.491275+0000","last_became_peered":"2024-03-22T01:34:15.491275+0000","last_unstale":"2024-03-22T01:36:38.501785+0000","last_undegraded":"2024-03-22T01:36:38.501785+0000","last_fullsized":"2024-03-22T01:36:38.501785+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:26:36.630811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"93'8","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500405+0000","last_change":"2024-03-22T01:34:15.488579+0000","last_active":"2024-03-22T01:36:38.500405+0000","last_peered":"2024-03-22T01:36:38.500405+0000","last_clean":"2024-03-22T01:36:38.500405+0000","last_became_active":"2024-03-22T01:34:15.488262+0000","last_became_peered":"2024-03-22T01:34:15.488262+0000","last_unstale":"2024-03-22T01:36:38.500405+0000","last_undegraded":"2024-03-22T01:36:38.500405+0000","last_fullsized":"2024-03-22T01:36:38.500405+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:58:21.788420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"93'4","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.307700+0000","last_change":"2024-03-22T01:34:15.485744+0000","last_active":"2024-03-22T01:36:38.307700+0000","last_peered":"2024-03-22T01:36:38.307700+0000","last_clean":"2024-03-22T01:36:38.307700+0000","last_became_active":"2024-03-22T01:34:15.485378+0000","last_became_peered":"2024-03-22T01:34:15.485378+0000","last_unstale":"2024-03-22T01:36:38.307700+0000","last_undegraded":"2024-03-22T01:36:38.307700+0000","last_fullsized":"2024-03-22T01:36:38.307700+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:06:40.979475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"93'6","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.192414+0000","last_change":"2024-03-22T01:34:15.495051+0000","last_active":"2024-03-22T01:36:38.192414+0000","last_peered":"2024-03-22T01:36:38.192414+0000","last_clean":"2024-03-22T01:36:38.192414+0000","last_became_active":"2024-03-22T01:34:15.494691+0000","last_became_peered":"2024-03-22T01:34:15.494691+0000","last_unstale":"2024-03-22T01:36:38.192414+0000","last_undegraded":"2024-03-22T01:36:38.192414+0000","last_fullsized":"2024-03-22T01:36:38.192414+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T01:44:28.336545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"93'5","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.300153+0000","last_change":"2024-03-22T01:34:15.493911+0000","last_active":"2024-03-22T01:36:38.300153+0000","last_peered":"2024-03-22T01:36:38.300153+0000","last_clean":"2024-03-22T01:36:38.300153+0000","last_became_active":"2024-03-22T01:34:15.493586+0000","last_became_peered":"2024-03-22T01:34:15.493586+0000","last_unstale":"2024-03-22T01:36:38.300153+0000","last_undegraded":"2024-03-22T01:36:38.300153+0000","last_fullsized":"2024-03-22T01:36:38.300153+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:22:41.391855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"93'9","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.307138+0000","last_change":"2024-03-22T01:34:15.481410+0000","last_active":"2024-03-22T01:36:38.307138+0000","last_peered":"2024-03-22T01:36:38.307138+0000","last_clean":"2024-03-22T01:36:38.307138+0000","last_became_active":"2024-03-22T01:34:15.481109+0000","last_became_peered":"2024-03-22T01:34:15.481109+0000","last_unstale":"2024-03-22T01:36:38.307138+0000","last_undegraded":"2024-03-22T01:36:38.307138+0000","last_fullsized":"2024-03-22T01:36:38.307138+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:27:10.004584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"93'11","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.460250+0000","last_change":"2024-03-22T01:34:15.485173+0000","last_active":"2024-03-22T01:36:38.460250+0000","last_peered":"2024-03-22T01:36:38.460250+0000","last_clean":"2024-03-22T01:36:38.460250+0000","last_became_active":"2024-03-22T01:34:15.484812+0000","last_became_peered":"2024-03-22T01:34:15.484812+0000","last_unstale":"2024-03-22T01:36:38.460250+0000","last_undegraded":"2024-03-22T01:36:38.460250+0000","last_fullsized":"2024-03-22T01:36:38.460250+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:44:06.383568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"93'16","reported_seq":47,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.192571+0000","last_change":"2024-03-22T01:34:15.497964+0000","last_active":"2024-03-22T01:36:38.192571+0000","last_peered":"2024-03-22T01:36:38.192571+0000","last_clean":"2024-03-22T01:36:38.192571+0000","last_became_active":"2024-03-22T01:34:15.496887+0000","last_became_peered":"2024-03-22T01:34:15.496887+0000","last_unstale":"2024-03-22T01:36:38.192571+0000","last_undegraded":"2024-03-22T01:36:38.192571+0000","last_fullsized":"2024-03-22T01:36:38.192571+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:13:40.420667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"93'6","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.300582+0000","last_change":"2024-03-22T01:34:15.496310+0000","last_active":"2024-03-22T01:36:38.300582+0000","last_peered":"2024-03-22T01:36:38.300582+0000","last_clean":"2024-03-22T01:36:38.300582+0000","last_became_active":"2024-03-22T01:34:15.494196+0000","last_became_peered":"2024-03-22T01:34:15.494196+0000","last_unstale":"2024-03-22T01:36:38.300582+0000","last_undegraded":"2024-03-22T01:36:38.300582+0000","last_fullsized":"2024-03-22T01:36:38.300582+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:33:02.798310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"93'7","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500458+0000","last_change":"2024-03-22T01:34:15.495380+0000","last_active":"2024-03-22T01:36:38.500458+0000","last_peered":"2024-03-22T01:36:38.500458+0000","last_clean":"2024-03-22T01:36:38.500458+0000","last_became_active":"2024-03-22T01:34:15.495063+0000","last_became_peered":"2024-03-22T01:34:15.495063+0000","last_unstale":"2024-03-22T01:36:38.500458+0000","last_undegraded":"2024-03-22T01:36:38.500458+0000","last_fullsized":"2024-03-22T01:36:38.500458+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:46:29.497329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"93'10","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.189104+0000","last_change":"2024-03-22T01:34:15.497533+0000","last_active":"2024-03-22T01:36:38.189104+0000","last_peered":"2024-03-22T01:36:38.189104+0000","last_clean":"2024-03-22T01:36:38.189104+0000","last_became_active":"2024-03-22T01:34:15.497223+0000","last_became_peered":"2024-03-22T01:34:15.497223+0000","last_unstale":"2024-03-22T01:36:38.189104+0000","last_undegraded":"2024-03-22T01:36:38.189104+0000","last_fullsized":"2024-03-22T01:36:38.189104+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:17:49.202842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"93'7","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.501331+0000","last_change":"2024-03-22T01:34:15.491317+0000","last_active":"2024-03-22T01:36:38.501331+0000","last_peered":"2024-03-22T01:36:38.501331+0000","last_clean":"2024-03-22T01:36:38.501331+0000","last_became_active":"2024-03-22T01:34:15.490982+0000","last_became_peered":"2024-03-22T01:34:15.490982+0000","last_unstale":"2024-03-22T01:36:38.501331+0000","last_undegraded":"2024-03-22T01:36:38.501331+0000","last_fullsized":"2024-03-22T01:36:38.501331+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:05:39.889964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"93'7","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.501112+0000","last_change":"2024-03-22T01:34:15.488502+0000","last_active":"2024-03-22T01:36:38.501112+0000","last_peered":"2024-03-22T01:36:38.501112+0000","last_clean":"2024-03-22T01:36:38.501112+0000","last_became_active":"2024-03-22T01:34:15.488200+0000","last_became_peered":"2024-03-22T01:34:15.488200+0000","last_unstale":"2024-03-22T01:36:38.501112+0000","last_undegraded":"2024-03-22T01:36:38.501112+0000","last_fullsized":"2024-03-22T01:36:38.501112+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:52:59.991591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"93'9","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.459533+0000","last_change":"2024-03-22T01:34:15.489991+0000","last_active":"2024-03-22T01:36:38.459533+0000","last_peered":"2024-03-22T01:36:38.459533+0000","last_clean":"2024-03-22T01:36:38.459533+0000","last_became_active":"2024-03-22T01:34:15.489696+0000","last_became_peered":"2024-03-22T01:34:15.489696+0000","last_unstale":"2024-03-22T01:36:38.459533+0000","last_undegraded":"2024-03-22T01:36:38.459533+0000","last_fullsized":"2024-03-22T01:36:38.459533+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:10:45.316422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"93'8","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500862+0000","last_change":"2024-03-22T01:34:15.488011+0000","last_active":"2024-03-22T01:36:38.500862+0000","last_peered":"2024-03-22T01:36:38.500862+0000","last_clean":"2024-03-22T01:36:38.500862+0000","last_became_active":"2024-03-22T01:34:15.487282+0000","last_became_peered":"2024-03-22T01:34:15.487282+0000","last_unstale":"2024-03-22T01:36:38.500862+0000","last_undegraded":"2024-03-22T01:36:38.500862+0000","last_fullsized":"2024-03-22T01:36:38.500862+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:16:21.396273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"93'10","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.305057+0000","last_change":"2024-03-22T01:34:15.488046+0000","last_active":"2024-03-22T01:36:38.305057+0000","last_peered":"2024-03-22T01:36:38.305057+0000","last_clean":"2024-03-22T01:36:38.305057+0000","last_became_active":"2024-03-22T01:34:15.487731+0000","last_became_peered":"2024-03-22T01:34:15.487731+0000","last_unstale":"2024-03-22T01:36:38.305057+0000","last_undegraded":"2024-03-22T01:36:38.305057+0000","last_fullsized":"2024-03-22T01:36:38.305057+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:44:16.931544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"93'7","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.191341+0000","last_change":"2024-03-22T01:34:15.503737+0000","last_active":"2024-03-22T01:36:38.191341+0000","last_peered":"2024-03-22T01:36:38.191341+0000","last_clean":"2024-03-22T01:36:38.191341+0000","last_became_active":"2024-03-22T01:34:15.502169+0000","last_became_peered":"2024-03-22T01:34:15.502169+0000","last_unstale":"2024-03-22T01:36:38.191341+0000","last_undegraded":"2024-03-22T01:36:38.191341+0000","last_fullsized":"2024-03-22T01:36:38.191341+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:05:53.027318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":24,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.191251+0000","last_change":"2024-03-22T01:34:19.527964+0000","last_active":"2024-03-22T01:36:38.191251+0000","last_peered":"2024-03-22T01:36:38.191251+0000","last_clean":"2024-03-22T01:36:38.191251+0000","last_became_active":"2024-03-22T01:34:19.527526+0000","last_became_peered":"2024-03-22T01:34:19.527526+0000","last_unstale":"2024-03-22T01:36:38.191251+0000","last_undegraded":"2024-03-22T01:36:38.191251+0000","last_fullsized":"2024-03-22T01:36:38.191251+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:35:39.404439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'4","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.305744+0000","last_change":"2024-03-22T01:34:14.473065+0000","last_active":"2024-03-22T01:36:38.305744+0000","last_peered":"2024-03-22T01:36:38.305744+0000","last_clean":"2024-03-22T01:36:38.305744+0000","last_became_active":"2024-03-22T01:34:14.472653+0000","last_became_peered":"2024-03-22T01:34:14.472653+0000","last_unstale":"2024-03-22T01:36:38.305744+0000","last_undegraded":"2024-03-22T01:36:38.305744+0000","last_fullsized":"2024-03-22T01:36:38.305744+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:11:09.525438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"93'5","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.308183+0000","last_change":"2024-03-22T01:34:15.497818+0000","last_active":"2024-03-22T01:36:38.308183+0000","last_peered":"2024-03-22T01:36:38.308183+0000","last_clean":"2024-03-22T01:36:38.308183+0000","last_became_active":"2024-03-22T01:34:15.497488+0000","last_became_peered":"2024-03-22T01:34:15.497488+0000","last_unstale":"2024-03-22T01:36:38.308183+0000","last_undegraded":"2024-03-22T01:36:38.308183+0000","last_fullsized":"2024-03-22T01:36:38.308183+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:21:09.340557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":24,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500006+0000","last_change":"2024-03-22T01:34:19.528435+0000","last_active":"2024-03-22T01:36:38.500006+0000","last_peered":"2024-03-22T01:36:38.500006+0000","last_clean":"2024-03-22T01:36:38.500006+0000","last_became_active":"2024-03-22T01:34:19.528044+0000","last_became_peered":"2024-03-22T01:34:19.528044+0000","last_unstale":"2024-03-22T01:36:38.500006+0000","last_undegraded":"2024-03-22T01:36:38.500006+0000","last_fullsized":"2024-03-22T01:36:38.500006+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:52:05.682173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"86'1","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.308372+0000","last_change":"2024-03-22T01:34:14.465817+0000","last_active":"2024-03-22T01:36:38.308372+0000","last_peered":"2024-03-22T01:36:38.308372+0000","last_clean":"2024-03-22T01:36:38.308372+0000","last_became_active":"2024-03-22T01:34:14.464338+0000","last_became_peered":"2024-03-22T01:34:14.464338+0000","last_unstale":"2024-03-22T01:36:38.308372+0000","last_undegraded":"2024-03-22T01:36:38.308372+0000","last_fullsized":"2024-03-22T01:36:38.308372+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:04:47.457092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"93'6","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.502083+0000","last_change":"2024-03-22T01:34:15.481988+0000","last_active":"2024-03-22T01:36:38.502083+0000","last_peered":"2024-03-22T01:36:38.502083+0000","last_clean":"2024-03-22T01:36:38.502083+0000","last_became_active":"2024-03-22T01:34:15.481676+0000","last_became_peered":"2024-03-22T01:34:15.481676+0000","last_unstale":"2024-03-22T01:36:38.502083+0000","last_undegraded":"2024-03-22T01:36:38.502083+0000","last_fullsized":"2024-03-22T01:36:38.502083+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:37:54.609731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":24,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500197+0000","last_change":"2024-03-22T01:34:19.528349+0000","last_active":"2024-03-22T01:36:38.500197+0000","last_peered":"2024-03-22T01:36:38.500197+0000","last_clean":"2024-03-22T01:36:38.500197+0000","last_became_active":"2024-03-22T01:34:19.527937+0000","last_became_peered":"2024-03-22T01:34:19.527937+0000","last_unstale":"2024-03-22T01:36:38.500197+0000","last_undegraded":"2024-03-22T01:36:38.500197+0000","last_fullsized":"2024-03-22T01:36:38.500197+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:48:34.242104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"86'2","reported_seq":44,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.308006+0000","last_change":"2024-03-22T01:35:16.756010+0000","last_active":"2024-03-22T01:36:38.308006+0000","last_peered":"2024-03-22T01:36:38.308006+0000","last_clean":"2024-03-22T01:36:38.308006+0000","last_became_active":"2024-03-22T01:35:16.751845+0000","last_became_peered":"2024-03-22T01:35:16.751845+0000","last_unstale":"2024-03-22T01:36:38.308006+0000","last_undegraded":"2024-03-22T01:36:38.308006+0000","last_fullsized":"2024-03-22T01:36:38.308006+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:23:18.305330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"93'9","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.459910+0000","last_change":"2024-03-22T01:34:15.491345+0000","last_active":"2024-03-22T01:36:38.459910+0000","last_peered":"2024-03-22T01:36:38.459910+0000","last_clean":"2024-03-22T01:36:38.459910+0000","last_became_active":"2024-03-22T01:34:15.490833+0000","last_became_peered":"2024-03-22T01:34:15.490833+0000","last_unstale":"2024-03-22T01:36:38.459910+0000","last_undegraded":"2024-03-22T01:36:38.459910+0000","last_fullsized":"2024-03-22T01:36:38.459910+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:30:43.950679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":24,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500078+0000","last_change":"2024-03-22T01:34:19.528351+0000","last_active":"2024-03-22T01:36:38.500078+0000","last_peered":"2024-03-22T01:36:38.500078+0000","last_clean":"2024-03-22T01:36:38.500078+0000","last_became_active":"2024-03-22T01:34:19.527939+0000","last_became_peered":"2024-03-22T01:34:19.527939+0000","last_unstale":"2024-03-22T01:36:38.500078+0000","last_undegraded":"2024-03-22T01:36:38.500078+0000","last_fullsized":"2024-03-22T01:36:38.500078+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:30:21.055848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":21,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500428+0000","last_change":"2024-03-22T01:35:16.755726+0000","last_active":"2024-03-22T01:36:38.500428+0000","last_peered":"2024-03-22T01:36:38.500428+0000","last_clean":"2024-03-22T01:36:38.500428+0000","last_became_active":"2024-03-22T01:35:16.749107+0000","last_became_peered":"2024-03-22T01:35:16.749107+0000","last_unstale":"2024-03-22T01:36:38.500428+0000","last_undegraded":"2024-03-22T01:36:38.500428+0000","last_fullsized":"2024-03-22T01:36:38.500428+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:49:37.500003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"93'15","reported_seq":44,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.314906+0000","last_change":"2024-03-22T01:34:15.479530+0000","last_active":"2024-03-22T01:36:38.314906+0000","last_peered":"2024-03-22T01:36:38.314906+0000","last_clean":"2024-03-22T01:36:38.314906+0000","last_became_active":"2024-03-22T01:34:15.479198+0000","last_became_peered":"2024-03-22T01:34:15.479198+0000","last_unstale":"2024-03-22T01:36:38.314906+0000","last_undegraded":"2024-03-22T01:36:38.314906+0000","last_fullsized":"2024-03-22T01:36:38.314906+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:28:51.624369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":24,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.191181+0000","last_change":"2024-03-22T01:34:19.524486+0000","last_active":"2024-03-22T01:36:38.191181+0000","last_peered":"2024-03-22T01:36:38.191181+0000","last_clean":"2024-03-22T01:36:38.191181+0000","last_became_active":"2024-03-22T01:34:19.524030+0000","last_became_peered":"2024-03-22T01:34:19.524030+0000","last_unstale":"2024-03-22T01:36:38.191181+0000","last_undegraded":"2024-03-22T01:36:38.191181+0000","last_fullsized":"2024-03-22T01:36:38.191181+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:31:10.157431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"86'1","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.314671+0000","last_change":"2024-03-22T01:34:14.469191+0000","last_active":"2024-03-22T01:36:38.314671+0000","last_peered":"2024-03-22T01:36:38.314671+0000","last_clean":"2024-03-22T01:36:38.314671+0000","last_became_active":"2024-03-22T01:34:14.468842+0000","last_became_peered":"2024-03-22T01:34:14.468842+0000","last_unstale":"2024-03-22T01:36:38.314671+0000","last_undegraded":"2024-03-22T01:36:38.314671+0000","last_fullsized":"2024-03-22T01:36:38.314671+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:29:56.566256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"93'7","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.307636+0000","last_change":"2024-03-22T01:34:15.483736+0000","last_active":"2024-03-22T01:36:38.307636+0000","last_peered":"2024-03-22T01:36:38.307636+0000","last_clean":"2024-03-22T01:36:38.307636+0000","last_became_active":"2024-03-22T01:34:15.483401+0000","last_became_peered":"2024-03-22T01:34:15.483401+0000","last_unstale":"2024-03-22T01:36:38.307636+0000","last_undegraded":"2024-03-22T01:36:38.307636+0000","last_fullsized":"2024-03-22T01:36:38.307636+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:26:51.974932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":24,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.191134+0000","last_change":"2024-03-22T01:34:19.534542+0000","last_active":"2024-03-22T01:36:38.191134+0000","last_peered":"2024-03-22T01:36:38.191134+0000","last_clean":"2024-03-22T01:36:38.191134+0000","last_became_active":"2024-03-22T01:34:19.534039+0000","last_became_peered":"2024-03-22T01:34:19.534039+0000","last_unstale":"2024-03-22T01:36:38.191134+0000","last_undegraded":"2024-03-22T01:36:38.191134+0000","last_fullsized":"2024-03-22T01:36:38.191134+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:59:37.363972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"86'1","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.192279+0000","last_change":"2024-03-22T01:34:14.468382+0000","last_active":"2024-03-22T01:36:38.192279+0000","last_peered":"2024-03-22T01:36:38.192279+0000","last_clean":"2024-03-22T01:36:38.192279+0000","last_became_active":"2024-03-22T01:34:14.467939+0000","last_became_peered":"2024-03-22T01:34:14.467939+0000","last_unstale":"2024-03-22T01:36:38.192279+0000","last_undegraded":"2024-03-22T01:36:38.192279+0000","last_fullsized":"2024-03-22T01:36:38.192279+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:57:17.031893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"93'8","reported_seq":43,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.314801+0000","last_change":"2024-03-22T01:35:17.206178+0000","last_active":"2024-03-22T01:36:38.314801+0000","last_peered":"2024-03-22T01:36:38.314801+0000","last_clean":"2024-03-22T01:36:38.314801+0000","last_became_active":"2024-03-22T01:35:17.205717+0000","last_became_peered":"2024-03-22T01:35:17.205717+0000","last_unstale":"2024-03-22T01:36:38.314801+0000","last_undegraded":"2024-03-22T01:36:38.314801+0000","last_fullsized":"2024-03-22T01:36:38.314801+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:14:15.920771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500689+0000","last_change":"2024-03-22T01:36:17.659141+0000","last_active":"2024-03-22T01:36:38.500689+0000","last_peered":"2024-03-22T01:36:38.500689+0000","last_clean":"2024-03-22T01:36:38.500689+0000","last_became_active":"2024-03-22T01:36:17.658611+0000","last_became_peered":"2024-03-22T01:36:17.658611+0000","last_unstale":"2024-03-22T01:36:38.500689+0000","last_undegraded":"2024-03-22T01:36:38.500689+0000","last_fullsized":"2024-03-22T01:36:38.500689+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:45:28.895847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,10,0],"acting":[11,4,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.189382+0000","last_change":"2024-03-22T01:34:14.464485+0000","last_active":"2024-03-22T01:36:38.189382+0000","last_peered":"2024-03-22T01:36:38.189382+0000","last_clean":"2024-03-22T01:36:38.189382+0000","last_became_active":"2024-03-22T01:34:14.463973+0000","last_became_peered":"2024-03-22T01:34:14.463973+0000","last_unstale":"2024-03-22T01:36:38.189382+0000","last_undegraded":"2024-03-22T01:36:38.189382+0000","last_fullsized":"2024-03-22T01:36:38.189382+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:11:13.265346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"93'8","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.307338+0000","last_change":"2024-03-22T01:34:15.483893+0000","last_active":"2024-03-22T01:36:38.307338+0000","last_peered":"2024-03-22T01:36:38.307338+0000","last_clean":"2024-03-22T01:36:38.307338+0000","last_became_active":"2024-03-22T01:34:15.482295+0000","last_became_peered":"2024-03-22T01:34:15.482295+0000","last_unstale":"2024-03-22T01:36:38.307338+0000","last_undegraded":"2024-03-22T01:36:38.307338+0000","last_fullsized":"2024-03-22T01:36:38.307338+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:17:33.274900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":25,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.501992+0000","last_change":"2024-03-22T01:34:19.522567+0000","last_active":"2024-03-22T01:36:38.501992+0000","last_peered":"2024-03-22T01:36:38.501992+0000","last_clean":"2024-03-22T01:36:38.501992+0000","last_became_active":"2024-03-22T01:34:19.522165+0000","last_became_peered":"2024-03-22T01:34:19.522165+0000","last_unstale":"2024-03-22T01:36:38.501992+0000","last_undegraded":"2024-03-22T01:36:38.501992+0000","last_fullsized":"2024-03-22T01:36:38.501992+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:41:37.829622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":31,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.314474+0000","last_change":"2024-03-22T01:34:14.463824+0000","last_active":"2024-03-22T01:36:38.314474+0000","last_peered":"2024-03-22T01:36:38.314474+0000","last_clean":"2024-03-22T01:36:38.314474+0000","last_became_active":"2024-03-22T01:34:14.463484+0000","last_became_peered":"2024-03-22T01:34:14.463484+0000","last_unstale":"2024-03-22T01:36:38.314474+0000","last_undegraded":"2024-03-22T01:36:38.314474+0000","last_fullsized":"2024-03-22T01:36:38.314474+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:19:31.568853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"93'11","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.304658+0000","last_change":"2024-03-22T01:34:15.488883+0000","last_active":"2024-03-22T01:36:38.304658+0000","last_peered":"2024-03-22T01:36:38.304658+0000","last_clean":"2024-03-22T01:36:38.304658+0000","last_became_active":"2024-03-22T01:34:15.488573+0000","last_became_peered":"2024-03-22T01:34:15.488573+0000","last_unstale":"2024-03-22T01:36:38.304658+0000","last_undegraded":"2024-03-22T01:36:38.304658+0000","last_fullsized":"2024-03-22T01:36:38.304658+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:28:46.388930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.307244+0000","last_change":"2024-03-22T01:34:19.527321+0000","last_active":"2024-03-22T01:36:38.307244+0000","last_peered":"2024-03-22T01:36:38.307244+0000","last_clean":"2024-03-22T01:36:38.307244+0000","last_became_active":"2024-03-22T01:34:19.526905+0000","last_became_peered":"2024-03-22T01:34:19.526905+0000","last_unstale":"2024-03-22T01:36:38.307244+0000","last_undegraded":"2024-03-22T01:36:38.307244+0000","last_fullsized":"2024-03-22T01:36:38.307244+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:49:31.023938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.192604+0000","last_change":"2024-03-22T01:34:14.471415+0000","last_active":"2024-03-22T01:36:38.192604+0000","last_peered":"2024-03-22T01:36:38.192604+0000","last_clean":"2024-03-22T01:36:38.192604+0000","last_became_active":"2024-03-22T01:34:14.470928+0000","last_became_peered":"2024-03-22T01:34:14.470928+0000","last_unstale":"2024-03-22T01:36:38.192604+0000","last_undegraded":"2024-03-22T01:36:38.192604+0000","last_fullsized":"2024-03-22T01:36:38.192604+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:09:10.286549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"93'10","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.460165+0000","last_change":"2024-03-22T01:34:15.489656+0000","last_active":"2024-03-22T01:36:38.460165+0000","last_peered":"2024-03-22T01:36:38.460165+0000","last_clean":"2024-03-22T01:36:38.460165+0000","last_became_active":"2024-03-22T01:34:15.488933+0000","last_became_peered":"2024-03-22T01:34:15.488933+0000","last_unstale":"2024-03-22T01:36:38.460165+0000","last_undegraded":"2024-03-22T01:36:38.460165+0000","last_fullsized":"2024-03-22T01:36:38.460165+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:17:38.789414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.304368+0000","last_change":"2024-03-22T01:34:19.534174+0000","last_active":"2024-03-22T01:36:38.304368+0000","last_peered":"2024-03-22T01:36:38.304368+0000","last_clean":"2024-03-22T01:36:38.304368+0000","last_became_active":"2024-03-22T01:34:19.533759+0000","last_became_peered":"2024-03-22T01:34:19.533759+0000","last_unstale":"2024-03-22T01:36:38.304368+0000","last_undegraded":"2024-03-22T01:36:38.304368+0000","last_fullsized":"2024-03-22T01:36:38.304368+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:24:50.363984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.300490+0000","last_change":"2024-03-22T01:34:14.465755+0000","last_active":"2024-03-22T01:36:38.300490+0000","last_peered":"2024-03-22T01:36:38.300490+0000","last_clean":"2024-03-22T01:36:38.300490+0000","last_became_active":"2024-03-22T01:34:14.465230+0000","last_became_peered":"2024-03-22T01:34:14.465230+0000","last_unstale":"2024-03-22T01:36:38.300490+0000","last_undegraded":"2024-03-22T01:36:38.300490+0000","last_fullsized":"2024-03-22T01:36:38.300490+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:56:29.286551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"93'9","reported_seq":45,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.502022+0000","last_change":"2024-03-22T01:35:16.748981+0000","last_active":"2024-03-22T01:36:38.502022+0000","last_peered":"2024-03-22T01:36:38.502022+0000","last_clean":"2024-03-22T01:36:38.502022+0000","last_became_active":"2024-03-22T01:35:16.748524+0000","last_became_peered":"2024-03-22T01:35:16.748524+0000","last_unstale":"2024-03-22T01:36:38.502022+0000","last_undegraded":"2024-03-22T01:36:38.502022+0000","last_fullsized":"2024-03-22T01:36:38.502022+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:23:37.763347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":31,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500236+0000","last_change":"2024-03-22T01:36:17.789745+0000","last_active":"2024-03-22T01:36:38.500236+0000","last_peered":"2024-03-22T01:36:38.500236+0000","last_clean":"2024-03-22T01:36:38.500236+0000","last_became_active":"2024-03-22T01:36:17.789277+0000","last_became_peered":"2024-03-22T01:36:17.789277+0000","last_unstale":"2024-03-22T01:36:38.500236+0000","last_undegraded":"2024-03-22T01:36:38.500236+0000","last_fullsized":"2024-03-22T01:36:38.500236+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:57:47.671252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"86'3","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.315060+0000","last_change":"2024-03-22T01:34:14.463365+0000","last_active":"2024-03-22T01:36:38.315060+0000","last_peered":"2024-03-22T01:36:38.315060+0000","last_clean":"2024-03-22T01:36:38.315060+0000","last_became_active":"2024-03-22T01:34:14.462984+0000","last_became_peered":"2024-03-22T01:34:14.462984+0000","last_unstale":"2024-03-22T01:36:38.315060+0000","last_undegraded":"2024-03-22T01:36:38.315060+0000","last_fullsized":"2024-03-22T01:36:38.315060+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:55:27.318854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"93'11","reported_seq":43,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.460074+0000","last_change":"2024-03-22T01:34:15.485309+0000","last_active":"2024-03-22T01:36:38.460074+0000","last_peered":"2024-03-22T01:36:38.460074+0000","last_clean":"2024-03-22T01:36:38.460074+0000","last_became_active":"2024-03-22T01:34:15.484964+0000","last_became_peered":"2024-03-22T01:34:15.484964+0000","last_unstale":"2024-03-22T01:36:38.460074+0000","last_undegraded":"2024-03-22T01:36:38.460074+0000","last_fullsized":"2024-03-22T01:36:38.460074+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:25:57.911623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":31,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.304733+0000","last_change":"2024-03-22T01:36:17.659080+0000","last_active":"2024-03-22T01:36:38.304733+0000","last_peered":"2024-03-22T01:36:38.304733+0000","last_clean":"2024-03-22T01:36:38.304733+0000","last_became_active":"2024-03-22T01:36:17.658738+0000","last_became_peered":"2024-03-22T01:36:17.658738+0000","last_unstale":"2024-03-22T01:36:38.304733+0000","last_undegraded":"2024-03-22T01:36:38.304733+0000","last_fullsized":"2024-03-22T01:36:38.304733+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:28:38.338235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,10],"acting":[9,8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"96'37","reported_seq":321,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.688979+0000","last_change":"2024-03-22T01:34:14.469378+0000","last_active":"2024-03-22T01:36:38.688979+0000","last_peered":"2024-03-22T01:36:38.688979+0000","last_clean":"2024-03-22T01:36:38.688979+0000","last_became_active":"2024-03-22T01:34:14.468956+0000","last_became_peered":"2024-03-22T01:34:14.468956+0000","last_unstale":"2024-03-22T01:36:38.688979+0000","last_undegraded":"2024-03-22T01:36:38.688979+0000","last_fullsized":"2024-03-22T01:36:38.688979+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:40:49.048793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70068,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":250,"num_read_kb":194,"num_write":37,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"93'6","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.307927+0000","last_change":"2024-03-22T01:34:15.488497+0000","last_active":"2024-03-22T01:36:38.307927+0000","last_peered":"2024-03-22T01:36:38.307927+0000","last_clean":"2024-03-22T01:36:38.307927+0000","last_became_active":"2024-03-22T01:34:15.488134+0000","last_became_peered":"2024-03-22T01:34:15.488134+0000","last_unstale":"2024-03-22T01:36:38.307927+0000","last_undegraded":"2024-03-22T01:36:38.307927+0000","last_fullsized":"2024-03-22T01:36:38.307927+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:50:41.206135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":24,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.315310+0000","last_change":"2024-03-22T01:34:19.528744+0000","last_active":"2024-03-22T01:36:38.315310+0000","last_peered":"2024-03-22T01:36:38.315310+0000","last_clean":"2024-03-22T01:36:38.315310+0000","last_became_active":"2024-03-22T01:34:19.528375+0000","last_became_peered":"2024-03-22T01:34:19.528375+0000","last_unstale":"2024-03-22T01:36:38.315310+0000","last_undegraded":"2024-03-22T01:36:38.315310+0000","last_fullsized":"2024-03-22T01:36:38.315310+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:47:53.753923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"86'2","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.191536+0000","last_change":"2024-03-22T01:34:14.463289+0000","last_active":"2024-03-22T01:36:38.191536+0000","last_peered":"2024-03-22T01:36:38.191536+0000","last_clean":"2024-03-22T01:36:38.191536+0000","last_became_active":"2024-03-22T01:34:14.462883+0000","last_became_peered":"2024-03-22T01:34:14.462883+0000","last_unstale":"2024-03-22T01:36:38.191536+0000","last_undegraded":"2024-03-22T01:36:38.191536+0000","last_fullsized":"2024-03-22T01:36:38.191536+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:06:14.452674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"93'9","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.188986+0000","last_change":"2024-03-22T01:34:15.484785+0000","last_active":"2024-03-22T01:36:38.188986+0000","last_peered":"2024-03-22T01:36:38.188986+0000","last_clean":"2024-03-22T01:36:38.188986+0000","last_became_active":"2024-03-22T01:34:15.484474+0000","last_became_peered":"2024-03-22T01:34:15.484474+0000","last_unstale":"2024-03-22T01:36:38.188986+0000","last_undegraded":"2024-03-22T01:36:38.188986+0000","last_fullsized":"2024-03-22T01:36:38.188986+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:11:13.338836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":24,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500472+0000","last_change":"2024-03-22T01:34:19.527379+0000","last_active":"2024-03-22T01:36:38.500472+0000","last_peered":"2024-03-22T01:36:38.500472+0000","last_clean":"2024-03-22T01:36:38.500472+0000","last_became_active":"2024-03-22T01:34:19.526957+0000","last_became_peered":"2024-03-22T01:34:19.526957+0000","last_unstale":"2024-03-22T01:36:38.500472+0000","last_undegraded":"2024-03-22T01:36:38.500472+0000","last_fullsized":"2024-03-22T01:36:38.500472+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:03:09.538334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.314794+0000","last_change":"2024-03-22T01:34:14.467372+0000","last_active":"2024-03-22T01:36:38.314794+0000","last_peered":"2024-03-22T01:36:38.314794+0000","last_clean":"2024-03-22T01:36:38.314794+0000","last_became_active":"2024-03-22T01:34:14.466124+0000","last_became_peered":"2024-03-22T01:34:14.466124+0000","last_unstale":"2024-03-22T01:36:38.314794+0000","last_undegraded":"2024-03-22T01:36:38.314794+0000","last_fullsized":"2024-03-22T01:36:38.314794+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:15:41.804735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"93'15","reported_seq":46,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.300521+0000","last_change":"2024-03-22T01:34:15.491533+0000","last_active":"2024-03-22T01:36:38.300521+0000","last_peered":"2024-03-22T01:36:38.300521+0000","last_clean":"2024-03-22T01:36:38.300521+0000","last_became_active":"2024-03-22T01:34:15.491208+0000","last_became_peered":"2024-03-22T01:34:15.491208+0000","last_unstale":"2024-03-22T01:36:38.300521+0000","last_undegraded":"2024-03-22T01:36:38.300521+0000","last_fullsized":"2024-03-22T01:36:38.300521+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:03:07.800607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":24,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.306930+0000","last_change":"2024-03-22T01:34:19.533419+0000","last_active":"2024-03-22T01:36:38.306930+0000","last_peered":"2024-03-22T01:36:38.306930+0000","last_clean":"2024-03-22T01:36:38.306930+0000","last_became_active":"2024-03-22T01:34:19.530042+0000","last_became_peered":"2024-03-22T01:34:19.530042+0000","last_unstale":"2024-03-22T01:36:38.306930+0000","last_undegraded":"2024-03-22T01:36:38.306930+0000","last_fullsized":"2024-03-22T01:36:38.306930+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:46:59.859742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.499888+0000","last_change":"2024-03-22T01:34:14.467033+0000","last_active":"2024-03-22T01:36:38.499888+0000","last_peered":"2024-03-22T01:36:38.499888+0000","last_clean":"2024-03-22T01:36:38.499888+0000","last_became_active":"2024-03-22T01:34:14.466649+0000","last_became_peered":"2024-03-22T01:34:14.466649+0000","last_unstale":"2024-03-22T01:36:38.499888+0000","last_undegraded":"2024-03-22T01:36:38.499888+0000","last_fullsized":"2024-03-22T01:36:38.499888+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:38:44.890899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"93'6","reported_seq":43,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.300199+0000","last_change":"2024-03-22T01:35:16.815302+0000","last_active":"2024-03-22T01:36:38.300199+0000","last_peered":"2024-03-22T01:36:38.300199+0000","last_clean":"2024-03-22T01:36:38.300199+0000","last_became_active":"2024-03-22T01:35:16.814988+0000","last_became_peered":"2024-03-22T01:35:16.814988+0000","last_unstale":"2024-03-22T01:36:38.300199+0000","last_undegraded":"2024-03-22T01:36:38.300199+0000","last_fullsized":"2024-03-22T01:36:38.300199+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:54:47.793437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":24,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.189238+0000","last_change":"2024-03-22T01:34:19.531417+0000","last_active":"2024-03-22T01:36:38.189238+0000","last_peered":"2024-03-22T01:36:38.189238+0000","last_clean":"2024-03-22T01:36:38.189238+0000","last_became_active":"2024-03-22T01:34:19.531003+0000","last_became_peered":"2024-03-22T01:34:19.531003+0000","last_unstale":"2024-03-22T01:36:38.189238+0000","last_undegraded":"2024-03-22T01:36:38.189238+0000","last_fullsized":"2024-03-22T01:36:38.189238+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:33:52.865110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"86'8","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.307580+0000","last_change":"2024-03-22T01:34:14.472904+0000","last_active":"2024-03-22T01:36:38.307580+0000","last_peered":"2024-03-22T01:36:38.307580+0000","last_clean":"2024-03-22T01:36:38.307580+0000","last_became_active":"2024-03-22T01:34:14.472442+0000","last_became_peered":"2024-03-22T01:34:14.472442+0000","last_unstale":"2024-03-22T01:36:38.307580+0000","last_undegraded":"2024-03-22T01:36:38.307580+0000","last_fullsized":"2024-03-22T01:36:38.307580+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:57:45.478316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"93'6","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.314972+0000","last_change":"2024-03-22T01:34:15.477411+0000","last_active":"2024-03-22T01:36:38.314972+0000","last_peered":"2024-03-22T01:36:38.314972+0000","last_clean":"2024-03-22T01:36:38.314972+0000","last_became_active":"2024-03-22T01:34:15.477035+0000","last_became_peered":"2024-03-22T01:34:15.477035+0000","last_unstale":"2024-03-22T01:36:38.314972+0000","last_undegraded":"2024-03-22T01:36:38.314972+0000","last_fullsized":"2024-03-22T01:36:38.314972+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:33:13.410872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.314593+0000","last_change":"2024-03-22T01:34:19.528258+0000","last_active":"2024-03-22T01:36:38.314593+0000","last_peered":"2024-03-22T01:36:38.314593+0000","last_clean":"2024-03-22T01:36:38.314593+0000","last_became_active":"2024-03-22T01:34:19.527668+0000","last_became_peered":"2024-03-22T01:34:19.527668+0000","last_unstale":"2024-03-22T01:36:38.314593+0000","last_undegraded":"2024-03-22T01:36:38.314593+0000","last_fullsized":"2024-03-22T01:36:38.314593+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:34:19.162609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'1","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.308046+0000","last_change":"2024-03-22T01:34:14.464735+0000","last_active":"2024-03-22T01:36:38.308046+0000","last_peered":"2024-03-22T01:36:38.308046+0000","last_clean":"2024-03-22T01:36:38.308046+0000","last_became_active":"2024-03-22T01:34:14.464359+0000","last_became_peered":"2024-03-22T01:34:14.464359+0000","last_unstale":"2024-03-22T01:36:38.308046+0000","last_undegraded":"2024-03-22T01:36:38.308046+0000","last_fullsized":"2024-03-22T01:36:38.308046+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:21:18.354922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"93'5","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500679+0000","last_change":"2024-03-22T01:35:16.752884+0000","last_active":"2024-03-22T01:36:38.500679+0000","last_peered":"2024-03-22T01:36:38.500679+0000","last_clean":"2024-03-22T01:36:38.500679+0000","last_became_active":"2024-03-22T01:35:16.751355+0000","last_became_peered":"2024-03-22T01:35:16.751355+0000","last_unstale":"2024-03-22T01:36:38.500679+0000","last_undegraded":"2024-03-22T01:36:38.500679+0000","last_fullsized":"2024-03-22T01:36:38.500679+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:49:33.261481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":25,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.307644+0000","last_change":"2024-03-22T01:34:19.668136+0000","last_active":"2024-03-22T01:36:38.307644+0000","last_peered":"2024-03-22T01:36:38.307644+0000","last_clean":"2024-03-22T01:36:38.307644+0000","last_became_active":"2024-03-22T01:34:19.667633+0000","last_became_peered":"2024-03-22T01:34:19.667633+0000","last_unstale":"2024-03-22T01:36:38.307644+0000","last_undegraded":"2024-03-22T01:36:38.307644+0000","last_fullsized":"2024-03-22T01:36:38.307644+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:07:26.455373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.305067+0000","last_change":"2024-03-22T01:34:14.471959+0000","last_active":"2024-03-22T01:36:38.305067+0000","last_peered":"2024-03-22T01:36:38.305067+0000","last_clean":"2024-03-22T01:36:38.305067+0000","last_became_active":"2024-03-22T01:34:14.471618+0000","last_became_peered":"2024-03-22T01:34:14.471618+0000","last_unstale":"2024-03-22T01:36:38.305067+0000","last_undegraded":"2024-03-22T01:36:38.305067+0000","last_fullsized":"2024-03-22T01:36:38.305067+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:45:03.537013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"93'4","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.192471+0000","last_change":"2024-03-22T01:34:15.495770+0000","last_active":"2024-03-22T01:36:38.192471+0000","last_peered":"2024-03-22T01:36:38.192471+0000","last_clean":"2024-03-22T01:36:38.192471+0000","last_became_active":"2024-03-22T01:34:15.495403+0000","last_became_peered":"2024-03-22T01:34:15.495403+0000","last_unstale":"2024-03-22T01:36:38.192471+0000","last_undegraded":"2024-03-22T01:36:38.192471+0000","last_fullsized":"2024-03-22T01:36:38.192471+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:48:05.184305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":25,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.189283+0000","last_change":"2024-03-22T01:34:19.520204+0000","last_active":"2024-03-22T01:36:38.189283+0000","last_peered":"2024-03-22T01:36:38.189283+0000","last_clean":"2024-03-22T01:36:38.189283+0000","last_became_active":"2024-03-22T01:34:19.519847+0000","last_became_peered":"2024-03-22T01:34:19.519847+0000","last_unstale":"2024-03-22T01:36:38.189283+0000","last_undegraded":"2024-03-22T01:36:38.189283+0000","last_fullsized":"2024-03-22T01:36:38.189283+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:06:21.389636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.300364+0000","last_change":"2024-03-22T01:34:14.464162+0000","last_active":"2024-03-22T01:36:38.300364+0000","last_peered":"2024-03-22T01:36:38.300364+0000","last_clean":"2024-03-22T01:36:38.300364+0000","last_became_active":"2024-03-22T01:34:14.462700+0000","last_became_peered":"2024-03-22T01:34:14.462700+0000","last_unstale":"2024-03-22T01:36:38.300364+0000","last_undegraded":"2024-03-22T01:36:38.300364+0000","last_fullsized":"2024-03-22T01:36:38.300364+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:45:32.262466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"93'7","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.192324+0000","last_change":"2024-03-22T01:34:15.488256+0000","last_active":"2024-03-22T01:36:38.192324+0000","last_peered":"2024-03-22T01:36:38.192324+0000","last_clean":"2024-03-22T01:36:38.192324+0000","last_became_active":"2024-03-22T01:34:15.487939+0000","last_became_peered":"2024-03-22T01:34:15.487939+0000","last_unstale":"2024-03-22T01:36:38.192324+0000","last_undegraded":"2024-03-22T01:36:38.192324+0000","last_fullsized":"2024-03-22T01:36:38.192324+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:30:49.323531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.189196+0000","last_change":"2024-03-22T01:34:19.667633+0000","last_active":"2024-03-22T01:36:38.189196+0000","last_peered":"2024-03-22T01:36:38.189196+0000","last_clean":"2024-03-22T01:36:38.189196+0000","last_became_active":"2024-03-22T01:34:19.667199+0000","last_became_peered":"2024-03-22T01:34:19.667199+0000","last_unstale":"2024-03-22T01:36:38.189196+0000","last_undegraded":"2024-03-22T01:36:38.189196+0000","last_fullsized":"2024-03-22T01:36:38.189196+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T01:58:13.459475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"93'8","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.314599+0000","last_change":"2024-03-22T01:34:15.489468+0000","last_active":"2024-03-22T01:36:38.314599+0000","last_peered":"2024-03-22T01:36:38.314599+0000","last_clean":"2024-03-22T01:36:38.314599+0000","last_became_active":"2024-03-22T01:34:15.489029+0000","last_became_peered":"2024-03-22T01:34:15.489029+0000","last_unstale":"2024-03-22T01:36:38.314599+0000","last_undegraded":"2024-03-22T01:36:38.314599+0000","last_fullsized":"2024-03-22T01:36:38.314599+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:00:01.370161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.188685+0000","last_change":"2024-03-22T01:34:14.464552+0000","last_active":"2024-03-22T01:36:38.188685+0000","last_peered":"2024-03-22T01:36:38.188685+0000","last_clean":"2024-03-22T01:36:38.188685+0000","last_became_active":"2024-03-22T01:34:14.463554+0000","last_became_peered":"2024-03-22T01:34:14.463554+0000","last_unstale":"2024-03-22T01:36:38.188685+0000","last_undegraded":"2024-03-22T01:36:38.188685+0000","last_fullsized":"2024-03-22T01:36:38.188685+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:51:16.709278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"91'1","reported_seq":26,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.300450+0000","last_change":"2024-03-22T01:34:19.526066+0000","last_active":"2024-03-22T01:36:38.300450+0000","last_peered":"2024-03-22T01:36:38.300450+0000","last_clean":"2024-03-22T01:36:38.300450+0000","last_became_active":"2024-03-22T01:34:19.525659+0000","last_became_peered":"2024-03-22T01:34:19.525659+0000","last_unstale":"2024-03-22T01:36:38.300450+0000","last_undegraded":"2024-03-22T01:36:38.300450+0000","last_fullsized":"2024-03-22T01:36:38.300450+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:06:26.287971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"93'8","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.307732+0000","last_change":"2024-03-22T01:34:15.477018+0000","last_active":"2024-03-22T01:36:38.307732+0000","last_peered":"2024-03-22T01:36:38.307732+0000","last_clean":"2024-03-22T01:36:38.307732+0000","last_became_active":"2024-03-22T01:34:15.476613+0000","last_became_peered":"2024-03-22T01:34:15.476613+0000","last_unstale":"2024-03-22T01:36:38.307732+0000","last_undegraded":"2024-03-22T01:36:38.307732+0000","last_fullsized":"2024-03-22T01:36:38.307732+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:10:07.980123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.499827+0000","last_change":"2024-03-22T01:34:14.470922+0000","last_active":"2024-03-22T01:36:38.499827+0000","last_peered":"2024-03-22T01:36:38.499827+0000","last_clean":"2024-03-22T01:36:38.499827+0000","last_became_active":"2024-03-22T01:34:14.470515+0000","last_became_peered":"2024-03-22T01:34:14.470515+0000","last_unstale":"2024-03-22T01:36:38.499827+0000","last_undegraded":"2024-03-22T01:36:38.499827+0000","last_fullsized":"2024-03-22T01:36:38.499827+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:52:50.203631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":365,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.300114+0000","last_change":"2024-03-22T01:33:03.615234+0000","last_active":"2024-03-22T01:36:38.300114+0000","last_peered":"2024-03-22T01:36:38.300114+0000","last_clean":"2024-03-22T01:36:38.300114+0000","last_became_active":"2024-03-22T01:33:03.614710+0000","last_became_peered":"2024-03-22T01:33:03.614710+0000","last_unstale":"2024-03-22T01:36:38.300114+0000","last_undegraded":"2024-03-22T01:36:38.300114+0000","last_fullsized":"2024-03-22T01:36:38.300114+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:29:38.312292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:29:38.312292+0000","last_clean_scrub_stamp":"2024-03-22T01:29:38.312292+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:56:42.348119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.314929+0000","last_change":"2024-03-22T01:34:19.668565+0000","last_active":"2024-03-22T01:36:38.314929+0000","last_peered":"2024-03-22T01:36:38.314929+0000","last_clean":"2024-03-22T01:36:38.314929+0000","last_became_active":"2024-03-22T01:34:19.667959+0000","last_became_peered":"2024-03-22T01:34:19.667959+0000","last_unstale":"2024-03-22T01:36:38.314929+0000","last_undegraded":"2024-03-22T01:36:38.314929+0000","last_fullsized":"2024-03-22T01:36:38.314929+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:13:04.447486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"93'5","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.188685+0000","last_change":"2024-03-22T01:34:15.489513+0000","last_active":"2024-03-22T01:36:38.188685+0000","last_peered":"2024-03-22T01:36:38.188685+0000","last_clean":"2024-03-22T01:36:38.188685+0000","last_became_active":"2024-03-22T01:34:15.489188+0000","last_became_peered":"2024-03-22T01:34:15.489188+0000","last_unstale":"2024-03-22T01:36:38.188685+0000","last_undegraded":"2024-03-22T01:36:38.188685+0000","last_fullsized":"2024-03-22T01:36:38.188685+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:08:24.859350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.300274+0000","last_change":"2024-03-22T01:34:14.466305+0000","last_active":"2024-03-22T01:36:38.300274+0000","last_peered":"2024-03-22T01:36:38.300274+0000","last_clean":"2024-03-22T01:36:38.300274+0000","last_became_active":"2024-03-22T01:34:14.465495+0000","last_became_peered":"2024-03-22T01:34:14.465495+0000","last_unstale":"2024-03-22T01:36:38.300274+0000","last_undegraded":"2024-03-22T01:36:38.300274+0000","last_fullsized":"2024-03-22T01:36:38.300274+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:45:13.481187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500369+0000","last_change":"2024-03-22T01:34:19.526970+0000","last_active":"2024-03-22T01:36:38.500369+0000","last_peered":"2024-03-22T01:36:38.500369+0000","last_clean":"2024-03-22T01:36:38.500369+0000","last_became_active":"2024-03-22T01:34:19.526462+0000","last_became_peered":"2024-03-22T01:34:19.526462+0000","last_unstale":"2024-03-22T01:36:38.500369+0000","last_undegraded":"2024-03-22T01:36:38.500369+0000","last_fullsized":"2024-03-22T01:36:38.500369+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:04:18.417358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500521+0000","last_change":"2024-03-22T01:35:16.750746+0000","last_active":"2024-03-22T01:36:38.500521+0000","last_peered":"2024-03-22T01:36:38.500521+0000","last_clean":"2024-03-22T01:36:38.500521+0000","last_became_active":"2024-03-22T01:35:16.750271+0000","last_became_peered":"2024-03-22T01:35:16.750271+0000","last_unstale":"2024-03-22T01:36:38.500521+0000","last_undegraded":"2024-03-22T01:36:38.500521+0000","last_fullsized":"2024-03-22T01:36:38.500521+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:18:10.839351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"93'9","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.189402+0000","last_change":"2024-03-22T01:34:15.490405+0000","last_active":"2024-03-22T01:36:38.189402+0000","last_peered":"2024-03-22T01:36:38.189402+0000","last_clean":"2024-03-22T01:36:38.189402+0000","last_became_active":"2024-03-22T01:34:15.489921+0000","last_became_peered":"2024-03-22T01:34:15.489921+0000","last_unstale":"2024-03-22T01:36:38.189402+0000","last_undegraded":"2024-03-22T01:36:38.189402+0000","last_fullsized":"2024-03-22T01:36:38.189402+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:27:28.804649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.307314+0000","last_change":"2024-03-22T01:34:19.667651+0000","last_active":"2024-03-22T01:36:38.307314+0000","last_peered":"2024-03-22T01:36:38.307314+0000","last_clean":"2024-03-22T01:36:38.307314+0000","last_became_active":"2024-03-22T01:34:19.667197+0000","last_became_peered":"2024-03-22T01:34:19.667197+0000","last_unstale":"2024-03-22T01:36:38.307314+0000","last_undegraded":"2024-03-22T01:36:38.307314+0000","last_fullsized":"2024-03-22T01:36:38.307314+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:32:55.874098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"96'43","reported_seq":100,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.424707+0000","last_change":"2024-03-22T01:34:14.474988+0000","last_active":"2024-03-22T01:36:38.424707+0000","last_peered":"2024-03-22T01:36:38.424707+0000","last_clean":"2024-03-22T01:36:38.424707+0000","last_became_active":"2024-03-22T01:34:14.474651+0000","last_became_peered":"2024-03-22T01:34:14.474651+0000","last_unstale":"2024-03-22T01:36:38.424707+0000","last_undegraded":"2024-03-22T01:36:38.424707+0000","last_fullsized":"2024-03-22T01:36:38.424707+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:14:59.846727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":77,"num_write_kb":837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"93'5","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.305472+0000","last_change":"2024-03-22T01:35:16.749643+0000","last_active":"2024-03-22T01:36:38.305472+0000","last_peered":"2024-03-22T01:36:38.305472+0000","last_clean":"2024-03-22T01:36:38.305472+0000","last_became_active":"2024-03-22T01:35:16.749311+0000","last_became_peered":"2024-03-22T01:35:16.749311+0000","last_unstale":"2024-03-22T01:36:38.305472+0000","last_undegraded":"2024-03-22T01:36:38.305472+0000","last_fullsized":"2024-03-22T01:36:38.305472+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:23:18.666007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,7],"acting":[9,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.190868+0000","last_change":"2024-03-22T01:34:19.523959+0000","last_active":"2024-03-22T01:36:38.190868+0000","last_peered":"2024-03-22T01:36:38.190868+0000","last_clean":"2024-03-22T01:36:38.190868+0000","last_became_active":"2024-03-22T01:34:19.523556+0000","last_became_peered":"2024-03-22T01:34:19.523556+0000","last_unstale":"2024-03-22T01:36:38.190868+0000","last_undegraded":"2024-03-22T01:36:38.190868+0000","last_fullsized":"2024-03-22T01:36:38.190868+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:22:21.623042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"86'2","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.499880+0000","last_change":"2024-03-22T01:34:14.463434+0000","last_active":"2024-03-22T01:36:38.499880+0000","last_peered":"2024-03-22T01:36:38.499880+0000","last_clean":"2024-03-22T01:36:38.499880+0000","last_became_active":"2024-03-22T01:34:14.463075+0000","last_became_peered":"2024-03-22T01:34:14.463075+0000","last_unstale":"2024-03-22T01:36:38.499880+0000","last_undegraded":"2024-03-22T01:36:38.499880+0000","last_fullsized":"2024-03-22T01:36:38.499880+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:43:18.223766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"93'7","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.502109+0000","last_change":"2024-03-22T01:34:15.482695+0000","last_active":"2024-03-22T01:36:38.502109+0000","last_peered":"2024-03-22T01:36:38.502109+0000","last_clean":"2024-03-22T01:36:38.502109+0000","last_became_active":"2024-03-22T01:34:15.481881+0000","last_became_peered":"2024-03-22T01:34:15.481881+0000","last_unstale":"2024-03-22T01:36:38.502109+0000","last_undegraded":"2024-03-22T01:36:38.502109+0000","last_fullsized":"2024-03-22T01:36:38.502109+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:44:26.290714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":25,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500458+0000","last_change":"2024-03-22T01:34:19.668252+0000","last_active":"2024-03-22T01:36:38.500458+0000","last_peered":"2024-03-22T01:36:38.500458+0000","last_clean":"2024-03-22T01:36:38.500458+0000","last_became_active":"2024-03-22T01:34:19.667830+0000","last_became_peered":"2024-03-22T01:34:19.667830+0000","last_unstale":"2024-03-22T01:36:38.500458+0000","last_undegraded":"2024-03-22T01:36:38.500458+0000","last_fullsized":"2024-03-22T01:36:38.500458+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:08:29.602899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.459520+0000","last_change":"2024-03-22T01:34:14.473020+0000","last_active":"2024-03-22T01:36:38.459520+0000","last_peered":"2024-03-22T01:36:38.459520+0000","last_clean":"2024-03-22T01:36:38.459520+0000","last_became_active":"2024-03-22T01:34:14.472689+0000","last_became_peered":"2024-03-22T01:34:14.472689+0000","last_unstale":"2024-03-22T01:36:38.459520+0000","last_undegraded":"2024-03-22T01:36:38.459520+0000","last_fullsized":"2024-03-22T01:36:38.459520+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:04:19.724685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"93'2","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.307503+0000","last_change":"2024-03-22T01:34:15.491613+0000","last_active":"2024-03-22T01:36:38.307503+0000","last_peered":"2024-03-22T01:36:38.307503+0000","last_clean":"2024-03-22T01:36:38.307503+0000","last_became_active":"2024-03-22T01:34:15.490922+0000","last_became_peered":"2024-03-22T01:34:15.490922+0000","last_unstale":"2024-03-22T01:36:38.307503+0000","last_undegraded":"2024-03-22T01:36:38.307503+0000","last_fullsized":"2024-03-22T01:36:38.307503+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:09:33.177646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":25,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.306867+0000","last_change":"2024-03-22T01:34:19.534537+0000","last_active":"2024-03-22T01:36:38.306867+0000","last_peered":"2024-03-22T01:36:38.306867+0000","last_clean":"2024-03-22T01:36:38.306867+0000","last_became_active":"2024-03-22T01:34:19.534038+0000","last_became_peered":"2024-03-22T01:34:19.534038+0000","last_unstale":"2024-03-22T01:36:38.306867+0000","last_undegraded":"2024-03-22T01:36:38.306867+0000","last_fullsized":"2024-03-22T01:36:38.306867+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:59:26.020000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.307492+0000","last_change":"2024-03-22T01:34:14.464724+0000","last_active":"2024-03-22T01:36:38.307492+0000","last_peered":"2024-03-22T01:36:38.307492+0000","last_clean":"2024-03-22T01:36:38.307492+0000","last_became_active":"2024-03-22T01:34:14.464056+0000","last_became_peered":"2024-03-22T01:34:14.464056+0000","last_unstale":"2024-03-22T01:36:38.307492+0000","last_undegraded":"2024-03-22T01:36:38.307492+0000","last_fullsized":"2024-03-22T01:36:38.307492+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:49:39.098126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"93'9","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500261+0000","last_change":"2024-03-22T01:34:15.488934+0000","last_active":"2024-03-22T01:36:38.500261+0000","last_peered":"2024-03-22T01:36:38.500261+0000","last_clean":"2024-03-22T01:36:38.500261+0000","last_became_active":"2024-03-22T01:34:15.488585+0000","last_became_peered":"2024-03-22T01:34:15.488585+0000","last_unstale":"2024-03-22T01:36:38.500261+0000","last_undegraded":"2024-03-22T01:36:38.500261+0000","last_fullsized":"2024-03-22T01:36:38.500261+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:59:04.554614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"91'1","reported_seq":26,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500321+0000","last_change":"2024-03-22T01:34:19.534096+0000","last_active":"2024-03-22T01:36:38.500321+0000","last_peered":"2024-03-22T01:36:38.500321+0000","last_clean":"2024-03-22T01:36:38.500321+0000","last_became_active":"2024-03-22T01:34:19.533645+0000","last_became_peered":"2024-03-22T01:34:19.533645+0000","last_unstale":"2024-03-22T01:36:38.500321+0000","last_undegraded":"2024-03-22T01:36:38.500321+0000","last_fullsized":"2024-03-22T01:36:38.500321+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:50:21.259146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.190987+0000","last_change":"2024-03-22T01:34:14.469529+0000","last_active":"2024-03-22T01:36:38.190987+0000","last_peered":"2024-03-22T01:36:38.190987+0000","last_clean":"2024-03-22T01:36:38.190987+0000","last_became_active":"2024-03-22T01:34:14.469204+0000","last_became_peered":"2024-03-22T01:34:14.469204+0000","last_unstale":"2024-03-22T01:36:38.190987+0000","last_undegraded":"2024-03-22T01:36:38.190987+0000","last_fullsized":"2024-03-22T01:36:38.190987+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:39:57.048537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"93'8","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.307968+0000","last_change":"2024-03-22T01:34:15.501324+0000","last_active":"2024-03-22T01:36:38.307968+0000","last_peered":"2024-03-22T01:36:38.307968+0000","last_clean":"2024-03-22T01:36:38.307968+0000","last_became_active":"2024-03-22T01:34:15.501015+0000","last_became_peered":"2024-03-22T01:34:15.501015+0000","last_unstale":"2024-03-22T01:36:38.307968+0000","last_undegraded":"2024-03-22T01:36:38.307968+0000","last_fullsized":"2024-03-22T01:36:38.307968+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:35:27.673019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500540+0000","last_change":"2024-03-22T01:34:19.667806+0000","last_active":"2024-03-22T01:36:38.500540+0000","last_peered":"2024-03-22T01:36:38.500540+0000","last_clean":"2024-03-22T01:36:38.500540+0000","last_became_active":"2024-03-22T01:34:19.667249+0000","last_became_peered":"2024-03-22T01:34:19.667249+0000","last_unstale":"2024-03-22T01:36:38.500540+0000","last_undegraded":"2024-03-22T01:36:38.500540+0000","last_fullsized":"2024-03-22T01:36:38.500540+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:44:22.476882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"86'2","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.307691+0000","last_change":"2024-03-22T01:34:14.471800+0000","last_active":"2024-03-22T01:36:38.307691+0000","last_peered":"2024-03-22T01:36:38.307691+0000","last_clean":"2024-03-22T01:36:38.307691+0000","last_became_active":"2024-03-22T01:34:14.471422+0000","last_became_peered":"2024-03-22T01:34:14.471422+0000","last_unstale":"2024-03-22T01:36:38.307691+0000","last_undegraded":"2024-03-22T01:36:38.307691+0000","last_fullsized":"2024-03-22T01:36:38.307691+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:25:58.546331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"93'5","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.300283+0000","last_change":"2024-03-22T01:34:15.487348+0000","last_active":"2024-03-22T01:36:38.300283+0000","last_peered":"2024-03-22T01:36:38.300283+0000","last_clean":"2024-03-22T01:36:38.300283+0000","last_became_active":"2024-03-22T01:34:15.486788+0000","last_became_peered":"2024-03-22T01:34:15.486788+0000","last_unstale":"2024-03-22T01:36:38.300283+0000","last_undegraded":"2024-03-22T01:36:38.300283+0000","last_fullsized":"2024-03-22T01:36:38.300283+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:10:31.489021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.459783+0000","last_change":"2024-03-22T01:34:19.521490+0000","last_active":"2024-03-22T01:36:38.459783+0000","last_peered":"2024-03-22T01:36:38.459783+0000","last_clean":"2024-03-22T01:36:38.459783+0000","last_became_active":"2024-03-22T01:34:19.521047+0000","last_became_peered":"2024-03-22T01:34:19.521047+0000","last_unstale":"2024-03-22T01:36:38.459783+0000","last_undegraded":"2024-03-22T01:36:38.459783+0000","last_fullsized":"2024-03-22T01:36:38.459783+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:36:50.176756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":17,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.459782+0000","last_change":"2024-03-22T01:35:16.749731+0000","last_active":"2024-03-22T01:36:38.459782+0000","last_peered":"2024-03-22T01:36:38.459782+0000","last_clean":"2024-03-22T01:36:38.459782+0000","last_became_active":"2024-03-22T01:35:16.749329+0000","last_became_peered":"2024-03-22T01:35:16.749329+0000","last_unstale":"2024-03-22T01:36:38.459782+0000","last_undegraded":"2024-03-22T01:36:38.459782+0000","last_fullsized":"2024-03-22T01:36:38.459782+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:23:27.386061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"93'7","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.300670+0000","last_change":"2024-03-22T01:34:15.491683+0000","last_active":"2024-03-22T01:36:38.300670+0000","last_peered":"2024-03-22T01:36:38.300670+0000","last_clean":"2024-03-22T01:36:38.300670+0000","last_became_active":"2024-03-22T01:34:15.491030+0000","last_became_peered":"2024-03-22T01:34:15.491030+0000","last_unstale":"2024-03-22T01:36:38.300670+0000","last_undegraded":"2024-03-22T01:36:38.300670+0000","last_fullsized":"2024-03-22T01:36:38.300670+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:02:10.115971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500471+0000","last_change":"2024-03-22T01:36:17.767646+0000","last_active":"2024-03-22T01:36:38.500471+0000","last_peered":"2024-03-22T01:36:38.500471+0000","last_clean":"2024-03-22T01:36:38.500471+0000","last_became_active":"2024-03-22T01:36:17.767257+0000","last_became_peered":"2024-03-22T01:36:17.767257+0000","last_unstale":"2024-03-22T01:36:38.500471+0000","last_undegraded":"2024-03-22T01:36:38.500471+0000","last_fullsized":"2024-03-22T01:36:38.500471+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:16:11.523941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,1,4],"acting":[0,8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"96'39","reported_seq":79,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.419218+0000","last_change":"2024-03-22T01:34:14.467755+0000","last_active":"2024-03-22T01:36:38.419218+0000","last_peered":"2024-03-22T01:36:38.419218+0000","last_clean":"2024-03-22T01:36:38.419218+0000","last_became_active":"2024-03-22T01:34:14.467426+0000","last_became_peered":"2024-03-22T01:34:14.467426+0000","last_unstale":"2024-03-22T01:36:38.419218+0000","last_undegraded":"2024-03-22T01:36:38.419218+0000","last_fullsized":"2024-03-22T01:36:38.419218+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:16:35.436872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"93'6","reported_seq":43,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.300373+0000","last_change":"2024-03-22T01:35:16.748667+0000","last_active":"2024-03-22T01:36:38.300373+0000","last_peered":"2024-03-22T01:36:38.300373+0000","last_clean":"2024-03-22T01:36:38.300373+0000","last_became_active":"2024-03-22T01:35:16.748303+0000","last_became_peered":"2024-03-22T01:35:16.748303+0000","last_unstale":"2024-03-22T01:36:38.300373+0000","last_undegraded":"2024-03-22T01:36:38.300373+0000","last_fullsized":"2024-03-22T01:36:38.300373+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:40:55.207127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.314390+0000","last_change":"2024-03-22T01:34:19.523631+0000","last_active":"2024-03-22T01:36:38.314390+0000","last_peered":"2024-03-22T01:36:38.314390+0000","last_clean":"2024-03-22T01:36:38.314390+0000","last_became_active":"2024-03-22T01:34:19.522968+0000","last_became_peered":"2024-03-22T01:34:19.522968+0000","last_unstale":"2024-03-22T01:36:38.314390+0000","last_undegraded":"2024-03-22T01:36:38.314390+0000","last_fullsized":"2024-03-22T01:36:38.314390+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:27:24.182886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.189345+0000","last_change":"2024-03-22T01:34:14.464113+0000","last_active":"2024-03-22T01:36:38.189345+0000","last_peered":"2024-03-22T01:36:38.189345+0000","last_clean":"2024-03-22T01:36:38.189345+0000","last_became_active":"2024-03-22T01:34:14.463389+0000","last_became_peered":"2024-03-22T01:34:14.463389+0000","last_unstale":"2024-03-22T01:36:38.189345+0000","last_undegraded":"2024-03-22T01:36:38.189345+0000","last_fullsized":"2024-03-22T01:36:38.189345+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:08:42.376753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"93'9","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.300652+0000","last_change":"2024-03-22T01:34:15.492941+0000","last_active":"2024-03-22T01:36:38.300652+0000","last_peered":"2024-03-22T01:36:38.300652+0000","last_clean":"2024-03-22T01:36:38.300652+0000","last_became_active":"2024-03-22T01:34:15.492598+0000","last_became_peered":"2024-03-22T01:34:15.492598+0000","last_unstale":"2024-03-22T01:36:38.300652+0000","last_undegraded":"2024-03-22T01:36:38.300652+0000","last_fullsized":"2024-03-22T01:36:38.300652+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:22:35.502404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"93'9","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500326+0000","last_change":"2024-03-22T01:34:15.494548+0000","last_active":"2024-03-22T01:36:38.500326+0000","last_peered":"2024-03-22T01:36:38.500326+0000","last_clean":"2024-03-22T01:36:38.500326+0000","last_became_active":"2024-03-22T01:34:15.494228+0000","last_became_peered":"2024-03-22T01:34:15.494228+0000","last_unstale":"2024-03-22T01:36:38.500326+0000","last_undegraded":"2024-03-22T01:36:38.500326+0000","last_fullsized":"2024-03-22T01:36:38.500326+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:43:26.011030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"93'12","reported_seq":44,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.191036+0000","last_change":"2024-03-22T01:34:15.490843+0000","last_active":"2024-03-22T01:36:38.191036+0000","last_peered":"2024-03-22T01:36:38.191036+0000","last_clean":"2024-03-22T01:36:38.191036+0000","last_became_active":"2024-03-22T01:34:15.490474+0000","last_became_peered":"2024-03-22T01:34:15.490474+0000","last_unstale":"2024-03-22T01:36:38.191036+0000","last_undegraded":"2024-03-22T01:36:38.191036+0000","last_fullsized":"2024-03-22T01:36:38.191036+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:11:31.488132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"93'8","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.190920+0000","last_change":"2024-03-22T01:34:15.502213+0000","last_active":"2024-03-22T01:36:38.190920+0000","last_peered":"2024-03-22T01:36:38.190920+0000","last_clean":"2024-03-22T01:36:38.190920+0000","last_became_active":"2024-03-22T01:34:15.501464+0000","last_became_peered":"2024-03-22T01:34:15.501464+0000","last_unstale":"2024-03-22T01:36:38.190920+0000","last_undegraded":"2024-03-22T01:36:38.190920+0000","last_fullsized":"2024-03-22T01:36:38.190920+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:38:02.090365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"93'7","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.501571+0000","last_change":"2024-03-22T01:34:15.481282+0000","last_active":"2024-03-22T01:36:38.501571+0000","last_peered":"2024-03-22T01:36:38.501571+0000","last_clean":"2024-03-22T01:36:38.501571+0000","last_became_active":"2024-03-22T01:34:15.480465+0000","last_became_peered":"2024-03-22T01:34:15.480465+0000","last_unstale":"2024-03-22T01:36:38.501571+0000","last_undegraded":"2024-03-22T01:36:38.501571+0000","last_fullsized":"2024-03-22T01:36:38.501571+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:55:59.398543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"93'4","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.189395+0000","last_change":"2024-03-22T01:34:15.480910+0000","last_active":"2024-03-22T01:36:38.189395+0000","last_peered":"2024-03-22T01:36:38.189395+0000","last_clean":"2024-03-22T01:36:38.189395+0000","last_became_active":"2024-03-22T01:34:15.480601+0000","last_became_peered":"2024-03-22T01:34:15.480601+0000","last_unstale":"2024-03-22T01:36:38.189395+0000","last_undegraded":"2024-03-22T01:36:38.189395+0000","last_fullsized":"2024-03-22T01:36:38.189395+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:19:51.563598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"93'6","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500686+0000","last_change":"2024-03-22T01:34:15.490859+0000","last_active":"2024-03-22T01:36:38.500686+0000","last_peered":"2024-03-22T01:36:38.500686+0000","last_clean":"2024-03-22T01:36:38.500686+0000","last_became_active":"2024-03-22T01:34:15.490337+0000","last_became_peered":"2024-03-22T01:34:15.490337+0000","last_unstale":"2024-03-22T01:36:38.500686+0000","last_undegraded":"2024-03-22T01:36:38.500686+0000","last_fullsized":"2024-03-22T01:36:38.500686+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:46:07.072015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"93'15","reported_seq":47,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.191416+0000","last_change":"2024-03-22T01:34:15.495531+0000","last_active":"2024-03-22T01:36:38.191416+0000","last_peered":"2024-03-22T01:36:38.191416+0000","last_clean":"2024-03-22T01:36:38.191416+0000","last_became_active":"2024-03-22T01:34:15.495216+0000","last_became_peered":"2024-03-22T01:34:15.495216+0000","last_unstale":"2024-03-22T01:36:38.191416+0000","last_undegraded":"2024-03-22T01:36:38.191416+0000","last_fullsized":"2024-03-22T01:36:38.191416+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:39:22.575926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"93'2","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.308446+0000","last_change":"2024-03-22T01:34:15.499430+0000","last_active":"2024-03-22T01:36:38.308446+0000","last_peered":"2024-03-22T01:36:38.308446+0000","last_clean":"2024-03-22T01:36:38.308446+0000","last_became_active":"2024-03-22T01:34:15.499114+0000","last_became_peered":"2024-03-22T01:34:15.499114+0000","last_unstale":"2024-03-22T01:36:38.308446+0000","last_undegraded":"2024-03-22T01:36:38.308446+0000","last_fullsized":"2024-03-22T01:36:38.308446+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:23:01.781963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"93'6","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.314538+0000","last_change":"2024-03-22T01:34:15.485449+0000","last_active":"2024-03-22T01:36:38.314538+0000","last_peered":"2024-03-22T01:36:38.314538+0000","last_clean":"2024-03-22T01:36:38.314538+0000","last_became_active":"2024-03-22T01:34:15.485084+0000","last_became_peered":"2024-03-22T01:34:15.485084+0000","last_unstale":"2024-03-22T01:36:38.314538+0000","last_undegraded":"2024-03-22T01:36:38.314538+0000","last_fullsized":"2024-03-22T01:36:38.314538+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:53:25.632719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"93'8","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.307592+0000","last_change":"2024-03-22T01:34:15.495521+0000","last_active":"2024-03-22T01:36:38.307592+0000","last_peered":"2024-03-22T01:36:38.307592+0000","last_clean":"2024-03-22T01:36:38.307592+0000","last_became_active":"2024-03-22T01:34:15.495218+0000","last_became_peered":"2024-03-22T01:34:15.495218+0000","last_unstale":"2024-03-22T01:36:38.307592+0000","last_undegraded":"2024-03-22T01:36:38.307592+0000","last_fullsized":"2024-03-22T01:36:38.307592+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:04:32.569934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"93'10","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.460006+0000","last_change":"2024-03-22T01:34:15.489213+0000","last_active":"2024-03-22T01:36:38.460006+0000","last_peered":"2024-03-22T01:36:38.460006+0000","last_clean":"2024-03-22T01:36:38.460006+0000","last_became_active":"2024-03-22T01:34:15.488887+0000","last_became_peered":"2024-03-22T01:34:15.488887+0000","last_unstale":"2024-03-22T01:36:38.460006+0000","last_undegraded":"2024-03-22T01:36:38.460006+0000","last_fullsized":"2024-03-22T01:36:38.460006+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:53:39.983144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"93'11","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.189182+0000","last_change":"2024-03-22T01:34:15.485632+0000","last_active":"2024-03-22T01:36:38.189182+0000","last_peered":"2024-03-22T01:36:38.189182+0000","last_clean":"2024-03-22T01:36:38.189182+0000","last_became_active":"2024-03-22T01:34:15.485341+0000","last_became_peered":"2024-03-22T01:34:15.485341+0000","last_unstale":"2024-03-22T01:36:38.189182+0000","last_undegraded":"2024-03-22T01:36:38.189182+0000","last_fullsized":"2024-03-22T01:36:38.189182+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:33:00.248555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"93'8","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500583+0000","last_change":"2024-03-22T01:34:15.483593+0000","last_active":"2024-03-22T01:36:38.500583+0000","last_peered":"2024-03-22T01:36:38.500583+0000","last_clean":"2024-03-22T01:36:38.500583+0000","last_became_active":"2024-03-22T01:34:15.483201+0000","last_became_peered":"2024-03-22T01:34:15.483201+0000","last_unstale":"2024-03-22T01:36:38.500583+0000","last_undegraded":"2024-03-22T01:36:38.500583+0000","last_fullsized":"2024-03-22T01:36:38.500583+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:03:10.955120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":595682,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":348,"num_read_kb":279,"num_write":200,"num_write_kb":1410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1880064,"data_stored":1787046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318314,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32244,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712908,"statfs":{"total":95995035648,"available":95962017792,"internally_reserved":0,"allocated":1089536,"data_stored":538835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579951,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37064,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708088,"statfs":{"total":95995035648,"available":95957082112,"internally_reserved":0,"allocated":1765376,"data_stored":1189214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776180,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28476,"kb_used_data":1456,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716676,"statfs":{"total":95995035648,"available":95965876224,"internally_reserved":0,"allocated":1490944,"data_stored":936729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037816,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32636,"kb_used_data":1456,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712516,"statfs":{"total":95995035648,"available":95961616384,"internally_reserved":0,"allocated":1490944,"data_stored":932655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299454,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28672,"kb_used_data":1652,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716480,"statfs":{"total":95995035648,"available":95965675520,"internally_reserved":0,"allocated":1691648,"data_stored":1133881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462978,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1085440,"data_stored":534739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574939,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28156,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716996,"statfs":{"total":95995035648,"available":95966203904,"internally_reserved":0,"allocated":1163264,"data_stored":604807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738464,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1073152,"data_stored":524905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411414,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28160,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716992,"statfs":{"total":95995035648,"available":95966199808,"internally_reserved":0,"allocated":1167360,"data_stored":604841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149777,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36920,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708232,"statfs":{"total":95995035648,"available":95957229568,"internally_reserved":0,"allocated":1622016,"data_stored":1059659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888141,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1093632,"data_stored":538857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724616,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37304,"kb_used_data":1788,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93707848,"statfs":{"total":95995035648,"available":95956836352,"internally_reserved":0,"allocated":1830912,"data_stored":1260985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":70068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":70102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":536576,"data_stored":524920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":131782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":393820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":397312,"data_stored":393798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":225280,"data_stored":201862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-22T01:36:40.807 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-03-22T01:36:40.807 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-22T01:36:40.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.948+0000 7f58216d4640 1 -- 172.21.15.72:0/2691768119 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f581c101cb0 msgr2=0x7f581c102130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:40.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.948+0000 7f58216d4640 1 --2- 172.21.15.72:0/2691768119 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f581c101cb0 0x7f581c102130 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f5810009900 tx=0x7f581002bc30 comp rx=0 tx=0).stop 2024-03-22T01:36:40.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.948+0000 7f58216d4640 1 -- 172.21.15.72:0/2691768119 shutdown_connections 2024-03-22T01:36:40.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.948+0000 7f58216d4640 1 --2- 172.21.15.72:0/2691768119 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f581c102670 0x7f581c108f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:40.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.948+0000 7f58216d4640 1 --2- 172.21.15.72:0/2691768119 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f581c101cb0 0x7f581c102130 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:40.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.948+0000 7f58216d4640 1 --2- 172.21.15.72:0/2691768119 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f581c100ac0 0x7f581c100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:40.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.948+0000 7f58216d4640 1 -- 172.21.15.72:0/2691768119 >> 172.21.15.72:0/2691768119 conn(0x7f581c0fc030 msgr2=0x7f581c0fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:40.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.948+0000 7f58216d4640 1 -- 172.21.15.72:0/2691768119 shutdown_connections 2024-03-22T01:36:40.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.948+0000 7f58216d4640 1 -- 172.21.15.72:0/2691768119 wait complete. 2024-03-22T01:36:40.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.949+0000 7f58216d4640 1 Processor -- start 2024-03-22T01:36:40.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.949+0000 7f58216d4640 1 -- start start 2024-03-22T01:36:40.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.949+0000 7f58216d4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f581c100ac0 0x7f581c10b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:40.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.949+0000 7f58216d4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f581c101cb0 0x7f581c109a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:40.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.949+0000 7f58216d4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f581c102670 0x7f581c109fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:40.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.949+0000 7f58216d4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f581c10d860 con 0x7f581c100ac0 2024-03-22T01:36:40.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.949+0000 7f58216d4640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f581c10d6e0 con 0x7f581c102670 2024-03-22T01:36:40.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.949+0000 7f58216d4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f581c10d9e0 con 0x7f581c101cb0 2024-03-22T01:36:40.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.949+0000 7f581affd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f581c100ac0 0x7f581c10b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:40.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.949+0000 7f581affd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f581c100ac0 0x7f581c10b440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46738/0 (socket says 172.21.15.72:46738) 2024-03-22T01:36:40.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.949+0000 7f581affd640 1 -- 172.21.15.72:0/1067801262 learned_addr learned my addr 172.21.15.72:0/1067801262 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:40.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.949+0000 7f581a7fc640 1 --2- 172.21.15.72:0/1067801262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f581c101cb0 0x7f581c109a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:40.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.950+0000 7f581b7fe640 1 --2- 172.21.15.72:0/1067801262 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f581c102670 0x7f581c109fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:40.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.950+0000 7f581affd640 1 -- 172.21.15.72:0/1067801262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f581c101cb0 msgr2=0x7f581c109a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:40.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.950+0000 7f581affd640 1 --2- 172.21.15.72:0/1067801262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f581c101cb0 0x7f581c109a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:40.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.950+0000 7f581affd640 1 -- 172.21.15.72:0/1067801262 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f581c102670 msgr2=0x7f581c109fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:40.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.950+0000 7f581affd640 1 --2- 172.21.15.72:0/1067801262 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f581c102670 0x7f581c109fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:40.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.950+0000 7f581affd640 1 -- 172.21.15.72:0/1067801262 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f581c10a850 con 0x7f581c100ac0 2024-03-22T01:36:40.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.950+0000 7f581a7fc640 1 --2- 172.21.15.72:0/1067801262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f581c101cb0 0x7f581c109a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:36:40.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.950+0000 7f581affd640 1 --2- 172.21.15.72:0/1067801262 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f581c100ac0 0x7f581c10b440 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f580c00e9c0 tx=0x7f580c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:40.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.950+0000 7f5803fff640 1 -- 172.21.15.72:0/1067801262 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f580c00cd80 con 0x7f581c100ac0 2024-03-22T01:36:40.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.950+0000 7f58216d4640 1 -- 172.21.15.72:0/1067801262 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f581c1a1fa0 con 0x7f581c100ac0 2024-03-22T01:36:40.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.950+0000 7f5803fff640 1 -- 172.21.15.72:0/1067801262 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f580c0035f0 con 0x7f581c100ac0 2024-03-22T01:36:40.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.950+0000 7f5803fff640 1 -- 172.21.15.72:0/1067801262 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f580c010690 con 0x7f581c100ac0 2024-03-22T01:36:40.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.950+0000 7f58216d4640 1 -- 172.21.15.72:0/1067801262 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f581c1a2350 con 0x7f581c100ac0 2024-03-22T01:36:40.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.951+0000 7f58216d4640 1 -- 172.21.15.72:0/1067801262 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f581c107d80 con 0x7f581c100ac0 2024-03-22T01:36:40.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.953+0000 7f5803fff640 1 -- 172.21.15.72:0/1067801262 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 21) ==== 137666+0+0 (secure 0 0 0) 0x7f580c0037a0 con 0x7f581c100ac0 2024-03-22T01:36:40.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.954+0000 7f5803fff640 1 --2- 172.21.15.72:0/1067801262 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f58040a2d20 0x7f58040a51e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:40.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.954+0000 7f5803fff640 1 -- 172.21.15.72:0/1067801262 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f580c0cf890 con 0x7f581c100ac0 2024-03-22T01:36:40.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.954+0000 7f581a7fc640 1 --2- 172.21.15.72:0/1067801262 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f58040a2d20 0x7f58040a51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:40.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.954+0000 7f581a7fc640 1 --2- 172.21.15.72:0/1067801262 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f58040a2d20 0x7f58040a51e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f581002c7b0 tx=0x7f5810005fd0 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:40.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:40.957+0000 7f5803fff640 1 -- 172.21.15.72:0/1067801262 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f580c09c4e0 con 0x7f581c100ac0 2024-03-22T01:36:41.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:40 smithi089 ceph-mon[35770]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:36:41.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:40 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/2728439601' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:36:41.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:40 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:36:41.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:40 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:36:41.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:40 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:36:41.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:40 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:36:41.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:40 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:36:41.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:40 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:36:41.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:40 smithi072 ceph-mon[32414]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:36:41.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:40 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/2728439601' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:36:41.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:40 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:36:41.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:40 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:36:41.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:40 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:36:41.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:40 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:36:41.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:40 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:36:41.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:40 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:36:41.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.139+0000 7f58216d4640 1 -- 172.21.15.72:0/1067801262 --> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f581c061fd0 con 0x7f58040a2d20 2024-03-22T01:36:41.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.149+0000 7f5803fff640 1 -- 172.21.15.72:0/1067801262 <== mgr.15381 v2:172.21.15.72:6800/1379139045 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339302 (secure 0 0 0) 0x7f581c061fd0 con 0x7f58040a2d20 2024-03-22T01:36:41.152 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:36:41.154 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-03-22T01:36:41.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.153+0000 7f58216d4640 1 -- 172.21.15.72:0/1067801262 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f58040a2d20 msgr2=0x7f58040a51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:41.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.153+0000 7f58216d4640 1 --2- 172.21.15.72:0/1067801262 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f58040a2d20 0x7f58040a51e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f581002c7b0 tx=0x7f5810005fd0 comp rx=0 tx=0).stop 2024-03-22T01:36:41.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.153+0000 7f58216d4640 1 -- 172.21.15.72:0/1067801262 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f581c100ac0 msgr2=0x7f581c10b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:41.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.153+0000 7f58216d4640 1 --2- 172.21.15.72:0/1067801262 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f581c100ac0 0x7f581c10b440 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f580c00e9c0 tx=0x7f580c00ee80 comp rx=0 tx=0).stop 2024-03-22T01:36:41.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.153+0000 7f58216d4640 1 -- 172.21.15.72:0/1067801262 shutdown_connections 2024-03-22T01:36:41.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.153+0000 7f58216d4640 1 --2- 172.21.15.72:0/1067801262 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f58040a2d20 0x7f58040a51e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:41.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.153+0000 7f58216d4640 1 --2- 172.21.15.72:0/1067801262 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f581c102670 0x7f581c109fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:41.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.153+0000 7f58216d4640 1 --2- 172.21.15.72:0/1067801262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f581c101cb0 0x7f581c109a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:41.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.153+0000 7f58216d4640 1 --2- 172.21.15.72:0/1067801262 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f581c100ac0 0x7f581c10b440 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:41.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.153+0000 7f58216d4640 1 -- 172.21.15.72:0/1067801262 >> 172.21.15.72:0/1067801262 conn(0x7f581c0fc030 msgr2=0x7f581c104f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:41.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.153+0000 7f58216d4640 1 -- 172.21.15.72:0/1067801262 shutdown_connections 2024-03-22T01:36:41.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.153+0000 7f58216d4640 1 -- 172.21.15.72:0/1067801262 wait complete. 2024-03-22T01:36:41.163 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":4,"stamp":"2024-03-22T01:36:40.202937+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1186284,"num_objects":28,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":484,"num_read_kb":398,"num_write":383,"num_write_kb":4350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":789,"ondisk_log_size":789,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":373868,"kb_used_data":16176,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124567956,"statfs":{"total":1151940427776,"available":1151557586944,"internally_reserved":0,"allocated":16564224,"data_stored":9860107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":365933882},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.990801"},"pg_stats":[{"pgid":"3.3f","version":"93'8","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500839+0000","last_change":"2024-03-22T01:34:15.487496+0000","last_active":"2024-03-22T01:36:38.500839+0000","last_peered":"2024-03-22T01:36:38.500839+0000","last_clean":"2024-03-22T01:36:38.500839+0000","last_became_active":"2024-03-22T01:34:15.486700+0000","last_became_peered":"2024-03-22T01:34:15.486700+0000","last_unstale":"2024-03-22T01:36:38.500839+0000","last_undegraded":"2024-03-22T01:36:38.500839+0000","last_fullsized":"2024-03-22T01:36:38.500839+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:03:19.146811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"93'4","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.501995+0000","last_change":"2024-03-22T01:34:15.489791+0000","last_active":"2024-03-22T01:36:38.501995+0000","last_peered":"2024-03-22T01:36:38.501995+0000","last_clean":"2024-03-22T01:36:38.501995+0000","last_became_active":"2024-03-22T01:34:15.489493+0000","last_became_peered":"2024-03-22T01:34:15.489493+0000","last_unstale":"2024-03-22T01:36:38.501995+0000","last_undegraded":"2024-03-22T01:36:38.501995+0000","last_fullsized":"2024-03-22T01:36:38.501995+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:45:05.060454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"93'7","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.501785+0000","last_change":"2024-03-22T01:34:15.491567+0000","last_active":"2024-03-22T01:36:38.501785+0000","last_peered":"2024-03-22T01:36:38.501785+0000","last_clean":"2024-03-22T01:36:38.501785+0000","last_became_active":"2024-03-22T01:34:15.491275+0000","last_became_peered":"2024-03-22T01:34:15.491275+0000","last_unstale":"2024-03-22T01:36:38.501785+0000","last_undegraded":"2024-03-22T01:36:38.501785+0000","last_fullsized":"2024-03-22T01:36:38.501785+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:26:36.630811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"93'8","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500405+0000","last_change":"2024-03-22T01:34:15.488579+0000","last_active":"2024-03-22T01:36:38.500405+0000","last_peered":"2024-03-22T01:36:38.500405+0000","last_clean":"2024-03-22T01:36:38.500405+0000","last_became_active":"2024-03-22T01:34:15.488262+0000","last_became_peered":"2024-03-22T01:34:15.488262+0000","last_unstale":"2024-03-22T01:36:38.500405+0000","last_undegraded":"2024-03-22T01:36:38.500405+0000","last_fullsized":"2024-03-22T01:36:38.500405+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:58:21.788420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"93'4","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.307700+0000","last_change":"2024-03-22T01:34:15.485744+0000","last_active":"2024-03-22T01:36:38.307700+0000","last_peered":"2024-03-22T01:36:38.307700+0000","last_clean":"2024-03-22T01:36:38.307700+0000","last_became_active":"2024-03-22T01:34:15.485378+0000","last_became_peered":"2024-03-22T01:34:15.485378+0000","last_unstale":"2024-03-22T01:36:38.307700+0000","last_undegraded":"2024-03-22T01:36:38.307700+0000","last_fullsized":"2024-03-22T01:36:38.307700+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:06:40.979475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"93'6","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.192414+0000","last_change":"2024-03-22T01:34:15.495051+0000","last_active":"2024-03-22T01:36:38.192414+0000","last_peered":"2024-03-22T01:36:38.192414+0000","last_clean":"2024-03-22T01:36:38.192414+0000","last_became_active":"2024-03-22T01:34:15.494691+0000","last_became_peered":"2024-03-22T01:34:15.494691+0000","last_unstale":"2024-03-22T01:36:38.192414+0000","last_undegraded":"2024-03-22T01:36:38.192414+0000","last_fullsized":"2024-03-22T01:36:38.192414+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T01:44:28.336545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"93'5","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.300153+0000","last_change":"2024-03-22T01:34:15.493911+0000","last_active":"2024-03-22T01:36:38.300153+0000","last_peered":"2024-03-22T01:36:38.300153+0000","last_clean":"2024-03-22T01:36:38.300153+0000","last_became_active":"2024-03-22T01:34:15.493586+0000","last_became_peered":"2024-03-22T01:34:15.493586+0000","last_unstale":"2024-03-22T01:36:38.300153+0000","last_undegraded":"2024-03-22T01:36:38.300153+0000","last_fullsized":"2024-03-22T01:36:38.300153+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:22:41.391855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"93'9","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.307138+0000","last_change":"2024-03-22T01:34:15.481410+0000","last_active":"2024-03-22T01:36:38.307138+0000","last_peered":"2024-03-22T01:36:38.307138+0000","last_clean":"2024-03-22T01:36:38.307138+0000","last_became_active":"2024-03-22T01:34:15.481109+0000","last_became_peered":"2024-03-22T01:34:15.481109+0000","last_unstale":"2024-03-22T01:36:38.307138+0000","last_undegraded":"2024-03-22T01:36:38.307138+0000","last_fullsized":"2024-03-22T01:36:38.307138+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:27:10.004584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"93'11","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.460250+0000","last_change":"2024-03-22T01:34:15.485173+0000","last_active":"2024-03-22T01:36:38.460250+0000","last_peered":"2024-03-22T01:36:38.460250+0000","last_clean":"2024-03-22T01:36:38.460250+0000","last_became_active":"2024-03-22T01:34:15.484812+0000","last_became_peered":"2024-03-22T01:34:15.484812+0000","last_unstale":"2024-03-22T01:36:38.460250+0000","last_undegraded":"2024-03-22T01:36:38.460250+0000","last_fullsized":"2024-03-22T01:36:38.460250+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:44:06.383568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"93'16","reported_seq":47,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.192571+0000","last_change":"2024-03-22T01:34:15.497964+0000","last_active":"2024-03-22T01:36:38.192571+0000","last_peered":"2024-03-22T01:36:38.192571+0000","last_clean":"2024-03-22T01:36:38.192571+0000","last_became_active":"2024-03-22T01:34:15.496887+0000","last_became_peered":"2024-03-22T01:34:15.496887+0000","last_unstale":"2024-03-22T01:36:38.192571+0000","last_undegraded":"2024-03-22T01:36:38.192571+0000","last_fullsized":"2024-03-22T01:36:38.192571+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:13:40.420667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"93'6","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.300582+0000","last_change":"2024-03-22T01:34:15.496310+0000","last_active":"2024-03-22T01:36:38.300582+0000","last_peered":"2024-03-22T01:36:38.300582+0000","last_clean":"2024-03-22T01:36:38.300582+0000","last_became_active":"2024-03-22T01:34:15.494196+0000","last_became_peered":"2024-03-22T01:34:15.494196+0000","last_unstale":"2024-03-22T01:36:38.300582+0000","last_undegraded":"2024-03-22T01:36:38.300582+0000","last_fullsized":"2024-03-22T01:36:38.300582+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:33:02.798310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"93'7","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500458+0000","last_change":"2024-03-22T01:34:15.495380+0000","last_active":"2024-03-22T01:36:38.500458+0000","last_peered":"2024-03-22T01:36:38.500458+0000","last_clean":"2024-03-22T01:36:38.500458+0000","last_became_active":"2024-03-22T01:34:15.495063+0000","last_became_peered":"2024-03-22T01:34:15.495063+0000","last_unstale":"2024-03-22T01:36:38.500458+0000","last_undegraded":"2024-03-22T01:36:38.500458+0000","last_fullsized":"2024-03-22T01:36:38.500458+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:46:29.497329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"93'10","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.189104+0000","last_change":"2024-03-22T01:34:15.497533+0000","last_active":"2024-03-22T01:36:38.189104+0000","last_peered":"2024-03-22T01:36:38.189104+0000","last_clean":"2024-03-22T01:36:38.189104+0000","last_became_active":"2024-03-22T01:34:15.497223+0000","last_became_peered":"2024-03-22T01:34:15.497223+0000","last_unstale":"2024-03-22T01:36:38.189104+0000","last_undegraded":"2024-03-22T01:36:38.189104+0000","last_fullsized":"2024-03-22T01:36:38.189104+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:17:49.202842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"93'7","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.501331+0000","last_change":"2024-03-22T01:34:15.491317+0000","last_active":"2024-03-22T01:36:38.501331+0000","last_peered":"2024-03-22T01:36:38.501331+0000","last_clean":"2024-03-22T01:36:38.501331+0000","last_became_active":"2024-03-22T01:34:15.490982+0000","last_became_peered":"2024-03-22T01:34:15.490982+0000","last_unstale":"2024-03-22T01:36:38.501331+0000","last_undegraded":"2024-03-22T01:36:38.501331+0000","last_fullsized":"2024-03-22T01:36:38.501331+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:05:39.889964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"93'7","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.501112+0000","last_change":"2024-03-22T01:34:15.488502+0000","last_active":"2024-03-22T01:36:38.501112+0000","last_peered":"2024-03-22T01:36:38.501112+0000","last_clean":"2024-03-22T01:36:38.501112+0000","last_became_active":"2024-03-22T01:34:15.488200+0000","last_became_peered":"2024-03-22T01:34:15.488200+0000","last_unstale":"2024-03-22T01:36:38.501112+0000","last_undegraded":"2024-03-22T01:36:38.501112+0000","last_fullsized":"2024-03-22T01:36:38.501112+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:52:59.991591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"93'9","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.459533+0000","last_change":"2024-03-22T01:34:15.489991+0000","last_active":"2024-03-22T01:36:38.459533+0000","last_peered":"2024-03-22T01:36:38.459533+0000","last_clean":"2024-03-22T01:36:38.459533+0000","last_became_active":"2024-03-22T01:34:15.489696+0000","last_became_peered":"2024-03-22T01:34:15.489696+0000","last_unstale":"2024-03-22T01:36:38.459533+0000","last_undegraded":"2024-03-22T01:36:38.459533+0000","last_fullsized":"2024-03-22T01:36:38.459533+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:10:45.316422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"93'8","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500862+0000","last_change":"2024-03-22T01:34:15.488011+0000","last_active":"2024-03-22T01:36:38.500862+0000","last_peered":"2024-03-22T01:36:38.500862+0000","last_clean":"2024-03-22T01:36:38.500862+0000","last_became_active":"2024-03-22T01:34:15.487282+0000","last_became_peered":"2024-03-22T01:34:15.487282+0000","last_unstale":"2024-03-22T01:36:38.500862+0000","last_undegraded":"2024-03-22T01:36:38.500862+0000","last_fullsized":"2024-03-22T01:36:38.500862+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:16:21.396273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"93'10","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.305057+0000","last_change":"2024-03-22T01:34:15.488046+0000","last_active":"2024-03-22T01:36:38.305057+0000","last_peered":"2024-03-22T01:36:38.305057+0000","last_clean":"2024-03-22T01:36:38.305057+0000","last_became_active":"2024-03-22T01:34:15.487731+0000","last_became_peered":"2024-03-22T01:34:15.487731+0000","last_unstale":"2024-03-22T01:36:38.305057+0000","last_undegraded":"2024-03-22T01:36:38.305057+0000","last_fullsized":"2024-03-22T01:36:38.305057+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:44:16.931544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"93'7","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.191341+0000","last_change":"2024-03-22T01:34:15.503737+0000","last_active":"2024-03-22T01:36:38.191341+0000","last_peered":"2024-03-22T01:36:38.191341+0000","last_clean":"2024-03-22T01:36:38.191341+0000","last_became_active":"2024-03-22T01:34:15.502169+0000","last_became_peered":"2024-03-22T01:34:15.502169+0000","last_unstale":"2024-03-22T01:36:38.191341+0000","last_undegraded":"2024-03-22T01:36:38.191341+0000","last_fullsized":"2024-03-22T01:36:38.191341+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:05:53.027318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":24,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.191251+0000","last_change":"2024-03-22T01:34:19.527964+0000","last_active":"2024-03-22T01:36:38.191251+0000","last_peered":"2024-03-22T01:36:38.191251+0000","last_clean":"2024-03-22T01:36:38.191251+0000","last_became_active":"2024-03-22T01:34:19.527526+0000","last_became_peered":"2024-03-22T01:34:19.527526+0000","last_unstale":"2024-03-22T01:36:38.191251+0000","last_undegraded":"2024-03-22T01:36:38.191251+0000","last_fullsized":"2024-03-22T01:36:38.191251+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:35:39.404439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'4","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.305744+0000","last_change":"2024-03-22T01:34:14.473065+0000","last_active":"2024-03-22T01:36:38.305744+0000","last_peered":"2024-03-22T01:36:38.305744+0000","last_clean":"2024-03-22T01:36:38.305744+0000","last_became_active":"2024-03-22T01:34:14.472653+0000","last_became_peered":"2024-03-22T01:34:14.472653+0000","last_unstale":"2024-03-22T01:36:38.305744+0000","last_undegraded":"2024-03-22T01:36:38.305744+0000","last_fullsized":"2024-03-22T01:36:38.305744+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:11:09.525438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"93'5","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.308183+0000","last_change":"2024-03-22T01:34:15.497818+0000","last_active":"2024-03-22T01:36:38.308183+0000","last_peered":"2024-03-22T01:36:38.308183+0000","last_clean":"2024-03-22T01:36:38.308183+0000","last_became_active":"2024-03-22T01:34:15.497488+0000","last_became_peered":"2024-03-22T01:34:15.497488+0000","last_unstale":"2024-03-22T01:36:38.308183+0000","last_undegraded":"2024-03-22T01:36:38.308183+0000","last_fullsized":"2024-03-22T01:36:38.308183+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:21:09.340557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":24,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500006+0000","last_change":"2024-03-22T01:34:19.528435+0000","last_active":"2024-03-22T01:36:38.500006+0000","last_peered":"2024-03-22T01:36:38.500006+0000","last_clean":"2024-03-22T01:36:38.500006+0000","last_became_active":"2024-03-22T01:34:19.528044+0000","last_became_peered":"2024-03-22T01:34:19.528044+0000","last_unstale":"2024-03-22T01:36:38.500006+0000","last_undegraded":"2024-03-22T01:36:38.500006+0000","last_fullsized":"2024-03-22T01:36:38.500006+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:52:05.682173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"86'1","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.308372+0000","last_change":"2024-03-22T01:34:14.465817+0000","last_active":"2024-03-22T01:36:38.308372+0000","last_peered":"2024-03-22T01:36:38.308372+0000","last_clean":"2024-03-22T01:36:38.308372+0000","last_became_active":"2024-03-22T01:34:14.464338+0000","last_became_peered":"2024-03-22T01:34:14.464338+0000","last_unstale":"2024-03-22T01:36:38.308372+0000","last_undegraded":"2024-03-22T01:36:38.308372+0000","last_fullsized":"2024-03-22T01:36:38.308372+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:04:47.457092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"93'6","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.502083+0000","last_change":"2024-03-22T01:34:15.481988+0000","last_active":"2024-03-22T01:36:38.502083+0000","last_peered":"2024-03-22T01:36:38.502083+0000","last_clean":"2024-03-22T01:36:38.502083+0000","last_became_active":"2024-03-22T01:34:15.481676+0000","last_became_peered":"2024-03-22T01:34:15.481676+0000","last_unstale":"2024-03-22T01:36:38.502083+0000","last_undegraded":"2024-03-22T01:36:38.502083+0000","last_fullsized":"2024-03-22T01:36:38.502083+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:37:54.609731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":24,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500197+0000","last_change":"2024-03-22T01:34:19.528349+0000","last_active":"2024-03-22T01:36:38.500197+0000","last_peered":"2024-03-22T01:36:38.500197+0000","last_clean":"2024-03-22T01:36:38.500197+0000","last_became_active":"2024-03-22T01:34:19.527937+0000","last_became_peered":"2024-03-22T01:34:19.527937+0000","last_unstale":"2024-03-22T01:36:38.500197+0000","last_undegraded":"2024-03-22T01:36:38.500197+0000","last_fullsized":"2024-03-22T01:36:38.500197+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:48:34.242104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"86'2","reported_seq":44,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.308006+0000","last_change":"2024-03-22T01:35:16.756010+0000","last_active":"2024-03-22T01:36:38.308006+0000","last_peered":"2024-03-22T01:36:38.308006+0000","last_clean":"2024-03-22T01:36:38.308006+0000","last_became_active":"2024-03-22T01:35:16.751845+0000","last_became_peered":"2024-03-22T01:35:16.751845+0000","last_unstale":"2024-03-22T01:36:38.308006+0000","last_undegraded":"2024-03-22T01:36:38.308006+0000","last_fullsized":"2024-03-22T01:36:38.308006+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:23:18.305330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"93'9","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.459910+0000","last_change":"2024-03-22T01:34:15.491345+0000","last_active":"2024-03-22T01:36:38.459910+0000","last_peered":"2024-03-22T01:36:38.459910+0000","last_clean":"2024-03-22T01:36:38.459910+0000","last_became_active":"2024-03-22T01:34:15.490833+0000","last_became_peered":"2024-03-22T01:34:15.490833+0000","last_unstale":"2024-03-22T01:36:38.459910+0000","last_undegraded":"2024-03-22T01:36:38.459910+0000","last_fullsized":"2024-03-22T01:36:38.459910+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:30:43.950679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":24,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500078+0000","last_change":"2024-03-22T01:34:19.528351+0000","last_active":"2024-03-22T01:36:38.500078+0000","last_peered":"2024-03-22T01:36:38.500078+0000","last_clean":"2024-03-22T01:36:38.500078+0000","last_became_active":"2024-03-22T01:34:19.527939+0000","last_became_peered":"2024-03-22T01:34:19.527939+0000","last_unstale":"2024-03-22T01:36:38.500078+0000","last_undegraded":"2024-03-22T01:36:38.500078+0000","last_fullsized":"2024-03-22T01:36:38.500078+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:30:21.055848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":21,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500428+0000","last_change":"2024-03-22T01:35:16.755726+0000","last_active":"2024-03-22T01:36:38.500428+0000","last_peered":"2024-03-22T01:36:38.500428+0000","last_clean":"2024-03-22T01:36:38.500428+0000","last_became_active":"2024-03-22T01:35:16.749107+0000","last_became_peered":"2024-03-22T01:35:16.749107+0000","last_unstale":"2024-03-22T01:36:38.500428+0000","last_undegraded":"2024-03-22T01:36:38.500428+0000","last_fullsized":"2024-03-22T01:36:38.500428+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:49:37.500003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"93'15","reported_seq":44,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.314906+0000","last_change":"2024-03-22T01:34:15.479530+0000","last_active":"2024-03-22T01:36:38.314906+0000","last_peered":"2024-03-22T01:36:38.314906+0000","last_clean":"2024-03-22T01:36:38.314906+0000","last_became_active":"2024-03-22T01:34:15.479198+0000","last_became_peered":"2024-03-22T01:34:15.479198+0000","last_unstale":"2024-03-22T01:36:38.314906+0000","last_undegraded":"2024-03-22T01:36:38.314906+0000","last_fullsized":"2024-03-22T01:36:38.314906+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:28:51.624369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":24,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.191181+0000","last_change":"2024-03-22T01:34:19.524486+0000","last_active":"2024-03-22T01:36:38.191181+0000","last_peered":"2024-03-22T01:36:38.191181+0000","last_clean":"2024-03-22T01:36:38.191181+0000","last_became_active":"2024-03-22T01:34:19.524030+0000","last_became_peered":"2024-03-22T01:34:19.524030+0000","last_unstale":"2024-03-22T01:36:38.191181+0000","last_undegraded":"2024-03-22T01:36:38.191181+0000","last_fullsized":"2024-03-22T01:36:38.191181+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:31:10.157431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"86'1","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.314671+0000","last_change":"2024-03-22T01:34:14.469191+0000","last_active":"2024-03-22T01:36:38.314671+0000","last_peered":"2024-03-22T01:36:38.314671+0000","last_clean":"2024-03-22T01:36:38.314671+0000","last_became_active":"2024-03-22T01:34:14.468842+0000","last_became_peered":"2024-03-22T01:34:14.468842+0000","last_unstale":"2024-03-22T01:36:38.314671+0000","last_undegraded":"2024-03-22T01:36:38.314671+0000","last_fullsized":"2024-03-22T01:36:38.314671+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:29:56.566256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"93'7","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.307636+0000","last_change":"2024-03-22T01:34:15.483736+0000","last_active":"2024-03-22T01:36:38.307636+0000","last_peered":"2024-03-22T01:36:38.307636+0000","last_clean":"2024-03-22T01:36:38.307636+0000","last_became_active":"2024-03-22T01:34:15.483401+0000","last_became_peered":"2024-03-22T01:34:15.483401+0000","last_unstale":"2024-03-22T01:36:38.307636+0000","last_undegraded":"2024-03-22T01:36:38.307636+0000","last_fullsized":"2024-03-22T01:36:38.307636+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:26:51.974932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":24,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.191134+0000","last_change":"2024-03-22T01:34:19.534542+0000","last_active":"2024-03-22T01:36:38.191134+0000","last_peered":"2024-03-22T01:36:38.191134+0000","last_clean":"2024-03-22T01:36:38.191134+0000","last_became_active":"2024-03-22T01:34:19.534039+0000","last_became_peered":"2024-03-22T01:34:19.534039+0000","last_unstale":"2024-03-22T01:36:38.191134+0000","last_undegraded":"2024-03-22T01:36:38.191134+0000","last_fullsized":"2024-03-22T01:36:38.191134+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:59:37.363972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"86'1","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.192279+0000","last_change":"2024-03-22T01:34:14.468382+0000","last_active":"2024-03-22T01:36:38.192279+0000","last_peered":"2024-03-22T01:36:38.192279+0000","last_clean":"2024-03-22T01:36:38.192279+0000","last_became_active":"2024-03-22T01:34:14.467939+0000","last_became_peered":"2024-03-22T01:34:14.467939+0000","last_unstale":"2024-03-22T01:36:38.192279+0000","last_undegraded":"2024-03-22T01:36:38.192279+0000","last_fullsized":"2024-03-22T01:36:38.192279+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:57:17.031893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"93'8","reported_seq":43,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.314801+0000","last_change":"2024-03-22T01:35:17.206178+0000","last_active":"2024-03-22T01:36:38.314801+0000","last_peered":"2024-03-22T01:36:38.314801+0000","last_clean":"2024-03-22T01:36:38.314801+0000","last_became_active":"2024-03-22T01:35:17.205717+0000","last_became_peered":"2024-03-22T01:35:17.205717+0000","last_unstale":"2024-03-22T01:36:38.314801+0000","last_undegraded":"2024-03-22T01:36:38.314801+0000","last_fullsized":"2024-03-22T01:36:38.314801+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:14:15.920771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500689+0000","last_change":"2024-03-22T01:36:17.659141+0000","last_active":"2024-03-22T01:36:38.500689+0000","last_peered":"2024-03-22T01:36:38.500689+0000","last_clean":"2024-03-22T01:36:38.500689+0000","last_became_active":"2024-03-22T01:36:17.658611+0000","last_became_peered":"2024-03-22T01:36:17.658611+0000","last_unstale":"2024-03-22T01:36:38.500689+0000","last_undegraded":"2024-03-22T01:36:38.500689+0000","last_fullsized":"2024-03-22T01:36:38.500689+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:45:28.895847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,10,0],"acting":[11,4,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.189382+0000","last_change":"2024-03-22T01:34:14.464485+0000","last_active":"2024-03-22T01:36:38.189382+0000","last_peered":"2024-03-22T01:36:38.189382+0000","last_clean":"2024-03-22T01:36:38.189382+0000","last_became_active":"2024-03-22T01:34:14.463973+0000","last_became_peered":"2024-03-22T01:34:14.463973+0000","last_unstale":"2024-03-22T01:36:38.189382+0000","last_undegraded":"2024-03-22T01:36:38.189382+0000","last_fullsized":"2024-03-22T01:36:38.189382+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:11:13.265346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"93'8","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.307338+0000","last_change":"2024-03-22T01:34:15.483893+0000","last_active":"2024-03-22T01:36:38.307338+0000","last_peered":"2024-03-22T01:36:38.307338+0000","last_clean":"2024-03-22T01:36:38.307338+0000","last_became_active":"2024-03-22T01:34:15.482295+0000","last_became_peered":"2024-03-22T01:34:15.482295+0000","last_unstale":"2024-03-22T01:36:38.307338+0000","last_undegraded":"2024-03-22T01:36:38.307338+0000","last_fullsized":"2024-03-22T01:36:38.307338+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:17:33.274900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":25,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.501992+0000","last_change":"2024-03-22T01:34:19.522567+0000","last_active":"2024-03-22T01:36:38.501992+0000","last_peered":"2024-03-22T01:36:38.501992+0000","last_clean":"2024-03-22T01:36:38.501992+0000","last_became_active":"2024-03-22T01:34:19.522165+0000","last_became_peered":"2024-03-22T01:34:19.522165+0000","last_unstale":"2024-03-22T01:36:38.501992+0000","last_undegraded":"2024-03-22T01:36:38.501992+0000","last_fullsized":"2024-03-22T01:36:38.501992+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:41:37.829622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":31,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.314474+0000","last_change":"2024-03-22T01:34:14.463824+0000","last_active":"2024-03-22T01:36:38.314474+0000","last_peered":"2024-03-22T01:36:38.314474+0000","last_clean":"2024-03-22T01:36:38.314474+0000","last_became_active":"2024-03-22T01:34:14.463484+0000","last_became_peered":"2024-03-22T01:34:14.463484+0000","last_unstale":"2024-03-22T01:36:38.314474+0000","last_undegraded":"2024-03-22T01:36:38.314474+0000","last_fullsized":"2024-03-22T01:36:38.314474+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:19:31.568853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"93'11","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.304658+0000","last_change":"2024-03-22T01:34:15.488883+0000","last_active":"2024-03-22T01:36:38.304658+0000","last_peered":"2024-03-22T01:36:38.304658+0000","last_clean":"2024-03-22T01:36:38.304658+0000","last_became_active":"2024-03-22T01:34:15.488573+0000","last_became_peered":"2024-03-22T01:34:15.488573+0000","last_unstale":"2024-03-22T01:36:38.304658+0000","last_undegraded":"2024-03-22T01:36:38.304658+0000","last_fullsized":"2024-03-22T01:36:38.304658+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:28:46.388930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.307244+0000","last_change":"2024-03-22T01:34:19.527321+0000","last_active":"2024-03-22T01:36:38.307244+0000","last_peered":"2024-03-22T01:36:38.307244+0000","last_clean":"2024-03-22T01:36:38.307244+0000","last_became_active":"2024-03-22T01:34:19.526905+0000","last_became_peered":"2024-03-22T01:34:19.526905+0000","last_unstale":"2024-03-22T01:36:38.307244+0000","last_undegraded":"2024-03-22T01:36:38.307244+0000","last_fullsized":"2024-03-22T01:36:38.307244+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:49:31.023938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.192604+0000","last_change":"2024-03-22T01:34:14.471415+0000","last_active":"2024-03-22T01:36:38.192604+0000","last_peered":"2024-03-22T01:36:38.192604+0000","last_clean":"2024-03-22T01:36:38.192604+0000","last_became_active":"2024-03-22T01:34:14.470928+0000","last_became_peered":"2024-03-22T01:34:14.470928+0000","last_unstale":"2024-03-22T01:36:38.192604+0000","last_undegraded":"2024-03-22T01:36:38.192604+0000","last_fullsized":"2024-03-22T01:36:38.192604+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:09:10.286549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"93'10","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.460165+0000","last_change":"2024-03-22T01:34:15.489656+0000","last_active":"2024-03-22T01:36:38.460165+0000","last_peered":"2024-03-22T01:36:38.460165+0000","last_clean":"2024-03-22T01:36:38.460165+0000","last_became_active":"2024-03-22T01:34:15.488933+0000","last_became_peered":"2024-03-22T01:34:15.488933+0000","last_unstale":"2024-03-22T01:36:38.460165+0000","last_undegraded":"2024-03-22T01:36:38.460165+0000","last_fullsized":"2024-03-22T01:36:38.460165+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:17:38.789414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.304368+0000","last_change":"2024-03-22T01:34:19.534174+0000","last_active":"2024-03-22T01:36:38.304368+0000","last_peered":"2024-03-22T01:36:38.304368+0000","last_clean":"2024-03-22T01:36:38.304368+0000","last_became_active":"2024-03-22T01:34:19.533759+0000","last_became_peered":"2024-03-22T01:34:19.533759+0000","last_unstale":"2024-03-22T01:36:38.304368+0000","last_undegraded":"2024-03-22T01:36:38.304368+0000","last_fullsized":"2024-03-22T01:36:38.304368+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:24:50.363984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.300490+0000","last_change":"2024-03-22T01:34:14.465755+0000","last_active":"2024-03-22T01:36:38.300490+0000","last_peered":"2024-03-22T01:36:38.300490+0000","last_clean":"2024-03-22T01:36:38.300490+0000","last_became_active":"2024-03-22T01:34:14.465230+0000","last_became_peered":"2024-03-22T01:34:14.465230+0000","last_unstale":"2024-03-22T01:36:38.300490+0000","last_undegraded":"2024-03-22T01:36:38.300490+0000","last_fullsized":"2024-03-22T01:36:38.300490+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:56:29.286551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"93'9","reported_seq":45,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.502022+0000","last_change":"2024-03-22T01:35:16.748981+0000","last_active":"2024-03-22T01:36:38.502022+0000","last_peered":"2024-03-22T01:36:38.502022+0000","last_clean":"2024-03-22T01:36:38.502022+0000","last_became_active":"2024-03-22T01:35:16.748524+0000","last_became_peered":"2024-03-22T01:35:16.748524+0000","last_unstale":"2024-03-22T01:36:38.502022+0000","last_undegraded":"2024-03-22T01:36:38.502022+0000","last_fullsized":"2024-03-22T01:36:38.502022+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:23:37.763347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":31,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500236+0000","last_change":"2024-03-22T01:36:17.789745+0000","last_active":"2024-03-22T01:36:38.500236+0000","last_peered":"2024-03-22T01:36:38.500236+0000","last_clean":"2024-03-22T01:36:38.500236+0000","last_became_active":"2024-03-22T01:36:17.789277+0000","last_became_peered":"2024-03-22T01:36:17.789277+0000","last_unstale":"2024-03-22T01:36:38.500236+0000","last_undegraded":"2024-03-22T01:36:38.500236+0000","last_fullsized":"2024-03-22T01:36:38.500236+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:57:47.671252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"86'3","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.315060+0000","last_change":"2024-03-22T01:34:14.463365+0000","last_active":"2024-03-22T01:36:38.315060+0000","last_peered":"2024-03-22T01:36:38.315060+0000","last_clean":"2024-03-22T01:36:38.315060+0000","last_became_active":"2024-03-22T01:34:14.462984+0000","last_became_peered":"2024-03-22T01:34:14.462984+0000","last_unstale":"2024-03-22T01:36:38.315060+0000","last_undegraded":"2024-03-22T01:36:38.315060+0000","last_fullsized":"2024-03-22T01:36:38.315060+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:55:27.318854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"93'11","reported_seq":43,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.460074+0000","last_change":"2024-03-22T01:34:15.485309+0000","last_active":"2024-03-22T01:36:38.460074+0000","last_peered":"2024-03-22T01:36:38.460074+0000","last_clean":"2024-03-22T01:36:38.460074+0000","last_became_active":"2024-03-22T01:34:15.484964+0000","last_became_peered":"2024-03-22T01:34:15.484964+0000","last_unstale":"2024-03-22T01:36:38.460074+0000","last_undegraded":"2024-03-22T01:36:38.460074+0000","last_fullsized":"2024-03-22T01:36:38.460074+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:25:57.911623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":31,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.304733+0000","last_change":"2024-03-22T01:36:17.659080+0000","last_active":"2024-03-22T01:36:38.304733+0000","last_peered":"2024-03-22T01:36:38.304733+0000","last_clean":"2024-03-22T01:36:38.304733+0000","last_became_active":"2024-03-22T01:36:17.658738+0000","last_became_peered":"2024-03-22T01:36:17.658738+0000","last_unstale":"2024-03-22T01:36:38.304733+0000","last_undegraded":"2024-03-22T01:36:38.304733+0000","last_fullsized":"2024-03-22T01:36:38.304733+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:28:38.338235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,10],"acting":[9,8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"96'37","reported_seq":321,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.688979+0000","last_change":"2024-03-22T01:34:14.469378+0000","last_active":"2024-03-22T01:36:38.688979+0000","last_peered":"2024-03-22T01:36:38.688979+0000","last_clean":"2024-03-22T01:36:38.688979+0000","last_became_active":"2024-03-22T01:34:14.468956+0000","last_became_peered":"2024-03-22T01:34:14.468956+0000","last_unstale":"2024-03-22T01:36:38.688979+0000","last_undegraded":"2024-03-22T01:36:38.688979+0000","last_fullsized":"2024-03-22T01:36:38.688979+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:40:49.048793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70068,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":250,"num_read_kb":194,"num_write":37,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"93'6","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.307927+0000","last_change":"2024-03-22T01:34:15.488497+0000","last_active":"2024-03-22T01:36:38.307927+0000","last_peered":"2024-03-22T01:36:38.307927+0000","last_clean":"2024-03-22T01:36:38.307927+0000","last_became_active":"2024-03-22T01:34:15.488134+0000","last_became_peered":"2024-03-22T01:34:15.488134+0000","last_unstale":"2024-03-22T01:36:38.307927+0000","last_undegraded":"2024-03-22T01:36:38.307927+0000","last_fullsized":"2024-03-22T01:36:38.307927+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:50:41.206135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":24,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.315310+0000","last_change":"2024-03-22T01:34:19.528744+0000","last_active":"2024-03-22T01:36:38.315310+0000","last_peered":"2024-03-22T01:36:38.315310+0000","last_clean":"2024-03-22T01:36:38.315310+0000","last_became_active":"2024-03-22T01:34:19.528375+0000","last_became_peered":"2024-03-22T01:34:19.528375+0000","last_unstale":"2024-03-22T01:36:38.315310+0000","last_undegraded":"2024-03-22T01:36:38.315310+0000","last_fullsized":"2024-03-22T01:36:38.315310+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:47:53.753923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"86'2","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.191536+0000","last_change":"2024-03-22T01:34:14.463289+0000","last_active":"2024-03-22T01:36:38.191536+0000","last_peered":"2024-03-22T01:36:38.191536+0000","last_clean":"2024-03-22T01:36:38.191536+0000","last_became_active":"2024-03-22T01:34:14.462883+0000","last_became_peered":"2024-03-22T01:34:14.462883+0000","last_unstale":"2024-03-22T01:36:38.191536+0000","last_undegraded":"2024-03-22T01:36:38.191536+0000","last_fullsized":"2024-03-22T01:36:38.191536+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:06:14.452674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"93'9","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.188986+0000","last_change":"2024-03-22T01:34:15.484785+0000","last_active":"2024-03-22T01:36:38.188986+0000","last_peered":"2024-03-22T01:36:38.188986+0000","last_clean":"2024-03-22T01:36:38.188986+0000","last_became_active":"2024-03-22T01:34:15.484474+0000","last_became_peered":"2024-03-22T01:34:15.484474+0000","last_unstale":"2024-03-22T01:36:38.188986+0000","last_undegraded":"2024-03-22T01:36:38.188986+0000","last_fullsized":"2024-03-22T01:36:38.188986+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:11:13.338836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":24,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500472+0000","last_change":"2024-03-22T01:34:19.527379+0000","last_active":"2024-03-22T01:36:38.500472+0000","last_peered":"2024-03-22T01:36:38.500472+0000","last_clean":"2024-03-22T01:36:38.500472+0000","last_became_active":"2024-03-22T01:34:19.526957+0000","last_became_peered":"2024-03-22T01:34:19.526957+0000","last_unstale":"2024-03-22T01:36:38.500472+0000","last_undegraded":"2024-03-22T01:36:38.500472+0000","last_fullsized":"2024-03-22T01:36:38.500472+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:03:09.538334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.314794+0000","last_change":"2024-03-22T01:34:14.467372+0000","last_active":"2024-03-22T01:36:38.314794+0000","last_peered":"2024-03-22T01:36:38.314794+0000","last_clean":"2024-03-22T01:36:38.314794+0000","last_became_active":"2024-03-22T01:34:14.466124+0000","last_became_peered":"2024-03-22T01:34:14.466124+0000","last_unstale":"2024-03-22T01:36:38.314794+0000","last_undegraded":"2024-03-22T01:36:38.314794+0000","last_fullsized":"2024-03-22T01:36:38.314794+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:15:41.804735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"93'15","reported_seq":46,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.300521+0000","last_change":"2024-03-22T01:34:15.491533+0000","last_active":"2024-03-22T01:36:38.300521+0000","last_peered":"2024-03-22T01:36:38.300521+0000","last_clean":"2024-03-22T01:36:38.300521+0000","last_became_active":"2024-03-22T01:34:15.491208+0000","last_became_peered":"2024-03-22T01:34:15.491208+0000","last_unstale":"2024-03-22T01:36:38.300521+0000","last_undegraded":"2024-03-22T01:36:38.300521+0000","last_fullsized":"2024-03-22T01:36:38.300521+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:03:07.800607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":24,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.306930+0000","last_change":"2024-03-22T01:34:19.533419+0000","last_active":"2024-03-22T01:36:38.306930+0000","last_peered":"2024-03-22T01:36:38.306930+0000","last_clean":"2024-03-22T01:36:38.306930+0000","last_became_active":"2024-03-22T01:34:19.530042+0000","last_became_peered":"2024-03-22T01:34:19.530042+0000","last_unstale":"2024-03-22T01:36:38.306930+0000","last_undegraded":"2024-03-22T01:36:38.306930+0000","last_fullsized":"2024-03-22T01:36:38.306930+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:46:59.859742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.499888+0000","last_change":"2024-03-22T01:34:14.467033+0000","last_active":"2024-03-22T01:36:38.499888+0000","last_peered":"2024-03-22T01:36:38.499888+0000","last_clean":"2024-03-22T01:36:38.499888+0000","last_became_active":"2024-03-22T01:34:14.466649+0000","last_became_peered":"2024-03-22T01:34:14.466649+0000","last_unstale":"2024-03-22T01:36:38.499888+0000","last_undegraded":"2024-03-22T01:36:38.499888+0000","last_fullsized":"2024-03-22T01:36:38.499888+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:38:44.890899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"93'6","reported_seq":43,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.300199+0000","last_change":"2024-03-22T01:35:16.815302+0000","last_active":"2024-03-22T01:36:38.300199+0000","last_peered":"2024-03-22T01:36:38.300199+0000","last_clean":"2024-03-22T01:36:38.300199+0000","last_became_active":"2024-03-22T01:35:16.814988+0000","last_became_peered":"2024-03-22T01:35:16.814988+0000","last_unstale":"2024-03-22T01:36:38.300199+0000","last_undegraded":"2024-03-22T01:36:38.300199+0000","last_fullsized":"2024-03-22T01:36:38.300199+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:54:47.793437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":24,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.189238+0000","last_change":"2024-03-22T01:34:19.531417+0000","last_active":"2024-03-22T01:36:38.189238+0000","last_peered":"2024-03-22T01:36:38.189238+0000","last_clean":"2024-03-22T01:36:38.189238+0000","last_became_active":"2024-03-22T01:34:19.531003+0000","last_became_peered":"2024-03-22T01:34:19.531003+0000","last_unstale":"2024-03-22T01:36:38.189238+0000","last_undegraded":"2024-03-22T01:36:38.189238+0000","last_fullsized":"2024-03-22T01:36:38.189238+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:33:52.865110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"86'8","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.307580+0000","last_change":"2024-03-22T01:34:14.472904+0000","last_active":"2024-03-22T01:36:38.307580+0000","last_peered":"2024-03-22T01:36:38.307580+0000","last_clean":"2024-03-22T01:36:38.307580+0000","last_became_active":"2024-03-22T01:34:14.472442+0000","last_became_peered":"2024-03-22T01:34:14.472442+0000","last_unstale":"2024-03-22T01:36:38.307580+0000","last_undegraded":"2024-03-22T01:36:38.307580+0000","last_fullsized":"2024-03-22T01:36:38.307580+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:57:45.478316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"93'6","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.314972+0000","last_change":"2024-03-22T01:34:15.477411+0000","last_active":"2024-03-22T01:36:38.314972+0000","last_peered":"2024-03-22T01:36:38.314972+0000","last_clean":"2024-03-22T01:36:38.314972+0000","last_became_active":"2024-03-22T01:34:15.477035+0000","last_became_peered":"2024-03-22T01:34:15.477035+0000","last_unstale":"2024-03-22T01:36:38.314972+0000","last_undegraded":"2024-03-22T01:36:38.314972+0000","last_fullsized":"2024-03-22T01:36:38.314972+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:33:13.410872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.314593+0000","last_change":"2024-03-22T01:34:19.528258+0000","last_active":"2024-03-22T01:36:38.314593+0000","last_peered":"2024-03-22T01:36:38.314593+0000","last_clean":"2024-03-22T01:36:38.314593+0000","last_became_active":"2024-03-22T01:34:19.527668+0000","last_became_peered":"2024-03-22T01:34:19.527668+0000","last_unstale":"2024-03-22T01:36:38.314593+0000","last_undegraded":"2024-03-22T01:36:38.314593+0000","last_fullsized":"2024-03-22T01:36:38.314593+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:34:19.162609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'1","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.308046+0000","last_change":"2024-03-22T01:34:14.464735+0000","last_active":"2024-03-22T01:36:38.308046+0000","last_peered":"2024-03-22T01:36:38.308046+0000","last_clean":"2024-03-22T01:36:38.308046+0000","last_became_active":"2024-03-22T01:34:14.464359+0000","last_became_peered":"2024-03-22T01:34:14.464359+0000","last_unstale":"2024-03-22T01:36:38.308046+0000","last_undegraded":"2024-03-22T01:36:38.308046+0000","last_fullsized":"2024-03-22T01:36:38.308046+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:21:18.354922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"93'5","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500679+0000","last_change":"2024-03-22T01:35:16.752884+0000","last_active":"2024-03-22T01:36:38.500679+0000","last_peered":"2024-03-22T01:36:38.500679+0000","last_clean":"2024-03-22T01:36:38.500679+0000","last_became_active":"2024-03-22T01:35:16.751355+0000","last_became_peered":"2024-03-22T01:35:16.751355+0000","last_unstale":"2024-03-22T01:36:38.500679+0000","last_undegraded":"2024-03-22T01:36:38.500679+0000","last_fullsized":"2024-03-22T01:36:38.500679+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:49:33.261481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":25,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.307644+0000","last_change":"2024-03-22T01:34:19.668136+0000","last_active":"2024-03-22T01:36:38.307644+0000","last_peered":"2024-03-22T01:36:38.307644+0000","last_clean":"2024-03-22T01:36:38.307644+0000","last_became_active":"2024-03-22T01:34:19.667633+0000","last_became_peered":"2024-03-22T01:34:19.667633+0000","last_unstale":"2024-03-22T01:36:38.307644+0000","last_undegraded":"2024-03-22T01:36:38.307644+0000","last_fullsized":"2024-03-22T01:36:38.307644+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:07:26.455373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.305067+0000","last_change":"2024-03-22T01:34:14.471959+0000","last_active":"2024-03-22T01:36:38.305067+0000","last_peered":"2024-03-22T01:36:38.305067+0000","last_clean":"2024-03-22T01:36:38.305067+0000","last_became_active":"2024-03-22T01:34:14.471618+0000","last_became_peered":"2024-03-22T01:34:14.471618+0000","last_unstale":"2024-03-22T01:36:38.305067+0000","last_undegraded":"2024-03-22T01:36:38.305067+0000","last_fullsized":"2024-03-22T01:36:38.305067+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:45:03.537013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"93'4","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.192471+0000","last_change":"2024-03-22T01:34:15.495770+0000","last_active":"2024-03-22T01:36:38.192471+0000","last_peered":"2024-03-22T01:36:38.192471+0000","last_clean":"2024-03-22T01:36:38.192471+0000","last_became_active":"2024-03-22T01:34:15.495403+0000","last_became_peered":"2024-03-22T01:34:15.495403+0000","last_unstale":"2024-03-22T01:36:38.192471+0000","last_undegraded":"2024-03-22T01:36:38.192471+0000","last_fullsized":"2024-03-22T01:36:38.192471+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:48:05.184305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":25,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.189283+0000","last_change":"2024-03-22T01:34:19.520204+0000","last_active":"2024-03-22T01:36:38.189283+0000","last_peered":"2024-03-22T01:36:38.189283+0000","last_clean":"2024-03-22T01:36:38.189283+0000","last_became_active":"2024-03-22T01:34:19.519847+0000","last_became_peered":"2024-03-22T01:34:19.519847+0000","last_unstale":"2024-03-22T01:36:38.189283+0000","last_undegraded":"2024-03-22T01:36:38.189283+0000","last_fullsized":"2024-03-22T01:36:38.189283+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:06:21.389636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.300364+0000","last_change":"2024-03-22T01:34:14.464162+0000","last_active":"2024-03-22T01:36:38.300364+0000","last_peered":"2024-03-22T01:36:38.300364+0000","last_clean":"2024-03-22T01:36:38.300364+0000","last_became_active":"2024-03-22T01:34:14.462700+0000","last_became_peered":"2024-03-22T01:34:14.462700+0000","last_unstale":"2024-03-22T01:36:38.300364+0000","last_undegraded":"2024-03-22T01:36:38.300364+0000","last_fullsized":"2024-03-22T01:36:38.300364+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:45:32.262466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"93'7","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.192324+0000","last_change":"2024-03-22T01:34:15.488256+0000","last_active":"2024-03-22T01:36:38.192324+0000","last_peered":"2024-03-22T01:36:38.192324+0000","last_clean":"2024-03-22T01:36:38.192324+0000","last_became_active":"2024-03-22T01:34:15.487939+0000","last_became_peered":"2024-03-22T01:34:15.487939+0000","last_unstale":"2024-03-22T01:36:38.192324+0000","last_undegraded":"2024-03-22T01:36:38.192324+0000","last_fullsized":"2024-03-22T01:36:38.192324+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:30:49.323531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.189196+0000","last_change":"2024-03-22T01:34:19.667633+0000","last_active":"2024-03-22T01:36:38.189196+0000","last_peered":"2024-03-22T01:36:38.189196+0000","last_clean":"2024-03-22T01:36:38.189196+0000","last_became_active":"2024-03-22T01:34:19.667199+0000","last_became_peered":"2024-03-22T01:34:19.667199+0000","last_unstale":"2024-03-22T01:36:38.189196+0000","last_undegraded":"2024-03-22T01:36:38.189196+0000","last_fullsized":"2024-03-22T01:36:38.189196+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T01:58:13.459475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"93'8","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.314599+0000","last_change":"2024-03-22T01:34:15.489468+0000","last_active":"2024-03-22T01:36:38.314599+0000","last_peered":"2024-03-22T01:36:38.314599+0000","last_clean":"2024-03-22T01:36:38.314599+0000","last_became_active":"2024-03-22T01:34:15.489029+0000","last_became_peered":"2024-03-22T01:34:15.489029+0000","last_unstale":"2024-03-22T01:36:38.314599+0000","last_undegraded":"2024-03-22T01:36:38.314599+0000","last_fullsized":"2024-03-22T01:36:38.314599+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:00:01.370161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.188685+0000","last_change":"2024-03-22T01:34:14.464552+0000","last_active":"2024-03-22T01:36:38.188685+0000","last_peered":"2024-03-22T01:36:38.188685+0000","last_clean":"2024-03-22T01:36:38.188685+0000","last_became_active":"2024-03-22T01:34:14.463554+0000","last_became_peered":"2024-03-22T01:34:14.463554+0000","last_unstale":"2024-03-22T01:36:38.188685+0000","last_undegraded":"2024-03-22T01:36:38.188685+0000","last_fullsized":"2024-03-22T01:36:38.188685+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:51:16.709278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"91'1","reported_seq":26,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.300450+0000","last_change":"2024-03-22T01:34:19.526066+0000","last_active":"2024-03-22T01:36:38.300450+0000","last_peered":"2024-03-22T01:36:38.300450+0000","last_clean":"2024-03-22T01:36:38.300450+0000","last_became_active":"2024-03-22T01:34:19.525659+0000","last_became_peered":"2024-03-22T01:34:19.525659+0000","last_unstale":"2024-03-22T01:36:38.300450+0000","last_undegraded":"2024-03-22T01:36:38.300450+0000","last_fullsized":"2024-03-22T01:36:38.300450+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:06:26.287971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"93'8","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.307732+0000","last_change":"2024-03-22T01:34:15.477018+0000","last_active":"2024-03-22T01:36:38.307732+0000","last_peered":"2024-03-22T01:36:38.307732+0000","last_clean":"2024-03-22T01:36:38.307732+0000","last_became_active":"2024-03-22T01:34:15.476613+0000","last_became_peered":"2024-03-22T01:34:15.476613+0000","last_unstale":"2024-03-22T01:36:38.307732+0000","last_undegraded":"2024-03-22T01:36:38.307732+0000","last_fullsized":"2024-03-22T01:36:38.307732+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:10:07.980123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.499827+0000","last_change":"2024-03-22T01:34:14.470922+0000","last_active":"2024-03-22T01:36:38.499827+0000","last_peered":"2024-03-22T01:36:38.499827+0000","last_clean":"2024-03-22T01:36:38.499827+0000","last_became_active":"2024-03-22T01:34:14.470515+0000","last_became_peered":"2024-03-22T01:34:14.470515+0000","last_unstale":"2024-03-22T01:36:38.499827+0000","last_undegraded":"2024-03-22T01:36:38.499827+0000","last_fullsized":"2024-03-22T01:36:38.499827+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:52:50.203631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":365,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.300114+0000","last_change":"2024-03-22T01:33:03.615234+0000","last_active":"2024-03-22T01:36:38.300114+0000","last_peered":"2024-03-22T01:36:38.300114+0000","last_clean":"2024-03-22T01:36:38.300114+0000","last_became_active":"2024-03-22T01:33:03.614710+0000","last_became_peered":"2024-03-22T01:33:03.614710+0000","last_unstale":"2024-03-22T01:36:38.300114+0000","last_undegraded":"2024-03-22T01:36:38.300114+0000","last_fullsized":"2024-03-22T01:36:38.300114+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:29:38.312292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:29:38.312292+0000","last_clean_scrub_stamp":"2024-03-22T01:29:38.312292+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:56:42.348119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.314929+0000","last_change":"2024-03-22T01:34:19.668565+0000","last_active":"2024-03-22T01:36:38.314929+0000","last_peered":"2024-03-22T01:36:38.314929+0000","last_clean":"2024-03-22T01:36:38.314929+0000","last_became_active":"2024-03-22T01:34:19.667959+0000","last_became_peered":"2024-03-22T01:34:19.667959+0000","last_unstale":"2024-03-22T01:36:38.314929+0000","last_undegraded":"2024-03-22T01:36:38.314929+0000","last_fullsized":"2024-03-22T01:36:38.314929+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:13:04.447486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"93'5","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.188685+0000","last_change":"2024-03-22T01:34:15.489513+0000","last_active":"2024-03-22T01:36:38.188685+0000","last_peered":"2024-03-22T01:36:38.188685+0000","last_clean":"2024-03-22T01:36:38.188685+0000","last_became_active":"2024-03-22T01:34:15.489188+0000","last_became_peered":"2024-03-22T01:34:15.489188+0000","last_unstale":"2024-03-22T01:36:38.188685+0000","last_undegraded":"2024-03-22T01:36:38.188685+0000","last_fullsized":"2024-03-22T01:36:38.188685+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:08:24.859350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.300274+0000","last_change":"2024-03-22T01:34:14.466305+0000","last_active":"2024-03-22T01:36:38.300274+0000","last_peered":"2024-03-22T01:36:38.300274+0000","last_clean":"2024-03-22T01:36:38.300274+0000","last_became_active":"2024-03-22T01:34:14.465495+0000","last_became_peered":"2024-03-22T01:34:14.465495+0000","last_unstale":"2024-03-22T01:36:38.300274+0000","last_undegraded":"2024-03-22T01:36:38.300274+0000","last_fullsized":"2024-03-22T01:36:38.300274+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:45:13.481187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500369+0000","last_change":"2024-03-22T01:34:19.526970+0000","last_active":"2024-03-22T01:36:38.500369+0000","last_peered":"2024-03-22T01:36:38.500369+0000","last_clean":"2024-03-22T01:36:38.500369+0000","last_became_active":"2024-03-22T01:34:19.526462+0000","last_became_peered":"2024-03-22T01:34:19.526462+0000","last_unstale":"2024-03-22T01:36:38.500369+0000","last_undegraded":"2024-03-22T01:36:38.500369+0000","last_fullsized":"2024-03-22T01:36:38.500369+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:04:18.417358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500521+0000","last_change":"2024-03-22T01:35:16.750746+0000","last_active":"2024-03-22T01:36:38.500521+0000","last_peered":"2024-03-22T01:36:38.500521+0000","last_clean":"2024-03-22T01:36:38.500521+0000","last_became_active":"2024-03-22T01:35:16.750271+0000","last_became_peered":"2024-03-22T01:35:16.750271+0000","last_unstale":"2024-03-22T01:36:38.500521+0000","last_undegraded":"2024-03-22T01:36:38.500521+0000","last_fullsized":"2024-03-22T01:36:38.500521+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:18:10.839351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"93'9","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.189402+0000","last_change":"2024-03-22T01:34:15.490405+0000","last_active":"2024-03-22T01:36:38.189402+0000","last_peered":"2024-03-22T01:36:38.189402+0000","last_clean":"2024-03-22T01:36:38.189402+0000","last_became_active":"2024-03-22T01:34:15.489921+0000","last_became_peered":"2024-03-22T01:34:15.489921+0000","last_unstale":"2024-03-22T01:36:38.189402+0000","last_undegraded":"2024-03-22T01:36:38.189402+0000","last_fullsized":"2024-03-22T01:36:38.189402+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:27:28.804649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.307314+0000","last_change":"2024-03-22T01:34:19.667651+0000","last_active":"2024-03-22T01:36:38.307314+0000","last_peered":"2024-03-22T01:36:38.307314+0000","last_clean":"2024-03-22T01:36:38.307314+0000","last_became_active":"2024-03-22T01:34:19.667197+0000","last_became_peered":"2024-03-22T01:34:19.667197+0000","last_unstale":"2024-03-22T01:36:38.307314+0000","last_undegraded":"2024-03-22T01:36:38.307314+0000","last_fullsized":"2024-03-22T01:36:38.307314+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:32:55.874098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"96'43","reported_seq":100,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.424707+0000","last_change":"2024-03-22T01:34:14.474988+0000","last_active":"2024-03-22T01:36:38.424707+0000","last_peered":"2024-03-22T01:36:38.424707+0000","last_clean":"2024-03-22T01:36:38.424707+0000","last_became_active":"2024-03-22T01:34:14.474651+0000","last_became_peered":"2024-03-22T01:34:14.474651+0000","last_unstale":"2024-03-22T01:36:38.424707+0000","last_undegraded":"2024-03-22T01:36:38.424707+0000","last_fullsized":"2024-03-22T01:36:38.424707+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:14:59.846727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":77,"num_write_kb":837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"93'5","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.305472+0000","last_change":"2024-03-22T01:35:16.749643+0000","last_active":"2024-03-22T01:36:38.305472+0000","last_peered":"2024-03-22T01:36:38.305472+0000","last_clean":"2024-03-22T01:36:38.305472+0000","last_became_active":"2024-03-22T01:35:16.749311+0000","last_became_peered":"2024-03-22T01:35:16.749311+0000","last_unstale":"2024-03-22T01:36:38.305472+0000","last_undegraded":"2024-03-22T01:36:38.305472+0000","last_fullsized":"2024-03-22T01:36:38.305472+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:23:18.666007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,7],"acting":[9,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.190868+0000","last_change":"2024-03-22T01:34:19.523959+0000","last_active":"2024-03-22T01:36:38.190868+0000","last_peered":"2024-03-22T01:36:38.190868+0000","last_clean":"2024-03-22T01:36:38.190868+0000","last_became_active":"2024-03-22T01:34:19.523556+0000","last_became_peered":"2024-03-22T01:34:19.523556+0000","last_unstale":"2024-03-22T01:36:38.190868+0000","last_undegraded":"2024-03-22T01:36:38.190868+0000","last_fullsized":"2024-03-22T01:36:38.190868+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:22:21.623042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"86'2","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.499880+0000","last_change":"2024-03-22T01:34:14.463434+0000","last_active":"2024-03-22T01:36:38.499880+0000","last_peered":"2024-03-22T01:36:38.499880+0000","last_clean":"2024-03-22T01:36:38.499880+0000","last_became_active":"2024-03-22T01:34:14.463075+0000","last_became_peered":"2024-03-22T01:34:14.463075+0000","last_unstale":"2024-03-22T01:36:38.499880+0000","last_undegraded":"2024-03-22T01:36:38.499880+0000","last_fullsized":"2024-03-22T01:36:38.499880+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:43:18.223766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"93'7","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.502109+0000","last_change":"2024-03-22T01:34:15.482695+0000","last_active":"2024-03-22T01:36:38.502109+0000","last_peered":"2024-03-22T01:36:38.502109+0000","last_clean":"2024-03-22T01:36:38.502109+0000","last_became_active":"2024-03-22T01:34:15.481881+0000","last_became_peered":"2024-03-22T01:34:15.481881+0000","last_unstale":"2024-03-22T01:36:38.502109+0000","last_undegraded":"2024-03-22T01:36:38.502109+0000","last_fullsized":"2024-03-22T01:36:38.502109+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:44:26.290714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":25,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500458+0000","last_change":"2024-03-22T01:34:19.668252+0000","last_active":"2024-03-22T01:36:38.500458+0000","last_peered":"2024-03-22T01:36:38.500458+0000","last_clean":"2024-03-22T01:36:38.500458+0000","last_became_active":"2024-03-22T01:34:19.667830+0000","last_became_peered":"2024-03-22T01:34:19.667830+0000","last_unstale":"2024-03-22T01:36:38.500458+0000","last_undegraded":"2024-03-22T01:36:38.500458+0000","last_fullsized":"2024-03-22T01:36:38.500458+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:08:29.602899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.459520+0000","last_change":"2024-03-22T01:34:14.473020+0000","last_active":"2024-03-22T01:36:38.459520+0000","last_peered":"2024-03-22T01:36:38.459520+0000","last_clean":"2024-03-22T01:36:38.459520+0000","last_became_active":"2024-03-22T01:34:14.472689+0000","last_became_peered":"2024-03-22T01:34:14.472689+0000","last_unstale":"2024-03-22T01:36:38.459520+0000","last_undegraded":"2024-03-22T01:36:38.459520+0000","last_fullsized":"2024-03-22T01:36:38.459520+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:04:19.724685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"93'2","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.307503+0000","last_change":"2024-03-22T01:34:15.491613+0000","last_active":"2024-03-22T01:36:38.307503+0000","last_peered":"2024-03-22T01:36:38.307503+0000","last_clean":"2024-03-22T01:36:38.307503+0000","last_became_active":"2024-03-22T01:34:15.490922+0000","last_became_peered":"2024-03-22T01:34:15.490922+0000","last_unstale":"2024-03-22T01:36:38.307503+0000","last_undegraded":"2024-03-22T01:36:38.307503+0000","last_fullsized":"2024-03-22T01:36:38.307503+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:09:33.177646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":25,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.306867+0000","last_change":"2024-03-22T01:34:19.534537+0000","last_active":"2024-03-22T01:36:38.306867+0000","last_peered":"2024-03-22T01:36:38.306867+0000","last_clean":"2024-03-22T01:36:38.306867+0000","last_became_active":"2024-03-22T01:34:19.534038+0000","last_became_peered":"2024-03-22T01:34:19.534038+0000","last_unstale":"2024-03-22T01:36:38.306867+0000","last_undegraded":"2024-03-22T01:36:38.306867+0000","last_fullsized":"2024-03-22T01:36:38.306867+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:59:26.020000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.307492+0000","last_change":"2024-03-22T01:34:14.464724+0000","last_active":"2024-03-22T01:36:38.307492+0000","last_peered":"2024-03-22T01:36:38.307492+0000","last_clean":"2024-03-22T01:36:38.307492+0000","last_became_active":"2024-03-22T01:34:14.464056+0000","last_became_peered":"2024-03-22T01:34:14.464056+0000","last_unstale":"2024-03-22T01:36:38.307492+0000","last_undegraded":"2024-03-22T01:36:38.307492+0000","last_fullsized":"2024-03-22T01:36:38.307492+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:49:39.098126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"93'9","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500261+0000","last_change":"2024-03-22T01:34:15.488934+0000","last_active":"2024-03-22T01:36:38.500261+0000","last_peered":"2024-03-22T01:36:38.500261+0000","last_clean":"2024-03-22T01:36:38.500261+0000","last_became_active":"2024-03-22T01:34:15.488585+0000","last_became_peered":"2024-03-22T01:34:15.488585+0000","last_unstale":"2024-03-22T01:36:38.500261+0000","last_undegraded":"2024-03-22T01:36:38.500261+0000","last_fullsized":"2024-03-22T01:36:38.500261+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:59:04.554614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"91'1","reported_seq":26,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500321+0000","last_change":"2024-03-22T01:34:19.534096+0000","last_active":"2024-03-22T01:36:38.500321+0000","last_peered":"2024-03-22T01:36:38.500321+0000","last_clean":"2024-03-22T01:36:38.500321+0000","last_became_active":"2024-03-22T01:34:19.533645+0000","last_became_peered":"2024-03-22T01:34:19.533645+0000","last_unstale":"2024-03-22T01:36:38.500321+0000","last_undegraded":"2024-03-22T01:36:38.500321+0000","last_fullsized":"2024-03-22T01:36:38.500321+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:50:21.259146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.190987+0000","last_change":"2024-03-22T01:34:14.469529+0000","last_active":"2024-03-22T01:36:38.190987+0000","last_peered":"2024-03-22T01:36:38.190987+0000","last_clean":"2024-03-22T01:36:38.190987+0000","last_became_active":"2024-03-22T01:34:14.469204+0000","last_became_peered":"2024-03-22T01:34:14.469204+0000","last_unstale":"2024-03-22T01:36:38.190987+0000","last_undegraded":"2024-03-22T01:36:38.190987+0000","last_fullsized":"2024-03-22T01:36:38.190987+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:39:57.048537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"93'8","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.307968+0000","last_change":"2024-03-22T01:34:15.501324+0000","last_active":"2024-03-22T01:36:38.307968+0000","last_peered":"2024-03-22T01:36:38.307968+0000","last_clean":"2024-03-22T01:36:38.307968+0000","last_became_active":"2024-03-22T01:34:15.501015+0000","last_became_peered":"2024-03-22T01:34:15.501015+0000","last_unstale":"2024-03-22T01:36:38.307968+0000","last_undegraded":"2024-03-22T01:36:38.307968+0000","last_fullsized":"2024-03-22T01:36:38.307968+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:35:27.673019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500540+0000","last_change":"2024-03-22T01:34:19.667806+0000","last_active":"2024-03-22T01:36:38.500540+0000","last_peered":"2024-03-22T01:36:38.500540+0000","last_clean":"2024-03-22T01:36:38.500540+0000","last_became_active":"2024-03-22T01:34:19.667249+0000","last_became_peered":"2024-03-22T01:34:19.667249+0000","last_unstale":"2024-03-22T01:36:38.500540+0000","last_undegraded":"2024-03-22T01:36:38.500540+0000","last_fullsized":"2024-03-22T01:36:38.500540+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:44:22.476882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"86'2","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.307691+0000","last_change":"2024-03-22T01:34:14.471800+0000","last_active":"2024-03-22T01:36:38.307691+0000","last_peered":"2024-03-22T01:36:38.307691+0000","last_clean":"2024-03-22T01:36:38.307691+0000","last_became_active":"2024-03-22T01:34:14.471422+0000","last_became_peered":"2024-03-22T01:34:14.471422+0000","last_unstale":"2024-03-22T01:36:38.307691+0000","last_undegraded":"2024-03-22T01:36:38.307691+0000","last_fullsized":"2024-03-22T01:36:38.307691+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:25:58.546331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"93'5","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.300283+0000","last_change":"2024-03-22T01:34:15.487348+0000","last_active":"2024-03-22T01:36:38.300283+0000","last_peered":"2024-03-22T01:36:38.300283+0000","last_clean":"2024-03-22T01:36:38.300283+0000","last_became_active":"2024-03-22T01:34:15.486788+0000","last_became_peered":"2024-03-22T01:34:15.486788+0000","last_unstale":"2024-03-22T01:36:38.300283+0000","last_undegraded":"2024-03-22T01:36:38.300283+0000","last_fullsized":"2024-03-22T01:36:38.300283+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:10:31.489021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.459783+0000","last_change":"2024-03-22T01:34:19.521490+0000","last_active":"2024-03-22T01:36:38.459783+0000","last_peered":"2024-03-22T01:36:38.459783+0000","last_clean":"2024-03-22T01:36:38.459783+0000","last_became_active":"2024-03-22T01:34:19.521047+0000","last_became_peered":"2024-03-22T01:34:19.521047+0000","last_unstale":"2024-03-22T01:36:38.459783+0000","last_undegraded":"2024-03-22T01:36:38.459783+0000","last_fullsized":"2024-03-22T01:36:38.459783+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:36:50.176756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":17,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.459782+0000","last_change":"2024-03-22T01:35:16.749731+0000","last_active":"2024-03-22T01:36:38.459782+0000","last_peered":"2024-03-22T01:36:38.459782+0000","last_clean":"2024-03-22T01:36:38.459782+0000","last_became_active":"2024-03-22T01:35:16.749329+0000","last_became_peered":"2024-03-22T01:35:16.749329+0000","last_unstale":"2024-03-22T01:36:38.459782+0000","last_undegraded":"2024-03-22T01:36:38.459782+0000","last_fullsized":"2024-03-22T01:36:38.459782+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:23:27.386061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"93'7","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.300670+0000","last_change":"2024-03-22T01:34:15.491683+0000","last_active":"2024-03-22T01:36:38.300670+0000","last_peered":"2024-03-22T01:36:38.300670+0000","last_clean":"2024-03-22T01:36:38.300670+0000","last_became_active":"2024-03-22T01:34:15.491030+0000","last_became_peered":"2024-03-22T01:34:15.491030+0000","last_unstale":"2024-03-22T01:36:38.300670+0000","last_undegraded":"2024-03-22T01:36:38.300670+0000","last_fullsized":"2024-03-22T01:36:38.300670+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:02:10.115971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500471+0000","last_change":"2024-03-22T01:36:17.767646+0000","last_active":"2024-03-22T01:36:38.500471+0000","last_peered":"2024-03-22T01:36:38.500471+0000","last_clean":"2024-03-22T01:36:38.500471+0000","last_became_active":"2024-03-22T01:36:17.767257+0000","last_became_peered":"2024-03-22T01:36:17.767257+0000","last_unstale":"2024-03-22T01:36:38.500471+0000","last_undegraded":"2024-03-22T01:36:38.500471+0000","last_fullsized":"2024-03-22T01:36:38.500471+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:16:11.523941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,1,4],"acting":[0,8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"96'39","reported_seq":79,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.419218+0000","last_change":"2024-03-22T01:34:14.467755+0000","last_active":"2024-03-22T01:36:38.419218+0000","last_peered":"2024-03-22T01:36:38.419218+0000","last_clean":"2024-03-22T01:36:38.419218+0000","last_became_active":"2024-03-22T01:34:14.467426+0000","last_became_peered":"2024-03-22T01:34:14.467426+0000","last_unstale":"2024-03-22T01:36:38.419218+0000","last_undegraded":"2024-03-22T01:36:38.419218+0000","last_fullsized":"2024-03-22T01:36:38.419218+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:16:35.436872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"93'6","reported_seq":43,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.300373+0000","last_change":"2024-03-22T01:35:16.748667+0000","last_active":"2024-03-22T01:36:38.300373+0000","last_peered":"2024-03-22T01:36:38.300373+0000","last_clean":"2024-03-22T01:36:38.300373+0000","last_became_active":"2024-03-22T01:35:16.748303+0000","last_became_peered":"2024-03-22T01:35:16.748303+0000","last_unstale":"2024-03-22T01:36:38.300373+0000","last_undegraded":"2024-03-22T01:36:38.300373+0000","last_fullsized":"2024-03-22T01:36:38.300373+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:40:55.207127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.314390+0000","last_change":"2024-03-22T01:34:19.523631+0000","last_active":"2024-03-22T01:36:38.314390+0000","last_peered":"2024-03-22T01:36:38.314390+0000","last_clean":"2024-03-22T01:36:38.314390+0000","last_became_active":"2024-03-22T01:34:19.522968+0000","last_became_peered":"2024-03-22T01:34:19.522968+0000","last_unstale":"2024-03-22T01:36:38.314390+0000","last_undegraded":"2024-03-22T01:36:38.314390+0000","last_fullsized":"2024-03-22T01:36:38.314390+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:18.497392+0000","last_clean_scrub_stamp":"2024-03-22T01:34:18.497392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:27:24.182886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.189345+0000","last_change":"2024-03-22T01:34:14.464113+0000","last_active":"2024-03-22T01:36:38.189345+0000","last_peered":"2024-03-22T01:36:38.189345+0000","last_clean":"2024-03-22T01:36:38.189345+0000","last_became_active":"2024-03-22T01:34:14.463389+0000","last_became_peered":"2024-03-22T01:34:14.463389+0000","last_unstale":"2024-03-22T01:36:38.189345+0000","last_undegraded":"2024-03-22T01:36:38.189345+0000","last_fullsized":"2024-03-22T01:36:38.189345+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:13.438188+0000","last_clean_scrub_stamp":"2024-03-22T01:34:13.438188+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:08:42.376753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"93'9","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.300652+0000","last_change":"2024-03-22T01:34:15.492941+0000","last_active":"2024-03-22T01:36:38.300652+0000","last_peered":"2024-03-22T01:36:38.300652+0000","last_clean":"2024-03-22T01:36:38.300652+0000","last_became_active":"2024-03-22T01:34:15.492598+0000","last_became_peered":"2024-03-22T01:34:15.492598+0000","last_unstale":"2024-03-22T01:36:38.300652+0000","last_undegraded":"2024-03-22T01:36:38.300652+0000","last_fullsized":"2024-03-22T01:36:38.300652+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:22:35.502404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"93'9","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500326+0000","last_change":"2024-03-22T01:34:15.494548+0000","last_active":"2024-03-22T01:36:38.500326+0000","last_peered":"2024-03-22T01:36:38.500326+0000","last_clean":"2024-03-22T01:36:38.500326+0000","last_became_active":"2024-03-22T01:34:15.494228+0000","last_became_peered":"2024-03-22T01:34:15.494228+0000","last_unstale":"2024-03-22T01:36:38.500326+0000","last_undegraded":"2024-03-22T01:36:38.500326+0000","last_fullsized":"2024-03-22T01:36:38.500326+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:43:26.011030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"93'12","reported_seq":44,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.191036+0000","last_change":"2024-03-22T01:34:15.490843+0000","last_active":"2024-03-22T01:36:38.191036+0000","last_peered":"2024-03-22T01:36:38.191036+0000","last_clean":"2024-03-22T01:36:38.191036+0000","last_became_active":"2024-03-22T01:34:15.490474+0000","last_became_peered":"2024-03-22T01:34:15.490474+0000","last_unstale":"2024-03-22T01:36:38.191036+0000","last_undegraded":"2024-03-22T01:36:38.191036+0000","last_fullsized":"2024-03-22T01:36:38.191036+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:11:31.488132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"93'8","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.190920+0000","last_change":"2024-03-22T01:34:15.502213+0000","last_active":"2024-03-22T01:36:38.190920+0000","last_peered":"2024-03-22T01:36:38.190920+0000","last_clean":"2024-03-22T01:36:38.190920+0000","last_became_active":"2024-03-22T01:34:15.501464+0000","last_became_peered":"2024-03-22T01:34:15.501464+0000","last_unstale":"2024-03-22T01:36:38.190920+0000","last_undegraded":"2024-03-22T01:36:38.190920+0000","last_fullsized":"2024-03-22T01:36:38.190920+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:38:02.090365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"93'7","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.501571+0000","last_change":"2024-03-22T01:34:15.481282+0000","last_active":"2024-03-22T01:36:38.501571+0000","last_peered":"2024-03-22T01:36:38.501571+0000","last_clean":"2024-03-22T01:36:38.501571+0000","last_became_active":"2024-03-22T01:34:15.480465+0000","last_became_peered":"2024-03-22T01:34:15.480465+0000","last_unstale":"2024-03-22T01:36:38.501571+0000","last_undegraded":"2024-03-22T01:36:38.501571+0000","last_fullsized":"2024-03-22T01:36:38.501571+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:55:59.398543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"93'4","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.189395+0000","last_change":"2024-03-22T01:34:15.480910+0000","last_active":"2024-03-22T01:36:38.189395+0000","last_peered":"2024-03-22T01:36:38.189395+0000","last_clean":"2024-03-22T01:36:38.189395+0000","last_became_active":"2024-03-22T01:34:15.480601+0000","last_became_peered":"2024-03-22T01:34:15.480601+0000","last_unstale":"2024-03-22T01:36:38.189395+0000","last_undegraded":"2024-03-22T01:36:38.189395+0000","last_fullsized":"2024-03-22T01:36:38.189395+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:19:51.563598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"93'6","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500686+0000","last_change":"2024-03-22T01:34:15.490859+0000","last_active":"2024-03-22T01:36:38.500686+0000","last_peered":"2024-03-22T01:36:38.500686+0000","last_clean":"2024-03-22T01:36:38.500686+0000","last_became_active":"2024-03-22T01:34:15.490337+0000","last_became_peered":"2024-03-22T01:34:15.490337+0000","last_unstale":"2024-03-22T01:36:38.500686+0000","last_undegraded":"2024-03-22T01:36:38.500686+0000","last_fullsized":"2024-03-22T01:36:38.500686+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:46:07.072015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"93'15","reported_seq":47,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.191416+0000","last_change":"2024-03-22T01:34:15.495531+0000","last_active":"2024-03-22T01:36:38.191416+0000","last_peered":"2024-03-22T01:36:38.191416+0000","last_clean":"2024-03-22T01:36:38.191416+0000","last_became_active":"2024-03-22T01:34:15.495216+0000","last_became_peered":"2024-03-22T01:34:15.495216+0000","last_unstale":"2024-03-22T01:36:38.191416+0000","last_undegraded":"2024-03-22T01:36:38.191416+0000","last_fullsized":"2024-03-22T01:36:38.191416+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:39:22.575926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"93'2","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.308446+0000","last_change":"2024-03-22T01:34:15.499430+0000","last_active":"2024-03-22T01:36:38.308446+0000","last_peered":"2024-03-22T01:36:38.308446+0000","last_clean":"2024-03-22T01:36:38.308446+0000","last_became_active":"2024-03-22T01:34:15.499114+0000","last_became_peered":"2024-03-22T01:34:15.499114+0000","last_unstale":"2024-03-22T01:36:38.308446+0000","last_undegraded":"2024-03-22T01:36:38.308446+0000","last_fullsized":"2024-03-22T01:36:38.308446+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:23:01.781963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"93'6","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.314538+0000","last_change":"2024-03-22T01:34:15.485449+0000","last_active":"2024-03-22T01:36:38.314538+0000","last_peered":"2024-03-22T01:36:38.314538+0000","last_clean":"2024-03-22T01:36:38.314538+0000","last_became_active":"2024-03-22T01:34:15.485084+0000","last_became_peered":"2024-03-22T01:34:15.485084+0000","last_unstale":"2024-03-22T01:36:38.314538+0000","last_undegraded":"2024-03-22T01:36:38.314538+0000","last_fullsized":"2024-03-22T01:36:38.314538+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:53:25.632719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"93'8","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.307592+0000","last_change":"2024-03-22T01:34:15.495521+0000","last_active":"2024-03-22T01:36:38.307592+0000","last_peered":"2024-03-22T01:36:38.307592+0000","last_clean":"2024-03-22T01:36:38.307592+0000","last_became_active":"2024-03-22T01:34:15.495218+0000","last_became_peered":"2024-03-22T01:34:15.495218+0000","last_unstale":"2024-03-22T01:36:38.307592+0000","last_undegraded":"2024-03-22T01:36:38.307592+0000","last_fullsized":"2024-03-22T01:36:38.307592+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:04:32.569934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"93'10","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.460006+0000","last_change":"2024-03-22T01:34:15.489213+0000","last_active":"2024-03-22T01:36:38.460006+0000","last_peered":"2024-03-22T01:36:38.460006+0000","last_clean":"2024-03-22T01:36:38.460006+0000","last_became_active":"2024-03-22T01:34:15.488887+0000","last_became_peered":"2024-03-22T01:34:15.488887+0000","last_unstale":"2024-03-22T01:36:38.460006+0000","last_undegraded":"2024-03-22T01:36:38.460006+0000","last_fullsized":"2024-03-22T01:36:38.460006+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:53:39.983144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"93'11","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.189182+0000","last_change":"2024-03-22T01:34:15.485632+0000","last_active":"2024-03-22T01:36:38.189182+0000","last_peered":"2024-03-22T01:36:38.189182+0000","last_clean":"2024-03-22T01:36:38.189182+0000","last_became_active":"2024-03-22T01:34:15.485341+0000","last_became_peered":"2024-03-22T01:34:15.485341+0000","last_unstale":"2024-03-22T01:36:38.189182+0000","last_undegraded":"2024-03-22T01:36:38.189182+0000","last_fullsized":"2024-03-22T01:36:38.189182+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:33:00.248555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"93'8","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-22T01:36:38.500583+0000","last_change":"2024-03-22T01:34:15.483593+0000","last_active":"2024-03-22T01:36:38.500583+0000","last_peered":"2024-03-22T01:36:38.500583+0000","last_clean":"2024-03-22T01:36:38.500583+0000","last_became_active":"2024-03-22T01:34:15.483201+0000","last_became_peered":"2024-03-22T01:34:15.483201+0000","last_unstale":"2024-03-22T01:36:38.500583+0000","last_undegraded":"2024-03-22T01:36:38.500583+0000","last_fullsized":"2024-03-22T01:36:38.500583+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T01:34:14.445286+0000","last_clean_scrub_stamp":"2024-03-22T01:34:14.445286+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:03:10.955120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":595682,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":348,"num_read_kb":279,"num_write":200,"num_write_kb":1410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1880064,"data_stored":1787046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318314,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32244,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712908,"statfs":{"total":95995035648,"available":95962017792,"internally_reserved":0,"allocated":1089536,"data_stored":538835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579951,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37064,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708088,"statfs":{"total":95995035648,"available":95957082112,"internally_reserved":0,"allocated":1765376,"data_stored":1189214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776180,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28476,"kb_used_data":1456,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716676,"statfs":{"total":95995035648,"available":95965876224,"internally_reserved":0,"allocated":1490944,"data_stored":936729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037816,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32636,"kb_used_data":1456,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712516,"statfs":{"total":95995035648,"available":95961616384,"internally_reserved":0,"allocated":1490944,"data_stored":932655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299454,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28672,"kb_used_data":1652,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716480,"statfs":{"total":95995035648,"available":95965675520,"internally_reserved":0,"allocated":1691648,"data_stored":1133881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462978,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1085440,"data_stored":534739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574939,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28156,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716996,"statfs":{"total":95995035648,"available":95966203904,"internally_reserved":0,"allocated":1163264,"data_stored":604807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738464,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1073152,"data_stored":524905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411414,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28160,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716992,"statfs":{"total":95995035648,"available":95966199808,"internally_reserved":0,"allocated":1167360,"data_stored":604841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149777,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36920,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708232,"statfs":{"total":95995035648,"available":95957229568,"internally_reserved":0,"allocated":1622016,"data_stored":1059659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888141,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1093632,"data_stored":538857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724616,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37304,"kb_used_data":1788,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93707848,"statfs":{"total":95995035648,"available":95956836352,"internally_reserved":0,"allocated":1830912,"data_stored":1260985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":70068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":70102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":536576,"data_stored":524920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":131782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":393820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":397312,"data_stored":393798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":225280,"data_stored":201862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-22T01:36:41.171 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T01:36:41.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:40 smithi098 ceph-mon[36468]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:36:41.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:40 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/2728439601' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:36:41.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:40 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:36:41.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:40 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:36:41.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:40 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:36:41.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:40 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:36:41.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:40 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:36:41.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:40 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:36:41.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.310+0000 7f91c413e640 1 -- 172.21.15.72:0/4105086595 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f91bc107940 msgr2=0x7f91bc109d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:41.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.310+0000 7f91c413e640 1 --2- 172.21.15.72:0/4105086595 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f91bc107940 0x7f91bc109d30 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f91b4009ca0 tx=0x7f91b402c0f0 comp rx=0 tx=0).stop 2024-03-22T01:36:41.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.310+0000 7f91c413e640 1 -- 172.21.15.72:0/4105086595 shutdown_connections 2024-03-22T01:36:41.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.310+0000 7f91c413e640 1 --2- 172.21.15.72:0/4105086595 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f91bc105010 0x7f91bc107400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:41.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.310+0000 7f91c413e640 1 --2- 172.21.15.72:0/4105086595 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f91bc107940 0x7f91bc109d30 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:41.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.310+0000 7f91c413e640 1 --2- 172.21.15.72:0/4105086595 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91bc1026e0 0x7f91bc104ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:41.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.310+0000 7f91c413e640 1 -- 172.21.15.72:0/4105086595 >> 172.21.15.72:0/4105086595 conn(0x7f91bc0fc030 msgr2=0x7f91bc0fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:41.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.310+0000 7f91c413e640 1 -- 172.21.15.72:0/4105086595 shutdown_connections 2024-03-22T01:36:41.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.310+0000 7f91c413e640 1 -- 172.21.15.72:0/4105086595 wait complete. 2024-03-22T01:36:41.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.311+0000 7f91c413e640 1 Processor -- start 2024-03-22T01:36:41.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.311+0000 7f91c413e640 1 -- start start 2024-03-22T01:36:41.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.311+0000 7f91c413e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f91bc1026e0 0x7f91bc197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:41.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.311+0000 7f91c413e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91bc105010 0x7f91bc1983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:41.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.311+0000 7f91c413e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f91bc107940 0x7f91bc19f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:41.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.311+0000 7f91c413e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f91bc10d860 con 0x7f91bc107940 2024-03-22T01:36:41.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.311+0000 7f91c413e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f91bc10d6e0 con 0x7f91bc1026e0 2024-03-22T01:36:41.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.311+0000 7f91c413e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f91bc10d9e0 con 0x7f91bc105010 2024-03-22T01:36:41.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.311+0000 7f91c26db640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f91bc107940 0x7f91bc19f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:41.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.311+0000 7f91c1eda640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f91bc1026e0 0x7f91bc197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:41.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.311+0000 7f91c26db640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f91bc107940 0x7f91bc19f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46746/0 (socket says 172.21.15.72:46746) 2024-03-22T01:36:41.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.311+0000 7f91c26db640 1 -- 172.21.15.72:0/2499245168 learned_addr learned my addr 172.21.15.72:0/2499245168 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:41.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.311+0000 7f91c16d9640 1 --2- 172.21.15.72:0/2499245168 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91bc105010 0x7f91bc1983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:41.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.311+0000 7f91c1eda640 1 -- 172.21.15.72:0/2499245168 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91bc105010 msgr2=0x7f91bc1983c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:41.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.311+0000 7f91c1eda640 1 --2- 172.21.15.72:0/2499245168 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91bc105010 0x7f91bc1983c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:41.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.311+0000 7f91c1eda640 1 -- 172.21.15.72:0/2499245168 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f91bc107940 msgr2=0x7f91bc19f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:41.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.311+0000 7f91c1eda640 1 --2- 172.21.15.72:0/2499245168 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f91bc107940 0x7f91bc19f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:41.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.311+0000 7f91c1eda640 1 -- 172.21.15.72:0/2499245168 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91bc19faa0 con 0x7f91bc1026e0 2024-03-22T01:36:41.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.311+0000 7f91c26db640 1 --2- 172.21.15.72:0/2499245168 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f91bc107940 0x7f91bc19f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:36:41.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.312+0000 7f91c1eda640 1 --2- 172.21.15.72:0/2499245168 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f91bc1026e0 0x7f91bc197e80 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f91ac00e9f0 tx=0x7f91ac00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:41.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.312+0000 7f91b2ffd640 1 -- 172.21.15.72:0/2499245168 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91ac00ccc0 con 0x7f91bc1026e0 2024-03-22T01:36:41.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.312+0000 7f91c413e640 1 -- 172.21.15.72:0/2499245168 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91bc19fd90 con 0x7f91bc1026e0 2024-03-22T01:36:41.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.312+0000 7f91b2ffd640 1 -- 172.21.15.72:0/2499245168 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f91ac005020 con 0x7f91bc1026e0 2024-03-22T01:36:41.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.312+0000 7f91b2ffd640 1 -- 172.21.15.72:0/2499245168 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91ac0049e0 con 0x7f91bc1026e0 2024-03-22T01:36:41.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.312+0000 7f91c413e640 1 -- 172.21.15.72:0/2499245168 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91bc1a0220 con 0x7f91bc1026e0 2024-03-22T01:36:41.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.313+0000 7f91c413e640 1 -- 172.21.15.72:0/2499245168 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91bc0fdb20 con 0x7f91bc1026e0 2024-03-22T01:36:41.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.314+0000 7f91b2ffd640 1 -- 172.21.15.72:0/2499245168 <== mon.2 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 22) ==== 137974+0+0 (secure 0 0 0) 0x7f91ac005560 con 0x7f91bc1026e0 2024-03-22T01:36:41.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.314+0000 7f91b2ffd640 1 --2- 172.21.15.72:0/2499245168 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f91a80a3090 0x7f91a80a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:41.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.314+0000 7f91c16d9640 1 --2- 172.21.15.72:0/2499245168 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f91a80a3090 0x7f91a80a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:41.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.314+0000 7f91b2ffd640 1 -- 172.21.15.72:0/2499245168 <== mon.2 v2:172.21.15.89:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f91ac0cfb10 con 0x7f91bc1026e0 2024-03-22T01:36:41.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.315+0000 7f91c16d9640 1 --2- 172.21.15.72:0/2499245168 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f91a80a3090 0x7f91a80a5550 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f91b80095f0 tx=0x7f91b8005f90 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:41.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.319+0000 7f91b2ffd640 1 -- 172.21.15.72:0/2499245168 <== mon.2 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f91ac0051d0 con 0x7f91bc1026e0 2024-03-22T01:36:41.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.503+0000 7f91c413e640 1 -- 172.21.15.72:0/2499245168 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f91bc062790 con 0x7f91bc1026e0 2024-03-22T01:36:41.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.504+0000 7f91b2ffd640 1 -- 172.21.15.72:0/2499245168 <== mon.2 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) ==== 74+0+25134 (secure 0 0 0) 0x7f91ac09c250 con 0x7f91bc1026e0 2024-03-22T01:36:41.504 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:36:41.504 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":96,"fsid":"022a99bc-e7eb-11ee-95cd-87774f69a715","created":"2024-03-22T01:26:23.893672+0000","modified":"2024-03-22T01:36:38.183837+0000","last_up_change":"2024-03-22T01:33:25.223684+0000","last_in_change":"2024-03-22T01:33:05.010331+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T01:29:37.438503+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-22T01:34:12.610422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-22T01:34:13.761686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-22T01:34:17.507039+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"16c15bb0-db92-4191-9063-38b4890e81e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6803","nonce":661906791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6805","nonce":661906791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6809","nonce":661906791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6807","nonce":661906791}]},"public_addr":"172.21.15.72:6803/661906791","cluster_addr":"172.21.15.72:6805/661906791","heartbeat_back_addr":"172.21.15.72:6809/661906791","heartbeat_front_addr":"172.21.15.72:6807/661906791","state":["exists","up"]},{"osd":1,"uuid":"981290ba-97dc-496b-aa22-d51704184b10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6801","nonce":2870590550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6803","nonce":2870590550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6807","nonce":2870590550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6805","nonce":2870590550}]},"public_addr":"172.21.15.89:6801/2870590550","cluster_addr":"172.21.15.89:6803/2870590550","heartbeat_back_addr":"172.21.15.89:6807/2870590550","heartbeat_front_addr":"172.21.15.89:6805/2870590550","state":["exists","up"]},{"osd":2,"uuid":"f07ed658-f545-46a1-8edd-8b1d5d49885c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6801","nonce":3865778118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6803","nonce":3865778118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6807","nonce":3865778118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6805","nonce":3865778118}]},"public_addr":"172.21.15.98:6801/3865778118","cluster_addr":"172.21.15.98:6803/3865778118","heartbeat_back_addr":"172.21.15.98:6807/3865778118","heartbeat_front_addr":"172.21.15.98:6805/3865778118","state":["exists","up"]},{"osd":3,"uuid":"ccc4bc40-2c48-46d2-8548-0c05b44f9f1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6811","nonce":1865951395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6813","nonce":1865951395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6817","nonce":1865951395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6815","nonce":1865951395}]},"public_addr":"172.21.15.72:6811/1865951395","cluster_addr":"172.21.15.72:6813/1865951395","heartbeat_back_addr":"172.21.15.72:6817/1865951395","heartbeat_front_addr":"172.21.15.72:6815/1865951395","state":["exists","up"]},{"osd":4,"uuid":"1812439f-810a-4ed2-9395-c00c9a085a13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6809","nonce":2683760795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6811","nonce":2683760795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6815","nonce":2683760795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6813","nonce":2683760795}]},"public_addr":"172.21.15.89:6809/2683760795","cluster_addr":"172.21.15.89:6811/2683760795","heartbeat_back_addr":"172.21.15.89:6815/2683760795","heartbeat_front_addr":"172.21.15.89:6813/2683760795","state":["exists","up"]},{"osd":5,"uuid":"e78954d8-f4e3-4d58-8536-d2d59c843386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6809","nonce":2635838572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6811","nonce":2635838572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6815","nonce":2635838572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6813","nonce":2635838572}]},"public_addr":"172.21.15.98:6809/2635838572","cluster_addr":"172.21.15.98:6811/2635838572","heartbeat_back_addr":"172.21.15.98:6815/2635838572","heartbeat_front_addr":"172.21.15.98:6813/2635838572","state":["exists","up"]},{"osd":6,"uuid":"dc6c8be2-b64c-4959-8015-7b0ea6d25599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6819","nonce":3158982661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6821","nonce":3158982661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6825","nonce":3158982661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6823","nonce":3158982661}]},"public_addr":"172.21.15.72:6819/3158982661","cluster_addr":"172.21.15.72:6821/3158982661","heartbeat_back_addr":"172.21.15.72:6825/3158982661","heartbeat_front_addr":"172.21.15.72:6823/3158982661","state":["exists","up"]},{"osd":7,"uuid":"7a73469f-cf3a-4306-ae93-ea46e21ebbbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6817","nonce":3075913234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6819","nonce":3075913234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6823","nonce":3075913234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6821","nonce":3075913234}]},"public_addr":"172.21.15.89:6817/3075913234","cluster_addr":"172.21.15.89:6819/3075913234","heartbeat_back_addr":"172.21.15.89:6823/3075913234","heartbeat_front_addr":"172.21.15.89:6821/3075913234","state":["exists","up"]},{"osd":8,"uuid":"c4b13bee-ea4d-4344-95fb-2622ecbb8196","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6817","nonce":1813255577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6819","nonce":1813255577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6823","nonce":1813255577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6821","nonce":1813255577}]},"public_addr":"172.21.15.98:6817/1813255577","cluster_addr":"172.21.15.98:6819/1813255577","heartbeat_back_addr":"172.21.15.98:6823/1813255577","heartbeat_front_addr":"172.21.15.98:6821/1813255577","state":["exists","up"]},{"osd":9,"uuid":"eefb0a44-4a36-4e55-9d04-0501bf718db6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6827","nonce":3377749206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6829","nonce":3377749206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6832","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6833","nonce":3377749206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6831","nonce":3377749206}]},"public_addr":"172.21.15.72:6827/3377749206","cluster_addr":"172.21.15.72:6829/3377749206","heartbeat_back_addr":"172.21.15.72:6833/3377749206","heartbeat_front_addr":"172.21.15.72:6831/3377749206","state":["exists","up"]},{"osd":10,"uuid":"8741d25f-40dd-40d7-b040-5cc5e411eb98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6825","nonce":885058786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6827","nonce":885058786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6831","nonce":885058786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6829","nonce":885058786}]},"public_addr":"172.21.15.89:6825/885058786","cluster_addr":"172.21.15.89:6827/885058786","heartbeat_back_addr":"172.21.15.89:6831/885058786","heartbeat_front_addr":"172.21.15.89:6829/885058786","state":["exists","up"]},{"osd":11,"uuid":"8d02e957-eda4-4184-b5c6-5eb3b8b38060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6825","nonce":1517095838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6827","nonce":1517095838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6831","nonce":1517095838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6829","nonce":1517095838}]},"public_addr":"172.21.15.98:6825/1517095838","cluster_addr":"172.21.15.98:6827/1517095838","heartbeat_back_addr":"172.21.15.98:6831/1517095838","heartbeat_front_addr":"172.21.15.98:6829/1517095838","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:28:47.307652+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:11.037135+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:34.619188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:59.350738+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:30:23.787337+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:30:48.994154+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:31:13.690350+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:31:39.628116+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:04.723912+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:30.689741+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:57.229508+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:33:23.154537+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":7}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:0/2032770149":"2024-03-23T01:36:38.183806+0000","172.21.15.72:6801/401403200":"2024-03-23T01:36:38.183806+0000","172.21.15.72:0/1161486989":"2024-03-23T01:36:38.183806+0000","172.21.15.72:6800/401403200":"2024-03-23T01:36:38.183806+0000","172.21.15.72:0/121087196":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/2451857761":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/1228477995":"2024-03-23T01:27:15.339809+0000","172.21.15.72:6801/1513232913":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/4223447038":"2024-03-23T01:36:38.183806+0000","172.21.15.72:0/3558958264":"2024-03-23T01:26:50.074871+0000","172.21.15.72:0/1510187198":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6800/3518012916":"2024-03-23T01:26:50.074871+0000","172.21.15.72:0/608566205":"2024-03-23T01:36:38.183806+0000","172.21.15.72:6801/3518012916":"2024-03-23T01:26:50.074871+0000","172.21.15.72:0/1372801041":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6800/1513232913":"2024-03-23T01:27:15.339809+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T01:36:41.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.508+0000 7f91c413e640 1 -- 172.21.15.72:0/2499245168 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f91a80a3090 msgr2=0x7f91a80a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:41.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.508+0000 7f91c413e640 1 --2- 172.21.15.72:0/2499245168 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f91a80a3090 0x7f91a80a5550 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f91b80095f0 tx=0x7f91b8005f90 comp rx=0 tx=0).stop 2024-03-22T01:36:41.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.508+0000 7f91c413e640 1 -- 172.21.15.72:0/2499245168 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f91bc1026e0 msgr2=0x7f91bc197e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:41.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.508+0000 7f91c413e640 1 --2- 172.21.15.72:0/2499245168 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f91bc1026e0 0x7f91bc197e80 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f91ac00e9f0 tx=0x7f91ac00eeb0 comp rx=0 tx=0).stop 2024-03-22T01:36:41.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.508+0000 7f91c413e640 1 -- 172.21.15.72:0/2499245168 shutdown_connections 2024-03-22T01:36:41.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.508+0000 7f91c413e640 1 --2- 172.21.15.72:0/2499245168 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f91a80a3090 0x7f91a80a5550 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:41.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.508+0000 7f91c413e640 1 --2- 172.21.15.72:0/2499245168 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f91bc107940 0x7f91bc19f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:41.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.508+0000 7f91c413e640 1 --2- 172.21.15.72:0/2499245168 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91bc105010 0x7f91bc1983c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:41.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.508+0000 7f91c413e640 1 --2- 172.21.15.72:0/2499245168 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f91bc1026e0 0x7f91bc197e80 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:41.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.508+0000 7f91c413e640 1 -- 172.21.15.72:0/2499245168 >> 172.21.15.72:0/2499245168 conn(0x7f91bc0fc030 msgr2=0x7f91bc1030e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:41.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.508+0000 7f91c413e640 1 -- 172.21.15.72:0/2499245168 shutdown_connections 2024-03-22T01:36:41.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.508+0000 7f91c413e640 1 -- 172.21.15.72:0/2499245168 wait complete. 2024-03-22T01:36:41.523 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-22T01:29:37.438503+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-22T01:34:12.610422+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-22T01:34:13.761686+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-22T01:34:17.507039+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-22T01:36:41.523 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-22T01:36:41.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.632+0000 7f3fded3e640 1 -- 172.21.15.72:0/295519075 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3fd8102630 msgr2=0x7f3fd8104a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:41.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.632+0000 7f3fded3e640 1 --2- 172.21.15.72:0/295519075 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3fd8102630 0x7f3fd8104a20 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc009900 tx=0x7f3fcc02bc30 comp rx=0 tx=0).stop 2024-03-22T01:36:41.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.632+0000 7f3fded3e640 1 -- 172.21.15.72:0/295519075 shutdown_connections 2024-03-22T01:36:41.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.632+0000 7f3fded3e640 1 --2- 172.21.15.72:0/295519075 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3fd8104f60 0x7f3fd8107350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:41.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.632+0000 7f3fded3e640 1 --2- 172.21.15.72:0/295519075 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fd8107890 0x7f3fd8109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:41.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.632+0000 7f3fded3e640 1 --2- 172.21.15.72:0/295519075 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3fd8102630 0x7f3fd8104a20 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:41.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.632+0000 7f3fded3e640 1 -- 172.21.15.72:0/295519075 >> 172.21.15.72:0/295519075 conn(0x7f3fd80fc020 msgr2=0x7f3fd80fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:41.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.632+0000 7f3fded3e640 1 -- 172.21.15.72:0/295519075 shutdown_connections 2024-03-22T01:36:41.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.632+0000 7f3fded3e640 1 -- 172.21.15.72:0/295519075 wait complete. 2024-03-22T01:36:41.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.633+0000 7f3fded3e640 1 Processor -- start 2024-03-22T01:36:41.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.633+0000 7f3fded3e640 1 -- start start 2024-03-22T01:36:41.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.633+0000 7f3fded3e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3fd8102630 0x7f3fd8197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:41.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.633+0000 7f3fded3e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fd8104f60 0x7f3fd8198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:41.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.633+0000 7f3fded3e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3fd8107890 0x7f3fd819f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:41.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.633+0000 7f3fded3e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f3fd810d850 con 0x7f3fd8107890 2024-03-22T01:36:41.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.633+0000 7f3fded3e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3fd810d6d0 con 0x7f3fd8102630 2024-03-22T01:36:41.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.633+0000 7f3fded3e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3fd810d9d0 con 0x7f3fd8104f60 2024-03-22T01:36:41.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.633+0000 7f3fdd2db640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3fd8107890 0x7f3fd819f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:41.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.633+0000 7f3fdcada640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3fd8102630 0x7f3fd8197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:41.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.633+0000 7f3fd7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fd8104f60 0x7f3fd8198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:41.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.633+0000 7f3fdd2db640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3fd8107890 0x7f3fd819f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46756/0 (socket says 172.21.15.72:46756) 2024-03-22T01:36:41.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.633+0000 7f3fdd2db640 1 -- 172.21.15.72:0/744148743 learned_addr learned my addr 172.21.15.72:0/744148743 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:41.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.633+0000 7f3fdd2db640 1 -- 172.21.15.72:0/744148743 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fd8104f60 msgr2=0x7f3fd8198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:41.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.633+0000 7f3fdd2db640 1 --2- 172.21.15.72:0/744148743 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fd8104f60 0x7f3fd8198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:41.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.633+0000 7f3fdd2db640 1 -- 172.21.15.72:0/744148743 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3fd8102630 msgr2=0x7f3fd8197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:41.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.633+0000 7f3fdd2db640 1 --2- 172.21.15.72:0/744148743 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3fd8102630 0x7f3fd8197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:41.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.633+0000 7f3fdd2db640 1 -- 172.21.15.72:0/744148743 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fd819faf0 con 0x7f3fd8107890 2024-03-22T01:36:41.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.633+0000 7f3fdd2db640 1 --2- 172.21.15.72:0/744148743 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3fd8107890 0x7f3fd819f480 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f3fd000eb10 tx=0x7f3fd000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:41.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.633+0000 7f3fd5ffb640 1 -- 172.21.15.72:0/744148743 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fd0004b20 con 0x7f3fd8107890 2024-03-22T01:36:41.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.634+0000 7f3fded3e640 1 -- 172.21.15.72:0/744148743 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fd819fde0 con 0x7f3fd8107890 2024-03-22T01:36:41.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.634+0000 7f3fd5ffb640 1 -- 172.21.15.72:0/744148743 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3fd0004cc0 con 0x7f3fd8107890 2024-03-22T01:36:41.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.634+0000 7f3fd5ffb640 1 -- 172.21.15.72:0/744148743 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fd0010760 con 0x7f3fd8107890 2024-03-22T01:36:41.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.634+0000 7f3fded3e640 1 -- 172.21.15.72:0/744148743 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fd81a02a0 con 0x7f3fd8107890 2024-03-22T01:36:41.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.635+0000 7f3fded3e640 1 -- 172.21.15.72:0/744148743 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3fd80fdb10 con 0x7f3fd8107890 2024-03-22T01:36:41.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.636+0000 7f3fd5ffb640 1 -- 172.21.15.72:0/744148743 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 22) ==== 137974+0+0 (secure 0 0 0) 0x7f3fd0005020 con 0x7f3fd8107890 2024-03-22T01:36:41.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.637+0000 7f3fd5ffb640 1 --2- 172.21.15.72:0/744148743 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f3fc80a3090 0x7f3fc80a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:41.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.637+0000 7f3fdcada640 1 --2- 172.21.15.72:0/744148743 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f3fc80a3090 0x7f3fc80a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:41.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.637+0000 7f3fd5ffb640 1 -- 172.21.15.72:0/744148743 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f3fd00d0250 con 0x7f3fd8107890 2024-03-22T01:36:41.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.637+0000 7f3fdcada640 1 --2- 172.21.15.72:0/744148743 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f3fc80a3090 0x7f3fc80a5550 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc02c7b0 tx=0x7f3fcc005fd0 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:41.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.641+0000 7f3fd5ffb640 1 -- 172.21.15.72:0/744148743 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3fd009c990 con 0x7f3fd8107890 2024-03-22T01:36:41.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.825+0000 7f3fded3e640 1 -- 172.21.15.72:0/744148743 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3fd802d600 con 0x7f3fd8107890 2024-03-22T01:36:41.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.825+0000 7f3fd5ffb640 1 -- 172.21.15.72:0/744148743 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v96) ==== 93+0+10 (secure 0 0 0) 0x7f3fd009c290 con 0x7f3fd8107890 2024-03-22T01:36:41.824 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-22T01:36:41.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.827+0000 7f3fded3e640 1 -- 172.21.15.72:0/744148743 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f3fc80a3090 msgr2=0x7f3fc80a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:41.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.827+0000 7f3fded3e640 1 --2- 172.21.15.72:0/744148743 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f3fc80a3090 0x7f3fc80a5550 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc02c7b0 tx=0x7f3fcc005fd0 comp rx=0 tx=0).stop 2024-03-22T01:36:41.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.827+0000 7f3fded3e640 1 -- 172.21.15.72:0/744148743 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3fd8107890 msgr2=0x7f3fd819f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:41.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.827+0000 7f3fded3e640 1 --2- 172.21.15.72:0/744148743 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3fd8107890 0x7f3fd819f480 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f3fd000eb10 tx=0x7f3fd000c2d0 comp rx=0 tx=0).stop 2024-03-22T01:36:41.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.827+0000 7f3fded3e640 1 -- 172.21.15.72:0/744148743 shutdown_connections 2024-03-22T01:36:41.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.827+0000 7f3fded3e640 1 --2- 172.21.15.72:0/744148743 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f3fc80a3090 0x7f3fc80a5550 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:41.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.827+0000 7f3fded3e640 1 --2- 172.21.15.72:0/744148743 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3fd8107890 0x7f3fd819f480 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:41.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.827+0000 7f3fded3e640 1 --2- 172.21.15.72:0/744148743 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3fd8104f60 0x7f3fd8198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:41.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.827+0000 7f3fded3e640 1 --2- 172.21.15.72:0/744148743 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3fd8102630 0x7f3fd8197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:41.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.827+0000 7f3fded3e640 1 -- 172.21.15.72:0/744148743 >> 172.21.15.72:0/744148743 conn(0x7f3fd80fc020 msgr2=0x7f3fd8103070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:41.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.828+0000 7f3fded3e640 1 -- 172.21.15.72:0/744148743 shutdown_connections 2024-03-22T01:36:41.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.828+0000 7f3fded3e640 1 -- 172.21.15.72:0/744148743 wait complete. 2024-03-22T01:36:41.840 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-22T01:36:41.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.949+0000 7f2da608d640 1 -- 172.21.15.72:0/2140944260 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2da0101ca0 msgr2=0x7f2da0102120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:41.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.949+0000 7f2da608d640 1 --2- 172.21.15.72:0/2140944260 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2da0101ca0 0x7f2da0102120 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2d94009860 tx=0x7f2d9402bb70 comp rx=0 tx=0).stop 2024-03-22T01:36:41.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.949+0000 7f2da608d640 1 -- 172.21.15.72:0/2140944260 shutdown_connections 2024-03-22T01:36:41.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.949+0000 7f2da608d640 1 --2- 172.21.15.72:0/2140944260 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2da0101ca0 0x7f2da0102120 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:41.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.949+0000 7f2da608d640 1 --2- 172.21.15.72:0/2140944260 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2da0102660 0x7f2da0108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:41.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.949+0000 7f2da608d640 1 --2- 172.21.15.72:0/2140944260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2da0100ab0 0x7f2da0100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:41.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.949+0000 7f2da608d640 1 -- 172.21.15.72:0/2140944260 >> 172.21.15.72:0/2140944260 conn(0x7f2da00fc020 msgr2=0x7f2da00fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:41.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.949+0000 7f2da608d640 1 -- 172.21.15.72:0/2140944260 shutdown_connections 2024-03-22T01:36:41.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.949+0000 7f2da608d640 1 -- 172.21.15.72:0/2140944260 wait complete. 2024-03-22T01:36:41.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.950+0000 7f2da608d640 1 Processor -- start 2024-03-22T01:36:41.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.950+0000 7f2da608d640 1 -- start start 2024-03-22T01:36:41.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.950+0000 7f2da608d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2da0100ab0 0x7f2da00699d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:41.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.950+0000 7f2da608d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2da0101ca0 0x7f2da019cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:41.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.950+0000 7f2da608d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2da0102660 0x7f2da019f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:41.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.950+0000 7f2da608d640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f2da010c710 con 0x7f2da0102660 2024-03-22T01:36:41.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.950+0000 7f2da608d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2da010c590 con 0x7f2da0101ca0 2024-03-22T01:36:41.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.950+0000 7f2da608d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2da010c890 con 0x7f2da0100ab0 2024-03-22T01:36:41.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.950+0000 7f2d9ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2da0102660 0x7f2da019f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:41.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.950+0000 7f2d9f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2da0100ab0 0x7f2da00699d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:41.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.950+0000 7f2d9effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2da0101ca0 0x7f2da019cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:41.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.950+0000 7f2d9ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2da0102660 0x7f2da019f220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46776/0 (socket says 172.21.15.72:46776) 2024-03-22T01:36:41.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.950+0000 7f2d9ffff640 1 -- 172.21.15.72:0/2388197290 learned_addr learned my addr 172.21.15.72:0/2388197290 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:41.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.950+0000 7f2d9f7fe640 1 -- 172.21.15.72:0/2388197290 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2da0101ca0 msgr2=0x7f2da019cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:41.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.950+0000 7f2d9f7fe640 1 --2- 172.21.15.72:0/2388197290 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2da0101ca0 0x7f2da019cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:41.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.950+0000 7f2d9f7fe640 1 -- 172.21.15.72:0/2388197290 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2da0102660 msgr2=0x7f2da019f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:41.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.950+0000 7f2d9f7fe640 1 --2- 172.21.15.72:0/2388197290 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2da0102660 0x7f2da019f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:41.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.950+0000 7f2d9f7fe640 1 -- 172.21.15.72:0/2388197290 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2da019fb30 con 0x7f2da0100ab0 2024-03-22T01:36:41.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.950+0000 7f2d9effd640 1 --2- 172.21.15.72:0/2388197290 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2da0101ca0 0x7f2da019cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:36:41.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.950+0000 7f2d9ffff640 1 --2- 172.21.15.72:0/2388197290 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2da0102660 0x7f2da019f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:36:41.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.951+0000 7f2d9f7fe640 1 --2- 172.21.15.72:0/2388197290 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2da0100ab0 0x7f2da00699d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2d9000e9c0 tx=0x7f2d9000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:41.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.951+0000 7f2d9cff9640 1 -- 172.21.15.72:0/2388197290 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d9000ccc0 con 0x7f2da0100ab0 2024-03-22T01:36:41.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.951+0000 7f2da608d640 1 -- 172.21.15.72:0/2388197290 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2da019fdc0 con 0x7f2da0100ab0 2024-03-22T01:36:41.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.951+0000 7f2d9cff9640 1 -- 172.21.15.72:0/2388197290 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d900035f0 con 0x7f2da0100ab0 2024-03-22T01:36:41.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.951+0000 7f2d9cff9640 1 -- 172.21.15.72:0/2388197290 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d9000ce60 con 0x7f2da0100ab0 2024-03-22T01:36:41.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.951+0000 7f2da608d640 1 -- 172.21.15.72:0/2388197290 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2da01a0220 con 0x7f2da0100ab0 2024-03-22T01:36:41.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.952+0000 7f2da608d640 1 -- 172.21.15.72:0/2388197290 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2da0068a10 con 0x7f2da0100ab0 2024-03-22T01:36:41.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.953+0000 7f2d9cff9640 1 -- 172.21.15.72:0/2388197290 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 22) ==== 137974+0+0 (secure 0 0 0) 0x7f2d900037a0 con 0x7f2da0100ab0 2024-03-22T01:36:41.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.953+0000 7f2d9cff9640 1 --2- 172.21.15.72:0/2388197290 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f2d880a3090 0x7f2d880a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:41.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.954+0000 7f2d9effd640 1 --2- 172.21.15.72:0/2388197290 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f2d880a3090 0x7f2d880a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:41.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.954+0000 7f2d9cff9640 1 -- 172.21.15.72:0/2388197290 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f2d900cff80 con 0x7f2da0100ab0 2024-03-22T01:36:41.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.954+0000 7f2d9effd640 1 --2- 172.21.15.72:0/2388197290 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f2d880a3090 0x7f2d880a5550 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2d9402c180 tx=0x7f2d9402e040 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:41.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:41.957+0000 7f2d9cff9640 1 -- 172.21.15.72:0/2388197290 <== mon.1 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2d9009c6c0 con 0x7f2da0100ab0 2024-03-22T01:36:42.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:41 smithi089 ceph-mon[35770]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:36:42.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:41 smithi089 ceph-mon[35770]: from='client.15510 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-22T01:36:42.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:41 smithi089 ceph-mon[35770]: from='client.25370 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-22T01:36:42.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:41 smithi089 ceph-mon[35770]: mgrmap e22: x(active, since 3s), standbys: y, z 2024-03-22T01:36:42.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:41 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:36:42.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:41 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:36:42.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:41 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:36:42.109 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:41 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:36:42.109 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:41 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/2499245168' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:36:42.109 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:41 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:36:42.109 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:41 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:36:42.109 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:41 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:36:42.109 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:41 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:36:42.109 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:41 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:36:42.109 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:41 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:36:42.109 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:41 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:36:42.109 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:41 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:36:42.109 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:41 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:36:42.109 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:41 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:36:42.109 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:41 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:36:42.110 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:41 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:36:42.110 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:41 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:36:42.110 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:41 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:36:42.110 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:41 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:36:42.110 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:41 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:36:42.110 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:41 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/744148743' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T01:36:42.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:41 smithi072 ceph-mon[32414]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:36:42.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:41 smithi072 ceph-mon[32414]: from='client.15510 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-22T01:36:42.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:41 smithi072 ceph-mon[32414]: from='client.25370 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-22T01:36:42.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:41 smithi072 ceph-mon[32414]: mgrmap e22: x(active, since 3s), standbys: y, z 2024-03-22T01:36:42.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:41 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:36:42.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:41 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:36:42.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:41 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:36:42.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:41 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:36:42.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:41 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/2499245168' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:36:42.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:41 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:36:42.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:41 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:36:42.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:41 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:36:42.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:41 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:36:42.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:41 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:36:42.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:41 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:36:42.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:41 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:36:42.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:41 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:36:42.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:41 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:36:42.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:41 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:36:42.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:41 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:36:42.124 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:41 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:36:42.124 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:41 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:36:42.124 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:41 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:36:42.124 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:41 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:36:42.124 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:41 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:36:42.124 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:41 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/744148743' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T01:36:42.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.142+0000 7f2da608d640 1 -- 172.21.15.72:0/2388197290 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f2da002d4c0 con 0x7f2da0100ab0 2024-03-22T01:36:42.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.142+0000 7f2d9cff9640 1 -- 172.21.15.72:0/2388197290 <== mon.1 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v96) ==== 104+0+11 (secure 0 0 0) 0x7f2d9009bfc0 con 0x7f2da0100ab0 2024-03-22T01:36:42.141 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-03-22T01:36:42.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.145+0000 7f2da608d640 1 -- 172.21.15.72:0/2388197290 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f2d880a3090 msgr2=0x7f2d880a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:42.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.145+0000 7f2da608d640 1 --2- 172.21.15.72:0/2388197290 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f2d880a3090 0x7f2d880a5550 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2d9402c180 tx=0x7f2d9402e040 comp rx=0 tx=0).stop 2024-03-22T01:36:42.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.145+0000 7f2da608d640 1 -- 172.21.15.72:0/2388197290 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2da0100ab0 msgr2=0x7f2da00699d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:42.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.145+0000 7f2da608d640 1 --2- 172.21.15.72:0/2388197290 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2da0100ab0 0x7f2da00699d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2d9000e9c0 tx=0x7f2d9000ee80 comp rx=0 tx=0).stop 2024-03-22T01:36:42.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.145+0000 7f2da608d640 1 -- 172.21.15.72:0/2388197290 shutdown_connections 2024-03-22T01:36:42.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.145+0000 7f2da608d640 1 --2- 172.21.15.72:0/2388197290 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f2d880a3090 0x7f2d880a5550 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:42.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.145+0000 7f2da608d640 1 --2- 172.21.15.72:0/2388197290 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2da0101ca0 0x7f2da019cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:42.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.145+0000 7f2da608d640 1 --2- 172.21.15.72:0/2388197290 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2da0102660 0x7f2da019f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:42.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.145+0000 7f2da608d640 1 --2- 172.21.15.72:0/2388197290 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2da0100ab0 0x7f2da00699d0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:42.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.145+0000 7f2da608d640 1 -- 172.21.15.72:0/2388197290 >> 172.21.15.72:0/2388197290 conn(0x7f2da00fc020 msgr2=0x7f2da0104fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:42.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.145+0000 7f2da608d640 1 -- 172.21.15.72:0/2388197290 shutdown_connections 2024-03-22T01:36:42.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.145+0000 7f2da608d640 1 -- 172.21.15.72:0/2388197290 wait complete. 2024-03-22T01:36:42.155 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-22T01:36:42.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:41 smithi098 ceph-mon[36468]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:36:42.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:41 smithi098 ceph-mon[36468]: from='client.15510 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-22T01:36:42.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:41 smithi098 ceph-mon[36468]: from='client.25370 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-22T01:36:42.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:41 smithi098 ceph-mon[36468]: mgrmap e22: x(active, since 3s), standbys: y, z 2024-03-22T01:36:42.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:41 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:36:42.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:41 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:36:42.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:41 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:36:42.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:41 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:36:42.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:41 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/2499245168' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:36:42.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:41 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:36:42.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:41 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:36:42.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:41 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:36:42.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:41 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:36:42.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:41 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:36:42.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:41 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:36:42.173 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:41 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:36:42.173 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:41 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:36:42.173 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:41 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:36:42.173 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:41 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:36:42.173 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:41 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:36:42.173 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:41 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:36:42.173 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:41 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:36:42.173 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:41 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:36:42.173 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:41 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:36:42.173 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:41 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:36:42.173 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:41 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/744148743' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T01:36:42.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.265+0000 7f2f20d8a640 1 -- 172.21.15.72:0/3240096825 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f1c0ffc30 msgr2=0x7f2f1c1000b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:42.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.265+0000 7f2f20d8a640 1 --2- 172.21.15.72:0/3240096825 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f1c0ffc30 0x7f2f1c1000b0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f2f04009900 tx=0x7f2f0402bc30 comp rx=0 tx=0).stop 2024-03-22T01:36:42.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.265+0000 7f2f20d8a640 1 -- 172.21.15.72:0/3240096825 shutdown_connections 2024-03-22T01:36:42.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.265+0000 7f2f20d8a640 1 --2- 172.21.15.72:0/3240096825 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f1c1005f0 0x7f2f1c10a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:42.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.265+0000 7f2f20d8a640 1 --2- 172.21.15.72:0/3240096825 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f1c0ffc30 0x7f2f1c1000b0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:42.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.265+0000 7f2f20d8a640 1 --2- 172.21.15.72:0/3240096825 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f1c1031c0 0x7f2f1c1035e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:42.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.265+0000 7f2f20d8a640 1 -- 172.21.15.72:0/3240096825 >> 172.21.15.72:0/3240096825 conn(0x7f2f1c0fb7b0 msgr2=0x7f2f1c0fdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:42.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.265+0000 7f2f20d8a640 1 -- 172.21.15.72:0/3240096825 shutdown_connections 2024-03-22T01:36:42.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.265+0000 7f2f20d8a640 1 -- 172.21.15.72:0/3240096825 wait complete. 2024-03-22T01:36:42.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.266+0000 7f2f20d8a640 1 Processor -- start 2024-03-22T01:36:42.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.266+0000 7f2f20d8a640 1 -- start start 2024-03-22T01:36:42.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.266+0000 7f2f20d8a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f1c0ffc30 0x7f2f1c197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:42.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.266+0000 7f2f20d8a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f1c1005f0 0x7f2f1c198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:42.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.266+0000 7f2f20d8a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f1c1031c0 0x7f2f1c19f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:42.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.266+0000 7f2f20d8a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f2f1c10d840 con 0x7f2f1c0ffc30 2024-03-22T01:36:42.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.266+0000 7f2f20d8a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2f1c10d6c0 con 0x7f2f1c1005f0 2024-03-22T01:36:42.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.266+0000 7f2f20d8a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2f1c10d9c0 con 0x7f2f1c1031c0 2024-03-22T01:36:42.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.266+0000 7f2f1a59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f1c0ffc30 0x7f2f1c197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:42.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.266+0000 7f2f1a59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f1c0ffc30 0x7f2f1c197ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46790/0 (socket says 172.21.15.72:46790) 2024-03-22T01:36:42.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.266+0000 7f2f1a59c640 1 -- 172.21.15.72:0/2856078791 learned_addr learned my addr 172.21.15.72:0/2856078791 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:42.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.266+0000 7f2f13fff640 1 --2- 172.21.15.72:0/2856078791 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f1c1005f0 0x7f2f1c198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:42.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.266+0000 7f2f1ad9d640 1 --2- 172.21.15.72:0/2856078791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f1c1031c0 0x7f2f1c19f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:42.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.266+0000 7f2f1a59c640 1 -- 172.21.15.72:0/2856078791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f1c1031c0 msgr2=0x7f2f1c19f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:42.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.266+0000 7f2f1a59c640 1 --2- 172.21.15.72:0/2856078791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f1c1031c0 0x7f2f1c19f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:42.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.266+0000 7f2f1a59c640 1 -- 172.21.15.72:0/2856078791 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f1c1005f0 msgr2=0x7f2f1c198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:42.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.266+0000 7f2f1a59c640 1 --2- 172.21.15.72:0/2856078791 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f1c1005f0 0x7f2f1c198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:42.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.266+0000 7f2f1a59c640 1 -- 172.21.15.72:0/2856078791 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f1c19fb00 con 0x7f2f1c0ffc30 2024-03-22T01:36:42.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.267+0000 7f2f1a59c640 1 --2- 172.21.15.72:0/2856078791 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f1c0ffc30 0x7f2f1c197ee0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f2f0c00e9c0 tx=0x7f2f0c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:42.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.267+0000 7f2f137fe640 1 -- 172.21.15.72:0/2856078791 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f0c00cd80 con 0x7f2f1c0ffc30 2024-03-22T01:36:42.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.267+0000 7f2f137fe640 1 -- 172.21.15.72:0/2856078791 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f0c0035f0 con 0x7f2f1c0ffc30 2024-03-22T01:36:42.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.267+0000 7f2f20d8a640 1 -- 172.21.15.72:0/2856078791 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f1c19fdf0 con 0x7f2f1c0ffc30 2024-03-22T01:36:42.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.267+0000 7f2f137fe640 1 -- 172.21.15.72:0/2856078791 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f0c010690 con 0x7f2f1c0ffc30 2024-03-22T01:36:42.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.267+0000 7f2f20d8a640 1 -- 172.21.15.72:0/2856078791 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f1c1a0200 con 0x7f2f1c0ffc30 2024-03-22T01:36:42.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.268+0000 7f2f20d8a640 1 -- 172.21.15.72:0/2856078791 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f1c109080 con 0x7f2f1c0ffc30 2024-03-22T01:36:42.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.270+0000 7f2f137fe640 1 -- 172.21.15.72:0/2856078791 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 22) ==== 137974+0+0 (secure 0 0 0) 0x7f2f0c0037a0 con 0x7f2f1c0ffc30 2024-03-22T01:36:42.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.270+0000 7f2f137fe640 1 --2- 172.21.15.72:0/2856078791 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f2f080a3090 0x7f2f080a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:42.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.270+0000 7f2f13fff640 1 --2- 172.21.15.72:0/2856078791 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f2f080a3090 0x7f2f080a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:42.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.270+0000 7f2f137fe640 1 -- 172.21.15.72:0/2856078791 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f2f0c0d0050 con 0x7f2f1c0ffc30 2024-03-22T01:36:42.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.271+0000 7f2f13fff640 1 --2- 172.21.15.72:0/2856078791 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f2f080a3090 0x7f2f080a5550 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2f0402c7b0 tx=0x7f2f04005fd0 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:42.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.274+0000 7f2f137fe640 1 -- 172.21.15.72:0/2856078791 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2f0c09c790 con 0x7f2f1c0ffc30 2024-03-22T01:36:42.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.463+0000 7f2f20d8a640 1 -- 172.21.15.72:0/2856078791 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f2f1c02d640 con 0x7f2f1c0ffc30 2024-03-22T01:36:42.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.463+0000 7f2f137fe640 1 -- 172.21.15.72:0/2856078791 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v96) ==== 100+0+11 (secure 0 0 0) 0x7f2f0c09c090 con 0x7f2f1c0ffc30 2024-03-22T01:36:42.462 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-03-22T01:36:42.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.465+0000 7f2f20d8a640 1 -- 172.21.15.72:0/2856078791 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f2f080a3090 msgr2=0x7f2f080a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:42.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.465+0000 7f2f20d8a640 1 --2- 172.21.15.72:0/2856078791 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f2f080a3090 0x7f2f080a5550 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2f0402c7b0 tx=0x7f2f04005fd0 comp rx=0 tx=0).stop 2024-03-22T01:36:42.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.465+0000 7f2f20d8a640 1 -- 172.21.15.72:0/2856078791 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f1c0ffc30 msgr2=0x7f2f1c197ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:42.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.465+0000 7f2f20d8a640 1 --2- 172.21.15.72:0/2856078791 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f1c0ffc30 0x7f2f1c197ee0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f2f0c00e9c0 tx=0x7f2f0c00ee80 comp rx=0 tx=0).stop 2024-03-22T01:36:42.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.465+0000 7f2f20d8a640 1 -- 172.21.15.72:0/2856078791 shutdown_connections 2024-03-22T01:36:42.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.465+0000 7f2f20d8a640 1 --2- 172.21.15.72:0/2856078791 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f2f080a3090 0x7f2f080a5550 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:42.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.465+0000 7f2f20d8a640 1 --2- 172.21.15.72:0/2856078791 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f1c1005f0 0x7f2f1c198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:42.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.465+0000 7f2f20d8a640 1 --2- 172.21.15.72:0/2856078791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f1c1031c0 0x7f2f1c19f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:42.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.465+0000 7f2f20d8a640 1 --2- 172.21.15.72:0/2856078791 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f1c0ffc30 0x7f2f1c197ee0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:42.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.465+0000 7f2f20d8a640 1 -- 172.21.15.72:0/2856078791 >> 172.21.15.72:0/2856078791 conn(0x7f2f1c0fb7b0 msgr2=0x7f2f1c106250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:42.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.465+0000 7f2f20d8a640 1 -- 172.21.15.72:0/2856078791 shutdown_connections 2024-03-22T01:36:42.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.465+0000 7f2f20d8a640 1 -- 172.21.15.72:0/2856078791 wait complete. 2024-03-22T01:36:42.475 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-22T01:36:42.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.584+0000 7f4b949a0640 1 -- 172.21.15.72:0/155376357 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4b90106d00 msgr2=0x7f4b9010e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:42.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.584+0000 7f4b949a0640 1 --2- 172.21.15.72:0/155376357 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4b90106d00 0x7f4b9010e8f0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f4b88009ca0 tx=0x7f4b8802c0f0 comp rx=0 tx=0).stop 2024-03-22T01:36:42.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.584+0000 7f4b949a0640 1 -- 172.21.15.72:0/155376357 shutdown_connections 2024-03-22T01:36:42.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.584+0000 7f4b949a0640 1 --2- 172.21.15.72:0/155376357 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4b90106d00 0x7f4b9010e8f0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:42.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.584+0000 7f4b949a0640 1 --2- 172.21.15.72:0/155376357 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b90100ef0 0x7f4b90106620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:42.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.584+0000 7f4b949a0640 1 --2- 172.21.15.72:0/155376357 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b90100540 0x7f4b90100920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:42.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.584+0000 7f4b949a0640 1 -- 172.21.15.72:0/155376357 >> 172.21.15.72:0/155376357 conn(0x7f4b90075b60 msgr2=0x7f4b90075f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:42.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.584+0000 7f4b949a0640 1 -- 172.21.15.72:0/155376357 shutdown_connections 2024-03-22T01:36:42.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.584+0000 7f4b949a0640 1 -- 172.21.15.72:0/155376357 wait complete. 2024-03-22T01:36:42.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.585+0000 7f4b949a0640 1 Processor -- start 2024-03-22T01:36:42.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.585+0000 7f4b949a0640 1 -- start start 2024-03-22T01:36:42.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.585+0000 7f4b949a0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b90100540 0x7f4b90198290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:42.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.585+0000 7f4b949a0640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4b90106d00 0x7f4b901987d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:42.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.585+0000 7f4b949a0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b9019cb50 0x7f4b9019d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:42.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.585+0000 7f4b949a0640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f4b90111dd0 con 0x7f4b90106d00 2024-03-22T01:36:42.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.585+0000 7f4b949a0640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4b90111c50 con 0x7f4b9019cb50 2024-03-22T01:36:42.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.585+0000 7f4b949a0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4b90111f50 con 0x7f4b90100540 2024-03-22T01:36:42.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.585+0000 7f4b8effd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4b90106d00 0x7f4b901987d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:42.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.585+0000 7f4b8effd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4b90106d00 0x7f4b901987d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46798/0 (socket says 172.21.15.72:46798) 2024-03-22T01:36:42.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.585+0000 7f4b8effd640 1 -- 172.21.15.72:0/3155955539 learned_addr learned my addr 172.21.15.72:0/3155955539 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:42.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.585+0000 7f4b8f7fe640 1 --2- 172.21.15.72:0/3155955539 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b90100540 0x7f4b90198290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:42.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.585+0000 7f4b8ffff640 1 --2- 172.21.15.72:0/3155955539 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b9019cb50 0x7f4b9019d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:42.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.586+0000 7f4b8effd640 1 -- 172.21.15.72:0/3155955539 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b90100540 msgr2=0x7f4b90198290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:42.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.586+0000 7f4b8effd640 1 --2- 172.21.15.72:0/3155955539 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b90100540 0x7f4b90198290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:42.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.586+0000 7f4b8effd640 1 -- 172.21.15.72:0/3155955539 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b9019cb50 msgr2=0x7f4b9019d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:42.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.586+0000 7f4b8effd640 1 --2- 172.21.15.72:0/3155955539 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b9019cb50 0x7f4b9019d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:42.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.586+0000 7f4b8effd640 1 -- 172.21.15.72:0/3155955539 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b9019d640 con 0x7f4b90106d00 2024-03-22T01:36:42.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.586+0000 7f4b8effd640 1 --2- 172.21.15.72:0/3155955539 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4b90106d00 0x7f4b901987d0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f4b8400e9f0 tx=0x7f4b8400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:42.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.586+0000 7f4b8cff9640 1 -- 172.21.15.72:0/3155955539 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b8400ccc0 con 0x7f4b90106d00 2024-03-22T01:36:42.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.586+0000 7f4b8cff9640 1 -- 172.21.15.72:0/3155955539 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b84005500 con 0x7f4b90106d00 2024-03-22T01:36:42.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.586+0000 7f4b949a0640 1 -- 172.21.15.72:0/3155955539 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b9019d930 con 0x7f4b90106d00 2024-03-22T01:36:42.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.586+0000 7f4b8cff9640 1 -- 172.21.15.72:0/3155955539 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b84010690 con 0x7f4b90106d00 2024-03-22T01:36:42.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.586+0000 7f4b949a0640 1 -- 172.21.15.72:0/3155955539 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b901a5120 con 0x7f4b90106d00 2024-03-22T01:36:42.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.587+0000 7f4b949a0640 1 -- 172.21.15.72:0/3155955539 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b9004f8f0 con 0x7f4b90106d00 2024-03-22T01:36:42.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.588+0000 7f4b8cff9640 1 -- 172.21.15.72:0/3155955539 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 22) ==== 137974+0+0 (secure 0 0 0) 0x7f4b840046c0 con 0x7f4b90106d00 2024-03-22T01:36:42.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.589+0000 7f4b8cff9640 1 --2- 172.21.15.72:0/3155955539 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f4b780a3110 0x7f4b780a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:42.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.589+0000 7f4b8f7fe640 1 --2- 172.21.15.72:0/3155955539 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f4b780a3110 0x7f4b780a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:42.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.589+0000 7f4b8cff9640 1 -- 172.21.15.72:0/3155955539 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f4b840cfe40 con 0x7f4b90106d00 2024-03-22T01:36:42.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.589+0000 7f4b8f7fe640 1 --2- 172.21.15.72:0/3155955539 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f4b780a3110 0x7f4b780a55d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4b80009bd0 tx=0x7f4b80005f90 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:42.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.593+0000 7f4b8cff9640 1 -- 172.21.15.72:0/3155955539 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4b8409c520 con 0x7f4b90106d00 2024-03-22T01:36:42.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.795+0000 7f4b949a0640 1 -- 172.21.15.72:0/3155955539 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f4b90100ef0 con 0x7f4b90106d00 2024-03-22T01:36:42.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.795+0000 7f4b8cff9640 1 -- 172.21.15.72:0/3155955539 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v96) ==== 103+0+11 (secure 0 0 0) 0x7f4b8409be20 con 0x7f4b90106d00 2024-03-22T01:36:42.795 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-03-22T01:36:42.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.798+0000 7f4b949a0640 1 -- 172.21.15.72:0/3155955539 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f4b780a3110 msgr2=0x7f4b780a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:42.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.798+0000 7f4b949a0640 1 --2- 172.21.15.72:0/3155955539 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f4b780a3110 0x7f4b780a55d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4b80009bd0 tx=0x7f4b80005f90 comp rx=0 tx=0).stop 2024-03-22T01:36:42.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.798+0000 7f4b949a0640 1 -- 172.21.15.72:0/3155955539 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4b90106d00 msgr2=0x7f4b901987d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:42.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.798+0000 7f4b949a0640 1 --2- 172.21.15.72:0/3155955539 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4b90106d00 0x7f4b901987d0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f4b8400e9f0 tx=0x7f4b8400eeb0 comp rx=0 tx=0).stop 2024-03-22T01:36:42.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.798+0000 7f4b949a0640 1 -- 172.21.15.72:0/3155955539 shutdown_connections 2024-03-22T01:36:42.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.798+0000 7f4b949a0640 1 --2- 172.21.15.72:0/3155955539 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f4b780a3110 0x7f4b780a55d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:42.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.798+0000 7f4b949a0640 1 --2- 172.21.15.72:0/3155955539 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b9019cb50 0x7f4b9019d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:42.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.798+0000 7f4b949a0640 1 --2- 172.21.15.72:0/3155955539 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4b90106d00 0x7f4b901987d0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:42.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.798+0000 7f4b949a0640 1 --2- 172.21.15.72:0/3155955539 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b90100540 0x7f4b90198290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:42.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.798+0000 7f4b949a0640 1 -- 172.21.15.72:0/3155955539 >> 172.21.15.72:0/3155955539 conn(0x7f4b90075b60 msgr2=0x7f4b900fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:42.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.798+0000 7f4b949a0640 1 -- 172.21.15.72:0/3155955539 shutdown_connections 2024-03-22T01:36:42.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.798+0000 7f4b949a0640 1 -- 172.21.15.72:0/3155955539 wait complete. 2024-03-22T01:36:42.811 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T01:36:42.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.922+0000 7f2d72c46640 1 -- 172.21.15.72:0/645959585 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d6c106a50 msgr2=0x7f2d6c106e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:42.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.922+0000 7f2d72c46640 1 --2- 172.21.15.72:0/645959585 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d6c106a50 0x7f2d6c106e30 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f2d60009930 tx=0x7f2d6002bc30 comp rx=0 tx=0).stop 2024-03-22T01:36:42.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.922+0000 7f2d72c46640 1 -- 172.21.15.72:0/645959585 shutdown_connections 2024-03-22T01:36:42.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.922+0000 7f2d72c46640 1 --2- 172.21.15.72:0/645959585 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d6c100a40 0x7f2d6c100ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:42.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.922+0000 7f2d72c46640 1 --2- 172.21.15.72:0/645959585 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d6c101400 0x7f2d6c10d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:42.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.922+0000 7f2d72c46640 1 --2- 172.21.15.72:0/645959585 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d6c106a50 0x7f2d6c106e30 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:42.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.922+0000 7f2d72c46640 1 -- 172.21.15.72:0/645959585 >> 172.21.15.72:0/645959585 conn(0x7f2d6c077c10 msgr2=0x7f2d6c078020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:42.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.922+0000 7f2d72c46640 1 -- 172.21.15.72:0/645959585 shutdown_connections 2024-03-22T01:36:42.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.922+0000 7f2d72c46640 1 -- 172.21.15.72:0/645959585 wait complete. 2024-03-22T01:36:42.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.923+0000 7f2d72c46640 1 Processor -- start 2024-03-22T01:36:42.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.923+0000 7f2d72c46640 1 -- start start 2024-03-22T01:36:42.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.923+0000 7f2d72c46640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d6c100a40 0x7f2d6c198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:42.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.923+0000 7f2d72c46640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d6c101400 0x7f2d6c1989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:42.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.923+0000 7f2d72c46640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d6c106a50 0x7f2d6c19ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:42.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.923+0000 7f2d72c46640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f2d6c111f00 con 0x7f2d6c100a40 2024-03-22T01:36:42.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.923+0000 7f2d72c46640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2d6c111d80 con 0x7f2d6c101400 2024-03-22T01:36:42.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.923+0000 7f2d72c46640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2d6c112080 con 0x7f2d6c106a50 2024-03-22T01:36:42.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.923+0000 7f2d709e2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d6c100a40 0x7f2d6c198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:42.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.923+0000 7f2d709e2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d6c100a40 0x7f2d6c198480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46804/0 (socket says 172.21.15.72:46804) 2024-03-22T01:36:42.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.923+0000 7f2d709e2640 1 -- 172.21.15.72:0/938625205 learned_addr learned my addr 172.21.15.72:0/938625205 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:42.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.923+0000 7f2d711e3640 1 --2- 172.21.15.72:0/938625205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d6c106a50 0x7f2d6c19ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:42.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.923+0000 7f2d6bfff640 1 --2- 172.21.15.72:0/938625205 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d6c101400 0x7f2d6c1989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:42.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.923+0000 7f2d709e2640 1 -- 172.21.15.72:0/938625205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d6c106a50 msgr2=0x7f2d6c19ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:42.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.923+0000 7f2d709e2640 1 --2- 172.21.15.72:0/938625205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d6c106a50 0x7f2d6c19ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:42.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.923+0000 7f2d709e2640 1 -- 172.21.15.72:0/938625205 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d6c101400 msgr2=0x7f2d6c1989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:42.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.923+0000 7f2d709e2640 1 --2- 172.21.15.72:0/938625205 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d6c101400 0x7f2d6c1989c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:42.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.923+0000 7f2d709e2640 1 -- 172.21.15.72:0/938625205 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d6c19d420 con 0x7f2d6c100a40 2024-03-22T01:36:42.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.924+0000 7f2d709e2640 1 --2- 172.21.15.72:0/938625205 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d6c100a40 0x7f2d6c198480 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f2d60009230 tx=0x7f2d6002cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:42.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.924+0000 7f2d69ffb640 1 -- 172.21.15.72:0/938625205 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d60041070 con 0x7f2d6c100a40 2024-03-22T01:36:42.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.924+0000 7f2d72c46640 1 -- 172.21.15.72:0/938625205 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d6c19d6b0 con 0x7f2d6c100a40 2024-03-22T01:36:42.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.924+0000 7f2d69ffb640 1 -- 172.21.15.72:0/938625205 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d60004450 con 0x7f2d6c100a40 2024-03-22T01:36:42.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.924+0000 7f2d69ffb640 1 -- 172.21.15.72:0/938625205 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d60003e50 con 0x7f2d6c100a40 2024-03-22T01:36:42.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.924+0000 7f2d72c46640 1 -- 172.21.15.72:0/938625205 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d6c1a4fa0 con 0x7f2d6c100a40 2024-03-22T01:36:42.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.925+0000 7f2d72c46640 1 -- 172.21.15.72:0/938625205 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d6c068a10 con 0x7f2d6c100a40 2024-03-22T01:36:42.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.926+0000 7f2d69ffb640 1 -- 172.21.15.72:0/938625205 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 23) ==== 138030+0+0 (secure 0 0 0) 0x7f2d60044070 con 0x7f2d6c100a40 2024-03-22T01:36:42.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.927+0000 7f2d69ffb640 1 --2- 172.21.15.72:0/938625205 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f2d5c0a32c0 0x7f2d5c0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:42.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.927+0000 7f2d6bfff640 1 --2- 172.21.15.72:0/938625205 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f2d5c0a32c0 0x7f2d5c0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:42.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.927+0000 7f2d69ffb640 1 -- 172.21.15.72:0/938625205 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f2d600f18d0 con 0x7f2d6c100a40 2024-03-22T01:36:42.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.927+0000 7f2d6bfff640 1 --2- 172.21.15.72:0/938625205 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f2d5c0a32c0 0x7f2d5c0a5780 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2d580095f0 tx=0x7f2d58005f90 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:42.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:42.930+0000 7f2d69ffb640 1 -- 172.21.15.72:0/938625205 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2d600bd960 con 0x7f2d6c100a40 2024-03-22T01:36:43.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:42 smithi089 ceph-mon[35770]: from='client.15528 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-22T01:36:43.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:42 smithi089 ceph-mon[35770]: Adjusting osd_memory_target on smithi098 to 174.1M 2024-03-22T01:36:43.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:42 smithi089 ceph-mon[35770]: Unable to set osd_memory_target on smithi098 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-22T01:36:43.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:42 smithi089 ceph-mon[35770]: Adjusting osd_memory_target on smithi072 to 176.1M 2024-03-22T01:36:43.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:42 smithi089 ceph-mon[35770]: Unable to set osd_memory_target on smithi072 to 184725811: error parsing value: Value '184725811' is below minimum 939524096 2024-03-22T01:36:43.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:42 smithi089 ceph-mon[35770]: Adjusting osd_memory_target on smithi089 to 174.1M 2024-03-22T01:36:43.109 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:42 smithi089 ceph-mon[35770]: Unable to set osd_memory_target on smithi089 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-22T01:36:43.109 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:42 smithi089 ceph-mon[35770]: Updating smithi072:/etc/ceph/ceph.conf 2024-03-22T01:36:43.109 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:42 smithi089 ceph-mon[35770]: Updating smithi089:/etc/ceph/ceph.conf 2024-03-22T01:36:43.109 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:42 smithi089 ceph-mon[35770]: Updating smithi098:/etc/ceph/ceph.conf 2024-03-22T01:36:43.109 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:42 smithi089 ceph-mon[35770]: Updating smithi072:/var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T01:36:43.109 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:42 smithi089 ceph-mon[35770]: Updating smithi089:/var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T01:36:43.109 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:42 smithi089 ceph-mon[35770]: Updating smithi098:/var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T01:36:43.109 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:42 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/2388197290' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T01:36:43.109 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:42 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/2856078791' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T01:36:43.109 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:42 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:36:43.109 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:42 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:36:43.109 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:42 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:36:43.109 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:42 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:36:43.109 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:42 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:36:43.110 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:42 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:36:43.110 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:42 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:36:43.110 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:42 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/3155955539' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T01:36:43.110 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:42 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:36:43.110 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:42 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:36:43.110 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:42 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:36:43.110 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:42 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:36:43.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:42 smithi072 ceph-mon[32414]: from='client.15528 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-22T01:36:43.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:42 smithi072 ceph-mon[32414]: Adjusting osd_memory_target on smithi098 to 174.1M 2024-03-22T01:36:43.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:42 smithi072 ceph-mon[32414]: Unable to set osd_memory_target on smithi098 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-22T01:36:43.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:42 smithi072 ceph-mon[32414]: Adjusting osd_memory_target on smithi072 to 176.1M 2024-03-22T01:36:43.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:42 smithi072 ceph-mon[32414]: Unable to set osd_memory_target on smithi072 to 184725811: error parsing value: Value '184725811' is below minimum 939524096 2024-03-22T01:36:43.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:42 smithi072 ceph-mon[32414]: Adjusting osd_memory_target on smithi089 to 174.1M 2024-03-22T01:36:43.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:42 smithi072 ceph-mon[32414]: Unable to set osd_memory_target on smithi089 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-22T01:36:43.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:42 smithi072 ceph-mon[32414]: Updating smithi072:/etc/ceph/ceph.conf 2024-03-22T01:36:43.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:42 smithi072 ceph-mon[32414]: Updating smithi089:/etc/ceph/ceph.conf 2024-03-22T01:36:43.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:42 smithi072 ceph-mon[32414]: Updating smithi098:/etc/ceph/ceph.conf 2024-03-22T01:36:43.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:42 smithi072 ceph-mon[32414]: Updating smithi072:/var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T01:36:43.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:42 smithi072 ceph-mon[32414]: Updating smithi089:/var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T01:36:43.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:42 smithi072 ceph-mon[32414]: Updating smithi098:/var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T01:36:43.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:42 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/2388197290' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T01:36:43.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:42 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/2856078791' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T01:36:43.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:42 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:36:43.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:42 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:36:43.124 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:42 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:36:43.124 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:42 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:36:43.124 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:42 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:36:43.124 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:42 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:36:43.124 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:42 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:36:43.124 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:42 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/3155955539' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T01:36:43.124 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:42 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:36:43.124 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:42 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:36:43.124 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:42 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:36:43.124 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:42 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:36:43.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.144+0000 7f2d72c46640 1 -- 172.21.15.72:0/938625205 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2d6c108fb0 con 0x7f2d6c100a40 2024-03-22T01:36:43.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.145+0000 7f2d69ffb640 1 -- 172.21.15.72:0/938625205 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11785 (secure 0 0 0) 0x7f2d600c1a00 con 0x7f2d6c100a40 2024-03-22T01:36:43.144 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:36:43.144 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6835/3077216308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3077216308},{"type":"v1","addr":"172.21.15.72:6835","nonce":3077216308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2460254143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":2460254143},{"type":"v1","addr":"172.21.15.72:6837","nonce":2460254143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6839/2026432819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6838","nonce":2026432819},{"type":"v1","addr":"172.21.15.72:6839","nonce":2026432819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6841/400094545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6840","nonce":400094545},{"type":"v1","addr":"172.21.15.72:6841","nonce":400094545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6833/1578701171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":1578701171},{"type":"v1","addr":"172.21.15.89:6833","nonce":1578701171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/2703047213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2703047213},{"type":"v1","addr":"172.21.15.98:6833","nonce":2703047213}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2589468787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2589468787},{"type":"v1","addr":"172.21.15.89:6835","nonce":2589468787}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/92657636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":92657636},{"type":"v1","addr":"172.21.15.98:6835","nonce":92657636}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1662941782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1662941782},{"type":"v1","addr":"172.21.15.89:6837","nonce":1662941782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24490,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/781860371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":781860371},{"type":"v1","addr":"172.21.15.98:6837","nonce":781860371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-22T01:34:14.815677+0000","modified":"2024-03-22T01:34:39.842959+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24524},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3192173650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3192173650},{"type":"v1","addr":"172.21.15.98:6839","nonce":3192173650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6839/149200280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":149200280},{"type":"v1","addr":"172.21.15.89:6839","nonce":149200280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24524]},"id":1}]} 2024-03-22T01:36:43.144 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 21 2024-03-22T01:36:43.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.147+0000 7f2d72c46640 1 -- 172.21.15.72:0/938625205 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f2d5c0a32c0 msgr2=0x7f2d5c0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:43.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.147+0000 7f2d72c46640 1 --2- 172.21.15.72:0/938625205 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f2d5c0a32c0 0x7f2d5c0a5780 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2d580095f0 tx=0x7f2d58005f90 comp rx=0 tx=0).stop 2024-03-22T01:36:43.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.147+0000 7f2d72c46640 1 -- 172.21.15.72:0/938625205 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d6c100a40 msgr2=0x7f2d6c198480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:43.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.147+0000 7f2d72c46640 1 --2- 172.21.15.72:0/938625205 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d6c100a40 0x7f2d6c198480 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f2d60009230 tx=0x7f2d6002cb30 comp rx=0 tx=0).stop 2024-03-22T01:36:43.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.147+0000 7f2d72c46640 1 -- 172.21.15.72:0/938625205 shutdown_connections 2024-03-22T01:36:43.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.147+0000 7f2d72c46640 1 --2- 172.21.15.72:0/938625205 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f2d5c0a32c0 0x7f2d5c0a5780 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.147+0000 7f2d72c46640 1 --2- 172.21.15.72:0/938625205 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d6c101400 0x7f2d6c1989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.147+0000 7f2d72c46640 1 --2- 172.21.15.72:0/938625205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d6c106a50 0x7f2d6c19ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.147+0000 7f2d72c46640 1 --2- 172.21.15.72:0/938625205 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d6c100a40 0x7f2d6c198480 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.147+0000 7f2d72c46640 1 -- 172.21.15.72:0/938625205 >> 172.21.15.72:0/938625205 conn(0x7f2d6c077c10 msgr2=0x7f2d6c103650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:43.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.147+0000 7f2d72c46640 1 -- 172.21.15.72:0/938625205 shutdown_connections 2024-03-22T01:36:43.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.147+0000 7f2d72c46640 1 -- 172.21.15.72:0/938625205 wait complete. 2024-03-22T01:36:43.157 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T01:36:43.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:42 smithi098 ceph-mon[36468]: from='client.15528 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-22T01:36:43.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:42 smithi098 ceph-mon[36468]: Adjusting osd_memory_target on smithi098 to 174.1M 2024-03-22T01:36:43.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:42 smithi098 ceph-mon[36468]: Unable to set osd_memory_target on smithi098 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-22T01:36:43.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:42 smithi098 ceph-mon[36468]: Adjusting osd_memory_target on smithi072 to 176.1M 2024-03-22T01:36:43.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:42 smithi098 ceph-mon[36468]: Unable to set osd_memory_target on smithi072 to 184725811: error parsing value: Value '184725811' is below minimum 939524096 2024-03-22T01:36:43.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:42 smithi098 ceph-mon[36468]: Adjusting osd_memory_target on smithi089 to 174.1M 2024-03-22T01:36:43.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:42 smithi098 ceph-mon[36468]: Unable to set osd_memory_target on smithi089 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-22T01:36:43.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:42 smithi098 ceph-mon[36468]: Updating smithi072:/etc/ceph/ceph.conf 2024-03-22T01:36:43.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:42 smithi098 ceph-mon[36468]: Updating smithi089:/etc/ceph/ceph.conf 2024-03-22T01:36:43.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:42 smithi098 ceph-mon[36468]: Updating smithi098:/etc/ceph/ceph.conf 2024-03-22T01:36:43.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:42 smithi098 ceph-mon[36468]: Updating smithi072:/var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T01:36:43.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:42 smithi098 ceph-mon[36468]: Updating smithi089:/var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T01:36:43.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:42 smithi098 ceph-mon[36468]: Updating smithi098:/var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T01:36:43.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:42 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/2388197290' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T01:36:43.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:42 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/2856078791' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T01:36:43.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:42 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:36:43.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:42 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:36:43.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:42 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:36:43.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:42 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:36:43.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:42 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:36:43.173 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:42 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:36:43.173 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:42 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:36:43.173 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:42 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/3155955539' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T01:36:43.174 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:42 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:36:43.174 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:42 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:36:43.174 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:42 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:36:43.174 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:42 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:36:43.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.260+0000 7fe892e6f640 1 -- 172.21.15.72:0/2663955848 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe88c100290 msgr2=0x7fe88c100670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:43.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.260+0000 7fe892e6f640 1 --2- 172.21.15.72:0/2663955848 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe88c100290 0x7fe88c100670 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fe880009900 tx=0x7fe88002bc30 comp rx=0 tx=0).stop 2024-03-22T01:36:43.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.260+0000 7fe892e6f640 1 -- 172.21.15.72:0/2663955848 shutdown_connections 2024-03-22T01:36:43.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.260+0000 7fe892e6f640 1 --2- 172.21.15.72:0/2663955848 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe88c106f00 0x7fe88c10ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.260+0000 7fe892e6f640 1 --2- 172.21.15.72:0/2663955848 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe88c100c40 0x7fe88c1067d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.260+0000 7fe892e6f640 1 --2- 172.21.15.72:0/2663955848 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe88c100290 0x7fe88c100670 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.260+0000 7fe892e6f640 1 -- 172.21.15.72:0/2663955848 >> 172.21.15.72:0/2663955848 conn(0x7fe88c076fa0 msgr2=0x7fe88c0fe330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:43.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.260+0000 7fe892e6f640 1 -- 172.21.15.72:0/2663955848 shutdown_connections 2024-03-22T01:36:43.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.260+0000 7fe892e6f640 1 -- 172.21.15.72:0/2663955848 wait complete. 2024-03-22T01:36:43.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.260+0000 7fe892e6f640 1 Processor -- start 2024-03-22T01:36:43.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.260+0000 7fe892e6f640 1 -- start start 2024-03-22T01:36:43.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.261+0000 7fe892e6f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe88c100290 0x7fe88c198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:43.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.261+0000 7fe892e6f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe88c106f00 0x7fe88c1989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:43.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.261+0000 7fe892e6f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe88c19cd20 0x7fe88c19d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:43.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.261+0000 7fe892e6f640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fe88c111f60 con 0x7fe88c106f00 2024-03-22T01:36:43.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.261+0000 7fe892e6f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe88c111de0 con 0x7fe88c19cd20 2024-03-22T01:36:43.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.261+0000 7fe892e6f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe88c1120e0 con 0x7fe88c100290 2024-03-22T01:36:43.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.261+0000 7fe88bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe88c106f00 0x7fe88c1989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:43.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.261+0000 7fe88bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe88c106f00 0x7fe88c1989a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46816/0 (socket says 172.21.15.72:46816) 2024-03-22T01:36:43.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.261+0000 7fe89140c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe88c19cd20 0x7fe88c19d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:43.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.261+0000 7fe88bfff640 1 -- 172.21.15.72:0/1758706804 learned_addr learned my addr 172.21.15.72:0/1758706804 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:43.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.261+0000 7fe890c0b640 1 --2- 172.21.15.72:0/1758706804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe88c100290 0x7fe88c198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:43.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.261+0000 7fe89140c640 1 -- 172.21.15.72:0/1758706804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe88c100290 msgr2=0x7fe88c198460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:43.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.261+0000 7fe89140c640 1 --2- 172.21.15.72:0/1758706804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe88c100290 0x7fe88c198460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.261+0000 7fe89140c640 1 -- 172.21.15.72:0/1758706804 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe88c106f00 msgr2=0x7fe88c1989a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:43.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.261+0000 7fe89140c640 1 --2- 172.21.15.72:0/1758706804 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe88c106f00 0x7fe88c1989a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.261+0000 7fe89140c640 1 -- 172.21.15.72:0/1758706804 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe88c19d810 con 0x7fe88c19cd20 2024-03-22T01:36:43.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.261+0000 7fe88bfff640 1 --2- 172.21.15.72:0/1758706804 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe88c106f00 0x7fe88c1989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:36:43.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.261+0000 7fe89140c640 1 --2- 172.21.15.72:0/1758706804 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe88c19cd20 0x7fe88c19d1d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe884009fd0 tx=0x7fe88400ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:43.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.261+0000 7fe889ffb640 1 -- 172.21.15.72:0/1758706804 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe88400ce20 con 0x7fe88c19cd20 2024-03-22T01:36:43.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.261+0000 7fe892e6f640 1 -- 172.21.15.72:0/1758706804 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe88c19db00 con 0x7fe88c19cd20 2024-03-22T01:36:43.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.261+0000 7fe889ffb640 1 -- 172.21.15.72:0/1758706804 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe884005020 con 0x7fe88c19cd20 2024-03-22T01:36:43.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.261+0000 7fe889ffb640 1 -- 172.21.15.72:0/1758706804 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe884010720 con 0x7fe88c19cd20 2024-03-22T01:36:43.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.262+0000 7fe892e6f640 1 -- 172.21.15.72:0/1758706804 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe88c1a5270 con 0x7fe88c19cd20 2024-03-22T01:36:43.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.263+0000 7fe892e6f640 1 -- 172.21.15.72:0/1758706804 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe88c068e60 con 0x7fe88c19cd20 2024-03-22T01:36:43.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.263+0000 7fe889ffb640 1 -- 172.21.15.72:0/1758706804 <== mon.2 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 23) ==== 138030+0+0 (secure 0 0 0) 0x7fe884005540 con 0x7fe88c19cd20 2024-03-22T01:36:43.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.264+0000 7fe889ffb640 1 --2- 172.21.15.72:0/1758706804 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fe87c0a3120 0x7fe87c0a55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:43.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.264+0000 7fe890c0b640 1 --2- 172.21.15.72:0/1758706804 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fe87c0a3120 0x7fe87c0a55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:43.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.264+0000 7fe889ffb640 1 -- 172.21.15.72:0/1758706804 <== mon.2 v2:172.21.15.89:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fe8840d0420 con 0x7fe88c19cd20 2024-03-22T01:36:43.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.264+0000 7fe890c0b640 1 --2- 172.21.15.72:0/1758706804 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fe87c0a3120 0x7fe87c0a55e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe880006010 tx=0x7fe880005dd0 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:43.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.269+0000 7fe889ffb640 1 -- 172.21.15.72:0/1758706804 <== mon.2 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe88409c990 con 0x7fe88c19cd20 2024-03-22T01:36:43.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.446+0000 7fe892e6f640 1 -- 172.21.15.72:0/1758706804 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe88c10a130 con 0x7fe88c19cd20 2024-03-22T01:36:43.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.447+0000 7fe889ffb640 1 -- 172.21.15.72:0/1758706804 <== mon.2 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) ==== 74+0+25134 (secure 0 0 0) 0x7fe88409c290 con 0x7fe88c19cd20 2024-03-22T01:36:43.447 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:36:43.447 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":96,"fsid":"022a99bc-e7eb-11ee-95cd-87774f69a715","created":"2024-03-22T01:26:23.893672+0000","modified":"2024-03-22T01:36:38.183837+0000","last_up_change":"2024-03-22T01:33:25.223684+0000","last_in_change":"2024-03-22T01:33:05.010331+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T01:29:37.438503+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-22T01:34:12.610422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-22T01:34:13.761686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-22T01:34:17.507039+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"16c15bb0-db92-4191-9063-38b4890e81e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6803","nonce":661906791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6805","nonce":661906791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6809","nonce":661906791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6807","nonce":661906791}]},"public_addr":"172.21.15.72:6803/661906791","cluster_addr":"172.21.15.72:6805/661906791","heartbeat_back_addr":"172.21.15.72:6809/661906791","heartbeat_front_addr":"172.21.15.72:6807/661906791","state":["exists","up"]},{"osd":1,"uuid":"981290ba-97dc-496b-aa22-d51704184b10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6801","nonce":2870590550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6803","nonce":2870590550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6807","nonce":2870590550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6805","nonce":2870590550}]},"public_addr":"172.21.15.89:6801/2870590550","cluster_addr":"172.21.15.89:6803/2870590550","heartbeat_back_addr":"172.21.15.89:6807/2870590550","heartbeat_front_addr":"172.21.15.89:6805/2870590550","state":["exists","up"]},{"osd":2,"uuid":"f07ed658-f545-46a1-8edd-8b1d5d49885c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6801","nonce":3865778118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6803","nonce":3865778118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6807","nonce":3865778118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6805","nonce":3865778118}]},"public_addr":"172.21.15.98:6801/3865778118","cluster_addr":"172.21.15.98:6803/3865778118","heartbeat_back_addr":"172.21.15.98:6807/3865778118","heartbeat_front_addr":"172.21.15.98:6805/3865778118","state":["exists","up"]},{"osd":3,"uuid":"ccc4bc40-2c48-46d2-8548-0c05b44f9f1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6811","nonce":1865951395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6813","nonce":1865951395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6817","nonce":1865951395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6815","nonce":1865951395}]},"public_addr":"172.21.15.72:6811/1865951395","cluster_addr":"172.21.15.72:6813/1865951395","heartbeat_back_addr":"172.21.15.72:6817/1865951395","heartbeat_front_addr":"172.21.15.72:6815/1865951395","state":["exists","up"]},{"osd":4,"uuid":"1812439f-810a-4ed2-9395-c00c9a085a13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6809","nonce":2683760795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6811","nonce":2683760795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6815","nonce":2683760795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6813","nonce":2683760795}]},"public_addr":"172.21.15.89:6809/2683760795","cluster_addr":"172.21.15.89:6811/2683760795","heartbeat_back_addr":"172.21.15.89:6815/2683760795","heartbeat_front_addr":"172.21.15.89:6813/2683760795","state":["exists","up"]},{"osd":5,"uuid":"e78954d8-f4e3-4d58-8536-d2d59c843386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6809","nonce":2635838572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6811","nonce":2635838572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6815","nonce":2635838572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6813","nonce":2635838572}]},"public_addr":"172.21.15.98:6809/2635838572","cluster_addr":"172.21.15.98:6811/2635838572","heartbeat_back_addr":"172.21.15.98:6815/2635838572","heartbeat_front_addr":"172.21.15.98:6813/2635838572","state":["exists","up"]},{"osd":6,"uuid":"dc6c8be2-b64c-4959-8015-7b0ea6d25599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6819","nonce":3158982661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6821","nonce":3158982661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6825","nonce":3158982661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6823","nonce":3158982661}]},"public_addr":"172.21.15.72:6819/3158982661","cluster_addr":"172.21.15.72:6821/3158982661","heartbeat_back_addr":"172.21.15.72:6825/3158982661","heartbeat_front_addr":"172.21.15.72:6823/3158982661","state":["exists","up"]},{"osd":7,"uuid":"7a73469f-cf3a-4306-ae93-ea46e21ebbbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6817","nonce":3075913234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6819","nonce":3075913234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6823","nonce":3075913234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6821","nonce":3075913234}]},"public_addr":"172.21.15.89:6817/3075913234","cluster_addr":"172.21.15.89:6819/3075913234","heartbeat_back_addr":"172.21.15.89:6823/3075913234","heartbeat_front_addr":"172.21.15.89:6821/3075913234","state":["exists","up"]},{"osd":8,"uuid":"c4b13bee-ea4d-4344-95fb-2622ecbb8196","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6817","nonce":1813255577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6819","nonce":1813255577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6823","nonce":1813255577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6821","nonce":1813255577}]},"public_addr":"172.21.15.98:6817/1813255577","cluster_addr":"172.21.15.98:6819/1813255577","heartbeat_back_addr":"172.21.15.98:6823/1813255577","heartbeat_front_addr":"172.21.15.98:6821/1813255577","state":["exists","up"]},{"osd":9,"uuid":"eefb0a44-4a36-4e55-9d04-0501bf718db6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6827","nonce":3377749206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6829","nonce":3377749206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6832","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6833","nonce":3377749206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6831","nonce":3377749206}]},"public_addr":"172.21.15.72:6827/3377749206","cluster_addr":"172.21.15.72:6829/3377749206","heartbeat_back_addr":"172.21.15.72:6833/3377749206","heartbeat_front_addr":"172.21.15.72:6831/3377749206","state":["exists","up"]},{"osd":10,"uuid":"8741d25f-40dd-40d7-b040-5cc5e411eb98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6825","nonce":885058786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6827","nonce":885058786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6831","nonce":885058786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6829","nonce":885058786}]},"public_addr":"172.21.15.89:6825/885058786","cluster_addr":"172.21.15.89:6827/885058786","heartbeat_back_addr":"172.21.15.89:6831/885058786","heartbeat_front_addr":"172.21.15.89:6829/885058786","state":["exists","up"]},{"osd":11,"uuid":"8d02e957-eda4-4184-b5c6-5eb3b8b38060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6825","nonce":1517095838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6827","nonce":1517095838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6831","nonce":1517095838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6829","nonce":1517095838}]},"public_addr":"172.21.15.98:6825/1517095838","cluster_addr":"172.21.15.98:6827/1517095838","heartbeat_back_addr":"172.21.15.98:6831/1517095838","heartbeat_front_addr":"172.21.15.98:6829/1517095838","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:28:47.307652+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:11.037135+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:34.619188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:59.350738+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:30:23.787337+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:30:48.994154+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:31:13.690350+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:31:39.628116+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:04.723912+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:30.689741+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:57.229508+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:33:23.154537+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":7}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:0/2032770149":"2024-03-23T01:36:38.183806+0000","172.21.15.72:6801/401403200":"2024-03-23T01:36:38.183806+0000","172.21.15.72:0/1161486989":"2024-03-23T01:36:38.183806+0000","172.21.15.72:6800/401403200":"2024-03-23T01:36:38.183806+0000","172.21.15.72:0/121087196":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/2451857761":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/1228477995":"2024-03-23T01:27:15.339809+0000","172.21.15.72:6801/1513232913":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/4223447038":"2024-03-23T01:36:38.183806+0000","172.21.15.72:0/3558958264":"2024-03-23T01:26:50.074871+0000","172.21.15.72:0/1510187198":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6800/3518012916":"2024-03-23T01:26:50.074871+0000","172.21.15.72:0/608566205":"2024-03-23T01:36:38.183806+0000","172.21.15.72:6801/3518012916":"2024-03-23T01:26:50.074871+0000","172.21.15.72:0/1372801041":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6800/1513232913":"2024-03-23T01:27:15.339809+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T01:36:43.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.449+0000 7fe892e6f640 1 -- 172.21.15.72:0/1758706804 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fe87c0a3120 msgr2=0x7fe87c0a55e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:43.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.449+0000 7fe892e6f640 1 --2- 172.21.15.72:0/1758706804 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fe87c0a3120 0x7fe87c0a55e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe880006010 tx=0x7fe880005dd0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.449+0000 7fe892e6f640 1 -- 172.21.15.72:0/1758706804 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe88c19cd20 msgr2=0x7fe88c19d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:43.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.449+0000 7fe892e6f640 1 --2- 172.21.15.72:0/1758706804 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe88c19cd20 0x7fe88c19d1d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe884009fd0 tx=0x7fe88400ef10 comp rx=0 tx=0).stop 2024-03-22T01:36:43.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.449+0000 7fe892e6f640 1 -- 172.21.15.72:0/1758706804 shutdown_connections 2024-03-22T01:36:43.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.449+0000 7fe892e6f640 1 --2- 172.21.15.72:0/1758706804 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fe87c0a3120 0x7fe87c0a55e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.449+0000 7fe892e6f640 1 --2- 172.21.15.72:0/1758706804 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe88c19cd20 0x7fe88c19d1d0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.449+0000 7fe892e6f640 1 --2- 172.21.15.72:0/1758706804 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe88c106f00 0x7fe88c1989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.449+0000 7fe892e6f640 1 --2- 172.21.15.72:0/1758706804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe88c100290 0x7fe88c198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.449+0000 7fe892e6f640 1 -- 172.21.15.72:0/1758706804 >> 172.21.15.72:0/1758706804 conn(0x7fe88c076fa0 msgr2=0x7fe88c105220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:43.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.450+0000 7fe892e6f640 1 -- 172.21.15.72:0/1758706804 shutdown_connections 2024-03-22T01:36:43.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.450+0000 7fe892e6f640 1 -- 172.21.15.72:0/1758706804 wait complete. 2024-03-22T01:36:43.460 DEBUG:tasks.fwd_scrub:Yielding 2024-03-22T01:36:43.460 INFO:teuthology.run_tasks:Running task exec... 2024-03-22T01:36:43.470 INFO:teuthology.task.exec:Executing custom commands... 2024-03-22T01:36:43.470 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi072.front.sepia.ceph.com 2024-03-22T01:36:43.471 DEBUG:teuthology.orchestra.run.smithi072:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-03-22T01:36:43.471 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-03-22T01:36:43.471 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-22T01:36:43.471 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T01:36:43.471 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-22T01:36:43.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.580+0000 7fc4acc29640 1 -- 172.21.15.72:0/2058049919 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc4a8069430 msgr2=0x7fc4a8069810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:43.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.580+0000 7fc4acc29640 1 --2- 172.21.15.72:0/2058049919 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc4a8069430 0x7fc4a8069810 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fc4a0009900 tx=0x7fc4a002bc30 comp rx=0 tx=0).stop 2024-03-22T01:36:43.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.580+0000 7fc4acc29640 1 -- 172.21.15.72:0/2058049919 shutdown_connections 2024-03-22T01:36:43.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.580+0000 7fc4acc29640 1 --2- 172.21.15.72:0/2058049919 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4a810e530 0x7fc4a8110920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.580+0000 7fc4acc29640 1 --2- 172.21.15.72:0/2058049919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4a8069d50 0x7fc4a810dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.580+0000 7fc4acc29640 1 --2- 172.21.15.72:0/2058049919 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc4a8069430 0x7fc4a8069810 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.580+0000 7fc4acc29640 1 -- 172.21.15.72:0/2058049919 >> 172.21.15.72:0/2058049919 conn(0x7fc4a8077b80 msgr2=0x7fc4a8077f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:43.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.580+0000 7fc4acc29640 1 -- 172.21.15.72:0/2058049919 shutdown_connections 2024-03-22T01:36:43.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.580+0000 7fc4acc29640 1 -- 172.21.15.72:0/2058049919 wait complete. 2024-03-22T01:36:43.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.580+0000 7f529f802640 1 -- 172.21.15.72:0/1273202329 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f52981036d0 msgr2=0x7f529810bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:43.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.580+0000 7f529f802640 1 --2- 172.21.15.72:0/1273202329 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f52981036d0 0x7f529810bbb0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f5288009860 tx=0x7f528802bc20 comp rx=0 tx=0).stop 2024-03-22T01:36:43.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.580+0000 7f529f802640 1 -- 172.21.15.72:0/1273202329 shutdown_connections 2024-03-22T01:36:43.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.580+0000 7f529f802640 1 --2- 172.21.15.72:0/1273202329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f529810c180 0x7f529810e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.580+0000 7f529f802640 1 --2- 172.21.15.72:0/1273202329 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f52981036d0 0x7f529810bbb0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.580+0000 7f529f802640 1 --2- 172.21.15.72:0/1273202329 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5298102db0 0x7f5298103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.580+0000 7f529f802640 1 -- 172.21.15.72:0/1273202329 >> 172.21.15.72:0/1273202329 conn(0x7f5298077c40 msgr2=0x7f5298078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:43.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.580+0000 7f529f802640 1 -- 172.21.15.72:0/1273202329 shutdown_connections 2024-03-22T01:36:43.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.580+0000 7f529f802640 1 -- 172.21.15.72:0/1273202329 wait complete. 2024-03-22T01:36:43.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.580+0000 7fc4acc29640 1 Processor -- start 2024-03-22T01:36:43.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.581+0000 7fc4acc29640 1 -- start start 2024-03-22T01:36:43.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.581+0000 7f529f802640 1 Processor -- start 2024-03-22T01:36:43.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.581+0000 7f529f802640 1 -- start start 2024-03-22T01:36:43.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.581+0000 7fc4acc29640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc4a8069430 0x7fc4a819a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:43.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.581+0000 7fc4acc29640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4a8069d50 0x7fc4a819ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:43.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.581+0000 7fc4acc29640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4a810e530 0x7fc4a819efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:43.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.581+0000 7fc4acc29640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fc4a81140e0 con 0x7fc4a8069430 2024-03-22T01:36:43.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.581+0000 7fc4acc29640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc4a8113f60 con 0x7fc4a8069d50 2024-03-22T01:36:43.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.581+0000 7fc4acc29640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc4a8114260 con 0x7fc4a810e530 2024-03-22T01:36:43.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.581+0000 7fc4a659c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc4a8069430 0x7fc4a819a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:43.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.581+0000 7f529f802640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5298102db0 0x7f5298198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:43.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.581+0000 7fc4a659c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc4a8069430 0x7fc4a819a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46832/0 (socket says 172.21.15.72:46832) 2024-03-22T01:36:43.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.581+0000 7fc4a659c640 1 -- 172.21.15.72:0/2884558694 learned_addr learned my addr 172.21.15.72:0/2884558694 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:43.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.581+0000 7f529f802640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52981036d0 0x7f5298198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:43.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.581+0000 7f529f802640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f529810c180 0x7f529819ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:43.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.581+0000 7f529f802640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f5298111f10 con 0x7f529810c180 2024-03-22T01:36:43.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.581+0000 7f529f802640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5298111d90 con 0x7f52981036d0 2024-03-22T01:36:43.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.581+0000 7f529f802640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5298112090 con 0x7f5298102db0 2024-03-22T01:36:43.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.581+0000 7fc4a5d9b640 1 --2- 172.21.15.72:0/2884558694 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4a8069d50 0x7fc4a819ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:43.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.581+0000 7fc4a6d9d640 1 --2- 172.21.15.72:0/2884558694 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4a810e530 0x7fc4a819efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:43.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.581+0000 7f529cd9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52981036d0 0x7f5298198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:43.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.581+0000 7f529d59e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5298102db0 0x7f5298198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:43.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.581+0000 7f529dd9f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f529810c180 0x7f529819ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:43.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.581+0000 7f529cd9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52981036d0 0x7f5298198970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.72:52844/0 (socket says 172.21.15.72:52844) 2024-03-22T01:36:43.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.581+0000 7f529cd9d640 1 -- 172.21.15.72:0/3724326006 learned_addr learned my addr 172.21.15.72:0/3724326006 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:43.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.581+0000 7f529d59e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5298102db0 0x7f5298198430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.72:55698/0 (socket says 172.21.15.72:55698) 2024-03-22T01:36:43.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.581+0000 7fc4a659c640 1 -- 172.21.15.72:0/2884558694 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4a810e530 msgr2=0x7fc4a819efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:43.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.581+0000 7fc4a659c640 1 --2- 172.21.15.72:0/2884558694 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4a810e530 0x7fc4a819efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.581+0000 7fc4a659c640 1 -- 172.21.15.72:0/2884558694 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4a8069d50 msgr2=0x7fc4a819ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:43.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.581+0000 7fc4a659c640 1 --2- 172.21.15.72:0/2884558694 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4a8069d50 0x7fc4a819ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.581+0000 7fc4a659c640 1 -- 172.21.15.72:0/2884558694 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4a819f6d0 con 0x7fc4a8069430 2024-03-22T01:36:43.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.581+0000 7f529cd9d640 1 -- 172.21.15.72:0/3724326006 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5298102db0 msgr2=0x7f5298198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:43.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.581+0000 7f529cd9d640 1 --2- 172.21.15.72:0/3724326006 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5298102db0 0x7f5298198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.581+0000 7f529cd9d640 1 -- 172.21.15.72:0/3724326006 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f529810c180 msgr2=0x7f529819ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:43.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.581+0000 7f529cd9d640 1 --2- 172.21.15.72:0/3724326006 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f529810c180 0x7f529819ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.581+0000 7f529cd9d640 1 -- 172.21.15.72:0/3724326006 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f529819d420 con 0x7f52981036d0 2024-03-22T01:36:43.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.581+0000 7f529dd9f640 1 --2- 172.21.15.72:0/3724326006 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f529810c180 0x7f529819ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:36:43.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.581+0000 7fc4a659c640 1 --2- 172.21.15.72:0/2884558694 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc4a8069430 0x7fc4a819a6e0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fc4a0006010 tx=0x7fc4a002cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:43.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.582+0000 7fc4977fe640 1 -- 172.21.15.72:0/2884558694 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4a0041070 con 0x7fc4a8069430 2024-03-22T01:36:43.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.582+0000 7f529cd9d640 1 --2- 172.21.15.72:0/3724326006 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52981036d0 0x7f5298198970 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5288037040 tx=0x7f52880344c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:43.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.582+0000 7fc4acc29640 1 -- 172.21.15.72:0/2884558694 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4a819f960 con 0x7fc4a8069430 2024-03-22T01:36:43.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.582+0000 7fc4977fe640 1 -- 172.21.15.72:0/2884558694 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4a0004450 con 0x7fc4a8069430 2024-03-22T01:36:43.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.582+0000 7fc4977fe640 1 -- 172.21.15.72:0/2884558694 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4a0003e00 con 0x7fc4a8069430 2024-03-22T01:36:43.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.582+0000 7fc4acc29640 1 -- 172.21.15.72:0/2884558694 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4a81a7250 con 0x7fc4a8069430 2024-03-22T01:36:43.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.582+0000 7f528e7fc640 1 -- 172.21.15.72:0/3724326006 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5288041020 con 0x7f52981036d0 2024-03-22T01:36:43.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.582+0000 7f528e7fc640 1 -- 172.21.15.72:0/3724326006 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f528802ce50 con 0x7f52981036d0 2024-03-22T01:36:43.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.582+0000 7f529f802640 1 -- 172.21.15.72:0/3724326006 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f529819d6b0 con 0x7f52981036d0 2024-03-22T01:36:43.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.582+0000 7f528e7fc640 1 -- 172.21.15.72:0/3724326006 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5288039040 con 0x7f52981036d0 2024-03-22T01:36:43.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.582+0000 7f529f802640 1 -- 172.21.15.72:0/3724326006 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5298100930 con 0x7f52981036d0 2024-03-22T01:36:43.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.583+0000 7fc4acc29640 1 -- 172.21.15.72:0/2884558694 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4a8050140 con 0x7fc4a8069430 2024-03-22T01:36:43.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.584+0000 7fc4977fe640 1 -- 172.21.15.72:0/2884558694 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 23) ==== 138030+0+0 (secure 0 0 0) 0x7fc4a0044070 con 0x7fc4a8069430 2024-03-22T01:36:43.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.585+0000 7f528e7fc640 1 -- 172.21.15.72:0/3724326006 <== mon.2 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 23) ==== 138030+0+0 (secure 0 0 0) 0x7f5288044020 con 0x7f52981036d0 2024-03-22T01:36:43.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.585+0000 7fc4977fe640 1 --2- 172.21.15.72:0/2884558694 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fc4900a3110 0x7fc4900a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:43.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.585+0000 7fc4a5d9b640 1 --2- 172.21.15.72:0/2884558694 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fc4900a3110 0x7fc4900a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:43.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.585+0000 7fc4977fe640 1 -- 172.21.15.72:0/2884558694 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fc4a00f1730 con 0x7fc4a8069430 2024-03-22T01:36:43.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.585+0000 7f52a23ed740 1 Processor -- start 2024-03-22T01:36:43.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.585+0000 7f52a23ed740 1 -- start start 2024-03-22T01:36:43.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.585+0000 7f52a23ed740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b563e7dba0 0x55b563e7df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:43.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.585+0000 7f52a23ed740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b563e7e4c0 0x55b563e86b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:43.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.585+0000 7f52a23ed740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b563e870a0 0x55b563e89490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:43.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.585+0000 7f52a23ed740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55b563de1570 con 0x55b563e7dba0 2024-03-22T01:36:43.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.585+0000 7f52a23ed740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55b563d89e10 con 0x55b563e870a0 2024-03-22T01:36:43.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.585+0000 7f52a23ed740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b563ddd520 con 0x55b563e7e4c0 2024-03-22T01:36:43.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.586+0000 7fc4a5d9b640 1 --2- 172.21.15.72:0/2884558694 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fc4900a3110 0x7fc4900a55d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc4a81020f0 tx=0x7fc49c00a590 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:43.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.585+0000 7f528e7fc640 1 --2- 172.21.15.72:0/3724326006 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f52840a3120 0x7f52840a55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:43.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.586+0000 7f529dd9f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b563e7dba0 0x55b563e7df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:43.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.586+0000 7f529dd9f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b563e7dba0 0x55b563e7df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46860/0 (socket says 172.21.15.72:46860) 2024-03-22T01:36:43.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.586+0000 7f529dd9f640 1 -- 172.21.15.72:0/1158930982 learned_addr learned my addr 172.21.15.72:0/1158930982 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:43.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.586+0000 7f529cd9d640 1 --2- 172.21.15.72:0/1158930982 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b563e870a0 0x55b563e89490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:43.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.586+0000 7f529dd9f640 1 --2- 172.21.15.72:0/3724326006 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f52840a3120 0x7f52840a55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:43.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.586+0000 7f529d59e640 1 --2- 172.21.15.72:0/1158930982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b563e7e4c0 0x55b563e86b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:43.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.586+0000 7f528e7fc640 1 -- 172.21.15.72:0/3724326006 <== mon.2 v2:172.21.15.89:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f52880f11d0 con 0x7f52981036d0 2024-03-22T01:36:43.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.586+0000 7f529dd9f640 1 -- 172.21.15.72:0/1158930982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b563e7e4c0 msgr2=0x55b563e86b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:36:43.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.586+0000 7f529dd9f640 1 --2- 172.21.15.72:0/1158930982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b563e7e4c0 0x55b563e86b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.586+0000 7f529dd9f640 1 -- 172.21.15.72:0/1158930982 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b563e870a0 msgr2=0x55b563e89490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:36:43.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.586+0000 7f529dd9f640 1 --2- 172.21.15.72:0/1158930982 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b563e870a0 0x55b563e89490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.586+0000 7f529dd9f640 1 -- 172.21.15.72:0/1158930982 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b563bbd7e0 con 0x55b563e7dba0 2024-03-22T01:36:43.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.586+0000 7f529cd9d640 1 --2- 172.21.15.72:0/1158930982 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b563e870a0 0x55b563e89490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:36:43.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.586+0000 7f529dd9f640 1 --2- 172.21.15.72:0/1158930982 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b563e7dba0 0x55b563e7df80 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f529001e8e0 tx=0x7f529001eda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b376b2849dac2d13 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:43.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.586+0000 7f529dd9f640 1 --2- 172.21.15.72:0/3724326006 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f52840a3120 0x7f52840a55e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5290009930 tx=0x7f5290022000 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:43.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.590+0000 7f52837fe640 1 -- 172.21.15.72:0/1158930982 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f529002c070 con 0x55b563e7dba0 2024-03-22T01:36:43.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.590+0000 7f52837fe640 1 -- 172.21.15.72:0/1158930982 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5290010830 con 0x55b563e7dba0 2024-03-22T01:36:43.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.590+0000 7f52837fe640 1 -- 172.21.15.72:0/1158930982 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5290004290 con 0x55b563e7dba0 2024-03-22T01:36:43.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.590+0000 7fc4977fe640 1 -- 172.21.15.72:0/2884558694 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc4a00bd7c0 con 0x7fc4a8069430 2024-03-22T01:36:43.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.590+0000 7f52a23ed740 1 -- 172.21.15.72:0/1158930982 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b563e7dba0 msgr2=0x55b563e7df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:43.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.590+0000 7f52a23ed740 1 --2- 172.21.15.72:0/1158930982 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b563e7dba0 0x55b563e7df80 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f529001e8e0 tx=0x7f529001eda0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.590+0000 7f52a23ed740 1 -- 172.21.15.72:0/1158930982 shutdown_connections 2024-03-22T01:36:43.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.590+0000 7f52a23ed740 1 --2- 172.21.15.72:0/1158930982 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b563e870a0 0x55b563e89490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.590+0000 7f52a23ed740 1 --2- 172.21.15.72:0/1158930982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b563e7e4c0 0x55b563e86b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.590+0000 7f52a23ed740 1 --2- 172.21.15.72:0/1158930982 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b563e7dba0 0x55b563e7df80 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.590+0000 7f52a23ed740 1 -- 172.21.15.72:0/1158930982 >> 172.21.15.72:0/1158930982 conn(0x55b563e782d0 msgr2=0x55b563e786b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:43.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.591+0000 7f52a23ed740 1 -- 172.21.15.72:0/1158930982 shutdown_connections 2024-03-22T01:36:43.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.591+0000 7f52a23ed740 1 -- 172.21.15.72:0/1158930982 wait complete. 2024-03-22T01:36:43.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.591+0000 7f52a23ed740 1 -- start start 2024-03-22T01:36:43.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.591+0000 7f52a23ed740 1 Processor -- start 2024-03-22T01:36:43.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.591+0000 7f52a23ed740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b563e7dba0 0x55b563e86b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:43.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.591+0000 7f52a23ed740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b563e7e4c0 0x55b563e83ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:43.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.591+0000 7f52a23ed740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b563e870a0 0x55b563e845b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:43.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.591+0000 7f52a23ed740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55b563e89e30 con 0x55b563e870a0 2024-03-22T01:36:43.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.591+0000 7f52a23ed740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55b563d9d110 con 0x55b563e7dba0 2024-03-22T01:36:43.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.591+0000 7f52a23ed740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b563e89fb0 con 0x55b563e7e4c0 2024-03-22T01:36:43.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.591+0000 7f52a23ed740 10 client.0 _ll_register_callbacks cb 0x55b563dd9be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:36:43.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.591+0000 7f529dd9f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b563e870a0 0x55b563e845b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:43.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.591+0000 7f529d59e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b563e7dba0 0x55b563e86b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:43.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.591+0000 7f529dd9f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b563e870a0 0x55b563e845b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46870/0 (socket says 172.21.15.72:46870) 2024-03-22T01:36:43.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.591+0000 7f529dd9f640 1 -- 172.21.15.72:0/596763465 learned_addr learned my addr 172.21.15.72:0/596763465 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:43.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.592+0000 7f529cd9d640 1 --2- 172.21.15.72:0/596763465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b563e7e4c0 0x55b563e83ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:43.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.592+0000 7f529d59e640 1 -- 172.21.15.72:0/596763465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b563e7e4c0 msgr2=0x55b563e83ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:36:43.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.592+0000 7f529d59e640 1 --2- 172.21.15.72:0/596763465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b563e7e4c0 0x55b563e83ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.592+0000 7f529d59e640 1 -- 172.21.15.72:0/596763465 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b563e870a0 msgr2=0x55b563e845b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:36:43.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.592+0000 7f529d59e640 1 --2- 172.21.15.72:0/596763465 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b563e870a0 0x55b563e845b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.592+0000 7f529d59e640 1 -- 172.21.15.72:0/596763465 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b563c1df70 con 0x55b563e7dba0 2024-03-22T01:36:43.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.592+0000 7f529dd9f640 1 --2- 172.21.15.72:0/596763465 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b563e870a0 0x55b563e845b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:36:43.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.592+0000 7f529d59e640 1 --2- 172.21.15.72:0/596763465 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b563e7dba0 0x55b563e86b30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5294009530 tx=0x7f5294011a90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=103dc4f44da45e7d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:43.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.592+0000 7f5280ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-03-22T01:36:43.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.592+0000 7f5280ff9640 1 -- 172.21.15.72:0/596763465 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5294004020 con 0x55b563e7dba0 2024-03-22T01:36:43.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.592+0000 7f5280ff9640 1 -- 172.21.15.72:0/596763465 <== mon.2 v2:172.21.15.89:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f52940041c0 con 0x55b563e7dba0 2024-03-22T01:36:43.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.592+0000 7f52a23ed740 1 -- 172.21.15.72:0/596763465 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5290010830 con 0x55b563e7dba0 2024-03-22T01:36:43.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.592+0000 7f5280ff9640 1 -- 172.21.15.72:0/596763465 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f5294024050 con 0x55b563e7dba0 2024-03-22T01:36:43.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.592+0000 7f52a23ed740 10 client.25436 fetch_fsmap learned FSMap version 21 2024-03-22T01:36:43.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.592+0000 7f52a23ed740 1 -- 172.21.15.72:0/596763465 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55b563ab5170 con 0x55b563e7dba0 2024-03-22T01:36:43.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.592+0000 7f5280ff9640 1 -- 172.21.15.72:0/596763465 <== mon.2 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5290010830 con 0x55b563e7dba0 2024-03-22T01:36:43.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.593+0000 7f5280ff9640 1 -- 172.21.15.72:0/596763465 <== mon.2 v2:172.21.15.89:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f5294019070 con 0x55b563e7dba0 2024-03-22T01:36:43.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.593+0000 7f52a23ed740 10 client.25436 fetch_fsmap finished waiting for FSMap version 21 2024-03-22T01:36:43.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.593+0000 7f52a23ed740 10 client.25436 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T01:36:43.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.593+0000 7f52a23ed740 20 client.25436 populate_metadata read hostname 'smithi072' 2024-03-22T01:36:43.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.593+0000 7f52a23ed740 1 --2- 172.21.15.72:0/596763465 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55b563e97210 0x55b563e99670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:43.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.593+0000 7f52a23ed740 4 client.25436 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T01:36:43.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.593+0000 7f52a23ed740 1 -- 172.21.15.72:0/596763465 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b563deeab0 con 0x55b563e97210 2024-03-22T01:36:43.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.593+0000 7f529cd9d640 1 --2- 172.21.15.72:0/596763465 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55b563e97210 0x55b563e99670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:43.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.593+0000 7f529cd9d640 1 --2- 172.21.15.72:0/596763465 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55b563e97210 0x55b563e99670 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9837997a73d5d8b server_cookie=3fe25f654e4148ab in_seq=0 out_seq=0 2024-03-22T01:36:43.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.593+0000 7f5280ff9640 10 client.25436 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:36:43.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.594+0000 7f5280ff9640 1 -- 172.21.15.72:0/596763465 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b563deeab0 con 0x55b563e97210 2024-03-22T01:36:43.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.594+0000 7f5280ff9640 10 client.25436 handle_command_reply: tid=0 2024-03-22T01:36:43.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.594+0000 7f52a23ed740 1 client.25436 shutdown 2024-03-22T01:36:43.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.594+0000 7f52a23ed740 1 -- 172.21.15.72:0/596763465 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b563e7dba0 msgr2=0x55b563e86b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:43.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.594+0000 7f52a23ed740 1 --2- 172.21.15.72:0/596763465 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b563e7dba0 0x55b563e86b30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5294009530 tx=0x7f5294011a90 comp rx=0 tx=0).stop 2024-03-22T01:36:43.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.594+0000 7f52a23ed740 1 -- 172.21.15.72:0/596763465 shutdown_connections 2024-03-22T01:36:43.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.594+0000 7f52a23ed740 1 --2- 172.21.15.72:0/596763465 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55b563e97210 0x55b563e99670 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.594+0000 7f52a23ed740 1 --2- 172.21.15.72:0/596763465 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b563e870a0 0x55b563e845b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.594+0000 7f5280ff9640 0 client.25436 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:36:43.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.594+0000 7f52a23ed740 1 --2- 172.21.15.72:0/596763465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b563e7e4c0 0x55b563e83ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.594+0000 7f52a23ed740 1 --2- 172.21.15.72:0/596763465 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b563e7dba0 0x55b563e86b30 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.594+0000 7f52a23ed740 1 -- 172.21.15.72:0/596763465 >> 172.21.15.72:0/596763465 conn(0x55b563e782d0 msgr2=0x55b563e7a020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:43.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.594+0000 7f52a23ed740 1 -- 172.21.15.72:0/596763465 shutdown_connections 2024-03-22T01:36:43.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.594+0000 7f52a23ed740 1 -- 172.21.15.72:0/596763465 wait complete. 2024-03-22T01:36:43.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.594+0000 7f52a23ed740 20 client.25436 trim_cache size 0 max 16384 2024-03-22T01:36:43.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.612+0000 7f52a23ed740 1 Processor -- start 2024-03-22T01:36:43.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.612+0000 7f52a23ed740 1 -- start start 2024-03-22T01:36:43.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.612+0000 7f52a23ed740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b563e782d0 0x55b563e9ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:43.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.612+0000 7f52a23ed740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b563e7dba0 0x55b563e8f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:43.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.612+0000 7f52a23ed740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b563e7e4c0 0x55b563e865f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:43.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.612+0000 7f52a23ed740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f52981aae40 con 0x55b563e7e4c0 2024-03-22T01:36:43.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.612+0000 7f52a23ed740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f52981ad6c0 con 0x55b563e782d0 2024-03-22T01:36:43.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.612+0000 7f52a23ed740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f52981ab4b0 con 0x55b563e7dba0 2024-03-22T01:36:43.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.612+0000 7f529dd9f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b563e7e4c0 0x55b563e865f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:43.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.612+0000 7f529d59e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b563e782d0 0x55b563e9ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:43.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.612+0000 7f529d59e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b563e782d0 0x55b563e9ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.72:52872/0 (socket says 172.21.15.72:52872) 2024-03-22T01:36:43.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.612+0000 7f529dd9f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b563e7e4c0 0x55b563e865f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46880/0 (socket says 172.21.15.72:46880) 2024-03-22T01:36:43.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.612+0000 7f529d59e640 1 -- 172.21.15.72:0/3724464197 learned_addr learned my addr 172.21.15.72:0/3724464197 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:43.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.612+0000 7f529cd9d640 1 --2- 172.21.15.72:0/3724464197 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b563e7dba0 0x55b563e8f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:43.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.612+0000 7f529d59e640 1 -- 172.21.15.72:0/3724464197 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b563e7dba0 msgr2=0x55b563e8f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:36:43.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.612+0000 7f529d59e640 1 --2- 172.21.15.72:0/3724464197 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b563e7dba0 0x55b563e8f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.612+0000 7f529d59e640 1 -- 172.21.15.72:0/3724464197 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b563e7e4c0 msgr2=0x55b563e865f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:36:43.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.612+0000 7f529d59e640 1 --2- 172.21.15.72:0/3724464197 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b563e7e4c0 0x55b563e865f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.612+0000 7f529d59e640 1 -- 172.21.15.72:0/3724464197 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b563e89ed0 con 0x55b563e782d0 2024-03-22T01:36:43.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.612+0000 7f529cd9d640 1 --2- 172.21.15.72:0/3724464197 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b563e7dba0 0x55b563e8f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:36:43.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.612+0000 7f529d59e640 1 --2- 172.21.15.72:0/3724464197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b563e782d0 0x55b563e9ef90 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5294009cb0 tx=0x7f5294005e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5165982f043f64ad server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:43.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.613+0000 7f5282ffd640 1 -- 172.21.15.72:0/3724464197 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f529402c020 con 0x55b563e782d0 2024-03-22T01:36:43.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.613+0000 7f5282ffd640 1 -- 172.21.15.72:0/3724464197 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5294009d50 con 0x55b563e782d0 2024-03-22T01:36:43.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.613+0000 7f5282ffd640 1 -- 172.21.15.72:0/3724464197 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5294003f70 con 0x55b563e782d0 2024-03-22T01:36:43.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.613+0000 7f52a23ed740 1 -- 172.21.15.72:0/3724464197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b563e782d0 msgr2=0x55b563e9ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:43.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.613+0000 7f52a23ed740 1 --2- 172.21.15.72:0/3724464197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b563e782d0 0x55b563e9ef90 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5294009cb0 tx=0x7f5294005e70 comp rx=0 tx=0).stop 2024-03-22T01:36:43.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.613+0000 7f52a23ed740 1 -- 172.21.15.72:0/3724464197 shutdown_connections 2024-03-22T01:36:43.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.613+0000 7f52a23ed740 1 --2- 172.21.15.72:0/3724464197 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b563e7e4c0 0x55b563e865f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.613+0000 7f52a23ed740 1 --2- 172.21.15.72:0/3724464197 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b563e7dba0 0x55b563e8f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.613+0000 7f52a23ed740 1 --2- 172.21.15.72:0/3724464197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b563e782d0 0x55b563e9ef90 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.613+0000 7f52a23ed740 1 -- 172.21.15.72:0/3724464197 >> 172.21.15.72:0/3724464197 conn(0x55b563e82b00 msgr2=0x55b563e795d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:43.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.613+0000 7f52a23ed740 1 -- 172.21.15.72:0/3724464197 shutdown_connections 2024-03-22T01:36:43.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.614+0000 7f52a23ed740 1 -- 172.21.15.72:0/3724464197 wait complete. 2024-03-22T01:36:43.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.614+0000 7f52a23ed740 1 -- start start 2024-03-22T01:36:43.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.614+0000 7f52a23ed740 1 Processor -- start 2024-03-22T01:36:43.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.614+0000 7f52a23ed740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b563e782d0 0x55b563e8f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:43.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.614+0000 7f52a23ed740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b563e7dba0 0x55b563e8c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:43.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.614+0000 7f52a23ed740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b563e7e4c0 0x55b563e8c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:43.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.614+0000 7f52a23ed740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f52981abb80 con 0x55b563e782d0 2024-03-22T01:36:43.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.614+0000 7f52a23ed740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f52981acff0 con 0x55b563e7dba0 2024-03-22T01:36:43.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.614+0000 7f52a23ed740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f52981ac250 con 0x55b563e7e4c0 2024-03-22T01:36:43.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.614+0000 7f52a23ed740 10 client.0 _ll_register_callbacks cb 0x7f5294009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:36:43.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.614+0000 7f529d59e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b563e782d0 0x55b563e8f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:43.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.614+0000 7f529d59e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b563e782d0 0x55b563e8f200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46896/0 (socket says 172.21.15.72:46896) 2024-03-22T01:36:43.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.614+0000 7f529d59e640 1 -- 172.21.15.72:0/2767705815 learned_addr learned my addr 172.21.15.72:0/2767705815 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:43.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.614+0000 7f529cd9d640 1 --2- 172.21.15.72:0/2767705815 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b563e7dba0 0x55b563e8c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:43.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.615+0000 7f529dd9f640 1 --2- 172.21.15.72:0/2767705815 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b563e7e4c0 0x55b563e8c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:43.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.615+0000 7f529d59e640 1 -- 172.21.15.72:0/2767705815 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b563e7e4c0 msgr2=0x55b563e8c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:36:43.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.615+0000 7f529d59e640 1 --2- 172.21.15.72:0/2767705815 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b563e7e4c0 0x55b563e8c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.615+0000 7f529d59e640 1 -- 172.21.15.72:0/2767705815 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b563e7dba0 msgr2=0x55b563e8c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:36:43.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.615+0000 7f529d59e640 1 --2- 172.21.15.72:0/2767705815 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b563e7dba0 0x55b563e8c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.615+0000 7f529d59e640 1 -- 172.21.15.72:0/2767705815 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b563e8cc50 con 0x55b563e782d0 2024-03-22T01:36:43.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.615+0000 7f529d59e640 1 --2- 172.21.15.72:0/2767705815 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b563e782d0 0x55b563e8f200 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f52940058c0 tx=0x7f5294011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64684b1f64984332 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:43.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.615+0000 7f5280ff9640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:36:43.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.615+0000 7f5280ff9640 1 -- 172.21.15.72:0/2767705815 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f529400fa00 con 0x55b563e782d0 2024-03-22T01:36:43.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.615+0000 7f5280ff9640 1 -- 172.21.15.72:0/2767705815 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f529400fba0 con 0x55b563e782d0 2024-03-22T01:36:43.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.615+0000 7f52a23ed740 1 -- 172.21.15.72:0/2767705815 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5294009d50 con 0x55b563e782d0 2024-03-22T01:36:43.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.615+0000 7f5280ff9640 1 -- 172.21.15.72:0/2767705815 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f5294043050 con 0x55b563e782d0 2024-03-22T01:36:43.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.615+0000 7f52a23ed740 10 client.15594 fetch_fsmap learned FSMap version 21 2024-03-22T01:36:43.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.615+0000 7f52a23ed740 1 -- 172.21.15.72:0/2767705815 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55b563efe210 con 0x55b563e782d0 2024-03-22T01:36:43.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.615+0000 7f5280ff9640 1 -- 172.21.15.72:0/2767705815 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5294009d50 con 0x55b563e782d0 2024-03-22T01:36:43.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.616+0000 7f5280ff9640 1 -- 172.21.15.72:0/2767705815 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f5294019040 con 0x55b563e782d0 2024-03-22T01:36:43.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.616+0000 7f52a23ed740 10 client.15594 fetch_fsmap finished waiting for FSMap version 21 2024-03-22T01:36:43.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.616+0000 7f52a23ed740 10 client.15594 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T01:36:43.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.616+0000 7f52a23ed740 20 client.15594 populate_metadata read hostname 'smithi072' 2024-03-22T01:36:43.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.616+0000 7f52a23ed740 1 --2- 172.21.15.72:0/2767705815 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55b563e870a0 0x55b563f007d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:43.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.616+0000 7f52a23ed740 4 client.15594 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-22T01:36:43.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.616+0000 7f52a23ed740 1 -- 172.21.15.72:0/2767705815 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55b563de8e00 con 0x55b563e870a0 2024-03-22T01:36:43.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.616+0000 7f529cd9d640 1 --2- 172.21.15.72:0/2767705815 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55b563e870a0 0x55b563f007d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:43.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.616+0000 7f529cd9d640 1 --2- 172.21.15.72:0/2767705815 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55b563e870a0 0x55b563f007d0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=99951d2f330c2750 server_cookie=9186af53fca90b19 in_seq=0 out_seq=0 2024-03-22T01:36:43.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.616+0000 7f5280ff9640 10 client.15594 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:36:43.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.617+0000 7f5280ff9640 1 -- 172.21.15.72:0/2767705815 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55b563de8e00 con 0x55b563e870a0 2024-03-22T01:36:43.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.617+0000 7f5280ff9640 10 client.15594 handle_command_reply: tid=0 2024-03-22T01:36:43.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.617+0000 7f52a23ed740 1 client.15594 shutdown 2024-03-22T01:36:43.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.617+0000 7f52a23ed740 1 -- 172.21.15.72:0/2767705815 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b563e782d0 msgr2=0x55b563e8f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:43.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.617+0000 7f52a23ed740 1 --2- 172.21.15.72:0/2767705815 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b563e782d0 0x55b563e8f200 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f52940058c0 tx=0x7f5294011a90 comp rx=0 tx=0).stop 2024-03-22T01:36:43.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.617+0000 7f52a23ed740 1 -- 172.21.15.72:0/2767705815 shutdown_connections 2024-03-22T01:36:43.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.617+0000 7f52a23ed740 1 --2- 172.21.15.72:0/2767705815 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55b563e870a0 0x55b563f007d0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.617+0000 7f5280ff9640 0 client.15594 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:36:43.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.617+0000 7f52a23ed740 1 --2- 172.21.15.72:0/2767705815 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b563e7dba0 0x55b563e8c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.617+0000 7f52a23ed740 1 --2- 172.21.15.72:0/2767705815 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b563e7e4c0 0x55b563e8c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.617+0000 7f52a23ed740 1 --2- 172.21.15.72:0/2767705815 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b563e782d0 0x55b563e8f200 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.617+0000 7f52a23ed740 1 -- 172.21.15.72:0/2767705815 >> 172.21.15.72:0/2767705815 conn(0x55b563e82b00 msgr2=0x55b563e79e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:43.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.617+0000 7f52a23ed740 1 -- 172.21.15.72:0/2767705815 shutdown_connections 2024-03-22T01:36:43.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.617+0000 7f52a23ed740 1 -- 172.21.15.72:0/2767705815 wait complete. 2024-03-22T01:36:43.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.617+0000 7f52a23ed740 20 client.15594 trim_cache size 0 max 16384 2024-03-22T01:36:43.643 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-22T01:36:43.643 INFO:teuthology.orchestra.run.smithi072.stdout: "return_code": 0, 2024-03-22T01:36:43.643 INFO:teuthology.orchestra.run.smithi072.stdout: "scrub_tag": "b2cb8289-218c-43e5-a000-ef96955b64f0", 2024-03-22T01:36:43.643 INFO:teuthology.orchestra.run.smithi072.stdout: "mode": "asynchronous" 2024-03-22T01:36:43.644 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-22T01:36:43.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.618+0000 7f529f802640 1 -- 172.21.15.72:0/3724326006 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f52840a3120 msgr2=0x7f52840a55e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:43.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.618+0000 7f529f802640 1 --2- 172.21.15.72:0/3724326006 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f52840a3120 0x7f52840a55e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5290009930 tx=0x7f5290022000 comp rx=0 tx=0).stop 2024-03-22T01:36:43.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.618+0000 7f529f802640 1 -- 172.21.15.72:0/3724326006 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52981036d0 msgr2=0x7f5298198970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:43.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.618+0000 7f529f802640 1 --2- 172.21.15.72:0/3724326006 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52981036d0 0x7f5298198970 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5288037040 tx=0x7f52880344c0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.618+0000 7f529f802640 1 -- 172.21.15.72:0/3724326006 shutdown_connections 2024-03-22T01:36:43.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.618+0000 7f529f802640 1 --2- 172.21.15.72:0/3724326006 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f52840a3120 0x7f52840a55e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.618+0000 7f529f802640 1 --2- 172.21.15.72:0/3724326006 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52981036d0 0x7f5298198970 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.618+0000 7f529f802640 1 --2- 172.21.15.72:0/3724326006 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f529810c180 0x7f529819ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.618+0000 7f529f802640 1 --2- 172.21.15.72:0/3724326006 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5298102db0 0x7f5298198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.618+0000 7f529f802640 1 -- 172.21.15.72:0/3724326006 >> 172.21.15.72:0/3724326006 conn(0x7f5298077c40 msgr2=0x7f52980fdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:43.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.618+0000 7f529f802640 1 -- 172.21.15.72:0/3724326006 shutdown_connections 2024-03-22T01:36:43.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.618+0000 7f529f802640 1 -- 172.21.15.72:0/3724326006 wait complete. 2024-03-22T01:36:43.648 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T01:36:43.648 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T01:36:43.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.754+0000 7f9ff7039640 1 -- 172.21.15.72:0/2352982581 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ff0100a30 msgr2=0x7f9ff0100eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:43.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.754+0000 7f9ff7039640 1 --2- 172.21.15.72:0/2352982581 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ff0100a30 0x7f9ff0100eb0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc009930 tx=0x7f9fdc02bc30 comp rx=0 tx=0).stop 2024-03-22T01:36:43.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.755+0000 7f9ff7039640 1 -- 172.21.15.72:0/2352982581 shutdown_connections 2024-03-22T01:36:43.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.755+0000 7f9ff7039640 1 --2- 172.21.15.72:0/2352982581 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ff01013f0 0x7f9ff010d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.755+0000 7f9ff7039640 1 --2- 172.21.15.72:0/2352982581 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ff0100a30 0x7f9ff0100eb0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.755+0000 7f9ff7039640 1 --2- 172.21.15.72:0/2352982581 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ff0106a40 0x7f9ff0106e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.755+0000 7f9ff7039640 1 -- 172.21.15.72:0/2352982581 >> 172.21.15.72:0/2352982581 conn(0x7f9ff0077c40 msgr2=0x7f9ff0078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:43.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.755+0000 7f9ff7039640 1 -- 172.21.15.72:0/2352982581 shutdown_connections 2024-03-22T01:36:43.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.755+0000 7f9ff7039640 1 -- 172.21.15.72:0/2352982581 wait complete. 2024-03-22T01:36:43.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.755+0000 7f9ff7039640 1 Processor -- start 2024-03-22T01:36:43.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.755+0000 7f9ff7039640 1 -- start start 2024-03-22T01:36:43.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.755+0000 7f9ff7039640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ff0100a30 0x7f9ff0199210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:43.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.755+0000 7f9ff7039640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ff01013f0 0x7f9ff0199750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:43.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.755+0000 7f9ff7039640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ff0106a40 0x7f9ff01a4290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:43.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.755+0000 7f9ff7039640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f9ff0110e50 con 0x7f9ff0100a30 2024-03-22T01:36:43.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.755+0000 7f9ff7039640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9ff0110cd0 con 0x7f9ff0106a40 2024-03-22T01:36:43.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.755+0000 7f9ff7039640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9ff0110fd0 con 0x7f9ff01013f0 2024-03-22T01:36:43.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.756+0000 7f9ff4dd5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ff0100a30 0x7f9ff0199210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:43.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.756+0000 7f9ff4dd5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ff0100a30 0x7f9ff0199210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46908/0 (socket says 172.21.15.72:46908) 2024-03-22T01:36:43.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.756+0000 7f9ff4dd5640 1 -- 172.21.15.72:0/1069202221 learned_addr learned my addr 172.21.15.72:0/1069202221 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:43.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.756+0000 7f9ff55d6640 1 --2- 172.21.15.72:0/1069202221 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ff0106a40 0x7f9ff01a4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:43.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.756+0000 7f9feffff640 1 --2- 172.21.15.72:0/1069202221 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ff01013f0 0x7f9ff0199750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:43.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.756+0000 7f9ff4dd5640 1 -- 172.21.15.72:0/1069202221 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ff01013f0 msgr2=0x7f9ff0199750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:43.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.756+0000 7f9ff4dd5640 1 --2- 172.21.15.72:0/1069202221 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ff01013f0 0x7f9ff0199750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.756+0000 7f9ff4dd5640 1 -- 172.21.15.72:0/1069202221 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ff0106a40 msgr2=0x7f9ff01a4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:43.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.756+0000 7f9ff4dd5640 1 --2- 172.21.15.72:0/1069202221 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ff0106a40 0x7f9ff01a4290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.756+0000 7f9ff4dd5640 1 -- 172.21.15.72:0/1069202221 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ff01a47d0 con 0x7f9ff0100a30 2024-03-22T01:36:43.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.756+0000 7f9feffff640 1 --2- 172.21.15.72:0/1069202221 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ff01013f0 0x7f9ff0199750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T01:36:43.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.756+0000 7f9ff4dd5640 1 --2- 172.21.15.72:0/1069202221 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ff0100a30 0x7f9ff0199210 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f9fe400e9f0 tx=0x7f9fe400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:43.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.756+0000 7f9fedffb640 1 -- 172.21.15.72:0/1069202221 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe400cd80 con 0x7f9ff0100a30 2024-03-22T01:36:43.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.756+0000 7f9ff7039640 1 -- 172.21.15.72:0/1069202221 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ff01a49a0 con 0x7f9ff0100a30 2024-03-22T01:36:43.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.756+0000 7f9fedffb640 1 -- 172.21.15.72:0/1069202221 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fe4010910 con 0x7f9ff0100a30 2024-03-22T01:36:43.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.756+0000 7f9fedffb640 1 -- 172.21.15.72:0/1069202221 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe40188f0 con 0x7f9ff0100a30 2024-03-22T01:36:43.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.756+0000 7f9ff7039640 1 -- 172.21.15.72:0/1069202221 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ff01a4f40 con 0x7f9ff0100a30 2024-03-22T01:36:43.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.759+0000 7f9fedffb640 1 -- 172.21.15.72:0/1069202221 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 23) ==== 138030+0+0 (secure 0 0 0) 0x7f9fe4010ac0 con 0x7f9ff0100a30 2024-03-22T01:36:43.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.759+0000 7f9ff9c29740 1 Processor -- start 2024-03-22T01:36:43.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.759+0000 7f9ff9c29740 1 -- start start 2024-03-22T01:36:43.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.759+0000 7f9fedffb640 1 --2- 172.21.15.72:0/1069202221 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f9fe00a3120 0x7f9fe00a55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:43.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.759+0000 7f9ff9c29740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56005de33410 0x56005de337f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:43.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.759+0000 7f9ff9c29740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56005de33d30 0x56005de3c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:43.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.759+0000 7f9ff9c29740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56005de3c910 0x56005de3ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:43.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.759+0000 7f9ff9c29740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x56005dd978c0 con 0x56005de3c910 2024-03-22T01:36:43.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.759+0000 7f9ff9c29740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56005dd3fc20 con 0x56005de33410 2024-03-22T01:36:43.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.759+0000 7f9ff9c29740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56005dd93870 con 0x56005de33d30 2024-03-22T01:36:43.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.759+0000 7f9ff55d6640 1 --2- 172.21.15.72:0/1069202221 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f9fe00a3120 0x7f9fe00a55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:43.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.759+0000 7f9ff4dd5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56005de33410 0x56005de337f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:43.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.759+0000 7f9ff4dd5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56005de33410 0x56005de337f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.72:52902/0 (socket says 172.21.15.72:52902) 2024-03-22T01:36:43.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.759+0000 7f9ff4dd5640 1 -- 172.21.15.72:0/2195559334 learned_addr learned my addr 172.21.15.72:0/2195559334 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:43.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.759+0000 7f9feffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56005de33d30 0x56005de3c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:43.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.759+0000 7f9ff55d6640 1 --2- 172.21.15.72:0/2195559334 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56005de3c910 0x56005de3ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:43.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.760+0000 7f9fedffb640 1 -- 172.21.15.72:0/1069202221 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f9fe40d0130 con 0x7f9ff0100a30 2024-03-22T01:36:43.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.760+0000 7f9ff4dd5640 1 -- 172.21.15.72:0/2195559334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56005de33d30 msgr2=0x56005de3c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:36:43.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.760+0000 7f9ff4dd5640 1 --2- 172.21.15.72:0/2195559334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56005de33d30 0x56005de3c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.760+0000 7f9ff4dd5640 1 -- 172.21.15.72:0/2195559334 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56005de3c910 msgr2=0x56005de3ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:36:43.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.760+0000 7f9ff4dd5640 1 --2- 172.21.15.72:0/2195559334 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56005de3c910 0x56005de3ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.760+0000 7f9ff4dd5640 1 -- 172.21.15.72:0/2195559334 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56005dd64150 con 0x56005de33410 2024-03-22T01:36:43.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.760+0000 7f9ff55d6640 1 --2- 172.21.15.72:0/1069202221 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f9fe00a3120 0x7f9fe00a55e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9fe8007310 tx=0x7f9fe8007de0 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:43.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.760+0000 7f9ff4dd5640 1 --2- 172.21.15.72:0/2195559334 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56005de33410 0x56005de337f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f9fe40cdad0 tx=0x7f9fe401f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=645c8098588ca320 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:43.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.760+0000 7f9fdaffd640 1 -- 172.21.15.72:0/2195559334 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe4065180 con 0x56005de33410 2024-03-22T01:36:43.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.760+0000 7f9fdaffd640 1 -- 172.21.15.72:0/2195559334 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fe409c700 con 0x56005de33410 2024-03-22T01:36:43.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.760+0000 7f9fdaffd640 1 -- 172.21.15.72:0/2195559334 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe406ea10 con 0x56005de33410 2024-03-22T01:36:43.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.761+0000 7f9ff9c29740 1 -- 172.21.15.72:0/2195559334 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56005de33410 msgr2=0x56005de337f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:43.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.761+0000 7f9ff9c29740 1 --2- 172.21.15.72:0/2195559334 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56005de33410 0x56005de337f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f9fe40cdad0 tx=0x7f9fe401f000 comp rx=0 tx=0).stop 2024-03-22T01:36:43.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.761+0000 7f9ff9c29740 1 -- 172.21.15.72:0/2195559334 shutdown_connections 2024-03-22T01:36:43.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.761+0000 7f9ff9c29740 1 --2- 172.21.15.72:0/2195559334 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56005de3c910 0x56005de3ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.761+0000 7f9ff9c29740 1 --2- 172.21.15.72:0/2195559334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56005de33d30 0x56005de3c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.761+0000 7f9ff9c29740 1 --2- 172.21.15.72:0/2195559334 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56005de33410 0x56005de337f0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.761+0000 7f9ff9c29740 1 -- 172.21.15.72:0/2195559334 >> 172.21.15.72:0/2195559334 conn(0x56005de2db40 msgr2=0x56005de2df20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:43.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.761+0000 7f9ff9c29740 1 -- 172.21.15.72:0/2195559334 shutdown_connections 2024-03-22T01:36:43.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.761+0000 7f9ff9c29740 1 -- 172.21.15.72:0/2195559334 wait complete. 2024-03-22T01:36:43.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.761+0000 7f9ff9c29740 1 -- start start 2024-03-22T01:36:43.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.761+0000 7f9ff9c29740 1 Processor -- start 2024-03-22T01:36:43.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.762+0000 7f9ff9c29740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56005de33410 0x56005de39a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:43.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.762+0000 7f9ff9c29740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56005de33d30 0x56005de39fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:43.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.762+0000 7f9ff9c29740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56005de3c910 0x56005de3a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:43.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.762+0000 7f9ff9c29740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x56005de43b10 con 0x56005de33410 2024-03-22T01:36:43.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.762+0000 7f9ff9c29740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56005dd53130 con 0x56005de3c910 2024-03-22T01:36:43.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.762+0000 7f9ff9c29740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56005de43c90 con 0x56005de33d30 2024-03-22T01:36:43.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.762+0000 7f9ff9c29740 10 client.0 _ll_register_callbacks cb 0x56005dd52b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:36:43.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.762+0000 7f9ff4dd5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56005de33410 0x56005de39a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:43.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.762+0000 7f9ff4dd5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56005de33410 0x56005de39a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46920/0 (socket says 172.21.15.72:46920) 2024-03-22T01:36:43.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.762+0000 7f9ff55d6640 1 --2- 172.21.15.72:0/3599494658 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56005de3c910 0x56005de3a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:43.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.762+0000 7f9ff4dd5640 1 -- 172.21.15.72:0/3599494658 learned_addr learned my addr 172.21.15.72:0/3599494658 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:43.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.762+0000 7f9feffff640 1 --2- 172.21.15.72:0/3599494658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56005de33d30 0x56005de39fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:43.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.762+0000 7f9ff55d6640 1 -- 172.21.15.72:0/3599494658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56005de33d30 msgr2=0x56005de39fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:36:43.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.762+0000 7f9ff55d6640 1 --2- 172.21.15.72:0/3599494658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56005de33d30 0x56005de39fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.762+0000 7f9ff55d6640 1 -- 172.21.15.72:0/3599494658 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56005de33410 msgr2=0x56005de39a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:36:43.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.762+0000 7f9ff55d6640 1 --2- 172.21.15.72:0/3599494658 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56005de33410 0x56005de39a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.762+0000 7f9ff55d6640 1 -- 172.21.15.72:0/3599494658 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56005db757d0 con 0x56005de3c910 2024-03-22T01:36:43.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.762+0000 7f9ff4dd5640 1 --2- 172.21.15.72:0/3599494658 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56005de33410 0x56005de39a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:36:43.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.762+0000 7f9ff55d6640 1 --2- 172.21.15.72:0/3599494658 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56005de3c910 0x56005de3a4e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9fe8005c40 tx=0x7f9fe8007470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78ff54cca2352e9e server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:43.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.762+0000 7f9fcffff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-03-22T01:36:43.781 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.762+0000 7f9fcffff640 1 -- 172.21.15.72:0/3599494658 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe8026070 con 0x56005de3c910 2024-03-22T01:36:43.781 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.762+0000 7f9ff9c29740 1 -- 172.21.15.72:0/3599494658 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9fe409c700 con 0x56005de3c910 2024-03-22T01:36:43.781 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.762+0000 7f9fcffff640 1 -- 172.21.15.72:0/3599494658 <== mon.2 v2:172.21.15.89:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f9fe8004b20 con 0x56005de3c910 2024-03-22T01:36:43.781 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.763+0000 7f9fcffff640 1 -- 172.21.15.72:0/3599494658 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f9fe8031050 con 0x56005de3c910 2024-03-22T01:36:43.781 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.763+0000 7f9ff9c29740 10 client.25454 fetch_fsmap learned FSMap version 21 2024-03-22T01:36:43.781 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.763+0000 7f9ff9c29740 1 -- 172.21.15.72:0/3599494658 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x56005daab3f0 con 0x56005de3c910 2024-03-22T01:36:43.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.763+0000 7f9fcffff640 1 -- 172.21.15.72:0/3599494658 <== mon.2 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fe409c700 con 0x56005de3c910 2024-03-22T01:36:43.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.763+0000 7f9fcffff640 1 -- 172.21.15.72:0/3599494658 <== mon.2 v2:172.21.15.89:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f9fe8005090 con 0x56005de3c910 2024-03-22T01:36:43.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.763+0000 7f9ff9c29740 10 client.25454 fetch_fsmap finished waiting for FSMap version 21 2024-03-22T01:36:43.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.763+0000 7f9ff9c29740 10 client.25454 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T01:36:43.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.763+0000 7f9ff9c29740 20 client.25454 populate_metadata read hostname 'smithi072' 2024-03-22T01:36:43.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.763+0000 7f9ff9c29740 1 --2- 172.21.15.72:0/3599494658 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x56005de51170 0x56005de535c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:43.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.763+0000 7f9ff9c29740 4 client.25454 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T01:36:43.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.763+0000 7f9ff9c29740 1 -- 172.21.15.72:0/3599494658 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56005dda4e00 con 0x56005de51170 2024-03-22T01:36:43.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.763+0000 7f9ff4dd5640 1 --2- 172.21.15.72:0/3599494658 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x56005de51170 0x56005de535c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:43.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.764+0000 7f9fcffff640 10 client.25454 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:36:43.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.764+0000 7f9ff4dd5640 1 --2- 172.21.15.72:0/3599494658 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x56005de51170 0x56005de535c0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89a34382d4bfa3f2 server_cookie=ae7bc1bb43eb7138 in_seq=0 out_seq=0 2024-03-22T01:36:43.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.764+0000 7f9fcffff640 1 -- 172.21.15.72:0/3599494658 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56005dda4e00 con 0x56005de51170 2024-03-22T01:36:43.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.765+0000 7f9fcffff640 10 client.25454 handle_command_reply: tid=0 2024-03-22T01:36:43.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.765+0000 7f9ff9c29740 1 client.25454 shutdown 2024-03-22T01:36:43.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.765+0000 7f9ff9c29740 1 -- 172.21.15.72:0/3599494658 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56005de3c910 msgr2=0x56005de3a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:43.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.765+0000 7f9ff9c29740 1 --2- 172.21.15.72:0/3599494658 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56005de3c910 0x56005de3a4e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9fe8005c40 tx=0x7f9fe8007470 comp rx=0 tx=0).stop 2024-03-22T01:36:43.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.765+0000 7f9ff9c29740 1 -- 172.21.15.72:0/3599494658 shutdown_connections 2024-03-22T01:36:43.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.765+0000 7f9ff9c29740 1 --2- 172.21.15.72:0/3599494658 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x56005de51170 0x56005de535c0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.765+0000 7f9ff9c29740 1 --2- 172.21.15.72:0/3599494658 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56005de3c910 0x56005de3a4e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.765+0000 7f9ff9c29740 1 --2- 172.21.15.72:0/3599494658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56005de33d30 0x56005de39fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.765+0000 7f9ff9c29740 1 --2- 172.21.15.72:0/3599494658 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56005de33410 0x56005de39a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.765+0000 7f9fcffff640 0 client.25454 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:36:43.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.765+0000 7f9ff9c29740 1 -- 172.21.15.72:0/3599494658 >> 172.21.15.72:0/3599494658 conn(0x56005de2db40 msgr2=0x56005de2f890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:43.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.765+0000 7f9ff9c29740 1 -- 172.21.15.72:0/3599494658 shutdown_connections 2024-03-22T01:36:43.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.765+0000 7f9ff9c29740 1 -- 172.21.15.72:0/3599494658 wait complete. 2024-03-22T01:36:43.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.765+0000 7f9ff9c29740 20 client.25454 trim_cache size 0 max 16384 2024-03-22T01:36:43.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.782+0000 7f9ff9c29740 1 Processor -- start 2024-03-22T01:36:43.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.782+0000 7f9ff9c29740 1 -- start start 2024-03-22T01:36:43.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.782+0000 7f9ff9c29740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56005de33410 0x56005de3c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:43.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.782+0000 7f9ff9c29740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56005de33d30 0x56005de4c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:43.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.782+0000 7f9ff9c29740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56005de3c910 0x56005de38bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:43.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.782+0000 7f9ff9c29740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f9ff01aaeb0 con 0x56005de33410 2024-03-22T01:36:43.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.782+0000 7f9ff9c29740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9ff01ad730 con 0x56005de33d30 2024-03-22T01:36:43.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.782+0000 7f9ff9c29740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9ff01ab520 con 0x56005de3c910 2024-03-22T01:36:43.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.782+0000 7f9ff4dd5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56005de33410 0x56005de3c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:43.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.782+0000 7f9ff4dd5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56005de33410 0x56005de3c190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46934/0 (socket says 172.21.15.72:46934) 2024-03-22T01:36:43.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.782+0000 7f9feffff640 1 --2- 172.21.15.72:0/1664234653 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56005de33d30 0x56005de4c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:43.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.782+0000 7f9ff4dd5640 1 -- 172.21.15.72:0/1664234653 learned_addr learned my addr 172.21.15.72:0/1664234653 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:43.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.782+0000 7f9ff55d6640 1 --2- 172.21.15.72:0/1664234653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56005de3c910 0x56005de38bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:43.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.782+0000 7f9feffff640 1 -- 172.21.15.72:0/1664234653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56005de3c910 msgr2=0x56005de38bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:36:43.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.782+0000 7f9feffff640 1 --2- 172.21.15.72:0/1664234653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56005de3c910 0x56005de38bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.782+0000 7f9feffff640 1 -- 172.21.15.72:0/1664234653 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56005de33410 msgr2=0x56005de3c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:36:43.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.782+0000 7f9feffff640 1 --2- 172.21.15.72:0/1664234653 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56005de33410 0x56005de3c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.782+0000 7f9feffff640 1 -- 172.21.15.72:0/1664234653 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56005de3f070 con 0x56005de33d30 2024-03-22T01:36:43.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.782+0000 7f9ff4dd5640 1 --2- 172.21.15.72:0/1664234653 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56005de33410 0x56005de3c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:36:43.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.782+0000 7f9ff55d6640 1 --2- 172.21.15.72:0/1664234653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56005de3c910 0x56005de38bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:36:43.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.783+0000 7f9feffff640 1 --2- 172.21.15.72:0/1664234653 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56005de33d30 0x56005de4c8b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc02c170 tx=0x7f9fdc02c750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2ba57bc04c554dd server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:43.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.783+0000 7f9fda7fc640 1 -- 172.21.15.72:0/1664234653 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fdc02cb00 con 0x56005de33d30 2024-03-22T01:36:43.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.783+0000 7f9fda7fc640 1 -- 172.21.15.72:0/1664234653 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fdc02cca0 con 0x56005de33d30 2024-03-22T01:36:43.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.783+0000 7f9fda7fc640 1 -- 172.21.15.72:0/1664234653 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fdc0049b0 con 0x56005de33d30 2024-03-22T01:36:43.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.783+0000 7f9ff9c29740 1 -- 172.21.15.72:0/1664234653 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56005de33d30 msgr2=0x56005de4c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:43.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.783+0000 7f9ff9c29740 1 --2- 172.21.15.72:0/1664234653 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56005de33d30 0x56005de4c8b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc02c170 tx=0x7f9fdc02c750 comp rx=0 tx=0).stop 2024-03-22T01:36:43.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.783+0000 7f9ff9c29740 1 -- 172.21.15.72:0/1664234653 shutdown_connections 2024-03-22T01:36:43.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.783+0000 7f9ff9c29740 1 --2- 172.21.15.72:0/1664234653 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56005de33d30 0x56005de4c8b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.783+0000 7f9ff9c29740 1 --2- 172.21.15.72:0/1664234653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56005de3c910 0x56005de38bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.783+0000 7f9ff9c29740 1 --2- 172.21.15.72:0/1664234653 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56005de33410 0x56005de3c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.783+0000 7f9ff9c29740 1 -- 172.21.15.72:0/1664234653 >> 172.21.15.72:0/1664234653 conn(0x56005de2db40 msgr2=0x56005de2c370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:43.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.784+0000 7f9ff9c29740 1 -- 172.21.15.72:0/1664234653 shutdown_connections 2024-03-22T01:36:43.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.784+0000 7f9ff9c29740 1 -- 172.21.15.72:0/1664234653 wait complete. 2024-03-22T01:36:43.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.784+0000 7f9ff9c29740 1 -- start start 2024-03-22T01:36:43.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.784+0000 7f9ff9c29740 1 Processor -- start 2024-03-22T01:36:43.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.784+0000 7f9ff9c29740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56005de33410 0x56005de36610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:43.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.784+0000 7f9ff9c29740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56005de33d30 0x56005de36b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:43.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.784+0000 7f9ff9c29740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56005de3c910 0x56005de37090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:43.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.784+0000 7f9ff9c29740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f9ff01abbf0 con 0x56005de3c910 2024-03-22T01:36:43.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.784+0000 7f9ff9c29740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9ff01ad060 con 0x56005de33410 2024-03-22T01:36:43.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.784+0000 7f9ff9c29740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9ff01ac2c0 con 0x56005de33d30 2024-03-22T01:36:43.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.784+0000 7f9ff9c29740 10 client.0 _ll_register_callbacks cb 0x7f9fe8011e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:36:43.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.784+0000 7f9ff55d6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56005de3c910 0x56005de37090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:43.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.784+0000 7f9ff55d6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56005de3c910 0x56005de37090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46946/0 (socket says 172.21.15.72:46946) 2024-03-22T01:36:43.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.784+0000 7f9ff4dd5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56005de33410 0x56005de36610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:43.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.784+0000 7f9ff55d6640 1 -- 172.21.15.72:0/3505017809 learned_addr learned my addr 172.21.15.72:0/3505017809 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:43.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.785+0000 7f9ff55d6640 1 -- 172.21.15.72:0/3505017809 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56005de33d30 msgr2=0x56005de36b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:36:43.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.785+0000 7f9ff55d6640 1 --2- 172.21.15.72:0/3505017809 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56005de33d30 0x56005de36b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.785+0000 7f9ff55d6640 1 -- 172.21.15.72:0/3505017809 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56005de33410 msgr2=0x56005de36610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:36:43.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.785+0000 7f9ff55d6640 1 --2- 172.21.15.72:0/3505017809 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56005de33410 0x56005de36610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.785+0000 7f9ff55d6640 1 -- 172.21.15.72:0/3505017809 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56005da6b0f0 con 0x56005de3c910 2024-03-22T01:36:43.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.785+0000 7f9ff55d6640 1 --2- 172.21.15.72:0/3505017809 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56005de3c910 0x56005de37090 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f9fe8007a80 tx=0x7f9fe8005980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f0e4a6d8a39985f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:43.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.785+0000 7f9fcffff640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:36:43.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.785+0000 7f9fcffff640 1 -- 172.21.15.72:0/3505017809 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe8026040 con 0x56005de3c910 2024-03-22T01:36:43.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.785+0000 7f9fcffff640 1 -- 172.21.15.72:0/3505017809 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f9fe8004d20 con 0x56005de3c910 2024-03-22T01:36:43.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.785+0000 7f9ff9c29740 1 -- 172.21.15.72:0/3505017809 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9fdc02cca0 con 0x56005de3c910 2024-03-22T01:36:43.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.785+0000 7f9fcffff640 1 -- 172.21.15.72:0/3505017809 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f9fe8043050 con 0x56005de3c910 2024-03-22T01:36:43.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.785+0000 7f9ff9c29740 10 client.15612 fetch_fsmap learned FSMap version 21 2024-03-22T01:36:43.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.785+0000 7f9ff9c29740 1 -- 172.21.15.72:0/3505017809 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x56005deb75e0 con 0x56005de3c910 2024-03-22T01:36:43.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.785+0000 7f9fcffff640 1 -- 172.21.15.72:0/3505017809 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fdc02cca0 con 0x56005de3c910 2024-03-22T01:36:43.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.785+0000 7f9fcffff640 1 -- 172.21.15.72:0/3505017809 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f9fe8020b20 con 0x56005de3c910 2024-03-22T01:36:43.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.785+0000 7f9ff9c29740 10 client.15612 fetch_fsmap finished waiting for FSMap version 21 2024-03-22T01:36:43.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.785+0000 7f9ff9c29740 10 client.15612 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T01:36:43.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.785+0000 7f9ff9c29740 20 client.15612 populate_metadata read hostname 'smithi072' 2024-03-22T01:36:43.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.786+0000 7f9ff9c29740 1 --2- 172.21.15.72:0/3505017809 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x56005deb7d50 0x56005deba170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:43.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.786+0000 7f9ff9c29740 4 client.15612 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T01:36:43.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.786+0000 7f9ff9c29740 1 -- 172.21.15.72:0/3505017809 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56005dd9f150 con 0x56005deb7d50 2024-03-22T01:36:43.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.786+0000 7f9ff4dd5640 1 --2- 172.21.15.72:0/3505017809 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x56005deb7d50 0x56005deba170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:43.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.786+0000 7f9fcffff640 10 client.15612 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:36:43.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.786+0000 7f9ff4dd5640 1 --2- 172.21.15.72:0/3505017809 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x56005deb7d50 0x56005deba170 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=11f8eace121b9da2 server_cookie=3073c0c33d1a365d in_seq=0 out_seq=0 2024-03-22T01:36:43.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.786+0000 7f9fcffff640 1 -- 172.21.15.72:0/3505017809 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x56005dd9f150 con 0x56005deb7d50 2024-03-22T01:36:43.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.786+0000 7f9fcffff640 10 client.15612 handle_command_reply: tid=0 2024-03-22T01:36:43.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.786+0000 7f9ff9c29740 1 client.15612 shutdown 2024-03-22T01:36:43.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.786+0000 7f9ff9c29740 1 -- 172.21.15.72:0/3505017809 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56005de3c910 msgr2=0x56005de37090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:43.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.786+0000 7f9ff9c29740 1 --2- 172.21.15.72:0/3505017809 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56005de3c910 0x56005de37090 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f9fe8007a80 tx=0x7f9fe8005980 comp rx=0 tx=0).stop 2024-03-22T01:36:43.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.786+0000 7f9ff9c29740 1 -- 172.21.15.72:0/3505017809 shutdown_connections 2024-03-22T01:36:43.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.786+0000 7f9ff9c29740 1 --2- 172.21.15.72:0/3505017809 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x56005deb7d50 0x56005deba170 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.786+0000 7f9ff9c29740 1 --2- 172.21.15.72:0/3505017809 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56005de3c910 0x56005de37090 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.786+0000 7f9fcffff640 0 client.15612 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:36:43.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.786+0000 7f9ff9c29740 1 --2- 172.21.15.72:0/3505017809 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56005de33d30 0x56005de36b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.786+0000 7f9ff9c29740 1 --2- 172.21.15.72:0/3505017809 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56005de33410 0x56005de36610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.786+0000 7f9ff9c29740 1 -- 172.21.15.72:0/3505017809 >> 172.21.15.72:0/3505017809 conn(0x56005de2db40 msgr2=0x56005de2c370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:43.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.786+0000 7f9ff9c29740 1 -- 172.21.15.72:0/3505017809 shutdown_connections 2024-03-22T01:36:43.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.787+0000 7f9ff9c29740 1 -- 172.21.15.72:0/3505017809 wait complete. 2024-03-22T01:36:43.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.787+0000 7f9ff9c29740 20 client.15612 trim_cache size 0 max 16384 2024-03-22T01:36:43.807 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-22T01:36:43.807 INFO:teuthology.orchestra.run.smithi072.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-22T01:36:43.807 INFO:teuthology.orchestra.run.smithi072.stdout: "scrubs": { 2024-03-22T01:36:43.807 INFO:teuthology.orchestra.run.smithi072.stdout: "b2cb8289-218c-43e5-a000-ef96955b64f0": { 2024-03-22T01:36:43.807 INFO:teuthology.orchestra.run.smithi072.stdout: "path": "/", 2024-03-22T01:36:43.808 INFO:teuthology.orchestra.run.smithi072.stdout: "tag": "b2cb8289-218c-43e5-a000-ef96955b64f0", 2024-03-22T01:36:43.808 INFO:teuthology.orchestra.run.smithi072.stdout: "options": "recursive,force" 2024-03-22T01:36:43.808 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:36:43.808 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:36:43.808 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-22T01:36:43.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.787+0000 7f9ff7039640 1 -- 172.21.15.72:0/1069202221 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f9fe00a3120 msgr2=0x7f9fe00a55e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:43.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.787+0000 7f9ff7039640 1 --2- 172.21.15.72:0/1069202221 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f9fe00a3120 0x7f9fe00a55e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9fe8007310 tx=0x7f9fe8007de0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.787+0000 7f9ff7039640 1 -- 172.21.15.72:0/1069202221 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ff0100a30 msgr2=0x7f9ff0199210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:43.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.787+0000 7f9ff7039640 1 --2- 172.21.15.72:0/1069202221 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ff0100a30 0x7f9ff0199210 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f9fe400e9f0 tx=0x7f9fe400eeb0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.787+0000 7f9ff7039640 1 -- 172.21.15.72:0/1069202221 shutdown_connections 2024-03-22T01:36:43.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.787+0000 7f9ff7039640 1 --2- 172.21.15.72:0/1069202221 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f9fe00a3120 0x7f9fe00a55e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.787+0000 7f9ff7039640 1 --2- 172.21.15.72:0/1069202221 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ff0106a40 0x7f9ff01a4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.787+0000 7f9ff7039640 1 --2- 172.21.15.72:0/1069202221 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ff01013f0 0x7f9ff0199750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.787+0000 7f9ff7039640 1 --2- 172.21.15.72:0/1069202221 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ff0100a30 0x7f9ff0199210 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.787+0000 7f9ff7039640 1 -- 172.21.15.72:0/1069202221 >> 172.21.15.72:0/1069202221 conn(0x7f9ff0077c40 msgr2=0x7f9ff0103600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:43.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.787+0000 7f9ff7039640 1 -- 172.21.15.72:0/1069202221 shutdown_connections 2024-03-22T01:36:43.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.787+0000 7f9ff7039640 1 -- 172.21.15.72:0/1069202221 wait complete. 2024-03-22T01:36:43.810 INFO:tasks.cephfs.filesystem:scrub status for tag:b2cb8289-218c-43e5-a000-ef96955b64f0 - {'path': '/', 'tag': 'b2cb8289-218c-43e5-a000-ef96955b64f0', 'options': 'recursive,force'} 2024-03-22T01:36:43.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.801+0000 7fc4acc29640 1 -- 172.21.15.72:0/2884558694 --> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7fc4a8061fd0 con 0x7fc4900a3110 2024-03-22T01:36:43.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.862+0000 7fc4977fe640 1 -- 172.21.15.72:0/2884558694 <== mgr.15381 v2:172.21.15.72:6800/1379139045 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fc4a8061fd0 con 0x7fc4900a3110 2024-03-22T01:36:43.861 INFO:teuthology.orchestra.run.smithi072.stdout:Schedule set for path / 2024-03-22T01:36:43.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.864+0000 7fc4acc29640 1 -- 172.21.15.72:0/2884558694 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fc4900a3110 msgr2=0x7fc4900a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:43.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.864+0000 7fc4acc29640 1 --2- 172.21.15.72:0/2884558694 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fc4900a3110 0x7fc4900a55d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc4a81020f0 tx=0x7fc49c00a590 comp rx=0 tx=0).stop 2024-03-22T01:36:43.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.864+0000 7fc4acc29640 1 -- 172.21.15.72:0/2884558694 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc4a8069430 msgr2=0x7fc4a819a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:43.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.864+0000 7fc4acc29640 1 --2- 172.21.15.72:0/2884558694 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc4a8069430 0x7fc4a819a6e0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fc4a0006010 tx=0x7fc4a002cb30 comp rx=0 tx=0).stop 2024-03-22T01:36:43.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.864+0000 7fc4acc29640 1 -- 172.21.15.72:0/2884558694 shutdown_connections 2024-03-22T01:36:43.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.864+0000 7fc4acc29640 1 --2- 172.21.15.72:0/2884558694 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fc4900a3110 0x7fc4900a55d0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.864+0000 7fc4acc29640 1 --2- 172.21.15.72:0/2884558694 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4a8069d50 0x7fc4a819ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.864+0000 7fc4acc29640 1 --2- 172.21.15.72:0/2884558694 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4a810e530 0x7fc4a819efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.864+0000 7fc4acc29640 1 --2- 172.21.15.72:0/2884558694 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc4a8069430 0x7fc4a819a6e0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.864+0000 7fc4acc29640 1 -- 172.21.15.72:0/2884558694 >> 172.21.15.72:0/2884558694 conn(0x7fc4a8077b80 msgr2=0x7fc4a810fae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:43.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.864+0000 7fc4acc29640 1 -- 172.21.15.72:0/2884558694 shutdown_connections 2024-03-22T01:36:43.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.864+0000 7fc4acc29640 1 -- 172.21.15.72:0/2884558694 wait complete. 2024-03-22T01:36:43.873 DEBUG:teuthology.orchestra.run.smithi072:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-03-22T01:36:43.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.979+0000 7fb3edcdf640 1 -- 172.21.15.72:0/1974079452 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3e8101dd0 msgr2=0x7fb3e81021b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:43.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.979+0000 7fb3edcdf640 1 --2- 172.21.15.72:0/1974079452 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3e8101dd0 0x7fb3e81021b0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fb3d8009900 tx=0x7fb3d802bc30 comp rx=0 tx=0).stop 2024-03-22T01:36:43.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.979+0000 7fb3edcdf640 1 -- 172.21.15.72:0/1974079452 shutdown_connections 2024-03-22T01:36:43.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.979+0000 7fb3edcdf640 1 --2- 172.21.15.72:0/1974079452 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3e8106ea0 0x7fb3e810ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.979+0000 7fb3edcdf640 1 --2- 172.21.15.72:0/1974079452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3e8102780 0x7fb3e81067c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.979+0000 7fb3edcdf640 1 --2- 172.21.15.72:0/1974079452 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3e8101dd0 0x7fb3e81021b0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.979+0000 7fb3edcdf640 1 -- 172.21.15.72:0/1974079452 >> 172.21.15.72:0/1974079452 conn(0x7fb3e80fd6b0 msgr2=0x7fb3e80ffad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:43.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.979+0000 7fb3edcdf640 1 -- 172.21.15.72:0/1974079452 shutdown_connections 2024-03-22T01:36:43.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.979+0000 7fb3edcdf640 1 -- 172.21.15.72:0/1974079452 wait complete. 2024-03-22T01:36:43.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.980+0000 7fb3edcdf640 1 Processor -- start 2024-03-22T01:36:43.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.980+0000 7fb3edcdf640 1 -- start start 2024-03-22T01:36:43.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.980+0000 7fb3edcdf640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3e8101dd0 0x7fb3e81984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:43.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.980+0000 7fb3edcdf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3e8106ea0 0x7fb3e8198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:43.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.980+0000 7fb3edcdf640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3e819cd90 0x7fb3e819d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:43.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.980+0000 7fb3edcdf640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fb3e8111f50 con 0x7fb3e8101dd0 2024-03-22T01:36:43.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.980+0000 7fb3edcdf640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb3e8111dd0 con 0x7fb3e819cd90 2024-03-22T01:36:43.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.980+0000 7fb3edcdf640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb3e81120d0 con 0x7fb3e8106ea0 2024-03-22T01:36:43.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.980+0000 7fb3e77fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3e8101dd0 0x7fb3e81984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:43.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.980+0000 7fb3e77fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3e8101dd0 0x7fb3e81984d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46966/0 (socket says 172.21.15.72:46966) 2024-03-22T01:36:43.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.980+0000 7fb3e77fe640 1 -- 172.21.15.72:0/1837910979 learned_addr learned my addr 172.21.15.72:0/1837910979 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:43.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.980+0000 7fb3e7fff640 1 --2- 172.21.15.72:0/1837910979 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3e819cd90 0x7fb3e819d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:43.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.980+0000 7fb3e6ffd640 1 --2- 172.21.15.72:0/1837910979 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3e8106ea0 0x7fb3e8198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:43.988 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.980+0000 7fb3e77fe640 1 -- 172.21.15.72:0/1837910979 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3e8106ea0 msgr2=0x7fb3e8198a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:43.988 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.980+0000 7fb3e77fe640 1 --2- 172.21.15.72:0/1837910979 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3e8106ea0 0x7fb3e8198a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.988 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.980+0000 7fb3e77fe640 1 -- 172.21.15.72:0/1837910979 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3e819cd90 msgr2=0x7fb3e819d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:43.988 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.980+0000 7fb3e77fe640 1 --2- 172.21.15.72:0/1837910979 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3e819cd90 0x7fb3e819d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:43.988 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.980+0000 7fb3e77fe640 1 -- 172.21.15.72:0/1837910979 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3e819d880 con 0x7fb3e8101dd0 2024-03-22T01:36:43.988 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.980+0000 7fb3e7fff640 1 --2- 172.21.15.72:0/1837910979 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3e819cd90 0x7fb3e819d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:36:43.988 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.981+0000 7fb3e77fe640 1 --2- 172.21.15.72:0/1837910979 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3e8101dd0 0x7fb3e81984d0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fb3d8006010 tx=0x7fb3d802cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:43.989 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.981+0000 7fb3e4ff9640 1 -- 172.21.15.72:0/1837910979 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3d8041070 con 0x7fb3e8101dd0 2024-03-22T01:36:43.989 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.981+0000 7fb3e4ff9640 1 -- 172.21.15.72:0/1837910979 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3d8004450 con 0x7fb3e8101dd0 2024-03-22T01:36:43.989 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.981+0000 7fb3e4ff9640 1 -- 172.21.15.72:0/1837910979 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3d8003d00 con 0x7fb3e8101dd0 2024-03-22T01:36:43.989 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.981+0000 7fb3edcdf640 1 -- 172.21.15.72:0/1837910979 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3e819db10 con 0x7fb3e8101dd0 2024-03-22T01:36:43.989 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.981+0000 7fb3edcdf640 1 -- 172.21.15.72:0/1837910979 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3e81a52d0 con 0x7fb3e8101dd0 2024-03-22T01:36:43.989 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.982+0000 7fb3edcdf640 1 -- 172.21.15.72:0/1837910979 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3e81032a0 con 0x7fb3e8101dd0 2024-03-22T01:36:43.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.983+0000 7fb3e4ff9640 1 -- 172.21.15.72:0/1837910979 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 23) ==== 138030+0+0 (secure 0 0 0) 0x7fb3d8044070 con 0x7fb3e8101dd0 2024-03-22T01:36:43.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.983+0000 7fb3e4ff9640 1 --2- 172.21.15.72:0/1837910979 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fb3d00a3110 0x7fb3d00a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:43.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.983+0000 7fb3e6ffd640 1 --2- 172.21.15.72:0/1837910979 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fb3d00a3110 0x7fb3d00a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:43.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.983+0000 7fb3e4ff9640 1 -- 172.21.15.72:0/1837910979 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fb3d80f17d0 con 0x7fb3e8101dd0 2024-03-22T01:36:43.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.984+0000 7fb3e6ffd640 1 --2- 172.21.15.72:0/1837910979 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fb3d00a3110 0x7fb3d00a55d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb3e80fdbd0 tx=0x7fb3dc00b390 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:43.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:43.988+0000 7fb3e4ff9640 1 -- 172.21.15.72:0/1837910979 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb3d80bdeb0 con 0x7fb3e8101dd0 2024-03-22T01:36:44.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:43 smithi089 ceph-mon[35770]: Updating smithi072:/etc/ceph/ceph.client.admin.keyring 2024-03-22T01:36:44.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:43 smithi089 ceph-mon[35770]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-03-22T01:36:44.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:43 smithi089 ceph-mon[35770]: Updating smithi089:/etc/ceph/ceph.client.admin.keyring 2024-03-22T01:36:44.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:43 smithi089 ceph-mon[35770]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:36:44.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:43 smithi089 ceph-mon[35770]: Updating smithi072:/var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T01:36:44.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:43 smithi089 ceph-mon[35770]: Updating smithi098:/var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T01:36:44.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:43 smithi089 ceph-mon[35770]: Updating smithi089:/var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T01:36:44.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:43 smithi089 ceph-mon[35770]: mgrmap e23: x(active, since 4s), standbys: y, z 2024-03-22T01:36:44.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:43 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/938625205' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:36:44.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:43 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/1758706804' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:36:44.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:43 smithi072 ceph-mon[32414]: Updating smithi072:/etc/ceph/ceph.client.admin.keyring 2024-03-22T01:36:44.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:43 smithi072 ceph-mon[32414]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-03-22T01:36:44.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:43 smithi072 ceph-mon[32414]: Updating smithi089:/etc/ceph/ceph.client.admin.keyring 2024-03-22T01:36:44.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:43 smithi072 ceph-mon[32414]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:36:44.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:43 smithi072 ceph-mon[32414]: Updating smithi072:/var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T01:36:44.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:43 smithi072 ceph-mon[32414]: Updating smithi098:/var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T01:36:44.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:43 smithi072 ceph-mon[32414]: Updating smithi089:/var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T01:36:44.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:43 smithi072 ceph-mon[32414]: mgrmap e23: x(active, since 4s), standbys: y, z 2024-03-22T01:36:44.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:43 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/938625205' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:36:44.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:43 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/1758706804' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:36:44.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:43 smithi098 ceph-mon[36468]: Updating smithi072:/etc/ceph/ceph.client.admin.keyring 2024-03-22T01:36:44.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:43 smithi098 ceph-mon[36468]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-03-22T01:36:44.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:43 smithi098 ceph-mon[36468]: Updating smithi089:/etc/ceph/ceph.client.admin.keyring 2024-03-22T01:36:44.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:43 smithi098 ceph-mon[36468]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:36:44.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:43 smithi098 ceph-mon[36468]: Updating smithi072:/var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T01:36:44.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:43 smithi098 ceph-mon[36468]: Updating smithi098:/var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T01:36:44.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:43 smithi098 ceph-mon[36468]: Updating smithi089:/var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T01:36:44.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:43 smithi098 ceph-mon[36468]: mgrmap e23: x(active, since 4s), standbys: y, z 2024-03-22T01:36:44.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:43 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/938625205' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:36:44.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:43 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/1758706804' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:36:44.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.188+0000 7fb3edcdf640 1 -- 172.21.15.72:0/1837910979 --> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7fb3e8102780 con 0x7fb3d00a3110 2024-03-22T01:36:44.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.245+0000 7fb3e4ff9640 1 -- 172.21.15.72:0/1837910979 <== mgr.15381 v2:172.21.15.72:6800/1379139045 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7fb3e8102780 con 0x7fb3d00a3110 2024-03-22T01:36:44.245 INFO:teuthology.orchestra.run.smithi072.stdout:Retention added to path / 2024-03-22T01:36:44.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.247+0000 7fb3edcdf640 1 -- 172.21.15.72:0/1837910979 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fb3d00a3110 msgr2=0x7fb3d00a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:44.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.247+0000 7fb3edcdf640 1 --2- 172.21.15.72:0/1837910979 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fb3d00a3110 0x7fb3d00a55d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb3e80fdbd0 tx=0x7fb3dc00b390 comp rx=0 tx=0).stop 2024-03-22T01:36:44.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.247+0000 7fb3edcdf640 1 -- 172.21.15.72:0/1837910979 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3e8101dd0 msgr2=0x7fb3e81984d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:44.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.247+0000 7fb3edcdf640 1 --2- 172.21.15.72:0/1837910979 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3e8101dd0 0x7fb3e81984d0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fb3d8006010 tx=0x7fb3d802cb30 comp rx=0 tx=0).stop 2024-03-22T01:36:44.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.247+0000 7fb3edcdf640 1 -- 172.21.15.72:0/1837910979 shutdown_connections 2024-03-22T01:36:44.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.247+0000 7fb3edcdf640 1 --2- 172.21.15.72:0/1837910979 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fb3d00a3110 0x7fb3d00a55d0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:44.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.248+0000 7fb3edcdf640 1 --2- 172.21.15.72:0/1837910979 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3e819cd90 0x7fb3e819d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:44.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.248+0000 7fb3edcdf640 1 --2- 172.21.15.72:0/1837910979 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3e8106ea0 0x7fb3e8198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:44.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.248+0000 7fb3edcdf640 1 --2- 172.21.15.72:0/1837910979 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3e8101dd0 0x7fb3e81984d0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:44.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.248+0000 7fb3edcdf640 1 -- 172.21.15.72:0/1837910979 >> 172.21.15.72:0/1837910979 conn(0x7fb3e80fd6b0 msgr2=0x7fb3e80fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:44.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.248+0000 7fb3edcdf640 1 -- 172.21.15.72:0/1837910979 shutdown_connections 2024-03-22T01:36:44.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.248+0000 7fb3edcdf640 1 -- 172.21.15.72:0/1837910979 wait complete. 2024-03-22T01:36:44.258 DEBUG:teuthology.orchestra.run.smithi072:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-03-22T01:36:44.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.364+0000 7fe7cbb3a640 1 -- 172.21.15.72:0/1407895436 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7c4100a30 msgr2=0x7fe7c4100eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:44.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.364+0000 7fe7cbb3a640 1 --2- 172.21.15.72:0/1407895436 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7c4100a30 0x7fe7c4100eb0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe7b4009860 tx=0x7fe7b402bb70 comp rx=0 tx=0).stop 2024-03-22T01:36:44.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.364+0000 7fe7cbb3a640 1 -- 172.21.15.72:0/1407895436 shutdown_connections 2024-03-22T01:36:44.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.364+0000 7fe7cbb3a640 1 --2- 172.21.15.72:0/1407895436 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7c4100a30 0x7fe7c4100eb0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:44.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.364+0000 7fe7cbb3a640 1 --2- 172.21.15.72:0/1407895436 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe7c41013f0 0x7fe7c410d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:44.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.364+0000 7fe7cbb3a640 1 --2- 172.21.15.72:0/1407895436 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7c4106a40 0x7fe7c4106e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:44.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.364+0000 7fe7cbb3a640 1 -- 172.21.15.72:0/1407895436 >> 172.21.15.72:0/1407895436 conn(0x7fe7c4077b80 msgr2=0x7fe7c4077f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:44.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.365+0000 7fe7cbb3a640 1 -- 172.21.15.72:0/1407895436 shutdown_connections 2024-03-22T01:36:44.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.365+0000 7fe7cbb3a640 1 -- 172.21.15.72:0/1407895436 wait complete. 2024-03-22T01:36:44.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.365+0000 7fe7cbb3a640 1 Processor -- start 2024-03-22T01:36:44.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.365+0000 7fe7cbb3a640 1 -- start start 2024-03-22T01:36:44.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.365+0000 7fe7cbb3a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7c4100a30 0x7fe7c4199210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:44.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.365+0000 7fe7cbb3a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe7c41013f0 0x7fe7c4199750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:44.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.365+0000 7fe7cbb3a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7c4106a40 0x7fe7c41a4290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:44.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.365+0000 7fe7cbb3a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fe7c4110e50 con 0x7fe7c41013f0 2024-03-22T01:36:44.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.365+0000 7fe7cbb3a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe7c4110cd0 con 0x7fe7c4106a40 2024-03-22T01:36:44.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.365+0000 7fe7cbb3a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe7c4110fd0 con 0x7fe7c4100a30 2024-03-22T01:36:44.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.365+0000 7fe7c90d5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe7c41013f0 0x7fe7c4199750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:44.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.365+0000 7fe7ca0d7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7c4106a40 0x7fe7c41a4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:44.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.365+0000 7fe7c90d5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe7c41013f0 0x7fe7c4199750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46982/0 (socket says 172.21.15.72:46982) 2024-03-22T01:36:44.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.365+0000 7fe7c90d5640 1 -- 172.21.15.72:0/2676751454 learned_addr learned my addr 172.21.15.72:0/2676751454 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:44.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.365+0000 7fe7c98d6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7c4100a30 0x7fe7c4199210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:44.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.366+0000 7fe7c90d5640 1 -- 172.21.15.72:0/2676751454 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7c4100a30 msgr2=0x7fe7c4199210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:44.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.366+0000 7fe7c90d5640 1 --2- 172.21.15.72:0/2676751454 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7c4100a30 0x7fe7c4199210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:44.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.366+0000 7fe7c90d5640 1 -- 172.21.15.72:0/2676751454 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7c4106a40 msgr2=0x7fe7c41a4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:44.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.366+0000 7fe7c90d5640 1 --2- 172.21.15.72:0/2676751454 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7c4106a40 0x7fe7c41a4290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:44.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.366+0000 7fe7c90d5640 1 -- 172.21.15.72:0/2676751454 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7c41a47d0 con 0x7fe7c41013f0 2024-03-22T01:36:44.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.366+0000 7fe7ca0d7640 1 --2- 172.21.15.72:0/2676751454 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7c4106a40 0x7fe7c41a4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:36:44.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.366+0000 7fe7c90d5640 1 --2- 172.21.15.72:0/2676751454 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe7c41013f0 0x7fe7c4199750 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fe7b4037040 tx=0x7fe7b40323f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:44.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.366+0000 7fe7baffd640 1 -- 172.21.15.72:0/2676751454 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7b4041020 con 0x7fe7c41013f0 2024-03-22T01:36:44.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.366+0000 7fe7cbb3a640 1 -- 172.21.15.72:0/2676751454 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7c41a49a0 con 0x7fe7c41013f0 2024-03-22T01:36:44.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.366+0000 7fe7baffd640 1 -- 172.21.15.72:0/2676751454 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7b402c9f0 con 0x7fe7c41013f0 2024-03-22T01:36:44.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.366+0000 7fe7baffd640 1 -- 172.21.15.72:0/2676751454 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7b4039040 con 0x7fe7c41013f0 2024-03-22T01:36:44.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.366+0000 7fe7cbb3a640 1 -- 172.21.15.72:0/2676751454 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7c41a4e00 con 0x7fe7c41013f0 2024-03-22T01:36:44.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.367+0000 7fe7cbb3a640 1 -- 172.21.15.72:0/2676751454 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7c4050140 con 0x7fe7c41013f0 2024-03-22T01:36:44.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.368+0000 7fe7baffd640 1 -- 172.21.15.72:0/2676751454 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 23) ==== 138030+0+0 (secure 0 0 0) 0x7fe7b4044020 con 0x7fe7c41013f0 2024-03-22T01:36:44.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.369+0000 7fe7baffd640 1 --2- 172.21.15.72:0/2676751454 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fe7b00a3110 0x7fe7b00a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:44.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.369+0000 7fe7baffd640 1 -- 172.21.15.72:0/2676751454 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fe7b40f1890 con 0x7fe7c41013f0 2024-03-22T01:36:44.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.369+0000 7fe7c98d6640 1 --2- 172.21.15.72:0/2676751454 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fe7b00a3110 0x7fe7b00a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:44.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.369+0000 7fe7c98d6640 1 --2- 172.21.15.72:0/2676751454 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fe7b00a3110 0x7fe7b00a55d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe7c0005d00 tx=0x7fe7c000b410 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:44.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.376+0000 7fe7baffd640 1 -- 172.21.15.72:0/2676751454 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe7b404b020 con 0x7fe7c41013f0 2024-03-22T01:36:44.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.579+0000 7fe7cbb3a640 1 -- 172.21.15.72:0/2676751454 --> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7fe7c4061fd0 con 0x7fe7b00a3110 2024-03-22T01:36:44.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.586+0000 7fe7baffd640 1 -- 172.21.15.72:0/2676751454 <== mgr.15381 v2:172.21.15.72:6800/1379139045 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7fe7c4061fd0 con 0x7fe7b00a3110 2024-03-22T01:36:44.585 INFO:teuthology.orchestra.run.smithi072.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-03-22T00:00:00", "created": "2024-03-22T01:36:43", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-03-22T01:36:44.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.588+0000 7fe7cbb3a640 1 -- 172.21.15.72:0/2676751454 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fe7b00a3110 msgr2=0x7fe7b00a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:44.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.588+0000 7fe7cbb3a640 1 --2- 172.21.15.72:0/2676751454 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fe7b00a3110 0x7fe7b00a55d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe7c0005d00 tx=0x7fe7c000b410 comp rx=0 tx=0).stop 2024-03-22T01:36:44.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.588+0000 7fe7cbb3a640 1 -- 172.21.15.72:0/2676751454 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe7c41013f0 msgr2=0x7fe7c4199750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:44.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.588+0000 7fe7cbb3a640 1 --2- 172.21.15.72:0/2676751454 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe7c41013f0 0x7fe7c4199750 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fe7b4037040 tx=0x7fe7b40323f0 comp rx=0 tx=0).stop 2024-03-22T01:36:44.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.588+0000 7fe7cbb3a640 1 -- 172.21.15.72:0/2676751454 shutdown_connections 2024-03-22T01:36:44.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.588+0000 7fe7cbb3a640 1 --2- 172.21.15.72:0/2676751454 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fe7b00a3110 0x7fe7b00a55d0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:44.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.588+0000 7fe7cbb3a640 1 --2- 172.21.15.72:0/2676751454 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7c4106a40 0x7fe7c41a4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:44.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.588+0000 7fe7cbb3a640 1 --2- 172.21.15.72:0/2676751454 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe7c41013f0 0x7fe7c4199750 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:44.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.588+0000 7fe7cbb3a640 1 --2- 172.21.15.72:0/2676751454 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7c4100a30 0x7fe7c4199210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:44.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.588+0000 7fe7cbb3a640 1 -- 172.21.15.72:0/2676751454 >> 172.21.15.72:0/2676751454 conn(0x7fe7c4077b80 msgr2=0x7fe7c4103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:44.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.588+0000 7fe7cbb3a640 1 -- 172.21.15.72:0/2676751454 shutdown_connections 2024-03-22T01:36:44.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.588+0000 7fe7cbb3a640 1 -- 172.21.15.72:0/2676751454 wait complete. 2024-03-22T01:36:44.598 DEBUG:teuthology.orchestra.run.smithi072:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-03-22T01:36:44.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.705+0000 7f570c5f7640 1 -- 172.21.15.72:0/3750663061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57041013f0 msgr2=0x7f570410d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:44.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.705+0000 7f570c5f7640 1 --2- 172.21.15.72:0/3750663061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57041013f0 0x7f570410d900 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f56fc009be0 tx=0x7f56fc02bef0 comp rx=0 tx=0).stop 2024-03-22T01:36:44.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.705+0000 7f570c5f7640 1 -- 172.21.15.72:0/3750663061 shutdown_connections 2024-03-22T01:36:44.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.705+0000 7f570c5f7640 1 --2- 172.21.15.72:0/3750663061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57041013f0 0x7f570410d900 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:44.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.705+0000 7f570c5f7640 1 --2- 172.21.15.72:0/3750663061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5704100a30 0x7f5704100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:44.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.705+0000 7f570c5f7640 1 --2- 172.21.15.72:0/3750663061 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5704106a40 0x7f5704106e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:44.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.705+0000 7f570c5f7640 1 -- 172.21.15.72:0/3750663061 >> 172.21.15.72:0/3750663061 conn(0x7f5704077b80 msgr2=0x7f5704077f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:44.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.705+0000 7f570c5f7640 1 -- 172.21.15.72:0/3750663061 shutdown_connections 2024-03-22T01:36:44.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.705+0000 7f570c5f7640 1 -- 172.21.15.72:0/3750663061 wait complete. 2024-03-22T01:36:44.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.706+0000 7f570c5f7640 1 Processor -- start 2024-03-22T01:36:44.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.706+0000 7f570c5f7640 1 -- start start 2024-03-22T01:36:44.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.706+0000 7f570c5f7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5704100a30 0x7f5704197330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:44.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.706+0000 7f570c5f7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57041013f0 0x7f5704197870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:44.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.706+0000 7f570c5f7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5704106a40 0x7f570419bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:44.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.706+0000 7f570c5f7640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f5704110e50 con 0x7f5704106a40 2024-03-22T01:36:44.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.706+0000 7f570c5f7640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5704110cd0 con 0x7f57041013f0 2024-03-22T01:36:44.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.706+0000 7f570c5f7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5704110fd0 con 0x7f5704100a30 2024-03-22T01:36:44.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.706+0000 7f570ab94640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5704106a40 0x7f570419bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:44.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.706+0000 7f5709b92640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57041013f0 0x7f5704197870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:44.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.706+0000 7f5709b92640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57041013f0 0x7f5704197870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.72:53016/0 (socket says 172.21.15.72:53016) 2024-03-22T01:36:44.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.706+0000 7f570ab94640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5704106a40 0x7f570419bba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47000/0 (socket says 172.21.15.72:47000) 2024-03-22T01:36:44.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.706+0000 7f5709b92640 1 -- 172.21.15.72:0/2974476592 learned_addr learned my addr 172.21.15.72:0/2974476592 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:44.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.706+0000 7f570a393640 1 --2- 172.21.15.72:0/2974476592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5704100a30 0x7f5704197330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:44.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.706+0000 7f5709b92640 1 -- 172.21.15.72:0/2974476592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5704100a30 msgr2=0x7f5704197330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:44.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.706+0000 7f5709b92640 1 --2- 172.21.15.72:0/2974476592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5704100a30 0x7f5704197330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:44.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.706+0000 7f5709b92640 1 -- 172.21.15.72:0/2974476592 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5704106a40 msgr2=0x7f570419bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:44.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.706+0000 7f5709b92640 1 --2- 172.21.15.72:0/2974476592 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5704106a40 0x7f570419bba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:44.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.706+0000 7f5709b92640 1 -- 172.21.15.72:0/2974476592 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f570419c2d0 con 0x7f57041013f0 2024-03-22T01:36:44.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.706+0000 7f570ab94640 1 --2- 172.21.15.72:0/2974476592 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5704106a40 0x7f570419bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:36:44.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.707+0000 7f5709b92640 1 --2- 172.21.15.72:0/2974476592 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57041013f0 0x7f5704197870 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f56f400e9f0 tx=0x7f56f400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:44.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.707+0000 7f56fb7fe640 1 -- 172.21.15.72:0/2974476592 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56f400ccc0 con 0x7f57041013f0 2024-03-22T01:36:44.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.707+0000 7f570c5f7640 1 -- 172.21.15.72:0/2974476592 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f570419c5c0 con 0x7f57041013f0 2024-03-22T01:36:44.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.707+0000 7f56fb7fe640 1 -- 172.21.15.72:0/2974476592 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f56f400ce60 con 0x7f57041013f0 2024-03-22T01:36:44.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.707+0000 7f56fb7fe640 1 -- 172.21.15.72:0/2974476592 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56f4010690 con 0x7f57041013f0 2024-03-22T01:36:44.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.707+0000 7f570c5f7640 1 -- 172.21.15.72:0/2974476592 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57041a3e50 con 0x7f57041013f0 2024-03-22T01:36:44.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.708+0000 7f570c5f7640 1 -- 172.21.15.72:0/2974476592 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5704050140 con 0x7f57041013f0 2024-03-22T01:36:44.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.709+0000 7f56fb7fe640 1 -- 172.21.15.72:0/2974476592 <== mon.2 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 23) ==== 138030+0+0 (secure 0 0 0) 0x7f56f40046c0 con 0x7f57041013f0 2024-03-22T01:36:44.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.709+0000 7f56fb7fe640 1 --2- 172.21.15.72:0/2974476592 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f56f00a3110 0x7f56f00a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:44.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.709+0000 7f570a393640 1 --2- 172.21.15.72:0/2974476592 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f56f00a3110 0x7f56f00a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:44.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.709+0000 7f56fb7fe640 1 -- 172.21.15.72:0/2974476592 <== mon.2 v2:172.21.15.89:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f56f40d0560 con 0x7f57041013f0 2024-03-22T01:36:44.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.710+0000 7f570a393640 1 --2- 172.21.15.72:0/2974476592 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f56f00a3110 0x7f56f00a55d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5700005fd0 tx=0x7f5700009260 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:44.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.713+0000 7f56fb7fe640 1 -- 172.21.15.72:0/2974476592 <== mon.2 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f56f409cad0 con 0x7f57041013f0 2024-03-22T01:36:44.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.919+0000 7f570c5f7640 1 -- 172.21.15.72:0/2974476592 --> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f5704061fd0 con 0x7f56f00a3110 2024-03-22T01:36:44.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.926+0000 7f56fb7fe640 1 -- 172.21.15.72:0/2974476592 <== mgr.15381 v2:172.21.15.72:6800/1379139045 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f5704061fd0 con 0x7f56f00a3110 2024-03-22T01:36:44.925 INFO:teuthology.orchestra.run.smithi072.stdout:/ 1m 6m3h 2024-03-22T01:36:44.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.928+0000 7f570c5f7640 1 -- 172.21.15.72:0/2974476592 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f56f00a3110 msgr2=0x7f56f00a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:44.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.928+0000 7f570c5f7640 1 --2- 172.21.15.72:0/2974476592 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f56f00a3110 0x7f56f00a55d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5700005fd0 tx=0x7f5700009260 comp rx=0 tx=0).stop 2024-03-22T01:36:44.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.928+0000 7f570c5f7640 1 -- 172.21.15.72:0/2974476592 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57041013f0 msgr2=0x7f5704197870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:44.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.928+0000 7f570c5f7640 1 --2- 172.21.15.72:0/2974476592 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57041013f0 0x7f5704197870 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f56f400e9f0 tx=0x7f56f400edf0 comp rx=0 tx=0).stop 2024-03-22T01:36:44.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.928+0000 7f570c5f7640 1 -- 172.21.15.72:0/2974476592 shutdown_connections 2024-03-22T01:36:44.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.928+0000 7f570c5f7640 1 --2- 172.21.15.72:0/2974476592 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f56f00a3110 0x7f56f00a55d0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:44.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.928+0000 7f570c5f7640 1 --2- 172.21.15.72:0/2974476592 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57041013f0 0x7f5704197870 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:44.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.928+0000 7f570c5f7640 1 --2- 172.21.15.72:0/2974476592 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5704106a40 0x7f570419bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:44.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.928+0000 7f570c5f7640 1 --2- 172.21.15.72:0/2974476592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5704100a30 0x7f5704197330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:44.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.928+0000 7f570c5f7640 1 -- 172.21.15.72:0/2974476592 >> 172.21.15.72:0/2974476592 conn(0x7f5704077b80 msgr2=0x7f5704103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:44.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.928+0000 7f570c5f7640 1 -- 172.21.15.72:0/2974476592 shutdown_connections 2024-03-22T01:36:44.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:36:44.928+0000 7f570c5f7640 1 -- 172.21.15.72:0/2974476592 wait complete. 2024-03-22T01:36:44.938 DEBUG:teuthology.orchestra.run.smithi072:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-03-22T01:36:44.966 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-03-22T01:36:44.993 INFO:teuthology.run_tasks:Running task background_exec... 2024-03-22T01:36:45.022 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi072.front.sepia.ceph.com 2024-03-22T01:36:45.022 DEBUG:teuthology.orchestra.run.smithi072:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-03-22T01:36:45.025 INFO:teuthology.run_tasks:Running task workunit... 2024-03-22T01:36:45.037 INFO:tasks.workunit:Pulling workunits from ref c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e 2024-03-22T01:36:45.038 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-22T01:36:45.038 INFO:tasks.workunit:timeout=3h 2024-03-22T01:36:45.038 INFO:tasks.workunit:cleanup=True 2024-03-22T01:36:45.038 DEBUG:teuthology.orchestra.run.smithi072:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-22T01:36:45.053 INFO:teuthology.orchestra.run.smithi072.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-22T01:36:45.053 INFO:teuthology.orchestra.run.smithi072.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-03-22T01:36:45.053 INFO:teuthology.orchestra.run.smithi072.stdout:Device: 77h/119d Inode: 1099511628283 Links: 2 2024-03-22T01:36:45.053 INFO:teuthology.orchestra.run.smithi072.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-22T01:36:45.053 INFO:teuthology.orchestra.run.smithi072.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-03-22T01:36:45.053 INFO:teuthology.orchestra.run.smithi072.stdout:Access: 2024-03-22 01:34:41.887657552 +0000 2024-03-22T01:36:45.053 INFO:teuthology.orchestra.run.smithi072.stdout:Modify: 2024-03-22 01:34:41.887657552 +0000 2024-03-22T01:36:45.054 INFO:teuthology.orchestra.run.smithi072.stdout:Change: 2024-03-22 01:36:36.040655594 +0000 2024-03-22T01:36:45.054 INFO:teuthology.orchestra.run.smithi072.stdout: Birth: 2024-03-22 01:34:41.887657552 +0000 2024-03-22T01:36:45.054 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-22T01:36:45.054 DEBUG:teuthology.orchestra.run.smithi072:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-22T01:36:45.107 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:44 smithi089 ceph-mon[35770]: scrub queued for path: / 2024-03-22T01:36:45.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:44 smithi089 ceph-mon[35770]: scrub summary: idle+waiting paths [/] 2024-03-22T01:36:45.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:44 smithi089 ceph-mon[35770]: scrub summary: active paths [/] 2024-03-22T01:36:45.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:44 smithi089 ceph-mon[35770]: scrub summary: idle+waiting paths [/] 2024-03-22T01:36:45.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:44 smithi089 ceph-mon[35770]: from='client.15579 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:36:45.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:44 smithi089 ceph-mon[35770]: scrub summary: idle 2024-03-22T01:36:45.118 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:44 smithi072 ceph-mon[32414]: scrub queued for path: / 2024-03-22T01:36:45.118 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:44 smithi072 ceph-mon[32414]: scrub summary: idle+waiting paths [/] 2024-03-22T01:36:45.118 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:44 smithi072 ceph-mon[32414]: scrub summary: active paths [/] 2024-03-22T01:36:45.118 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:44 smithi072 ceph-mon[32414]: scrub summary: idle+waiting paths [/] 2024-03-22T01:36:45.118 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:44 smithi072 ceph-mon[32414]: from='client.15579 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:36:45.118 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:44 smithi072 ceph-mon[32414]: scrub summary: idle 2024-03-22T01:36:45.127 DEBUG:teuthology.orchestra.run.smithi072:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e 2024-03-22T01:36:45.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:44 smithi098 ceph-mon[36468]: scrub queued for path: / 2024-03-22T01:36:45.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:44 smithi098 ceph-mon[36468]: scrub summary: idle+waiting paths [/] 2024-03-22T01:36:45.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:44 smithi098 ceph-mon[36468]: scrub summary: active paths [/] 2024-03-22T01:36:45.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:44 smithi098 ceph-mon[36468]: scrub summary: idle+waiting paths [/] 2024-03-22T01:36:45.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:44 smithi098 ceph-mon[36468]: from='client.15579 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:36:45.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:44 smithi098 ceph-mon[36468]: scrub summary: idle 2024-03-22T01:36:45.182 INFO:tasks.workunit.client.0.smithi072.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-22T01:36:46.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:45 smithi089 ceph-mon[35770]: from='client.15618 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:36:46.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:45 smithi089 ceph-mon[35770]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:36:46.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:45 smithi089 ceph-mon[35770]: from='client.15624 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:36:46.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:45 smithi089 ceph-mon[35770]: from='client.25478 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:36:46.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:45 smithi072 ceph-mon[32414]: from='client.15618 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:36:46.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:45 smithi072 ceph-mon[32414]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:36:46.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:45 smithi072 ceph-mon[32414]: from='client.15624 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:36:46.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:45 smithi072 ceph-mon[32414]: from='client.25478 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:36:46.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:45 smithi098 ceph-mon[36468]: from='client.15618 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:36:46.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:45 smithi098 ceph-mon[36468]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:36:46.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:45 smithi098 ceph-mon[36468]: from='client.15624 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:36:46.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:45 smithi098 ceph-mon[36468]: from='client.25478 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:36:48.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:47 smithi089 ceph-mon[35770]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 112 KiB/s wr, 48 op/s 2024-03-22T01:36:48.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:47 smithi072 ceph-mon[32414]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 112 KiB/s wr, 48 op/s 2024-03-22T01:36:48.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:47 smithi098 ceph-mon[36468]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 112 KiB/s wr, 48 op/s 2024-03-22T01:36:50.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:49 smithi089 ceph-mon[35770]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 87 KiB/s wr, 37 op/s 2024-03-22T01:36:50.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:49 smithi072 ceph-mon[32414]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 87 KiB/s wr, 37 op/s 2024-03-22T01:36:50.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:49 smithi098 ceph-mon[36468]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 87 KiB/s wr, 37 op/s 2024-03-22T01:36:52.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:51 smithi072 ceph-mon[32414]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 84 KiB/s wr, 47 op/s 2024-03-22T01:36:52.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:51 smithi098 ceph-mon[36468]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 84 KiB/s wr, 47 op/s 2024-03-22T01:36:52.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:51 smithi089 ceph-mon[35770]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 84 KiB/s wr, 47 op/s 2024-03-22T01:36:54.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:53 smithi072 ceph-mon[32414]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 77 KiB/s wr, 43 op/s 2024-03-22T01:36:54.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:53 smithi098 ceph-mon[36468]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 77 KiB/s wr, 43 op/s 2024-03-22T01:36:54.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:53 smithi089 ceph-mon[35770]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 77 KiB/s wr, 43 op/s 2024-03-22T01:36:56.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:55 smithi072 ceph-mon[32414]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 77 KiB/s wr, 43 op/s 2024-03-22T01:36:56.160 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:55 smithi089 ceph-mon[35770]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 77 KiB/s wr, 43 op/s 2024-03-22T01:36:56.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:55 smithi098 ceph-mon[36468]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 77 KiB/s wr, 43 op/s 2024-03-22T01:36:58.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:57 smithi072 ceph-mon[32414]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 77 KiB/s wr, 44 op/s 2024-03-22T01:36:58.167 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.165+0000 7f97e756b640 1 -- 172.21.15.72:0/2765976432 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f97e0106a40 msgr2=0x7f97e0106e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:58.167 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.165+0000 7f97e756b640 1 --2- 172.21.15.72:0/2765976432 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f97e0106a40 0x7f97e0106e20 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f97dc009930 tx=0x7f97dc02bc30 comp rx=0 tx=0).stop 2024-03-22T01:36:58.168 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.166+0000 7f97e756b640 1 -- 172.21.15.72:0/2765976432 shutdown_connections 2024-03-22T01:36:58.168 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.166+0000 7f97e756b640 1 --2- 172.21.15.72:0/2765976432 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97e0100a30 0x7f97e0100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:58.168 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.166+0000 7f97e756b640 1 --2- 172.21.15.72:0/2765976432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97e01013f0 0x7f97e010d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:58.168 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.166+0000 7f97e756b640 1 --2- 172.21.15.72:0/2765976432 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f97e0106a40 0x7f97e0106e20 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:58.168 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.166+0000 7f97e756b640 1 -- 172.21.15.72:0/2765976432 >> 172.21.15.72:0/2765976432 conn(0x7f97e0077b80 msgr2=0x7f97e0077f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:58.168 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.166+0000 7f97e756b640 1 -- 172.21.15.72:0/2765976432 shutdown_connections 2024-03-22T01:36:58.168 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.166+0000 7f97e756b640 1 -- 172.21.15.72:0/2765976432 wait complete. 2024-03-22T01:36:58.168 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.166+0000 7f97e756b640 1 Processor -- start 2024-03-22T01:36:58.168 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.166+0000 7f97e756b640 1 -- start start 2024-03-22T01:36:58.168 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.166+0000 7f97e756b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97e0100a30 0x7f97e01992a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:58.168 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.166+0000 7f97e756b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97e01013f0 0x7f97e01997e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:58.169 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.166+0000 7f97e756b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f97e0106a40 0x7f97e01a4320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:58.169 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.166+0000 7f97e756b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f97e0110e20 con 0x7f97e0106a40 2024-03-22T01:36:58.169 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.166+0000 7f97e756b640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f97e0110ca0 con 0x7f97e0100a30 2024-03-22T01:36:58.169 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.166+0000 7f97e756b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f97e0110fa0 con 0x7f97e01013f0 2024-03-22T01:36:58.169 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.167+0000 7f97e5b08640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f97e0106a40 0x7f97e01a4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:58.169 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.167+0000 7f97e5b08640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f97e0106a40 0x7f97e01a4320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:51902/0 (socket says 172.21.15.72:51902) 2024-03-22T01:36:58.169 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.167+0000 7f97e5b08640 1 -- 172.21.15.72:0/1661228648 learned_addr learned my addr 172.21.15.72:0/1661228648 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:58.169 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.167+0000 7f97e5307640 1 --2- 172.21.15.72:0/1661228648 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97e0100a30 0x7f97e01992a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:58.169 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.167+0000 7f97e4b06640 1 --2- 172.21.15.72:0/1661228648 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97e01013f0 0x7f97e01997e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:58.169 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.167+0000 7f97e5b08640 1 -- 172.21.15.72:0/1661228648 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97e01013f0 msgr2=0x7f97e01997e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:58.169 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.167+0000 7f97e5b08640 1 --2- 172.21.15.72:0/1661228648 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97e01013f0 0x7f97e01997e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:58.169 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.167+0000 7f97e5b08640 1 -- 172.21.15.72:0/1661228648 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97e0100a30 msgr2=0x7f97e01992a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:58.169 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.167+0000 7f97e5b08640 1 --2- 172.21.15.72:0/1661228648 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97e0100a30 0x7f97e01992a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:58.169 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.167+0000 7f97e5b08640 1 -- 172.21.15.72:0/1661228648 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97e01a4860 con 0x7f97e0106a40 2024-03-22T01:36:58.170 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.167+0000 7f97e5307640 1 --2- 172.21.15.72:0/1661228648 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97e0100a30 0x7f97e01992a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:36:58.170 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.167+0000 7f97e4b06640 1 --2- 172.21.15.72:0/1661228648 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97e01013f0 0x7f97e01997e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T01:36:58.170 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.167+0000 7f97e5b08640 1 --2- 172.21.15.72:0/1661228648 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f97e0106a40 0x7f97e01a4320 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f97d8009fd0 tx=0x7f97d800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:58.170 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.167+0000 7f97d67fc640 1 -- 172.21.15.72:0/1661228648 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97d8017070 con 0x7f97e0106a40 2024-03-22T01:36:58.170 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.167+0000 7f97d67fc640 1 -- 172.21.15.72:0/1661228648 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97d8003e50 con 0x7f97e0106a40 2024-03-22T01:36:58.170 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.167+0000 7f97e756b640 1 -- 172.21.15.72:0/1661228648 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97e01a4a30 con 0x7f97e0106a40 2024-03-22T01:36:58.170 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.167+0000 7f97d67fc640 1 -- 172.21.15.72:0/1661228648 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97d80106c0 con 0x7f97e0106a40 2024-03-22T01:36:58.170 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.167+0000 7f97e756b640 1 -- 172.21.15.72:0/1661228648 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97e01a4f30 con 0x7f97e0106a40 2024-03-22T01:36:58.176 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:57 smithi098 ceph-mon[36468]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 77 KiB/s wr, 44 op/s 2024-03-22T01:36:58.178 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.170+0000 7f97d67fc640 1 -- 172.21.15.72:0/1661228648 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 23) ==== 138030+0+0 (secure 0 0 0) 0x7f97d8003970 con 0x7f97e0106a40 2024-03-22T01:36:58.178 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.170+0000 7f97ea22a740 1 Processor -- start 2024-03-22T01:36:58.178 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.170+0000 7f97ea22a740 1 -- start start 2024-03-22T01:36:58.178 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.171+0000 7f97ea22a740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560bd746d2d0 0x560bd746d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:58.178 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.171+0000 7f97ea22a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560bd746dbf0 0x560bd7476480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:58.178 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.171+0000 7f97ea22a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bd74769c0 0x560bd7478db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:58.178 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.171+0000 7f97ea22a740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x560bd73d1040 con 0x560bd746d2d0 2024-03-22T01:36:58.178 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.171+0000 7f97ea22a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x560bd73799e0 con 0x560bd746dbf0 2024-03-22T01:36:58.178 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.171+0000 7f97ea22a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560bd73ccff0 con 0x560bd74769c0 2024-03-22T01:36:58.179 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.171+0000 7f97d67fc640 1 --2- 172.21.15.72:0/1661228648 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f97cc0a3110 0x7f97cc0a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:58.179 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.171+0000 7f97e4b06640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560bd746d2d0 0x560bd746d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:58.179 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.171+0000 7f97e5307640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560bd746dbf0 0x560bd7476480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:58.179 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.171+0000 7f97e4b06640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560bd746d2d0 0x560bd746d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:51918/0 (socket says 172.21.15.72:51918) 2024-03-22T01:36:58.179 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.171+0000 7f97e5307640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560bd746dbf0 0x560bd7476480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.72:53204/0 (socket says 172.21.15.72:53204) 2024-03-22T01:36:58.179 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.171+0000 7f97e4b06640 1 -- 172.21.15.72:0/3160486101 learned_addr learned my addr 172.21.15.72:0/3160486101 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:58.179 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.171+0000 7f97e5b08640 1 --2- 172.21.15.72:0/1661228648 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f97cc0a3110 0x7f97cc0a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:58.179 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.171+0000 7f97e4b06640 1 --2- 172.21.15.72:0/3160486101 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bd74769c0 0x560bd7478db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:58.179 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.171+0000 7f97d67fc640 1 -- 172.21.15.72:0/1661228648 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f97d80cfad0 con 0x7f97e0106a40 2024-03-22T01:36:58.179 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.171+0000 7f97e5307640 1 -- 172.21.15.72:0/3160486101 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bd74769c0 msgr2=0x560bd7478db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:36:58.179 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.171+0000 7f97e5307640 1 --2- 172.21.15.72:0/3160486101 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bd74769c0 0x560bd7478db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:58.179 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.171+0000 7f97e5307640 1 -- 172.21.15.72:0/3160486101 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560bd746d2d0 msgr2=0x560bd746d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:36:58.179 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.171+0000 7f97e5307640 1 --2- 172.21.15.72:0/3160486101 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560bd746d2d0 0x560bd746d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:58.179 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.171+0000 7f97e5307640 1 -- 172.21.15.72:0/3160486101 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560bd70e5210 con 0x560bd746dbf0 2024-03-22T01:36:58.180 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.171+0000 7f97e5b08640 1 --2- 172.21.15.72:0/1661228648 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f97cc0a3110 0x7f97cc0a55d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f97d80ccdf0 tx=0x7f97d806e660 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:58.180 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.171+0000 7f97e5307640 1 --2- 172.21.15.72:0/3160486101 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560bd746dbf0 0x560bd7476480 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f97dc009230 tx=0x7f97dc005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ebb9e6b844e8767 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:58.180 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.172+0000 7f97cb7fe640 1 -- 172.21.15.72:0/3160486101 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97dc042070 con 0x560bd746dbf0 2024-03-22T01:36:58.180 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.172+0000 7f97cb7fe640 1 -- 172.21.15.72:0/3160486101 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97dc004a70 con 0x560bd746dbf0 2024-03-22T01:36:58.180 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.172+0000 7f97cb7fe640 1 -- 172.21.15.72:0/3160486101 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97dc005130 con 0x560bd746dbf0 2024-03-22T01:36:58.180 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.172+0000 7f97ea22a740 1 -- 172.21.15.72:0/3160486101 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560bd746dbf0 msgr2=0x560bd7476480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:58.180 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.172+0000 7f97ea22a740 1 --2- 172.21.15.72:0/3160486101 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560bd746dbf0 0x560bd7476480 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f97dc009230 tx=0x7f97dc005d50 comp rx=0 tx=0).stop 2024-03-22T01:36:58.180 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.172+0000 7f97ea22a740 1 -- 172.21.15.72:0/3160486101 shutdown_connections 2024-03-22T01:36:58.180 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.172+0000 7f97ea22a740 1 --2- 172.21.15.72:0/3160486101 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560bd746dbf0 0x560bd7476480 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:58.180 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.172+0000 7f97ea22a740 1 --2- 172.21.15.72:0/3160486101 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bd74769c0 0x560bd7478db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:58.180 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.172+0000 7f97ea22a740 1 --2- 172.21.15.72:0/3160486101 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560bd746d2d0 0x560bd746d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:58.181 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.172+0000 7f97ea22a740 1 -- 172.21.15.72:0/3160486101 >> 172.21.15.72:0/3160486101 conn(0x560bd7467bf0 msgr2=0x560bd7467fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:58.181 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.172+0000 7f97ea22a740 1 -- 172.21.15.72:0/3160486101 shutdown_connections 2024-03-22T01:36:58.181 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.172+0000 7f97ea22a740 1 -- 172.21.15.72:0/3160486101 wait complete. 2024-03-22T01:36:58.181 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.173+0000 7f97ea22a740 1 -- start start 2024-03-22T01:36:58.181 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.173+0000 7f97ea22a740 1 Processor -- start 2024-03-22T01:36:58.181 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.173+0000 7f97ea22a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bd746d2d0 0x560bd7476310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:58.181 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.173+0000 7f97ea22a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560bd746dbf0 0x560bd74736a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:58.182 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.173+0000 7f97ea22a740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560bd74769c0 0x560bd7473c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:58.182 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.173+0000 7f97ea22a740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x560bd7479590 con 0x560bd74769c0 2024-03-22T01:36:58.182 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.173+0000 7f97ea22a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x560bd738cdb0 con 0x560bd746dbf0 2024-03-22T01:36:58.182 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.173+0000 7f97ea22a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560bd7479710 con 0x560bd746d2d0 2024-03-22T01:36:58.182 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.173+0000 7f97ea22a740 10 client.0 _ll_register_callbacks cb 0x560bd70e4a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:36:58.182 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.173+0000 7f97e4b06640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560bd74769c0 0x560bd7473c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:58.182 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.173+0000 7f97e5307640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560bd746dbf0 0x560bd74736a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:58.182 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.173+0000 7f97e4b06640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560bd74769c0 0x560bd7473c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:51924/0 (socket says 172.21.15.72:51924) 2024-03-22T01:36:58.182 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.173+0000 7f97e4b06640 1 -- 172.21.15.72:0/1276006339 learned_addr learned my addr 172.21.15.72:0/1276006339 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:58.182 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.173+0000 7f97e4b06640 1 --2- 172.21.15.72:0/1276006339 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bd746d2d0 0x560bd7476310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:58.182 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.174+0000 7f97e5307640 1 -- 172.21.15.72:0/1276006339 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bd746d2d0 msgr2=0x560bd7476310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:36:58.182 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.174+0000 7f97e5307640 1 --2- 172.21.15.72:0/1276006339 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bd746d2d0 0x560bd7476310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:58.182 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.174+0000 7f97e5307640 1 -- 172.21.15.72:0/1276006339 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560bd74769c0 msgr2=0x560bd7473c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:36:58.183 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.174+0000 7f97e5307640 1 --2- 172.21.15.72:0/1276006339 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560bd74769c0 0x560bd7473c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:58.183 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.174+0000 7f97e5307640 1 -- 172.21.15.72:0/1276006339 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560bd71b1620 con 0x560bd746dbf0 2024-03-22T01:36:58.183 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.174+0000 7f97e5307640 1 --2- 172.21.15.72:0/1276006339 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560bd746dbf0 0x560bd74736a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f97dc009900 tx=0x7f97dc02cb30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e506aa369d2ebb2 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:58.183 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.174+0000 7f97c8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-03-22T01:36:58.183 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.174+0000 7f97c8ff9640 1 -- 172.21.15.72:0/1276006339 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97dc042050 con 0x560bd746dbf0 2024-03-22T01:36:58.183 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.174+0000 7f97c8ff9640 1 -- 172.21.15.72:0/1276006339 <== mon.2 v2:172.21.15.89:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f97dc005140 con 0x560bd746dbf0 2024-03-22T01:36:58.183 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.174+0000 7f97ea22a740 1 -- 172.21.15.72:0/1276006339 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f97dc004a70 con 0x560bd746dbf0 2024-03-22T01:36:58.183 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.174+0000 7f97c8ff9640 1 -- 172.21.15.72:0/1276006339 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f97dc004c20 con 0x560bd746dbf0 2024-03-22T01:36:58.183 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.174+0000 7f97ea22a740 10 client.25490 fetch_fsmap learned FSMap version 21 2024-03-22T01:36:58.183 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.174+0000 7f97ea22a740 1 -- 172.21.15.72:0/1276006339 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x560bd7485500 con 0x560bd746dbf0 2024-03-22T01:36:58.183 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.174+0000 7f97c8ff9640 1 -- 172.21.15.72:0/1276006339 <== mon.2 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97dc004a70 con 0x560bd746dbf0 2024-03-22T01:36:58.183 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.175+0000 7f97c8ff9640 1 -- 172.21.15.72:0/1276006339 <== mon.2 v2:172.21.15.89:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f97dc003bc0 con 0x560bd746dbf0 2024-03-22T01:36:58.183 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.175+0000 7f97ea22a740 10 client.25490 fetch_fsmap finished waiting for FSMap version 21 2024-03-22T01:36:58.184 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.175+0000 7f97ea22a740 10 client.25490 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T01:36:58.184 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.175+0000 7f97ea22a740 20 client.25490 populate_metadata read hostname 'smithi072' 2024-03-22T01:36:58.184 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.175+0000 7f97ea22a740 1 --2- 172.21.15.72:0/1276006339 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x560bd7486850 0x560bd7488c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:58.184 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.175+0000 7f97ea22a740 4 client.25490 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T01:36:58.184 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.175+0000 7f97ea22a740 1 -- 172.21.15.72:0/1276006339 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560bd73de580 con 0x560bd7486850 2024-03-22T01:36:58.184 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.175+0000 7f97e4b06640 1 --2- 172.21.15.72:0/1276006339 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x560bd7486850 0x560bd7488c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:58.184 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.175+0000 7f97e4b06640 1 --2- 172.21.15.72:0/1276006339 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x560bd7486850 0x560bd7488c40 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d690a41309c89312 server_cookie=85a6da2303de10c in_seq=0 out_seq=0 2024-03-22T01:36:58.184 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.175+0000 7f97c8ff9640 10 client.25490 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:36:58.184 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.176+0000 7f97c8ff9640 1 -- 172.21.15.72:0/1276006339 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560bd73de580 con 0x560bd7486850 2024-03-22T01:36:58.184 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.176+0000 7f97c8ff9640 10 client.25490 handle_command_reply: tid=0 2024-03-22T01:36:58.184 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.176+0000 7f97ea22a740 1 client.25490 shutdown 2024-03-22T01:36:58.184 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.176+0000 7f97ea22a740 1 -- 172.21.15.72:0/1276006339 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560bd746dbf0 msgr2=0x560bd74736a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:58.184 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.176+0000 7f97ea22a740 1 --2- 172.21.15.72:0/1276006339 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560bd746dbf0 0x560bd74736a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f97dc009900 tx=0x7f97dc02cb30 comp rx=0 tx=0).stop 2024-03-22T01:36:58.184 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.176+0000 7f97ea22a740 1 -- 172.21.15.72:0/1276006339 shutdown_connections 2024-03-22T01:36:58.184 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.176+0000 7f97ea22a740 1 --2- 172.21.15.72:0/1276006339 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x560bd7486850 0x560bd7488c40 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:58.184 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.176+0000 7f97ea22a740 1 --2- 172.21.15.72:0/1276006339 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560bd746dbf0 0x560bd74736a0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:58.185 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.176+0000 7f97ea22a740 1 --2- 172.21.15.72:0/1276006339 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560bd74769c0 0x560bd7473c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:58.185 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.176+0000 7f97c8ff9640 0 client.25490 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:36:58.185 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.176+0000 7f97ea22a740 1 --2- 172.21.15.72:0/1276006339 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bd746d2d0 0x560bd7476310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:58.185 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.176+0000 7f97ea22a740 1 -- 172.21.15.72:0/1276006339 >> 172.21.15.72:0/1276006339 conn(0x560bd7467bf0 msgr2=0x560bd7469940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:58.185 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.176+0000 7f97ea22a740 1 -- 172.21.15.72:0/1276006339 shutdown_connections 2024-03-22T01:36:58.185 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.176+0000 7f97ea22a740 1 -- 172.21.15.72:0/1276006339 wait complete. 2024-03-22T01:36:58.185 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.176+0000 7f97ea22a740 20 client.25490 trim_cache size 0 max 16384 2024-03-22T01:36:58.195 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.191+0000 7f97ea22a740 1 Processor -- start 2024-03-22T01:36:58.195 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.191+0000 7f97ea22a740 1 -- start start 2024-03-22T01:36:58.195 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.191+0000 7f97ea22a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bd746d2d0 0x560bd7466ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:58.195 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.191+0000 7f97ea22a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560bd746dbf0 0x560bd7475dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:58.195 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.191+0000 7f97ea22a740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560bd74769c0 0x560bd7476310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:58.195 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.191+0000 7f97ea22a740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f97e01ad070 con 0x560bd74769c0 2024-03-22T01:36:58.196 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.191+0000 7f97ea22a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f97e01af8f0 con 0x560bd746dbf0 2024-03-22T01:36:58.196 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.191+0000 7f97ea22a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f97e01ad6e0 con 0x560bd746d2d0 2024-03-22T01:36:58.196 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.191+0000 7f97e4b06640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560bd74769c0 0x560bd7476310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:58.196 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.191+0000 7f97e4b06640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560bd74769c0 0x560bd7476310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:51932/0 (socket says 172.21.15.72:51932) 2024-03-22T01:36:58.196 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.191+0000 7f97e4b06640 1 -- 172.21.15.72:0/1483276756 learned_addr learned my addr 172.21.15.72:0/1483276756 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:58.196 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.191+0000 7f97e5307640 1 --2- 172.21.15.72:0/1483276756 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560bd746dbf0 0x560bd7475dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:58.196 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.191+0000 7f97e4b06640 1 --2- 172.21.15.72:0/1483276756 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bd746d2d0 0x560bd7466ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:58.196 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.191+0000 7f97e4b06640 1 -- 172.21.15.72:0/1483276756 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bd746d2d0 msgr2=0x560bd7466ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:36:58.196 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.191+0000 7f97e4b06640 1 --2- 172.21.15.72:0/1483276756 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bd746d2d0 0x560bd7466ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:58.196 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.191+0000 7f97e4b06640 1 -- 172.21.15.72:0/1483276756 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560bd746dbf0 msgr2=0x560bd7475dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:36:58.196 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.191+0000 7f97e4b06640 1 --2- 172.21.15.72:0/1483276756 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560bd746dbf0 0x560bd7475dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:58.196 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.191+0000 7f97e4b06640 1 -- 172.21.15.72:0/1483276756 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560bd7471bf0 con 0x560bd74769c0 2024-03-22T01:36:58.196 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.192+0000 7f97e5307640 1 --2- 172.21.15.72:0/1483276756 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560bd746dbf0 0x560bd7475dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:36:58.196 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.192+0000 7f97e4b06640 1 --2- 172.21.15.72:0/1483276756 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560bd74769c0 0x560bd7476310 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x560bd7379b30 tx=0x7f97d00096e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7447c4275620a970 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:58.197 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.192+0000 7f97caffd640 1 -- 172.21.15.72:0/1483276756 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97d001d040 con 0x560bd74769c0 2024-03-22T01:36:58.197 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.192+0000 7f97caffd640 1 -- 172.21.15.72:0/1483276756 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97d00035f0 con 0x560bd74769c0 2024-03-22T01:36:58.197 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.192+0000 7f97caffd640 1 -- 172.21.15.72:0/1483276756 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97d0027350 con 0x560bd74769c0 2024-03-22T01:36:58.197 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.192+0000 7f97ea22a740 1 -- 172.21.15.72:0/1483276756 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560bd74769c0 msgr2=0x560bd7476310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:58.197 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.192+0000 7f97ea22a740 1 --2- 172.21.15.72:0/1483276756 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560bd74769c0 0x560bd7476310 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x560bd7379b30 tx=0x7f97d00096e0 comp rx=0 tx=0).stop 2024-03-22T01:36:58.197 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.192+0000 7f97ea22a740 1 -- 172.21.15.72:0/1483276756 shutdown_connections 2024-03-22T01:36:58.197 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.192+0000 7f97ea22a740 1 --2- 172.21.15.72:0/1483276756 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560bd746dbf0 0x560bd7475dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:58.197 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.192+0000 7f97ea22a740 1 --2- 172.21.15.72:0/1483276756 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560bd74769c0 0x560bd7476310 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:58.197 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.192+0000 7f97ea22a740 1 --2- 172.21.15.72:0/1483276756 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bd746d2d0 0x560bd7466ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:58.197 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.192+0000 7f97ea22a740 1 -- 172.21.15.72:0/1483276756 >> 172.21.15.72:0/1483276756 conn(0x560bd7467bf0 msgr2=0x560bd74663c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:58.197 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.193+0000 7f97ea22a740 1 -- 172.21.15.72:0/1483276756 shutdown_connections 2024-03-22T01:36:58.197 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.193+0000 7f97ea22a740 1 -- 172.21.15.72:0/1483276756 wait complete. 2024-03-22T01:36:58.197 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.193+0000 7f97ea22a740 1 -- start start 2024-03-22T01:36:58.197 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.193+0000 7f97ea22a740 1 Processor -- start 2024-03-22T01:36:58.198 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.193+0000 7f97ea22a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bd746d2d0 0x560bd7489840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:58.198 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.193+0000 7f97ea22a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560bd746dbf0 0x560bd7489d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:58.198 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.193+0000 7f97ea22a740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560bd74769c0 0x560bd748a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:58.198 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.193+0000 7f97ea22a740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f97e01addb0 con 0x560bd74769c0 2024-03-22T01:36:58.198 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.193+0000 7f97ea22a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f97e01af220 con 0x560bd746dbf0 2024-03-22T01:36:58.198 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.193+0000 7f97ea22a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f97e01ae480 con 0x560bd746d2d0 2024-03-22T01:36:58.198 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.193+0000 7f97ea22a740 10 client.0 _ll_register_callbacks cb 0x7f97d0009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:36:58.198 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.193+0000 7f97e4b06640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560bd74769c0 0x560bd748a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:58.198 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.193+0000 7f97e5307640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560bd746dbf0 0x560bd7489d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:58.198 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.193+0000 7f97e5307640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560bd746dbf0 0x560bd7489d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.72:53252/0 (socket says 172.21.15.72:53252) 2024-03-22T01:36:58.198 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.193+0000 7f97e4b06640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560bd74769c0 0x560bd748a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:51934/0 (socket says 172.21.15.72:51934) 2024-03-22T01:36:58.198 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.194+0000 7f97e5307640 1 -- 172.21.15.72:0/3314231215 learned_addr learned my addr 172.21.15.72:0/3314231215 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:36:58.198 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.194+0000 7f97e4b06640 1 --2- 172.21.15.72:0/3314231215 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bd746d2d0 0x560bd7489840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:58.199 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.194+0000 7f97e5307640 1 -- 172.21.15.72:0/3314231215 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bd746d2d0 msgr2=0x560bd7489840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:36:58.199 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.194+0000 7f97e5307640 1 --2- 172.21.15.72:0/3314231215 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bd746d2d0 0x560bd7489840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:58.199 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.194+0000 7f97e5307640 1 -- 172.21.15.72:0/3314231215 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560bd74769c0 msgr2=0x560bd748a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:36:58.199 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.194+0000 7f97e5307640 1 --2- 172.21.15.72:0/3314231215 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560bd74769c0 0x560bd748a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:58.199 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.194+0000 7f97e5307640 1 -- 172.21.15.72:0/3314231215 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560bd74eba40 con 0x560bd746dbf0 2024-03-22T01:36:58.199 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.194+0000 7f97e4b06640 1 --2- 172.21.15.72:0/3314231215 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bd746d2d0 0x560bd7489840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:36:58.199 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.194+0000 7f97e5307640 1 --2- 172.21.15.72:0/3314231215 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560bd746dbf0 0x560bd7489d80 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f97dc000c00 tx=0x7f97dc02ca90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf4495e4028a6500 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:36:58.199 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.194+0000 7f97c8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-03-22T01:36:58.199 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.194+0000 7f97c8ff9640 1 -- 172.21.15.72:0/3314231215 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97dc004340 con 0x560bd746dbf0 2024-03-22T01:36:58.199 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.194+0000 7f97c8ff9640 1 -- 172.21.15.72:0/3314231215 <== mon.2 v2:172.21.15.89:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f97dc004a70 con 0x560bd746dbf0 2024-03-22T01:36:58.199 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.194+0000 7f97ea22a740 1 -- 172.21.15.72:0/3314231215 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f97d00035f0 con 0x560bd746dbf0 2024-03-22T01:36:58.199 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.194+0000 7f97c8ff9640 1 -- 172.21.15.72:0/3314231215 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f97dc0349b0 con 0x560bd746dbf0 2024-03-22T01:36:58.199 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.194+0000 7f97ea22a740 10 client.25496 fetch_fsmap learned FSMap version 21 2024-03-22T01:36:58.199 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.194+0000 7f97ea22a740 1 -- 172.21.15.72:0/3314231215 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x560bd74ecd50 con 0x560bd746dbf0 2024-03-22T01:36:58.200 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.194+0000 7f97c8ff9640 1 -- 172.21.15.72:0/3314231215 <== mon.2 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97d00035f0 con 0x560bd746dbf0 2024-03-22T01:36:58.200 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.195+0000 7f97c8ff9640 1 -- 172.21.15.72:0/3314231215 <== mon.2 v2:172.21.15.89:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f97dc0035f0 con 0x560bd746dbf0 2024-03-22T01:36:58.200 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.195+0000 7f97ea22a740 10 client.25496 fetch_fsmap finished waiting for FSMap version 21 2024-03-22T01:36:58.200 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.195+0000 7f97ea22a740 10 client.25496 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T01:36:58.200 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.195+0000 7f97ea22a740 20 client.25496 populate_metadata read hostname 'smithi072' 2024-03-22T01:36:58.200 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.195+0000 7f97ea22a740 1 --2- 172.21.15.72:0/3314231215 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x560bd74ed1e0 0x560bd74ef600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:36:58.200 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.195+0000 7f97ea22a740 4 client.25496 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T01:36:58.200 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.195+0000 7f97ea22a740 1 -- 172.21.15.72:0/3314231215 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560bd73d88d0 con 0x560bd74ed1e0 2024-03-22T01:36:58.200 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.195+0000 7f97e4b06640 1 --2- 172.21.15.72:0/3314231215 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x560bd74ed1e0 0x560bd74ef600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:36:58.200 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.195+0000 7f97c8ff9640 10 client.25496 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:36:58.200 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.195+0000 7f97e4b06640 1 --2- 172.21.15.72:0/3314231215 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x560bd74ed1e0 0x560bd74ef600 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f8ef1d48512ab11 server_cookie=ebc96b81ba401bed in_seq=0 out_seq=0 2024-03-22T01:36:58.206 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.206+0000 7f97c8ff9640 1 -- 172.21.15.72:0/3314231215 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560bd73d88d0 con 0x560bd74ed1e0 2024-03-22T01:36:58.206 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.207+0000 7f97c8ff9640 10 client.25496 handle_command_reply: tid=0 2024-03-22T01:36:58.206 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.207+0000 7f97ea22a740 1 client.25496 shutdown 2024-03-22T01:36:58.206 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.207+0000 7f97ea22a740 1 -- 172.21.15.72:0/3314231215 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560bd746dbf0 msgr2=0x560bd7489d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:58.207 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.207+0000 7f97ea22a740 1 --2- 172.21.15.72:0/3314231215 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560bd746dbf0 0x560bd7489d80 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f97dc000c00 tx=0x7f97dc02ca90 comp rx=0 tx=0).stop 2024-03-22T01:36:58.207 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.207+0000 7f97ea22a740 1 -- 172.21.15.72:0/3314231215 shutdown_connections 2024-03-22T01:36:58.207 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.207+0000 7f97ea22a740 1 --2- 172.21.15.72:0/3314231215 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x560bd74ed1e0 0x560bd74ef600 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:58.207 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.207+0000 7f97ea22a740 1 --2- 172.21.15.72:0/3314231215 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560bd746dbf0 0x560bd7489d80 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:58.207 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.207+0000 7f97ea22a740 1 --2- 172.21.15.72:0/3314231215 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560bd74769c0 0x560bd748a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:58.207 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.207+0000 7f97c8ff9640 0 client.25496 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:36:58.207 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.207+0000 7f97ea22a740 1 --2- 172.21.15.72:0/3314231215 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bd746d2d0 0x560bd7489840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:58.207 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.207+0000 7f97ea22a740 1 -- 172.21.15.72:0/3314231215 >> 172.21.15.72:0/3314231215 conn(0x560bd7467bf0 msgr2=0x560bd74821f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:58.207 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.207+0000 7f97ea22a740 1 -- 172.21.15.72:0/3314231215 shutdown_connections 2024-03-22T01:36:58.207 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.207+0000 7f97ea22a740 1 -- 172.21.15.72:0/3314231215 wait complete. 2024-03-22T01:36:58.207 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.207+0000 7f97ea22a740 20 client.25496 trim_cache size 0 max 16384 2024-03-22T01:36:58.208 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout:{ 2024-03-22T01:36:58.208 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout: "message": "", 2024-03-22T01:36:58.208 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout: "return_code": 0 2024-03-22T01:36:58.208 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout:} 2024-03-22T01:36:58.208 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.208+0000 7f97e756b640 1 -- 172.21.15.72:0/1661228648 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f97cc0a3110 msgr2=0x7f97cc0a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:58.209 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.208+0000 7f97e756b640 1 --2- 172.21.15.72:0/1661228648 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f97cc0a3110 0x7f97cc0a55d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f97d80ccdf0 tx=0x7f97d806e660 comp rx=0 tx=0).stop 2024-03-22T01:36:58.209 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.208+0000 7f97e756b640 1 -- 172.21.15.72:0/1661228648 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f97e0106a40 msgr2=0x7f97e01a4320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:36:58.209 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.208+0000 7f97e756b640 1 --2- 172.21.15.72:0/1661228648 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f97e0106a40 0x7f97e01a4320 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f97d8009fd0 tx=0x7f97d800b260 comp rx=0 tx=0).stop 2024-03-22T01:36:58.209 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.208+0000 7f97e756b640 1 -- 172.21.15.72:0/1661228648 shutdown_connections 2024-03-22T01:36:58.209 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.208+0000 7f97e756b640 1 --2- 172.21.15.72:0/1661228648 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f97cc0a3110 0x7f97cc0a55d0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:58.209 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.208+0000 7f97e756b640 1 --2- 172.21.15.72:0/1661228648 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f97e0106a40 0x7f97e01a4320 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:58.209 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.208+0000 7f97e756b640 1 --2- 172.21.15.72:0/1661228648 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97e01013f0 0x7f97e01997e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:58.209 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.208+0000 7f97e756b640 1 --2- 172.21.15.72:0/1661228648 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97e0100a30 0x7f97e01992a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:36:58.209 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.208+0000 7f97e756b640 1 -- 172.21.15.72:0/1661228648 >> 172.21.15.72:0/1661228648 conn(0x7f97e0077b80 msgr2=0x7f97e0103760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:36:58.209 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.208+0000 7f97e756b640 1 -- 172.21.15.72:0/1661228648 shutdown_connections 2024-03-22T01:36:58.209 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:36:58.208+0000 7f97e756b640 1 -- 172.21.15.72:0/1661228648 wait complete. 2024-03-22T01:36:58.357 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:57 smithi089 ceph-mon[35770]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 77 KiB/s wr, 44 op/s 2024-03-22T01:37:00.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:36:59 smithi072 ceph-mon[32414]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 16 op/s 2024-03-22T01:37:00.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:36:59 smithi098 ceph-mon[36468]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 16 op/s 2024-03-22T01:37:00.357 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:36:59 smithi089 ceph-mon[35770]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 16 op/s 2024-03-22T01:37:00.622 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:37:00 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:37:00.256+0000 7fa4535f4700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-22T01:37:00.622 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:37:00 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:37:00.256+0000 7fa4535f4700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-22T01:37:00.622 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:37:00 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:37:00.256+0000 7fa4535f4700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-22T01:37:00.623 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:37:00 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:37:00.256+0000 7fa4535f4700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-22T01:37:00.623 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:37:00 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:37:00.256+0000 7fa4535f4700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-22T01:37:02.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:37:01 smithi098 ceph-mon[36468]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 13 KiB/s wr, 21 op/s 2024-03-22T01:37:02.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:37:01 smithi098 ceph-mon[36468]: mgrmap e24: x(active, since 22s), standbys: y, z 2024-03-22T01:37:02.357 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:37:01 smithi089 ceph-mon[35770]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 13 KiB/s wr, 21 op/s 2024-03-22T01:37:02.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:37:01 smithi089 ceph-mon[35770]: mgrmap e24: x(active, since 22s), standbys: y, z 2024-03-22T01:37:02.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:37:01 smithi072 ceph-mon[32414]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 13 KiB/s wr, 21 op/s 2024-03-22T01:37:02.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:37:01 smithi072 ceph-mon[32414]: mgrmap e24: x(active, since 22s), standbys: y, z 2024-03-22T01:37:04.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:37:03 smithi098 ceph-mon[36468]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.6 KiB/s wr, 6 op/s 2024-03-22T01:37:04.357 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:37:03 smithi089 ceph-mon[35770]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.6 KiB/s wr, 6 op/s 2024-03-22T01:37:04.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:37:03 smithi072 ceph-mon[32414]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.6 KiB/s wr, 6 op/s 2024-03-22T01:37:06.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:37:05 smithi098 ceph-mon[36468]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 12 KiB/s wr, 19 op/s 2024-03-22T01:37:06.357 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:37:05 smithi089 ceph-mon[35770]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 12 KiB/s wr, 19 op/s 2024-03-22T01:37:06.374 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:37:05 smithi072 ceph-mon[32414]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 12 KiB/s wr, 19 op/s 2024-03-22T01:37:08.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:37:07 smithi098 ceph-mon[36468]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-03-22T01:37:08.357 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:37:07 smithi089 ceph-mon[35770]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-03-22T01:37:08.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:37:07 smithi072 ceph-mon[32414]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-03-22T01:37:10.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:37:09 smithi098 ceph-mon[36468]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 24 KiB/s wr, 21 op/s 2024-03-22T01:37:10.357 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:37:09 smithi089 ceph-mon[35770]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 24 KiB/s wr, 21 op/s 2024-03-22T01:37:10.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:37:09 smithi072 ceph-mon[32414]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 24 KiB/s wr, 21 op/s 2024-03-22T01:37:11.305 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.303+0000 7f425d732640 1 -- 172.21.15.72:0/3951623990 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4258107c10 msgr2=0x7f4258107ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:11.305 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.303+0000 7f425d732640 1 --2- 172.21.15.72:0/3951623990 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4258107c10 0x7f4258107ff0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f4248009900 tx=0x7f424802bc30 comp rx=0 tx=0).stop 2024-03-22T01:37:11.305 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.303+0000 7f425d732640 1 -- 172.21.15.72:0/3951623990 shutdown_connections 2024-03-22T01:37:11.306 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.303+0000 7f425d732640 1 --2- 172.21.15.72:0/3951623990 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4258101b50 0x7f4258101fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:11.306 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.303+0000 7f425d732640 1 --2- 172.21.15.72:0/3951623990 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4258102510 0x7f425810e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:11.306 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.303+0000 7f425d732640 1 --2- 172.21.15.72:0/3951623990 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4258107c10 0x7f4258107ff0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:11.306 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.303+0000 7f425d732640 1 -- 172.21.15.72:0/3951623990 >> 172.21.15.72:0/3951623990 conn(0x7f42580fd5e0 msgr2=0x7f42580ffa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:11.306 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.303+0000 7f425d732640 1 -- 172.21.15.72:0/3951623990 shutdown_connections 2024-03-22T01:37:11.306 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.304+0000 7f425d732640 1 -- 172.21.15.72:0/3951623990 wait complete. 2024-03-22T01:37:11.306 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.304+0000 7f425d732640 1 Processor -- start 2024-03-22T01:37:11.306 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.304+0000 7f425d732640 1 -- start start 2024-03-22T01:37:11.306 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.304+0000 7f425d732640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4258101b50 0x7f4258198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:11.306 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.304+0000 7f425d732640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4258102510 0x7f4258198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:11.306 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.304+0000 7f425d732640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4258107c10 0x7f425819cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:11.306 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.304+0000 7f425d732640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f4258111f50 con 0x7f4258107c10 2024-03-22T01:37:11.307 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.304+0000 7f425d732640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4258111dd0 con 0x7f4258101b50 2024-03-22T01:37:11.307 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.304+0000 7f425d732640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f42581120d0 con 0x7f4258102510 2024-03-22T01:37:11.307 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.304+0000 7f42577fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4258107c10 0x7f425819cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:11.307 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.304+0000 7f42577fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4258107c10 0x7f425819cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59298/0 (socket says 172.21.15.72:59298) 2024-03-22T01:37:11.307 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.304+0000 7f42577fe640 1 -- 172.21.15.72:0/4060369308 learned_addr learned my addr 172.21.15.72:0/4060369308 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:37:11.307 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.304+0000 7f4256ffd640 1 --2- 172.21.15.72:0/4060369308 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4258101b50 0x7f4258198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:11.307 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.305+0000 7f424ffff640 1 --2- 172.21.15.72:0/4060369308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4258102510 0x7f4258198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:11.307 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.305+0000 7f42577fe640 1 -- 172.21.15.72:0/4060369308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4258102510 msgr2=0x7f4258198950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:11.307 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.305+0000 7f42577fe640 1 --2- 172.21.15.72:0/4060369308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4258102510 0x7f4258198950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:11.307 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.305+0000 7f42577fe640 1 -- 172.21.15.72:0/4060369308 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4258101b50 msgr2=0x7f4258198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:11.307 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.305+0000 7f42577fe640 1 --2- 172.21.15.72:0/4060369308 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4258101b50 0x7f4258198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:11.307 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.305+0000 7f42577fe640 1 -- 172.21.15.72:0/4060369308 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f425819d3b0 con 0x7f4258107c10 2024-03-22T01:37:11.308 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.305+0000 7f42577fe640 1 --2- 172.21.15.72:0/4060369308 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4258107c10 0x7f425819cc80 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f425000cad0 tx=0x7f425000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:11.308 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.305+0000 7f4254ff9640 1 -- 172.21.15.72:0/4060369308 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f425000ae20 con 0x7f4258107c10 2024-03-22T01:37:11.308 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.305+0000 7f4254ff9640 1 -- 172.21.15.72:0/4060369308 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4250005020 con 0x7f4258107c10 2024-03-22T01:37:11.308 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.305+0000 7f425d732640 1 -- 172.21.15.72:0/4060369308 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f425819d6a0 con 0x7f4258107c10 2024-03-22T01:37:11.308 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.305+0000 7f4254ff9640 1 -- 172.21.15.72:0/4060369308 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f425000f6f0 con 0x7f4258107c10 2024-03-22T01:37:11.308 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.305+0000 7f425d732640 1 -- 172.21.15.72:0/4060369308 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42581a4f30 con 0x7f4258107c10 2024-03-22T01:37:11.315 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.307+0000 7f4254ff9640 1 -- 172.21.15.72:0/4060369308 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 24) ==== 138087+0+0 (secure 0 0 0) 0x7f42500051d0 con 0x7f4258107c10 2024-03-22T01:37:11.315 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.308+0000 7f42603f1740 1 Processor -- start 2024-03-22T01:37:11.315 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.308+0000 7f42603f1740 1 -- start start 2024-03-22T01:37:11.315 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.308+0000 7f4254ff9640 1 --2- 172.21.15.72:0/4060369308 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f42440a3220 0x7f42440a56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:11.315 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.308+0000 7f424ffff640 1 --2- 172.21.15.72:0/4060369308 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f42440a3220 0x7f42440a56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:11.315 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.308+0000 7f42603f1740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a165c742d0 0x55a165c746b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:11.316 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.308+0000 7f42603f1740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a165c74bf0 0x55a165c7d480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:11.316 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.308+0000 7f42603f1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a165c7d9c0 0x55a165c7fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:11.316 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.308+0000 7f42603f1740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55a165bd8040 con 0x55a165c74bf0 2024-03-22T01:37:11.316 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.308+0000 7f4254ff9640 1 -- 172.21.15.72:0/4060369308 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f42500d01e0 con 0x7f4258107c10 2024-03-22T01:37:11.316 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.308+0000 7f42603f1740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55a165b809e0 con 0x55a165c742d0 2024-03-22T01:37:11.316 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.308+0000 7f42603f1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a165bd3ff0 con 0x55a165c7d9c0 2024-03-22T01:37:11.316 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.309+0000 7f4256ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a165c742d0 0x55a165c746b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:11.316 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.309+0000 7f4256ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a165c742d0 0x55a165c746b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.72:33770/0 (socket says 172.21.15.72:33770) 2024-03-22T01:37:11.316 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.309+0000 7f4256ffd640 1 -- 172.21.15.72:0/935863930 learned_addr learned my addr 172.21.15.72:0/935863930 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:37:11.316 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.309+0000 7f424ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a165c74bf0 0x55a165c7d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:11.316 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.309+0000 7f42577fe640 1 --2- 172.21.15.72:0/935863930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a165c7d9c0 0x55a165c7fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:11.316 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.309+0000 7f424ffff640 1 --2- 172.21.15.72:0/4060369308 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f42440a3220 0x7f42440a56e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f42580fe590 tx=0x7f42400113e0 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:11.316 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.309+0000 7f4256ffd640 1 -- 172.21.15.72:0/935863930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a165c7d9c0 msgr2=0x55a165c7fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:11.316 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.309+0000 7f4256ffd640 1 --2- 172.21.15.72:0/935863930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a165c7d9c0 0x55a165c7fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:11.317 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.309+0000 7f4256ffd640 1 -- 172.21.15.72:0/935863930 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a165c74bf0 msgr2=0x55a165c7d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:11.317 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.309+0000 7f4256ffd640 1 --2- 172.21.15.72:0/935863930 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a165c74bf0 0x55a165c7d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:11.317 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.309+0000 7f4256ffd640 1 -- 172.21.15.72:0/935863930 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a1658ec210 con 0x55a165c742d0 2024-03-22T01:37:11.317 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.309+0000 7f4256ffd640 1 --2- 172.21.15.72:0/935863930 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a165c742d0 0x55a165c746b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4248009900 tx=0x7f424802c5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=974d7d24637240cb server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:11.317 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.309+0000 7f424d7fa640 1 -- 172.21.15.72:0/935863930 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4248042070 con 0x55a165c742d0 2024-03-22T01:37:11.317 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.309+0000 7f424d7fa640 1 -- 172.21.15.72:0/935863930 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4248004240 con 0x55a165c742d0 2024-03-22T01:37:11.317 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.309+0000 7f424d7fa640 1 -- 172.21.15.72:0/935863930 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4248004ec0 con 0x55a165c742d0 2024-03-22T01:37:11.317 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.310+0000 7f42603f1740 1 -- 172.21.15.72:0/935863930 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a165c742d0 msgr2=0x55a165c746b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:11.317 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.310+0000 7f42603f1740 1 --2- 172.21.15.72:0/935863930 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a165c742d0 0x55a165c746b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4248009900 tx=0x7f424802c5d0 comp rx=0 tx=0).stop 2024-03-22T01:37:11.317 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.310+0000 7f42603f1740 1 -- 172.21.15.72:0/935863930 shutdown_connections 2024-03-22T01:37:11.317 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.310+0000 7f42603f1740 1 --2- 172.21.15.72:0/935863930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a165c7d9c0 0x55a165c7fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:11.317 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.310+0000 7f42603f1740 1 --2- 172.21.15.72:0/935863930 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a165c74bf0 0x55a165c7d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:11.317 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.310+0000 7f42603f1740 1 --2- 172.21.15.72:0/935863930 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a165c742d0 0x55a165c746b0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:11.317 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.310+0000 7f42603f1740 1 -- 172.21.15.72:0/935863930 >> 172.21.15.72:0/935863930 conn(0x55a165c6ebf0 msgr2=0x55a165c6efd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:11.318 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.310+0000 7f42603f1740 1 -- 172.21.15.72:0/935863930 shutdown_connections 2024-03-22T01:37:11.318 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.310+0000 7f42603f1740 1 -- 172.21.15.72:0/935863930 wait complete. 2024-03-22T01:37:11.318 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.310+0000 7f42603f1740 1 -- start start 2024-03-22T01:37:11.318 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.311+0000 7f42603f1740 1 Processor -- start 2024-03-22T01:37:11.318 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.311+0000 7f42603f1740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a165c742d0 0x55a165c7d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:11.318 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.311+0000 7f42603f1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a165c74bf0 0x55a165c7a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:11.318 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.311+0000 7f42603f1740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a165c7d9c0 0x55a165c7ab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:11.318 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.311+0000 7f42603f1740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55a165c80590 con 0x55a165c7d9c0 2024-03-22T01:37:11.318 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.311+0000 7f42603f1740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55a165b93db0 con 0x55a165c742d0 2024-03-22T01:37:11.318 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.311+0000 7f42603f1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a165c80710 con 0x55a165c74bf0 2024-03-22T01:37:11.318 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.311+0000 7f42603f1740 10 client.0 _ll_register_callbacks cb 0x55a1658eba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:37:11.318 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.311+0000 7f4256ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a165c742d0 0x55a165c7d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:11.318 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.311+0000 7f4256ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a165c742d0 0x55a165c7d2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.72:33786/0 (socket says 172.21.15.72:33786) 2024-03-22T01:37:11.318 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.311+0000 7f4256ffd640 1 -- 172.21.15.72:0/2407445987 learned_addr learned my addr 172.21.15.72:0/2407445987 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:37:11.319 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.311+0000 7f42577fe640 1 --2- 172.21.15.72:0/2407445987 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a165c7d9c0 0x55a165c7ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:11.319 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.311+0000 7f424ffff640 1 --2- 172.21.15.72:0/2407445987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a165c74bf0 0x55a165c7a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:11.319 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.311+0000 7f4256ffd640 1 -- 172.21.15.72:0/2407445987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a165c74bf0 msgr2=0x55a165c7a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:11.319 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.311+0000 7f4256ffd640 1 --2- 172.21.15.72:0/2407445987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a165c74bf0 0x55a165c7a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:11.319 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.311+0000 7f4256ffd640 1 -- 172.21.15.72:0/2407445987 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a165c7d9c0 msgr2=0x55a165c7ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:11.319 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.311+0000 7f4256ffd640 1 --2- 172.21.15.72:0/2407445987 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a165c7d9c0 0x55a165c7ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:11.319 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.311+0000 7f4256ffd640 1 -- 172.21.15.72:0/2407445987 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a1659b8620 con 0x55a165c742d0 2024-03-22T01:37:11.319 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.311+0000 7f4256ffd640 1 --2- 172.21.15.72:0/2407445987 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a165c742d0 0x55a165c7d2c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f424802f020 tx=0x7f4248005b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a12ee4da16ac678a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:11.319 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.311+0000 7f423e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-03-22T01:37:11.320 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.312+0000 7f423e7fc640 1 -- 172.21.15.72:0/2407445987 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f424804f060 con 0x55a165c742d0 2024-03-22T01:37:11.320 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.312+0000 7f423e7fc640 1 -- 172.21.15.72:0/2407445987 <== mon.2 v2:172.21.15.89:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f42480050c0 con 0x55a165c742d0 2024-03-22T01:37:11.320 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.312+0000 7f42603f1740 1 -- 172.21.15.72:0/2407445987 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4248004240 con 0x55a165c742d0 2024-03-22T01:37:11.320 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.312+0000 7f423e7fc640 1 -- 172.21.15.72:0/2407445987 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f4248056050 con 0x55a165c742d0 2024-03-22T01:37:11.320 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.312+0000 7f42603f1740 10 client.25514 fetch_fsmap learned FSMap version 21 2024-03-22T01:37:11.320 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.312+0000 7f42603f1740 1 -- 172.21.15.72:0/2407445987 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55a165c8c460 con 0x55a165c742d0 2024-03-22T01:37:11.320 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.312+0000 7f423e7fc640 1 -- 172.21.15.72:0/2407445987 <== mon.2 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4248004240 con 0x55a165c742d0 2024-03-22T01:37:11.320 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.312+0000 7f423e7fc640 1 -- 172.21.15.72:0/2407445987 <== mon.2 v2:172.21.15.89:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f4248047b50 con 0x55a165c742d0 2024-03-22T01:37:11.320 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.312+0000 7f42603f1740 10 client.25514 fetch_fsmap finished waiting for FSMap version 21 2024-03-22T01:37:11.320 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.312+0000 7f42603f1740 10 client.25514 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T01:37:11.321 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.312+0000 7f42603f1740 20 client.25514 populate_metadata read hostname 'smithi072' 2024-03-22T01:37:11.321 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.312+0000 7f42603f1740 1 --2- 172.21.15.72:0/2407445987 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55a165c8d7b0 0x55a165c8fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:11.321 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.312+0000 7f42603f1740 4 client.25514 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T01:37:11.321 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.312+0000 7f42603f1740 1 -- 172.21.15.72:0/2407445987 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a165be5580 con 0x55a165c8d7b0 2024-03-22T01:37:11.321 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.312+0000 7f424ffff640 1 --2- 172.21.15.72:0/2407445987 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55a165c8d7b0 0x55a165c8fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:11.321 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.313+0000 7f424ffff640 1 --2- 172.21.15.72:0/2407445987 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55a165c8d7b0 0x55a165c8fba0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ea95e57783c43dce server_cookie=55c9e6269d239939 in_seq=0 out_seq=0 2024-03-22T01:37:11.321 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.313+0000 7f423e7fc640 10 client.25514 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:37:11.321 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.313+0000 7f423e7fc640 1 -- 172.21.15.72:0/2407445987 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a165be5580 con 0x55a165c8d7b0 2024-03-22T01:37:11.321 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.313+0000 7f423e7fc640 10 client.25514 handle_command_reply: tid=0 2024-03-22T01:37:11.321 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.313+0000 7f42603f1740 1 client.25514 shutdown 2024-03-22T01:37:11.321 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.314+0000 7f42603f1740 1 -- 172.21.15.72:0/2407445987 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a165c742d0 msgr2=0x55a165c7d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:11.322 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.314+0000 7f42603f1740 1 --2- 172.21.15.72:0/2407445987 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a165c742d0 0x55a165c7d2c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f424802f020 tx=0x7f4248005b50 comp rx=0 tx=0).stop 2024-03-22T01:37:11.322 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.314+0000 7f42603f1740 1 -- 172.21.15.72:0/2407445987 shutdown_connections 2024-03-22T01:37:11.322 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.314+0000 7f42603f1740 1 --2- 172.21.15.72:0/2407445987 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55a165c8d7b0 0x55a165c8fba0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:11.322 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.314+0000 7f42603f1740 1 --2- 172.21.15.72:0/2407445987 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a165c7d9c0 0x55a165c7ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:11.322 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.314+0000 7f42603f1740 1 --2- 172.21.15.72:0/2407445987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a165c74bf0 0x55a165c7a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:11.322 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.314+0000 7f423e7fc640 0 client.25514 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:37:11.322 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.314+0000 7f42603f1740 1 --2- 172.21.15.72:0/2407445987 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a165c742d0 0x55a165c7d2c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:11.322 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.314+0000 7f42603f1740 1 -- 172.21.15.72:0/2407445987 >> 172.21.15.72:0/2407445987 conn(0x55a165c6ebf0 msgr2=0x55a165c70940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:11.322 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.314+0000 7f42603f1740 1 -- 172.21.15.72:0/2407445987 shutdown_connections 2024-03-22T01:37:11.322 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.314+0000 7f42603f1740 1 -- 172.21.15.72:0/2407445987 wait complete. 2024-03-22T01:37:11.322 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.314+0000 7f42603f1740 20 client.25514 trim_cache size 0 max 16384 2024-03-22T01:37:11.334 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.328+0000 7f42603f1740 1 Processor -- start 2024-03-22T01:37:11.334 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.328+0000 7f42603f1740 1 -- start start 2024-03-22T01:37:11.334 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.329+0000 7f42603f1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a165c6ebf0 0x55a165c89330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:11.334 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.329+0000 7f42603f1740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a165c742d0 0x55a165c95140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:11.334 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.329+0000 7f42603f1740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a165c74bf0 0x55a165c7cd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:11.334 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.329+0000 7f42603f1740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f42581acfe0 con 0x55a165c742d0 2024-03-22T01:37:11.334 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.329+0000 7f42603f1740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f42581af860 con 0x55a165c74bf0 2024-03-22T01:37:11.335 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.329+0000 7f42603f1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f42581ad650 con 0x55a165c6ebf0 2024-03-22T01:37:11.335 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.329+0000 7f424ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a165c742d0 0x55a165c95140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:11.335 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.329+0000 7f424ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a165c742d0 0x55a165c95140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59324/0 (socket says 172.21.15.72:59324) 2024-03-22T01:37:11.335 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.329+0000 7f424ffff640 1 -- 172.21.15.72:0/778712429 learned_addr learned my addr 172.21.15.72:0/778712429 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:37:11.335 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.329+0000 7f42577fe640 1 --2- 172.21.15.72:0/778712429 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a165c74bf0 0x55a165c7cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:11.335 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.329+0000 7f4256ffd640 1 --2- 172.21.15.72:0/778712429 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a165c6ebf0 0x55a165c89330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:11.335 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.329+0000 7f424ffff640 1 -- 172.21.15.72:0/778712429 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a165c6ebf0 msgr2=0x55a165c89330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:11.335 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.329+0000 7f424ffff640 1 --2- 172.21.15.72:0/778712429 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a165c6ebf0 0x55a165c89330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:11.335 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.329+0000 7f424ffff640 1 -- 172.21.15.72:0/778712429 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a165c74bf0 msgr2=0x55a165c7cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:11.335 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.329+0000 7f424ffff640 1 --2- 172.21.15.72:0/778712429 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a165c74bf0 0x55a165c7cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:11.335 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.329+0000 7f424ffff640 1 -- 172.21.15.72:0/778712429 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a165c74a20 con 0x55a165c742d0 2024-03-22T01:37:11.335 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.329+0000 7f42577fe640 1 --2- 172.21.15.72:0/778712429 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a165c74bf0 0x55a165c7cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:37:11.335 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.329+0000 7f424ffff640 1 --2- 172.21.15.72:0/778712429 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a165c742d0 0x55a165c95140 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f4240005d00 tx=0x7f4240016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aad315d2e5320468 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:11.336 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.329+0000 7f424cff9640 1 -- 172.21.15.72:0/778712429 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4240004020 con 0x55a165c742d0 2024-03-22T01:37:11.336 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.330+0000 7f424cff9640 1 -- 172.21.15.72:0/778712429 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f424002b5b0 con 0x55a165c742d0 2024-03-22T01:37:11.336 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.330+0000 7f424cff9640 1 -- 172.21.15.72:0/778712429 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4240003990 con 0x55a165c742d0 2024-03-22T01:37:11.336 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.330+0000 7f42603f1740 1 -- 172.21.15.72:0/778712429 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a165c742d0 msgr2=0x55a165c95140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:11.336 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.330+0000 7f42603f1740 1 --2- 172.21.15.72:0/778712429 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a165c742d0 0x55a165c95140 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f4240005d00 tx=0x7f4240016000 comp rx=0 tx=0).stop 2024-03-22T01:37:11.336 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.330+0000 7f42603f1740 1 -- 172.21.15.72:0/778712429 shutdown_connections 2024-03-22T01:37:11.336 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.330+0000 7f42603f1740 1 --2- 172.21.15.72:0/778712429 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a165c74bf0 0x55a165c7cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:11.336 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.330+0000 7f42603f1740 1 --2- 172.21.15.72:0/778712429 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a165c742d0 0x55a165c95140 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:11.336 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.330+0000 7f42603f1740 1 --2- 172.21.15.72:0/778712429 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a165c6ebf0 0x55a165c89330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:11.336 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.330+0000 7f42603f1740 1 -- 172.21.15.72:0/778712429 >> 172.21.15.72:0/778712429 conn(0x55a165c7acf0 msgr2=0x55a165c6dc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:11.336 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.330+0000 7f42603f1740 1 -- 172.21.15.72:0/778712429 shutdown_connections 2024-03-22T01:37:11.336 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.330+0000 7f42603f1740 1 -- 172.21.15.72:0/778712429 wait complete. 2024-03-22T01:37:11.336 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.330+0000 7f42603f1740 1 -- start start 2024-03-22T01:37:11.336 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.331+0000 7f42603f1740 1 Processor -- start 2024-03-22T01:37:11.336 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.331+0000 7f42603f1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a165c6ebf0 0x55a165c85ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:11.337 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.331+0000 7f42603f1740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a165c742d0 0x55a165c80f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:11.337 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.331+0000 7f42603f1740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a165c74bf0 0x55a165c814a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:11.337 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.331+0000 7f42603f1740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f42581add20 con 0x55a165c742d0 2024-03-22T01:37:11.337 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.331+0000 7f42603f1740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f42581af190 con 0x55a165c74bf0 2024-03-22T01:37:11.337 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.331+0000 7f42603f1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f42581ae3f0 con 0x55a165c6ebf0 2024-03-22T01:37:11.337 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.331+0000 7f42603f1740 10 client.0 _ll_register_callbacks cb 0x7f42480095f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:37:11.337 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.331+0000 7f424ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a165c742d0 0x55a165c80f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:11.337 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.331+0000 7f424ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a165c742d0 0x55a165c80f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59334/0 (socket says 172.21.15.72:59334) 2024-03-22T01:37:11.337 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.331+0000 7f424ffff640 1 -- 172.21.15.72:0/72574914 learned_addr learned my addr 172.21.15.72:0/72574914 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:37:11.337 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.331+0000 7f42577fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a165c74bf0 0x55a165c814a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:11.337 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.331+0000 7f4256ffd640 1 --2- 172.21.15.72:0/72574914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a165c6ebf0 0x55a165c85ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:11.337 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.331+0000 7f424ffff640 1 -- 172.21.15.72:0/72574914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a165c6ebf0 msgr2=0x55a165c85ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:11.337 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.331+0000 7f424ffff640 1 --2- 172.21.15.72:0/72574914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a165c6ebf0 0x55a165c85ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:11.337 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.331+0000 7f424ffff640 1 -- 172.21.15.72:0/72574914 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a165c74bf0 msgr2=0x55a165c814a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:11.338 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.331+0000 7f424ffff640 1 --2- 172.21.15.72:0/72574914 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a165c74bf0 0x55a165c814a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:11.338 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.331+0000 7f424ffff640 1 -- 172.21.15.72:0/72574914 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a165c81c70 con 0x55a165c742d0 2024-03-22T01:37:11.338 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.331+0000 7f42577fe640 1 --2- 172.21.15.72:0/72574914 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a165c74bf0 0x55a165c814a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:37:11.338 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.331+0000 7f424ffff640 1 --2- 172.21.15.72:0/72574914 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a165c742d0 0x55a165c80f60 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f4240003630 tx=0x7f424002b950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21f3789de71a1c26 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:11.338 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.331+0000 7f423e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:37:11.338 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.332+0000 7f423e7fc640 1 -- 172.21.15.72:0/72574914 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f424002ce10 con 0x55a165c742d0 2024-03-22T01:37:11.338 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.332+0000 7f423e7fc640 1 -- 172.21.15.72:0/72574914 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f424002b0d0 con 0x55a165c742d0 2024-03-22T01:37:11.338 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.332+0000 7f42603f1740 1 -- 172.21.15.72:0/72574914 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f424002b5b0 con 0x55a165c742d0 2024-03-22T01:37:11.338 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.332+0000 7f423e7fc640 1 -- 172.21.15.72:0/72574914 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f424002b2d0 con 0x55a165c742d0 2024-03-22T01:37:11.338 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.332+0000 7f42603f1740 10 client.15672 fetch_fsmap learned FSMap version 21 2024-03-22T01:37:11.338 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.332+0000 7f42603f1740 1 -- 172.21.15.72:0/72574914 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55a165cf3720 con 0x55a165c742d0 2024-03-22T01:37:11.338 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.332+0000 7f423e7fc640 1 -- 172.21.15.72:0/72574914 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f424002b5b0 con 0x55a165c742d0 2024-03-22T01:37:11.338 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.332+0000 7f423e7fc640 1 -- 172.21.15.72:0/72574914 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f4240004b60 con 0x55a165c742d0 2024-03-22T01:37:11.338 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.332+0000 7f42603f1740 10 client.15672 fetch_fsmap finished waiting for FSMap version 21 2024-03-22T01:37:11.339 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.332+0000 7f42603f1740 10 client.15672 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T01:37:11.339 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.332+0000 7f42603f1740 20 client.15672 populate_metadata read hostname 'smithi072' 2024-03-22T01:37:11.339 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.332+0000 7f42603f1740 1 --2- 172.21.15.72:0/72574914 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55a165c7d9c0 0x55a165cf5f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:11.339 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.332+0000 7f42603f1740 4 client.15672 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T01:37:11.339 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.332+0000 7f42603f1740 1 -- 172.21.15.72:0/72574914 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a165bdf8d0 con 0x55a165c7d9c0 2024-03-22T01:37:11.339 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.332+0000 7f4256ffd640 1 --2- 172.21.15.72:0/72574914 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55a165c7d9c0 0x55a165cf5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:11.339 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.333+0000 7f423e7fc640 10 client.15672 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:37:11.339 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.333+0000 7f4256ffd640 1 --2- 172.21.15.72:0/72574914 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55a165c7d9c0 0x55a165cf5f00 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d8e710c6f067315 server_cookie=6182e89bace07bb7 in_seq=0 out_seq=0 2024-03-22T01:37:11.340 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.339+0000 7f423e7fc640 1 -- 172.21.15.72:0/72574914 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a165bdf8d0 con 0x55a165c7d9c0 2024-03-22T01:37:11.340 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.339+0000 7f423e7fc640 10 client.15672 handle_command_reply: tid=0 2024-03-22T01:37:11.340 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.339+0000 7f42603f1740 1 client.15672 shutdown 2024-03-22T01:37:11.341 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.339+0000 7f42603f1740 1 -- 172.21.15.72:0/72574914 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a165c742d0 msgr2=0x55a165c80f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:11.341 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.339+0000 7f42603f1740 1 --2- 172.21.15.72:0/72574914 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a165c742d0 0x55a165c80f60 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f4240003630 tx=0x7f424002b950 comp rx=0 tx=0).stop 2024-03-22T01:37:11.341 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.340+0000 7f42603f1740 1 -- 172.21.15.72:0/72574914 shutdown_connections 2024-03-22T01:37:11.341 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.340+0000 7f42603f1740 1 --2- 172.21.15.72:0/72574914 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55a165c7d9c0 0x55a165cf5f00 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:11.341 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.340+0000 7f42603f1740 1 --2- 172.21.15.72:0/72574914 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a165c74bf0 0x55a165c814a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:11.341 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.340+0000 7f423e7fc640 0 client.15672 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:37:11.341 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.340+0000 7f42603f1740 1 --2- 172.21.15.72:0/72574914 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a165c742d0 0x55a165c80f60 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:11.341 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.340+0000 7f42603f1740 1 --2- 172.21.15.72:0/72574914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a165c6ebf0 0x55a165c85ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:11.341 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.340+0000 7f42603f1740 1 -- 172.21.15.72:0/72574914 >> 172.21.15.72:0/72574914 conn(0x55a165c7acf0 msgr2=0x55a165c75150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:11.341 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.340+0000 7f42603f1740 1 -- 172.21.15.72:0/72574914 shutdown_connections 2024-03-22T01:37:11.341 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.340+0000 7f42603f1740 1 -- 172.21.15.72:0/72574914 wait complete. 2024-03-22T01:37:11.341 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.340+0000 7f42603f1740 20 client.15672 trim_cache size 0 max 16384 2024-03-22T01:37:11.342 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout:{ 2024-03-22T01:37:11.342 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout: "message": "", 2024-03-22T01:37:11.342 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout: "return_code": 0 2024-03-22T01:37:11.342 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout:} 2024-03-22T01:37:11.343 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.340+0000 7f425d732640 1 -- 172.21.15.72:0/4060369308 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f42440a3220 msgr2=0x7f42440a56e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:11.343 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.340+0000 7f425d732640 1 --2- 172.21.15.72:0/4060369308 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f42440a3220 0x7f42440a56e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f42580fe590 tx=0x7f42400113e0 comp rx=0 tx=0).stop 2024-03-22T01:37:11.343 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.340+0000 7f425d732640 1 -- 172.21.15.72:0/4060369308 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4258107c10 msgr2=0x7f425819cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:11.343 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.340+0000 7f425d732640 1 --2- 172.21.15.72:0/4060369308 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4258107c10 0x7f425819cc80 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f425000cad0 tx=0x7f425000cf90 comp rx=0 tx=0).stop 2024-03-22T01:37:11.343 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.340+0000 7f425d732640 1 -- 172.21.15.72:0/4060369308 shutdown_connections 2024-03-22T01:37:11.343 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.340+0000 7f425d732640 1 --2- 172.21.15.72:0/4060369308 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f42440a3220 0x7f42440a56e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:11.343 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.340+0000 7f425d732640 1 --2- 172.21.15.72:0/4060369308 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4258107c10 0x7f425819cc80 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:11.343 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.340+0000 7f425d732640 1 --2- 172.21.15.72:0/4060369308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4258102510 0x7f4258198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:11.343 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.340+0000 7f425d732640 1 --2- 172.21.15.72:0/4060369308 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4258101b50 0x7f4258198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:11.343 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.340+0000 7f425d732640 1 -- 172.21.15.72:0/4060369308 >> 172.21.15.72:0/4060369308 conn(0x7f42580fd5e0 msgr2=0x7f42580feda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:11.344 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.340+0000 7f425d732640 1 -- 172.21.15.72:0/4060369308 shutdown_connections 2024-03-22T01:37:11.344 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:11.340+0000 7f425d732640 1 -- 172.21.15.72:0/4060369308 wait complete. 2024-03-22T01:37:12.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:37:11 smithi098 ceph-mon[36468]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-03-22T01:37:12.357 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:37:11 smithi089 ceph-mon[35770]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-03-22T01:37:12.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:37:11 smithi072 ceph-mon[32414]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-03-22T01:37:13.811 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T01:37:13.811 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T01:37:13.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.931+0000 7f7fc45b6640 1 -- 172.21.15.72:0/3244681266 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7fbc100a30 msgr2=0x7f7fbc100eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:13.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.931+0000 7f7fc45b6640 1 --2- 172.21.15.72:0/3244681266 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7fbc100a30 0x7f7fbc100eb0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f7fac009930 tx=0x7f7fac02bc30 comp rx=0 tx=0).stop 2024-03-22T01:37:13.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.931+0000 7f7fc45b6640 1 -- 172.21.15.72:0/3244681266 shutdown_connections 2024-03-22T01:37:13.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.931+0000 7f7fc45b6640 1 --2- 172.21.15.72:0/3244681266 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7fbc1013f0 0x7f7fbc10d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:13.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.931+0000 7f7fc45b6640 1 --2- 172.21.15.72:0/3244681266 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7fbc100a30 0x7f7fbc100eb0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:13.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.931+0000 7f7fc45b6640 1 --2- 172.21.15.72:0/3244681266 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7fbc106a40 0x7f7fbc106e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:13.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.931+0000 7f7fc45b6640 1 -- 172.21.15.72:0/3244681266 >> 172.21.15.72:0/3244681266 conn(0x7f7fbc077c40 msgr2=0x7f7fbc078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:13.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.931+0000 7f7fc45b6640 1 -- 172.21.15.72:0/3244681266 shutdown_connections 2024-03-22T01:37:13.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.931+0000 7f7fc45b6640 1 -- 172.21.15.72:0/3244681266 wait complete. 2024-03-22T01:37:13.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.931+0000 7f7fc45b6640 1 Processor -- start 2024-03-22T01:37:13.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.932+0000 7f7fc45b6640 1 -- start start 2024-03-22T01:37:13.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.932+0000 7f7fc45b6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7fbc100a30 0x7f7fbc197370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:13.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.932+0000 7f7fc45b6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7fbc1013f0 0x7f7fbc1978b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:13.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.932+0000 7f7fc45b6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7fbc106a40 0x7f7fbc19bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:13.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.932+0000 7f7fc45b6640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f7fbc110e50 con 0x7f7fbc100a30 2024-03-22T01:37:13.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.932+0000 7f7fc45b6640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f7fbc110cd0 con 0x7f7fbc106a40 2024-03-22T01:37:13.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.932+0000 7f7fc45b6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7fbc110fd0 con 0x7f7fbc1013f0 2024-03-22T01:37:13.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.932+0000 7f7fc2352640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7fbc100a30 0x7f7fbc197370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:13.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.932+0000 7f7fc2352640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7fbc100a30 0x7f7fbc197370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59342/0 (socket says 172.21.15.72:59342) 2024-03-22T01:37:13.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.932+0000 7f7fc2352640 1 -- 172.21.15.72:0/2371809301 learned_addr learned my addr 172.21.15.72:0/2371809301 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:37:13.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.932+0000 7f7fc2b53640 1 --2- 172.21.15.72:0/2371809301 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7fbc106a40 0x7f7fbc19bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:13.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.932+0000 7f7fc1b51640 1 --2- 172.21.15.72:0/2371809301 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7fbc1013f0 0x7f7fbc1978b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:13.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.932+0000 7f7fc2352640 1 -- 172.21.15.72:0/2371809301 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7fbc1013f0 msgr2=0x7f7fbc1978b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:13.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.932+0000 7f7fc2352640 1 --2- 172.21.15.72:0/2371809301 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7fbc1013f0 0x7f7fbc1978b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:13.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.932+0000 7f7fc2352640 1 -- 172.21.15.72:0/2371809301 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7fbc106a40 msgr2=0x7f7fbc19bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:13.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.932+0000 7f7fc2352640 1 --2- 172.21.15.72:0/2371809301 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7fbc106a40 0x7f7fbc19bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:13.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.932+0000 7f7fc2352640 1 -- 172.21.15.72:0/2371809301 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fbc19c310 con 0x7f7fbc100a30 2024-03-22T01:37:13.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.932+0000 7f7fc2352640 1 --2- 172.21.15.72:0/2371809301 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7fbc100a30 0x7f7fbc197370 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f7fb800eb90 tx=0x7f7fb8007590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:13.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.932+0000 7f7fb37fe640 1 -- 172.21.15.72:0/2371809301 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb800ec80 con 0x7f7fbc100a30 2024-03-22T01:37:13.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.933+0000 7f7fb37fe640 1 -- 172.21.15.72:0/2371809301 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fb800ee20 con 0x7f7fbc100a30 2024-03-22T01:37:13.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.933+0000 7f7fc45b6640 1 -- 172.21.15.72:0/2371809301 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fbc19c5a0 con 0x7f7fbc100a30 2024-03-22T01:37:13.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.933+0000 7f7fb37fe640 1 -- 172.21.15.72:0/2371809301 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb80058b0 con 0x7f7fbc100a30 2024-03-22T01:37:13.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.933+0000 7f7fc45b6640 1 -- 172.21.15.72:0/2371809301 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fbc068e60 con 0x7f7fbc100a30 2024-03-22T01:37:13.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.935+0000 7f7fb37fe640 1 -- 172.21.15.72:0/2371809301 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 24) ==== 138087+0+0 (secure 0 0 0) 0x7f7fb8005020 con 0x7f7fbc100a30 2024-03-22T01:37:13.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.935+0000 7f7fc7199740 1 Processor -- start 2024-03-22T01:37:13.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.936+0000 7f7fc7199740 1 -- start start 2024-03-22T01:37:13.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.936+0000 7f7fb37fe640 1 --2- 172.21.15.72:0/2371809301 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f7fa80a31b0 0x7f7fa80a5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:13.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.936+0000 7f7fc7199740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557e8a28a410 0x557e8a28a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:13.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.936+0000 7f7fc7199740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557e8a28ad30 0x557e8a2933d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:13.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.936+0000 7f7fc7199740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557e8a293910 0x557e8a295d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:13.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.936+0000 7f7fc7199740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x557e8a1ee8c0 con 0x557e8a28a410 2024-03-22T01:37:13.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.936+0000 7f7fc7199740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x557e8a196c20 con 0x557e8a28ad30 2024-03-22T01:37:13.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.936+0000 7f7fc7199740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557e8a1ea870 con 0x557e8a293910 2024-03-22T01:37:13.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.936+0000 7f7fc2b53640 1 --2- 172.21.15.72:0/2371809301 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f7fa80a31b0 0x7f7fa80a5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:13.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.936+0000 7f7fc2352640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557e8a28a410 0x557e8a28a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:13.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.936+0000 7f7fb37fe640 1 -- 172.21.15.72:0/2371809301 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f7fb80d1320 con 0x7f7fbc100a30 2024-03-22T01:37:13.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.936+0000 7f7fc1b51640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557e8a28ad30 0x557e8a2933d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:13.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.936+0000 7f7fc2352640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557e8a28a410 0x557e8a28a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59344/0 (socket says 172.21.15.72:59344) 2024-03-22T01:37:13.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.936+0000 7f7fc2352640 1 -- 172.21.15.72:0/81795506 learned_addr learned my addr 172.21.15.72:0/81795506 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:37:13.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.936+0000 7f7fc2b53640 1 --2- 172.21.15.72:0/81795506 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557e8a293910 0x557e8a295d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:13.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.936+0000 7f7fc1b51640 1 -- 172.21.15.72:0/81795506 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557e8a293910 msgr2=0x557e8a295d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:13.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.936+0000 7f7fc1b51640 1 --2- 172.21.15.72:0/81795506 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557e8a293910 0x557e8a295d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:13.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.936+0000 7f7fc1b51640 1 -- 172.21.15.72:0/81795506 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557e8a28a410 msgr2=0x557e8a28a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:13.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.936+0000 7f7fc1b51640 1 --2- 172.21.15.72:0/81795506 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557e8a28a410 0x557e8a28a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:13.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.938+0000 7f7fc1b51640 1 -- 172.21.15.72:0/81795506 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557e8a1bb150 con 0x557e8a28ad30 2024-03-22T01:37:13.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.938+0000 7f7fc2b53640 1 --2- 172.21.15.72:0/2371809301 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f7fa80a31b0 0x7f7fa80a5670 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7fb4009660 tx=0x7f7fb4013830 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:13.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.938+0000 7f7fc1b51640 1 --2- 172.21.15.72:0/81795506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557e8a28ad30 0x557e8a2933d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7fac009230 tx=0x7f7fac005bf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f555e965e161573 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:13.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.938+0000 7f7fb0bfe640 1 -- 172.21.15.72:0/81795506 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fac042070 con 0x557e8a28ad30 2024-03-22T01:37:13.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.938+0000 7f7fb0bfe640 1 -- 172.21.15.72:0/81795506 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fac004030 con 0x557e8a28ad30 2024-03-22T01:37:13.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.938+0000 7f7fb0bfe640 1 -- 172.21.15.72:0/81795506 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fac004c10 con 0x557e8a28ad30 2024-03-22T01:37:13.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.939+0000 7f7fc7199740 1 -- 172.21.15.72:0/81795506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557e8a28ad30 msgr2=0x557e8a2933d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:13.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.939+0000 7f7fc7199740 1 --2- 172.21.15.72:0/81795506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557e8a28ad30 0x557e8a2933d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7fac009230 tx=0x7f7fac005bf0 comp rx=0 tx=0).stop 2024-03-22T01:37:13.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.939+0000 7f7fc7199740 1 -- 172.21.15.72:0/81795506 shutdown_connections 2024-03-22T01:37:13.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.939+0000 7f7fc7199740 1 --2- 172.21.15.72:0/81795506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557e8a28ad30 0x557e8a2933d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:13.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.939+0000 7f7fc7199740 1 --2- 172.21.15.72:0/81795506 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557e8a293910 0x557e8a295d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:13.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.939+0000 7f7fc7199740 1 --2- 172.21.15.72:0/81795506 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557e8a28a410 0x557e8a28a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:13.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.939+0000 7f7fc7199740 1 -- 172.21.15.72:0/81795506 >> 172.21.15.72:0/81795506 conn(0x557e8a284b40 msgr2=0x557e8a284f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:13.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.939+0000 7f7fc7199740 1 -- 172.21.15.72:0/81795506 shutdown_connections 2024-03-22T01:37:13.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.939+0000 7f7fc7199740 1 -- 172.21.15.72:0/81795506 wait complete. 2024-03-22T01:37:13.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.939+0000 7f7fc7199740 1 -- start start 2024-03-22T01:37:13.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.939+0000 7f7fc7199740 1 Processor -- start 2024-03-22T01:37:13.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.939+0000 7f7fc7199740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557e8a28a410 0x557e8a290930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:13.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.939+0000 7f7fc7199740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557e8a28ad30 0x557e8a290e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:13.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.939+0000 7f7fc7199740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557e8a293910 0x557e8a2913b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:13.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.939+0000 7f7fc7199740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x557e8a2966a0 con 0x557e8a28ad30 2024-03-22T01:37:13.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.939+0000 7f7fc7199740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x557e8a1aa130 con 0x557e8a293910 2024-03-22T01:37:13.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.939+0000 7f7fc7199740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557e8a296820 con 0x557e8a28a410 2024-03-22T01:37:13.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.939+0000 7f7fc7199740 10 client.0 _ll_register_callbacks cb 0x557e8a1a9b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:37:13.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.939+0000 7f7fc1b51640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557e8a28ad30 0x557e8a290e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:13.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.939+0000 7f7fc1b51640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557e8a28ad30 0x557e8a290e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59358/0 (socket says 172.21.15.72:59358) 2024-03-22T01:37:13.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.939+0000 7f7fc1b51640 1 -- 172.21.15.72:0/234467749 learned_addr learned my addr 172.21.15.72:0/234467749 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:37:13.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.939+0000 7f7fc2b53640 1 --2- 172.21.15.72:0/234467749 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557e8a293910 0x557e8a2913b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:13.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.939+0000 7f7fc2352640 1 --2- 172.21.15.72:0/234467749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557e8a28a410 0x557e8a290930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:13.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.939+0000 7f7fc1b51640 1 -- 172.21.15.72:0/234467749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557e8a28a410 msgr2=0x557e8a290930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:13.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.939+0000 7f7fc1b51640 1 --2- 172.21.15.72:0/234467749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557e8a28a410 0x557e8a290930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:13.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.939+0000 7f7fc1b51640 1 -- 172.21.15.72:0/234467749 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557e8a293910 msgr2=0x557e8a2913b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:13.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.939+0000 7f7fc1b51640 1 --2- 172.21.15.72:0/234467749 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557e8a293910 0x557e8a2913b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:13.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.939+0000 7f7fc1b51640 1 -- 172.21.15.72:0/234467749 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557e89fcc7d0 con 0x557e8a28ad30 2024-03-22T01:37:13.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.939+0000 7f7fc2b53640 1 --2- 172.21.15.72:0/234467749 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557e8a293910 0x557e8a2913b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:37:13.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.939+0000 7f7fc1b51640 1 --2- 172.21.15.72:0/234467749 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557e8a28ad30 0x557e8a290e70 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f7fac005ab0 tx=0x7f7fac004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3936d4841899b824 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:13.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.939+0000 7f7fa5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:37:13.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.939+0000 7f7fa5ffb640 1 -- 172.21.15.72:0/234467749 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fac042070 con 0x557e8a28ad30 2024-03-22T01:37:13.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.939+0000 7f7fa5ffb640 1 -- 172.21.15.72:0/234467749 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f7fac02ca90 con 0x557e8a28ad30 2024-03-22T01:37:13.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.939+0000 7f7fc7199740 1 -- 172.21.15.72:0/234467749 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7fac004030 con 0x557e8a28ad30 2024-03-22T01:37:13.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.939+0000 7f7fa5ffb640 1 -- 172.21.15.72:0/234467749 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f7fac057050 con 0x557e8a28ad30 2024-03-22T01:37:13.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.939+0000 7f7fc7199740 10 client.15684 fetch_fsmap learned FSMap version 21 2024-03-22T01:37:13.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.939+0000 7f7fc7199740 1 -- 172.21.15.72:0/234467749 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x557e89f023f0 con 0x557e8a28ad30 2024-03-22T01:37:13.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.940+0000 7f7fa5ffb640 1 -- 172.21.15.72:0/234467749 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fac004030 con 0x557e8a28ad30 2024-03-22T01:37:13.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.940+0000 7f7fa5ffb640 1 -- 172.21.15.72:0/234467749 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f7fac003790 con 0x557e8a28ad30 2024-03-22T01:37:13.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.940+0000 7f7fc7199740 10 client.15684 fetch_fsmap finished waiting for FSMap version 21 2024-03-22T01:37:13.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.940+0000 7f7fc7199740 10 client.15684 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T01:37:13.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.940+0000 7f7fc7199740 20 client.15684 populate_metadata read hostname 'smithi072' 2024-03-22T01:37:13.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.940+0000 7f7fc7199740 1 --2- 172.21.15.72:0/234467749 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x557e8a2a3aa0 0x557e8a2a5e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:13.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.940+0000 7f7fc7199740 4 client.15684 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T01:37:13.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.940+0000 7f7fc7199740 1 -- 172.21.15.72:0/234467749 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557e8a1fbe00 con 0x557e8a2a3aa0 2024-03-22T01:37:13.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.940+0000 7f7fc2352640 1 --2- 172.21.15.72:0/234467749 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x557e8a2a3aa0 0x557e8a2a5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:13.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.940+0000 7f7fa5ffb640 10 client.15684 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:37:13.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.940+0000 7f7fc2352640 1 --2- 172.21.15.72:0/234467749 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x557e8a2a3aa0 0x557e8a2a5e90 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c0ab311c4ba0dc36 server_cookie=6ee171376a1bda86 in_seq=0 out_seq=0 2024-03-22T01:37:13.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.941+0000 7f7fa5ffb640 1 -- 172.21.15.72:0/234467749 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557e8a1fbe00 con 0x557e8a2a3aa0 2024-03-22T01:37:13.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.941+0000 7f7fa5ffb640 10 client.15684 handle_command_reply: tid=0 2024-03-22T01:37:13.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.941+0000 7f7fc7199740 1 client.15684 shutdown 2024-03-22T01:37:13.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.941+0000 7f7fc7199740 1 -- 172.21.15.72:0/234467749 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557e8a28ad30 msgr2=0x557e8a290e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:13.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.941+0000 7f7fc7199740 1 --2- 172.21.15.72:0/234467749 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557e8a28ad30 0x557e8a290e70 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f7fac005ab0 tx=0x7f7fac004b40 comp rx=0 tx=0).stop 2024-03-22T01:37:13.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.941+0000 7f7fc7199740 1 -- 172.21.15.72:0/234467749 shutdown_connections 2024-03-22T01:37:13.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.941+0000 7f7fc7199740 1 --2- 172.21.15.72:0/234467749 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x557e8a2a3aa0 0x557e8a2a5e90 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:13.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.941+0000 7f7fc7199740 1 --2- 172.21.15.72:0/234467749 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557e8a293910 0x557e8a2913b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:13.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.941+0000 7f7fa5ffb640 0 client.15684 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:37:13.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.941+0000 7f7fc7199740 1 --2- 172.21.15.72:0/234467749 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557e8a28ad30 0x557e8a290e70 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:13.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.941+0000 7f7fc7199740 1 --2- 172.21.15.72:0/234467749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557e8a28a410 0x557e8a290930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:13.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.941+0000 7f7fc7199740 1 -- 172.21.15.72:0/234467749 >> 172.21.15.72:0/234467749 conn(0x557e8a284b40 msgr2=0x557e8a286890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:13.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.941+0000 7f7fc7199740 1 -- 172.21.15.72:0/234467749 shutdown_connections 2024-03-22T01:37:13.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.941+0000 7f7fc7199740 1 -- 172.21.15.72:0/234467749 wait complete. 2024-03-22T01:37:13.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.942+0000 7f7fc7199740 20 client.15684 trim_cache size 0 max 16384 2024-03-22T01:37:13.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.959+0000 7f7fc7199740 1 Processor -- start 2024-03-22T01:37:13.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.959+0000 7f7fc7199740 1 -- start start 2024-03-22T01:37:13.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.959+0000 7f7fc7199740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557e8a28a410 0x557e8a292ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:13.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.959+0000 7f7fc7199740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557e8a28ad30 0x557e8a29f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:13.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.959+0000 7f7fc7199740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557e8a293910 0x557e8a2ab540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:13.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.959+0000 7f7fc7199740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f7fbc1aae00 con 0x557e8a293910 2024-03-22T01:37:13.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.959+0000 7f7fc7199740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f7fbc1ad680 con 0x557e8a28a410 2024-03-22T01:37:13.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.959+0000 7f7fc7199740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7fbc1ab470 con 0x557e8a28ad30 2024-03-22T01:37:13.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.959+0000 7f7fc2b53640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557e8a293910 0x557e8a2ab540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:13.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.959+0000 7f7fc2b53640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557e8a293910 0x557e8a2ab540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59366/0 (socket says 172.21.15.72:59366) 2024-03-22T01:37:13.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.959+0000 7f7fc2b53640 1 -- 172.21.15.72:0/535716792 learned_addr learned my addr 172.21.15.72:0/535716792 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:37:13.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.959+0000 7f7fc2352640 1 --2- 172.21.15.72:0/535716792 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557e8a28a410 0x557e8a292ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:13.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.959+0000 7f7fc2b53640 1 -- 172.21.15.72:0/535716792 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557e8a28ad30 msgr2=0x557e8a29f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:13.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.959+0000 7f7fc2b53640 1 --2- 172.21.15.72:0/535716792 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557e8a28ad30 0x557e8a29f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:13.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.959+0000 7f7fc2b53640 1 -- 172.21.15.72:0/535716792 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557e8a28a410 msgr2=0x557e8a292ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:13.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.959+0000 7f7fc2b53640 1 --2- 172.21.15.72:0/535716792 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557e8a28a410 0x557e8a292ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:13.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.959+0000 7f7fc2b53640 1 -- 172.21.15.72:0/535716792 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557e89ec20f0 con 0x557e8a293910 2024-03-22T01:37:13.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.959+0000 7f7fc2b53640 1 --2- 172.21.15.72:0/535716792 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557e8a293910 0x557e8a2ab540 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f7fb4004370 tx=0x7f7fb401f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e85d2c2cd8b2384 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:13.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.959+0000 7f7fa7fff640 1 -- 172.21.15.72:0/535716792 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb400f480 con 0x557e8a293910 2024-03-22T01:37:13.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.960+0000 7f7fa7fff640 1 -- 172.21.15.72:0/535716792 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fb400fb00 con 0x557e8a293910 2024-03-22T01:37:13.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.960+0000 7f7fa7fff640 1 -- 172.21.15.72:0/535716792 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb4010a80 con 0x557e8a293910 2024-03-22T01:37:13.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.960+0000 7f7fc7199740 1 -- 172.21.15.72:0/535716792 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557e8a293910 msgr2=0x557e8a2ab540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:13.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.960+0000 7f7fc7199740 1 --2- 172.21.15.72:0/535716792 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557e8a293910 0x557e8a2ab540 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f7fb4004370 tx=0x7f7fb401f000 comp rx=0 tx=0).stop 2024-03-22T01:37:13.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.960+0000 7f7fc7199740 1 -- 172.21.15.72:0/535716792 shutdown_connections 2024-03-22T01:37:13.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.960+0000 7f7fc7199740 1 --2- 172.21.15.72:0/535716792 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557e8a293910 0x557e8a2ab540 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:13.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.960+0000 7f7fc7199740 1 --2- 172.21.15.72:0/535716792 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557e8a28ad30 0x557e8a29f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:13.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.960+0000 7f7fc7199740 1 --2- 172.21.15.72:0/535716792 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557e8a28a410 0x557e8a292ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:13.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.960+0000 7f7fc7199740 1 -- 172.21.15.72:0/535716792 >> 172.21.15.72:0/535716792 conn(0x557e8a284b40 msgr2=0x557e8a28bc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:13.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.960+0000 7f7fc7199740 1 -- 172.21.15.72:0/535716792 shutdown_connections 2024-03-22T01:37:13.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.960+0000 7f7fc7199740 1 -- 172.21.15.72:0/535716792 wait complete. 2024-03-22T01:37:13.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.960+0000 7f7fc7199740 1 -- start start 2024-03-22T01:37:13.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.960+0000 7f7fc7199740 1 Processor -- start 2024-03-22T01:37:13.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.962+0000 7f7fc7199740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557e8a28a410 0x557e8a29bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:13.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.962+0000 7f7fc7199740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557e8a28ad30 0x557e8a28cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:13.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.962+0000 7f7fc7199740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557e8a293910 0x557e8a28d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:13.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.962+0000 7f7fc7199740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f7fbc1abb40 con 0x557e8a293910 2024-03-22T01:37:13.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.962+0000 7f7fc7199740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f7fbc1acfb0 con 0x557e8a28ad30 2024-03-22T01:37:13.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.962+0000 7f7fc7199740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7fbc1ac210 con 0x557e8a28a410 2024-03-22T01:37:13.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.962+0000 7f7fc7199740 10 client.0 _ll_register_callbacks cb 0x7f7fac004530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:37:13.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.962+0000 7f7fc2b53640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557e8a293910 0x557e8a28d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:13.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.962+0000 7f7fc2b53640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557e8a293910 0x557e8a28d4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59368/0 (socket says 172.21.15.72:59368) 2024-03-22T01:37:13.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.962+0000 7f7fc2b53640 1 -- 172.21.15.72:0/958686075 learned_addr learned my addr 172.21.15.72:0/958686075 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:37:13.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.962+0000 7f7fc1b51640 1 --2- 172.21.15.72:0/958686075 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557e8a28ad30 0x557e8a28cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:13.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.962+0000 7f7fc2352640 1 --2- 172.21.15.72:0/958686075 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557e8a28a410 0x557e8a29bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:13.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.962+0000 7f7fc2b53640 1 -- 172.21.15.72:0/958686075 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557e8a28a410 msgr2=0x557e8a29bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:13.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.962+0000 7f7fc2b53640 1 --2- 172.21.15.72:0/958686075 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557e8a28a410 0x557e8a29bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:13.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.962+0000 7f7fc2b53640 1 -- 172.21.15.72:0/958686075 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557e8a28ad30 msgr2=0x557e8a28cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:13.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.962+0000 7f7fc2b53640 1 --2- 172.21.15.72:0/958686075 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557e8a28ad30 0x557e8a28cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:13.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.962+0000 7f7fc2b53640 1 -- 172.21.15.72:0/958686075 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557e8a28daa0 con 0x557e8a293910 2024-03-22T01:37:13.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.962+0000 7f7fc2b53640 1 --2- 172.21.15.72:0/958686075 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557e8a293910 0x557e8a28d4b0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f7fb4003d90 tx=0x7f7fb40045d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=235e2cdb6a2dafe1 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:13.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.962+0000 7f7fa5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:37:13.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.962+0000 7f7fa5ffb640 1 -- 172.21.15.72:0/958686075 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb402d070 con 0x557e8a293910 2024-03-22T01:37:13.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.962+0000 7f7fa5ffb640 1 -- 172.21.15.72:0/958686075 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f7fb400f890 con 0x557e8a293910 2024-03-22T01:37:13.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.962+0000 7f7fc7199740 1 -- 172.21.15.72:0/958686075 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7fb400fb00 con 0x557e8a293910 2024-03-22T01:37:13.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.962+0000 7f7fa5ffb640 1 -- 172.21.15.72:0/958686075 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f7fb4039050 con 0x557e8a293910 2024-03-22T01:37:13.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.962+0000 7f7fc7199740 10 client.15690 fetch_fsmap learned FSMap version 21 2024-03-22T01:37:13.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.962+0000 7f7fc7199740 1 -- 172.21.15.72:0/958686075 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x557e8a309980 con 0x557e8a293910 2024-03-22T01:37:13.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.963+0000 7f7fa5ffb640 1 -- 172.21.15.72:0/958686075 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fb400fb00 con 0x557e8a293910 2024-03-22T01:37:13.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.963+0000 7f7fa5ffb640 1 -- 172.21.15.72:0/958686075 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f7fb401f070 con 0x557e8a293910 2024-03-22T01:37:13.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.963+0000 7f7fc7199740 10 client.15690 fetch_fsmap finished waiting for FSMap version 21 2024-03-22T01:37:13.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.963+0000 7f7fc7199740 10 client.15690 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T01:37:13.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.963+0000 7f7fc7199740 20 client.15690 populate_metadata read hostname 'smithi072' 2024-03-22T01:37:13.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.963+0000 7f7fc7199740 1 --2- 172.21.15.72:0/958686075 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x557e8a30a020 0x557e8a30c440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:13.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.963+0000 7f7fc7199740 4 client.15690 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T01:37:13.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.963+0000 7f7fc7199740 1 -- 172.21.15.72:0/958686075 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "scrub status"}) -- 0x557e8a1f6150 con 0x557e8a30a020 2024-03-22T01:37:13.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.963+0000 7f7fc2352640 1 --2- 172.21.15.72:0/958686075 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x557e8a30a020 0x557e8a30c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:13.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.963+0000 7f7fa5ffb640 10 client.15690 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:37:13.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.963+0000 7f7fc2352640 1 --2- 172.21.15.72:0/958686075 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x557e8a30a020 0x557e8a30c440 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a8a3fdcaaa54c3b server_cookie=396a231f68a803cc in_seq=0 out_seq=0 2024-03-22T01:37:13.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.963+0000 7f7fa5ffb640 1 -- 172.21.15.72:0/958686075 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x557e8a1f6150 con 0x557e8a30a020 2024-03-22T01:37:13.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.963+0000 7f7fa5ffb640 10 client.15690 handle_command_reply: tid=0 2024-03-22T01:37:13.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.963+0000 7f7fc7199740 1 client.15690 shutdown 2024-03-22T01:37:13.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.963+0000 7f7fc7199740 1 -- 172.21.15.72:0/958686075 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557e8a293910 msgr2=0x557e8a28d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:13.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.963+0000 7f7fc7199740 1 --2- 172.21.15.72:0/958686075 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557e8a293910 0x557e8a28d4b0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f7fb4003d90 tx=0x7f7fb40045d0 comp rx=0 tx=0).stop 2024-03-22T01:37:13.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.964+0000 7f7fc7199740 1 -- 172.21.15.72:0/958686075 shutdown_connections 2024-03-22T01:37:13.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.964+0000 7f7fc7199740 1 --2- 172.21.15.72:0/958686075 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x557e8a30a020 0x557e8a30c440 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:13.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.964+0000 7f7fc7199740 1 --2- 172.21.15.72:0/958686075 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557e8a28ad30 0x557e8a28cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:13.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.964+0000 7f7fc7199740 1 --2- 172.21.15.72:0/958686075 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557e8a293910 0x557e8a28d4b0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:13.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.964+0000 7f7fa5ffb640 0 client.15690 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:37:13.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.964+0000 7f7fc7199740 1 --2- 172.21.15.72:0/958686075 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557e8a28a410 0x557e8a29bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:13.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.964+0000 7f7fc7199740 1 -- 172.21.15.72:0/958686075 >> 172.21.15.72:0/958686075 conn(0x557e8a284b40 msgr2=0x557e8a2908d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:13.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.964+0000 7f7fc7199740 1 -- 172.21.15.72:0/958686075 shutdown_connections 2024-03-22T01:37:13.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.964+0000 7f7fc7199740 1 -- 172.21.15.72:0/958686075 wait complete. 2024-03-22T01:37:13.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.964+0000 7f7fc7199740 20 client.15690 trim_cache size 0 max 16384 2024-03-22T01:37:13.970 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-22T01:37:13.970 INFO:teuthology.orchestra.run.smithi072.stdout: "status": "no active scrubs running", 2024-03-22T01:37:13.970 INFO:teuthology.orchestra.run.smithi072.stdout: "scrubs": {} 2024-03-22T01:37:13.970 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-22T01:37:13.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.964+0000 7f7fc45b6640 1 -- 172.21.15.72:0/2371809301 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f7fa80a31b0 msgr2=0x7f7fa80a5670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:13.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.964+0000 7f7fc45b6640 1 --2- 172.21.15.72:0/2371809301 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f7fa80a31b0 0x7f7fa80a5670 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7fb4009660 tx=0x7f7fb4013830 comp rx=0 tx=0).stop 2024-03-22T01:37:13.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.964+0000 7f7fc45b6640 1 -- 172.21.15.72:0/2371809301 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7fbc100a30 msgr2=0x7f7fbc197370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:13.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.964+0000 7f7fc45b6640 1 --2- 172.21.15.72:0/2371809301 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7fbc100a30 0x7f7fbc197370 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f7fb800eb90 tx=0x7f7fb8007590 comp rx=0 tx=0).stop 2024-03-22T01:37:13.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.964+0000 7f7fc45b6640 1 -- 172.21.15.72:0/2371809301 shutdown_connections 2024-03-22T01:37:13.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.964+0000 7f7fc45b6640 1 --2- 172.21.15.72:0/2371809301 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f7fa80a31b0 0x7f7fa80a5670 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:13.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.964+0000 7f7fc45b6640 1 --2- 172.21.15.72:0/2371809301 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7fbc106a40 0x7f7fbc19bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:13.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.964+0000 7f7fc45b6640 1 --2- 172.21.15.72:0/2371809301 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7fbc1013f0 0x7f7fbc1978b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:13.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.964+0000 7f7fc45b6640 1 --2- 172.21.15.72:0/2371809301 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7fbc100a30 0x7f7fbc197370 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:13.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.964+0000 7f7fc45b6640 1 -- 172.21.15.72:0/2371809301 >> 172.21.15.72:0/2371809301 conn(0x7f7fbc077c40 msgr2=0x7f7fbc103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:13.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.964+0000 7f7fc45b6640 1 -- 172.21.15.72:0/2371809301 shutdown_connections 2024-03-22T01:37:13.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:13.964+0000 7f7fc45b6640 1 -- 172.21.15.72:0/2371809301 wait complete. 2024-03-22T01:37:13.974 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-22T01:37:13.974 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T01:37:14.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.076+0000 7fe3f044e640 1 -- 172.21.15.72:0/4136644438 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3e8106a30 msgr2=0x7fe3e8106e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:14.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.076+0000 7fe3f044e640 1 --2- 172.21.15.72:0/4136644438 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3e8106a30 0x7fe3e8106e10 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc009930 tx=0x7fe3dc02bc30 comp rx=0 tx=0).stop 2024-03-22T01:37:14.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.077+0000 7fe3f044e640 1 -- 172.21.15.72:0/4136644438 shutdown_connections 2024-03-22T01:37:14.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.077+0000 7fe3f044e640 1 --2- 172.21.15.72:0/4136644438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3e81013e0 0x7fe3e810d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:14.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.077+0000 7fe3f044e640 1 --2- 172.21.15.72:0/4136644438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3e8100a20 0x7fe3e8100ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:14.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.077+0000 7fe3f044e640 1 --2- 172.21.15.72:0/4136644438 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3e8106a30 0x7fe3e8106e10 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:14.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.077+0000 7fe3f044e640 1 -- 172.21.15.72:0/4136644438 >> 172.21.15.72:0/4136644438 conn(0x7fe3e8077ca0 msgr2=0x7fe3e80780b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:14.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.077+0000 7fe3f044e640 1 -- 172.21.15.72:0/4136644438 shutdown_connections 2024-03-22T01:37:14.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.077+0000 7fe3f044e640 1 -- 172.21.15.72:0/4136644438 wait complete. 2024-03-22T01:37:14.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.077+0000 7fe3f044e640 1 Processor -- start 2024-03-22T01:37:14.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.077+0000 7fe3f044e640 1 -- start start 2024-03-22T01:37:14.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.077+0000 7fe3f044e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3e8100a20 0x7fe3e8198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:14.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.077+0000 7fe3f044e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3e81013e0 0x7fe3e8198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:14.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.077+0000 7fe3f044e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3e8106a30 0x7fe3e819ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:14.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.077+0000 7fe3f044e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fe3e8110e10 con 0x7fe3e8106a30 2024-03-22T01:37:14.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.077+0000 7fe3f044e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe3e8110c90 con 0x7fe3e81013e0 2024-03-22T01:37:14.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.077+0000 7fe3f044e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe3e8110f90 con 0x7fe3e8100a20 2024-03-22T01:37:14.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.077+0000 7fe3ee9eb640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3e8106a30 0x7fe3e819ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:14.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.078+0000 7fe3ee9eb640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3e8106a30 0x7fe3e819ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59386/0 (socket says 172.21.15.72:59386) 2024-03-22T01:37:14.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.078+0000 7fe3ee9eb640 1 -- 172.21.15.72:0/2773683432 learned_addr learned my addr 172.21.15.72:0/2773683432 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:37:14.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.078+0000 7fe3ed9e9640 1 --2- 172.21.15.72:0/2773683432 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3e81013e0 0x7fe3e8198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:14.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.078+0000 7fe3ee1ea640 1 --2- 172.21.15.72:0/2773683432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3e8100a20 0x7fe3e8198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:14.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.078+0000 7fe3ee9eb640 1 -- 172.21.15.72:0/2773683432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3e8100a20 msgr2=0x7fe3e8198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:14.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.078+0000 7fe3ee9eb640 1 --2- 172.21.15.72:0/2773683432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3e8100a20 0x7fe3e8198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:14.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.078+0000 7fe3ee9eb640 1 -- 172.21.15.72:0/2773683432 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3e81013e0 msgr2=0x7fe3e8198950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:14.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.078+0000 7fe3ee9eb640 1 --2- 172.21.15.72:0/2773683432 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3e81013e0 0x7fe3e8198950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:14.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.078+0000 7fe3ee9eb640 1 -- 172.21.15.72:0/2773683432 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3e819d400 con 0x7fe3e8106a30 2024-03-22T01:37:14.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.078+0000 7fe3ee1ea640 1 --2- 172.21.15.72:0/2773683432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3e8100a20 0x7fe3e8198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T01:37:14.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.078+0000 7fe3ee9eb640 1 --2- 172.21.15.72:0/2773683432 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3e8106a30 0x7fe3e819ccd0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fe3e000eb10 tx=0x7fe3e000ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:14.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.078+0000 7fe3db7fe640 1 -- 172.21.15.72:0/2773683432 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e000ce20 con 0x7fe3e8106a30 2024-03-22T01:37:14.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.078+0000 7fe3db7fe640 1 -- 172.21.15.72:0/2773683432 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3e0005020 con 0x7fe3e8106a30 2024-03-22T01:37:14.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.078+0000 7fe3db7fe640 1 -- 172.21.15.72:0/2773683432 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e0010690 con 0x7fe3e8106a30 2024-03-22T01:37:14.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.078+0000 7fe3f044e640 1 -- 172.21.15.72:0/2773683432 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3e819d6f0 con 0x7fe3e8106a30 2024-03-22T01:37:14.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.078+0000 7fe3f044e640 1 -- 172.21.15.72:0/2773683432 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3e81a4f80 con 0x7fe3e8106a30 2024-03-22T01:37:14.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.079+0000 7fe3f044e640 1 -- 172.21.15.72:0/2773683432 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3e8102160 con 0x7fe3e8106a30 2024-03-22T01:37:14.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.080+0000 7fe3db7fe640 1 -- 172.21.15.72:0/2773683432 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 24) ==== 138087+0+0 (secure 0 0 0) 0x7fe3e0005540 con 0x7fe3e8106a30 2024-03-22T01:37:14.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.081+0000 7fe3db7fe640 1 --2- 172.21.15.72:0/2773683432 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fe3d40a31b0 0x7fe3d40a5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:14.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.081+0000 7fe3ee1ea640 1 --2- 172.21.15.72:0/2773683432 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fe3d40a31b0 0x7fe3d40a5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:14.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.081+0000 7fe3db7fe640 1 -- 172.21.15.72:0/2773683432 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fe3e00d03f0 con 0x7fe3e8106a30 2024-03-22T01:37:14.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.081+0000 7fe3ee1ea640 1 --2- 172.21.15.72:0/2773683432 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fe3d40a31b0 0x7fe3d40a5670 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc036040 tx=0x7fe3dc000bc0 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:14.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.086+0000 7fe3db7fe640 1 -- 172.21.15.72:0/2773683432 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe3e009c8e0 con 0x7fe3e8106a30 2024-03-22T01:37:14.121 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:37:13 smithi072 ceph-mon[32414]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 KiB/s wr, 16 op/s 2024-03-22T01:37:14.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:37:13 smithi098 ceph-mon[36468]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 KiB/s wr, 16 op/s 2024-03-22T01:37:14.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.308+0000 7fe3f044e640 1 -- 172.21.15.72:0/2773683432 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe3e8199760 con 0x7fe3e8106a30 2024-03-22T01:37:14.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.309+0000 7fe3db7fe640 1 -- 172.21.15.72:0/2773683432 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11785 (secure 0 0 0) 0x7fe3e009c1e0 con 0x7fe3e8106a30 2024-03-22T01:37:14.308 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:37:14.308 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6835/3077216308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3077216308},{"type":"v1","addr":"172.21.15.72:6835","nonce":3077216308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2460254143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":2460254143},{"type":"v1","addr":"172.21.15.72:6837","nonce":2460254143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6839/2026432819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6838","nonce":2026432819},{"type":"v1","addr":"172.21.15.72:6839","nonce":2026432819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6841/400094545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6840","nonce":400094545},{"type":"v1","addr":"172.21.15.72:6841","nonce":400094545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6833/1578701171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":1578701171},{"type":"v1","addr":"172.21.15.89:6833","nonce":1578701171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/2703047213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2703047213},{"type":"v1","addr":"172.21.15.98:6833","nonce":2703047213}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2589468787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2589468787},{"type":"v1","addr":"172.21.15.89:6835","nonce":2589468787}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/92657636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":92657636},{"type":"v1","addr":"172.21.15.98:6835","nonce":92657636}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1662941782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1662941782},{"type":"v1","addr":"172.21.15.89:6837","nonce":1662941782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24490,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/781860371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":781860371},{"type":"v1","addr":"172.21.15.98:6837","nonce":781860371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-22T01:34:14.815677+0000","modified":"2024-03-22T01:34:39.842959+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24524},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3192173650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3192173650},{"type":"v1","addr":"172.21.15.98:6839","nonce":3192173650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6839/149200280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":149200280},{"type":"v1","addr":"172.21.15.89:6839","nonce":149200280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24524]},"id":1}]} 2024-03-22T01:37:14.308 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 21 2024-03-22T01:37:14.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.311+0000 7fe3f044e640 1 -- 172.21.15.72:0/2773683432 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fe3d40a31b0 msgr2=0x7fe3d40a5670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:14.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.311+0000 7fe3f044e640 1 --2- 172.21.15.72:0/2773683432 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fe3d40a31b0 0x7fe3d40a5670 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc036040 tx=0x7fe3dc000bc0 comp rx=0 tx=0).stop 2024-03-22T01:37:14.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.311+0000 7fe3f044e640 1 -- 172.21.15.72:0/2773683432 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3e8106a30 msgr2=0x7fe3e819ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:14.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.311+0000 7fe3f044e640 1 --2- 172.21.15.72:0/2773683432 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3e8106a30 0x7fe3e819ccd0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fe3e000eb10 tx=0x7fe3e000ef10 comp rx=0 tx=0).stop 2024-03-22T01:37:14.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.311+0000 7fe3f044e640 1 -- 172.21.15.72:0/2773683432 shutdown_connections 2024-03-22T01:37:14.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.311+0000 7fe3f044e640 1 --2- 172.21.15.72:0/2773683432 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fe3d40a31b0 0x7fe3d40a5670 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:14.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.311+0000 7fe3f044e640 1 --2- 172.21.15.72:0/2773683432 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3e81013e0 0x7fe3e8198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:14.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.311+0000 7fe3f044e640 1 --2- 172.21.15.72:0/2773683432 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3e8106a30 0x7fe3e819ccd0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:14.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.311+0000 7fe3f044e640 1 --2- 172.21.15.72:0/2773683432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3e8100a20 0x7fe3e8198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:14.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.311+0000 7fe3f044e640 1 -- 172.21.15.72:0/2773683432 >> 172.21.15.72:0/2773683432 conn(0x7fe3e8077ca0 msgr2=0x7fe3e806a5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:14.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.311+0000 7fe3f044e640 1 -- 172.21.15.72:0/2773683432 shutdown_connections 2024-03-22T01:37:14.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.311+0000 7fe3f044e640 1 -- 172.21.15.72:0/2773683432 wait complete. 2024-03-22T01:37:14.321 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-22T01:37:14.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:37:13 smithi089 ceph-mon[35770]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 KiB/s wr, 16 op/s 2024-03-22T01:37:14.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.430+0000 7fde5b484640 1 -- 172.21.15.72:0/3991185374 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde54102db0 msgr2=0x7fde54103190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:14.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.430+0000 7fde5b484640 1 --2- 172.21.15.72:0/3991185374 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde54102db0 0x7fde54103190 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fde44009860 tx=0x7fde4402bc00 comp rx=0 tx=0).stop 2024-03-22T01:37:14.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.430+0000 7fde5b484640 1 -- 172.21.15.72:0/3991185374 shutdown_connections 2024-03-22T01:37:14.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.430+0000 7fde5b484640 1 --2- 172.21.15.72:0/3991185374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde5410c180 0x7fde5410e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:14.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.430+0000 7fde5b484640 1 --2- 172.21.15.72:0/3991185374 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde541036d0 0x7fde5410bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:14.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.430+0000 7fde5b484640 1 --2- 172.21.15.72:0/3991185374 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde54102db0 0x7fde54103190 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:14.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.430+0000 7fde5b484640 1 -- 172.21.15.72:0/3991185374 >> 172.21.15.72:0/3991185374 conn(0x7fde54077c40 msgr2=0x7fde54078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:14.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.431+0000 7fde5b484640 1 -- 172.21.15.72:0/3991185374 shutdown_connections 2024-03-22T01:37:14.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.431+0000 7fde5b484640 1 -- 172.21.15.72:0/3991185374 wait complete. 2024-03-22T01:37:14.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.431+0000 7fde5b484640 1 Processor -- start 2024-03-22T01:37:14.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.431+0000 7fde5b484640 1 -- start start 2024-03-22T01:37:14.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.431+0000 7fde5b484640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde54102db0 0x7fde54198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:14.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.431+0000 7fde5b484640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde541036d0 0x7fde541989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:14.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.431+0000 7fde5b484640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde5410c180 0x7fde5419cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:14.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.431+0000 7fde5b484640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fde54111f10 con 0x7fde54102db0 2024-03-22T01:37:14.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.431+0000 7fde5b484640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fde54111d90 con 0x7fde541036d0 2024-03-22T01:37:14.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.431+0000 7fde5b484640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fde54112090 con 0x7fde5410c180 2024-03-22T01:37:14.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.431+0000 7fde59220640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde54102db0 0x7fde54198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:14.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.431+0000 7fde59220640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde54102db0 0x7fde54198490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59414/0 (socket says 172.21.15.72:59414) 2024-03-22T01:37:14.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.431+0000 7fde59220640 1 -- 172.21.15.72:0/2517144905 learned_addr learned my addr 172.21.15.72:0/2517144905 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:37:14.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.432+0000 7fde58a1f640 1 --2- 172.21.15.72:0/2517144905 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde541036d0 0x7fde541989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:14.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.432+0000 7fde59a21640 1 --2- 172.21.15.72:0/2517144905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde5410c180 0x7fde5419cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:14.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.432+0000 7fde59220640 1 -- 172.21.15.72:0/2517144905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde5410c180 msgr2=0x7fde5419cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:14.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.432+0000 7fde59220640 1 --2- 172.21.15.72:0/2517144905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde5410c180 0x7fde5419cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:14.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.432+0000 7fde59220640 1 -- 172.21.15.72:0/2517144905 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde541036d0 msgr2=0x7fde541989d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:14.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.432+0000 7fde59220640 1 --2- 172.21.15.72:0/2517144905 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde541036d0 0x7fde541989d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:14.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.432+0000 7fde59220640 1 -- 172.21.15.72:0/2517144905 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde5419d480 con 0x7fde54102db0 2024-03-22T01:37:14.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.432+0000 7fde58a1f640 1 --2- 172.21.15.72:0/2517144905 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde541036d0 0x7fde541989d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:37:14.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.432+0000 7fde59220640 1 --2- 172.21.15.72:0/2517144905 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde54102db0 0x7fde54198490 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fde44037040 tx=0x7fde440346f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:14.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.432+0000 7fde4a7fc640 1 -- 172.21.15.72:0/2517144905 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde44041020 con 0x7fde54102db0 2024-03-22T01:37:14.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.432+0000 7fde5b484640 1 -- 172.21.15.72:0/2517144905 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde5419d710 con 0x7fde54102db0 2024-03-22T01:37:14.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.432+0000 7fde4a7fc640 1 -- 172.21.15.72:0/2517144905 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde440044d0 con 0x7fde54102db0 2024-03-22T01:37:14.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.432+0000 7fde4a7fc640 1 -- 172.21.15.72:0/2517144905 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde44039040 con 0x7fde54102db0 2024-03-22T01:37:14.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.432+0000 7fde5b484640 1 -- 172.21.15.72:0/2517144905 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde541a5000 con 0x7fde54102db0 2024-03-22T01:37:14.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.434+0000 7fde4a7fc640 1 -- 172.21.15.72:0/2517144905 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 24) ==== 138087+0+0 (secure 0 0 0) 0x7fde44044020 con 0x7fde54102db0 2024-03-22T01:37:14.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.435+0000 7fde5e06b740 1 Processor -- start 2024-03-22T01:37:14.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.435+0000 7fde5e06b740 1 -- start start 2024-03-22T01:37:14.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.435+0000 7fde5e06b740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c6e009f800 0x55c6e009fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:14.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.435+0000 7fde5e06b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c6e00a0120 0x55c6e00a87c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:14.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.435+0000 7fde5e06b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c6e00a8d00 0x55c6e00ab0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:14.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.435+0000 7fde5e06b740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55c6e00038d0 con 0x55c6e009f800 2024-03-22T01:37:14.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.435+0000 7fde5e06b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55c6dffabe00 con 0x55c6e00a0120 2024-03-22T01:37:14.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.435+0000 7fde5e06b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c6dffff880 con 0x55c6e00a8d00 2024-03-22T01:37:14.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.435+0000 7fde4a7fc640 1 --2- 172.21.15.72:0/2517144905 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fde400a31b0 0x7fde400a5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:14.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.435+0000 7fde59a21640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c6e009f800 0x55c6e009fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:14.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.435+0000 7fde59a21640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c6e009f800 0x55c6e009fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59428/0 (socket says 172.21.15.72:59428) 2024-03-22T01:37:14.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.435+0000 7fde59a21640 1 -- 172.21.15.72:0/1919652748 learned_addr learned my addr 172.21.15.72:0/1919652748 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:37:14.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.435+0000 7fde59220640 1 --2- 172.21.15.72:0/1919652748 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c6e00a0120 0x55c6e00a87c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:14.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.435+0000 7fde59a21640 1 --2- 172.21.15.72:0/2517144905 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fde400a31b0 0x7fde400a5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:14.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.435+0000 7fde4a7fc640 1 -- 172.21.15.72:0/2517144905 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fde440f14e0 con 0x7fde54102db0 2024-03-22T01:37:14.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.435+0000 7fde58a1f640 1 --2- 172.21.15.72:0/1919652748 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c6e00a8d00 0x55c6e00ab0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:14.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.436+0000 7fde59220640 1 -- 172.21.15.72:0/1919652748 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c6e00a8d00 msgr2=0x55c6e00ab0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:14.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.436+0000 7fde59220640 1 --2- 172.21.15.72:0/1919652748 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c6e00a8d00 0x55c6e00ab0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:14.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.436+0000 7fde59220640 1 -- 172.21.15.72:0/1919652748 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c6e009f800 msgr2=0x55c6e009fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:14.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.436+0000 7fde59220640 1 --2- 172.21.15.72:0/1919652748 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c6e009f800 0x55c6e009fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:14.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.436+0000 7fde59220640 1 -- 172.21.15.72:0/1919652748 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c6dfd173f0 con 0x55c6e00a0120 2024-03-22T01:37:14.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.436+0000 7fde59a21640 1 --2- 172.21.15.72:0/1919652748 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c6e009f800 0x55c6e009fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:37:14.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.436+0000 7fde58a1f640 1 --2- 172.21.15.72:0/1919652748 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c6e00a8d00 0x55c6e00ab0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:37:14.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.436+0000 7fde59220640 1 --2- 172.21.15.72:0/1919652748 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c6e00a0120 0x55c6e00a87c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fde440ee850 tx=0x7fde4408cef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c3ff4b404b588b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:14.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.436+0000 7fde59a21640 1 --2- 172.21.15.72:0/2517144905 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fde400a31b0 0x7fde400a5670 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fde4c00afd0 tx=0x7fde4c00a420 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:14.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.436+0000 7fde3f7fe640 1 -- 172.21.15.72:0/1919652748 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde440bcf60 con 0x55c6e00a0120 2024-03-22T01:37:14.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.436+0000 7fde3f7fe640 1 -- 172.21.15.72:0/1919652748 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde440bdb20 con 0x55c6e00a0120 2024-03-22T01:37:14.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.436+0000 7fde3f7fe640 1 -- 172.21.15.72:0/1919652748 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde440bcd80 con 0x55c6e00a0120 2024-03-22T01:37:14.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.437+0000 7fde5e06b740 1 -- 172.21.15.72:0/1919652748 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c6e00a0120 msgr2=0x55c6e00a87c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:14.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.437+0000 7fde5e06b740 1 --2- 172.21.15.72:0/1919652748 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c6e00a0120 0x55c6e00a87c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fde440ee850 tx=0x7fde4408cef0 comp rx=0 tx=0).stop 2024-03-22T01:37:14.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.437+0000 7fde5e06b740 1 -- 172.21.15.72:0/1919652748 shutdown_connections 2024-03-22T01:37:14.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.437+0000 7fde5e06b740 1 --2- 172.21.15.72:0/1919652748 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c6e00a0120 0x55c6e00a87c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:14.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.437+0000 7fde5e06b740 1 --2- 172.21.15.72:0/1919652748 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c6e00a8d00 0x55c6e00ab0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:14.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.437+0000 7fde5e06b740 1 --2- 172.21.15.72:0/1919652748 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c6e009f800 0x55c6e009fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:14.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.437+0000 7fde5e06b740 1 -- 172.21.15.72:0/1919652748 >> 172.21.15.72:0/1919652748 conn(0x55c6e0099f30 msgr2=0x55c6e009a310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:14.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.437+0000 7fde5e06b740 1 -- 172.21.15.72:0/1919652748 shutdown_connections 2024-03-22T01:37:14.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.437+0000 7fde5e06b740 1 -- 172.21.15.72:0/1919652748 wait complete. 2024-03-22T01:37:14.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.437+0000 7fde5e06b740 1 -- start start 2024-03-22T01:37:14.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.437+0000 7fde5e06b740 1 Processor -- start 2024-03-22T01:37:14.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.437+0000 7fde5e06b740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c6e009f800 0x55c6e00a5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:14.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.437+0000 7fde5e06b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c6e00a0120 0x55c6e00a6220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:14.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.437+0000 7fde5e06b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c6e00a8d00 0x55c6e00a6760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:14.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.437+0000 7fde5e06b740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55c6e00ab8b0 con 0x55c6e009f800 2024-03-22T01:37:14.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.437+0000 7fde5e06b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55c6dffbf300 con 0x55c6e00a8d00 2024-03-22T01:37:14.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.437+0000 7fde5e06b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c6e00aba30 con 0x55c6e00a0120 2024-03-22T01:37:14.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.438+0000 7fde5e06b740 10 client.0 _ll_register_callbacks cb 0x55c6dfffae30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:37:14.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.438+0000 7fde59220640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c6e009f800 0x55c6e00a5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:14.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.438+0000 7fde59220640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c6e009f800 0x55c6e00a5ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59444/0 (socket says 172.21.15.72:59444) 2024-03-22T01:37:14.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.438+0000 7fde59220640 1 -- 172.21.15.72:0/4006954678 learned_addr learned my addr 172.21.15.72:0/4006954678 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:37:14.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.438+0000 7fde59a21640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c6e00a8d00 0x55c6e00a6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:14.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.438+0000 7fde58a1f640 1 --2- 172.21.15.72:0/4006954678 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c6e00a0120 0x55c6e00a6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:14.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.438+0000 7fde59220640 1 -- 172.21.15.72:0/4006954678 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c6e00a0120 msgr2=0x55c6e00a6220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:14.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.438+0000 7fde59220640 1 --2- 172.21.15.72:0/4006954678 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c6e00a0120 0x55c6e00a6220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:14.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.438+0000 7fde59220640 1 -- 172.21.15.72:0/4006954678 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c6e00a8d00 msgr2=0x55c6e00a6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:14.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.438+0000 7fde59220640 1 --2- 172.21.15.72:0/4006954678 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c6e00a8d00 0x55c6e00a6760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:14.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.438+0000 7fde59220640 1 -- 172.21.15.72:0/4006954678 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c6dfde17d0 con 0x55c6e009f800 2024-03-22T01:37:14.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.438+0000 7fde59a21640 1 --2- 172.21.15.72:0/4006954678 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c6e00a8d00 0x55c6e00a6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:37:14.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.438+0000 7fde59220640 1 --2- 172.21.15.72:0/4006954678 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c6e009f800 0x55c6e00a5ce0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fde440bd3b0 tx=0x7fde440bcf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da47ab8a4582e5ed server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:14.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.438+0000 7fde3cff9640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:37:14.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.438+0000 7fde3cff9640 1 -- 172.21.15.72:0/4006954678 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde440bb720 con 0x55c6e009f800 2024-03-22T01:37:14.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.438+0000 7fde3cff9640 1 -- 172.21.15.72:0/4006954678 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fde440f1120 con 0x55c6e009f800 2024-03-22T01:37:14.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.438+0000 7fde5e06b740 1 -- 172.21.15.72:0/4006954678 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde440bdb20 con 0x55c6e009f800 2024-03-22T01:37:14.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.438+0000 7fde3cff9640 1 -- 172.21.15.72:0/4006954678 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fde440f1320 con 0x55c6e009f800 2024-03-22T01:37:14.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.438+0000 7fde5e06b740 10 client.15708 fetch_fsmap learned FSMap version 21 2024-03-22T01:37:14.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.438+0000 7fde5e06b740 1 -- 172.21.15.72:0/4006954678 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55c6e00ab8b0 con 0x55c6e009f800 2024-03-22T01:37:14.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.438+0000 7fde3cff9640 1 -- 172.21.15.72:0/4006954678 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde440bdb20 con 0x55c6e009f800 2024-03-22T01:37:14.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.439+0000 7fde3cff9640 1 -- 172.21.15.72:0/4006954678 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fde4403c030 con 0x55c6e009f800 2024-03-22T01:37:14.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.439+0000 7fde5e06b740 10 client.15708 fetch_fsmap finished waiting for FSMap version 21 2024-03-22T01:37:14.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.439+0000 7fde5e06b740 10 client.15708 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T01:37:14.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.439+0000 7fde5e06b740 20 client.15708 populate_metadata read hostname 'smithi072' 2024-03-22T01:37:14.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.439+0000 7fde5e06b740 1 --2- 172.21.15.72:0/4006954678 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55c6e00b8bf0 0x55c6e00bafe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:14.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.439+0000 7fde5e06b740 4 client.15708 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T01:37:14.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.439+0000 7fde5e06b740 1 -- 172.21.15.72:0/4006954678 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c6e0010e10 con 0x55c6e00b8bf0 2024-03-22T01:37:14.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.439+0000 7fde58a1f640 1 --2- 172.21.15.72:0/4006954678 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55c6e00b8bf0 0x55c6e00bafe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:14.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.439+0000 7fde3cff9640 10 client.15708 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:37:14.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.439+0000 7fde58a1f640 1 --2- 172.21.15.72:0/4006954678 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55c6e00b8bf0 0x55c6e00bafe0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ffcc1df6e459e6eb server_cookie=322f1a2a1c01c4ea in_seq=0 out_seq=0 2024-03-22T01:37:14.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.440+0000 7fde3cff9640 1 -- 172.21.15.72:0/4006954678 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c6e0010e10 con 0x55c6e00b8bf0 2024-03-22T01:37:14.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.440+0000 7fde3cff9640 10 client.15708 handle_command_reply: tid=0 2024-03-22T01:37:14.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.440+0000 7fde5e06b740 1 client.15708 shutdown 2024-03-22T01:37:14.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.440+0000 7fde5e06b740 1 -- 172.21.15.72:0/4006954678 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c6e009f800 msgr2=0x55c6e00a5ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:14.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.440+0000 7fde5e06b740 1 --2- 172.21.15.72:0/4006954678 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c6e009f800 0x55c6e00a5ce0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fde440bd3b0 tx=0x7fde440bcf60 comp rx=0 tx=0).stop 2024-03-22T01:37:14.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.440+0000 7fde5e06b740 1 -- 172.21.15.72:0/4006954678 shutdown_connections 2024-03-22T01:37:14.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.440+0000 7fde5e06b740 1 --2- 172.21.15.72:0/4006954678 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55c6e00b8bf0 0x55c6e00bafe0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:14.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.440+0000 7fde5e06b740 1 --2- 172.21.15.72:0/4006954678 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c6e00a8d00 0x55c6e00a6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:14.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.440+0000 7fde5e06b740 1 --2- 172.21.15.72:0/4006954678 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c6e00a0120 0x55c6e00a6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:14.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.440+0000 7fde3cff9640 0 client.15708 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:37:14.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.440+0000 7fde5e06b740 1 --2- 172.21.15.72:0/4006954678 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c6e009f800 0x55c6e00a5ce0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:14.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.440+0000 7fde5e06b740 1 -- 172.21.15.72:0/4006954678 >> 172.21.15.72:0/4006954678 conn(0x55c6e0099f30 msgr2=0x55c6e009bc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:14.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.441+0000 7fde5e06b740 1 -- 172.21.15.72:0/4006954678 shutdown_connections 2024-03-22T01:37:14.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.441+0000 7fde5e06b740 1 -- 172.21.15.72:0/4006954678 wait complete. 2024-03-22T01:37:14.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.441+0000 7fde5e06b740 20 client.15708 trim_cache size 0 max 16384 2024-03-22T01:37:14.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.454+0000 7fde5e06b740 1 Processor -- start 2024-03-22T01:37:14.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.454+0000 7fde5e06b740 1 -- start start 2024-03-22T01:37:14.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.454+0000 7fde5e06b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c6e009f800 0x55c6e00a4e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:14.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.454+0000 7fde5e06b740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c6e00a0120 0x55c6e00b45c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:14.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.454+0000 7fde5e06b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c6e00a8d00 0x55c6e00a6580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:14.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.454+0000 7fde5e06b740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fde541aae60 con 0x55c6e00a0120 2024-03-22T01:37:14.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.454+0000 7fde5e06b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fde541ad6e0 con 0x55c6e009f800 2024-03-22T01:37:14.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.454+0000 7fde5e06b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fde541ab4d0 con 0x55c6e00a8d00 2024-03-22T01:37:14.456 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.454+0000 7fde58a1f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c6e00a0120 0x55c6e00b45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:14.456 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.455+0000 7fde59220640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c6e009f800 0x55c6e00a4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:14.456 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.455+0000 7fde58a1f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c6e00a0120 0x55c6e00b45c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59460/0 (socket says 172.21.15.72:59460) 2024-03-22T01:37:14.456 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.455+0000 7fde59220640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c6e009f800 0x55c6e00a4e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.72:33916/0 (socket says 172.21.15.72:33916) 2024-03-22T01:37:14.456 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.455+0000 7fde58a1f640 1 -- 172.21.15.72:0/1083314055 learned_addr learned my addr 172.21.15.72:0/1083314055 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:37:14.456 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.455+0000 7fde59a21640 1 --2- 172.21.15.72:0/1083314055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c6e00a8d00 0x55c6e00a6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:14.456 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.455+0000 7fde59220640 1 -- 172.21.15.72:0/1083314055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c6e00a8d00 msgr2=0x55c6e00a6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:14.456 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.455+0000 7fde59220640 1 --2- 172.21.15.72:0/1083314055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c6e00a8d00 0x55c6e00a6580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:14.456 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.455+0000 7fde59220640 1 -- 172.21.15.72:0/1083314055 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c6e00a0120 msgr2=0x55c6e00b45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:14.456 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.455+0000 7fde59220640 1 --2- 172.21.15.72:0/1083314055 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c6e00a0120 0x55c6e00b45c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:14.456 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.455+0000 7fde59220640 1 -- 172.21.15.72:0/1083314055 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c6e0099d60 con 0x55c6e009f800 2024-03-22T01:37:14.456 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.455+0000 7fde58a1f640 1 --2- 172.21.15.72:0/1083314055 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c6e00a0120 0x55c6e00b45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:37:14.456 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.455+0000 7fde59220640 1 --2- 172.21.15.72:0/1083314055 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c6e009f800 0x55c6e00a4e40 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fde44009980 tx=0x7fde44005dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8b08062f4bb3460 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:14.456 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.455+0000 7fde3effd640 1 -- 172.21.15.72:0/1083314055 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde440b96c0 con 0x55c6e009f800 2024-03-22T01:37:14.456 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.455+0000 7fde3effd640 1 -- 172.21.15.72:0/1083314055 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde440ee790 con 0x55c6e009f800 2024-03-22T01:37:14.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.455+0000 7fde3effd640 1 -- 172.21.15.72:0/1083314055 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde440bdb20 con 0x55c6e009f800 2024-03-22T01:37:14.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.456+0000 7fde5e06b740 1 -- 172.21.15.72:0/1083314055 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c6e009f800 msgr2=0x55c6e00a4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:14.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.456+0000 7fde5e06b740 1 --2- 172.21.15.72:0/1083314055 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c6e009f800 0x55c6e00a4e40 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fde44009980 tx=0x7fde44005dc0 comp rx=0 tx=0).stop 2024-03-22T01:37:14.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.456+0000 7fde5e06b740 1 -- 172.21.15.72:0/1083314055 shutdown_connections 2024-03-22T01:37:14.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.456+0000 7fde5e06b740 1 --2- 172.21.15.72:0/1083314055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c6e00a8d00 0x55c6e00a6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:14.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.456+0000 7fde5e06b740 1 --2- 172.21.15.72:0/1083314055 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c6e00a0120 0x55c6e00b45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:14.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.456+0000 7fde5e06b740 1 --2- 172.21.15.72:0/1083314055 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c6e009f800 0x55c6e00a4e40 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:14.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.456+0000 7fde5e06b740 1 -- 172.21.15.72:0/1083314055 >> 172.21.15.72:0/1083314055 conn(0x55c6e0099f30 msgr2=0x55c6e009b360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:14.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.456+0000 7fde5e06b740 1 -- 172.21.15.72:0/1083314055 shutdown_connections 2024-03-22T01:37:14.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.456+0000 7fde5e06b740 1 -- 172.21.15.72:0/1083314055 wait complete. 2024-03-22T01:37:14.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.456+0000 7fde5e06b740 1 -- start start 2024-03-22T01:37:14.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.456+0000 7fde5e06b740 1 Processor -- start 2024-03-22T01:37:14.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.457+0000 7fde5e06b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c6e009f800 0x55c6e01170f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:14.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.457+0000 7fde5e06b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c6e00a0120 0x55c6e0117630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:14.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.457+0000 7fde5e06b740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c6e00a8d00 0x55c6e00abd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:14.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.457+0000 7fde5e06b740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fde541abba0 con 0x55c6e00a8d00 2024-03-22T01:37:14.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.457+0000 7fde5e06b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fde541ad010 con 0x55c6e009f800 2024-03-22T01:37:14.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.457+0000 7fde5e06b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fde541ac270 con 0x55c6e00a0120 2024-03-22T01:37:14.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.457+0000 7fde5e06b740 10 client.0 _ll_register_callbacks cb 0x7fde440bcd80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:37:14.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.457+0000 7fde59a21640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c6e00a8d00 0x55c6e00abd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:14.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.457+0000 7fde59220640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c6e009f800 0x55c6e01170f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:14.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.457+0000 7fde59220640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c6e009f800 0x55c6e01170f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.72:33924/0 (socket says 172.21.15.72:33924) 2024-03-22T01:37:14.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.457+0000 7fde59220640 1 -- 172.21.15.72:0/2687714243 learned_addr learned my addr 172.21.15.72:0/2687714243 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:37:14.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.457+0000 7fde58a1f640 1 --2- 172.21.15.72:0/2687714243 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c6e00a0120 0x55c6e0117630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:14.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.457+0000 7fde59a21640 1 -- 172.21.15.72:0/2687714243 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c6e00a0120 msgr2=0x55c6e0117630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:14.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.457+0000 7fde59a21640 1 --2- 172.21.15.72:0/2687714243 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c6e00a0120 0x55c6e0117630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:14.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.457+0000 7fde59a21640 1 -- 172.21.15.72:0/2687714243 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c6e009f800 msgr2=0x55c6e01170f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:14.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.457+0000 7fde59a21640 1 --2- 172.21.15.72:0/2687714243 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c6e009f800 0x55c6e01170f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:14.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.457+0000 7fde59a21640 1 -- 172.21.15.72:0/2687714243 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c6e00ac400 con 0x55c6e00a8d00 2024-03-22T01:37:14.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.457+0000 7fde59220640 1 --2- 172.21.15.72:0/2687714243 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c6e009f800 0x55c6e01170f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:37:14.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.457+0000 7fde59a21640 1 --2- 172.21.15.72:0/2687714243 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c6e00a8d00 0x55c6e00abd10 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fde4c00a6d0 tx=0x7fde4c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d21cc53c75426100 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:14.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.457+0000 7fde3cff9640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:37:14.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.457+0000 7fde3cff9640 1 -- 172.21.15.72:0/2687714243 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde4c02c030 con 0x55c6e00a8d00 2024-03-22T01:37:14.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.457+0000 7fde5e06b740 1 -- 172.21.15.72:0/2687714243 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde440ee790 con 0x55c6e00a8d00 2024-03-22T01:37:14.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.457+0000 7fde3cff9640 1 -- 172.21.15.72:0/2687714243 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fde4c004b20 con 0x55c6e00a8d00 2024-03-22T01:37:14.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.458+0000 7fde3cff9640 1 -- 172.21.15.72:0/2687714243 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fde4c023040 con 0x55c6e00a8d00 2024-03-22T01:37:14.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.458+0000 7fde5e06b740 10 client.15714 fetch_fsmap learned FSMap version 21 2024-03-22T01:37:14.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.458+0000 7fde5e06b740 1 -- 172.21.15.72:0/2687714243 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55c6e011e890 con 0x55c6e00a8d00 2024-03-22T01:37:14.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.458+0000 7fde3cff9640 1 -- 172.21.15.72:0/2687714243 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde440ee790 con 0x55c6e00a8d00 2024-03-22T01:37:14.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.458+0000 7fde3cff9640 1 -- 172.21.15.72:0/2687714243 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fde4c013460 con 0x55c6e00a8d00 2024-03-22T01:37:14.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.458+0000 7fde5e06b740 10 client.15714 fetch_fsmap finished waiting for FSMap version 21 2024-03-22T01:37:14.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.458+0000 7fde5e06b740 10 client.15714 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T01:37:14.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.458+0000 7fde5e06b740 20 client.15714 populate_metadata read hostname 'smithi072' 2024-03-22T01:37:14.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.458+0000 7fde5e06b740 1 --2- 172.21.15.72:0/2687714243 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55c6e011f090 0x55c6e01214b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:14.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.458+0000 7fde5e06b740 4 client.15714 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T01:37:14.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.458+0000 7fde5e06b740 1 -- 172.21.15.72:0/2687714243 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c6e000b160 con 0x55c6e011f090 2024-03-22T01:37:14.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.458+0000 7fde59220640 1 --2- 172.21.15.72:0/2687714243 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55c6e011f090 0x55c6e01214b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:14.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.459+0000 7fde3cff9640 10 client.15714 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:37:14.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.459+0000 7fde59220640 1 --2- 172.21.15.72:0/2687714243 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55c6e011f090 0x55c6e01214b0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a91f17f74c430d10 server_cookie=2a2a692de4159fd9 in_seq=0 out_seq=0 2024-03-22T01:37:14.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.459+0000 7fde3cff9640 1 -- 172.21.15.72:0/2687714243 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c6e000b160 con 0x55c6e011f090 2024-03-22T01:37:14.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.459+0000 7fde3cff9640 10 client.15714 handle_command_reply: tid=0 2024-03-22T01:37:14.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.459+0000 7fde5e06b740 1 client.15714 shutdown 2024-03-22T01:37:14.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.459+0000 7fde5e06b740 1 -- 172.21.15.72:0/2687714243 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c6e00a8d00 msgr2=0x55c6e00abd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:14.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.459+0000 7fde5e06b740 1 --2- 172.21.15.72:0/2687714243 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c6e00a8d00 0x55c6e00abd10 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fde4c00a6d0 tx=0x7fde4c004a40 comp rx=0 tx=0).stop 2024-03-22T01:37:14.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.459+0000 7fde5e06b740 1 -- 172.21.15.72:0/2687714243 shutdown_connections 2024-03-22T01:37:14.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.459+0000 7fde5e06b740 1 --2- 172.21.15.72:0/2687714243 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55c6e011f090 0x55c6e01214b0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:14.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.459+0000 7fde5e06b740 1 --2- 172.21.15.72:0/2687714243 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c6e00a8d00 0x55c6e00abd10 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:14.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.459+0000 7fde5e06b740 1 --2- 172.21.15.72:0/2687714243 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c6e00a0120 0x55c6e0117630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:14.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.459+0000 7fde5e06b740 1 --2- 172.21.15.72:0/2687714243 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c6e009f800 0x55c6e01170f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:14.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.459+0000 7fde5e06b740 1 -- 172.21.15.72:0/2687714243 >> 172.21.15.72:0/2687714243 conn(0x55c6e0099f30 msgr2=0x55c6e00a65c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:14.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.459+0000 7fde3cff9640 0 client.15714 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:37:14.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.459+0000 7fde5e06b740 1 -- 172.21.15.72:0/2687714243 shutdown_connections 2024-03-22T01:37:14.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.459+0000 7fde5e06b740 1 -- 172.21.15.72:0/2687714243 wait complete. 2024-03-22T01:37:14.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.459+0000 7fde5e06b740 20 client.15714 trim_cache size 0 max 16384 2024-03-22T01:37:14.465 INFO:teuthology.orchestra.run.smithi072.stdout:[] 2024-03-22T01:37:14.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.460+0000 7fde5b484640 1 -- 172.21.15.72:0/2517144905 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fde400a31b0 msgr2=0x7fde400a5670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:14.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.460+0000 7fde5b484640 1 --2- 172.21.15.72:0/2517144905 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fde400a31b0 0x7fde400a5670 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fde4c00afd0 tx=0x7fde4c00a420 comp rx=0 tx=0).stop 2024-03-22T01:37:14.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.460+0000 7fde5b484640 1 -- 172.21.15.72:0/2517144905 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde54102db0 msgr2=0x7fde54198490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:14.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.460+0000 7fde5b484640 1 --2- 172.21.15.72:0/2517144905 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde54102db0 0x7fde54198490 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fde44037040 tx=0x7fde440346f0 comp rx=0 tx=0).stop 2024-03-22T01:37:14.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.460+0000 7fde5b484640 1 -- 172.21.15.72:0/2517144905 shutdown_connections 2024-03-22T01:37:14.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.460+0000 7fde5b484640 1 --2- 172.21.15.72:0/2517144905 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fde400a31b0 0x7fde400a5670 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:14.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.460+0000 7fde5b484640 1 --2- 172.21.15.72:0/2517144905 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde541036d0 0x7fde541989d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:14.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.460+0000 7fde5b484640 1 --2- 172.21.15.72:0/2517144905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde5410c180 0x7fde5419cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:14.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.460+0000 7fde5b484640 1 --2- 172.21.15.72:0/2517144905 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde54102db0 0x7fde54198490 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:14.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.460+0000 7fde5b484640 1 -- 172.21.15.72:0/2517144905 >> 172.21.15.72:0/2517144905 conn(0x7fde54077c40 msgr2=0x7fde540fedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:14.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.460+0000 7fde5b484640 1 -- 172.21.15.72:0/2517144905 shutdown_connections 2024-03-22T01:37:14.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:14.460+0000 7fde5b484640 1 -- 172.21.15.72:0/2517144905 wait complete. 2024-03-22T01:37:15.175 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:37:14 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/2773683432' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:37:15.335 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:37:14 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/2773683432' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:37:15.357 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:37:14 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/2773683432' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:37:15.470 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-22T01:37:15.471 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T01:37:15.471 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-22T01:37:15.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.579+0000 7f0d34df7640 1 -- 172.21.15.72:0/3925539554 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d3010c110 msgr2=0x7f0d3010e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:15.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.579+0000 7f0d34df7640 1 --2- 172.21.15.72:0/3925539554 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d3010c110 0x7f0d3010e5b0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f0d28009ca0 tx=0x7f0d2802c180 comp rx=0 tx=0).stop 2024-03-22T01:37:15.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.579+0000 7f0d34df7640 1 -- 172.21.15.72:0/3925539554 shutdown_connections 2024-03-22T01:37:15.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.579+0000 7f0d34df7640 1 --2- 172.21.15.72:0/3925539554 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d30103610 0x7f0d3010baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.579+0000 7f0d34df7640 1 --2- 172.21.15.72:0/3925539554 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d3010c110 0x7f0d3010e5b0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.579+0000 7f0d34df7640 1 --2- 172.21.15.72:0/3925539554 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d30102cf0 0x7f0d301030d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.579+0000 7f0d34df7640 1 -- 172.21.15.72:0/3925539554 >> 172.21.15.72:0/3925539554 conn(0x7f0d300fc4f0 msgr2=0x7f0d300fe910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:15.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.579+0000 7f0d34df7640 1 -- 172.21.15.72:0/3925539554 shutdown_connections 2024-03-22T01:37:15.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.579+0000 7f0d34df7640 1 -- 172.21.15.72:0/3925539554 wait complete. 2024-03-22T01:37:15.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.580+0000 7f0d34df7640 1 Processor -- start 2024-03-22T01:37:15.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.580+0000 7f0d34df7640 1 -- start start 2024-03-22T01:37:15.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.580+0000 7f0d34df7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d30102cf0 0x7f0d30198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:15.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.580+0000 7f0d34df7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d30103610 0x7f0d30198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:15.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.580+0000 7f0d34df7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d3010c110 0x7f0d3019cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:15.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.580+0000 7f0d34df7640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f0d30111f10 con 0x7f0d30102cf0 2024-03-22T01:37:15.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.580+0000 7f0d34df7640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f0d30111d90 con 0x7f0d3010c110 2024-03-22T01:37:15.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.580+0000 7f0d34df7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0d30112090 con 0x7f0d30103610 2024-03-22T01:37:15.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.580+0000 7f0d2e59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d30102cf0 0x7f0d30198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:15.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.580+0000 7f0d2e59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d30102cf0 0x7f0d30198430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59490/0 (socket says 172.21.15.72:59490) 2024-03-22T01:37:15.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.580+0000 7f0d2e59c640 1 -- 172.21.15.72:0/2481282508 learned_addr learned my addr 172.21.15.72:0/2481282508 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:37:15.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.580+0000 7f0d2ed9d640 1 --2- 172.21.15.72:0/2481282508 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d3010c110 0x7f0d3019cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:15.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.580+0000 7f0d27fff640 1 --2- 172.21.15.72:0/2481282508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d30103610 0x7f0d30198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:15.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.580+0000 7f0d2e59c640 1 -- 172.21.15.72:0/2481282508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d30103610 msgr2=0x7f0d30198970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:15.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.580+0000 7f0d2e59c640 1 --2- 172.21.15.72:0/2481282508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d30103610 0x7f0d30198970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.580+0000 7f0d2e59c640 1 -- 172.21.15.72:0/2481282508 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d3010c110 msgr2=0x7f0d3019cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:15.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.580+0000 7f0d2e59c640 1 --2- 172.21.15.72:0/2481282508 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d3010c110 0x7f0d3019cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.580+0000 7f0d2e59c640 1 -- 172.21.15.72:0/2481282508 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d3019d3d0 con 0x7f0d30102cf0 2024-03-22T01:37:15.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.580+0000 7f0d27fff640 1 --2- 172.21.15.72:0/2481282508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d30103610 0x7f0d30198970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T01:37:15.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.580+0000 7f0d2e59c640 1 --2- 172.21.15.72:0/2481282508 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d30102cf0 0x7f0d30198430 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f0d2000d770 tx=0x7f0d2000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:15.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.580+0000 7f0d277fe640 1 -- 172.21.15.72:0/2481282508 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d20017070 con 0x7f0d30102cf0 2024-03-22T01:37:15.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.581+0000 7f0d34df7640 1 -- 172.21.15.72:0/2481282508 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d3019d6c0 con 0x7f0d30102cf0 2024-03-22T01:37:15.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.581+0000 7f0d277fe640 1 -- 172.21.15.72:0/2481282508 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d20005500 con 0x7f0d30102cf0 2024-03-22T01:37:15.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.581+0000 7f0d277fe640 1 -- 172.21.15.72:0/2481282508 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d20004e00 con 0x7f0d30102cf0 2024-03-22T01:37:15.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.581+0000 7f0d34df7640 1 -- 172.21.15.72:0/2481282508 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d301a4f50 con 0x7f0d30102cf0 2024-03-22T01:37:15.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.583+0000 7f0d277fe640 1 -- 172.21.15.72:0/2481282508 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 24) ==== 138087+0+0 (secure 0 0 0) 0x7f0d20004fa0 con 0x7f0d30102cf0 2024-03-22T01:37:15.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.583+0000 7f0d379e3740 1 Processor -- start 2024-03-22T01:37:15.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.584+0000 7f0d379e3740 1 -- start start 2024-03-22T01:37:15.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.584+0000 7f0d379e3740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f86d62eba0 0x55f86d62ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:15.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.584+0000 7f0d379e3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f86d62f4c0 0x55f86d637b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:15.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.584+0000 7f0d379e3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f86d6380a0 0x55f86d63a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:15.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.584+0000 7f0d379e3740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55f86d592570 con 0x55f86d62eba0 2024-03-22T01:37:15.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.584+0000 7f0d379e3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55f86d53ae10 con 0x55f86d62f4c0 2024-03-22T01:37:15.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.584+0000 7f0d379e3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f86d58e520 con 0x55f86d6380a0 2024-03-22T01:37:15.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.584+0000 7f0d277fe640 1 --2- 172.21.15.72:0/2481282508 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f0d1c0a31b0 0x7f0d1c0a5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:15.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.584+0000 7f0d2ed9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f86d62eba0 0x55f86d62ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:15.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.584+0000 7f0d2e59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f86d62f4c0 0x55f86d637b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:15.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.584+0000 7f0d2ed9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f86d62eba0 0x55f86d62ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59506/0 (socket says 172.21.15.72:59506) 2024-03-22T01:37:15.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.584+0000 7f0d2ed9d640 1 -- 172.21.15.72:0/1054786145 learned_addr learned my addr 172.21.15.72:0/1054786145 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:37:15.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.584+0000 7f0d27fff640 1 --2- 172.21.15.72:0/1054786145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f86d6380a0 0x55f86d63a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:15.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.584+0000 7f0d2ed9d640 1 --2- 172.21.15.72:0/2481282508 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f0d1c0a31b0 0x7f0d1c0a5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:15.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.584+0000 7f0d277fe640 1 -- 172.21.15.72:0/2481282508 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f0d200cf8c0 con 0x7f0d30102cf0 2024-03-22T01:37:15.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.584+0000 7f0d2ed9d640 1 -- 172.21.15.72:0/1054786145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f86d6380a0 msgr2=0x55f86d63a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:15.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.584+0000 7f0d2ed9d640 1 --2- 172.21.15.72:0/1054786145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f86d6380a0 0x55f86d63a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.584+0000 7f0d2ed9d640 1 -- 172.21.15.72:0/1054786145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f86d62f4c0 msgr2=0x55f86d637b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:15.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.584+0000 7f0d2ed9d640 1 --2- 172.21.15.72:0/1054786145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f86d62f4c0 0x55f86d637b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.584+0000 7f0d2ed9d640 1 -- 172.21.15.72:0/1054786145 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f86d36e7e0 con 0x55f86d62eba0 2024-03-22T01:37:15.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.584+0000 7f0d2e59c640 1 --2- 172.21.15.72:0/1054786145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f86d62f4c0 0x55f86d637b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:37:15.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.584+0000 7f0d27fff640 1 --2- 172.21.15.72:0/1054786145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f86d6380a0 0x55f86d63a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:37:15.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.585+0000 7f0d2ed9d640 1 --2- 172.21.15.72:0/1054786145 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f86d62eba0 0x55f86d62ef80 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f0d28037040 tx=0x7f0d28003b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4b00b2ef09fa2d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:15.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.585+0000 7f0d24ff9640 1 -- 172.21.15.72:0/1054786145 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d28050070 con 0x55f86d62eba0 2024-03-22T01:37:15.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.585+0000 7f0d24ff9640 1 -- 172.21.15.72:0/1054786145 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d28004610 con 0x55f86d62eba0 2024-03-22T01:37:15.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.585+0000 7f0d24ff9640 1 -- 172.21.15.72:0/1054786145 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d28039040 con 0x55f86d62eba0 2024-03-22T01:37:15.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.585+0000 7f0d379e3740 1 -- 172.21.15.72:0/1054786145 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f86d62eba0 msgr2=0x55f86d62ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:15.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.585+0000 7f0d2ed9d640 1 --2- 172.21.15.72:0/2481282508 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f0d1c0a31b0 0x7f0d1c0a5670 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0d2804d000 tx=0x7f0d28048000 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:15.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.585+0000 7f0d379e3740 1 --2- 172.21.15.72:0/1054786145 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f86d62eba0 0x55f86d62ef80 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f0d28037040 tx=0x7f0d28003b00 comp rx=0 tx=0).stop 2024-03-22T01:37:15.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.586+0000 7f0d379e3740 1 -- 172.21.15.72:0/1054786145 shutdown_connections 2024-03-22T01:37:15.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.586+0000 7f0d379e3740 1 --2- 172.21.15.72:0/1054786145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f86d62f4c0 0x55f86d637b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.586+0000 7f0d379e3740 1 --2- 172.21.15.72:0/1054786145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f86d6380a0 0x55f86d63a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.586+0000 7f0d379e3740 1 --2- 172.21.15.72:0/1054786145 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f86d62eba0 0x55f86d62ef80 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.586+0000 7f0d379e3740 1 -- 172.21.15.72:0/1054786145 >> 172.21.15.72:0/1054786145 conn(0x55f86d6292d0 msgr2=0x55f86d6296b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:15.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.586+0000 7f0d379e3740 1 -- 172.21.15.72:0/1054786145 shutdown_connections 2024-03-22T01:37:15.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.586+0000 7f0d379e3740 1 -- 172.21.15.72:0/1054786145 wait complete. 2024-03-22T01:37:15.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.586+0000 7f0d379e3740 1 -- start start 2024-03-22T01:37:15.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.586+0000 7f0d379e3740 1 Processor -- start 2024-03-22T01:37:15.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.586+0000 7f0d379e3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f86d62eba0 0x55f86d6379f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:15.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.586+0000 7f0d379e3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f86d62f4c0 0x55f86d634d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:15.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.586+0000 7f0d379e3740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f86d6380a0 0x55f86d6352e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:15.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.586+0000 7f0d379e3740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55f86d63ae90 con 0x55f86d6380a0 2024-03-22T01:37:15.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.586+0000 7f0d379e3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55f86d54e110 con 0x55f86d62eba0 2024-03-22T01:37:15.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.586+0000 7f0d379e3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f86d63b010 con 0x55f86d62f4c0 2024-03-22T01:37:15.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.586+0000 7f0d379e3740 10 client.0 _ll_register_callbacks cb 0x55f86d58abe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:37:15.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.586+0000 7f0d2ed9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f86d6380a0 0x55f86d6352e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:15.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.586+0000 7f0d2ed9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f86d6380a0 0x55f86d6352e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59520/0 (socket says 172.21.15.72:59520) 2024-03-22T01:37:15.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.586+0000 7f0d2ed9d640 1 -- 172.21.15.72:0/3067404221 learned_addr learned my addr 172.21.15.72:0/3067404221 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:37:15.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.586+0000 7f0d2e59c640 1 --2- 172.21.15.72:0/3067404221 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f86d62eba0 0x55f86d6379f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:15.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.587+0000 7f0d27fff640 1 --2- 172.21.15.72:0/3067404221 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f86d62f4c0 0x55f86d634d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:15.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.587+0000 7f0d2ed9d640 1 -- 172.21.15.72:0/3067404221 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f86d62f4c0 msgr2=0x55f86d634d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:15.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.587+0000 7f0d2ed9d640 1 --2- 172.21.15.72:0/3067404221 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f86d62f4c0 0x55f86d634d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.587+0000 7f0d2ed9d640 1 -- 172.21.15.72:0/3067404221 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f86d62eba0 msgr2=0x55f86d6379f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:15.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.587+0000 7f0d2ed9d640 1 --2- 172.21.15.72:0/3067404221 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f86d62eba0 0x55f86d6379f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.587+0000 7f0d2ed9d640 1 -- 172.21.15.72:0/3067404221 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f86d3cef70 con 0x55f86d6380a0 2024-03-22T01:37:15.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.587+0000 7f0d2ed9d640 1 --2- 172.21.15.72:0/3067404221 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f86d6380a0 0x55f86d6352e0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f0d2802c790 tx=0x7f0d28003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=715c137fe299df86 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:15.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.587+0000 7f0d15ffb640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:37:15.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.587+0000 7f0d15ffb640 1 -- 172.21.15.72:0/3067404221 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d28058020 con 0x55f86d6380a0 2024-03-22T01:37:15.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.587+0000 7f0d15ffb640 1 -- 172.21.15.72:0/3067404221 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f0d280347e0 con 0x55f86d6380a0 2024-03-22T01:37:15.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.587+0000 7f0d379e3740 1 -- 172.21.15.72:0/3067404221 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0d28004610 con 0x55f86d6380a0 2024-03-22T01:37:15.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.587+0000 7f0d15ffb640 1 -- 172.21.15.72:0/3067404221 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f0d28063050 con 0x55f86d6380a0 2024-03-22T01:37:15.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.587+0000 7f0d379e3740 10 client.15726 fetch_fsmap learned FSMap version 21 2024-03-22T01:37:15.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.587+0000 7f0d379e3740 1 -- 172.21.15.72:0/3067404221 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55f86d266170 con 0x55f86d6380a0 2024-03-22T01:37:15.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.587+0000 7f0d15ffb640 1 -- 172.21.15.72:0/3067404221 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d28004610 con 0x55f86d6380a0 2024-03-22T01:37:15.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.587+0000 7f0d15ffb640 1 -- 172.21.15.72:0/3067404221 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f0d2803f070 con 0x55f86d6380a0 2024-03-22T01:37:15.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.587+0000 7f0d379e3740 10 client.15726 fetch_fsmap finished waiting for FSMap version 21 2024-03-22T01:37:15.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.587+0000 7f0d379e3740 10 client.15726 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T01:37:15.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.587+0000 7f0d379e3740 20 client.15726 populate_metadata read hostname 'smithi072' 2024-03-22T01:37:15.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.588+0000 7f0d379e3740 1 --2- 172.21.15.72:0/3067404221 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55f86d647fe0 0x55f86d64a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:15.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.588+0000 7f0d379e3740 4 client.15726 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T01:37:15.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.588+0000 7f0d379e3740 1 -- 172.21.15.72:0/3067404221 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f86d59fab0 con 0x55f86d647fe0 2024-03-22T01:37:15.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.588+0000 7f0d2e59c640 1 --2- 172.21.15.72:0/3067404221 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55f86d647fe0 0x55f86d64a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:15.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.588+0000 7f0d15ffb640 10 client.15726 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:37:15.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.588+0000 7f0d2e59c640 1 --2- 172.21.15.72:0/3067404221 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55f86d647fe0 0x55f86d64a3d0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8178f656530adcb server_cookie=ed140d464a9a6d8d in_seq=0 out_seq=0 2024-03-22T01:37:15.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.589+0000 7f0d15ffb640 1 -- 172.21.15.72:0/3067404221 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f86d59fab0 con 0x55f86d647fe0 2024-03-22T01:37:15.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.589+0000 7f0d15ffb640 10 client.15726 handle_command_reply: tid=0 2024-03-22T01:37:15.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.589+0000 7f0d379e3740 1 client.15726 shutdown 2024-03-22T01:37:15.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.589+0000 7f0d379e3740 1 -- 172.21.15.72:0/3067404221 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f86d6380a0 msgr2=0x55f86d6352e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:15.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.589+0000 7f0d379e3740 1 --2- 172.21.15.72:0/3067404221 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f86d6380a0 0x55f86d6352e0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f0d2802c790 tx=0x7f0d28003f90 comp rx=0 tx=0).stop 2024-03-22T01:37:15.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.589+0000 7f0d379e3740 1 -- 172.21.15.72:0/3067404221 shutdown_connections 2024-03-22T01:37:15.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.589+0000 7f0d379e3740 1 --2- 172.21.15.72:0/3067404221 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55f86d647fe0 0x55f86d64a3d0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.589+0000 7f0d379e3740 1 --2- 172.21.15.72:0/3067404221 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f86d6380a0 0x55f86d6352e0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.589+0000 7f0d379e3740 1 --2- 172.21.15.72:0/3067404221 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f86d62f4c0 0x55f86d634d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.589+0000 7f0d15ffb640 0 client.15726 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:37:15.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.589+0000 7f0d379e3740 1 --2- 172.21.15.72:0/3067404221 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f86d62eba0 0x55f86d6379f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.589+0000 7f0d379e3740 1 -- 172.21.15.72:0/3067404221 >> 172.21.15.72:0/3067404221 conn(0x55f86d6292d0 msgr2=0x55f86d62b020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:15.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.589+0000 7f0d379e3740 1 -- 172.21.15.72:0/3067404221 shutdown_connections 2024-03-22T01:37:15.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.589+0000 7f0d379e3740 1 -- 172.21.15.72:0/3067404221 wait complete. 2024-03-22T01:37:15.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.589+0000 7f0d379e3740 20 client.15726 trim_cache size 0 max 16384 2024-03-22T01:37:15.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.606+0000 7f0d379e3740 1 Processor -- start 2024-03-22T01:37:15.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.606+0000 7f0d379e3740 1 -- start start 2024-03-22T01:37:15.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.606+0000 7f0d379e3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f86d62eba0 0x55f86d643c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:15.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.606+0000 7f0d379e3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f86d62f4c0 0x55f86d64faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:15.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.606+0000 7f0d379e3740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f86d6380a0 0x55f86d6374b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:15.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.606+0000 7f0d379e3740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f0d301acf90 con 0x55f86d6380a0 2024-03-22T01:37:15.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.606+0000 7f0d379e3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f0d301af810 con 0x55f86d62f4c0 2024-03-22T01:37:15.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.606+0000 7f0d379e3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0d301ad600 con 0x55f86d62eba0 2024-03-22T01:37:15.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.606+0000 7f0d2ed9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f86d6380a0 0x55f86d6374b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:15.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.606+0000 7f0d2ed9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f86d6380a0 0x55f86d6374b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59524/0 (socket says 172.21.15.72:59524) 2024-03-22T01:37:15.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.606+0000 7f0d2ed9d640 1 -- 172.21.15.72:0/3481983154 learned_addr learned my addr 172.21.15.72:0/3481983154 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:37:15.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.606+0000 7f0d27fff640 1 --2- 172.21.15.72:0/3481983154 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f86d62f4c0 0x55f86d64faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:15.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.606+0000 7f0d2ed9d640 1 -- 172.21.15.72:0/3481983154 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f86d62eba0 msgr2=0x55f86d643c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:15.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.606+0000 7f0d2e59c640 1 --2- 172.21.15.72:0/3481983154 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f86d62eba0 0x55f86d643c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:15.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.607+0000 7f0d2ed9d640 1 --2- 172.21.15.72:0/3481983154 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f86d62eba0 0x55f86d643c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.607+0000 7f0d2ed9d640 1 -- 172.21.15.72:0/3481983154 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f86d62f4c0 msgr2=0x55f86d64faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:15.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.607+0000 7f0d2ed9d640 1 --2- 172.21.15.72:0/3481983154 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f86d62f4c0 0x55f86d64faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.607+0000 7f0d2ed9d640 1 -- 172.21.15.72:0/3481983154 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f86d62df50 con 0x55f86d6380a0 2024-03-22T01:37:15.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.607+0000 7f0d2e59c640 1 --2- 172.21.15.72:0/3481983154 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f86d62eba0 0x55f86d643c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T01:37:15.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.607+0000 7f0d2ed9d640 1 --2- 172.21.15.72:0/3481983154 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f86d6380a0 0x55f86d6374b0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f0d28002230 tx=0x7f0d28009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6d954b2b12d714 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:15.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.607+0000 7f0d17fff640 1 -- 172.21.15.72:0/3481983154 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d28058040 con 0x55f86d6380a0 2024-03-22T01:37:15.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.607+0000 7f0d17fff640 1 -- 172.21.15.72:0/3481983154 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d28003b00 con 0x55f86d6380a0 2024-03-22T01:37:15.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.607+0000 7f0d17fff640 1 -- 172.21.15.72:0/3481983154 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d28039070 con 0x55f86d6380a0 2024-03-22T01:37:15.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.607+0000 7f0d379e3740 1 -- 172.21.15.72:0/3481983154 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f86d6380a0 msgr2=0x55f86d6374b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:15.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.607+0000 7f0d379e3740 1 --2- 172.21.15.72:0/3481983154 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f86d6380a0 0x55f86d6374b0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f0d28002230 tx=0x7f0d28009b60 comp rx=0 tx=0).stop 2024-03-22T01:37:15.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.607+0000 7f0d379e3740 1 -- 172.21.15.72:0/3481983154 shutdown_connections 2024-03-22T01:37:15.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.607+0000 7f0d379e3740 1 --2- 172.21.15.72:0/3481983154 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f86d62f4c0 0x55f86d64faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.607+0000 7f0d379e3740 1 --2- 172.21.15.72:0/3481983154 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f86d6380a0 0x55f86d6374b0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.607+0000 7f0d379e3740 1 --2- 172.21.15.72:0/3481983154 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f86d62eba0 0x55f86d643c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.607+0000 7f0d379e3740 1 -- 172.21.15.72:0/3481983154 >> 172.21.15.72:0/3481983154 conn(0x55f86d6292d0 msgr2=0x55f86d6283f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:15.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.607+0000 7f0d379e3740 1 -- 172.21.15.72:0/3481983154 shutdown_connections 2024-03-22T01:37:15.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.607+0000 7f0d379e3740 1 -- 172.21.15.72:0/3481983154 wait complete. 2024-03-22T01:37:15.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.608+0000 7f0d379e3740 1 -- start start 2024-03-22T01:37:15.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.608+0000 7f0d379e3740 1 Processor -- start 2024-03-22T01:37:15.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.608+0000 7f0d379e3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f86d62eba0 0x55f86d6318d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:15.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.608+0000 7f0d379e3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f86d62f4c0 0x55f86d631e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:15.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.608+0000 7f0d379e3740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f86d6380a0 0x55f86d632350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:15.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.608+0000 7f0d379e3740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f0d301adcd0 con 0x55f86d6380a0 2024-03-22T01:37:15.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.608+0000 7f0d379e3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f0d301af140 con 0x55f86d62eba0 2024-03-22T01:37:15.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.608+0000 7f0d379e3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0d301ae3a0 con 0x55f86d62f4c0 2024-03-22T01:37:15.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.608+0000 7f0d379e3740 10 client.0 _ll_register_callbacks cb 0x7f0d2804e070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:37:15.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.608+0000 7f0d2ed9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f86d6380a0 0x55f86d632350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:15.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.608+0000 7f0d2ed9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f86d6380a0 0x55f86d632350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59528/0 (socket says 172.21.15.72:59528) 2024-03-22T01:37:15.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.608+0000 7f0d2ed9d640 1 -- 172.21.15.72:0/1860070546 learned_addr learned my addr 172.21.15.72:0/1860070546 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:37:15.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.608+0000 7f0d2e59c640 1 --2- 172.21.15.72:0/1860070546 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f86d62eba0 0x55f86d6318d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:15.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.608+0000 7f0d2ed9d640 1 -- 172.21.15.72:0/1860070546 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f86d62f4c0 msgr2=0x55f86d631e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:15.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.608+0000 7f0d2ed9d640 1 --2- 172.21.15.72:0/1860070546 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f86d62f4c0 0x55f86d631e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.608+0000 7f0d2ed9d640 1 -- 172.21.15.72:0/1860070546 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f86d62eba0 msgr2=0x55f86d6318d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:15.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.608+0000 7f0d2ed9d640 1 --2- 172.21.15.72:0/1860070546 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f86d62eba0 0x55f86d6318d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.608+0000 7f0d2ed9d640 1 -- 172.21.15.72:0/1860070546 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f86d6ad120 con 0x55f86d6380a0 2024-03-22T01:37:15.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.608+0000 7f0d2ed9d640 1 --2- 172.21.15.72:0/1860070546 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f86d6380a0 0x55f86d632350 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f0d2802c6c0 tx=0x7f0d28009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b3c91033db7c7a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:15.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.608+0000 7f0d15ffb640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:37:15.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.609+0000 7f0d15ffb640 1 -- 172.21.15.72:0/1860070546 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d28058070 con 0x55f86d6380a0 2024-03-22T01:37:15.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.609+0000 7f0d15ffb640 1 -- 172.21.15.72:0/1860070546 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f0d28054a70 con 0x55f86d6380a0 2024-03-22T01:37:15.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.609+0000 7f0d379e3740 1 -- 172.21.15.72:0/1860070546 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0d28003b00 con 0x55f86d6380a0 2024-03-22T01:37:15.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.609+0000 7f0d15ffb640 1 -- 172.21.15.72:0/1860070546 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f0d28039070 con 0x55f86d6380a0 2024-03-22T01:37:15.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.609+0000 7f0d379e3740 10 client.15732 fetch_fsmap learned FSMap version 21 2024-03-22T01:37:15.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.609+0000 7f0d379e3740 1 -- 172.21.15.72:0/1860070546 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55f86d6ae2f0 con 0x55f86d6380a0 2024-03-22T01:37:15.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.609+0000 7f0d15ffb640 1 -- 172.21.15.72:0/1860070546 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d28003b00 con 0x55f86d6380a0 2024-03-22T01:37:15.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.609+0000 7f0d15ffb640 1 -- 172.21.15.72:0/1860070546 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f0d2803f070 con 0x55f86d6380a0 2024-03-22T01:37:15.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.609+0000 7f0d379e3740 10 client.15732 fetch_fsmap finished waiting for FSMap version 21 2024-03-22T01:37:15.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.609+0000 7f0d379e3740 10 client.15732 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T01:37:15.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.609+0000 7f0d379e3740 20 client.15732 populate_metadata read hostname 'smithi072' 2024-03-22T01:37:15.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.609+0000 7f0d379e3740 1 --2- 172.21.15.72:0/1860070546 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55f86d647fe0 0x55f86d6b08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:15.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.609+0000 7f0d379e3740 4 client.15732 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-22T01:37:15.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.609+0000 7f0d379e3740 1 -- 172.21.15.72:0/1860070546 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55f86d599e00 con 0x55f86d647fe0 2024-03-22T01:37:15.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.609+0000 7f0d2e59c640 1 --2- 172.21.15.72:0/1860070546 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55f86d647fe0 0x55f86d6b08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:15.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.610+0000 7f0d15ffb640 10 client.15732 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:37:15.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.610+0000 7f0d2e59c640 1 --2- 172.21.15.72:0/1860070546 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55f86d647fe0 0x55f86d6b08b0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6b272c5185023a37 server_cookie=f309bbc1bbe85783 in_seq=0 out_seq=0 2024-03-22T01:37:15.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.611+0000 7f0d15ffb640 1 -- 172.21.15.72:0/1860070546 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55f86d599e00 con 0x55f86d647fe0 2024-03-22T01:37:15.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.611+0000 7f0d15ffb640 10 client.15732 handle_command_reply: tid=0 2024-03-22T01:37:15.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.611+0000 7f0d379e3740 1 client.15732 shutdown 2024-03-22T01:37:15.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.611+0000 7f0d379e3740 1 -- 172.21.15.72:0/1860070546 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f86d6380a0 msgr2=0x55f86d632350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:15.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.611+0000 7f0d379e3740 1 --2- 172.21.15.72:0/1860070546 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f86d6380a0 0x55f86d632350 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f0d2802c6c0 tx=0x7f0d28009360 comp rx=0 tx=0).stop 2024-03-22T01:37:15.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.611+0000 7f0d379e3740 1 -- 172.21.15.72:0/1860070546 shutdown_connections 2024-03-22T01:37:15.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.611+0000 7f0d379e3740 1 --2- 172.21.15.72:0/1860070546 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55f86d647fe0 0x55f86d6b08b0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.611+0000 7f0d379e3740 1 --2- 172.21.15.72:0/1860070546 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f86d6380a0 0x55f86d632350 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.611+0000 7f0d15ffb640 0 client.15732 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:37:15.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.611+0000 7f0d379e3740 1 --2- 172.21.15.72:0/1860070546 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f86d62f4c0 0x55f86d631e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.611+0000 7f0d379e3740 1 --2- 172.21.15.72:0/1860070546 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f86d62eba0 0x55f86d6318d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.611+0000 7f0d379e3740 1 -- 172.21.15.72:0/1860070546 >> 172.21.15.72:0/1860070546 conn(0x55f86d6292d0 msgr2=0x55f86d634610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:15.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.611+0000 7f0d379e3740 1 -- 172.21.15.72:0/1860070546 shutdown_connections 2024-03-22T01:37:15.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.611+0000 7f0d379e3740 1 -- 172.21.15.72:0/1860070546 wait complete. 2024-03-22T01:37:15.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.611+0000 7f0d379e3740 20 client.15732 trim_cache size 0 max 16384 2024-03-22T01:37:15.617 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-22T01:37:15.617 INFO:teuthology.orchestra.run.smithi072.stdout: "return_code": 0, 2024-03-22T01:37:15.617 INFO:teuthology.orchestra.run.smithi072.stdout: "scrub_tag": "3f4326e8-5739-4f91-8e04-25bea8d7f196", 2024-03-22T01:37:15.617 INFO:teuthology.orchestra.run.smithi072.stdout: "mode": "asynchronous" 2024-03-22T01:37:15.617 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-22T01:37:15.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.612+0000 7f0d34df7640 1 -- 172.21.15.72:0/2481282508 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f0d1c0a31b0 msgr2=0x7f0d1c0a5670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:15.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.612+0000 7f0d34df7640 1 --2- 172.21.15.72:0/2481282508 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f0d1c0a31b0 0x7f0d1c0a5670 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0d2804d000 tx=0x7f0d28048000 comp rx=0 tx=0).stop 2024-03-22T01:37:15.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.612+0000 7f0d34df7640 1 -- 172.21.15.72:0/2481282508 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d30102cf0 msgr2=0x7f0d30198430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:15.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.612+0000 7f0d34df7640 1 --2- 172.21.15.72:0/2481282508 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d30102cf0 0x7f0d30198430 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f0d2000d770 tx=0x7f0d2000dc30 comp rx=0 tx=0).stop 2024-03-22T01:37:15.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.612+0000 7f0d34df7640 1 -- 172.21.15.72:0/2481282508 shutdown_connections 2024-03-22T01:37:15.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.612+0000 7f0d34df7640 1 --2- 172.21.15.72:0/2481282508 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f0d1c0a31b0 0x7f0d1c0a5670 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.612+0000 7f0d34df7640 1 --2- 172.21.15.72:0/2481282508 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d3010c110 0x7f0d3019cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.612+0000 7f0d34df7640 1 --2- 172.21.15.72:0/2481282508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d30103610 0x7f0d30198970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.612+0000 7f0d34df7640 1 --2- 172.21.15.72:0/2481282508 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d30102cf0 0x7f0d30198430 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.612+0000 7f0d34df7640 1 -- 172.21.15.72:0/2481282508 >> 172.21.15.72:0/2481282508 conn(0x7f0d300fc4f0 msgr2=0x7f0d300fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:15.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.612+0000 7f0d34df7640 1 -- 172.21.15.72:0/2481282508 shutdown_connections 2024-03-22T01:37:15.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.612+0000 7f0d34df7640 1 -- 172.21.15.72:0/2481282508 wait complete. 2024-03-22T01:37:15.621 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T01:37:15.621 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T01:37:15.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.726+0000 7fdba5866640 1 -- 172.21.15.72:0/2939660607 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdba0107c00 msgr2=0x7fdba0107fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:15.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.726+0000 7fdba5866640 1 --2- 172.21.15.72:0/2939660607 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdba0107c00 0x7fdba0107fe0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fdb90009890 tx=0x7fdb9002bb70 comp rx=0 tx=0).stop 2024-03-22T01:37:15.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.726+0000 7fdba5866640 1 -- 172.21.15.72:0/2939660607 shutdown_connections 2024-03-22T01:37:15.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.726+0000 7fdba5866640 1 --2- 172.21.15.72:0/2939660607 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdba0102500 0x7fdba010e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.726+0000 7fdba5866640 1 --2- 172.21.15.72:0/2939660607 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdba0101b40 0x7fdba0101fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.726+0000 7fdba5866640 1 --2- 172.21.15.72:0/2939660607 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdba0107c00 0x7fdba0107fe0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.726+0000 7fdba5866640 1 -- 172.21.15.72:0/2939660607 >> 172.21.15.72:0/2939660607 conn(0x7fdba00fd630 msgr2=0x7fdba00ffa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:15.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.726+0000 7fdba5866640 1 -- 172.21.15.72:0/2939660607 shutdown_connections 2024-03-22T01:37:15.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.726+0000 7fdba5866640 1 -- 172.21.15.72:0/2939660607 wait complete. 2024-03-22T01:37:15.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.727+0000 7fdba5866640 1 Processor -- start 2024-03-22T01:37:15.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.727+0000 7fdba5866640 1 -- start start 2024-03-22T01:37:15.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.727+0000 7fdba5866640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdba0101b40 0x7fdba01983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:15.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.727+0000 7fdba5866640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdba0102500 0x7fdba0198920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:15.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.727+0000 7fdba5866640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdba0107c00 0x7fdba019cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:15.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.727+0000 7fdba5866640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fdba0111f40 con 0x7fdba0101b40 2024-03-22T01:37:15.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.727+0000 7fdba5866640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fdba0111dc0 con 0x7fdba0107c00 2024-03-22T01:37:15.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.727+0000 7fdba5866640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdba01120c0 con 0x7fdba0102500 2024-03-22T01:37:15.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.727+0000 7fdb9effd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdba0101b40 0x7fdba01983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:15.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.727+0000 7fdb9effd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdba0101b40 0x7fdba01983e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59552/0 (socket says 172.21.15.72:59552) 2024-03-22T01:37:15.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.727+0000 7fdb9effd640 1 -- 172.21.15.72:0/4128470256 learned_addr learned my addr 172.21.15.72:0/4128470256 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:37:15.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.727+0000 7fdb9f7fe640 1 --2- 172.21.15.72:0/4128470256 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdba0107c00 0x7fdba019cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:15.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.727+0000 7fdb97fff640 1 --2- 172.21.15.72:0/4128470256 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdba0102500 0x7fdba0198920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:15.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.727+0000 7fdb9effd640 1 -- 172.21.15.72:0/4128470256 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdba0102500 msgr2=0x7fdba0198920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:15.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.727+0000 7fdb9effd640 1 --2- 172.21.15.72:0/4128470256 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdba0102500 0x7fdba0198920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.727+0000 7fdb9effd640 1 -- 172.21.15.72:0/4128470256 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdba0107c00 msgr2=0x7fdba019cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:15.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.727+0000 7fdb9effd640 1 --2- 172.21.15.72:0/4128470256 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdba0107c00 0x7fdba019cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.727+0000 7fdb9effd640 1 -- 172.21.15.72:0/4128470256 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdba019d3d0 con 0x7fdba0101b40 2024-03-22T01:37:15.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.728+0000 7fdb9effd640 1 --2- 172.21.15.72:0/4128470256 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdba0101b40 0x7fdba01983e0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fdb90000c00 tx=0x7fdb90004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:15.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.728+0000 7fdba4864640 1 -- 172.21.15.72:0/4128470256 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb90041020 con 0x7fdba0101b40 2024-03-22T01:37:15.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.728+0000 7fdba5866640 1 -- 172.21.15.72:0/4128470256 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdba019d660 con 0x7fdba0101b40 2024-03-22T01:37:15.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.728+0000 7fdba4864640 1 -- 172.21.15.72:0/4128470256 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb9002c9f0 con 0x7fdba0101b40 2024-03-22T01:37:15.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.728+0000 7fdba4864640 1 -- 172.21.15.72:0/4128470256 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb90039040 con 0x7fdba0101b40 2024-03-22T01:37:15.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.728+0000 7fdba5866640 1 -- 172.21.15.72:0/4128470256 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdba01a4f50 con 0x7fdba0101b40 2024-03-22T01:37:15.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.730+0000 7fdba4864640 1 -- 172.21.15.72:0/4128470256 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 24) ==== 138087+0+0 (secure 0 0 0) 0x7fdb90044020 con 0x7fdba0101b40 2024-03-22T01:37:15.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.731+0000 7fdba8449740 1 Processor -- start 2024-03-22T01:37:15.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.731+0000 7fdba8449740 1 -- start start 2024-03-22T01:37:15.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.731+0000 7fdba8449740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55822eac7410 0x55822eac77f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:15.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.731+0000 7fdba8449740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55822eac7d30 0x55822ead03d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:15.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.731+0000 7fdba8449740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55822ead0910 0x55822ead2d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:15.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.731+0000 7fdba8449740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55822ea2b8c0 con 0x55822eac7410 2024-03-22T01:37:15.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.731+0000 7fdba8449740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55822e9d3c20 con 0x55822ead0910 2024-03-22T01:37:15.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.731+0000 7fdba8449740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55822ea27870 con 0x55822eac7d30 2024-03-22T01:37:15.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.731+0000 7fdb9f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55822eac7410 0x55822eac77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:15.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.731+0000 7fdb9f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55822eac7410 0x55822eac77f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59560/0 (socket says 172.21.15.72:59560) 2024-03-22T01:37:15.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.731+0000 7fdb9f7fe640 1 -- 172.21.15.72:0/3273196903 learned_addr learned my addr 172.21.15.72:0/3273196903 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:37:15.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.731+0000 7fdb97fff640 1 --2- 172.21.15.72:0/3273196903 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55822ead0910 0x55822ead2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:15.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.731+0000 7fdba4864640 1 --2- 172.21.15.72:0/4128470256 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fdb8c0a31b0 0x7fdb8c0a5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:15.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.731+0000 7fdb9effd640 1 --2- 172.21.15.72:0/3273196903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55822eac7d30 0x55822ead03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:15.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.731+0000 7fdb9f7fe640 1 --2- 172.21.15.72:0/4128470256 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fdb8c0a31b0 0x7fdb8c0a5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:15.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.731+0000 7fdb9f7fe640 1 -- 172.21.15.72:0/3273196903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55822eac7d30 msgr2=0x55822ead03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:15.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.731+0000 7fdb9f7fe640 1 --2- 172.21.15.72:0/3273196903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55822eac7d30 0x55822ead03d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.731+0000 7fdb9f7fe640 1 -- 172.21.15.72:0/3273196903 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55822ead0910 msgr2=0x55822ead2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:15.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.731+0000 7fdb9f7fe640 1 --2- 172.21.15.72:0/3273196903 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55822ead0910 0x55822ead2d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.731+0000 7fdb9f7fe640 1 -- 172.21.15.72:0/3273196903 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55822e9f8150 con 0x55822eac7410 2024-03-22T01:37:15.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.731+0000 7fdba4864640 1 -- 172.21.15.72:0/4128470256 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fdb900f15f0 con 0x7fdba0101b40 2024-03-22T01:37:15.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.731+0000 7fdb97fff640 1 --2- 172.21.15.72:0/3273196903 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55822ead0910 0x55822ead2d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:37:15.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.732+0000 7fdb9f7fe640 1 --2- 172.21.15.72:0/3273196903 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55822eac7410 0x55822eac77f0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fdb980139c0 tx=0x7fdb98013e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f52a7a5f9b0226b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:15.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.732+0000 7fdb95ffb640 1 -- 172.21.15.72:0/3273196903 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb98021030 con 0x55822eac7410 2024-03-22T01:37:15.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.732+0000 7fdb95ffb640 1 -- 172.21.15.72:0/3273196903 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb98011c80 con 0x55822eac7410 2024-03-22T01:37:15.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.732+0000 7fdb95ffb640 1 -- 172.21.15.72:0/3273196903 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb98012450 con 0x55822eac7410 2024-03-22T01:37:15.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.732+0000 7fdba8449740 1 -- 172.21.15.72:0/3273196903 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55822eac7410 msgr2=0x55822eac77f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:15.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.732+0000 7fdb9f7fe640 1 --2- 172.21.15.72:0/4128470256 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fdb8c0a31b0 0x7fdb8c0a5670 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fdb98021000 tx=0x7fdb98020000 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:15.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.732+0000 7fdba8449740 1 --2- 172.21.15.72:0/3273196903 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55822eac7410 0x55822eac77f0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fdb980139c0 tx=0x7fdb98013e80 comp rx=0 tx=0).stop 2024-03-22T01:37:15.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.732+0000 7fdba8449740 1 -- 172.21.15.72:0/3273196903 shutdown_connections 2024-03-22T01:37:15.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.732+0000 7fdba8449740 1 --2- 172.21.15.72:0/3273196903 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55822ead0910 0x55822ead2d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.733+0000 7fdba8449740 1 --2- 172.21.15.72:0/3273196903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55822eac7d30 0x55822ead03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.733+0000 7fdba8449740 1 --2- 172.21.15.72:0/3273196903 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55822eac7410 0x55822eac77f0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.733+0000 7fdba8449740 1 -- 172.21.15.72:0/3273196903 >> 172.21.15.72:0/3273196903 conn(0x55822eac1b40 msgr2=0x55822eac1f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:15.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.733+0000 7fdba8449740 1 -- 172.21.15.72:0/3273196903 shutdown_connections 2024-03-22T01:37:15.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.733+0000 7fdba8449740 1 -- 172.21.15.72:0/3273196903 wait complete. 2024-03-22T01:37:15.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.733+0000 7fdba8449740 1 -- start start 2024-03-22T01:37:15.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.733+0000 7fdba8449740 1 Processor -- start 2024-03-22T01:37:15.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.733+0000 7fdba8449740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55822eac7410 0x55822eacd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:15.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.733+0000 7fdba8449740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55822eac7d30 0x55822eacde70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:15.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.733+0000 7fdba8449740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55822ead0910 0x55822eace3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:15.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.733+0000 7fdba8449740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55822ead3700 con 0x55822eac7d30 2024-03-22T01:37:15.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.733+0000 7fdba8449740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55822e9e7130 con 0x55822ead0910 2024-03-22T01:37:15.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.733+0000 7fdba8449740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55822ead3880 con 0x55822eac7410 2024-03-22T01:37:15.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.733+0000 7fdba8449740 10 client.0 _ll_register_callbacks cb 0x55822e9e6b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:37:15.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.733+0000 7fdb97fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55822eac7d30 0x55822eacde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:15.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.733+0000 7fdb97fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55822eac7d30 0x55822eacde70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59564/0 (socket says 172.21.15.72:59564) 2024-03-22T01:37:15.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.733+0000 7fdb97fff640 1 -- 172.21.15.72:0/1646620157 learned_addr learned my addr 172.21.15.72:0/1646620157 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:37:15.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.733+0000 7fdb9f7fe640 1 --2- 172.21.15.72:0/1646620157 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55822ead0910 0x55822eace3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:15.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.734+0000 7fdb9effd640 1 --2- 172.21.15.72:0/1646620157 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55822eac7410 0x55822eacd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:15.744 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.734+0000 7fdb97fff640 1 -- 172.21.15.72:0/1646620157 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55822eac7410 msgr2=0x55822eacd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:15.744 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.734+0000 7fdb97fff640 1 --2- 172.21.15.72:0/1646620157 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55822eac7410 0x55822eacd930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.744 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.734+0000 7fdb97fff640 1 -- 172.21.15.72:0/1646620157 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55822ead0910 msgr2=0x55822eace3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:15.744 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.734+0000 7fdb97fff640 1 --2- 172.21.15.72:0/1646620157 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55822ead0910 0x55822eace3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.744 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.734+0000 7fdb97fff640 1 -- 172.21.15.72:0/1646620157 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55822e8097d0 con 0x55822eac7d30 2024-03-22T01:37:15.744 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.734+0000 7fdb9effd640 1 --2- 172.21.15.72:0/1646620157 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55822eac7410 0x55822eacd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T01:37:15.744 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.734+0000 7fdb97fff640 1 --2- 172.21.15.72:0/1646620157 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55822eac7d30 0x55822eacde70 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fdb88009bc0 tx=0x7fdb880039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=489cb4a3adbd32d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:15.744 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.734+0000 7fdb7effd640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:37:15.744 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.734+0000 7fdb7effd640 1 -- 172.21.15.72:0/1646620157 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb88012860 con 0x55822eac7d30 2024-03-22T01:37:15.744 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.734+0000 7fdb7effd640 1 -- 172.21.15.72:0/1646620157 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fdb88012a00 con 0x55822eac7d30 2024-03-22T01:37:15.744 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.734+0000 7fdba8449740 1 -- 172.21.15.72:0/1646620157 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdb98011c80 con 0x55822eac7d30 2024-03-22T01:37:15.744 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.734+0000 7fdb7effd640 1 -- 172.21.15.72:0/1646620157 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fdb88025050 con 0x55822eac7d30 2024-03-22T01:37:15.744 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.734+0000 7fdba8449740 10 client.15744 fetch_fsmap learned FSMap version 21 2024-03-22T01:37:15.744 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.734+0000 7fdba8449740 1 -- 172.21.15.72:0/1646620157 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55822e73f3f0 con 0x55822eac7d30 2024-03-22T01:37:15.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.734+0000 7fdb7effd640 1 -- 172.21.15.72:0/1646620157 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb98011c80 con 0x55822eac7d30 2024-03-22T01:37:15.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.734+0000 7fdb7effd640 1 -- 172.21.15.72:0/1646620157 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fdb8801a020 con 0x55822eac7d30 2024-03-22T01:37:15.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.735+0000 7fdba8449740 10 client.15744 fetch_fsmap finished waiting for FSMap version 21 2024-03-22T01:37:15.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.735+0000 7fdba8449740 10 client.15744 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T01:37:15.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.735+0000 7fdba8449740 20 client.15744 populate_metadata read hostname 'smithi072' 2024-03-22T01:37:15.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.735+0000 7fdba8449740 1 --2- 172.21.15.72:0/1646620157 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55822eae0c70 0x55822eae3060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:15.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.735+0000 7fdba8449740 4 client.15744 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T01:37:15.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.735+0000 7fdba8449740 1 -- 172.21.15.72:0/1646620157 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55822ea38e00 con 0x55822eae0c70 2024-03-22T01:37:15.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.735+0000 7fdb9effd640 1 --2- 172.21.15.72:0/1646620157 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55822eae0c70 0x55822eae3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:15.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.735+0000 7fdb7effd640 10 client.15744 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:37:15.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.735+0000 7fdb9effd640 1 --2- 172.21.15.72:0/1646620157 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55822eae0c70 0x55822eae3060 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f70763fe3051f298 server_cookie=50aac30bb32c9d2a in_seq=0 out_seq=0 2024-03-22T01:37:15.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.736+0000 7fdb7effd640 1 -- 172.21.15.72:0/1646620157 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55822ea38e00 con 0x55822eae0c70 2024-03-22T01:37:15.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.736+0000 7fdb7effd640 10 client.15744 handle_command_reply: tid=0 2024-03-22T01:37:15.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.736+0000 7fdba8449740 1 client.15744 shutdown 2024-03-22T01:37:15.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.736+0000 7fdba8449740 1 -- 172.21.15.72:0/1646620157 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55822eac7d30 msgr2=0x55822eacde70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:15.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.736+0000 7fdba8449740 1 --2- 172.21.15.72:0/1646620157 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55822eac7d30 0x55822eacde70 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fdb88009bc0 tx=0x7fdb880039c0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.736+0000 7fdba8449740 1 -- 172.21.15.72:0/1646620157 shutdown_connections 2024-03-22T01:37:15.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.736+0000 7fdba8449740 1 --2- 172.21.15.72:0/1646620157 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55822eae0c70 0x55822eae3060 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.736+0000 7fdba8449740 1 --2- 172.21.15.72:0/1646620157 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55822ead0910 0x55822eace3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.736+0000 7fdba8449740 1 --2- 172.21.15.72:0/1646620157 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55822eac7d30 0x55822eacde70 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.736+0000 7fdb7effd640 0 client.15744 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:37:15.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.736+0000 7fdba8449740 1 --2- 172.21.15.72:0/1646620157 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55822eac7410 0x55822eacd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.736+0000 7fdba8449740 1 -- 172.21.15.72:0/1646620157 >> 172.21.15.72:0/1646620157 conn(0x55822eac1b40 msgr2=0x55822eac3890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:15.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.736+0000 7fdba8449740 1 -- 172.21.15.72:0/1646620157 shutdown_connections 2024-03-22T01:37:15.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.736+0000 7fdba8449740 1 -- 172.21.15.72:0/1646620157 wait complete. 2024-03-22T01:37:15.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.736+0000 7fdba8449740 20 client.15744 trim_cache size 0 max 16384 2024-03-22T01:37:15.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.753+0000 7fdba8449740 1 Processor -- start 2024-03-22T01:37:15.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.753+0000 7fdba8449740 1 -- start start 2024-03-22T01:37:15.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.753+0000 7fdba8449740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55822eac7410 0x55822eadc4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:15.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.753+0000 7fdba8449740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55822eac7d30 0x55822eae8970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:15.760 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.754+0000 7fdba8449740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55822ead0910 0x55822ead8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:15.760 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.754+0000 7fdba8449740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fdba01ad090 con 0x55822eac7d30 2024-03-22T01:37:15.760 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.754+0000 7fdba8449740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fdba01af910 con 0x55822ead0910 2024-03-22T01:37:15.760 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.754+0000 7fdba8449740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdba01ad700 con 0x55822eac7410 2024-03-22T01:37:15.760 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.754+0000 7fdb97fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55822eac7d30 0x55822eae8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:15.760 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.754+0000 7fdb97fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55822eac7d30 0x55822eae8970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59580/0 (socket says 172.21.15.72:59580) 2024-03-22T01:37:15.760 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.754+0000 7fdb97fff640 1 -- 172.21.15.72:0/1517772676 learned_addr learned my addr 172.21.15.72:0/1517772676 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:37:15.760 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.754+0000 7fdb9f7fe640 1 --2- 172.21.15.72:0/1517772676 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55822ead0910 0x55822ead8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:15.760 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.754+0000 7fdb97fff640 1 -- 172.21.15.72:0/1517772676 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55822eac7410 msgr2=0x55822eadc4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:15.760 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.754+0000 7fdb97fff640 1 --2- 172.21.15.72:0/1517772676 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55822eac7410 0x55822eadc4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.760 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.754+0000 7fdb97fff640 1 -- 172.21.15.72:0/1517772676 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55822ead0910 msgr2=0x55822ead8ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:15.760 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.754+0000 7fdb97fff640 1 --2- 172.21.15.72:0/1517772676 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55822ead0910 0x55822ead8ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.760 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.754+0000 7fdb97fff640 1 -- 172.21.15.72:0/1517772676 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55822e6ff0f0 con 0x55822eac7d30 2024-03-22T01:37:15.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.754+0000 7fdb97fff640 1 --2- 172.21.15.72:0/1517772676 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55822eac7d30 0x55822eae8970 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fdb88006010 tx=0x7fdb88003ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52093b426fff8378 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:15.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.754+0000 7fdb957fa640 1 -- 172.21.15.72:0/1517772676 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb88011540 con 0x55822eac7d30 2024-03-22T01:37:15.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.754+0000 7fdb957fa640 1 -- 172.21.15.72:0/1517772676 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb88030070 con 0x55822eac7d30 2024-03-22T01:37:15.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.754+0000 7fdb957fa640 1 -- 172.21.15.72:0/1517772676 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb88004680 con 0x55822eac7d30 2024-03-22T01:37:15.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.755+0000 7fdba8449740 1 -- 172.21.15.72:0/1517772676 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55822eac7d30 msgr2=0x55822eae8970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:15.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.755+0000 7fdba8449740 1 --2- 172.21.15.72:0/1517772676 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55822eac7d30 0x55822eae8970 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fdb88006010 tx=0x7fdb88003ec0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.755+0000 7fdba8449740 1 -- 172.21.15.72:0/1517772676 shutdown_connections 2024-03-22T01:37:15.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.755+0000 7fdba8449740 1 --2- 172.21.15.72:0/1517772676 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55822ead0910 0x55822ead8ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.755+0000 7fdba8449740 1 --2- 172.21.15.72:0/1517772676 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55822eac7d30 0x55822eae8970 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.755+0000 7fdba8449740 1 --2- 172.21.15.72:0/1517772676 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55822eac7410 0x55822eadc4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.755+0000 7fdba8449740 1 -- 172.21.15.72:0/1517772676 >> 172.21.15.72:0/1517772676 conn(0x55822eac1b40 msgr2=0x55822eac0370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:15.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.755+0000 7fdba8449740 1 -- 172.21.15.72:0/1517772676 shutdown_connections 2024-03-22T01:37:15.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.755+0000 7fdba8449740 1 -- 172.21.15.72:0/1517772676 wait complete. 2024-03-22T01:37:15.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.755+0000 7fdba8449740 1 -- start start 2024-03-22T01:37:15.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.755+0000 7fdba8449740 1 Processor -- start 2024-03-22T01:37:15.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.755+0000 7fdba8449740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55822eac7410 0x55822ead8960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:15.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.755+0000 7fdba8449740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55822eac7d30 0x55822ead8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:15.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.755+0000 7fdba8449740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55822ead0910 0x55822ead4db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:15.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.756+0000 7fdba8449740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fdba01addd0 con 0x55822eac7410 2024-03-22T01:37:15.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.756+0000 7fdba8449740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fdba01af240 con 0x55822eac7d30 2024-03-22T01:37:15.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.756+0000 7fdba8449740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdba01ae4a0 con 0x55822ead0910 2024-03-22T01:37:15.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.756+0000 7fdba8449740 10 client.0 _ll_register_callbacks cb 0x7fdb88004560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:37:15.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.756+0000 7fdb9effd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55822eac7410 0x55822ead8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:15.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.756+0000 7fdb9effd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55822eac7410 0x55822ead8960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59596/0 (socket says 172.21.15.72:59596) 2024-03-22T01:37:15.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.756+0000 7fdb9effd640 1 -- 172.21.15.72:0/1895851680 learned_addr learned my addr 172.21.15.72:0/1895851680 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:37:15.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.756+0000 7fdb97fff640 1 --2- 172.21.15.72:0/1895851680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55822eac7d30 0x55822ead8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:15.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.756+0000 7fdb9f7fe640 1 --2- 172.21.15.72:0/1895851680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55822ead0910 0x55822ead4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:15.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.756+0000 7fdb9effd640 1 -- 172.21.15.72:0/1895851680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55822ead0910 msgr2=0x55822ead4db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:15.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.756+0000 7fdb9effd640 1 --2- 172.21.15.72:0/1895851680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55822ead0910 0x55822ead4db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.756+0000 7fdb9effd640 1 -- 172.21.15.72:0/1895851680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55822eac7d30 msgr2=0x55822ead8ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:15.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.756+0000 7fdb9effd640 1 --2- 172.21.15.72:0/1895851680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55822eac7d30 0x55822ead8ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.756+0000 7fdb9effd640 1 -- 172.21.15.72:0/1895851680 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55822ead55e0 con 0x55822eac7410 2024-03-22T01:37:15.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.756+0000 7fdb9f7fe640 1 --2- 172.21.15.72:0/1895851680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55822ead0910 0x55822ead4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T01:37:15.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.756+0000 7fdb97fff640 1 --2- 172.21.15.72:0/1895851680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55822eac7d30 0x55822ead8ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:37:15.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.756+0000 7fdb9effd640 1 --2- 172.21.15.72:0/1895851680 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55822eac7410 0x55822ead8960 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fdb900b9950 tx=0x7fdb9008c090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93205da9174af04f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:15.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.756+0000 7fdb7effd640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:37:15.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.756+0000 7fdb7effd640 1 -- 172.21.15.72:0/1895851680 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb900a6020 con 0x55822eac7410 2024-03-22T01:37:15.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.756+0000 7fdb7effd640 1 -- 172.21.15.72:0/1895851680 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fdb90034640 con 0x55822eac7410 2024-03-22T01:37:15.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.756+0000 7fdba8449740 1 -- 172.21.15.72:0/1895851680 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdb88030070 con 0x55822eac7410 2024-03-22T01:37:15.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.756+0000 7fdb7effd640 1 -- 172.21.15.72:0/1895851680 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fdb90088450 con 0x55822eac7410 2024-03-22T01:37:15.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.756+0000 7fdba8449740 10 client.15750 fetch_fsmap learned FSMap version 21 2024-03-22T01:37:15.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.756+0000 7fdba8449740 1 -- 172.21.15.72:0/1895851680 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55822eb47c20 con 0x55822eac7410 2024-03-22T01:37:15.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.757+0000 7fdb7effd640 1 -- 172.21.15.72:0/1895851680 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb88030070 con 0x55822eac7410 2024-03-22T01:37:15.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.757+0000 7fdb7effd640 1 -- 172.21.15.72:0/1895851680 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fdb9003c030 con 0x55822eac7410 2024-03-22T01:37:15.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.757+0000 7fdba8449740 10 client.15750 fetch_fsmap finished waiting for FSMap version 21 2024-03-22T01:37:15.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.757+0000 7fdba8449740 10 client.15750 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T01:37:15.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.757+0000 7fdba8449740 20 client.15750 populate_metadata read hostname 'smithi072' 2024-03-22T01:37:15.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.757+0000 7fdba8449740 1 --2- 172.21.15.72:0/1895851680 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55822eb482f0 0x55822eb4a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:15.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.757+0000 7fdba8449740 4 client.15750 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T01:37:15.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.757+0000 7fdba8449740 1 -- 172.21.15.72:0/1895851680 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55822ea33150 con 0x55822eb482f0 2024-03-22T01:37:15.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.757+0000 7fdb97fff640 1 --2- 172.21.15.72:0/1895851680 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55822eb482f0 0x55822eb4a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:15.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.757+0000 7fdb7effd640 10 client.15750 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:37:15.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.757+0000 7fdb97fff640 1 --2- 172.21.15.72:0/1895851680 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55822eb482f0 0x55822eb4a6e0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b735b52f71407759 server_cookie=5766bbca55cc4849 in_seq=0 out_seq=0 2024-03-22T01:37:15.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.757+0000 7fdb7effd640 1 -- 172.21.15.72:0/1895851680 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55822ea33150 con 0x55822eb482f0 2024-03-22T01:37:15.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.757+0000 7fdb7effd640 10 client.15750 handle_command_reply: tid=0 2024-03-22T01:37:15.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.757+0000 7fdba8449740 1 client.15750 shutdown 2024-03-22T01:37:15.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.757+0000 7fdba8449740 1 -- 172.21.15.72:0/1895851680 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55822eac7410 msgr2=0x55822ead8960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:15.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.757+0000 7fdba8449740 1 --2- 172.21.15.72:0/1895851680 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55822eac7410 0x55822ead8960 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fdb900b9950 tx=0x7fdb9008c090 comp rx=0 tx=0).stop 2024-03-22T01:37:15.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.757+0000 7fdba8449740 1 -- 172.21.15.72:0/1895851680 shutdown_connections 2024-03-22T01:37:15.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.758+0000 7fdba8449740 1 --2- 172.21.15.72:0/1895851680 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55822eb482f0 0x55822eb4a6e0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.758+0000 7fdba8449740 1 --2- 172.21.15.72:0/1895851680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55822eac7d30 0x55822ead8ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.758+0000 7fdba8449740 1 --2- 172.21.15.72:0/1895851680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55822ead0910 0x55822ead4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.758+0000 7fdba8449740 1 --2- 172.21.15.72:0/1895851680 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55822eac7410 0x55822ead8960 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.758+0000 7fdb7effd640 0 client.15750 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:37:15.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.758+0000 7fdba8449740 1 -- 172.21.15.72:0/1895851680 >> 172.21.15.72:0/1895851680 conn(0x55822eac1b40 msgr2=0x55822eac0c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:15.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.758+0000 7fdba8449740 1 -- 172.21.15.72:0/1895851680 shutdown_connections 2024-03-22T01:37:15.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.758+0000 7fdba8449740 1 -- 172.21.15.72:0/1895851680 wait complete. 2024-03-22T01:37:15.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.758+0000 7fdba8449740 20 client.15750 trim_cache size 0 max 16384 2024-03-22T01:37:15.766 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-22T01:37:15.766 INFO:teuthology.orchestra.run.smithi072.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-22T01:37:15.766 INFO:teuthology.orchestra.run.smithi072.stdout: "scrubs": { 2024-03-22T01:37:15.766 INFO:teuthology.orchestra.run.smithi072.stdout: "3f4326e8-5739-4f91-8e04-25bea8d7f196": { 2024-03-22T01:37:15.766 INFO:teuthology.orchestra.run.smithi072.stdout: "path": "/", 2024-03-22T01:37:15.767 INFO:teuthology.orchestra.run.smithi072.stdout: "tag": "3f4326e8-5739-4f91-8e04-25bea8d7f196", 2024-03-22T01:37:15.767 INFO:teuthology.orchestra.run.smithi072.stdout: "options": "recursive,force" 2024-03-22T01:37:15.767 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:37:15.767 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:37:15.767 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-22T01:37:15.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.758+0000 7fdba5866640 1 -- 172.21.15.72:0/4128470256 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fdb8c0a31b0 msgr2=0x7fdb8c0a5670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:15.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.758+0000 7fdba5866640 1 --2- 172.21.15.72:0/4128470256 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fdb8c0a31b0 0x7fdb8c0a5670 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fdb98021000 tx=0x7fdb98020000 comp rx=0 tx=0).stop 2024-03-22T01:37:15.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.758+0000 7fdba5866640 1 -- 172.21.15.72:0/4128470256 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdba0101b40 msgr2=0x7fdba01983e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:15.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.758+0000 7fdba5866640 1 --2- 172.21.15.72:0/4128470256 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdba0101b40 0x7fdba01983e0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fdb90000c00 tx=0x7fdb90004600 comp rx=0 tx=0).stop 2024-03-22T01:37:15.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.758+0000 7fdba5866640 1 -- 172.21.15.72:0/4128470256 shutdown_connections 2024-03-22T01:37:15.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.758+0000 7fdba5866640 1 --2- 172.21.15.72:0/4128470256 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fdb8c0a31b0 0x7fdb8c0a5670 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.758+0000 7fdba5866640 1 --2- 172.21.15.72:0/4128470256 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdba0107c00 0x7fdba019cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.758+0000 7fdba5866640 1 --2- 172.21.15.72:0/4128470256 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdba0102500 0x7fdba0198920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.758+0000 7fdba5866640 1 --2- 172.21.15.72:0/4128470256 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdba0101b40 0x7fdba01983e0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:15.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.758+0000 7fdba5866640 1 -- 172.21.15.72:0/4128470256 >> 172.21.15.72:0/4128470256 conn(0x7fdba00fd630 msgr2=0x7fdba00feda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:15.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.758+0000 7fdba5866640 1 -- 172.21.15.72:0/4128470256 shutdown_connections 2024-03-22T01:37:15.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:15.758+0000 7fdba5866640 1 -- 172.21.15.72:0/4128470256 wait complete. 2024-03-22T01:37:15.770 INFO:tasks.cephfs.filesystem:scrub status for tag:3f4326e8-5739-4f91-8e04-25bea8d7f196 - {'path': '/', 'tag': '3f4326e8-5739-4f91-8e04-25bea8d7f196', 'options': 'recursive,force'} 2024-03-22T01:37:16.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:37:15 smithi098 ceph-mon[36468]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 24 KiB/s wr, 19 op/s 2024-03-22T01:37:16.357 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:37:15 smithi089 ceph-mon[35770]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 24 KiB/s wr, 19 op/s 2024-03-22T01:37:16.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:37:15 smithi072 ceph-mon[32414]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 24 KiB/s wr, 19 op/s 2024-03-22T01:37:17.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:37:16 smithi098 ceph-mon[36468]: scrub queued for path: / 2024-03-22T01:37:17.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:37:16 smithi098 ceph-mon[36468]: scrub summary: idle+waiting paths [/] 2024-03-22T01:37:17.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:37:16 smithi098 ceph-mon[36468]: scrub summary: active paths [/] 2024-03-22T01:37:17.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:37:16 smithi098 ceph-mon[36468]: scrub summary: idle+waiting paths [/] 2024-03-22T01:37:17.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:37:16 smithi089 ceph-mon[35770]: scrub queued for path: / 2024-03-22T01:37:17.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:37:16 smithi089 ceph-mon[35770]: scrub summary: idle+waiting paths [/] 2024-03-22T01:37:17.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:37:16 smithi089 ceph-mon[35770]: scrub summary: active paths [/] 2024-03-22T01:37:17.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:37:16 smithi089 ceph-mon[35770]: scrub summary: idle+waiting paths [/] 2024-03-22T01:37:17.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:37:16 smithi072 ceph-mon[32414]: scrub queued for path: / 2024-03-22T01:37:17.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:37:16 smithi072 ceph-mon[32414]: scrub summary: idle+waiting paths [/] 2024-03-22T01:37:17.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:37:16 smithi072 ceph-mon[32414]: scrub summary: active paths [/] 2024-03-22T01:37:17.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:37:16 smithi072 ceph-mon[32414]: scrub summary: idle+waiting paths [/] 2024-03-22T01:37:18.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:37:17 smithi098 ceph-mon[36468]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 13 KiB/s wr, 6 op/s 2024-03-22T01:37:18.357 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:37:17 smithi089 ceph-mon[35770]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 13 KiB/s wr, 6 op/s 2024-03-22T01:37:18.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:37:17 smithi072 ceph-mon[32414]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 13 KiB/s wr, 6 op/s 2024-03-22T01:37:20.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:37:19 smithi098 ceph-mon[36468]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-22T01:37:20.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:37:19 smithi098 ceph-mon[36468]: scrub summary: idle 2024-03-22T01:37:20.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:37:19 smithi089 ceph-mon[35770]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-22T01:37:20.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:37:19 smithi089 ceph-mon[35770]: scrub summary: idle 2024-03-22T01:37:20.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:37:19 smithi072 ceph-mon[32414]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-22T01:37:20.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:37:19 smithi072 ceph-mon[32414]: scrub summary: idle 2024-03-22T01:37:22.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:37:21 smithi098 ceph-mon[36468]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-03-22T01:37:22.357 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:37:21 smithi089 ceph-mon[35770]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-03-22T01:37:22.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:37:21 smithi072 ceph-mon[32414]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-03-22T01:37:24.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:37:23 smithi098 ceph-mon[36468]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-22T01:37:24.357 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:37:23 smithi089 ceph-mon[35770]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-22T01:37:24.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:37:23 smithi072 ceph-mon[32414]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-22T01:37:24.433 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.432+0000 7fdd9f381640 1 -- 172.21.15.72:0/2461387459 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd98102770 msgr2=0x7fdd981067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:24.433 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.432+0000 7fdd9f381640 1 --2- 172.21.15.72:0/2461387459 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd98102770 0x7fdd981067b0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fdd88009900 tx=0x7fdd8802bc30 comp rx=0 tx=0).stop 2024-03-22T01:37:24.433 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.432+0000 7fdd9f381640 1 -- 172.21.15.72:0/2461387459 shutdown_connections 2024-03-22T01:37:24.433 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.432+0000 7fdd9f381640 1 --2- 172.21.15.72:0/2461387459 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd98106e90 0x7fdd9810ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:24.433 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.432+0000 7fdd9f381640 1 --2- 172.21.15.72:0/2461387459 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd98102770 0x7fdd981067b0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:24.433 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.432+0000 7fdd9f381640 1 --2- 172.21.15.72:0/2461387459 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd98101dc0 0x7fdd981021a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:24.433 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.432+0000 7fdd9f381640 1 -- 172.21.15.72:0/2461387459 >> 172.21.15.72:0/2461387459 conn(0x7fdd980fd620 msgr2=0x7fdd980ffa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:24.433 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.432+0000 7fdd9f381640 1 -- 172.21.15.72:0/2461387459 shutdown_connections 2024-03-22T01:37:24.433 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.432+0000 7fdd9f381640 1 -- 172.21.15.72:0/2461387459 wait complete. 2024-03-22T01:37:24.433 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.432+0000 7fdd9f381640 1 Processor -- start 2024-03-22T01:37:24.433 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.432+0000 7fdd9f381640 1 -- start start 2024-03-22T01:37:24.433 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.433+0000 7fdd9f381640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd98101dc0 0x7fdd981984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:24.434 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.433+0000 7fdd9f381640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd98106e90 0x7fdd98198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:24.434 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.433+0000 7fdd9f381640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd9819cd50 0x7fdd9819d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:24.434 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.433+0000 7fdd9f381640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fdd98111f40 con 0x7fdd9819cd50 2024-03-22T01:37:24.434 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.433+0000 7fdd9f381640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fdd98111dc0 con 0x7fdd98106e90 2024-03-22T01:37:24.434 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.433+0000 7fdd9f381640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdd981120c0 con 0x7fdd98101dc0 2024-03-22T01:37:24.434 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.433+0000 7fdd9d91e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd9819cd50 0x7fdd9819d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:24.434 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.433+0000 7fdd9c91c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd98106e90 0x7fdd98198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:24.434 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.433+0000 7fdd9d91e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd9819cd50 0x7fdd9819d200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:35174/0 (socket says 172.21.15.72:35174) 2024-03-22T01:37:24.434 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.433+0000 7fdd9d91e640 1 -- 172.21.15.72:0/1872496916 learned_addr learned my addr 172.21.15.72:0/1872496916 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:37:24.434 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.433+0000 7fdd9d11d640 1 --2- 172.21.15.72:0/1872496916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd98101dc0 0x7fdd981984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:24.434 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.433+0000 7fdd9c91c640 1 -- 172.21.15.72:0/1872496916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd98101dc0 msgr2=0x7fdd981984e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:24.434 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.433+0000 7fdd9c91c640 1 --2- 172.21.15.72:0/1872496916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd98101dc0 0x7fdd981984e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:24.434 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.433+0000 7fdd9c91c640 1 -- 172.21.15.72:0/1872496916 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd9819cd50 msgr2=0x7fdd9819d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:24.435 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.433+0000 7fdd9c91c640 1 --2- 172.21.15.72:0/1872496916 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd9819cd50 0x7fdd9819d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:24.435 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.433+0000 7fdd9c91c640 1 -- 172.21.15.72:0/1872496916 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd9819d840 con 0x7fdd98106e90 2024-03-22T01:37:24.435 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.433+0000 7fdd9d91e640 1 --2- 172.21.15.72:0/1872496916 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd9819cd50 0x7fdd9819d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:37:24.435 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.433+0000 7fdd9d11d640 1 --2- 172.21.15.72:0/1872496916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd98101dc0 0x7fdd981984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:37:24.435 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.433+0000 7fdd9c91c640 1 --2- 172.21.15.72:0/1872496916 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd98106e90 0x7fdd98198a20 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fdd88036040 tx=0x7fdd8802cbf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:24.435 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.434+0000 7fdd8e7fc640 1 -- 172.21.15.72:0/1872496916 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd88041070 con 0x7fdd98106e90 2024-03-22T01:37:24.435 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.434+0000 7fdd9f381640 1 -- 172.21.15.72:0/1872496916 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd9819dad0 con 0x7fdd98106e90 2024-03-22T01:37:24.435 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.434+0000 7fdd8e7fc640 1 -- 172.21.15.72:0/1872496916 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd88004450 con 0x7fdd98106e90 2024-03-22T01:37:24.435 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.434+0000 7fdd8e7fc640 1 -- 172.21.15.72:0/1872496916 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd88003e30 con 0x7fdd98106e90 2024-03-22T01:37:24.435 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.434+0000 7fdd9f381640 1 -- 172.21.15.72:0/1872496916 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd981a52f0 con 0x7fdd98106e90 2024-03-22T01:37:24.441 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.436+0000 7fdd8e7fc640 1 -- 172.21.15.72:0/1872496916 <== mon.2 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 24) ==== 138087+0+0 (secure 0 0 0) 0x7fdd88044070 con 0x7fdd98106e90 2024-03-22T01:37:24.441 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.436+0000 7fdda1f69740 1 Processor -- start 2024-03-22T01:37:24.442 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.436+0000 7fdda1f69740 1 -- start start 2024-03-22T01:37:24.442 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.436+0000 7fdda1f69740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e2bfed2d0 0x563e2bfed6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:24.442 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.436+0000 7fdda1f69740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563e2bfedbf0 0x563e2bff6480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:24.442 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.436+0000 7fdda1f69740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e2bff69c0 0x563e2bff8db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:24.442 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.436+0000 7fdda1f69740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x563e2bf51040 con 0x563e2bfedbf0 2024-03-22T01:37:24.442 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.436+0000 7fdda1f69740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x563e2bef99e0 con 0x563e2bff69c0 2024-03-22T01:37:24.442 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.436+0000 7fdda1f69740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563e2bf4cff0 con 0x563e2bfed2d0 2024-03-22T01:37:24.442 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.437+0000 7fdd9d11d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563e2bfedbf0 0x563e2bff6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:24.442 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.437+0000 7fdd9d11d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563e2bfedbf0 0x563e2bff6480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:35190/0 (socket says 172.21.15.72:35190) 2024-03-22T01:37:24.442 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.437+0000 7fdd9d11d640 1 -- 172.21.15.72:0/1094140667 learned_addr learned my addr 172.21.15.72:0/1094140667 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:37:24.442 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.437+0000 7fdd9c91c640 1 --2- 172.21.15.72:0/1094140667 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e2bff69c0 0x563e2bff8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:24.442 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.437+0000 7fdd8e7fc640 1 --2- 172.21.15.72:0/1872496916 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fdd840a31a0 0x7fdd840a5660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:24.442 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.437+0000 7fdd9d91e640 1 --2- 172.21.15.72:0/1094140667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e2bfed2d0 0x563e2bfed6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:24.443 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.437+0000 7fdd9d11d640 1 -- 172.21.15.72:0/1094140667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e2bfed2d0 msgr2=0x563e2bfed6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:24.443 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.437+0000 7fdd9d11d640 1 --2- 172.21.15.72:0/1094140667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e2bfed2d0 0x563e2bfed6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:24.443 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.437+0000 7fdd9d91e640 1 --2- 172.21.15.72:0/1872496916 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fdd840a31a0 0x7fdd840a5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:24.443 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.437+0000 7fdd9d11d640 1 -- 172.21.15.72:0/1094140667 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e2bff69c0 msgr2=0x563e2bff8db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:24.443 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.437+0000 7fdd9d11d640 1 --2- 172.21.15.72:0/1094140667 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e2bff69c0 0x563e2bff8db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:24.443 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.437+0000 7fdd9d11d640 1 -- 172.21.15.72:0/1094140667 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e2bc65210 con 0x563e2bfedbf0 2024-03-22T01:37:24.443 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.437+0000 7fdd8e7fc640 1 -- 172.21.15.72:0/1872496916 <== mon.2 v2:172.21.15.89:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fdd880f1c00 con 0x7fdd98106e90 2024-03-22T01:37:24.443 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.437+0000 7fdd9d11d640 1 --2- 172.21.15.72:0/1094140667 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563e2bfedbf0 0x563e2bff6480 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fdd9400bd40 tx=0x7fdd9400f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92c96124faee65ce server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:24.443 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.437+0000 7fdd837fe640 1 -- 172.21.15.72:0/1094140667 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd94015070 con 0x563e2bfedbf0 2024-03-22T01:37:24.443 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.437+0000 7fdd9d91e640 1 --2- 172.21.15.72:0/1872496916 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fdd840a31a0 0x7fdd840a5660 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fdd90005c40 tx=0x7fdd90004ab0 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:24.443 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.438+0000 7fdd837fe640 1 -- 172.21.15.72:0/1094140667 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd9400be30 con 0x563e2bfedbf0 2024-03-22T01:37:24.443 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.438+0000 7fdd837fe640 1 -- 172.21.15.72:0/1094140667 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd940107f0 con 0x563e2bfedbf0 2024-03-22T01:37:24.443 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.438+0000 7fdda1f69740 1 -- 172.21.15.72:0/1094140667 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563e2bfedbf0 msgr2=0x563e2bff6480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:24.443 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.438+0000 7fdda1f69740 1 --2- 172.21.15.72:0/1094140667 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563e2bfedbf0 0x563e2bff6480 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fdd9400bd40 tx=0x7fdd9400f730 comp rx=0 tx=0).stop 2024-03-22T01:37:24.443 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.438+0000 7fdda1f69740 1 -- 172.21.15.72:0/1094140667 shutdown_connections 2024-03-22T01:37:24.444 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.438+0000 7fdda1f69740 1 --2- 172.21.15.72:0/1094140667 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e2bff69c0 0x563e2bff8db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:24.444 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.438+0000 7fdda1f69740 1 --2- 172.21.15.72:0/1094140667 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563e2bfedbf0 0x563e2bff6480 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:24.444 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.438+0000 7fdda1f69740 1 --2- 172.21.15.72:0/1094140667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e2bfed2d0 0x563e2bfed6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:24.444 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.438+0000 7fdda1f69740 1 -- 172.21.15.72:0/1094140667 >> 172.21.15.72:0/1094140667 conn(0x563e2bfe7bf0 msgr2=0x563e2bfe7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:24.444 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.438+0000 7fdda1f69740 1 -- 172.21.15.72:0/1094140667 shutdown_connections 2024-03-22T01:37:24.444 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.438+0000 7fdda1f69740 1 -- 172.21.15.72:0/1094140667 wait complete. 2024-03-22T01:37:24.444 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.439+0000 7fdda1f69740 1 -- start start 2024-03-22T01:37:24.444 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.439+0000 7fdda1f69740 1 Processor -- start 2024-03-22T01:37:24.444 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.439+0000 7fdda1f69740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e2bfed2d0 0x563e2bff6360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:24.444 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.439+0000 7fdda1f69740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e2bfedbf0 0x563e2bff36f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:24.444 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.439+0000 7fdda1f69740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563e2bff69c0 0x563e2bff3c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:24.444 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.439+0000 7fdda1f69740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x563e2bff95f0 con 0x563e2bff69c0 2024-03-22T01:37:24.444 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.439+0000 7fdda1f69740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x563e2bf0cdb0 con 0x563e2bfedbf0 2024-03-22T01:37:24.444 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.439+0000 7fdda1f69740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563e2bff9770 con 0x563e2bfed2d0 2024-03-22T01:37:24.445 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.439+0000 7fdda1f69740 10 client.0 _ll_register_callbacks cb 0x563e2bc64a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:37:24.445 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.439+0000 7fdd9d91e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563e2bff69c0 0x563e2bff3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:24.445 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.439+0000 7fdd9c91c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e2bfedbf0 0x563e2bff36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:24.445 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.439+0000 7fdd9d91e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563e2bff69c0 0x563e2bff3c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:35198/0 (socket says 172.21.15.72:35198) 2024-03-22T01:37:24.445 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.439+0000 7fdd9d91e640 1 -- 172.21.15.72:0/2794509229 learned_addr learned my addr 172.21.15.72:0/2794509229 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:37:24.445 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.439+0000 7fdd9d91e640 1 -- 172.21.15.72:0/2794509229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e2bfed2d0 msgr2=0x563e2bff6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:24.445 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.439+0000 7fdd9d91e640 1 --2- 172.21.15.72:0/2794509229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e2bfed2d0 0x563e2bff6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:24.445 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.439+0000 7fdd9d91e640 1 -- 172.21.15.72:0/2794509229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e2bfedbf0 msgr2=0x563e2bff36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:24.445 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.439+0000 7fdd9d91e640 1 --2- 172.21.15.72:0/2794509229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e2bfedbf0 0x563e2bff36f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:24.445 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.439+0000 7fdd9d91e640 1 -- 172.21.15.72:0/2794509229 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e2bd31620 con 0x563e2bff69c0 2024-03-22T01:37:24.445 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.440+0000 7fdd9d91e640 1 --2- 172.21.15.72:0/2794509229 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563e2bff69c0 0x563e2bff3c90 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fdd900106b0 tx=0x7fdd90004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76459139496e28af server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:24.445 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.440+0000 7fdd80ff9640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:37:24.445 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.440+0000 7fdd80ff9640 1 -- 172.21.15.72:0/2794509229 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd9002d060 con 0x563e2bff69c0 2024-03-22T01:37:24.445 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.440+0000 7fdd80ff9640 1 -- 172.21.15.72:0/2794509229 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fdd9001e580 con 0x563e2bff69c0 2024-03-22T01:37:24.445 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.440+0000 7fdda1f69740 1 -- 172.21.15.72:0/2794509229 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd9400be30 con 0x563e2bff69c0 2024-03-22T01:37:24.446 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.440+0000 7fdd80ff9640 1 -- 172.21.15.72:0/2794509229 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fdd90019030 con 0x563e2bff69c0 2024-03-22T01:37:24.446 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.440+0000 7fdda1f69740 10 client.15762 fetch_fsmap learned FSMap version 21 2024-03-22T01:37:24.446 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.440+0000 7fdda1f69740 1 -- 172.21.15.72:0/2794509229 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x563e2c0055c0 con 0x563e2bff69c0 2024-03-22T01:37:24.446 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.440+0000 7fdd80ff9640 1 -- 172.21.15.72:0/2794509229 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd9400be30 con 0x563e2bff69c0 2024-03-22T01:37:24.446 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.440+0000 7fdd80ff9640 1 -- 172.21.15.72:0/2794509229 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fdd90028960 con 0x563e2bff69c0 2024-03-22T01:37:24.446 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.440+0000 7fdda1f69740 10 client.15762 fetch_fsmap finished waiting for FSMap version 21 2024-03-22T01:37:24.446 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.440+0000 7fdda1f69740 10 client.15762 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T01:37:24.446 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.440+0000 7fdda1f69740 20 client.15762 populate_metadata read hostname 'smithi072' 2024-03-22T01:37:24.446 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.440+0000 7fdda1f69740 1 --2- 172.21.15.72:0/2794509229 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x563e2c006a80 0x563e2c008f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:24.446 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.440+0000 7fdda1f69740 4 client.15762 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T01:37:24.446 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.440+0000 7fdda1f69740 1 -- 172.21.15.72:0/2794509229 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563e2bf5e580 con 0x563e2c006a80 2024-03-22T01:37:24.446 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.441+0000 7fdd9d11d640 1 --2- 172.21.15.72:0/2794509229 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x563e2c006a80 0x563e2c008f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:24.446 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.441+0000 7fdd80ff9640 10 client.15762 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:37:24.446 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.441+0000 7fdd9d11d640 1 --2- 172.21.15.72:0/2794509229 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x563e2c006a80 0x563e2c008f10 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8cd7bc4f9bbb797e server_cookie=8d7c8e14a2b7ec19 in_seq=0 out_seq=0 2024-03-22T01:37:24.447 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.441+0000 7fdd80ff9640 1 -- 172.21.15.72:0/2794509229 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563e2bf5e580 con 0x563e2c006a80 2024-03-22T01:37:24.447 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.441+0000 7fdd80ff9640 10 client.15762 handle_command_reply: tid=0 2024-03-22T01:37:24.447 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.442+0000 7fdda1f69740 1 client.15762 shutdown 2024-03-22T01:37:24.447 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.442+0000 7fdda1f69740 1 -- 172.21.15.72:0/2794509229 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563e2bff69c0 msgr2=0x563e2bff3c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:24.447 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.442+0000 7fdda1f69740 1 --2- 172.21.15.72:0/2794509229 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563e2bff69c0 0x563e2bff3c90 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fdd900106b0 tx=0x7fdd90004a40 comp rx=0 tx=0).stop 2024-03-22T01:37:24.447 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.442+0000 7fdda1f69740 1 -- 172.21.15.72:0/2794509229 shutdown_connections 2024-03-22T01:37:24.447 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.442+0000 7fdda1f69740 1 --2- 172.21.15.72:0/2794509229 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x563e2c006a80 0x563e2c008f10 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:24.447 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.442+0000 7fdda1f69740 1 --2- 172.21.15.72:0/2794509229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e2bfedbf0 0x563e2bff36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:24.447 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.442+0000 7fdda1f69740 1 --2- 172.21.15.72:0/2794509229 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563e2bff69c0 0x563e2bff3c90 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:24.447 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.442+0000 7fdda1f69740 1 --2- 172.21.15.72:0/2794509229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e2bfed2d0 0x563e2bff6360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:24.447 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.442+0000 7fdda1f69740 1 -- 172.21.15.72:0/2794509229 >> 172.21.15.72:0/2794509229 conn(0x563e2bfe7bf0 msgr2=0x563e2bfe9940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:24.447 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.442+0000 7fdd80ff9640 0 client.15762 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:37:24.447 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.442+0000 7fdda1f69740 1 -- 172.21.15.72:0/2794509229 shutdown_connections 2024-03-22T01:37:24.448 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.442+0000 7fdda1f69740 1 -- 172.21.15.72:0/2794509229 wait complete. 2024-03-22T01:37:24.448 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.442+0000 7fdda1f69740 20 client.15762 trim_cache size 0 max 16384 2024-03-22T01:37:24.459 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.458+0000 7fdda1f69740 1 Processor -- start 2024-03-22T01:37:24.459 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.458+0000 7fdda1f69740 1 -- start start 2024-03-22T01:37:24.459 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.458+0000 7fdda1f69740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e2bfed2d0 0x563e2c002390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:24.459 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.458+0000 7fdda1f69740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563e2bfedbf0 0x563e2c00e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:24.459 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.458+0000 7fdda1f69740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e2bff69c0 0x563e2bff5e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:24.459 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.458+0000 7fdda1f69740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fdd981ad0a0 con 0x563e2bfedbf0 2024-03-22T01:37:24.459 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.458+0000 7fdda1f69740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fdd981af920 con 0x563e2bfed2d0 2024-03-22T01:37:24.460 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.458+0000 7fdda1f69740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdd981ad710 con 0x563e2bff69c0 2024-03-22T01:37:24.460 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.458+0000 7fdd9c91c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563e2bfedbf0 0x563e2c00e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:24.460 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.458+0000 7fdd9c91c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563e2bfedbf0 0x563e2c00e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:35212/0 (socket says 172.21.15.72:35212) 2024-03-22T01:37:24.460 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.458+0000 7fdd9c91c640 1 -- 172.21.15.72:0/2241715978 learned_addr learned my addr 172.21.15.72:0/2241715978 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:37:24.460 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.458+0000 7fdd9d11d640 1 --2- 172.21.15.72:0/2241715978 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e2bfed2d0 0x563e2c002390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:24.460 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.458+0000 7fdd9d91e640 1 --2- 172.21.15.72:0/2241715978 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e2bff69c0 0x563e2bff5e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:24.460 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.458+0000 7fdd9c91c640 1 -- 172.21.15.72:0/2241715978 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e2bff69c0 msgr2=0x563e2bff5e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:24.460 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.458+0000 7fdd9c91c640 1 --2- 172.21.15.72:0/2241715978 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e2bff69c0 0x563e2bff5e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:24.460 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.458+0000 7fdd9c91c640 1 -- 172.21.15.72:0/2241715978 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e2bfed2d0 msgr2=0x563e2c002390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:24.460 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.458+0000 7fdd9c91c640 1 --2- 172.21.15.72:0/2241715978 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e2bfed2d0 0x563e2c002390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:24.460 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.458+0000 7fdd9c91c640 1 -- 172.21.15.72:0/2241715978 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e2bfe7720 con 0x563e2bfedbf0 2024-03-22T01:37:24.460 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.458+0000 7fdd9d91e640 1 --2- 172.21.15.72:0/2241715978 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e2bff69c0 0x563e2bff5e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T01:37:24.460 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.459+0000 7fdd9c91c640 1 --2- 172.21.15.72:0/2241715978 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563e2bfedbf0 0x563e2c00e870 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fdd8802c170 tx=0x7fdd880b9ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40894fdd6b9075ad server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:24.460 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.459+0000 7fdd82ffd640 1 -- 172.21.15.72:0/2241715978 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd8804b040 con 0x563e2bfedbf0 2024-03-22T01:37:24.461 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.459+0000 7fdd82ffd640 1 -- 172.21.15.72:0/2241715978 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd880be200 con 0x563e2bfedbf0 2024-03-22T01:37:24.461 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.459+0000 7fdd82ffd640 1 -- 172.21.15.72:0/2241715978 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd880c6510 con 0x563e2bfedbf0 2024-03-22T01:37:24.461 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.460+0000 7fdda1f69740 1 -- 172.21.15.72:0/2241715978 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563e2bfedbf0 msgr2=0x563e2c00e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:24.461 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.460+0000 7fdda1f69740 1 --2- 172.21.15.72:0/2241715978 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563e2bfedbf0 0x563e2c00e870 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fdd8802c170 tx=0x7fdd880b9ed0 comp rx=0 tx=0).stop 2024-03-22T01:37:24.461 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.460+0000 7fdda1f69740 1 -- 172.21.15.72:0/2241715978 shutdown_connections 2024-03-22T01:37:24.461 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.460+0000 7fdda1f69740 1 --2- 172.21.15.72:0/2241715978 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e2bff69c0 0x563e2bff5e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:24.461 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.460+0000 7fdda1f69740 1 --2- 172.21.15.72:0/2241715978 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563e2bfedbf0 0x563e2c00e870 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:24.461 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.460+0000 7fdda1f69740 1 --2- 172.21.15.72:0/2241715978 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e2bfed2d0 0x563e2c002390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:24.461 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.460+0000 7fdda1f69740 1 -- 172.21.15.72:0/2241715978 >> 172.21.15.72:0/2241715978 conn(0x563e2bfe7bf0 msgr2=0x563e2bfe6c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:24.461 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.460+0000 7fdda1f69740 1 -- 172.21.15.72:0/2241715978 shutdown_connections 2024-03-22T01:37:24.461 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.460+0000 7fdda1f69740 1 -- 172.21.15.72:0/2241715978 wait complete. 2024-03-22T01:37:24.462 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.460+0000 7fdda1f69740 1 -- start start 2024-03-22T01:37:24.462 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.460+0000 7fdda1f69740 1 Processor -- start 2024-03-22T01:37:24.465 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.461+0000 7fdda1f69740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e2bfed2d0 0x563e2bffa8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:24.465 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.461+0000 7fdda1f69740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563e2bfedbf0 0x563e2bffae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:24.465 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.461+0000 7fdda1f69740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e2bff69c0 0x563e2c06bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:24.465 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.461+0000 7fdda1f69740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fdd981adde0 con 0x563e2bfedbf0 2024-03-22T01:37:24.466 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.461+0000 7fdda1f69740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fdd981af250 con 0x563e2bfed2d0 2024-03-22T01:37:24.466 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.461+0000 7fdda1f69740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdd981ae4b0 con 0x563e2bff69c0 2024-03-22T01:37:24.466 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.461+0000 7fdda1f69740 10 client.0 _ll_register_callbacks cb 0x7fdd90009d70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:37:24.466 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.461+0000 7fdd9c91c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563e2bfedbf0 0x563e2bffae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:24.466 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.461+0000 7fdd9c91c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563e2bfedbf0 0x563e2bffae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:35214/0 (socket says 172.21.15.72:35214) 2024-03-22T01:37:24.466 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.461+0000 7fdd9d11d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e2bfed2d0 0x563e2bffa8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:24.466 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.461+0000 7fdd9c91c640 1 -- 172.21.15.72:0/3349275750 learned_addr learned my addr 172.21.15.72:0/3349275750 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:37:24.466 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.461+0000 7fdd9d91e640 1 --2- 172.21.15.72:0/3349275750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e2bff69c0 0x563e2c06bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:24.466 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.461+0000 7fdd9c91c640 1 -- 172.21.15.72:0/3349275750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e2bff69c0 msgr2=0x563e2c06bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:24.466 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.461+0000 7fdd9c91c640 1 --2- 172.21.15.72:0/3349275750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e2bff69c0 0x563e2c06bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:24.466 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.461+0000 7fdd9c91c640 1 -- 172.21.15.72:0/3349275750 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e2bfed2d0 msgr2=0x563e2bffa8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:24.466 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.461+0000 7fdd9c91c640 1 --2- 172.21.15.72:0/3349275750 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e2bfed2d0 0x563e2bffa8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:24.466 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.461+0000 7fdd9c91c640 1 -- 172.21.15.72:0/3349275750 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e2c06c3e0 con 0x563e2bfedbf0 2024-03-22T01:37:24.467 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.461+0000 7fdd9d91e640 1 --2- 172.21.15.72:0/3349275750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e2bff69c0 0x563e2c06bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T01:37:24.467 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.461+0000 7fdd9d11d640 1 --2- 172.21.15.72:0/3349275750 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e2bfed2d0 0x563e2bffa8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:37:24.467 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.461+0000 7fdd9c91c640 1 --2- 172.21.15.72:0/3349275750 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563e2bfedbf0 0x563e2bffae30 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fdd8802c820 tx=0x7fdd880eefa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6aa5ca6c2a837cdb server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:24.467 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.461+0000 7fdd80ff9640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:37:24.467 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.462+0000 7fdd80ff9640 1 -- 172.21.15.72:0/3349275750 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd880db060 con 0x563e2bfedbf0 2024-03-22T01:37:24.467 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.462+0000 7fdd80ff9640 1 -- 172.21.15.72:0/3349275750 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fdd880bdf60 con 0x563e2bfedbf0 2024-03-22T01:37:24.467 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.462+0000 7fdda1f69740 1 -- 172.21.15.72:0/3349275750 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd880be200 con 0x563e2bfedbf0 2024-03-22T01:37:24.467 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.462+0000 7fdd80ff9640 1 -- 172.21.15.72:0/3349275750 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fdd880d6070 con 0x563e2bfedbf0 2024-03-22T01:37:24.467 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.462+0000 7fdda1f69740 10 client.15768 fetch_fsmap learned FSMap version 21 2024-03-22T01:37:24.467 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.462+0000 7fdda1f69740 1 -- 172.21.15.72:0/3349275750 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x563e2c06c3e0 con 0x563e2bfedbf0 2024-03-22T01:37:24.467 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.462+0000 7fdd80ff9640 1 -- 172.21.15.72:0/3349275750 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd880be200 con 0x563e2bfedbf0 2024-03-22T01:37:24.467 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.462+0000 7fdd80ff9640 1 -- 172.21.15.72:0/3349275750 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fdd880f9040 con 0x563e2bfedbf0 2024-03-22T01:37:24.467 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.462+0000 7fdda1f69740 10 client.15768 fetch_fsmap finished waiting for FSMap version 21 2024-03-22T01:37:24.467 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.462+0000 7fdda1f69740 10 client.15768 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T01:37:24.467 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.462+0000 7fdda1f69740 20 client.15768 populate_metadata read hostname 'smithi072' 2024-03-22T01:37:24.468 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.462+0000 7fdda1f69740 1 --2- 172.21.15.72:0/3349275750 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x563e2c06d8f0 0x563e2c06fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:24.468 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.462+0000 7fdda1f69740 4 client.15768 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T01:37:24.468 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.462+0000 7fdda1f69740 1 -- 172.21.15.72:0/3349275750 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563e2bf588d0 con 0x563e2c06d8f0 2024-03-22T01:37:24.468 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.462+0000 7fdd9d11d640 1 --2- 172.21.15.72:0/3349275750 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x563e2c06d8f0 0x563e2c06fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:24.468 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.463+0000 7fdd80ff9640 10 client.15768 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:37:24.468 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.463+0000 7fdd9d11d640 1 --2- 172.21.15.72:0/3349275750 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x563e2c06d8f0 0x563e2c06fd10 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c7f0ed0814fbb1f server_cookie=1c1d040c7975ef9a in_seq=0 out_seq=0 2024-03-22T01:37:24.470 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.469+0000 7fdd80ff9640 1 -- 172.21.15.72:0/3349275750 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563e2bf588d0 con 0x563e2c06d8f0 2024-03-22T01:37:24.470 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.469+0000 7fdd80ff9640 10 client.15768 handle_command_reply: tid=0 2024-03-22T01:37:24.470 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.469+0000 7fdda1f69740 1 client.15768 shutdown 2024-03-22T01:37:24.470 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.469+0000 7fdda1f69740 1 -- 172.21.15.72:0/3349275750 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563e2bfedbf0 msgr2=0x563e2bffae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:24.470 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.469+0000 7fdda1f69740 1 --2- 172.21.15.72:0/3349275750 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563e2bfedbf0 0x563e2bffae30 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fdd8802c820 tx=0x7fdd880eefa0 comp rx=0 tx=0).stop 2024-03-22T01:37:24.470 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.469+0000 7fdda1f69740 1 -- 172.21.15.72:0/3349275750 shutdown_connections 2024-03-22T01:37:24.471 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.469+0000 7fdda1f69740 1 --2- 172.21.15.72:0/3349275750 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x563e2c06d8f0 0x563e2c06fd10 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:24.471 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.469+0000 7fdda1f69740 1 --2- 172.21.15.72:0/3349275750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e2bff69c0 0x563e2c06bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:24.471 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.469+0000 7fdd80ff9640 0 client.15768 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:37:24.471 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.469+0000 7fdda1f69740 1 --2- 172.21.15.72:0/3349275750 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563e2bfedbf0 0x563e2bffae30 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:24.471 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.469+0000 7fdda1f69740 1 --2- 172.21.15.72:0/3349275750 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e2bfed2d0 0x563e2bffa8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:24.471 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.469+0000 7fdda1f69740 1 -- 172.21.15.72:0/3349275750 >> 172.21.15.72:0/3349275750 conn(0x563e2bfe7bf0 msgr2=0x563e2bfee280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:24.471 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.470+0000 7fdda1f69740 1 -- 172.21.15.72:0/3349275750 shutdown_connections 2024-03-22T01:37:24.471 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.470+0000 7fdda1f69740 1 -- 172.21.15.72:0/3349275750 wait complete. 2024-03-22T01:37:24.471 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.470+0000 7fdda1f69740 20 client.15768 trim_cache size 0 max 16384 2024-03-22T01:37:24.472 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout:{ 2024-03-22T01:37:24.472 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout: "message": "", 2024-03-22T01:37:24.472 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout: "return_code": 0 2024-03-22T01:37:24.472 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout:} 2024-03-22T01:37:24.473 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.470+0000 7fdd9f381640 1 -- 172.21.15.72:0/1872496916 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fdd840a31a0 msgr2=0x7fdd840a5660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:24.473 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.470+0000 7fdd9f381640 1 --2- 172.21.15.72:0/1872496916 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fdd840a31a0 0x7fdd840a5660 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fdd90005c40 tx=0x7fdd90004ab0 comp rx=0 tx=0).stop 2024-03-22T01:37:24.473 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.470+0000 7fdd9f381640 1 -- 172.21.15.72:0/1872496916 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd98106e90 msgr2=0x7fdd98198a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:24.473 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.470+0000 7fdd9f381640 1 --2- 172.21.15.72:0/1872496916 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd98106e90 0x7fdd98198a20 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fdd88036040 tx=0x7fdd8802cbf0 comp rx=0 tx=0).stop 2024-03-22T01:37:24.473 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.470+0000 7fdd9f381640 1 -- 172.21.15.72:0/1872496916 shutdown_connections 2024-03-22T01:37:24.473 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.470+0000 7fdd9f381640 1 --2- 172.21.15.72:0/1872496916 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fdd840a31a0 0x7fdd840a5660 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:24.473 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.470+0000 7fdd9f381640 1 --2- 172.21.15.72:0/1872496916 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd98106e90 0x7fdd98198a20 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:24.473 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.470+0000 7fdd9f381640 1 --2- 172.21.15.72:0/1872496916 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd9819cd50 0x7fdd9819d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:24.473 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.470+0000 7fdd9f381640 1 --2- 172.21.15.72:0/1872496916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd98101dc0 0x7fdd981984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:24.473 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.470+0000 7fdd9f381640 1 -- 172.21.15.72:0/1872496916 >> 172.21.15.72:0/1872496916 conn(0x7fdd980fd620 msgr2=0x7fdd980ff9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:24.473 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.470+0000 7fdd9f381640 1 -- 172.21.15.72:0/1872496916 shutdown_connections 2024-03-22T01:37:24.473 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:24.470+0000 7fdd9f381640 1 -- 172.21.15.72:0/1872496916 wait complete. 2024-03-22T01:37:26.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:37:25 smithi098 ceph-mon[36468]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 2.5 KiB/s wr, 5 op/s 2024-03-22T01:37:26.174 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:37:25 smithi089 ceph-mon[35770]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 2.5 KiB/s wr, 5 op/s 2024-03-22T01:37:26.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:37:25 smithi072 ceph-mon[32414]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 2.5 KiB/s wr, 5 op/s 2024-03-22T01:37:28.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:37:27 smithi098 ceph-mon[36468]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.4 KiB/s wr, 3 op/s 2024-03-22T01:37:28.357 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:37:27 smithi089 ceph-mon[35770]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.4 KiB/s wr, 3 op/s 2024-03-22T01:37:28.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:37:27 smithi072 ceph-mon[32414]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.4 KiB/s wr, 3 op/s 2024-03-22T01:37:30.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:37:29 smithi089 ceph-mon[35770]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-03-22T01:37:30.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:37:29 smithi072 ceph-mon[32414]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-03-22T01:37:30.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:37:29 smithi098 ceph-mon[36468]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-03-22T01:37:32.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:37:31 smithi089 ceph-mon[35770]: pgmap v29: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.5 KiB/s wr, 5 op/s 2024-03-22T01:37:32.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:37:31 smithi072 ceph-mon[32414]: pgmap v29: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.5 KiB/s wr, 5 op/s 2024-03-22T01:37:32.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:37:31 smithi098 ceph-mon[36468]: pgmap v29: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.5 KiB/s wr, 5 op/s 2024-03-22T01:37:34.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:37:33 smithi089 ceph-mon[35770]: pgmap v30: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 2.5 KiB/s wr, 3 op/s 2024-03-22T01:37:34.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:37:33 smithi072 ceph-mon[32414]: pgmap v30: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 2.5 KiB/s wr, 3 op/s 2024-03-22T01:37:34.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:37:33 smithi098 ceph-mon[36468]: pgmap v30: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 2.5 KiB/s wr, 3 op/s 2024-03-22T01:37:36.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:37:35 smithi089 ceph-mon[35770]: pgmap v31: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 2.5 KiB/s wr, 4 op/s 2024-03-22T01:37:36.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:37:35 smithi072 ceph-mon[32414]: pgmap v31: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 2.5 KiB/s wr, 4 op/s 2024-03-22T01:37:36.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:37:35 smithi098 ceph-mon[36468]: pgmap v31: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 2.5 KiB/s wr, 4 op/s 2024-03-22T01:37:37.575 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.574+0000 7f56bb7c7640 1 -- 172.21.15.72:0/3916736793 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f56b4106a40 msgr2=0x7f56b4106e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:37.575 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.574+0000 7f56bb7c7640 1 --2- 172.21.15.72:0/3916736793 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f56b4106a40 0x7f56b4106e20 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f56b0009930 tx=0x7f56b002bc30 comp rx=0 tx=0).stop 2024-03-22T01:37:37.576 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.574+0000 7f56bb7c7640 1 -- 172.21.15.72:0/3916736793 shutdown_connections 2024-03-22T01:37:37.576 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.574+0000 7f56bb7c7640 1 --2- 172.21.15.72:0/3916736793 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56b41013f0 0x7f56b410d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:37.576 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.574+0000 7f56bb7c7640 1 --2- 172.21.15.72:0/3916736793 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56b4100a30 0x7f56b4100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:37.576 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.574+0000 7f56bb7c7640 1 --2- 172.21.15.72:0/3916736793 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f56b4106a40 0x7f56b4106e20 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:37.576 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.574+0000 7f56bb7c7640 1 -- 172.21.15.72:0/3916736793 >> 172.21.15.72:0/3916736793 conn(0x7f56b4077b80 msgr2=0x7f56b4077f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:37.576 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.574+0000 7f56bb7c7640 1 -- 172.21.15.72:0/3916736793 shutdown_connections 2024-03-22T01:37:37.576 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.574+0000 7f56bb7c7640 1 -- 172.21.15.72:0/3916736793 wait complete. 2024-03-22T01:37:37.576 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.575+0000 7f56bb7c7640 1 Processor -- start 2024-03-22T01:37:37.576 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.575+0000 7f56bb7c7640 1 -- start start 2024-03-22T01:37:37.576 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.575+0000 7f56bb7c7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56b4100a30 0x7f56b41986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:37.576 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.575+0000 7f56bb7c7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f56b41013f0 0x7f56b4198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:37.576 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.575+0000 7f56bb7c7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56b419cf10 0x7f56b419d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:37.576 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.575+0000 7f56bb7c7640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f56b4110e50 con 0x7f56b41013f0 2024-03-22T01:37:37.577 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.575+0000 7f56bb7c7640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f56b4110cd0 con 0x7f56b4100a30 2024-03-22T01:37:37.577 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.575+0000 7f56bb7c7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f56b4110fd0 con 0x7f56b419cf10 2024-03-22T01:37:37.577 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.575+0000 7f56b8d62640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f56b41013f0 0x7f56b4198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:37.577 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.575+0000 7f56b8d62640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f56b41013f0 0x7f56b4198be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42456/0 (socket says 172.21.15.72:42456) 2024-03-22T01:37:37.577 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.575+0000 7f56b8d62640 1 -- 172.21.15.72:0/2397123517 learned_addr learned my addr 172.21.15.72:0/2397123517 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:37:37.577 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.575+0000 7f56b9563640 1 --2- 172.21.15.72:0/2397123517 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56b4100a30 0x7f56b41986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:37.577 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.576+0000 7f56b9d64640 1 --2- 172.21.15.72:0/2397123517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56b419cf10 0x7f56b419d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:37.577 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.576+0000 7f56b8d62640 1 -- 172.21.15.72:0/2397123517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56b419cf10 msgr2=0x7f56b419d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:37.577 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.576+0000 7f56b8d62640 1 --2- 172.21.15.72:0/2397123517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56b419cf10 0x7f56b419d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:37.577 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.576+0000 7f56b8d62640 1 -- 172.21.15.72:0/2397123517 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56b4100a30 msgr2=0x7f56b41986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:37.577 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.576+0000 7f56b8d62640 1 --2- 172.21.15.72:0/2397123517 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56b4100a30 0x7f56b41986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:37.577 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.576+0000 7f56b8d62640 1 -- 172.21.15.72:0/2397123517 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56b419da00 con 0x7f56b41013f0 2024-03-22T01:37:37.577 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.576+0000 7f56b9563640 1 --2- 172.21.15.72:0/2397123517 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56b4100a30 0x7f56b41986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:37:37.577 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.576+0000 7f56b9d64640 1 --2- 172.21.15.72:0/2397123517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56b419cf10 0x7f56b419d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:37:37.578 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.576+0000 7f56b8d62640 1 --2- 172.21.15.72:0/2397123517 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f56b41013f0 0x7f56b4198be0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f56a400e9f0 tx=0x7f56a400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:37.578 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.576+0000 7f56aa7fc640 1 -- 172.21.15.72:0/2397123517 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56a400cd80 con 0x7f56b41013f0 2024-03-22T01:37:37.578 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.576+0000 7f56bb7c7640 1 -- 172.21.15.72:0/2397123517 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56b419dcf0 con 0x7f56b41013f0 2024-03-22T01:37:37.578 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.576+0000 7f56aa7fc640 1 -- 172.21.15.72:0/2397123517 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f56a4005500 con 0x7f56b41013f0 2024-03-22T01:37:37.578 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.576+0000 7f56aa7fc640 1 -- 172.21.15.72:0/2397123517 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56a4010690 con 0x7f56b41013f0 2024-03-22T01:37:37.578 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.576+0000 7f56bb7c7640 1 -- 172.21.15.72:0/2397123517 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56b41a54e0 con 0x7f56b41013f0 2024-03-22T01:37:37.580 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.578+0000 7f56aa7fc640 1 -- 172.21.15.72:0/2397123517 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 24) ==== 138087+0+0 (secure 0 0 0) 0x7f56a40046c0 con 0x7f56b41013f0 2024-03-22T01:37:37.580 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.579+0000 7f56aa7fc640 1 --2- 172.21.15.72:0/2397123517 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f56a00a31a0 0x7f56a00a5660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:37.580 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.579+0000 7f56b9563640 1 --2- 172.21.15.72:0/2397123517 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f56a00a31a0 0x7f56a00a5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:37.580 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.579+0000 7f56aa7fc640 1 -- 172.21.15.72:0/2397123517 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f56a40d02d0 con 0x7f56b41013f0 2024-03-22T01:37:37.580 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.580+0000 7f56be3a9740 1 Processor -- start 2024-03-22T01:37:37.580 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.580+0000 7f56be3a9740 1 -- start start 2024-03-22T01:37:37.580 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.580+0000 7f56be3a9740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c7d093d2d0 0x55c7d093d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:37.580 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.580+0000 7f56b9563640 1 --2- 172.21.15.72:0/2397123517 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f56a00a31a0 0x7f56a00a5660 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f56b0009230 tx=0x7f56b0005fd0 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:37.580 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.580+0000 7f56be3a9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7d093dbf0 0x55c7d0946480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:37.580 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.580+0000 7f56be3a9740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c7d09469c0 0x55c7d0948db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:37.581 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.580+0000 7f56be3a9740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55c7d08a1040 con 0x55c7d093d2d0 2024-03-22T01:37:37.581 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.580+0000 7f56be3a9740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55c7d08499e0 con 0x55c7d09469c0 2024-03-22T01:37:37.581 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.580+0000 7f56be3a9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c7d089cff0 con 0x55c7d093dbf0 2024-03-22T01:37:37.581 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.580+0000 7f56b9563640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c7d093d2d0 0x55c7d093d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:37.581 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.580+0000 7f56b9563640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c7d093d2d0 0x55c7d093d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42460/0 (socket says 172.21.15.72:42460) 2024-03-22T01:37:37.581 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.580+0000 7f56b9563640 1 -- 172.21.15.72:0/2580090377 learned_addr learned my addr 172.21.15.72:0/2580090377 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:37:37.581 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.580+0000 7f56b9d64640 1 --2- 172.21.15.72:0/2580090377 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c7d09469c0 0x55c7d0948db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:37.581 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.580+0000 7f56b8d62640 1 --2- 172.21.15.72:0/2580090377 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7d093dbf0 0x55c7d0946480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:37.581 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.580+0000 7f56b9563640 1 -- 172.21.15.72:0/2580090377 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7d093dbf0 msgr2=0x55c7d0946480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:37.581 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.580+0000 7f56b9563640 1 --2- 172.21.15.72:0/2580090377 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7d093dbf0 0x55c7d0946480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:37.581 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.580+0000 7f56b9563640 1 -- 172.21.15.72:0/2580090377 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c7d09469c0 msgr2=0x55c7d0948db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:37.581 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.580+0000 7f56b9563640 1 --2- 172.21.15.72:0/2580090377 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c7d09469c0 0x55c7d0948db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:37.581 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.580+0000 7f56b9563640 1 -- 172.21.15.72:0/2580090377 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c7d05b5210 con 0x55c7d093d2d0 2024-03-22T01:37:37.581 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.581+0000 7f56b9d64640 1 --2- 172.21.15.72:0/2580090377 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c7d09469c0 0x55c7d0948db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:37:37.582 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.581+0000 7f56b9563640 1 --2- 172.21.15.72:0/2580090377 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c7d093d2d0 0x55c7d093d6b0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f56b0035000 tx=0x7f56b002e050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f44481622a065905 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:37.582 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.581+0000 7f569f7fe640 1 -- 172.21.15.72:0/2580090377 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56b0004040 con 0x55c7d093d2d0 2024-03-22T01:37:37.582 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.581+0000 7f569f7fe640 1 -- 172.21.15.72:0/2580090377 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f56b0050070 con 0x55c7d093d2d0 2024-03-22T01:37:37.582 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.581+0000 7f569f7fe640 1 -- 172.21.15.72:0/2580090377 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56b0003850 con 0x55c7d093d2d0 2024-03-22T01:37:37.586 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.582+0000 7f56be3a9740 1 -- 172.21.15.72:0/2580090377 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c7d093d2d0 msgr2=0x55c7d093d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:37.586 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.582+0000 7f56be3a9740 1 --2- 172.21.15.72:0/2580090377 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c7d093d2d0 0x55c7d093d6b0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f56b0035000 tx=0x7f56b002e050 comp rx=0 tx=0).stop 2024-03-22T01:37:37.586 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.582+0000 7f56be3a9740 1 -- 172.21.15.72:0/2580090377 shutdown_connections 2024-03-22T01:37:37.586 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.582+0000 7f56be3a9740 1 --2- 172.21.15.72:0/2580090377 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c7d09469c0 0x55c7d0948db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:37.587 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.582+0000 7f56be3a9740 1 --2- 172.21.15.72:0/2580090377 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7d093dbf0 0x55c7d0946480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:37.587 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.582+0000 7f56be3a9740 1 --2- 172.21.15.72:0/2580090377 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c7d093d2d0 0x55c7d093d6b0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:37.587 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.582+0000 7f56be3a9740 1 -- 172.21.15.72:0/2580090377 >> 172.21.15.72:0/2580090377 conn(0x55c7d0937bf0 msgr2=0x55c7d0937fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:37.587 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.582+0000 7f56be3a9740 1 -- 172.21.15.72:0/2580090377 shutdown_connections 2024-03-22T01:37:37.587 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.582+0000 7f56be3a9740 1 -- 172.21.15.72:0/2580090377 wait complete. 2024-03-22T01:37:37.587 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.582+0000 7f56be3a9740 1 -- start start 2024-03-22T01:37:37.587 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.582+0000 7f56be3a9740 1 Processor -- start 2024-03-22T01:37:37.587 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.583+0000 7f56be3a9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7d093d2d0 0x55c7d09462f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:37.587 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.583+0000 7f56be3a9740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c7d093dbf0 0x55c7d0943680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:37.587 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.583+0000 7f56be3a9740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c7d09469c0 0x55c7d0943bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:37.587 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.583+0000 7f56be3a9740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55c7d094d9a0 con 0x55c7d093dbf0 2024-03-22T01:37:37.587 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.583+0000 7f56be3a9740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55c7d085cdb0 con 0x55c7d09469c0 2024-03-22T01:37:37.587 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.583+0000 7f56be3a9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c7d094db20 con 0x55c7d093d2d0 2024-03-22T01:37:37.587 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.583+0000 7f56be3a9740 10 client.0 _ll_register_callbacks cb 0x55c7d05b4a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:37:37.587 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.583+0000 7f56b8d62640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c7d093dbf0 0x55c7d0943680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:37.588 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.583+0000 7f56b8d62640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c7d093dbf0 0x55c7d0943680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42468/0 (socket says 172.21.15.72:42468) 2024-03-22T01:37:37.588 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.583+0000 7f56b8d62640 1 -- 172.21.15.72:0/2785164412 learned_addr learned my addr 172.21.15.72:0/2785164412 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:37:37.588 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.583+0000 7f56b9563640 1 --2- 172.21.15.72:0/2785164412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7d093d2d0 0x55c7d09462f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:37.588 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.583+0000 7f56b9d64640 1 --2- 172.21.15.72:0/2785164412 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c7d09469c0 0x55c7d0943bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:37.588 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.583+0000 7f56b8d62640 1 -- 172.21.15.72:0/2785164412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7d093d2d0 msgr2=0x55c7d09462f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:37.588 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.583+0000 7f56b8d62640 1 --2- 172.21.15.72:0/2785164412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7d093d2d0 0x55c7d09462f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:37.588 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.583+0000 7f56b8d62640 1 -- 172.21.15.72:0/2785164412 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c7d09469c0 msgr2=0x55c7d0943bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:37.588 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.584+0000 7f56b8d62640 1 --2- 172.21.15.72:0/2785164412 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c7d09469c0 0x55c7d0943bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:37.588 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.584+0000 7f56b8d62640 1 -- 172.21.15.72:0/2785164412 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c7d0681620 con 0x55c7d093dbf0 2024-03-22T01:37:37.588 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.584+0000 7f56b9d64640 1 --2- 172.21.15.72:0/2785164412 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c7d09469c0 0x55c7d0943bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:37:37.588 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.584+0000 7f56b8d62640 1 --2- 172.21.15.72:0/2785164412 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c7d093dbf0 0x55c7d0943680 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f56a4098520 tx=0x7f56a409c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=963c701adad89e74 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:37.588 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.584+0000 7f569cff9640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:37:37.588 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.584+0000 7f569cff9640 1 -- 172.21.15.72:0/2785164412 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56a40b4070 con 0x55c7d093dbf0 2024-03-22T01:37:37.588 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.584+0000 7f56be3a9740 1 -- 172.21.15.72:0/2785164412 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f56b0050070 con 0x55c7d093dbf0 2024-03-22T01:37:37.589 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.584+0000 7f569cff9640 1 -- 172.21.15.72:0/2785164412 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f56a409c520 con 0x55c7d093dbf0 2024-03-22T01:37:37.589 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.584+0000 7f569cff9640 1 -- 172.21.15.72:0/2785164412 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f56a40bc2d0 con 0x55c7d093dbf0 2024-03-22T01:37:37.589 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.584+0000 7f56be3a9740 10 client.15780 fetch_fsmap learned FSMap version 21 2024-03-22T01:37:37.589 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.584+0000 7f56be3a9740 1 -- 172.21.15.72:0/2785164412 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55c7d0959960 con 0x55c7d093dbf0 2024-03-22T01:37:37.589 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.584+0000 7f569cff9640 1 -- 172.21.15.72:0/2785164412 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f56b0050070 con 0x55c7d093dbf0 2024-03-22T01:37:37.589 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.584+0000 7f569cff9640 1 -- 172.21.15.72:0/2785164412 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f56a40cd7d0 con 0x55c7d093dbf0 2024-03-22T01:37:37.589 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.585+0000 7f56be3a9740 10 client.15780 fetch_fsmap finished waiting for FSMap version 21 2024-03-22T01:37:37.590 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.585+0000 7f56be3a9740 10 client.15780 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T01:37:37.590 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.585+0000 7f56be3a9740 20 client.15780 populate_metadata read hostname 'smithi072' 2024-03-22T01:37:37.590 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.585+0000 7f56be3a9740 1 --2- 172.21.15.72:0/2785164412 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55c7d095ae70 0x55c7d095d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:37.590 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.585+0000 7f56be3a9740 4 client.15780 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T01:37:37.590 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.585+0000 7f56be3a9740 1 -- 172.21.15.72:0/2785164412 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c7d08ae580 con 0x55c7d095ae70 2024-03-22T01:37:37.590 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.585+0000 7f56b9563640 1 --2- 172.21.15.72:0/2785164412 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55c7d095ae70 0x55c7d095d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:37.590 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.585+0000 7f569cff9640 10 client.15780 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:37:37.590 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.585+0000 7f56b9563640 1 --2- 172.21.15.72:0/2785164412 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55c7d095ae70 0x55c7d095d260 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c83d7834564ab8ea server_cookie=2dda457ff2a94955 in_seq=0 out_seq=0 2024-03-22T01:37:37.590 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.586+0000 7f569cff9640 1 -- 172.21.15.72:0/2785164412 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c7d08ae580 con 0x55c7d095ae70 2024-03-22T01:37:37.591 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.586+0000 7f569cff9640 10 client.15780 handle_command_reply: tid=0 2024-03-22T01:37:37.591 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.586+0000 7f56be3a9740 1 client.15780 shutdown 2024-03-22T01:37:37.591 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.586+0000 7f56be3a9740 1 -- 172.21.15.72:0/2785164412 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c7d093dbf0 msgr2=0x55c7d0943680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:37.591 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.586+0000 7f56be3a9740 1 --2- 172.21.15.72:0/2785164412 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c7d093dbf0 0x55c7d0943680 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f56a4098520 tx=0x7f56a409c880 comp rx=0 tx=0).stop 2024-03-22T01:37:37.591 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.586+0000 7f56be3a9740 1 -- 172.21.15.72:0/2785164412 shutdown_connections 2024-03-22T01:37:37.591 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.586+0000 7f56be3a9740 1 --2- 172.21.15.72:0/2785164412 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55c7d095ae70 0x55c7d095d260 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:37.591 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.586+0000 7f569cff9640 0 client.15780 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:37:37.591 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.586+0000 7f56be3a9740 1 --2- 172.21.15.72:0/2785164412 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c7d09469c0 0x55c7d0943bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:37.591 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.586+0000 7f56be3a9740 1 --2- 172.21.15.72:0/2785164412 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c7d093dbf0 0x55c7d0943680 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:37.591 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.586+0000 7f56be3a9740 1 --2- 172.21.15.72:0/2785164412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7d093d2d0 0x55c7d09462f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:37.591 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.586+0000 7f56be3a9740 1 -- 172.21.15.72:0/2785164412 >> 172.21.15.72:0/2785164412 conn(0x55c7d0937bf0 msgr2=0x55c7d0939940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:37.591 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.586+0000 7f56be3a9740 1 -- 172.21.15.72:0/2785164412 shutdown_connections 2024-03-22T01:37:37.591 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.586+0000 7f56be3a9740 1 -- 172.21.15.72:0/2785164412 wait complete. 2024-03-22T01:37:37.591 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.586+0000 7f56be3a9740 20 client.15780 trim_cache size 0 max 16384 2024-03-22T01:37:37.606 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.601+0000 7f56be3a9740 1 Processor -- start 2024-03-22T01:37:37.606 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.601+0000 7f56be3a9740 1 -- start start 2024-03-22T01:37:37.606 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.601+0000 7f56be3a9740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c7d093d2d0 0x55c7d0943ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:37.606 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.601+0000 7f56be3a9740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c7d093dbf0 0x55c7d0956740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:37.607 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.601+0000 7f56be3a9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7d09469c0 0x55c7d0962b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:37.607 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.601+0000 7f56be3a9740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f56b41ab400 con 0x55c7d093d2d0 2024-03-22T01:37:37.607 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.601+0000 7f56be3a9740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f56b41adc80 con 0x55c7d093dbf0 2024-03-22T01:37:37.607 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.601+0000 7f56be3a9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f56b41aba70 con 0x55c7d09469c0 2024-03-22T01:37:37.607 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.601+0000 7f56b9563640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c7d093d2d0 0x55c7d0943ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:37.607 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.601+0000 7f56b9563640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c7d093d2d0 0x55c7d0943ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42484/0 (socket says 172.21.15.72:42484) 2024-03-22T01:37:37.607 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.601+0000 7f56b9563640 1 -- 172.21.15.72:0/1205426461 learned_addr learned my addr 172.21.15.72:0/1205426461 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:37:37.607 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.601+0000 7f56b8d62640 1 --2- 172.21.15.72:0/1205426461 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c7d093dbf0 0x55c7d0956740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:37.607 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.602+0000 7f56b9563640 1 -- 172.21.15.72:0/1205426461 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7d09469c0 msgr2=0x55c7d0962b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:37.607 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.602+0000 7f56b9d64640 1 --2- 172.21.15.72:0/1205426461 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7d09469c0 0x55c7d0962b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:37.607 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.602+0000 7f56b9563640 1 --2- 172.21.15.72:0/1205426461 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7d09469c0 0x55c7d0962b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:37.608 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.602+0000 7f56b9563640 1 -- 172.21.15.72:0/1205426461 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c7d093dbf0 msgr2=0x55c7d0956740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:37.608 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.602+0000 7f56b9563640 1 --2- 172.21.15.72:0/1205426461 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c7d093dbf0 0x55c7d0956740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:37.608 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.602+0000 7f56b9563640 1 -- 172.21.15.72:0/1205426461 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c7d093da20 con 0x55c7d093d2d0 2024-03-22T01:37:37.608 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.602+0000 7f56b8d62640 1 --2- 172.21.15.72:0/1205426461 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c7d093dbf0 0x55c7d0956740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:37:37.608 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.602+0000 7f56b9d64640 1 --2- 172.21.15.72:0/1205426461 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7d09469c0 0x55c7d0962b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T01:37:37.608 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.602+0000 7f56b9563640 1 --2- 172.21.15.72:0/1205426461 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c7d093d2d0 0x55c7d0943ba0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f56b0005c40 tx=0x7f56b002ce00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c194a1f183d8bb5d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:37.608 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.602+0000 7f569effd640 1 -- 172.21.15.72:0/1205426461 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56b0042050 con 0x55c7d093d2d0 2024-03-22T01:37:37.608 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.602+0000 7f569effd640 1 -- 172.21.15.72:0/1205426461 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f56b005e070 con 0x55c7d093d2d0 2024-03-22T01:37:37.608 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.602+0000 7f569effd640 1 -- 172.21.15.72:0/1205426461 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56b0005430 con 0x55c7d093d2d0 2024-03-22T01:37:37.609 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.603+0000 7f56be3a9740 1 -- 172.21.15.72:0/1205426461 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c7d093d2d0 msgr2=0x55c7d0943ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:37.609 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.603+0000 7f56be3a9740 1 --2- 172.21.15.72:0/1205426461 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c7d093d2d0 0x55c7d0943ba0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f56b0005c40 tx=0x7f56b002ce00 comp rx=0 tx=0).stop 2024-03-22T01:37:37.609 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.603+0000 7f56be3a9740 1 -- 172.21.15.72:0/1205426461 shutdown_connections 2024-03-22T01:37:37.609 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.603+0000 7f56be3a9740 1 --2- 172.21.15.72:0/1205426461 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c7d093dbf0 0x55c7d0956740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:37.609 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.603+0000 7f56be3a9740 1 --2- 172.21.15.72:0/1205426461 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7d09469c0 0x55c7d0962b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:37.609 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.603+0000 7f56be3a9740 1 --2- 172.21.15.72:0/1205426461 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c7d093d2d0 0x55c7d0943ba0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:37.609 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.603+0000 7f56be3a9740 1 -- 172.21.15.72:0/1205426461 >> 172.21.15.72:0/1205426461 conn(0x55c7d0937bf0 msgr2=0x55c7d0936c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:37.609 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.603+0000 7f56be3a9740 1 -- 172.21.15.72:0/1205426461 shutdown_connections 2024-03-22T01:37:37.609 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.603+0000 7f56be3a9740 1 -- 172.21.15.72:0/1205426461 wait complete. 2024-03-22T01:37:37.610 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.603+0000 7f56be3a9740 1 -- start start 2024-03-22T01:37:37.610 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.603+0000 7f56be3a9740 1 Processor -- start 2024-03-22T01:37:37.610 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.604+0000 7f56be3a9740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c7d093d2d0 0x55c7d0952f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:37.610 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.604+0000 7f56be3a9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7d093dbf0 0x55c7d094e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:37.610 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.604+0000 7f56be3a9740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c7d09469c0 0x55c7d094e890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:37.610 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.604+0000 7f56be3a9740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f56b41ac140 con 0x55c7d093d2d0 2024-03-22T01:37:37.610 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.604+0000 7f56be3a9740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f56b41ad5b0 con 0x55c7d09469c0 2024-03-22T01:37:37.610 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.604+0000 7f56be3a9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f56b41ac810 con 0x55c7d093dbf0 2024-03-22T01:37:37.610 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.604+0000 7f56be3a9740 10 client.0 _ll_register_callbacks cb 0x7f56a400e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:37:37.610 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.604+0000 7f56b9563640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c7d093d2d0 0x55c7d0952f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:37.610 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.604+0000 7f56b9563640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c7d093d2d0 0x55c7d0952f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42494/0 (socket says 172.21.15.72:42494) 2024-03-22T01:37:37.611 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.604+0000 7f56b9563640 1 -- 172.21.15.72:0/3867891395 learned_addr learned my addr 172.21.15.72:0/3867891395 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:37:37.611 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.604+0000 7f56b9d64640 1 --2- 172.21.15.72:0/3867891395 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c7d09469c0 0x55c7d094e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:37.611 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.604+0000 7f56b8d62640 1 --2- 172.21.15.72:0/3867891395 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7d093dbf0 0x55c7d094e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:37.611 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.604+0000 7f56b9d64640 1 -- 172.21.15.72:0/3867891395 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7d093dbf0 msgr2=0x55c7d094e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:37.611 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.604+0000 7f56b9d64640 1 --2- 172.21.15.72:0/3867891395 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7d093dbf0 0x55c7d094e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:37.611 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.604+0000 7f56b9d64640 1 -- 172.21.15.72:0/3867891395 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c7d093d2d0 msgr2=0x55c7d0952f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:37.611 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.604+0000 7f56b9d64640 1 --2- 172.21.15.72:0/3867891395 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c7d093d2d0 0x55c7d0952f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:37.611 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.604+0000 7f56b9d64640 1 -- 172.21.15.72:0/3867891395 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c7d094ef00 con 0x55c7d09469c0 2024-03-22T01:37:37.612 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.604+0000 7f56b9563640 1 --2- 172.21.15.72:0/3867891395 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c7d093d2d0 0x55c7d0952f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:37:37.612 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.605+0000 7f56b9d64640 1 --2- 172.21.15.72:0/3867891395 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c7d09469c0 0x55c7d094e890 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x55c7d0946990 tx=0x7f56ac014f50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9c6de5d97497efd server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:37.612 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.605+0000 7f569cff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-03-22T01:37:37.612 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.605+0000 7f569cff9640 1 -- 172.21.15.72:0/3867891395 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56ac014340 con 0x55c7d09469c0 2024-03-22T01:37:37.612 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.605+0000 7f56be3a9740 1 -- 172.21.15.72:0/3867891395 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f56b005e070 con 0x55c7d09469c0 2024-03-22T01:37:37.612 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.605+0000 7f569cff9640 1 -- 172.21.15.72:0/3867891395 <== mon.2 v2:172.21.15.89:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f56ac004b20 con 0x55c7d09469c0 2024-03-22T01:37:37.612 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.605+0000 7f569cff9640 1 -- 172.21.15.72:0/3867891395 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f56ac027050 con 0x55c7d09469c0 2024-03-22T01:37:37.612 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.605+0000 7f56be3a9740 10 client.25634 fetch_fsmap learned FSMap version 21 2024-03-22T01:37:37.613 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.605+0000 7f56be3a9740 1 -- 172.21.15.72:0/3867891395 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55c7d09c1060 con 0x55c7d09469c0 2024-03-22T01:37:37.613 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.605+0000 7f569cff9640 1 -- 172.21.15.72:0/3867891395 <== mon.2 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f56b005e070 con 0x55c7d09469c0 2024-03-22T01:37:37.613 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.605+0000 7f569cff9640 1 -- 172.21.15.72:0/3867891395 <== mon.2 v2:172.21.15.89:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f56ac00f040 con 0x55c7d09469c0 2024-03-22T01:37:37.613 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.605+0000 7f56be3a9740 10 client.25634 fetch_fsmap finished waiting for FSMap version 21 2024-03-22T01:37:37.613 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.605+0000 7f56be3a9740 10 client.25634 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T01:37:37.613 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.605+0000 7f56be3a9740 20 client.25634 populate_metadata read hostname 'smithi072' 2024-03-22T01:37:37.613 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.605+0000 7f56be3a9740 1 --2- 172.21.15.72:0/3867891395 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55c7d095ae70 0x55c7d09c3770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:37.613 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.606+0000 7f56be3a9740 4 client.25634 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T01:37:37.614 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.606+0000 7f56be3a9740 1 -- 172.21.15.72:0/3867891395 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c7d08a88d0 con 0x55c7d095ae70 2024-03-22T01:37:37.614 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.606+0000 7f56b9563640 1 --2- 172.21.15.72:0/3867891395 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55c7d095ae70 0x55c7d09c3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:37.614 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.606+0000 7f569cff9640 10 client.25634 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:37:37.614 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.606+0000 7f56b9563640 1 --2- 172.21.15.72:0/3867891395 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55c7d095ae70 0x55c7d09c3770 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa3452d19985397d server_cookie=769416710e4c118a in_seq=0 out_seq=0 2024-03-22T01:37:37.616 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.609+0000 7f569cff9640 1 -- 172.21.15.72:0/3867891395 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c7d08a88d0 con 0x55c7d095ae70 2024-03-22T01:37:37.616 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.609+0000 7f569cff9640 10 client.25634 handle_command_reply: tid=0 2024-03-22T01:37:37.616 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.609+0000 7f56be3a9740 1 client.25634 shutdown 2024-03-22T01:37:37.616 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.609+0000 7f56be3a9740 1 -- 172.21.15.72:0/3867891395 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c7d09469c0 msgr2=0x55c7d094e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:37.616 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.609+0000 7f56be3a9740 1 --2- 172.21.15.72:0/3867891395 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c7d09469c0 0x55c7d094e890 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x55c7d0946990 tx=0x7f56ac014f50 comp rx=0 tx=0).stop 2024-03-22T01:37:37.616 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.609+0000 7f56be3a9740 1 -- 172.21.15.72:0/3867891395 shutdown_connections 2024-03-22T01:37:37.616 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.609+0000 7f56be3a9740 1 --2- 172.21.15.72:0/3867891395 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55c7d095ae70 0x55c7d09c3770 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:37.616 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.610+0000 7f569cff9640 0 client.25634 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:37:37.616 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.609+0000 7f56be3a9740 1 --2- 172.21.15.72:0/3867891395 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c7d09469c0 0x55c7d094e890 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:37.616 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.610+0000 7f56be3a9740 1 --2- 172.21.15.72:0/3867891395 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c7d093dbf0 0x55c7d094e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:37.616 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.610+0000 7f56be3a9740 1 --2- 172.21.15.72:0/3867891395 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55c7d093d2d0 0x55c7d0952f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:37.617 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.610+0000 7f56be3a9740 1 -- 172.21.15.72:0/3867891395 >> 172.21.15.72:0/3867891395 conn(0x55c7d0937bf0 msgr2=0x55c7d09398d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:37.617 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.610+0000 7f56be3a9740 1 -- 172.21.15.72:0/3867891395 shutdown_connections 2024-03-22T01:37:37.617 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.610+0000 7f56be3a9740 1 -- 172.21.15.72:0/3867891395 wait complete. 2024-03-22T01:37:37.617 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.610+0000 7f56be3a9740 20 client.25634 trim_cache size 0 max 16384 2024-03-22T01:37:37.618 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout:{ 2024-03-22T01:37:37.618 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout: "message": "", 2024-03-22T01:37:37.618 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout: "return_code": 0 2024-03-22T01:37:37.618 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout:} 2024-03-22T01:37:37.620 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.610+0000 7f56bb7c7640 1 -- 172.21.15.72:0/2397123517 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f56a00a31a0 msgr2=0x7f56a00a5660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:37.620 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.610+0000 7f56bb7c7640 1 --2- 172.21.15.72:0/2397123517 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f56a00a31a0 0x7f56a00a5660 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f56b0009230 tx=0x7f56b0005fd0 comp rx=0 tx=0).stop 2024-03-22T01:37:37.620 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.610+0000 7f56bb7c7640 1 -- 172.21.15.72:0/2397123517 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f56b41013f0 msgr2=0x7f56b4198be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:37.620 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.610+0000 7f56bb7c7640 1 --2- 172.21.15.72:0/2397123517 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f56b41013f0 0x7f56b4198be0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f56a400e9f0 tx=0x7f56a400edf0 comp rx=0 tx=0).stop 2024-03-22T01:37:37.620 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.610+0000 7f56bb7c7640 1 -- 172.21.15.72:0/2397123517 shutdown_connections 2024-03-22T01:37:37.620 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.610+0000 7f56bb7c7640 1 --2- 172.21.15.72:0/2397123517 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f56a00a31a0 0x7f56a00a5660 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:37.620 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.610+0000 7f56bb7c7640 1 --2- 172.21.15.72:0/2397123517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56b419cf10 0x7f56b419d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:37.620 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.610+0000 7f56bb7c7640 1 --2- 172.21.15.72:0/2397123517 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f56b41013f0 0x7f56b4198be0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:37.620 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.610+0000 7f56bb7c7640 1 --2- 172.21.15.72:0/2397123517 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56b4100a30 0x7f56b41986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:37.620 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.610+0000 7f56bb7c7640 1 -- 172.21.15.72:0/2397123517 >> 172.21.15.72:0/2397123517 conn(0x7f56b4077b80 msgr2=0x7f56b4069800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:37.621 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.610+0000 7f56bb7c7640 1 -- 172.21.15.72:0/2397123517 shutdown_connections 2024-03-22T01:37:37.621 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:37.611+0000 7f56bb7c7640 1 -- 172.21.15.72:0/2397123517 wait complete. 2024-03-22T01:37:38.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:37:37 smithi089 ceph-mon[35770]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-22T01:37:38.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:37:37 smithi072 ceph-mon[32414]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-22T01:37:38.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:37:37 smithi098 ceph-mon[36468]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-22T01:37:40.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:37:39 smithi089 ceph-mon[35770]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-22T01:37:40.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:37:39 smithi072 ceph-mon[32414]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-22T01:37:40.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:37:39 smithi098 ceph-mon[36468]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-22T01:37:42.357 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:37:41 smithi089 ceph-mon[35770]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-03-22T01:37:42.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:37:41 smithi072 ceph-mon[32414]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-03-22T01:37:42.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:37:41 smithi098 ceph-mon[36468]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-03-22T01:37:44.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:37:43 smithi089 ceph-mon[35770]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-22T01:37:44.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:37:43 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:37:44.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:37:43 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:37:44.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:37:43 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:37:44.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:37:43 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:37:44.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:37:43 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:37:44.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:37:43 smithi072 ceph-mon[32414]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-22T01:37:44.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:37:43 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:37:44.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:37:43 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:37:44.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:37:43 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:37:44.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:37:43 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:37:44.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:37:43 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:37:44.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:37:43 smithi098 ceph-mon[36468]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-22T01:37:44.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:37:43 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:37:44.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:37:43 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:37:44.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:37:43 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:37:44.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:37:43 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:37:44.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:37:43 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:37:45.357 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:37:44 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:37:45.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:37:44 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:37:45.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:37:44 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:37:45.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:37:44 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:37:45.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:37:44 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:37:45.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:37:44 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:37:45.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:37:44 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:37:45.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:37:44 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:37:45.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:37:44 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:37:45.770 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T01:37:45.771 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T01:37:45.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.880+0000 7fdcc0efb640 1 -- 172.21.15.72:0/2432304124 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcbc10c180 msgr2=0x7fdcbc10e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:45.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.880+0000 7fdcc0efb640 1 --2- 172.21.15.72:0/2432304124 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcbc10c180 0x7fdcbc10e5b0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fdcb4009be0 tx=0x7fdcb402bf80 comp rx=0 tx=0).stop 2024-03-22T01:37:45.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.880+0000 7fdcc0efb640 1 -- 172.21.15.72:0/2432304124 shutdown_connections 2024-03-22T01:37:45.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.880+0000 7fdcc0efb640 1 --2- 172.21.15.72:0/2432304124 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcbc10c180 0x7fdcbc10e5b0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:45.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.880+0000 7fdcc0efb640 1 --2- 172.21.15.72:0/2432304124 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdcbc1036d0 0x7fdcbc10bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:45.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.880+0000 7fdcc0efb640 1 --2- 172.21.15.72:0/2432304124 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcbc102db0 0x7fdcbc103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:45.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.880+0000 7fdcc0efb640 1 -- 172.21.15.72:0/2432304124 >> 172.21.15.72:0/2432304124 conn(0x7fdcbc077c40 msgr2=0x7fdcbc078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:45.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.880+0000 7fdcc0efb640 1 -- 172.21.15.72:0/2432304124 shutdown_connections 2024-03-22T01:37:45.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.880+0000 7fdcc0efb640 1 -- 172.21.15.72:0/2432304124 wait complete. 2024-03-22T01:37:45.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.881+0000 7fdcc0efb640 1 Processor -- start 2024-03-22T01:37:45.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.881+0000 7fdcc0efb640 1 -- start start 2024-03-22T01:37:45.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.881+0000 7fdcc0efb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcbc102db0 0x7fdcbc198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:45.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.881+0000 7fdcc0efb640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcbc1036d0 0x7fdcbc198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:45.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.881+0000 7fdcc0efb640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdcbc10c180 0x7fdcbc19cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:45.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.881+0000 7fdcc0efb640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fdcbc111f10 con 0x7fdcbc10c180 2024-03-22T01:37:45.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.881+0000 7fdcc0efb640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fdcbc111d90 con 0x7fdcbc1036d0 2024-03-22T01:37:45.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.881+0000 7fdcc0efb640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdcbc112090 con 0x7fdcbc102db0 2024-03-22T01:37:45.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.881+0000 7fdcb9d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcbc1036d0 0x7fdcbc198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:45.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.881+0000 7fdcb9d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcbc1036d0 0x7fdcbc198950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.72:47808/0 (socket says 172.21.15.72:47808) 2024-03-22T01:37:45.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.881+0000 7fdcb9d9b640 1 -- 172.21.15.72:0/3774623684 learned_addr learned my addr 172.21.15.72:0/3774623684 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:37:45.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.881+0000 7fdcbad9d640 1 --2- 172.21.15.72:0/3774623684 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdcbc10c180 0x7fdcbc19cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:45.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.881+0000 7fdcba59c640 1 --2- 172.21.15.72:0/3774623684 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcbc102db0 0x7fdcbc198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:45.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.881+0000 7fdcb9d9b640 1 -- 172.21.15.72:0/3774623684 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcbc102db0 msgr2=0x7fdcbc198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:45.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.881+0000 7fdcb9d9b640 1 --2- 172.21.15.72:0/3774623684 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcbc102db0 0x7fdcbc198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:45.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.881+0000 7fdcb9d9b640 1 -- 172.21.15.72:0/3774623684 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdcbc10c180 msgr2=0x7fdcbc19cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:45.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.881+0000 7fdcb9d9b640 1 --2- 172.21.15.72:0/3774623684 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdcbc10c180 0x7fdcbc19cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:45.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.881+0000 7fdcb9d9b640 1 -- 172.21.15.72:0/3774623684 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcbc19d3b0 con 0x7fdcbc1036d0 2024-03-22T01:37:45.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.881+0000 7fdcba59c640 1 --2- 172.21.15.72:0/3774623684 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcbc102db0 0x7fdcbc198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T01:37:45.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.882+0000 7fdcb9d9b640 1 --2- 172.21.15.72:0/3774623684 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcbc1036d0 0x7fdcbc198950 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fdca4012dd0 tx=0x7fdca400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:45.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.882+0000 7fdcb37fe640 1 -- 172.21.15.72:0/3774623684 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdca4004020 con 0x7fdcbc1036d0 2024-03-22T01:37:45.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.882+0000 7fdcb37fe640 1 -- 172.21.15.72:0/3774623684 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdca40041c0 con 0x7fdcbc1036d0 2024-03-22T01:37:45.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.882+0000 7fdcc0efb640 1 -- 172.21.15.72:0/3774623684 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcbc19d6a0 con 0x7fdcbc1036d0 2024-03-22T01:37:45.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.882+0000 7fdcb37fe640 1 -- 172.21.15.72:0/3774623684 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdca401a800 con 0x7fdcbc1036d0 2024-03-22T01:37:45.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.882+0000 7fdcc0efb640 1 -- 172.21.15.72:0/3774623684 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcbc100930 con 0x7fdcbc1036d0 2024-03-22T01:37:45.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.884+0000 7fdcb37fe640 1 -- 172.21.15.72:0/3774623684 <== mon.2 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 24) ==== 138087+0+0 (secure 0 0 0) 0x7fdca4021020 con 0x7fdcbc1036d0 2024-03-22T01:37:45.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.884+0000 7fdcb37fe640 1 --2- 172.21.15.72:0/3774623684 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fdca80a31b0 0x7fdca80a5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:45.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.885+0000 7fdcb37fe640 1 -- 172.21.15.72:0/3774623684 <== mon.2 v2:172.21.15.89:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fdca40d3970 con 0x7fdcbc1036d0 2024-03-22T01:37:45.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.885+0000 7fdcba59c640 1 --2- 172.21.15.72:0/3774623684 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fdca80a31b0 0x7fdca80a5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:45.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.885+0000 7fdcba59c640 1 --2- 172.21.15.72:0/3774623684 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fdca80a31b0 0x7fdca80a5670 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fdcac005d00 tx=0x7fdcac009550 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:45.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.886+0000 7fdcc3bc2740 1 Processor -- start 2024-03-22T01:37:45.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.886+0000 7fdcc3bc2740 1 -- start start 2024-03-22T01:37:45.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.886+0000 7fdcc3bc2740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56024cd22410 0x56024cd227f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:45.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.886+0000 7fdcc3bc2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56024cd22d30 0x56024cd2b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:45.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.886+0000 7fdcc3bc2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56024cd2b910 0x56024cd2dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:45.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.886+0000 7fdcc3bc2740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x56024cc868c0 con 0x56024cd22410 2024-03-22T01:37:45.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.886+0000 7fdcc3bc2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56024cc2ec20 con 0x56024cd22d30 2024-03-22T01:37:45.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.886+0000 7fdcc3bc2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56024cc82870 con 0x56024cd2b910 2024-03-22T01:37:45.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.886+0000 7fdcba59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56024cd22410 0x56024cd227f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:45.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.886+0000 7fdcba59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56024cd22410 0x56024cd227f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37944/0 (socket says 172.21.15.72:37944) 2024-03-22T01:37:45.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.887+0000 7fdcba59c640 1 -- 172.21.15.72:0/3116001927 learned_addr learned my addr 172.21.15.72:0/3116001927 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:37:45.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.887+0000 7fdcb9d9b640 1 --2- 172.21.15.72:0/3116001927 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56024cd22d30 0x56024cd2b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:45.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.887+0000 7fdcba59c640 1 -- 172.21.15.72:0/3116001927 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56024cd2b910 msgr2=0x56024cd2dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:45.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.887+0000 7fdcba59c640 1 --2- 172.21.15.72:0/3116001927 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56024cd2b910 0x56024cd2dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:45.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.887+0000 7fdcba59c640 1 -- 172.21.15.72:0/3116001927 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56024cd22d30 msgr2=0x56024cd2b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:45.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.887+0000 7fdcba59c640 1 --2- 172.21.15.72:0/3116001927 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56024cd22d30 0x56024cd2b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:45.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.887+0000 7fdcba59c640 1 -- 172.21.15.72:0/3116001927 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56024cc53150 con 0x56024cd22410 2024-03-22T01:37:45.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.887+0000 7fdcba59c640 1 --2- 172.21.15.72:0/3116001927 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56024cd22410 0x56024cd227f0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fdcac00eeb0 tx=0x7fdcac011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57c37a64e62470d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:45.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.887+0000 7fdcb0ff9640 1 -- 172.21.15.72:0/3116001927 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcac01e640 con 0x56024cd22410 2024-03-22T01:37:45.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.887+0000 7fdcb0ff9640 1 -- 172.21.15.72:0/3116001927 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdcac023070 con 0x56024cd22410 2024-03-22T01:37:45.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.887+0000 7fdcb0ff9640 1 -- 172.21.15.72:0/3116001927 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcac010a50 con 0x56024cd22410 2024-03-22T01:37:45.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.888+0000 7fdcc3bc2740 1 -- 172.21.15.72:0/3116001927 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56024cd22410 msgr2=0x56024cd227f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:45.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.888+0000 7fdcc3bc2740 1 --2- 172.21.15.72:0/3116001927 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56024cd22410 0x56024cd227f0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fdcac00eeb0 tx=0x7fdcac011000 comp rx=0 tx=0).stop 2024-03-22T01:37:45.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.888+0000 7fdcc3bc2740 1 -- 172.21.15.72:0/3116001927 shutdown_connections 2024-03-22T01:37:45.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.888+0000 7fdcc3bc2740 1 --2- 172.21.15.72:0/3116001927 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56024cd22d30 0x56024cd2b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:45.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.888+0000 7fdcc3bc2740 1 --2- 172.21.15.72:0/3116001927 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56024cd2b910 0x56024cd2dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:45.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.888+0000 7fdcc3bc2740 1 --2- 172.21.15.72:0/3116001927 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56024cd22410 0x56024cd227f0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:45.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.888+0000 7fdcc3bc2740 1 -- 172.21.15.72:0/3116001927 >> 172.21.15.72:0/3116001927 conn(0x56024cd1cb40 msgr2=0x56024cd1cf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:45.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.888+0000 7fdcc3bc2740 1 -- 172.21.15.72:0/3116001927 shutdown_connections 2024-03-22T01:37:45.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.888+0000 7fdcc3bc2740 1 -- 172.21.15.72:0/3116001927 wait complete. 2024-03-22T01:37:45.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.888+0000 7fdcc3bc2740 1 -- start start 2024-03-22T01:37:45.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.889+0000 7fdcc3bc2740 1 Processor -- start 2024-03-22T01:37:45.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.889+0000 7fdcc3bc2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56024cd22410 0x56024cd289e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:45.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.889+0000 7fdcc3bc2740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56024cd22d30 0x56024cd28f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:45.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.889+0000 7fdcc3bc2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56024cd2b910 0x56024cd29460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:45.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.889+0000 7fdcc3bc2740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x56024cd2e6a0 con 0x56024cd22d30 2024-03-22T01:37:45.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.889+0000 7fdcc3bc2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56024cc42130 con 0x56024cd2b910 2024-03-22T01:37:45.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.889+0000 7fdcc3bc2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56024cd2e820 con 0x56024cd22410 2024-03-22T01:37:45.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.889+0000 7fdcc3bc2740 10 client.0 _ll_register_callbacks cb 0x56024cc41b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:37:45.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.889+0000 7fdcb9d9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56024cd22d30 0x56024cd28f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:45.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.889+0000 7fdcbad9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56024cd2b910 0x56024cd29460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:45.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.889+0000 7fdcbad9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56024cd2b910 0x56024cd29460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.72:47818/0 (socket says 172.21.15.72:47818) 2024-03-22T01:37:45.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.889+0000 7fdcb9d9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56024cd22d30 0x56024cd28f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37946/0 (socket says 172.21.15.72:37946) 2024-03-22T01:37:45.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.889+0000 7fdcbad9d640 1 -- 172.21.15.72:0/4222676186 learned_addr learned my addr 172.21.15.72:0/4222676186 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:37:45.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.889+0000 7fdcba59c640 1 --2- 172.21.15.72:0/4222676186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56024cd22410 0x56024cd289e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:45.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.889+0000 7fdcbad9d640 1 -- 172.21.15.72:0/4222676186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56024cd22410 msgr2=0x56024cd289e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:45.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.889+0000 7fdcbad9d640 1 --2- 172.21.15.72:0/4222676186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56024cd22410 0x56024cd289e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:45.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.889+0000 7fdcbad9d640 1 -- 172.21.15.72:0/4222676186 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56024cd22d30 msgr2=0x56024cd28f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:45.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.889+0000 7fdcbad9d640 1 --2- 172.21.15.72:0/4222676186 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56024cd22d30 0x56024cd28f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:45.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.889+0000 7fdcbad9d640 1 -- 172.21.15.72:0/4222676186 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56024ca647d0 con 0x56024cd2b910 2024-03-22T01:37:45.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.889+0000 7fdcb9d9b640 1 --2- 172.21.15.72:0/4222676186 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56024cd22d30 0x56024cd28f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:37:45.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.890+0000 7fdcbad9d640 1 --2- 172.21.15.72:0/4222676186 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56024cd2b910 0x56024cd29460 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fdcb4036040 tx=0x7fdcb402cb50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4f33ffaa810644c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:45.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.890+0000 7fdca1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-03-22T01:37:45.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.890+0000 7fdca1ffb640 1 -- 172.21.15.72:0/4222676186 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcb4043030 con 0x56024cd2b910 2024-03-22T01:37:45.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.890+0000 7fdcc3bc2740 1 -- 172.21.15.72:0/4222676186 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdcac023070 con 0x56024cd2b910 2024-03-22T01:37:45.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.890+0000 7fdca1ffb640 1 -- 172.21.15.72:0/4222676186 <== mon.2 v2:172.21.15.89:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fdcb403a040 con 0x56024cd2b910 2024-03-22T01:37:45.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.890+0000 7fdca1ffb640 1 -- 172.21.15.72:0/4222676186 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fdcb4050050 con 0x56024cd2b910 2024-03-22T01:37:45.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.890+0000 7fdcc3bc2740 10 client.25646 fetch_fsmap learned FSMap version 21 2024-03-22T01:37:45.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.890+0000 7fdcc3bc2740 1 -- 172.21.15.72:0/4222676186 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x56024c99a3f0 con 0x56024cd2b910 2024-03-22T01:37:45.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.890+0000 7fdca1ffb640 1 -- 172.21.15.72:0/4222676186 <== mon.2 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdcac023070 con 0x56024cd2b910 2024-03-22T01:37:45.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.890+0000 7fdca1ffb640 1 -- 172.21.15.72:0/4222676186 <== mon.2 v2:172.21.15.89:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fdcb4033640 con 0x56024cd2b910 2024-03-22T01:37:45.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.890+0000 7fdcc3bc2740 10 client.25646 fetch_fsmap finished waiting for FSMap version 21 2024-03-22T01:37:45.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.890+0000 7fdcc3bc2740 10 client.25646 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T01:37:45.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.890+0000 7fdcc3bc2740 20 client.25646 populate_metadata read hostname 'smithi072' 2024-03-22T01:37:45.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.890+0000 7fdcc3bc2740 1 --2- 172.21.15.72:0/4222676186 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x56024cd3bcb0 0x56024cd3e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:45.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.890+0000 7fdcc3bc2740 4 client.25646 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T01:37:45.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.890+0000 7fdcc3bc2740 1 -- 172.21.15.72:0/4222676186 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56024cc93e00 con 0x56024cd3bcb0 2024-03-22T01:37:45.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.891+0000 7fdcba59c640 1 --2- 172.21.15.72:0/4222676186 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x56024cd3bcb0 0x56024cd3e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:45.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.891+0000 7fdca1ffb640 10 client.25646 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:37:45.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.891+0000 7fdcba59c640 1 --2- 172.21.15.72:0/4222676186 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x56024cd3bcb0 0x56024cd3e0a0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46f8720026284365 server_cookie=8c8831c0d0d6807c in_seq=0 out_seq=0 2024-03-22T01:37:45.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.892+0000 7fdca1ffb640 1 -- 172.21.15.72:0/4222676186 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56024cc93e00 con 0x56024cd3bcb0 2024-03-22T01:37:45.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.892+0000 7fdca1ffb640 10 client.25646 handle_command_reply: tid=0 2024-03-22T01:37:45.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.892+0000 7fdcc3bc2740 1 client.25646 shutdown 2024-03-22T01:37:45.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.892+0000 7fdcc3bc2740 1 -- 172.21.15.72:0/4222676186 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56024cd2b910 msgr2=0x56024cd29460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:45.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.892+0000 7fdcc3bc2740 1 --2- 172.21.15.72:0/4222676186 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56024cd2b910 0x56024cd29460 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fdcb4036040 tx=0x7fdcb402cb50 comp rx=0 tx=0).stop 2024-03-22T01:37:45.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.892+0000 7fdcc3bc2740 1 -- 172.21.15.72:0/4222676186 shutdown_connections 2024-03-22T01:37:45.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.892+0000 7fdcc3bc2740 1 --2- 172.21.15.72:0/4222676186 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x56024cd3bcb0 0x56024cd3e0a0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:45.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.892+0000 7fdcc3bc2740 1 --2- 172.21.15.72:0/4222676186 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56024cd2b910 0x56024cd29460 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:45.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.892+0000 7fdca1ffb640 0 client.25646 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:37:45.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.892+0000 7fdcc3bc2740 1 --2- 172.21.15.72:0/4222676186 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56024cd22d30 0x56024cd28f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:45.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.892+0000 7fdcc3bc2740 1 --2- 172.21.15.72:0/4222676186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56024cd22410 0x56024cd289e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:45.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.892+0000 7fdcc3bc2740 1 -- 172.21.15.72:0/4222676186 >> 172.21.15.72:0/4222676186 conn(0x56024cd1cb40 msgr2=0x56024cd1e890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:45.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.892+0000 7fdcc3bc2740 1 -- 172.21.15.72:0/4222676186 shutdown_connections 2024-03-22T01:37:45.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.892+0000 7fdcc3bc2740 1 -- 172.21.15.72:0/4222676186 wait complete. 2024-03-22T01:37:45.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.892+0000 7fdcc3bc2740 20 client.25646 trim_cache size 0 max 16384 2024-03-22T01:37:45.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.909+0000 7fdcc3bc2740 1 Processor -- start 2024-03-22T01:37:45.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.909+0000 7fdcc3bc2740 1 -- start start 2024-03-22T01:37:45.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.909+0000 7fdcc3bc2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56024cd22410 0x56024cd2b110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:45.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.909+0000 7fdcc3bc2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56024cd22d30 0x56024cd37440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:45.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.909+0000 7fdcc3bc2740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56024cd2b910 0x56024cd1e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:45.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.909+0000 7fdcc3bc2740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fdcbc1aae90 con 0x56024cd2b910 2024-03-22T01:37:45.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.909+0000 7fdcc3bc2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fdcbc1ad710 con 0x56024cd22410 2024-03-22T01:37:45.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.909+0000 7fdcc3bc2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdcbc1ab500 con 0x56024cd22d30 2024-03-22T01:37:45.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.910+0000 7fdcba59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56024cd22410 0x56024cd2b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:45.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.910+0000 7fdcbad9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56024cd2b910 0x56024cd1e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:45.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.910+0000 7fdcba59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56024cd22410 0x56024cd2b110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.72:47832/0 (socket says 172.21.15.72:47832) 2024-03-22T01:37:45.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.910+0000 7fdcba59c640 1 -- 172.21.15.72:0/3824685755 learned_addr learned my addr 172.21.15.72:0/3824685755 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:37:45.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.910+0000 7fdcb9d9b640 1 --2- 172.21.15.72:0/3824685755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56024cd22d30 0x56024cd37440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:45.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.910+0000 7fdcba59c640 1 -- 172.21.15.72:0/3824685755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56024cd22d30 msgr2=0x56024cd37440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:45.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.910+0000 7fdcba59c640 1 --2- 172.21.15.72:0/3824685755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56024cd22d30 0x56024cd37440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:45.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.910+0000 7fdcba59c640 1 -- 172.21.15.72:0/3824685755 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56024cd2b910 msgr2=0x56024cd1e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:45.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.910+0000 7fdcba59c640 1 --2- 172.21.15.72:0/3824685755 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56024cd2b910 0x56024cd1e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:45.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.910+0000 7fdcba59c640 1 -- 172.21.15.72:0/3824685755 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56024cd22b60 con 0x56024cd22410 2024-03-22T01:37:45.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.910+0000 7fdcba59c640 1 --2- 172.21.15.72:0/3824685755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56024cd22410 0x56024cd2b110 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fdcac013040 tx=0x7fdcac0052d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6c4eb6e9968621b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:45.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.910+0000 7fdca3fff640 1 -- 172.21.15.72:0/3824685755 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcac0055c0 con 0x56024cd22410 2024-03-22T01:37:45.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.910+0000 7fdca3fff640 1 -- 172.21.15.72:0/3824685755 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdcac037070 con 0x56024cd22410 2024-03-22T01:37:45.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.910+0000 7fdca3fff640 1 -- 172.21.15.72:0/3824685755 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcac0047e0 con 0x56024cd22410 2024-03-22T01:37:45.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.911+0000 7fdcc3bc2740 1 -- 172.21.15.72:0/3824685755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56024cd22410 msgr2=0x56024cd2b110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:45.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.911+0000 7fdcc3bc2740 1 --2- 172.21.15.72:0/3824685755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56024cd22410 0x56024cd2b110 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fdcac013040 tx=0x7fdcac0052d0 comp rx=0 tx=0).stop 2024-03-22T01:37:45.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.911+0000 7fdcc3bc2740 1 -- 172.21.15.72:0/3824685755 shutdown_connections 2024-03-22T01:37:45.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.911+0000 7fdcc3bc2740 1 --2- 172.21.15.72:0/3824685755 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56024cd2b910 0x56024cd1e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:45.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.911+0000 7fdcc3bc2740 1 --2- 172.21.15.72:0/3824685755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56024cd22d30 0x56024cd37440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:45.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.911+0000 7fdcc3bc2740 1 --2- 172.21.15.72:0/3824685755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56024cd22410 0x56024cd2b110 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:45.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.911+0000 7fdcc3bc2740 1 -- 172.21.15.72:0/3824685755 >> 172.21.15.72:0/3824685755 conn(0x56024cd1cb40 msgr2=0x56024cd1df70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:45.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.911+0000 7fdcc3bc2740 1 -- 172.21.15.72:0/3824685755 shutdown_connections 2024-03-22T01:37:45.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.911+0000 7fdcc3bc2740 1 -- 172.21.15.72:0/3824685755 wait complete. 2024-03-22T01:37:45.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.911+0000 7fdcc3bc2740 1 -- start start 2024-03-22T01:37:45.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.911+0000 7fdcc3bc2740 1 Processor -- start 2024-03-22T01:37:45.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.912+0000 7fdcc3bc2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56024cd22410 0x56024cd2fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:45.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.912+0000 7fdcc3bc2740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56024cd22d30 0x56024cd2ff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:45.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.912+0000 7fdcc3bc2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56024cd2b910 0x56024cda0f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:45.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.912+0000 7fdcc3bc2740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fdcbc1abbd0 con 0x56024cd22d30 2024-03-22T01:37:45.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.912+0000 7fdcc3bc2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fdcbc1ad040 con 0x56024cd2b910 2024-03-22T01:37:45.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.912+0000 7fdcc3bc2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdcbc1ac2a0 con 0x56024cd22410 2024-03-22T01:37:45.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.912+0000 7fdcc3bc2740 10 client.0 _ll_register_callbacks cb 0x7fdcb4003da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:37:45.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.912+0000 7fdcb9d9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56024cd22d30 0x56024cd2ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:45.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.912+0000 7fdcbad9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56024cd2b910 0x56024cda0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:45.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.912+0000 7fdcb9d9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56024cd22d30 0x56024cd2ff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37958/0 (socket says 172.21.15.72:37958) 2024-03-22T01:37:45.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.912+0000 7fdcb9d9b640 1 -- 172.21.15.72:0/336481354 learned_addr learned my addr 172.21.15.72:0/336481354 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:37:45.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.912+0000 7fdcba59c640 1 --2- 172.21.15.72:0/336481354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56024cd22410 0x56024cd2fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:45.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.912+0000 7fdcbad9d640 1 -- 172.21.15.72:0/336481354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56024cd22410 msgr2=0x56024cd2fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:45.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.912+0000 7fdcbad9d640 1 --2- 172.21.15.72:0/336481354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56024cd22410 0x56024cd2fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:45.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.912+0000 7fdcbad9d640 1 -- 172.21.15.72:0/336481354 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56024cd22d30 msgr2=0x56024cd2ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:45.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.912+0000 7fdcbad9d640 1 --2- 172.21.15.72:0/336481354 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56024cd22d30 0x56024cd2ff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:45.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.912+0000 7fdcbad9d640 1 -- 172.21.15.72:0/336481354 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56024c95a0f0 con 0x56024cd2b910 2024-03-22T01:37:45.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.912+0000 7fdcbad9d640 1 --2- 172.21.15.72:0/336481354 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56024cd2b910 0x56024cda0f80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fdcb4033bb0 tx=0x7fdcb4033e60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8209caca199c4d52 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:45.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.912+0000 7fdca1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-03-22T01:37:45.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.912+0000 7fdca1ffb640 1 -- 172.21.15.72:0/336481354 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcb4043070 con 0x56024cd2b910 2024-03-22T01:37:45.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.913+0000 7fdca1ffb640 1 -- 172.21.15.72:0/336481354 <== mon.2 v2:172.21.15.89:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fdcb4003e50 con 0x56024cd2b910 2024-03-22T01:37:45.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.913+0000 7fdcc3bc2740 1 -- 172.21.15.72:0/336481354 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdcac037070 con 0x56024cd2b910 2024-03-22T01:37:45.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.913+0000 7fdca1ffb640 1 -- 172.21.15.72:0/336481354 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fdcb4009800 con 0x56024cd2b910 2024-03-22T01:37:45.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.913+0000 7fdcc3bc2740 10 client.25652 fetch_fsmap learned FSMap version 21 2024-03-22T01:37:45.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.913+0000 7fdcc3bc2740 1 -- 172.21.15.72:0/336481354 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x56024cda2570 con 0x56024cd2b910 2024-03-22T01:37:45.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.913+0000 7fdca1ffb640 1 -- 172.21.15.72:0/336481354 <== mon.2 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdcac037070 con 0x56024cd2b910 2024-03-22T01:37:45.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.913+0000 7fdca1ffb640 1 -- 172.21.15.72:0/336481354 <== mon.2 v2:172.21.15.89:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fdcb403f4d0 con 0x56024cd2b910 2024-03-22T01:37:45.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.913+0000 7fdcc3bc2740 10 client.25652 fetch_fsmap finished waiting for FSMap version 21 2024-03-22T01:37:45.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.913+0000 7fdcc3bc2740 10 client.25652 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T01:37:45.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.913+0000 7fdcc3bc2740 20 client.25652 populate_metadata read hostname 'smithi072' 2024-03-22T01:37:45.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.913+0000 7fdcc3bc2740 1 --2- 172.21.15.72:0/336481354 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x56024cd3bcb0 0x56024cda4d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:45.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.913+0000 7fdcc3bc2740 4 client.25652 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T01:37:45.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.913+0000 7fdcc3bc2740 1 -- 172.21.15.72:0/336481354 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56024cc8e150 con 0x56024cd3bcb0 2024-03-22T01:37:45.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.913+0000 7fdcba59c640 1 --2- 172.21.15.72:0/336481354 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x56024cd3bcb0 0x56024cda4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:45.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.914+0000 7fdca1ffb640 10 client.25652 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:37:45.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.914+0000 7fdcba59c640 1 --2- 172.21.15.72:0/336481354 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x56024cd3bcb0 0x56024cda4d20 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c697e0c19fed58a1 server_cookie=8e89835c91faabfd in_seq=0 out_seq=0 2024-03-22T01:37:45.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.914+0000 7fdca1ffb640 1 -- 172.21.15.72:0/336481354 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56024cc8e150 con 0x56024cd3bcb0 2024-03-22T01:37:45.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.914+0000 7fdca1ffb640 10 client.25652 handle_command_reply: tid=0 2024-03-22T01:37:45.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.914+0000 7fdcc3bc2740 1 client.25652 shutdown 2024-03-22T01:37:45.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.914+0000 7fdcc3bc2740 1 -- 172.21.15.72:0/336481354 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56024cd2b910 msgr2=0x56024cda0f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:45.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.914+0000 7fdcc3bc2740 1 --2- 172.21.15.72:0/336481354 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56024cd2b910 0x56024cda0f80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fdcb4033bb0 tx=0x7fdcb4033e60 comp rx=0 tx=0).stop 2024-03-22T01:37:45.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.914+0000 7fdcc3bc2740 1 -- 172.21.15.72:0/336481354 shutdown_connections 2024-03-22T01:37:45.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.914+0000 7fdcc3bc2740 1 --2- 172.21.15.72:0/336481354 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x56024cd3bcb0 0x56024cda4d20 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:45.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.914+0000 7fdca1ffb640 0 client.25652 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:37:45.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.914+0000 7fdcc3bc2740 1 --2- 172.21.15.72:0/336481354 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56024cd2b910 0x56024cda0f80 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:45.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.914+0000 7fdcc3bc2740 1 --2- 172.21.15.72:0/336481354 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56024cd22d30 0x56024cd2ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:45.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.914+0000 7fdcc3bc2740 1 --2- 172.21.15.72:0/336481354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56024cd22410 0x56024cd2fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:45.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.914+0000 7fdcc3bc2740 1 -- 172.21.15.72:0/336481354 >> 172.21.15.72:0/336481354 conn(0x56024cd1cb40 msgr2=0x56024cd1df00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:45.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.914+0000 7fdcc3bc2740 1 -- 172.21.15.72:0/336481354 shutdown_connections 2024-03-22T01:37:45.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.914+0000 7fdcc3bc2740 1 -- 172.21.15.72:0/336481354 wait complete. 2024-03-22T01:37:45.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.914+0000 7fdcc3bc2740 20 client.25652 trim_cache size 0 max 16384 2024-03-22T01:37:45.922 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-22T01:37:45.922 INFO:teuthology.orchestra.run.smithi072.stdout: "status": "no active scrubs running", 2024-03-22T01:37:45.922 INFO:teuthology.orchestra.run.smithi072.stdout: "scrubs": {} 2024-03-22T01:37:45.922 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-22T01:37:45.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.915+0000 7fdcc0efb640 1 -- 172.21.15.72:0/3774623684 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fdca80a31b0 msgr2=0x7fdca80a5670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:45.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.915+0000 7fdcc0efb640 1 --2- 172.21.15.72:0/3774623684 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fdca80a31b0 0x7fdca80a5670 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fdcac005d00 tx=0x7fdcac009550 comp rx=0 tx=0).stop 2024-03-22T01:37:45.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.915+0000 7fdcc0efb640 1 -- 172.21.15.72:0/3774623684 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcbc1036d0 msgr2=0x7fdcbc198950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:45.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.915+0000 7fdcc0efb640 1 --2- 172.21.15.72:0/3774623684 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcbc1036d0 0x7fdcbc198950 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fdca4012dd0 tx=0x7fdca400c6a0 comp rx=0 tx=0).stop 2024-03-22T01:37:45.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.915+0000 7fdcc0efb640 1 -- 172.21.15.72:0/3774623684 shutdown_connections 2024-03-22T01:37:45.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.915+0000 7fdcc0efb640 1 --2- 172.21.15.72:0/3774623684 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fdca80a31b0 0x7fdca80a5670 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:45.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.915+0000 7fdcc0efb640 1 --2- 172.21.15.72:0/3774623684 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcbc1036d0 0x7fdcbc198950 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:45.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.915+0000 7fdcc0efb640 1 --2- 172.21.15.72:0/3774623684 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdcbc10c180 0x7fdcbc19cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:45.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.915+0000 7fdcc0efb640 1 --2- 172.21.15.72:0/3774623684 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcbc102db0 0x7fdcbc198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:45.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.915+0000 7fdcc0efb640 1 -- 172.21.15.72:0/3774623684 >> 172.21.15.72:0/3774623684 conn(0x7fdcbc077c40 msgr2=0x7fdcbc0fdaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:45.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.915+0000 7fdcc0efb640 1 -- 172.21.15.72:0/3774623684 shutdown_connections 2024-03-22T01:37:45.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:45.915+0000 7fdcc0efb640 1 -- 172.21.15.72:0/3774623684 wait complete. 2024-03-22T01:37:45.930 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-22T01:37:45.930 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T01:37:46.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.037+0000 7f816cf25640 1 -- 172.21.15.72:0/2703818678 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8168102db0 msgr2=0x7f8168103190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:46.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.037+0000 7f816cf25640 1 --2- 172.21.15.72:0/2703818678 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8168102db0 0x7f8168103190 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8158009860 tx=0x7f815802bc00 comp rx=0 tx=0).stop 2024-03-22T01:37:46.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.037+0000 7f816cf25640 1 -- 172.21.15.72:0/2703818678 shutdown_connections 2024-03-22T01:37:46.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.037+0000 7f816cf25640 1 --2- 172.21.15.72:0/2703818678 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f816810c180 0x7f816810e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:46.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.037+0000 7f816cf25640 1 --2- 172.21.15.72:0/2703818678 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81681036d0 0x7f816810bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:46.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.037+0000 7f816cf25640 1 --2- 172.21.15.72:0/2703818678 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8168102db0 0x7f8168103190 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:46.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.037+0000 7f816cf25640 1 -- 172.21.15.72:0/2703818678 >> 172.21.15.72:0/2703818678 conn(0x7f8168077c10 msgr2=0x7f8168078020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:46.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.037+0000 7f816cf25640 1 -- 172.21.15.72:0/2703818678 shutdown_connections 2024-03-22T01:37:46.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.037+0000 7f816cf25640 1 -- 172.21.15.72:0/2703818678 wait complete. 2024-03-22T01:37:46.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.038+0000 7f816cf25640 1 Processor -- start 2024-03-22T01:37:46.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.038+0000 7f816cf25640 1 -- start start 2024-03-22T01:37:46.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.038+0000 7f816cf25640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8168102db0 0x7f81681983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:46.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.038+0000 7f816cf25640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81681036d0 0x7f81681988e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:46.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.038+0000 7f816cf25640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f816810c180 0x7f816819cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:46.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.038+0000 7f816cf25640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f8168111ee0 con 0x7f8168102db0 2024-03-22T01:37:46.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.038+0000 7f816cf25640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8168111d60 con 0x7f816810c180 2024-03-22T01:37:46.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.038+0000 7f816cf25640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8168112060 con 0x7f81681036d0 2024-03-22T01:37:46.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.038+0000 7f8166d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f816810c180 0x7f816819cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:46.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.038+0000 7f816659c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8168102db0 0x7f81681983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:46.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.038+0000 7f8166d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f816810c180 0x7f816819cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.72:47850/0 (socket says 172.21.15.72:47850) 2024-03-22T01:37:46.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.038+0000 7f8166d9d640 1 -- 172.21.15.72:0/2888108026 learned_addr learned my addr 172.21.15.72:0/2888108026 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:37:46.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.038+0000 7f816659c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8168102db0 0x7f81681983a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37970/0 (socket says 172.21.15.72:37970) 2024-03-22T01:37:46.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.038+0000 7f8165d9b640 1 --2- 172.21.15.72:0/2888108026 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81681036d0 0x7f81681988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:46.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.039+0000 7f8166d9d640 1 -- 172.21.15.72:0/2888108026 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81681036d0 msgr2=0x7f81681988e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:46.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.039+0000 7f8166d9d640 1 --2- 172.21.15.72:0/2888108026 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81681036d0 0x7f81681988e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:46.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.039+0000 7f8166d9d640 1 -- 172.21.15.72:0/2888108026 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8168102db0 msgr2=0x7f81681983a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:46.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.039+0000 7f8166d9d640 1 --2- 172.21.15.72:0/2888108026 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8168102db0 0x7f81681983a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:46.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.039+0000 7f8166d9d640 1 -- 172.21.15.72:0/2888108026 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f816819d340 con 0x7f816810c180 2024-03-22T01:37:46.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.039+0000 7f8166d9d640 1 --2- 172.21.15.72:0/2888108026 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f816810c180 0x7f816819cc10 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f816000b680 tx=0x7f816000bb40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:46.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.039+0000 7f81577fe640 1 -- 172.21.15.72:0/2888108026 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8160004b20 con 0x7f816810c180 2024-03-22T01:37:46.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.039+0000 7f816cf25640 1 -- 172.21.15.72:0/2888108026 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f816819d630 con 0x7f816810c180 2024-03-22T01:37:46.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.039+0000 7f81577fe640 1 -- 172.21.15.72:0/2888108026 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f816000ad30 con 0x7f816810c180 2024-03-22T01:37:46.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.039+0000 7f81577fe640 1 -- 172.21.15.72:0/2888108026 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81600058b0 con 0x7f816810c180 2024-03-22T01:37:46.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.039+0000 7f816cf25640 1 -- 172.21.15.72:0/2888108026 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8168100930 con 0x7f816810c180 2024-03-22T01:37:46.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.040+0000 7f816cf25640 1 -- 172.21.15.72:0/2888108026 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8168068a10 con 0x7f816810c180 2024-03-22T01:37:46.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.041+0000 7f81577fe640 1 -- 172.21.15.72:0/2888108026 <== mon.2 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 24) ==== 138087+0+0 (secure 0 0 0) 0x7f8160020020 con 0x7f816810c180 2024-03-22T01:37:46.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.042+0000 7f81577fe640 1 --2- 172.21.15.72:0/2888108026 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f81500a3230 0x7f81500a56f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:46.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.042+0000 7f816659c640 1 --2- 172.21.15.72:0/2888108026 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f81500a3230 0x7f81500a56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:46.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.042+0000 7f81577fe640 1 -- 172.21.15.72:0/2888108026 <== mon.2 v2:172.21.15.89:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f81600d3e30 con 0x7f816810c180 2024-03-22T01:37:46.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.043+0000 7f816659c640 1 --2- 172.21.15.72:0/2888108026 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f81500a3230 0x7f81500a56f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f815802c210 tx=0x7f81580058d0 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:46.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.047+0000 7f81577fe640 1 -- 172.21.15.72:0/2888108026 <== mon.2 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f81600a0320 con 0x7f816810c180 2024-03-22T01:37:46.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:37:45 smithi072 ceph-mon[32414]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-22T01:37:46.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.271+0000 7f816cf25640 1 -- 172.21.15.72:0/2888108026 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8168100c50 con 0x7f816810c180 2024-03-22T01:37:46.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.271+0000 7f81577fe640 1 -- 172.21.15.72:0/2888108026 <== mon.2 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11785 (secure 0 0 0) 0x7f816009fc20 con 0x7f816810c180 2024-03-22T01:37:46.271 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:37:46.271 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6835/3077216308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3077216308},{"type":"v1","addr":"172.21.15.72:6835","nonce":3077216308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2460254143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":2460254143},{"type":"v1","addr":"172.21.15.72:6837","nonce":2460254143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6839/2026432819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6838","nonce":2026432819},{"type":"v1","addr":"172.21.15.72:6839","nonce":2026432819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6841/400094545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6840","nonce":400094545},{"type":"v1","addr":"172.21.15.72:6841","nonce":400094545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6833/1578701171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":1578701171},{"type":"v1","addr":"172.21.15.89:6833","nonce":1578701171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/2703047213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2703047213},{"type":"v1","addr":"172.21.15.98:6833","nonce":2703047213}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2589468787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2589468787},{"type":"v1","addr":"172.21.15.89:6835","nonce":2589468787}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/92657636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":92657636},{"type":"v1","addr":"172.21.15.98:6835","nonce":92657636}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1662941782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1662941782},{"type":"v1","addr":"172.21.15.89:6837","nonce":1662941782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24490,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/781860371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":781860371},{"type":"v1","addr":"172.21.15.98:6837","nonce":781860371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-22T01:34:14.815677+0000","modified":"2024-03-22T01:34:39.842959+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24524},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3192173650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3192173650},{"type":"v1","addr":"172.21.15.98:6839","nonce":3192173650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6839/149200280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":149200280},{"type":"v1","addr":"172.21.15.89:6839","nonce":149200280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24524]},"id":1}]} 2024-03-22T01:37:46.271 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 21 2024-03-22T01:37:46.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.274+0000 7f816cf25640 1 -- 172.21.15.72:0/2888108026 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f81500a3230 msgr2=0x7f81500a56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:46.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.274+0000 7f816cf25640 1 --2- 172.21.15.72:0/2888108026 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f81500a3230 0x7f81500a56f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f815802c210 tx=0x7f81580058d0 comp rx=0 tx=0).stop 2024-03-22T01:37:46.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.274+0000 7f816cf25640 1 -- 172.21.15.72:0/2888108026 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f816810c180 msgr2=0x7f816819cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:46.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.274+0000 7f816cf25640 1 --2- 172.21.15.72:0/2888108026 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f816810c180 0x7f816819cc10 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f816000b680 tx=0x7f816000bb40 comp rx=0 tx=0).stop 2024-03-22T01:37:46.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.274+0000 7f816cf25640 1 -- 172.21.15.72:0/2888108026 shutdown_connections 2024-03-22T01:37:46.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.274+0000 7f816cf25640 1 --2- 172.21.15.72:0/2888108026 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f81500a3230 0x7f81500a56f0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:46.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.274+0000 7f816cf25640 1 --2- 172.21.15.72:0/2888108026 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f816810c180 0x7f816819cc10 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:46.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.274+0000 7f816cf25640 1 --2- 172.21.15.72:0/2888108026 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81681036d0 0x7f81681988e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:46.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.274+0000 7f816cf25640 1 --2- 172.21.15.72:0/2888108026 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8168102db0 0x7f81681983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:46.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.274+0000 7f816cf25640 1 -- 172.21.15.72:0/2888108026 >> 172.21.15.72:0/2888108026 conn(0x7f8168077c10 msgr2=0x7f81680fdaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:46.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.274+0000 7f816cf25640 1 -- 172.21.15.72:0/2888108026 shutdown_connections 2024-03-22T01:37:46.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.274+0000 7f816cf25640 1 -- 172.21.15.72:0/2888108026 wait complete. 2024-03-22T01:37:46.284 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-22T01:37:46.357 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:37:45 smithi089 ceph-mon[35770]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-22T01:37:46.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.390+0000 7fc23bdd7640 1 -- 172.21.15.72:0/3150362090 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2341036d0 msgr2=0x7fc23410bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:46.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.390+0000 7fc23bdd7640 1 --2- 172.21.15.72:0/3150362090 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2341036d0 0x7fc23410bbb0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fc224009860 tx=0x7fc22402bc00 comp rx=0 tx=0).stop 2024-03-22T01:37:46.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.391+0000 7fc23bdd7640 1 -- 172.21.15.72:0/3150362090 shutdown_connections 2024-03-22T01:37:46.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.391+0000 7fc23bdd7640 1 --2- 172.21.15.72:0/3150362090 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2341036d0 0x7fc23410bbb0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:46.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.391+0000 7fc23bdd7640 1 --2- 172.21.15.72:0/3150362090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc23410c180 0x7fc23410e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:46.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.391+0000 7fc23bdd7640 1 --2- 172.21.15.72:0/3150362090 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc234102db0 0x7fc234103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:46.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.391+0000 7fc23bdd7640 1 -- 172.21.15.72:0/3150362090 >> 172.21.15.72:0/3150362090 conn(0x7fc234077c40 msgr2=0x7fc234078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:46.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.391+0000 7fc23bdd7640 1 -- 172.21.15.72:0/3150362090 shutdown_connections 2024-03-22T01:37:46.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.391+0000 7fc23bdd7640 1 -- 172.21.15.72:0/3150362090 wait complete. 2024-03-22T01:37:46.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.391+0000 7fc23bdd7640 1 Processor -- start 2024-03-22T01:37:46.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.391+0000 7fc23bdd7640 1 -- start start 2024-03-22T01:37:46.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.391+0000 7fc23bdd7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc234102db0 0x7fc23419a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:46.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.391+0000 7fc23bdd7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2341036d0 0x7fc23419ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:46.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.392+0000 7fc23bdd7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc23410c180 0x7fc23419ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:46.393 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.392+0000 7fc23bdd7640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fc234111f10 con 0x7fc234102db0 2024-03-22T01:37:46.393 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.392+0000 7fc239b73640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc234102db0 0x7fc23419a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:46.393 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.392+0000 7fc239b73640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc234102db0 0x7fc23419a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38000/0 (socket says 172.21.15.72:38000) 2024-03-22T01:37:46.393 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.392+0000 7fc239b73640 1 -- 172.21.15.72:0/1051010808 learned_addr learned my addr 172.21.15.72:0/1051010808 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:37:46.393 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.392+0000 7fc23bdd7640 1 -- 172.21.15.72:0/1051010808 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc234111d90 con 0x7fc2341036d0 2024-03-22T01:37:46.393 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.392+0000 7fc23bdd7640 1 -- 172.21.15.72:0/1051010808 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc234112090 con 0x7fc23410c180 2024-03-22T01:37:46.393 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.392+0000 7fc239372640 1 --2- 172.21.15.72:0/1051010808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2341036d0 0x7fc23419ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:46.393 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.392+0000 7fc23a374640 1 --2- 172.21.15.72:0/1051010808 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc23410c180 0x7fc23419ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:46.393 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.392+0000 7fc239372640 1 -- 172.21.15.72:0/1051010808 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc23410c180 msgr2=0x7fc23419ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:46.393 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.392+0000 7fc239372640 1 --2- 172.21.15.72:0/1051010808 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc23410c180 0x7fc23419ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:46.393 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.392+0000 7fc239372640 1 -- 172.21.15.72:0/1051010808 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc234102db0 msgr2=0x7fc23419a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:46.393 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.392+0000 7fc239372640 1 --2- 172.21.15.72:0/1051010808 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc234102db0 0x7fc23419a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:46.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.392+0000 7fc239372640 1 -- 172.21.15.72:0/1051010808 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc23419f630 con 0x7fc2341036d0 2024-03-22T01:37:46.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.392+0000 7fc23a374640 1 --2- 172.21.15.72:0/1051010808 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc23410c180 0x7fc23419ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:37:46.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.392+0000 7fc239372640 1 --2- 172.21.15.72:0/1051010808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2341036d0 0x7fc23419ab80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fc224037040 tx=0x7fc2240346f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:46.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.392+0000 7fc22affd640 1 -- 172.21.15.72:0/1051010808 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc224041020 con 0x7fc2341036d0 2024-03-22T01:37:46.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.392+0000 7fc22affd640 1 -- 172.21.15.72:0/1051010808 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2240044d0 con 0x7fc2341036d0 2024-03-22T01:37:46.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.392+0000 7fc22affd640 1 -- 172.21.15.72:0/1051010808 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc224039040 con 0x7fc2341036d0 2024-03-22T01:37:46.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.392+0000 7fc23bdd7640 1 -- 172.21.15.72:0/1051010808 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc23419f8c0 con 0x7fc2341036d0 2024-03-22T01:37:46.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.393+0000 7fc23bdd7640 1 -- 172.21.15.72:0/1051010808 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2341a71b0 con 0x7fc2341036d0 2024-03-22T01:37:46.395 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.395+0000 7fc23e9c2740 1 Processor -- start 2024-03-22T01:37:46.395 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.395+0000 7fc23e9c2740 1 -- start start 2024-03-22T01:37:46.395 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.396+0000 7fc23e9c2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560fa98b7800 0x560fa98b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:46.395 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.396+0000 7fc23e9c2740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560fa98b8120 0x560fa98c07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:46.395 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.396+0000 7fc23e9c2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560fa98c0d00 0x560fa98c30f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:46.395 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.396+0000 7fc23e9c2740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x560fa981b8d0 con 0x560fa98b8120 2024-03-22T01:37:46.395 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.396+0000 7fc22affd640 1 -- 172.21.15.72:0/1051010808 <== mon.2 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 24) ==== 138087+0+0 (secure 0 0 0) 0x7fc224044020 con 0x7fc2341036d0 2024-03-22T01:37:46.395 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.396+0000 7fc23e9c2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x560fa97c3e00 con 0x560fa98c0d00 2024-03-22T01:37:46.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.396+0000 7fc23e9c2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560fa9817880 con 0x560fa98b7800 2024-03-22T01:37:46.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.396+0000 7fc239b73640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560fa98b8120 0x560fa98c07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:46.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.396+0000 7fc239b73640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560fa98b8120 0x560fa98c07c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38010/0 (socket says 172.21.15.72:38010) 2024-03-22T01:37:46.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.396+0000 7fc239b73640 1 -- 172.21.15.72:0/1198065846 learned_addr learned my addr 172.21.15.72:0/1198065846 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:37:46.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.396+0000 7fc239372640 1 --2- 172.21.15.72:0/1198065846 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560fa98c0d00 0x560fa98c30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:46.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.396+0000 7fc23a374640 1 --2- 172.21.15.72:0/1198065846 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560fa98b7800 0x560fa98b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:46.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.396+0000 7fc239b73640 1 -- 172.21.15.72:0/1198065846 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560fa98b7800 msgr2=0x560fa98b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:46.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.396+0000 7fc239b73640 1 --2- 172.21.15.72:0/1198065846 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560fa98b7800 0x560fa98b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:46.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.396+0000 7fc239b73640 1 -- 172.21.15.72:0/1198065846 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560fa98c0d00 msgr2=0x560fa98c30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:46.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.396+0000 7fc239b73640 1 --2- 172.21.15.72:0/1198065846 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560fa98c0d00 0x560fa98c30f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:46.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.396+0000 7fc239b73640 1 -- 172.21.15.72:0/1198065846 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560fa952f3f0 con 0x560fa98b8120 2024-03-22T01:37:46.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.396+0000 7fc239b73640 1 --2- 172.21.15.72:0/1198065846 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560fa98b8120 0x560fa98c07c0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fc230017fb0 tx=0x7fc230009db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b786bfef01408b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:46.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.397+0000 7fc21ffff640 1 -- 172.21.15.72:0/1198065846 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc230004030 con 0x560fa98b8120 2024-03-22T01:37:46.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.397+0000 7fc21ffff640 1 -- 172.21.15.72:0/1198065846 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2300041d0 con 0x560fa98b8120 2024-03-22T01:37:46.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.397+0000 7fc21ffff640 1 -- 172.21.15.72:0/1198065846 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc230003d20 con 0x560fa98b8120 2024-03-22T01:37:46.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.397+0000 7fc22affd640 1 --2- 172.21.15.72:0/1051010808 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fc2200a31b0 0x7fc2200a5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:46.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.397+0000 7fc22affd640 1 -- 172.21.15.72:0/1051010808 <== mon.2 v2:172.21.15.89:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fc2240f14a0 con 0x7fc2341036d0 2024-03-22T01:37:46.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.397+0000 7fc23a374640 1 --2- 172.21.15.72:0/1051010808 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fc2200a31b0 0x7fc2200a5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:46.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.397+0000 7fc23e9c2740 1 -- 172.21.15.72:0/1198065846 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560fa98b8120 msgr2=0x560fa98c07c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:46.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.397+0000 7fc23e9c2740 1 --2- 172.21.15.72:0/1198065846 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560fa98b8120 0x560fa98c07c0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fc230017fb0 tx=0x7fc230009db0 comp rx=0 tx=0).stop 2024-03-22T01:37:46.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.397+0000 7fc23e9c2740 1 -- 172.21.15.72:0/1198065846 shutdown_connections 2024-03-22T01:37:46.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.397+0000 7fc23e9c2740 1 --2- 172.21.15.72:0/1198065846 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560fa98c0d00 0x560fa98c30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:46.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.397+0000 7fc23e9c2740 1 --2- 172.21.15.72:0/1198065846 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560fa98b8120 0x560fa98c07c0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:46.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.397+0000 7fc23e9c2740 1 --2- 172.21.15.72:0/1198065846 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560fa98b7800 0x560fa98b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:46.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.397+0000 7fc23e9c2740 1 -- 172.21.15.72:0/1198065846 >> 172.21.15.72:0/1198065846 conn(0x560fa98b1f30 msgr2=0x560fa98b2310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:46.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.398+0000 7fc23a374640 1 --2- 172.21.15.72:0/1051010808 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fc2200a31b0 0x7fc2200a5670 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc22c005fd0 tx=0x7fc22c009970 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:46.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.398+0000 7fc23e9c2740 1 -- 172.21.15.72:0/1198065846 shutdown_connections 2024-03-22T01:37:46.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.398+0000 7fc23e9c2740 1 -- 172.21.15.72:0/1198065846 wait complete. 2024-03-22T01:37:46.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.398+0000 7fc23e9c2740 1 -- start start 2024-03-22T01:37:46.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.398+0000 7fc23e9c2740 1 Processor -- start 2024-03-22T01:37:46.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.399+0000 7fc23e9c2740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560fa98b7800 0x560fa98bdd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:46.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.399+0000 7fc23e9c2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560fa98c0d00 0x560fa98be2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:46.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.399+0000 7fc23e9c2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560fa98be800 0x560fa98ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:46.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.399+0000 7fc23e9c2740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x560fa98c3af0 con 0x560fa98b7800 2024-03-22T01:37:46.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.399+0000 7fc23e9c2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x560fa97d7300 con 0x560fa98be800 2024-03-22T01:37:46.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.399+0000 7fc23e9c2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560fa98c3c70 con 0x560fa98c0d00 2024-03-22T01:37:46.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.399+0000 7fc239b73640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560fa98b7800 0x560fa98bdd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:46.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.399+0000 7fc239b73640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560fa98b7800 0x560fa98bdd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38014/0 (socket says 172.21.15.72:38014) 2024-03-22T01:37:46.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.399+0000 7fc23e9c2740 10 client.0 _ll_register_callbacks cb 0x560fa9812e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:37:46.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.399+0000 7fc239b73640 1 -- 172.21.15.72:0/2766942069 learned_addr learned my addr 172.21.15.72:0/2766942069 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:37:46.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.399+0000 7fc239372640 1 --2- 172.21.15.72:0/2766942069 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560fa98c0d00 0x560fa98be2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:46.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.399+0000 7fc23a374640 1 --2- 172.21.15.72:0/2766942069 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560fa98be800 0x560fa98ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:46.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.399+0000 7fc239b73640 1 -- 172.21.15.72:0/2766942069 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560fa98c0d00 msgr2=0x560fa98be2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:46.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.399+0000 7fc239b73640 1 --2- 172.21.15.72:0/2766942069 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560fa98c0d00 0x560fa98be2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:46.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.399+0000 7fc239b73640 1 -- 172.21.15.72:0/2766942069 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560fa98be800 msgr2=0x560fa98ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:46.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.399+0000 7fc239b73640 1 --2- 172.21.15.72:0/2766942069 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560fa98be800 0x560fa98ccf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:46.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.399+0000 7fc239b73640 1 -- 172.21.15.72:0/2766942069 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560fa95f97d0 con 0x560fa98b7800 2024-03-22T01:37:46.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.399+0000 7fc23a374640 1 --2- 172.21.15.72:0/2766942069 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560fa98be800 0x560fa98ccf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:37:46.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.399+0000 7fc239b73640 1 --2- 172.21.15.72:0/2766942069 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560fa98b7800 0x560fa98bdd80 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fc230009880 tx=0x7fc2300056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=836825e0477ee515 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:46.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.399+0000 7fc21d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:37:46.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.400+0000 7fc21d7fa640 1 -- 172.21.15.72:0/2766942069 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc23002e020 con 0x560fa98b7800 2024-03-22T01:37:46.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.400+0000 7fc23e9c2740 1 -- 172.21.15.72:0/2766942069 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc2300041d0 con 0x560fa98b7800 2024-03-22T01:37:46.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.400+0000 7fc21d7fa640 1 -- 172.21.15.72:0/2766942069 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fc230003c30 con 0x560fa98b7800 2024-03-22T01:37:46.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.400+0000 7fc21d7fa640 1 -- 172.21.15.72:0/2766942069 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fc230039050 con 0x560fa98b7800 2024-03-22T01:37:46.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.400+0000 7fc23e9c2740 10 client.15822 fetch_fsmap learned FSMap version 21 2024-03-22T01:37:46.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.400+0000 7fc23e9c2740 1 -- 172.21.15.72:0/2766942069 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x560fa98d02a0 con 0x560fa98b7800 2024-03-22T01:37:46.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.400+0000 7fc21d7fa640 1 -- 172.21.15.72:0/2766942069 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2300041d0 con 0x560fa98b7800 2024-03-22T01:37:46.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.400+0000 7fc21d7fa640 1 -- 172.21.15.72:0/2766942069 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fc23002ab00 con 0x560fa98b7800 2024-03-22T01:37:46.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.400+0000 7fc23e9c2740 10 client.15822 fetch_fsmap finished waiting for FSMap version 21 2024-03-22T01:37:46.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.400+0000 7fc23e9c2740 10 client.15822 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T01:37:46.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.400+0000 7fc23e9c2740 20 client.15822 populate_metadata read hostname 'smithi072' 2024-03-22T01:37:46.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.400+0000 7fc23e9c2740 1 --2- 172.21.15.72:0/2766942069 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x560fa98d1650 0x560fa98d3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:46.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.400+0000 7fc23e9c2740 4 client.15822 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T01:37:46.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.400+0000 7fc23e9c2740 1 -- 172.21.15.72:0/2766942069 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560fa9828e10 con 0x560fa98d1650 2024-03-22T01:37:46.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.401+0000 7fc239372640 1 --2- 172.21.15.72:0/2766942069 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x560fa98d1650 0x560fa98d3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:46.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.401+0000 7fc239372640 1 --2- 172.21.15.72:0/2766942069 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x560fa98d1650 0x560fa98d3a40 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=453a62f963875495 server_cookie=e27abb91c5796701 in_seq=0 out_seq=0 2024-03-22T01:37:46.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.401+0000 7fc21d7fa640 10 client.15822 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:37:46.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.402+0000 7fc21d7fa640 1 -- 172.21.15.72:0/2766942069 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560fa9828e10 con 0x560fa98d1650 2024-03-22T01:37:46.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.402+0000 7fc21d7fa640 10 client.15822 handle_command_reply: tid=0 2024-03-22T01:37:46.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.402+0000 7fc23e9c2740 1 client.15822 shutdown 2024-03-22T01:37:46.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.402+0000 7fc23e9c2740 1 -- 172.21.15.72:0/2766942069 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560fa98b7800 msgr2=0x560fa98bdd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:46.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.402+0000 7fc23e9c2740 1 --2- 172.21.15.72:0/2766942069 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560fa98b7800 0x560fa98bdd80 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fc230009880 tx=0x7fc2300056d0 comp rx=0 tx=0).stop 2024-03-22T01:37:46.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.402+0000 7fc23e9c2740 1 -- 172.21.15.72:0/2766942069 shutdown_connections 2024-03-22T01:37:46.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.402+0000 7fc23e9c2740 1 --2- 172.21.15.72:0/2766942069 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x560fa98d1650 0x560fa98d3a40 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:46.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.402+0000 7fc21d7fa640 0 client.15822 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:37:46.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.402+0000 7fc23e9c2740 1 --2- 172.21.15.72:0/2766942069 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560fa98be800 0x560fa98ccf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:46.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.402+0000 7fc23e9c2740 1 --2- 172.21.15.72:0/2766942069 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560fa98c0d00 0x560fa98be2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:46.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.402+0000 7fc23e9c2740 1 --2- 172.21.15.72:0/2766942069 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560fa98b7800 0x560fa98bdd80 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:46.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.402+0000 7fc23e9c2740 1 -- 172.21.15.72:0/2766942069 >> 172.21.15.72:0/2766942069 conn(0x560fa98b1f30 msgr2=0x560fa98b3c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:46.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.402+0000 7fc23e9c2740 1 -- 172.21.15.72:0/2766942069 shutdown_connections 2024-03-22T01:37:46.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.402+0000 7fc23e9c2740 1 -- 172.21.15.72:0/2766942069 wait complete. 2024-03-22T01:37:46.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.402+0000 7fc23e9c2740 20 client.15822 trim_cache size 0 max 16384 2024-03-22T01:37:46.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.416+0000 7fc23e9c2740 1 Processor -- start 2024-03-22T01:37:46.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.416+0000 7fc23e9c2740 1 -- start start 2024-03-22T01:37:46.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.416+0000 7fc23e9c2740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560fa98b7800 0x560fa98d9100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:46.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.416+0000 7fc23e9c2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560fa98be800 0x560fa98cc890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:46.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.416+0000 7fc23e9c2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560fa98c0d00 0x560fa98ccdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:46.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.416+0000 7fc23e9c2740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fc2341aaea0 con 0x560fa98b7800 2024-03-22T01:37:46.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.416+0000 7fc239372640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560fa98be800 0x560fa98cc890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:46.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.416+0000 7fc239b73640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560fa98b7800 0x560fa98d9100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:46.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.416+0000 7fc239372640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560fa98be800 0x560fa98cc890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.72:47892/0 (socket says 172.21.15.72:47892) 2024-03-22T01:37:46.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.416+0000 7fc239372640 1 -- 172.21.15.72:0/3398049283 learned_addr learned my addr 172.21.15.72:0/3398049283 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:37:46.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.416+0000 7fc23e9c2740 1 -- 172.21.15.72:0/3398049283 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc2341ad700 con 0x560fa98be800 2024-03-22T01:37:46.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.416+0000 7fc23e9c2740 1 -- 172.21.15.72:0/3398049283 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc2341ab4f0 con 0x560fa98c0d00 2024-03-22T01:37:46.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.416+0000 7fc23a374640 1 --2- 172.21.15.72:0/3398049283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560fa98c0d00 0x560fa98ccdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:46.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.416+0000 7fc239372640 1 -- 172.21.15.72:0/3398049283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560fa98c0d00 msgr2=0x560fa98ccdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:46.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.416+0000 7fc239372640 1 --2- 172.21.15.72:0/3398049283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560fa98c0d00 0x560fa98ccdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:46.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.416+0000 7fc239372640 1 -- 172.21.15.72:0/3398049283 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560fa98b7800 msgr2=0x560fa98d9100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:46.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.416+0000 7fc239372640 1 --2- 172.21.15.72:0/3398049283 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560fa98b7800 0x560fa98d9100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:46.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.416+0000 7fc239372640 1 -- 172.21.15.72:0/3398049283 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560fa98be620 con 0x560fa98be800 2024-03-22T01:37:46.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.416+0000 7fc23a374640 1 --2- 172.21.15.72:0/3398049283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560fa98c0d00 0x560fa98ccdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T01:37:46.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.417+0000 7fc239372640 1 --2- 172.21.15.72:0/3398049283 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560fa98be800 0x560fa98cc890 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fc2240b9e10 tx=0x7fc2240f3040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2f18cc834e0c4bc server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:46.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.417+0000 7fc21f7fe640 1 -- 172.21.15.72:0/3398049283 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2240bcf80 con 0x560fa98be800 2024-03-22T01:37:46.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.417+0000 7fc21f7fe640 1 -- 172.21.15.72:0/3398049283 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2240bdb40 con 0x560fa98be800 2024-03-22T01:37:46.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.417+0000 7fc21f7fe640 1 -- 172.21.15.72:0/3398049283 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2240bcda0 con 0x560fa98be800 2024-03-22T01:37:46.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.418+0000 7fc23e9c2740 1 -- 172.21.15.72:0/3398049283 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560fa98be800 msgr2=0x560fa98cc890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:46.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.418+0000 7fc23e9c2740 1 --2- 172.21.15.72:0/3398049283 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560fa98be800 0x560fa98cc890 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fc2240b9e10 tx=0x7fc2240f3040 comp rx=0 tx=0).stop 2024-03-22T01:37:46.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.418+0000 7fc23e9c2740 1 -- 172.21.15.72:0/3398049283 shutdown_connections 2024-03-22T01:37:46.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.418+0000 7fc23e9c2740 1 --2- 172.21.15.72:0/3398049283 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560fa98be800 0x560fa98cc890 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:46.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.418+0000 7fc23e9c2740 1 --2- 172.21.15.72:0/3398049283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560fa98c0d00 0x560fa98ccdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:46.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.418+0000 7fc23e9c2740 1 --2- 172.21.15.72:0/3398049283 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560fa98b7800 0x560fa98d9100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:46.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.418+0000 7fc23e9c2740 1 -- 172.21.15.72:0/3398049283 >> 172.21.15.72:0/3398049283 conn(0x560fa98b1f30 msgr2=0x560fa98b0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:46.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.418+0000 7fc23e9c2740 1 -- 172.21.15.72:0/3398049283 shutdown_connections 2024-03-22T01:37:46.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.418+0000 7fc23e9c2740 1 -- 172.21.15.72:0/3398049283 wait complete. 2024-03-22T01:37:46.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.418+0000 7fc23e9c2740 1 -- start start 2024-03-22T01:37:46.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.418+0000 7fc23e9c2740 1 Processor -- start 2024-03-22T01:37:46.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.419+0000 7fc23e9c2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560fa98b7800 0x560fa98c90b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:46.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.419+0000 7fc23e9c2740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560fa98be800 0x560fa98ba610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:46.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.419+0000 7fc23e9c2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560fa98c0d00 0x560fa98bab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:46.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.419+0000 7fc23e9c2740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fc2341abbc0 con 0x560fa98be800 2024-03-22T01:37:46.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.419+0000 7fc23e9c2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc2341ad030 con 0x560fa98c0d00 2024-03-22T01:37:46.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.419+0000 7fc23e9c2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc2341ac290 con 0x560fa98b7800 2024-03-22T01:37:46.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.419+0000 7fc23e9c2740 10 client.0 _ll_register_callbacks cb 0x7fc230017eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:37:46.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.419+0000 7fc23a374640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560fa98c0d00 0x560fa98bab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:46.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.419+0000 7fc239372640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560fa98be800 0x560fa98ba610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:46.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.419+0000 7fc239b73640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560fa98b7800 0x560fa98c90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:46.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.419+0000 7fc239372640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560fa98be800 0x560fa98ba610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38024/0 (socket says 172.21.15.72:38024) 2024-03-22T01:37:46.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.419+0000 7fc239372640 1 -- 172.21.15.72:0/485613684 learned_addr learned my addr 172.21.15.72:0/485613684 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:37:46.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.419+0000 7fc239372640 1 -- 172.21.15.72:0/485613684 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560fa98b7800 msgr2=0x560fa98c90b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:46.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.419+0000 7fc239372640 1 --2- 172.21.15.72:0/485613684 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560fa98b7800 0x560fa98c90b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:46.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.419+0000 7fc239372640 1 -- 172.21.15.72:0/485613684 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560fa98c0d00 msgr2=0x560fa98bab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:46.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.419+0000 7fc239372640 1 --2- 172.21.15.72:0/485613684 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560fa98c0d00 0x560fa98bab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:46.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.419+0000 7fc239372640 1 -- 172.21.15.72:0/485613684 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560fa98bb350 con 0x560fa98be800 2024-03-22T01:37:46.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.419+0000 7fc239b73640 1 --2- 172.21.15.72:0/485613684 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560fa98b7800 0x560fa98c90b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:37:46.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.419+0000 7fc239372640 1 --2- 172.21.15.72:0/485613684 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560fa98be800 0x560fa98ba610 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fc22402c140 tx=0x7fc22402bc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f71efd8a4ab6f928 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:46.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.419+0000 7fc21d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:37:46.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.419+0000 7fc21d7fa640 1 -- 172.21.15.72:0/485613684 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2240bcf80 con 0x560fa98be800 2024-03-22T01:37:46.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.419+0000 7fc23e9c2740 1 -- 172.21.15.72:0/485613684 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc2240bdb40 con 0x560fa98be800 2024-03-22T01:37:46.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.419+0000 7fc21d7fa640 1 -- 172.21.15.72:0/485613684 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fc224003680 con 0x560fa98be800 2024-03-22T01:37:46.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.420+0000 7fc21d7fa640 1 -- 172.21.15.72:0/485613684 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fc2240bd440 con 0x560fa98be800 2024-03-22T01:37:46.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.420+0000 7fc23e9c2740 10 client.15828 fetch_fsmap learned FSMap version 21 2024-03-22T01:37:46.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.420+0000 7fc23e9c2740 1 -- 172.21.15.72:0/485613684 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x560fa9937560 con 0x560fa98be800 2024-03-22T01:37:46.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.420+0000 7fc21d7fa640 1 -- 172.21.15.72:0/485613684 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2240bdb40 con 0x560fa98be800 2024-03-22T01:37:46.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.420+0000 7fc21d7fa640 1 -- 172.21.15.72:0/485613684 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fc22403c030 con 0x560fa98be800 2024-03-22T01:37:46.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.421+0000 7fc23e9c2740 10 client.15828 fetch_fsmap finished waiting for FSMap version 21 2024-03-22T01:37:46.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.421+0000 7fc23e9c2740 10 client.15828 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T01:37:46.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.421+0000 7fc23e9c2740 20 client.15828 populate_metadata read hostname 'smithi072' 2024-03-22T01:37:46.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.421+0000 7fc23e9c2740 1 --2- 172.21.15.72:0/485613684 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x560fa98d1650 0x560fa9939a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:46.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.421+0000 7fc23e9c2740 4 client.15828 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T01:37:46.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.421+0000 7fc23e9c2740 1 -- 172.21.15.72:0/485613684 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560fa9823160 con 0x560fa98d1650 2024-03-22T01:37:46.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.421+0000 7fc239b73640 1 --2- 172.21.15.72:0/485613684 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x560fa98d1650 0x560fa9939a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:46.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.421+0000 7fc239b73640 1 --2- 172.21.15.72:0/485613684 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x560fa98d1650 0x560fa9939a90 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dbc7a41cb0b3b950 server_cookie=942480fa008f99c4 in_seq=0 out_seq=0 2024-03-22T01:37:46.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.421+0000 7fc21d7fa640 10 client.15828 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:37:46.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.421+0000 7fc21d7fa640 1 -- 172.21.15.72:0/485613684 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560fa9823160 con 0x560fa98d1650 2024-03-22T01:37:46.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.421+0000 7fc21d7fa640 10 client.15828 handle_command_reply: tid=0 2024-03-22T01:37:46.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.421+0000 7fc23e9c2740 1 client.15828 shutdown 2024-03-22T01:37:46.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.421+0000 7fc23e9c2740 1 -- 172.21.15.72:0/485613684 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560fa98be800 msgr2=0x560fa98ba610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:46.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.421+0000 7fc23e9c2740 1 --2- 172.21.15.72:0/485613684 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560fa98be800 0x560fa98ba610 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fc22402c140 tx=0x7fc22402bc00 comp rx=0 tx=0).stop 2024-03-22T01:37:46.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.421+0000 7fc23e9c2740 1 -- 172.21.15.72:0/485613684 shutdown_connections 2024-03-22T01:37:46.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.421+0000 7fc23e9c2740 1 --2- 172.21.15.72:0/485613684 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x560fa98d1650 0x560fa9939a90 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:46.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.421+0000 7fc23e9c2740 1 --2- 172.21.15.72:0/485613684 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560fa98c0d00 0x560fa98bab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:46.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.421+0000 7fc23e9c2740 1 --2- 172.21.15.72:0/485613684 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560fa98be800 0x560fa98ba610 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:46.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.421+0000 7fc21d7fa640 0 client.15828 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:37:46.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.421+0000 7fc23e9c2740 1 --2- 172.21.15.72:0/485613684 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560fa98b7800 0x560fa98c90b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:46.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.421+0000 7fc23e9c2740 1 -- 172.21.15.72:0/485613684 >> 172.21.15.72:0/485613684 conn(0x560fa98b1f30 msgr2=0x560fa98b2f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:46.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.422+0000 7fc23e9c2740 1 -- 172.21.15.72:0/485613684 shutdown_connections 2024-03-22T01:37:46.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.422+0000 7fc23e9c2740 1 -- 172.21.15.72:0/485613684 wait complete. 2024-03-22T01:37:46.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.422+0000 7fc23e9c2740 20 client.15828 trim_cache size 0 max 16384 2024-03-22T01:37:46.426 INFO:teuthology.orchestra.run.smithi072.stdout:[] 2024-03-22T01:37:46.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.422+0000 7fc23bdd7640 1 -- 172.21.15.72:0/1051010808 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fc2200a31b0 msgr2=0x7fc2200a5670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:46.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.422+0000 7fc23bdd7640 1 --2- 172.21.15.72:0/1051010808 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fc2200a31b0 0x7fc2200a5670 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc22c005fd0 tx=0x7fc22c009970 comp rx=0 tx=0).stop 2024-03-22T01:37:46.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.422+0000 7fc23bdd7640 1 -- 172.21.15.72:0/1051010808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2341036d0 msgr2=0x7fc23419ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:46.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.422+0000 7fc23bdd7640 1 --2- 172.21.15.72:0/1051010808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2341036d0 0x7fc23419ab80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fc224037040 tx=0x7fc2240346f0 comp rx=0 tx=0).stop 2024-03-22T01:37:46.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.422+0000 7fc23bdd7640 1 -- 172.21.15.72:0/1051010808 shutdown_connections 2024-03-22T01:37:46.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.422+0000 7fc23bdd7640 1 --2- 172.21.15.72:0/1051010808 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fc2200a31b0 0x7fc2200a5670 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:46.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.422+0000 7fc23bdd7640 1 --2- 172.21.15.72:0/1051010808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2341036d0 0x7fc23419ab80 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:46.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.422+0000 7fc23bdd7640 1 --2- 172.21.15.72:0/1051010808 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc23410c180 0x7fc23419ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:46.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.422+0000 7fc23bdd7640 1 --2- 172.21.15.72:0/1051010808 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc234102db0 0x7fc23419a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:46.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.422+0000 7fc23bdd7640 1 -- 172.21.15.72:0/1051010808 >> 172.21.15.72:0/1051010808 conn(0x7fc234077c40 msgr2=0x7fc2340fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:46.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.422+0000 7fc23bdd7640 1 -- 172.21.15.72:0/1051010808 shutdown_connections 2024-03-22T01:37:46.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:46.423+0000 7fc23bdd7640 1 -- 172.21.15.72:0/1051010808 wait complete. 2024-03-22T01:37:46.428 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:37:45 smithi098 ceph-mon[36468]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-22T01:37:47.357 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:37:46 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/2888108026' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:37:47.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:37:46 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/2888108026' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:37:47.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:37:46 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/2888108026' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:37:47.433 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-22T01:37:47.433 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T01:37:47.433 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-22T01:37:47.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.589+0000 7f2ea58e1640 1 -- 172.21.15.72:0/1856487133 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ea0100930 msgr2=0x7f2ea0100d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:47.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.589+0000 7f2ea58e1640 1 --2- 172.21.15.72:0/1856487133 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ea0100930 0x7f2ea0100d10 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2e90009860 tx=0x7f2e9002bc00 comp rx=0 tx=0).stop 2024-03-22T01:37:47.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.589+0000 7f2ea58e1640 1 -- 172.21.15.72:0/1856487133 shutdown_connections 2024-03-22T01:37:47.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.589+0000 7f2ea58e1640 1 --2- 172.21.15.72:0/1856487133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ea010e4e0 0x7f2ea01108d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.589+0000 7f2ea58e1640 1 --2- 172.21.15.72:0/1856487133 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2ea0101250 0x7f2ea010dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.589+0000 7f2ea58e1640 1 --2- 172.21.15.72:0/1856487133 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ea0100930 0x7f2ea0100d10 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.589+0000 7f2ea58e1640 1 -- 172.21.15.72:0/1856487133 >> 172.21.15.72:0/1856487133 conn(0x7f2ea0077c40 msgr2=0x7f2ea0078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:47.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.589+0000 7f2ea58e1640 1 -- 172.21.15.72:0/1856487133 shutdown_connections 2024-03-22T01:37:47.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.589+0000 7f2ea58e1640 1 -- 172.21.15.72:0/1856487133 wait complete. 2024-03-22T01:37:47.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.590+0000 7f2ea58e1640 1 Processor -- start 2024-03-22T01:37:47.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.590+0000 7f2ea58e1640 1 -- start start 2024-03-22T01:37:47.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.590+0000 7f2ea58e1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ea0100930 0x7f2ea019a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:47.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.590+0000 7f2ea58e1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2ea0101250 0x7f2ea019ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:47.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.590+0000 7f2ea58e1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ea010e4e0 0x7f2ea019ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:47.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.590+0000 7f2ea58e1640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f2ea0114160 con 0x7f2ea0101250 2024-03-22T01:37:47.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.590+0000 7f2ea58e1640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2ea0113fe0 con 0x7f2ea0100930 2024-03-22T01:37:47.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.591+0000 7f2ea58e1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2ea01142e0 con 0x7f2ea010e4e0 2024-03-22T01:37:47.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.591+0000 7f2e9e7fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2ea0101250 0x7f2ea019ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:47.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.591+0000 7f2e9effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ea0100930 0x7f2ea019a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:47.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.591+0000 7f2e9e7fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2ea0101250 0x7f2ea019ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38042/0 (socket says 172.21.15.72:38042) 2024-03-22T01:37:47.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.591+0000 7f2e9e7fc640 1 -- 172.21.15.72:0/1605816487 learned_addr learned my addr 172.21.15.72:0/1605816487 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:37:47.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.591+0000 7f2e9effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ea0100930 0x7f2ea019a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.72:47918/0 (socket says 172.21.15.72:47918) 2024-03-22T01:37:47.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.591+0000 7f2e9f7fe640 1 --2- 172.21.15.72:0/1605816487 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ea010e4e0 0x7f2ea019ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:47.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.591+0000 7f2e9e7fc640 1 -- 172.21.15.72:0/1605816487 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ea010e4e0 msgr2=0x7f2ea019ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:47.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.591+0000 7f2e9e7fc640 1 --2- 172.21.15.72:0/1605816487 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ea010e4e0 0x7f2ea019ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.591+0000 7f2e9e7fc640 1 -- 172.21.15.72:0/1605816487 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ea0100930 msgr2=0x7f2ea019a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:47.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.591+0000 7f2e9e7fc640 1 --2- 172.21.15.72:0/1605816487 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ea0100930 0x7f2ea019a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.591+0000 7f2e9e7fc640 1 -- 172.21.15.72:0/1605816487 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ea019f680 con 0x7f2ea0101250 2024-03-22T01:37:47.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.591+0000 7f2e9effd640 1 --2- 172.21.15.72:0/1605816487 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ea0100930 0x7f2ea019a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:37:47.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.591+0000 7f2e9e7fc640 1 --2- 172.21.15.72:0/1605816487 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2ea0101250 0x7f2ea019ac20 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f2e9400e9c0 tx=0x7f2e9400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:47.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.591+0000 7f2ea48df640 1 -- 172.21.15.72:0/1605816487 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e9400cde0 con 0x7f2ea0101250 2024-03-22T01:37:47.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.591+0000 7f2ea48df640 1 -- 172.21.15.72:0/1605816487 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e940035f0 con 0x7f2ea0101250 2024-03-22T01:37:47.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.591+0000 7f2ea58e1640 1 -- 172.21.15.72:0/1605816487 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ea019f970 con 0x7f2ea0101250 2024-03-22T01:37:47.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.591+0000 7f2ea48df640 1 -- 172.21.15.72:0/1605816487 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e94010690 con 0x7f2ea0101250 2024-03-22T01:37:47.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.591+0000 7f2ea58e1640 1 -- 172.21.15.72:0/1605816487 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ea01a7200 con 0x7f2ea0101250 2024-03-22T01:37:47.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.594+0000 7f2ea48df640 1 -- 172.21.15.72:0/1605816487 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 24) ==== 138087+0+0 (secure 0 0 0) 0x7f2e940037a0 con 0x7f2ea0101250 2024-03-22T01:37:47.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.595+0000 7f2ea859c740 1 Processor -- start 2024-03-22T01:37:47.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.595+0000 7f2ea859c740 1 -- start start 2024-03-22T01:37:47.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.595+0000 7f2ea48df640 1 --2- 172.21.15.72:0/1605816487 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f2e880a3230 0x7f2e880a56f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:47.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.595+0000 7f2ea859c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559dad616ba0 0x559dad616f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:47.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.595+0000 7f2ea859c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559dad6174c0 0x559dad61fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:47.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.595+0000 7f2ea859c740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559dad6200a0 0x559dad622490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:47.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.595+0000 7f2ea859c740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x559dad57a570 con 0x559dad6200a0 2024-03-22T01:37:47.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.595+0000 7f2ea859c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x559dad522e10 con 0x559dad6174c0 2024-03-22T01:37:47.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.595+0000 7f2ea859c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x559dad576520 con 0x559dad616ba0 2024-03-22T01:37:47.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.595+0000 7f2ea48df640 1 -- 172.21.15.72:0/1605816487 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f2e940d0100 con 0x7f2ea0101250 2024-03-22T01:37:47.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.595+0000 7f2e9f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559dad6200a0 0x559dad622490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:47.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.595+0000 7f2e9e7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559dad6174c0 0x559dad61fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:47.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.595+0000 7f2e9effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559dad616ba0 0x559dad616f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:47.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.595+0000 7f2e9f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559dad6200a0 0x559dad622490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38050/0 (socket says 172.21.15.72:38050) 2024-03-22T01:37:47.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.595+0000 7f2e9f7fe640 1 -- 172.21.15.72:0/817097316 learned_addr learned my addr 172.21.15.72:0/817097316 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:37:47.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.595+0000 7f2e9f7fe640 1 --2- 172.21.15.72:0/1605816487 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f2e880a3230 0x7f2e880a56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:47.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.595+0000 7f2e9e7fc640 1 -- 172.21.15.72:0/817097316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559dad616ba0 msgr2=0x559dad616f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:47.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.595+0000 7f2e9e7fc640 1 --2- 172.21.15.72:0/817097316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559dad616ba0 0x559dad616f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.595+0000 7f2e9e7fc640 1 -- 172.21.15.72:0/817097316 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559dad6200a0 msgr2=0x559dad622490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:47.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.595+0000 7f2e9e7fc640 1 --2- 172.21.15.72:0/817097316 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559dad6200a0 0x559dad622490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.595+0000 7f2e9e7fc640 1 -- 172.21.15.72:0/817097316 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559dad3567e0 con 0x559dad6174c0 2024-03-22T01:37:47.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.596+0000 7f2e9e7fc640 1 --2- 172.21.15.72:0/817097316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559dad6174c0 0x559dad61fb60 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2e940cd320 tx=0x7f2e9409bd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7add9a59ea0c724a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:47.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.596+0000 7f2e9f7fe640 1 --2- 172.21.15.72:0/1605816487 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f2e880a3230 0x7f2e880a56f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x559dad612460 tx=0x7f2e98009550 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:47.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.596+0000 7f2e85ffb640 1 -- 172.21.15.72:0/817097316 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e940bc110 con 0x559dad6174c0 2024-03-22T01:37:47.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.596+0000 7f2e85ffb640 1 -- 172.21.15.72:0/817097316 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e9406c180 con 0x559dad6174c0 2024-03-22T01:37:47.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.596+0000 7f2e85ffb640 1 -- 172.21.15.72:0/817097316 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e94098560 con 0x559dad6174c0 2024-03-22T01:37:47.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.596+0000 7f2ea859c740 1 -- 172.21.15.72:0/817097316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559dad6174c0 msgr2=0x559dad61fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:47.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.597+0000 7f2ea859c740 1 --2- 172.21.15.72:0/817097316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559dad6174c0 0x559dad61fb60 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2e940cd320 tx=0x7f2e9409bd80 comp rx=0 tx=0).stop 2024-03-22T01:37:47.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.597+0000 7f2ea859c740 1 -- 172.21.15.72:0/817097316 shutdown_connections 2024-03-22T01:37:47.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.597+0000 7f2ea859c740 1 --2- 172.21.15.72:0/817097316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559dad6174c0 0x559dad61fb60 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.597+0000 7f2ea859c740 1 --2- 172.21.15.72:0/817097316 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559dad6200a0 0x559dad622490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.597+0000 7f2ea859c740 1 --2- 172.21.15.72:0/817097316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559dad616ba0 0x559dad616f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.597+0000 7f2ea859c740 1 -- 172.21.15.72:0/817097316 >> 172.21.15.72:0/817097316 conn(0x559dad6112d0 msgr2=0x559dad6116b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:47.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.597+0000 7f2ea859c740 1 -- 172.21.15.72:0/817097316 shutdown_connections 2024-03-22T01:37:47.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.597+0000 7f2ea859c740 1 -- 172.21.15.72:0/817097316 wait complete. 2024-03-22T01:37:47.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.597+0000 7f2ea859c740 1 -- start start 2024-03-22T01:37:47.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.597+0000 7f2ea859c740 1 Processor -- start 2024-03-22T01:37:47.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.597+0000 7f2ea859c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559dad616ba0 0x559dad61fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:47.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.598+0000 7f2ea859c740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559dad6174c0 0x559dad61cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:47.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.598+0000 7f2ea859c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559dad6200a0 0x559dad61d380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:47.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.598+0000 7f2e9effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559dad616ba0 0x559dad61fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:47.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.598+0000 7f2e9e7fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559dad6174c0 0x559dad61cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:47.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.598+0000 7f2e9effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559dad616ba0 0x559dad61fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.72:47934/0 (socket says 172.21.15.72:47934) 2024-03-22T01:37:47.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.598+0000 7f2e9e7fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559dad6174c0 0x559dad61cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38054/0 (socket says 172.21.15.72:38054) 2024-03-22T01:37:47.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.598+0000 7f2e9effd640 1 -- 172.21.15.72:0/2549589116 learned_addr learned my addr 172.21.15.72:0/2549589116 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:37:47.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.598+0000 7f2ea859c740 1 -- 172.21.15.72:0/2549589116 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x559dad622e90 con 0x559dad6174c0 2024-03-22T01:37:47.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.598+0000 7f2ea859c740 1 -- 172.21.15.72:0/2549589116 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x559dad536110 con 0x559dad616ba0 2024-03-22T01:37:47.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.598+0000 7f2ea859c740 1 -- 172.21.15.72:0/2549589116 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x559dad623010 con 0x559dad6200a0 2024-03-22T01:37:47.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.598+0000 7f2ea859c740 10 client.0 _ll_register_callbacks cb 0x559dad572be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:37:47.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.598+0000 7f2e9f7fe640 1 --2- 172.21.15.72:0/2549589116 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559dad6200a0 0x559dad61d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:47.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.598+0000 7f2e9effd640 1 -- 172.21.15.72:0/2549589116 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559dad6200a0 msgr2=0x559dad61d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:47.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.598+0000 7f2e9effd640 1 --2- 172.21.15.72:0/2549589116 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559dad6200a0 0x559dad61d380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.598+0000 7f2e9effd640 1 -- 172.21.15.72:0/2549589116 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559dad6174c0 msgr2=0x559dad61cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:47.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.598+0000 7f2e9effd640 1 --2- 172.21.15.72:0/2549589116 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559dad6174c0 0x559dad61cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.598+0000 7f2e9effd640 1 -- 172.21.15.72:0/2549589116 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559dad3b6f70 con 0x559dad616ba0 2024-03-22T01:37:47.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.598+0000 7f2e9e7fc640 1 --2- 172.21.15.72:0/2549589116 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559dad6174c0 0x559dad61cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:37:47.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.598+0000 7f2e9effd640 1 --2- 172.21.15.72:0/2549589116 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559dad616ba0 0x559dad61fa50 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f2e9002c140 tx=0x7f2e90004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ed6828cced19db2 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:47.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.598+0000 7f2e7effd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-03-22T01:37:47.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.598+0000 7f2e7effd640 1 -- 172.21.15.72:0/2549589116 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e90039040 con 0x559dad616ba0 2024-03-22T01:37:47.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.599+0000 7f2e7effd640 1 -- 172.21.15.72:0/2549589116 <== mon.2 v2:172.21.15.89:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f2e90034640 con 0x559dad616ba0 2024-03-22T01:37:47.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.599+0000 7f2ea859c740 1 -- 172.21.15.72:0/2549589116 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2e9406c180 con 0x559dad616ba0 2024-03-22T01:37:47.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.599+0000 7f2e7effd640 1 -- 172.21.15.72:0/2549589116 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f2e90059050 con 0x559dad616ba0 2024-03-22T01:37:47.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.599+0000 7f2ea859c740 10 client.25688 fetch_fsmap learned FSMap version 21 2024-03-22T01:37:47.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.599+0000 7f2ea859c740 1 -- 172.21.15.72:0/2549589116 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x559dad24e170 con 0x559dad616ba0 2024-03-22T01:37:47.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.599+0000 7f2e7effd640 1 -- 172.21.15.72:0/2549589116 <== mon.2 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e9406c180 con 0x559dad616ba0 2024-03-22T01:37:47.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.599+0000 7f2e7effd640 1 -- 172.21.15.72:0/2549589116 <== mon.2 v2:172.21.15.89:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f2e90054030 con 0x559dad616ba0 2024-03-22T01:37:47.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.599+0000 7f2ea859c740 10 client.25688 fetch_fsmap finished waiting for FSMap version 21 2024-03-22T01:37:47.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.599+0000 7f2ea859c740 10 client.25688 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T01:37:47.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.599+0000 7f2ea859c740 20 client.25688 populate_metadata read hostname 'smithi072' 2024-03-22T01:37:47.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.599+0000 7f2ea859c740 1 --2- 172.21.15.72:0/2549589116 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x559dad630160 0x559dad6325d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:47.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.599+0000 7f2ea859c740 4 client.25688 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T01:37:47.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.599+0000 7f2ea859c740 1 -- 172.21.15.72:0/2549589116 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559dad587ab0 con 0x559dad630160 2024-03-22T01:37:47.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.599+0000 7f2e9e7fc640 1 --2- 172.21.15.72:0/2549589116 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x559dad630160 0x559dad6325d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:47.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.600+0000 7f2e7effd640 10 client.25688 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:37:47.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.600+0000 7f2e9e7fc640 1 --2- 172.21.15.72:0/2549589116 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x559dad630160 0x559dad6325d0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9a56b666fc9de75 server_cookie=e831261744b687f4 in_seq=0 out_seq=0 2024-03-22T01:37:47.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.600+0000 7f2e7effd640 1 -- 172.21.15.72:0/2549589116 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559dad587ab0 con 0x559dad630160 2024-03-22T01:37:47.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.600+0000 7f2e7effd640 10 client.25688 handle_command_reply: tid=0 2024-03-22T01:37:47.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.600+0000 7f2ea859c740 1 client.25688 shutdown 2024-03-22T01:37:47.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.601+0000 7f2ea859c740 1 -- 172.21.15.72:0/2549589116 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559dad616ba0 msgr2=0x559dad61fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:47.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.601+0000 7f2ea859c740 1 --2- 172.21.15.72:0/2549589116 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559dad616ba0 0x559dad61fa50 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f2e9002c140 tx=0x7f2e90004270 comp rx=0 tx=0).stop 2024-03-22T01:37:47.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.601+0000 7f2ea859c740 1 -- 172.21.15.72:0/2549589116 shutdown_connections 2024-03-22T01:37:47.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.601+0000 7f2ea859c740 1 --2- 172.21.15.72:0/2549589116 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x559dad630160 0x559dad6325d0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.601+0000 7f2e7effd640 0 client.25688 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:37:47.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.601+0000 7f2ea859c740 1 --2- 172.21.15.72:0/2549589116 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559dad6200a0 0x559dad61d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.601+0000 7f2ea859c740 1 --2- 172.21.15.72:0/2549589116 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559dad6174c0 0x559dad61cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.601+0000 7f2ea859c740 1 --2- 172.21.15.72:0/2549589116 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559dad616ba0 0x559dad61fa50 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.601+0000 7f2ea859c740 1 -- 172.21.15.72:0/2549589116 >> 172.21.15.72:0/2549589116 conn(0x559dad6112d0 msgr2=0x559dad613020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:47.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.601+0000 7f2ea859c740 1 -- 172.21.15.72:0/2549589116 shutdown_connections 2024-03-22T01:37:47.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.601+0000 7f2ea859c740 1 -- 172.21.15.72:0/2549589116 wait complete. 2024-03-22T01:37:47.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.601+0000 7f2ea859c740 20 client.25688 trim_cache size 0 max 16384 2024-03-22T01:37:47.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.618+0000 7f2ea859c740 1 Processor -- start 2024-03-22T01:37:47.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.618+0000 7f2ea859c740 1 -- start start 2024-03-22T01:37:47.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.618+0000 7f2ea859c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559dad616ba0 0x559dad62bc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:47.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.618+0000 7f2ea859c740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559dad6174c0 0x559dad637ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:47.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.618+0000 7f2ea859c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559dad6200a0 0x559dad61f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:47.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.618+0000 7f2ea859c740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f2ea01aaf80 con 0x559dad6174c0 2024-03-22T01:37:47.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.618+0000 7f2ea859c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2ea01ad800 con 0x559dad6200a0 2024-03-22T01:37:47.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.618+0000 7f2ea859c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2ea01ab5f0 con 0x559dad616ba0 2024-03-22T01:37:47.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.619+0000 7f2e9e7fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559dad6174c0 0x559dad637ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:47.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.619+0000 7f2e9e7fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559dad6174c0 0x559dad637ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38056/0 (socket says 172.21.15.72:38056) 2024-03-22T01:37:47.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.619+0000 7f2e9e7fc640 1 -- 172.21.15.72:0/318839908 learned_addr learned my addr 172.21.15.72:0/318839908 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:37:47.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.619+0000 7f2e9f7fe640 1 --2- 172.21.15.72:0/318839908 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559dad6200a0 0x559dad61f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:47.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.619+0000 7f2e9effd640 1 --2- 172.21.15.72:0/318839908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559dad616ba0 0x559dad62bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:47.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.619+0000 7f2e9e7fc640 1 -- 172.21.15.72:0/318839908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559dad616ba0 msgr2=0x559dad62bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:47.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.619+0000 7f2e9e7fc640 1 --2- 172.21.15.72:0/318839908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559dad616ba0 0x559dad62bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.619+0000 7f2e9e7fc640 1 -- 172.21.15.72:0/318839908 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559dad6200a0 msgr2=0x559dad61f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:47.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.619+0000 7f2e9e7fc640 1 --2- 172.21.15.72:0/318839908 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559dad6200a0 0x559dad61f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.619+0000 7f2e9e7fc640 1 -- 172.21.15.72:0/318839908 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559dad611b10 con 0x559dad6174c0 2024-03-22T01:37:47.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.619+0000 7f2e9e7fc640 1 --2- 172.21.15.72:0/318839908 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559dad6174c0 0x559dad637ea0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f2e94098350 tx=0x7f2e94000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d96172deb3f6c0ee server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:47.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.619+0000 7f2e857fa640 1 -- 172.21.15.72:0/318839908 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e9407f070 con 0x559dad6174c0 2024-03-22T01:37:47.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.619+0000 7f2e857fa640 1 -- 172.21.15.72:0/318839908 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e940bc110 con 0x559dad6174c0 2024-03-22T01:37:47.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.619+0000 7f2e857fa640 1 -- 172.21.15.72:0/318839908 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e940a46d0 con 0x559dad6174c0 2024-03-22T01:37:47.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.620+0000 7f2ea859c740 1 -- 172.21.15.72:0/318839908 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559dad6174c0 msgr2=0x559dad637ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:47.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.620+0000 7f2ea859c740 1 --2- 172.21.15.72:0/318839908 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559dad6174c0 0x559dad637ea0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f2e94098350 tx=0x7f2e94000bc0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.620+0000 7f2ea859c740 1 -- 172.21.15.72:0/318839908 shutdown_connections 2024-03-22T01:37:47.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.620+0000 7f2ea859c740 1 --2- 172.21.15.72:0/318839908 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559dad6200a0 0x559dad61f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.620+0000 7f2ea859c740 1 --2- 172.21.15.72:0/318839908 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559dad6174c0 0x559dad637ea0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.620+0000 7f2ea859c740 1 --2- 172.21.15.72:0/318839908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559dad616ba0 0x559dad62bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.620+0000 7f2ea859c740 1 -- 172.21.15.72:0/318839908 >> 172.21.15.72:0/318839908 conn(0x559dad6112d0 msgr2=0x559dad60fb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:47.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.620+0000 7f2ea859c740 1 -- 172.21.15.72:0/318839908 shutdown_connections 2024-03-22T01:37:47.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.620+0000 7f2ea859c740 1 -- 172.21.15.72:0/318839908 wait complete. 2024-03-22T01:37:47.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.621+0000 7f2ea859c740 1 -- start start 2024-03-22T01:37:47.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.621+0000 7f2ea859c740 1 Processor -- start 2024-03-22T01:37:47.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.621+0000 7f2ea859c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559dad616ba0 0x559dad628260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:47.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.621+0000 7f2ea859c740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559dad6174c0 0x559dad619460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:47.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.622+0000 7f2e9e7fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559dad6174c0 0x559dad619460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:47.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.622+0000 7f2e9e7fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559dad6174c0 0x559dad619460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38062/0 (socket says 172.21.15.72:38062) 2024-03-22T01:37:47.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.622+0000 7f2ea859c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559dad6200a0 0x559dad6199a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:47.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.622+0000 7f2e9effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559dad616ba0 0x559dad628260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:47.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.622+0000 7f2e9e7fc640 1 -- 172.21.15.72:0/1936523681 learned_addr learned my addr 172.21.15.72:0/1936523681 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:37:47.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.622+0000 7f2ea859c740 1 -- 172.21.15.72:0/1936523681 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f2ea01abcc0 con 0x559dad6174c0 2024-03-22T01:37:47.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.622+0000 7f2ea859c740 1 -- 172.21.15.72:0/1936523681 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2ea01ad130 con 0x559dad616ba0 2024-03-22T01:37:47.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.622+0000 7f2ea859c740 1 -- 172.21.15.72:0/1936523681 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2ea01ac390 con 0x559dad6200a0 2024-03-22T01:37:47.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.622+0000 7f2ea859c740 10 client.0 _ll_register_callbacks cb 0x7f2e900096b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:37:47.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.622+0000 7f2e9f7fe640 1 --2- 172.21.15.72:0/1936523681 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559dad6200a0 0x559dad6199a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:47.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.622+0000 7f2e9effd640 1 -- 172.21.15.72:0/1936523681 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559dad6200a0 msgr2=0x559dad6199a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:47.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.622+0000 7f2e9effd640 1 --2- 172.21.15.72:0/1936523681 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559dad6200a0 0x559dad6199a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.622+0000 7f2e9effd640 1 -- 172.21.15.72:0/1936523681 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559dad6174c0 msgr2=0x559dad619460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:47.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.622+0000 7f2e9effd640 1 --2- 172.21.15.72:0/1936523681 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559dad6174c0 0x559dad619460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.622+0000 7f2e9effd640 1 -- 172.21.15.72:0/1936523681 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559dad619ee0 con 0x559dad616ba0 2024-03-22T01:37:47.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.622+0000 7f2e9f7fe640 1 --2- 172.21.15.72:0/1936523681 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559dad6200a0 0x559dad6199a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T01:37:47.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.622+0000 7f2e9e7fc640 1 --2- 172.21.15.72:0/1936523681 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559dad6174c0 0x559dad619460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:37:47.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.622+0000 7f2e9effd640 1 --2- 172.21.15.72:0/1936523681 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559dad616ba0 0x559dad628260 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2e90003700 tx=0x7f2e9002c300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c739682976e8b557 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:47.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.622+0000 7f2e7effd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-03-22T01:37:47.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.622+0000 7f2e7effd640 1 -- 172.21.15.72:0/1936523681 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e90039040 con 0x559dad616ba0 2024-03-22T01:37:47.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.623+0000 7f2e7effd640 1 -- 172.21.15.72:0/1936523681 <== mon.2 v2:172.21.15.89:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f2e90034440 con 0x559dad616ba0 2024-03-22T01:37:47.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.623+0000 7f2ea859c740 1 -- 172.21.15.72:0/1936523681 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2e940bc110 con 0x559dad616ba0 2024-03-22T01:37:47.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.623+0000 7f2e7effd640 1 -- 172.21.15.72:0/1936523681 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f2e90034640 con 0x559dad616ba0 2024-03-22T01:37:47.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.623+0000 7f2ea859c740 10 client.25694 fetch_fsmap learned FSMap version 21 2024-03-22T01:37:47.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.623+0000 7f2ea859c740 1 -- 172.21.15.72:0/1936523681 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x559dad696110 con 0x559dad616ba0 2024-03-22T01:37:47.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.623+0000 7f2e7effd640 1 -- 172.21.15.72:0/1936523681 <== mon.2 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e940bc110 con 0x559dad616ba0 2024-03-22T01:37:47.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.623+0000 7f2e7effd640 1 -- 172.21.15.72:0/1936523681 <== mon.2 v2:172.21.15.89:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f2e90054040 con 0x559dad616ba0 2024-03-22T01:37:47.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.623+0000 7f2ea859c740 10 client.25694 fetch_fsmap finished waiting for FSMap version 21 2024-03-22T01:37:47.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.623+0000 7f2ea859c740 10 client.25694 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T01:37:47.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.623+0000 7f2ea859c740 20 client.25694 populate_metadata read hostname 'smithi072' 2024-03-22T01:37:47.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.623+0000 7f2ea859c740 1 --2- 172.21.15.72:0/1936523681 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x559dad630160 0x559dad698700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:47.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.624+0000 7f2ea859c740 4 client.25694 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-22T01:37:47.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.624+0000 7f2ea859c740 1 -- 172.21.15.72:0/1936523681 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x559dad581e00 con 0x559dad630160 2024-03-22T01:37:47.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.624+0000 7f2e9e7fc640 1 --2- 172.21.15.72:0/1936523681 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x559dad630160 0x559dad698700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:47.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.624+0000 7f2e7effd640 10 client.25694 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:37:47.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.624+0000 7f2e9e7fc640 1 --2- 172.21.15.72:0/1936523681 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x559dad630160 0x559dad698700 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60e8df5e2d9f56d2 server_cookie=a8a3dd9699c4cb2d in_seq=0 out_seq=0 2024-03-22T01:37:47.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.625+0000 7f2e7effd640 1 -- 172.21.15.72:0/1936523681 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x559dad581e00 con 0x559dad630160 2024-03-22T01:37:47.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.625+0000 7f2e7effd640 10 client.25694 handle_command_reply: tid=0 2024-03-22T01:37:47.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.625+0000 7f2ea859c740 1 client.25694 shutdown 2024-03-22T01:37:47.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.625+0000 7f2ea859c740 1 -- 172.21.15.72:0/1936523681 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559dad616ba0 msgr2=0x559dad628260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:47.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.625+0000 7f2ea859c740 1 --2- 172.21.15.72:0/1936523681 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559dad616ba0 0x559dad628260 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2e90003700 tx=0x7f2e9002c300 comp rx=0 tx=0).stop 2024-03-22T01:37:47.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.625+0000 7f2ea859c740 1 -- 172.21.15.72:0/1936523681 shutdown_connections 2024-03-22T01:37:47.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.625+0000 7f2ea859c740 1 --2- 172.21.15.72:0/1936523681 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x559dad630160 0x559dad698700 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.625+0000 7f2ea859c740 1 --2- 172.21.15.72:0/1936523681 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559dad6200a0 0x559dad6199a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.625+0000 7f2ea859c740 1 --2- 172.21.15.72:0/1936523681 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x559dad6174c0 0x559dad619460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.625+0000 7f2e7effd640 0 client.25694 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:37:47.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.625+0000 7f2ea859c740 1 --2- 172.21.15.72:0/1936523681 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559dad616ba0 0x559dad628260 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.625+0000 7f2ea859c740 1 -- 172.21.15.72:0/1936523681 >> 172.21.15.72:0/1936523681 conn(0x559dad6112d0 msgr2=0x559dad6103f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:47.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.626+0000 7f2ea859c740 1 -- 172.21.15.72:0/1936523681 shutdown_connections 2024-03-22T01:37:47.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.626+0000 7f2ea859c740 1 -- 172.21.15.72:0/1936523681 wait complete. 2024-03-22T01:37:47.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.626+0000 7f2ea859c740 20 client.25694 trim_cache size 0 max 16384 2024-03-22T01:37:47.631 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-22T01:37:47.631 INFO:teuthology.orchestra.run.smithi072.stdout: "return_code": 0, 2024-03-22T01:37:47.631 INFO:teuthology.orchestra.run.smithi072.stdout: "scrub_tag": "617534d6-6eb6-4588-aa60-be1239fe5dc2", 2024-03-22T01:37:47.631 INFO:teuthology.orchestra.run.smithi072.stdout: "mode": "asynchronous" 2024-03-22T01:37:47.631 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-22T01:37:47.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.626+0000 7f2ea58e1640 1 -- 172.21.15.72:0/1605816487 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f2e880a3230 msgr2=0x7f2e880a56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:47.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.626+0000 7f2ea58e1640 1 --2- 172.21.15.72:0/1605816487 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f2e880a3230 0x7f2e880a56f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x559dad612460 tx=0x7f2e98009550 comp rx=0 tx=0).stop 2024-03-22T01:37:47.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.626+0000 7f2ea58e1640 1 -- 172.21.15.72:0/1605816487 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2ea0101250 msgr2=0x7f2ea019ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:47.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.626+0000 7f2ea58e1640 1 --2- 172.21.15.72:0/1605816487 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2ea0101250 0x7f2ea019ac20 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f2e9400e9c0 tx=0x7f2e9400ee80 comp rx=0 tx=0).stop 2024-03-22T01:37:47.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.627+0000 7f2ea58e1640 1 -- 172.21.15.72:0/1605816487 shutdown_connections 2024-03-22T01:37:47.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.627+0000 7f2ea58e1640 1 --2- 172.21.15.72:0/1605816487 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f2e880a3230 0x7f2e880a56f0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.627+0000 7f2ea58e1640 1 --2- 172.21.15.72:0/1605816487 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ea010e4e0 0x7f2ea019ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.627+0000 7f2ea58e1640 1 --2- 172.21.15.72:0/1605816487 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2ea0101250 0x7f2ea019ac20 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.627+0000 7f2ea58e1640 1 --2- 172.21.15.72:0/1605816487 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ea0100930 0x7f2ea019a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.627+0000 7f2ea58e1640 1 -- 172.21.15.72:0/1605816487 >> 172.21.15.72:0/1605816487 conn(0x7f2ea0077c40 msgr2=0x7f2ea010fa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:47.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.627+0000 7f2ea58e1640 1 -- 172.21.15.72:0/1605816487 shutdown_connections 2024-03-22T01:37:47.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.627+0000 7f2ea58e1640 1 -- 172.21.15.72:0/1605816487 wait complete. 2024-03-22T01:37:47.640 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T01:37:47.640 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T01:37:47.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.749+0000 7ff672ec0640 1 -- 172.21.15.72:0/618273167 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff66c100a30 msgr2=0x7ff66c100eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:47.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.749+0000 7ff672ec0640 1 --2- 172.21.15.72:0/618273167 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff66c100a30 0x7ff66c100eb0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7ff658009930 tx=0x7ff65802bc30 comp rx=0 tx=0).stop 2024-03-22T01:37:47.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.750+0000 7ff672ec0640 1 -- 172.21.15.72:0/618273167 shutdown_connections 2024-03-22T01:37:47.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.750+0000 7ff672ec0640 1 --2- 172.21.15.72:0/618273167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff66c1013f0 0x7ff66c10d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.750+0000 7ff672ec0640 1 --2- 172.21.15.72:0/618273167 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff66c100a30 0x7ff66c100eb0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.750+0000 7ff672ec0640 1 --2- 172.21.15.72:0/618273167 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff66c106a40 0x7ff66c106e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.750+0000 7ff672ec0640 1 -- 172.21.15.72:0/618273167 >> 172.21.15.72:0/618273167 conn(0x7ff66c077c40 msgr2=0x7ff66c078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:47.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.750+0000 7ff672ec0640 1 -- 172.21.15.72:0/618273167 shutdown_connections 2024-03-22T01:37:47.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.750+0000 7ff672ec0640 1 -- 172.21.15.72:0/618273167 wait complete. 2024-03-22T01:37:47.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.750+0000 7ff672ec0640 1 Processor -- start 2024-03-22T01:37:47.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.750+0000 7ff672ec0640 1 -- start start 2024-03-22T01:37:47.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.750+0000 7ff672ec0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff66c100a30 0x7ff66c198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:47.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.750+0000 7ff672ec0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff66c1013f0 0x7ff66c1989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:47.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.750+0000 7ff672ec0640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff66c106a40 0x7ff66c19cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:47.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.751+0000 7ff672ec0640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7ff66c111f50 con 0x7ff66c106a40 2024-03-22T01:37:47.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.751+0000 7ff672ec0640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff66c111dd0 con 0x7ff66c1013f0 2024-03-22T01:37:47.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.751+0000 7ff672ec0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff66c1120d0 con 0x7ff66c100a30 2024-03-22T01:37:47.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.751+0000 7ff66bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff66c1013f0 0x7ff66c1989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:47.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.751+0000 7ff67145d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff66c106a40 0x7ff66c19cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:47.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.751+0000 7ff66bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff66c1013f0 0x7ff66c1989b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.72:47978/0 (socket says 172.21.15.72:47978) 2024-03-22T01:37:47.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.751+0000 7ff66bfff640 1 -- 172.21.15.72:0/2671297949 learned_addr learned my addr 172.21.15.72:0/2671297949 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:37:47.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.751+0000 7ff670c5c640 1 --2- 172.21.15.72:0/2671297949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff66c100a30 0x7ff66c198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:47.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.751+0000 7ff66bfff640 1 -- 172.21.15.72:0/2671297949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff66c100a30 msgr2=0x7ff66c198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:47.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.751+0000 7ff66bfff640 1 --2- 172.21.15.72:0/2671297949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff66c100a30 0x7ff66c198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.751+0000 7ff66bfff640 1 -- 172.21.15.72:0/2671297949 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff66c106a40 msgr2=0x7ff66c19cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:47.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.751+0000 7ff66bfff640 1 --2- 172.21.15.72:0/2671297949 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff66c106a40 0x7ff66c19cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.751+0000 7ff66bfff640 1 -- 172.21.15.72:0/2671297949 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff66c19d410 con 0x7ff66c1013f0 2024-03-22T01:37:47.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.751+0000 7ff66bfff640 1 --2- 172.21.15.72:0/2671297949 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff66c1013f0 0x7ff66c1989b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff658009230 tx=0x7ff65802cba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:47.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.751+0000 7ff669ffb640 1 -- 172.21.15.72:0/2671297949 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff658041070 con 0x7ff66c1013f0 2024-03-22T01:37:47.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.751+0000 7ff672ec0640 1 -- 172.21.15.72:0/2671297949 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff66c19d6a0 con 0x7ff66c1013f0 2024-03-22T01:37:47.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.751+0000 7ff669ffb640 1 -- 172.21.15.72:0/2671297949 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff658004450 con 0x7ff66c1013f0 2024-03-22T01:37:47.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.751+0000 7ff669ffb640 1 -- 172.21.15.72:0/2671297949 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff658034440 con 0x7ff66c1013f0 2024-03-22T01:37:47.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.752+0000 7ff672ec0640 1 -- 172.21.15.72:0/2671297949 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff66c1a4f90 con 0x7ff66c1013f0 2024-03-22T01:37:47.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.754+0000 7ff669ffb640 1 -- 172.21.15.72:0/2671297949 <== mon.2 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 24) ==== 138087+0+0 (secure 0 0 0) 0x7ff658044070 con 0x7ff66c1013f0 2024-03-22T01:37:47.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.755+0000 7ff675b83740 1 Processor -- start 2024-03-22T01:37:47.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.755+0000 7ff675b83740 1 -- start start 2024-03-22T01:37:47.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.755+0000 7ff669ffb640 1 --2- 172.21.15.72:0/2671297949 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7ff65c0a31b0 0x7ff65c0a5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:47.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.755+0000 7ff669ffb640 1 -- 172.21.15.72:0/2671297949 <== mon.2 v2:172.21.15.89:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7ff6580f16c0 con 0x7ff66c1013f0 2024-03-22T01:37:47.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.755+0000 7ff675b83740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564318ed1410 0x564318ed17f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:47.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.755+0000 7ff675b83740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564318ed1d30 0x564318eda3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:47.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.755+0000 7ff675b83740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564318eda910 0x564318edcd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:47.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.755+0000 7ff675b83740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x564318e358c0 con 0x564318ed1d30 2024-03-22T01:37:47.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.755+0000 7ff675b83740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x564318dddc20 con 0x564318eda910 2024-03-22T01:37:47.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.755+0000 7ff675b83740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x564318e31870 con 0x564318ed1410 2024-03-22T01:37:47.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.755+0000 7ff67145d640 1 --2- 172.21.15.72:0/2671297949 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7ff65c0a31b0 0x7ff65c0a5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:47.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.755+0000 7ff670c5c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564318ed1410 0x564318ed17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:47.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.755+0000 7ff66bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564318ed1d30 0x564318eda3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:47.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.755+0000 7ff670c5c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564318ed1410 0x564318ed17f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.72:56722/0 (socket says 172.21.15.72:56722) 2024-03-22T01:37:47.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.755+0000 7ff670c5c640 1 -- 172.21.15.72:0/630945962 learned_addr learned my addr 172.21.15.72:0/630945962 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:37:47.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.755+0000 7ff670c5c640 1 -- 172.21.15.72:0/630945962 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564318eda910 msgr2=0x564318edcd00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-22T01:37:47.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.755+0000 7ff670c5c640 1 --2- 172.21.15.72:0/630945962 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564318eda910 0x564318edcd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.755+0000 7ff670c5c640 1 -- 172.21.15.72:0/630945962 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564318ed1d30 msgr2=0x564318eda3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:47.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.755+0000 7ff670c5c640 1 --2- 172.21.15.72:0/630945962 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564318ed1d30 0x564318eda3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.755+0000 7ff670c5c640 1 -- 172.21.15.72:0/630945962 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564318e02150 con 0x564318ed1410 2024-03-22T01:37:47.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.755+0000 7ff66bfff640 1 --2- 172.21.15.72:0/630945962 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564318ed1d30 0x564318eda3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:37:47.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.756+0000 7ff670c5c640 1 --2- 172.21.15.72:0/630945962 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564318ed1410 0x564318ed17f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff660009c70 tx=0x7ff660011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69bfbbd7ae8d4782 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:47.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.756+0000 7ff67145d640 1 --2- 172.21.15.72:0/2671297949 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7ff65c0a31b0 0x7ff65c0a5670 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff664005fd0 tx=0x7ff664009550 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:47.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.756+0000 7ff656ffd640 1 -- 172.21.15.72:0/630945962 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff660004230 con 0x564318ed1410 2024-03-22T01:37:47.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.756+0000 7ff656ffd640 1 -- 172.21.15.72:0/630945962 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6600043d0 con 0x564318ed1410 2024-03-22T01:37:47.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.756+0000 7ff656ffd640 1 -- 172.21.15.72:0/630945962 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6600197a0 con 0x564318ed1410 2024-03-22T01:37:47.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.757+0000 7ff675b83740 1 -- 172.21.15.72:0/630945962 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564318ed1410 msgr2=0x564318ed17f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:47.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.757+0000 7ff675b83740 1 --2- 172.21.15.72:0/630945962 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564318ed1410 0x564318ed17f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff660009c70 tx=0x7ff660011810 comp rx=0 tx=0).stop 2024-03-22T01:37:47.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.757+0000 7ff675b83740 1 -- 172.21.15.72:0/630945962 shutdown_connections 2024-03-22T01:37:47.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.757+0000 7ff675b83740 1 --2- 172.21.15.72:0/630945962 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564318eda910 0x564318edcd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.757+0000 7ff675b83740 1 --2- 172.21.15.72:0/630945962 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564318ed1d30 0x564318eda3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.757+0000 7ff675b83740 1 --2- 172.21.15.72:0/630945962 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564318ed1410 0x564318ed17f0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.757+0000 7ff675b83740 1 -- 172.21.15.72:0/630945962 >> 172.21.15.72:0/630945962 conn(0x564318ecbb40 msgr2=0x564318ecbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:47.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.757+0000 7ff675b83740 1 -- 172.21.15.72:0/630945962 shutdown_connections 2024-03-22T01:37:47.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.757+0000 7ff675b83740 1 -- 172.21.15.72:0/630945962 wait complete. 2024-03-22T01:37:47.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.757+0000 7ff675b83740 1 -- start start 2024-03-22T01:37:47.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.757+0000 7ff675b83740 1 Processor -- start 2024-03-22T01:37:47.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.757+0000 7ff675b83740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564318ed1410 0x564318ed7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:47.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.757+0000 7ff675b83740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564318ed1d30 0x564318ed7ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:47.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.757+0000 7ff675b83740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564318eda910 0x564318ed83e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:47.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.758+0000 7ff675b83740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x564318edd700 con 0x564318ed1d30 2024-03-22T01:37:47.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.758+0000 7ff675b83740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x564318df1130 con 0x564318eda910 2024-03-22T01:37:47.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.758+0000 7ff675b83740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x564318edd880 con 0x564318ed1410 2024-03-22T01:37:47.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.758+0000 7ff670c5c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564318ed1410 0x564318ed7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:47.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.758+0000 7ff66bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564318ed1d30 0x564318ed7ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:47.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.758+0000 7ff670c5c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564318ed1410 0x564318ed7960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.72:56738/0 (socket says 172.21.15.72:56738) 2024-03-22T01:37:47.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.758+0000 7ff670c5c640 1 -- 172.21.15.72:0/844937218 learned_addr learned my addr 172.21.15.72:0/844937218 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:37:47.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.758+0000 7ff675b83740 10 client.0 _ll_register_callbacks cb 0x564318df0b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:37:47.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.758+0000 7ff67145d640 1 --2- 172.21.15.72:0/844937218 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564318eda910 0x564318ed83e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:47.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.758+0000 7ff670c5c640 1 -- 172.21.15.72:0/844937218 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564318eda910 msgr2=0x564318ed83e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:47.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.758+0000 7ff670c5c640 1 --2- 172.21.15.72:0/844937218 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564318eda910 0x564318ed83e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.758+0000 7ff670c5c640 1 -- 172.21.15.72:0/844937218 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564318ed1d30 msgr2=0x564318ed7ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:47.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.758+0000 7ff670c5c640 1 --2- 172.21.15.72:0/844937218 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564318ed1d30 0x564318ed7ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.758+0000 7ff670c5c640 1 -- 172.21.15.72:0/844937218 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564318c137d0 con 0x564318ed1410 2024-03-22T01:37:47.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.758+0000 7ff66bfff640 1 --2- 172.21.15.72:0/844937218 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564318ed1d30 0x564318ed7ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:37:47.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.758+0000 7ff670c5c640 1 --2- 172.21.15.72:0/844937218 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564318ed1410 0x564318ed7960 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff660000c00 tx=0x7ff660009f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc9741bb8a5f550c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:47.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.758+0000 7ff64bfff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-03-22T01:37:47.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.758+0000 7ff64bfff640 1 -- 172.21.15.72:0/844937218 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff660027070 con 0x564318ed1410 2024-03-22T01:37:47.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.759+0000 7ff64bfff640 1 -- 172.21.15.72:0/844937218 <== mon.1 v2:172.21.15.98:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7ff660003800 con 0x564318ed1410 2024-03-22T01:37:47.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.759+0000 7ff675b83740 1 -- 172.21.15.72:0/844937218 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff6600043d0 con 0x564318ed1410 2024-03-22T01:37:47.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.759+0000 7ff64bfff640 1 -- 172.21.15.72:0/844937218 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7ff660033050 con 0x564318ed1410 2024-03-22T01:37:47.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.759+0000 7ff675b83740 10 client.25369 fetch_fsmap learned FSMap version 21 2024-03-22T01:37:47.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.759+0000 7ff675b83740 1 -- 172.21.15.72:0/844937218 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x564318b493f0 con 0x564318ed1410 2024-03-22T01:37:47.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.759+0000 7ff64bfff640 1 -- 172.21.15.72:0/844937218 <== mon.1 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff660005610 con 0x564318ed1410 2024-03-22T01:37:47.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.760+0000 7ff64bfff640 1 -- 172.21.15.72:0/844937218 <== mon.1 v2:172.21.15.98:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7ff660018070 con 0x564318ed1410 2024-03-22T01:37:47.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.760+0000 7ff675b83740 10 client.25369 fetch_fsmap finished waiting for FSMap version 21 2024-03-22T01:37:47.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.760+0000 7ff675b83740 10 client.25369 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T01:37:47.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.760+0000 7ff675b83740 20 client.25369 populate_metadata read hostname 'smithi072' 2024-03-22T01:37:47.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.760+0000 7ff675b83740 1 --2- 172.21.15.72:0/844937218 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x564318ef3330 0x564318ef5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:47.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.760+0000 7ff675b83740 4 client.25369 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T01:37:47.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.760+0000 7ff675b83740 1 -- 172.21.15.72:0/844937218 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7ff6600043d0 con 0x564318ef3330 2024-03-22T01:37:47.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.760+0000 7ff66bfff640 1 --2- 172.21.15.72:0/844937218 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x564318ef3330 0x564318ef5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:47.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.760+0000 7ff64bfff640 10 client.25369 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:37:47.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.760+0000 7ff66bfff640 1 --2- 172.21.15.72:0/844937218 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x564318ef3330 0x564318ef5720 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e268f7b2a91bab79 server_cookie=a4a90a2719119a5d in_seq=0 out_seq=0 2024-03-22T01:37:47.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.761+0000 7ff64bfff640 1 -- 172.21.15.72:0/844937218 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7ff6600043d0 con 0x564318ef3330 2024-03-22T01:37:47.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.761+0000 7ff64bfff640 10 client.25369 handle_command_reply: tid=0 2024-03-22T01:37:47.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.761+0000 7ff675b83740 1 client.25369 shutdown 2024-03-22T01:37:47.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.761+0000 7ff675b83740 1 -- 172.21.15.72:0/844937218 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564318ed1410 msgr2=0x564318ed7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:47.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.761+0000 7ff675b83740 1 --2- 172.21.15.72:0/844937218 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564318ed1410 0x564318ed7960 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff660000c00 tx=0x7ff660009f60 comp rx=0 tx=0).stop 2024-03-22T01:37:47.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.761+0000 7ff675b83740 1 -- 172.21.15.72:0/844937218 shutdown_connections 2024-03-22T01:37:47.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.761+0000 7ff675b83740 1 --2- 172.21.15.72:0/844937218 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x564318ef3330 0x564318ef5720 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.761+0000 7ff64bfff640 0 client.25369 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:37:47.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.761+0000 7ff675b83740 1 --2- 172.21.15.72:0/844937218 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564318eda910 0x564318ed83e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.761+0000 7ff675b83740 1 --2- 172.21.15.72:0/844937218 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564318ed1d30 0x564318ed7ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.761+0000 7ff675b83740 1 --2- 172.21.15.72:0/844937218 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564318ed1410 0x564318ed7960 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.761+0000 7ff675b83740 1 -- 172.21.15.72:0/844937218 >> 172.21.15.72:0/844937218 conn(0x564318ecbb40 msgr2=0x564318ecd890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:47.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.761+0000 7ff675b83740 1 -- 172.21.15.72:0/844937218 shutdown_connections 2024-03-22T01:37:47.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.761+0000 7ff675b83740 1 -- 172.21.15.72:0/844937218 wait complete. 2024-03-22T01:37:47.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.761+0000 7ff675b83740 20 client.25369 trim_cache size 0 max 16384 2024-03-22T01:37:47.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.779+0000 7ff675b83740 1 Processor -- start 2024-03-22T01:37:47.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.779+0000 7ff675b83740 1 -- start start 2024-03-22T01:37:47.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.779+0000 7ff675b83740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564318ed1410 0x564318eda020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:47.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.779+0000 7ff675b83740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564318ed1d30 0x564318ed6b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:47.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.779+0000 7ff675b83740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564318eda910 0x564318eeed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:47.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.779+0000 7ff675b83740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7ff66c1aaeb0 con 0x564318ed1410 2024-03-22T01:37:47.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.779+0000 7ff675b83740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff66c1ad730 con 0x564318ed1d30 2024-03-22T01:37:47.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.779+0000 7ff675b83740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff66c1ab520 con 0x564318eda910 2024-03-22T01:37:47.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.779+0000 7ff670c5c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564318ed1410 0x564318eda020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:47.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.779+0000 7ff670c5c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564318ed1410 0x564318eda020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38122/0 (socket says 172.21.15.72:38122) 2024-03-22T01:37:47.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.779+0000 7ff670c5c640 1 -- 172.21.15.72:0/39934275 learned_addr learned my addr 172.21.15.72:0/39934275 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:37:47.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.779+0000 7ff66bfff640 1 --2- 172.21.15.72:0/39934275 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564318ed1d30 0x564318ed6b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:47.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.779+0000 7ff67145d640 1 --2- 172.21.15.72:0/39934275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564318eda910 0x564318eeed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:47.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.779+0000 7ff670c5c640 1 -- 172.21.15.72:0/39934275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564318eda910 msgr2=0x564318eeed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:47.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.779+0000 7ff670c5c640 1 --2- 172.21.15.72:0/39934275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564318eda910 0x564318eeed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.779+0000 7ff670c5c640 1 -- 172.21.15.72:0/39934275 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564318ed1d30 msgr2=0x564318ed6b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:47.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.779+0000 7ff670c5c640 1 --2- 172.21.15.72:0/39934275 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564318ed1d30 0x564318ed6b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.779+0000 7ff670c5c640 1 -- 172.21.15.72:0/39934275 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564318b090f0 con 0x564318ed1410 2024-03-22T01:37:47.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.779+0000 7ff66bfff640 1 --2- 172.21.15.72:0/39934275 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564318ed1d30 0x564318ed6b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T01:37:47.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.780+0000 7ff670c5c640 1 --2- 172.21.15.72:0/39934275 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564318ed1410 0x564318eda020 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7ff66000a010 tx=0x7ff660011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9679383d06d3de86 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:47.781 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.780+0000 7ff6567fc640 1 -- 172.21.15.72:0/39934275 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff660027070 con 0x564318ed1410 2024-03-22T01:37:47.781 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.780+0000 7ff6567fc640 1 -- 172.21.15.72:0/39934275 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff660015030 con 0x564318ed1410 2024-03-22T01:37:47.781 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.780+0000 7ff6567fc640 1 -- 172.21.15.72:0/39934275 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff660005400 con 0x564318ed1410 2024-03-22T01:37:47.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.781+0000 7ff675b83740 1 -- 172.21.15.72:0/39934275 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564318ed1410 msgr2=0x564318eda020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:47.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.781+0000 7ff675b83740 1 --2- 172.21.15.72:0/39934275 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564318ed1410 0x564318eda020 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7ff66000a010 tx=0x7ff660011810 comp rx=0 tx=0).stop 2024-03-22T01:37:47.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.781+0000 7ff675b83740 1 -- 172.21.15.72:0/39934275 shutdown_connections 2024-03-22T01:37:47.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.781+0000 7ff675b83740 1 --2- 172.21.15.72:0/39934275 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564318ed1d30 0x564318ed6b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.781+0000 7ff675b83740 1 --2- 172.21.15.72:0/39934275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564318eda910 0x564318eeed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.781+0000 7ff675b83740 1 --2- 172.21.15.72:0/39934275 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564318ed1410 0x564318eda020 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.781+0000 7ff675b83740 1 -- 172.21.15.72:0/39934275 >> 172.21.15.72:0/39934275 conn(0x564318ecbb40 msgr2=0x564318eccf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:47.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.781+0000 7ff675b83740 1 -- 172.21.15.72:0/39934275 shutdown_connections 2024-03-22T01:37:47.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.781+0000 7ff675b83740 1 -- 172.21.15.72:0/39934275 wait complete. 2024-03-22T01:37:47.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.781+0000 7ff675b83740 1 -- start start 2024-03-22T01:37:47.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.781+0000 7ff675b83740 1 Processor -- start 2024-03-22T01:37:47.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.782+0000 7ff675b83740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564318ed1410 0x564318eeb670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:47.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.782+0000 7ff675b83740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564318ed1d30 0x564318ed42d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:47.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.782+0000 7ff675b83740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564318eda910 0x564318ed4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:47.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.782+0000 7ff675b83740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7ff66c1abbf0 con 0x564318eda910 2024-03-22T01:37:47.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.782+0000 7ff675b83740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff66c1ad060 con 0x564318ed1d30 2024-03-22T01:37:47.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.782+0000 7ff675b83740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff66c1ac2c0 con 0x564318ed1410 2024-03-22T01:37:47.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.782+0000 7ff675b83740 10 client.0 _ll_register_callbacks cb 0x7ff660003d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:37:47.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.782+0000 7ff67145d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564318eda910 0x564318ed4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:47.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.782+0000 7ff67145d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564318eda910 0x564318ed4810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38124/0 (socket says 172.21.15.72:38124) 2024-03-22T01:37:47.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.782+0000 7ff67145d640 1 -- 172.21.15.72:0/1943753760 learned_addr learned my addr 172.21.15.72:0/1943753760 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:37:47.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.782+0000 7ff66bfff640 1 --2- 172.21.15.72:0/1943753760 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564318ed1d30 0x564318ed42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:47.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.782+0000 7ff670c5c640 1 --2- 172.21.15.72:0/1943753760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564318ed1410 0x564318eeb670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:47.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.782+0000 7ff66bfff640 1 -- 172.21.15.72:0/1943753760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564318ed1410 msgr2=0x564318eeb670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:47.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.782+0000 7ff66bfff640 1 --2- 172.21.15.72:0/1943753760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564318ed1410 0x564318eeb670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.782+0000 7ff66bfff640 1 -- 172.21.15.72:0/1943753760 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564318eda910 msgr2=0x564318ed4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:47.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.782+0000 7ff66bfff640 1 --2- 172.21.15.72:0/1943753760 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564318eda910 0x564318ed4810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.782+0000 7ff66bfff640 1 -- 172.21.15.72:0/1943753760 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564318ed5040 con 0x564318ed1d30 2024-03-22T01:37:47.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.782+0000 7ff67145d640 1 --2- 172.21.15.72:0/1943753760 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564318eda910 0x564318ed4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:37:47.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.782+0000 7ff66bfff640 1 --2- 172.21.15.72:0/1943753760 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564318ed1d30 0x564318ed42d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff6580ee9f0 tx=0x7ff658009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24d74dd7d3126fe1 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:47.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.782+0000 7ff64bfff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-03-22T01:37:47.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.782+0000 7ff64bfff640 1 -- 172.21.15.72:0/1943753760 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff65804b040 con 0x564318ed1d30 2024-03-22T01:37:47.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.782+0000 7ff675b83740 1 -- 172.21.15.72:0/1943753760 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff660015030 con 0x564318ed1d30 2024-03-22T01:37:47.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.783+0000 7ff64bfff640 1 -- 172.21.15.72:0/1943753760 <== mon.2 v2:172.21.15.89:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7ff6580035f0 con 0x564318ed1d30 2024-03-22T01:37:47.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.783+0000 7ff64bfff640 1 -- 172.21.15.72:0/1943753760 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7ff6580bcfd0 con 0x564318ed1d30 2024-03-22T01:37:47.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.783+0000 7ff675b83740 10 client.25706 fetch_fsmap learned FSMap version 21 2024-03-22T01:37:47.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.783+0000 7ff675b83740 1 -- 172.21.15.72:0/1943753760 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x564318f594c0 con 0x564318ed1d30 2024-03-22T01:37:47.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.783+0000 7ff64bfff640 1 -- 172.21.15.72:0/1943753760 <== mon.2 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff658004c80 con 0x564318ed1d30 2024-03-22T01:37:47.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.783+0000 7ff64bfff640 1 -- 172.21.15.72:0/1943753760 <== mon.2 v2:172.21.15.89:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7ff6580c5b40 con 0x564318ed1d30 2024-03-22T01:37:47.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.783+0000 7ff675b83740 10 client.25706 fetch_fsmap finished waiting for FSMap version 21 2024-03-22T01:37:47.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.783+0000 7ff675b83740 10 client.25706 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T01:37:47.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.783+0000 7ff675b83740 20 client.25706 populate_metadata read hostname 'smithi072' 2024-03-22T01:37:47.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.783+0000 7ff675b83740 1 --2- 172.21.15.72:0/1943753760 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x564318f59b90 0x564318f5bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:47.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.783+0000 7ff675b83740 4 client.25706 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T01:37:47.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.783+0000 7ff675b83740 1 -- 172.21.15.72:0/1943753760 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7ff660015030 con 0x564318f59b90 2024-03-22T01:37:47.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.783+0000 7ff670c5c640 1 --2- 172.21.15.72:0/1943753760 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x564318f59b90 0x564318f5bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:47.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.784+0000 7ff64bfff640 10 client.25706 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:37:47.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.784+0000 7ff670c5c640 1 --2- 172.21.15.72:0/1943753760 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x564318f59b90 0x564318f5bf80 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=165a5b5d32df9a1c server_cookie=47469218b72a144a in_seq=0 out_seq=0 2024-03-22T01:37:47.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.784+0000 7ff64bfff640 1 -- 172.21.15.72:0/1943753760 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x7ff660015030 con 0x564318f59b90 2024-03-22T01:37:47.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.784+0000 7ff64bfff640 10 client.25706 handle_command_reply: tid=0 2024-03-22T01:37:47.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.784+0000 7ff675b83740 1 client.25706 shutdown 2024-03-22T01:37:47.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.784+0000 7ff675b83740 1 -- 172.21.15.72:0/1943753760 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564318ed1d30 msgr2=0x564318ed42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:47.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.784+0000 7ff675b83740 1 --2- 172.21.15.72:0/1943753760 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564318ed1d30 0x564318ed42d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff6580ee9f0 tx=0x7ff658009260 comp rx=0 tx=0).stop 2024-03-22T01:37:47.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.784+0000 7ff675b83740 1 -- 172.21.15.72:0/1943753760 shutdown_connections 2024-03-22T01:37:47.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.784+0000 7ff675b83740 1 --2- 172.21.15.72:0/1943753760 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x564318f59b90 0x564318f5bf80 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.784+0000 7ff64bfff640 0 client.25706 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:37:47.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.784+0000 7ff675b83740 1 --2- 172.21.15.72:0/1943753760 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564318ed1d30 0x564318ed42d0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.784+0000 7ff675b83740 1 --2- 172.21.15.72:0/1943753760 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564318eda910 0x564318ed4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.784+0000 7ff675b83740 1 --2- 172.21.15.72:0/1943753760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564318ed1410 0x564318eeb670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.784+0000 7ff675b83740 1 -- 172.21.15.72:0/1943753760 >> 172.21.15.72:0/1943753760 conn(0x564318ecbb40 msgr2=0x564318ecac00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:47.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.784+0000 7ff675b83740 1 -- 172.21.15.72:0/1943753760 shutdown_connections 2024-03-22T01:37:47.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.784+0000 7ff675b83740 1 -- 172.21.15.72:0/1943753760 wait complete. 2024-03-22T01:37:47.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.784+0000 7ff675b83740 20 client.25706 trim_cache size 0 max 16384 2024-03-22T01:37:47.790 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-22T01:37:47.790 INFO:teuthology.orchestra.run.smithi072.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-22T01:37:47.790 INFO:teuthology.orchestra.run.smithi072.stdout: "scrubs": { 2024-03-22T01:37:47.790 INFO:teuthology.orchestra.run.smithi072.stdout: "617534d6-6eb6-4588-aa60-be1239fe5dc2": { 2024-03-22T01:37:47.790 INFO:teuthology.orchestra.run.smithi072.stdout: "path": "/", 2024-03-22T01:37:47.790 INFO:teuthology.orchestra.run.smithi072.stdout: "tag": "617534d6-6eb6-4588-aa60-be1239fe5dc2", 2024-03-22T01:37:47.790 INFO:teuthology.orchestra.run.smithi072.stdout: "options": "recursive,force" 2024-03-22T01:37:47.790 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:37:47.790 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:37:47.791 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-22T01:37:47.791 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.785+0000 7ff672ec0640 1 -- 172.21.15.72:0/2671297949 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7ff65c0a31b0 msgr2=0x7ff65c0a5670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:47.791 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.785+0000 7ff672ec0640 1 --2- 172.21.15.72:0/2671297949 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7ff65c0a31b0 0x7ff65c0a5670 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff664005fd0 tx=0x7ff664009550 comp rx=0 tx=0).stop 2024-03-22T01:37:47.791 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.785+0000 7ff672ec0640 1 -- 172.21.15.72:0/2671297949 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff66c1013f0 msgr2=0x7ff66c1989b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:47.791 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.785+0000 7ff672ec0640 1 --2- 172.21.15.72:0/2671297949 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff66c1013f0 0x7ff66c1989b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff658009230 tx=0x7ff65802cba0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.791 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.785+0000 7ff672ec0640 1 -- 172.21.15.72:0/2671297949 shutdown_connections 2024-03-22T01:37:47.791 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.785+0000 7ff672ec0640 1 --2- 172.21.15.72:0/2671297949 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7ff65c0a31b0 0x7ff65c0a5670 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.791 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.785+0000 7ff672ec0640 1 --2- 172.21.15.72:0/2671297949 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff66c1013f0 0x7ff66c1989b0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.791 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.785+0000 7ff672ec0640 1 --2- 172.21.15.72:0/2671297949 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff66c106a40 0x7ff66c19cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.785+0000 7ff672ec0640 1 --2- 172.21.15.72:0/2671297949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff66c100a30 0x7ff66c198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:47.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.785+0000 7ff672ec0640 1 -- 172.21.15.72:0/2671297949 >> 172.21.15.72:0/2671297949 conn(0x7ff66c077c40 msgr2=0x7ff66c103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:47.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.785+0000 7ff672ec0640 1 -- 172.21.15.72:0/2671297949 shutdown_connections 2024-03-22T01:37:47.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:37:47.785+0000 7ff672ec0640 1 -- 172.21.15.72:0/2671297949 wait complete. 2024-03-22T01:37:47.795 INFO:tasks.cephfs.filesystem:scrub status for tag:617534d6-6eb6-4588-aa60-be1239fe5dc2 - {'path': '/', 'tag': '617534d6-6eb6-4588-aa60-be1239fe5dc2', 'options': 'recursive,force'} 2024-03-22T01:37:48.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:37:47 smithi072 ceph-mon[32414]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-22T01:37:48.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:37:47 smithi072 ceph-mon[32414]: scrub queued for path: / 2024-03-22T01:37:48.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:37:47 smithi072 ceph-mon[32414]: scrub summary: idle+waiting paths [/] 2024-03-22T01:37:48.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:37:47 smithi072 ceph-mon[32414]: scrub summary: active paths [/] 2024-03-22T01:37:48.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:37:47 smithi072 ceph-mon[32414]: scrub summary: idle+waiting paths [/] 2024-03-22T01:37:48.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:37:47 smithi089 ceph-mon[35770]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-22T01:37:48.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:37:47 smithi089 ceph-mon[35770]: scrub queued for path: / 2024-03-22T01:37:48.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:37:47 smithi089 ceph-mon[35770]: scrub summary: idle+waiting paths [/] 2024-03-22T01:37:48.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:37:47 smithi089 ceph-mon[35770]: scrub summary: active paths [/] 2024-03-22T01:37:48.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:37:47 smithi089 ceph-mon[35770]: scrub summary: idle+waiting paths [/] 2024-03-22T01:37:48.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:37:47 smithi098 ceph-mon[36468]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-22T01:37:48.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:37:47 smithi098 ceph-mon[36468]: scrub queued for path: / 2024-03-22T01:37:48.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:37:47 smithi098 ceph-mon[36468]: scrub summary: idle+waiting paths [/] 2024-03-22T01:37:48.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:37:47 smithi098 ceph-mon[36468]: scrub summary: active paths [/] 2024-03-22T01:37:48.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:37:47 smithi098 ceph-mon[36468]: scrub summary: idle+waiting paths [/] 2024-03-22T01:37:49.357 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:37:48 smithi089 ceph-mon[35770]: scrub summary: idle 2024-03-22T01:37:49.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:37:48 smithi072 ceph-mon[32414]: scrub summary: idle 2024-03-22T01:37:49.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:37:48 smithi098 ceph-mon[36468]: scrub summary: idle 2024-03-22T01:37:50.357 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:37:49 smithi089 ceph-mon[35770]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-22T01:37:50.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:37:49 smithi072 ceph-mon[32414]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-22T01:37:50.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:37:49 smithi098 ceph-mon[36468]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-22T01:37:50.745 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.746+0000 7f8bd5765640 1 -- 172.21.15.72:0/790334690 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8bd01013f0 msgr2=0x7f8bd010d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:50.745 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.746+0000 7f8bd5765640 1 --2- 172.21.15.72:0/790334690 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8bd01013f0 0x7f8bd010d900 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f8bc8009ca0 tx=0x7f8bc802c100 comp rx=0 tx=0).stop 2024-03-22T01:37:50.745 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.746+0000 7f8bd5765640 1 -- 172.21.15.72:0/790334690 shutdown_connections 2024-03-22T01:37:50.746 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.746+0000 7f8bd5765640 1 --2- 172.21.15.72:0/790334690 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8bd01013f0 0x7f8bd010d900 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:50.746 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.746+0000 7f8bd5765640 1 --2- 172.21.15.72:0/790334690 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bd0100a30 0x7f8bd0100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:50.746 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.746+0000 7f8bd5765640 1 --2- 172.21.15.72:0/790334690 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8bd0106a40 0x7f8bd0106e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:50.746 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.746+0000 7f8bd5765640 1 -- 172.21.15.72:0/790334690 >> 172.21.15.72:0/790334690 conn(0x7f8bd0077b80 msgr2=0x7f8bd0077f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:50.746 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.746+0000 7f8bd5765640 1 -- 172.21.15.72:0/790334690 shutdown_connections 2024-03-22T01:37:50.746 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.746+0000 7f8bd5765640 1 -- 172.21.15.72:0/790334690 wait complete. 2024-03-22T01:37:50.748 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.747+0000 7f8bd5765640 1 Processor -- start 2024-03-22T01:37:50.748 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.747+0000 7f8bd5765640 1 -- start start 2024-03-22T01:37:50.748 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.747+0000 7f8bd5765640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8bd0100a30 0x7f8bd0198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:50.748 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.747+0000 7f8bd5765640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8bd01013f0 0x7f8bd01986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:50.748 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.747+0000 7f8bd5765640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bd0106a40 0x7f8bd019ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:50.748 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.747+0000 7f8bd5765640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd0111f10 con 0x7f8bd0100a30 2024-03-22T01:37:50.748 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.747+0000 7f8bd5765640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd0111d90 con 0x7f8bd01013f0 2024-03-22T01:37:50.748 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.747+0000 7f8bd5765640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd0112090 con 0x7f8bd0106a40 2024-03-22T01:37:50.748 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.748+0000 7f8bceffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8bd0100a30 0x7f8bd0198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:50.748 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.748+0000 7f8bce7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8bd01013f0 0x7f8bd01986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:50.749 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.748+0000 7f8bceffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8bd0100a30 0x7f8bd0198190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:55612/0 (socket says 172.21.15.72:55612) 2024-03-22T01:37:50.749 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.748+0000 7f8bceffd640 1 -- 172.21.15.72:0/2746446490 learned_addr learned my addr 172.21.15.72:0/2746446490 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:37:50.749 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.748+0000 7f8bcf7fe640 1 --2- 172.21.15.72:0/2746446490 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bd0106a40 0x7f8bd019ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:50.749 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.748+0000 7f8bce7fc640 1 -- 172.21.15.72:0/2746446490 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bd0106a40 msgr2=0x7f8bd019ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:50.749 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.748+0000 7f8bce7fc640 1 --2- 172.21.15.72:0/2746446490 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bd0106a40 0x7f8bd019ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:50.749 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.748+0000 7f8bce7fc640 1 -- 172.21.15.72:0/2746446490 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8bd0100a30 msgr2=0x7f8bd0198190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:50.749 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.748+0000 7f8bce7fc640 1 --2- 172.21.15.72:0/2746446490 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8bd0100a30 0x7f8bd0198190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:50.749 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.748+0000 7f8bce7fc640 1 -- 172.21.15.72:0/2746446490 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bd019d180 con 0x7f8bd01013f0 2024-03-22T01:37:50.749 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.748+0000 7f8bce7fc640 1 --2- 172.21.15.72:0/2746446490 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8bd01013f0 0x7f8bd01986d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f8bc4005fd0 tx=0x7f8bc400dde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:50.749 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.748+0000 7f8bb7fff640 1 -- 172.21.15.72:0/2746446490 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bc4009c30 con 0x7f8bd01013f0 2024-03-22T01:37:50.749 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.748+0000 7f8bb7fff640 1 -- 172.21.15.72:0/2746446490 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8bc4009dd0 con 0x7f8bd01013f0 2024-03-22T01:37:50.749 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.748+0000 7f8bb7fff640 1 -- 172.21.15.72:0/2746446490 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bc4010460 con 0x7f8bd01013f0 2024-03-22T01:37:50.749 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.748+0000 7f8bd5765640 1 -- 172.21.15.72:0/2746446490 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bd019d470 con 0x7f8bd01013f0 2024-03-22T01:37:50.749 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.748+0000 7f8bd5765640 1 -- 172.21.15.72:0/2746446490 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bd01a4d00 con 0x7f8bd01013f0 2024-03-22T01:37:50.750 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.751+0000 7f8bb7fff640 1 -- 172.21.15.72:0/2746446490 <== mon.2 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 24) ==== 138087+0+0 (secure 0 0 0) 0x7f8bc4003f50 con 0x7f8bd01013f0 2024-03-22T01:37:50.752 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.752+0000 7f8bd842b740 1 Processor -- start 2024-03-22T01:37:50.752 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.752+0000 7f8bd842b740 1 -- start start 2024-03-22T01:37:50.752 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.752+0000 7f8bb7fff640 1 --2- 172.21.15.72:0/2746446490 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f8bb80a3220 0x7f8bb80a56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:50.752 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.752+0000 7f8bd842b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bc0dee42d0 0x55bc0dee46b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:50.752 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.752+0000 7f8bd842b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bc0dee4bf0 0x55bc0deed480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:50.752 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.752+0000 7f8bd842b740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55bc0deed9c0 0x55bc0deefdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:50.752 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.752+0000 7f8bd842b740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55bc0de48040 con 0x55bc0deed9c0 2024-03-22T01:37:50.752 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.752+0000 7f8bd842b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55bc0ddf09e0 con 0x55bc0dee42d0 2024-03-22T01:37:50.753 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.752+0000 7f8bd842b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55bc0de43ff0 con 0x55bc0dee4bf0 2024-03-22T01:37:50.753 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.752+0000 7f8bb7fff640 1 -- 172.21.15.72:0/2746446490 <== mon.2 v2:172.21.15.89:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f8bc40cfd50 con 0x7f8bd01013f0 2024-03-22T01:37:50.753 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.752+0000 7f8bceffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bc0dee42d0 0x55bc0dee46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:50.753 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.752+0000 7f8bceffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bc0dee42d0 0x55bc0dee46b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.72:38902/0 (socket says 172.21.15.72:38902) 2024-03-22T01:37:50.753 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.752+0000 7f8bceffd640 1 -- 172.21.15.72:0/337579508 learned_addr learned my addr 172.21.15.72:0/337579508 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:37:50.753 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.752+0000 7f8bcf7fe640 1 --2- 172.21.15.72:0/2746446490 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f8bb80a3220 0x7f8bb80a56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:50.753 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.752+0000 7f8bce7fc640 1 --2- 172.21.15.72:0/337579508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bc0dee4bf0 0x55bc0deed480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:50.753 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.753+0000 7f8bceffd640 1 -- 172.21.15.72:0/337579508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bc0dee4bf0 msgr2=0x55bc0deed480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:50.753 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.753+0000 7f8bceffd640 1 --2- 172.21.15.72:0/337579508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bc0dee4bf0 0x55bc0deed480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:50.753 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.753+0000 7f8bceffd640 1 -- 172.21.15.72:0/337579508 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55bc0deed9c0 msgr2=0x55bc0deefdb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-22T01:37:50.753 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.753+0000 7f8bceffd640 1 --2- 172.21.15.72:0/337579508 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55bc0deed9c0 0x55bc0deefdb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:50.753 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.753+0000 7f8bceffd640 1 -- 172.21.15.72:0/337579508 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bc0db5c210 con 0x55bc0dee42d0 2024-03-22T01:37:50.753 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.753+0000 7f8bceffd640 1 --2- 172.21.15.72:0/337579508 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bc0dee42d0 0x55bc0dee46b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f8bc0009d40 tx=0x7f8bc0011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2bf27fcacb7e4069 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:50.753 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.753+0000 7f8bcf7fe640 1 --2- 172.21.15.72:0/2746446490 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f8bb80a3220 0x7f8bb80a56e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8bc8005e30 tx=0x7f8bc8005d80 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:50.753 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.753+0000 7f8bb57fa640 1 -- 172.21.15.72:0/337579508 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bc0004220 con 0x55bc0dee42d0 2024-03-22T01:37:50.754 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.753+0000 7f8bb57fa640 1 -- 172.21.15.72:0/337579508 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8bc00043c0 con 0x55bc0dee42d0 2024-03-22T01:37:50.754 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.753+0000 7f8bb57fa640 1 -- 172.21.15.72:0/337579508 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bc00035f0 con 0x55bc0dee42d0 2024-03-22T01:37:50.758 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.754+0000 7f8bd842b740 1 -- 172.21.15.72:0/337579508 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bc0dee42d0 msgr2=0x55bc0dee46b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:50.758 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.754+0000 7f8bd842b740 1 --2- 172.21.15.72:0/337579508 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bc0dee42d0 0x55bc0dee46b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f8bc0009d40 tx=0x7f8bc0011810 comp rx=0 tx=0).stop 2024-03-22T01:37:50.758 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.754+0000 7f8bd842b740 1 -- 172.21.15.72:0/337579508 shutdown_connections 2024-03-22T01:37:50.759 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.754+0000 7f8bd842b740 1 --2- 172.21.15.72:0/337579508 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55bc0deed9c0 0x55bc0deefdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:50.759 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.754+0000 7f8bd842b740 1 --2- 172.21.15.72:0/337579508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bc0dee4bf0 0x55bc0deed480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:50.759 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.754+0000 7f8bd842b740 1 --2- 172.21.15.72:0/337579508 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bc0dee42d0 0x55bc0dee46b0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:50.759 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.754+0000 7f8bd842b740 1 -- 172.21.15.72:0/337579508 >> 172.21.15.72:0/337579508 conn(0x55bc0dedebf0 msgr2=0x55bc0dedefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:50.759 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.754+0000 7f8bd842b740 1 -- 172.21.15.72:0/337579508 shutdown_connections 2024-03-22T01:37:50.759 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.754+0000 7f8bd842b740 1 -- 172.21.15.72:0/337579508 wait complete. 2024-03-22T01:37:50.759 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.754+0000 7f8bd842b740 1 -- start start 2024-03-22T01:37:50.759 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.754+0000 7f8bd842b740 1 Processor -- start 2024-03-22T01:37:50.759 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.755+0000 7f8bd842b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bc0dee42d0 0x55bc0deed340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:50.759 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.755+0000 7f8bd842b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bc0dee4bf0 0x55bc0deea6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:50.759 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.755+0000 7f8bd842b740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55bc0deed9c0 0x55bc0deeac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:50.759 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.755+0000 7f8bceffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bc0dee42d0 0x55bc0deed340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:50.759 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.755+0000 7f8bceffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bc0dee42d0 0x55bc0deed340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.72:38914/0 (socket says 172.21.15.72:38914) 2024-03-22T01:37:50.759 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.755+0000 7f8bcf7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55bc0deed9c0 0x55bc0deeac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:50.760 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.755+0000 7f8bce7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bc0dee4bf0 0x55bc0deea6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:50.760 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.755+0000 7f8bce7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bc0dee4bf0 0x55bc0deea6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.72:57754/0 (socket says 172.21.15.72:57754) 2024-03-22T01:37:50.760 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.755+0000 7f8bcf7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55bc0deed9c0 0x55bc0deeac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:55626/0 (socket says 172.21.15.72:55626) 2024-03-22T01:37:50.760 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.755+0000 7f8bceffd640 1 -- 172.21.15.72:0/913492212 learned_addr learned my addr 172.21.15.72:0/913492212 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:37:50.760 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.755+0000 7f8bd842b740 1 -- 172.21.15.72:0/913492212 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55bc0def05f0 con 0x55bc0deed9c0 2024-03-22T01:37:50.760 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.755+0000 7f8bd842b740 1 -- 172.21.15.72:0/913492212 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55bc0de03db0 con 0x55bc0dee42d0 2024-03-22T01:37:50.760 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.755+0000 7f8bd842b740 1 -- 172.21.15.72:0/913492212 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55bc0def0770 con 0x55bc0dee4bf0 2024-03-22T01:37:50.760 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.756+0000 7f8bd842b740 10 client.0 _ll_register_callbacks cb 0x55bc0db5ba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:37:50.760 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.756+0000 7f8bceffd640 1 -- 172.21.15.72:0/913492212 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bc0dee4bf0 msgr2=0x55bc0deea6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:50.760 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.756+0000 7f8bceffd640 1 --2- 172.21.15.72:0/913492212 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bc0dee4bf0 0x55bc0deea6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:50.760 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.756+0000 7f8bceffd640 1 -- 172.21.15.72:0/913492212 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55bc0deed9c0 msgr2=0x55bc0deeac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:50.760 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.756+0000 7f8bceffd640 1 --2- 172.21.15.72:0/913492212 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55bc0deed9c0 0x55bc0deeac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:50.760 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.756+0000 7f8bceffd640 1 -- 172.21.15.72:0/913492212 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bc0dc28620 con 0x55bc0dee42d0 2024-03-22T01:37:50.760 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.756+0000 7f8bd842b740 1 -- 172.21.15.72:0/913492212 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8bc00043c0 con 0x55bc0dee42d0 2024-03-22T01:37:50.760 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.756+0000 7f8bceffd640 1 --2- 172.21.15.72:0/913492212 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bc0dee42d0 0x55bc0deed340 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8bc0004ac0 tx=0x7f8bc00053e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20ab886b3864d4bd server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:50.761 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.756+0000 7f8bb2ff5640 10 client.25718 ms_handle_connect on v2:172.21.15.89:3300/0 2024-03-22T01:37:50.761 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.756+0000 7f8bb2ff5640 1 -- 172.21.15.72:0/913492212 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bc0028020 con 0x55bc0dee42d0 2024-03-22T01:37:50.761 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.756+0000 7f8bb2ff5640 1 -- 172.21.15.72:0/913492212 <== mon.2 v2:172.21.15.89:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f8bc0024640 con 0x55bc0dee42d0 2024-03-22T01:37:50.761 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.756+0000 7f8bb2ff5640 1 -- 172.21.15.72:0/913492212 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f8bc001d030 con 0x55bc0dee42d0 2024-03-22T01:37:50.761 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.756+0000 7f8bd842b740 10 client.25718 fetch_fsmap learned FSMap version 21 2024-03-22T01:37:50.761 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.756+0000 7f8bd842b740 1 -- 172.21.15.72:0/913492212 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({fsmap=21}) -- 0x55bc0defc310 con 0x55bc0dee42d0 2024-03-22T01:37:50.761 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.757+0000 7f8bb2ff5640 1 -- 172.21.15.72:0/913492212 <== mon.2 v2:172.21.15.89:3300/0 4 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f8bc0023070 con 0x55bc0dee42d0 2024-03-22T01:37:50.761 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.757+0000 7f8bd842b740 10 client.25718 fetch_fsmap finished waiting for FSMap version 21 2024-03-22T01:37:50.761 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.757+0000 7f8bd842b740 10 client.25718 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T01:37:50.761 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.757+0000 7f8bd842b740 20 client.25718 populate_metadata read hostname 'smithi072' 2024-03-22T01:37:50.761 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.757+0000 7f8bd842b740 1 --2- 172.21.15.72:0/913492212 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55bc0defd660 0x55bc0deffa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:50.761 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.757+0000 7f8bd842b740 4 client.25718 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T01:37:50.761 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.757+0000 7f8bd842b740 1 -- 172.21.15.72:0/913492212 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bc0de55580 con 0x55bc0defd660 2024-03-22T01:37:50.761 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.757+0000 7f8bce7fc640 1 --2- 172.21.15.72:0/913492212 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55bc0defd660 0x55bc0deffa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:50.762 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.757+0000 7f8bb2ff5640 10 client.25718 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:37:50.762 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.757+0000 7f8bce7fc640 1 --2- 172.21.15.72:0/913492212 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55bc0defd660 0x55bc0deffa50 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7403addd6b0a59c6 server_cookie=ebc85b282e93958d in_seq=0 out_seq=0 2024-03-22T01:37:50.762 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.758+0000 7f8bb2ff5640 1 -- 172.21.15.72:0/913492212 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55bc0de55580 con 0x55bc0defd660 2024-03-22T01:37:50.762 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.758+0000 7f8bb2ff5640 10 client.25718 handle_command_reply: tid=0 2024-03-22T01:37:50.762 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.758+0000 7f8bd842b740 1 client.25718 shutdown 2024-03-22T01:37:50.762 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.758+0000 7f8bd842b740 1 -- 172.21.15.72:0/913492212 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bc0dee42d0 msgr2=0x55bc0deed340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:50.762 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.758+0000 7f8bd842b740 1 --2- 172.21.15.72:0/913492212 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bc0dee42d0 0x55bc0deed340 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8bc0004ac0 tx=0x7f8bc00053e0 comp rx=0 tx=0).stop 2024-03-22T01:37:50.762 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.758+0000 7f8bd842b740 1 -- 172.21.15.72:0/913492212 shutdown_connections 2024-03-22T01:37:50.762 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.758+0000 7f8bd842b740 1 --2- 172.21.15.72:0/913492212 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55bc0defd660 0x55bc0deffa50 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:50.762 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.759+0000 7f8bd842b740 1 --2- 172.21.15.72:0/913492212 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55bc0deed9c0 0x55bc0deeac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:50.762 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.759+0000 7f8bb2ff5640 0 client.25718 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:37:50.762 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.759+0000 7f8bd842b740 1 --2- 172.21.15.72:0/913492212 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bc0dee4bf0 0x55bc0deea6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:50.762 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.759+0000 7f8bd842b740 1 --2- 172.21.15.72:0/913492212 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bc0dee42d0 0x55bc0deed340 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:50.762 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.759+0000 7f8bd842b740 1 -- 172.21.15.72:0/913492212 >> 172.21.15.72:0/913492212 conn(0x55bc0dedebf0 msgr2=0x55bc0dee0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:50.762 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.759+0000 7f8bd842b740 1 -- 172.21.15.72:0/913492212 shutdown_connections 2024-03-22T01:37:50.763 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.759+0000 7f8bd842b740 1 -- 172.21.15.72:0/913492212 wait complete. 2024-03-22T01:37:50.763 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.759+0000 7f8bd842b740 20 client.25718 trim_cache size 0 max 16384 2024-03-22T01:37:50.786 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.786+0000 7f8bd842b740 1 Processor -- start 2024-03-22T01:37:50.787 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.786+0000 7f8bd842b740 1 -- start start 2024-03-22T01:37:50.787 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.787+0000 7f8bd842b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bc0dee42d0 0x55bc0def9390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:50.787 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.787+0000 7f8bd842b740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55bc0dee4bf0 0x55bc0df05220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:50.787 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.787+0000 7f8bce7fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55bc0dee4bf0 0x55bc0df05220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:50.787 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.787+0000 7f8bce7fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55bc0dee4bf0 0x55bc0df05220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:55638/0 (socket says 172.21.15.72:55638) 2024-03-22T01:37:50.787 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.787+0000 7f8bce7fc640 1 -- 172.21.15.72:0/1320402609 learned_addr learned my addr 172.21.15.72:0/1320402609 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:37:50.787 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.787+0000 7f8bd842b740 1 --2- 172.21.15.72:0/1320402609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bc0deed9c0 0x55bc0deece00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:50.787 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.787+0000 7f8bd842b740 1 -- 172.21.15.72:0/1320402609 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd01acec0 con 0x55bc0dee4bf0 2024-03-22T01:37:50.787 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.787+0000 7f8bd842b740 1 -- 172.21.15.72:0/1320402609 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd01af740 con 0x55bc0deed9c0 2024-03-22T01:37:50.787 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.787+0000 7f8bd842b740 1 -- 172.21.15.72:0/1320402609 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd01ad530 con 0x55bc0dee42d0 2024-03-22T01:37:50.787 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.787+0000 7f8bceffd640 1 --2- 172.21.15.72:0/1320402609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bc0dee42d0 0x55bc0def9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:50.787 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.787+0000 7f8bcf7fe640 1 --2- 172.21.15.72:0/1320402609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bc0deed9c0 0x55bc0deece00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:50.787 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.787+0000 7f8bce7fc640 1 -- 172.21.15.72:0/1320402609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bc0dee42d0 msgr2=0x55bc0def9390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:50.788 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.787+0000 7f8bce7fc640 1 --2- 172.21.15.72:0/1320402609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bc0dee42d0 0x55bc0def9390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:50.788 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.787+0000 7f8bce7fc640 1 -- 172.21.15.72:0/1320402609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bc0deed9c0 msgr2=0x55bc0deece00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:50.788 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.787+0000 7f8bce7fc640 1 --2- 172.21.15.72:0/1320402609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bc0deed9c0 0x55bc0deece00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:50.788 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.787+0000 7f8bce7fc640 1 -- 172.21.15.72:0/1320402609 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bc0dee3a90 con 0x55bc0dee4bf0 2024-03-22T01:37:50.788 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.787+0000 7f8bce7fc640 1 --2- 172.21.15.72:0/1320402609 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55bc0dee4bf0 0x55bc0df05220 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f8bc4097fa0 tx=0x7f8bc4009740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54e574b3f50d35b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:50.788 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.788+0000 7f8bb4ff9640 1 -- 172.21.15.72:0/1320402609 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bc4020040 con 0x55bc0dee4bf0 2024-03-22T01:37:50.788 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.788+0000 7f8bb4ff9640 1 -- 172.21.15.72:0/1320402609 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8bc409bb40 con 0x55bc0dee4bf0 2024-03-22T01:37:50.788 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.788+0000 7f8bb4ff9640 1 -- 172.21.15.72:0/1320402609 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bc4003710 con 0x55bc0dee4bf0 2024-03-22T01:37:50.791 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.789+0000 7f8bd842b740 1 -- 172.21.15.72:0/1320402609 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55bc0dee4bf0 msgr2=0x55bc0df05220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:50.791 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.789+0000 7f8bd842b740 1 --2- 172.21.15.72:0/1320402609 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55bc0dee4bf0 0x55bc0df05220 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f8bc4097fa0 tx=0x7f8bc4009740 comp rx=0 tx=0).stop 2024-03-22T01:37:50.792 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.789+0000 7f8bd842b740 1 -- 172.21.15.72:0/1320402609 shutdown_connections 2024-03-22T01:37:50.792 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.789+0000 7f8bd842b740 1 --2- 172.21.15.72:0/1320402609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bc0deed9c0 0x55bc0deece00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:50.792 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.789+0000 7f8bd842b740 1 --2- 172.21.15.72:0/1320402609 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55bc0dee4bf0 0x55bc0df05220 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:50.792 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.789+0000 7f8bd842b740 1 --2- 172.21.15.72:0/1320402609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bc0dee42d0 0x55bc0def9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:50.792 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.789+0000 7f8bd842b740 1 -- 172.21.15.72:0/1320402609 >> 172.21.15.72:0/1320402609 conn(0x55bc0dedebf0 msgr2=0x55bc0dee08a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:50.792 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.789+0000 7f8bd842b740 1 -- 172.21.15.72:0/1320402609 shutdown_connections 2024-03-22T01:37:50.792 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.789+0000 7f8bd842b740 1 -- 172.21.15.72:0/1320402609 wait complete. 2024-03-22T01:37:50.792 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.789+0000 7f8bd842b740 1 -- start start 2024-03-22T01:37:50.793 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.789+0000 7f8bd842b740 1 Processor -- start 2024-03-22T01:37:50.793 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.790+0000 7f8bd842b740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55bc0dee42d0 0x55bc0def5d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:50.793 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.790+0000 7f8bd842b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bc0dee4bf0 0x55bc0dee7090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:50.793 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.790+0000 7f8bd842b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bc0deed9c0 0x55bc0dee76c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:50.793 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.790+0000 7f8bceffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55bc0dee42d0 0x55bc0def5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:50.793 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.790+0000 7f8bd842b740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd01adc00 con 0x55bc0dee42d0 2024-03-22T01:37:50.793 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.790+0000 7f8bceffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55bc0dee42d0 0x55bc0def5d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:55640/0 (socket says 172.21.15.72:55640) 2024-03-22T01:37:50.793 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.790+0000 7f8bd842b740 1 -- 172.21.15.72:0/2720809272 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd01af070 con 0x55bc0dee4bf0 2024-03-22T01:37:50.793 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.790+0000 7f8bceffd640 1 -- 172.21.15.72:0/2720809272 learned_addr learned my addr 172.21.15.72:0/2720809272 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:37:50.793 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.790+0000 7f8bd842b740 1 -- 172.21.15.72:0/2720809272 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd01ae2d0 con 0x55bc0deed9c0 2024-03-22T01:37:50.793 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.790+0000 7f8bce7fc640 1 --2- 172.21.15.72:0/2720809272 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bc0dee4bf0 0x55bc0dee7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:50.793 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.790+0000 7f8bcf7fe640 1 --2- 172.21.15.72:0/2720809272 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bc0deed9c0 0x55bc0dee76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:50.793 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.790+0000 7f8bd842b740 10 client.0 _ll_register_callbacks cb 0x7f8bc0009bc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:37:50.793 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.790+0000 7f8bce7fc640 1 -- 172.21.15.72:0/2720809272 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bc0deed9c0 msgr2=0x55bc0dee76c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:50.794 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.790+0000 7f8bce7fc640 1 --2- 172.21.15.72:0/2720809272 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bc0deed9c0 0x55bc0dee76c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:50.794 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.790+0000 7f8bce7fc640 1 -- 172.21.15.72:0/2720809272 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55bc0dee42d0 msgr2=0x55bc0def5d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:37:50.794 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.790+0000 7f8bce7fc640 1 --2- 172.21.15.72:0/2720809272 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55bc0dee42d0 0x55bc0def5d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:50.794 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.790+0000 7f8bce7fc640 1 -- 172.21.15.72:0/2720809272 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bc0dee7e90 con 0x55bc0dee4bf0 2024-03-22T01:37:50.794 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.791+0000 7f8bd842b740 1 -- 172.21.15.72:0/2720809272 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8bc409bb40 con 0x55bc0dee4bf0 2024-03-22T01:37:50.794 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.791+0000 7f8bce7fc640 1 --2- 172.21.15.72:0/2720809272 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bc0dee4bf0 0x55bc0dee7090 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8bc40a4710 tx=0x7f8bc4003820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97cb1ee4518842cc server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:37:50.794 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.791+0000 7f8bb2ff5640 10 client.25724 ms_handle_connect on v2:172.21.15.89:3300/0 2024-03-22T01:37:50.794 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.791+0000 7f8bb2ff5640 1 -- 172.21.15.72:0/2720809272 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bc4020070 con 0x55bc0dee4bf0 2024-03-22T01:37:50.794 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.791+0000 7f8bb2ff5640 1 -- 172.21.15.72:0/2720809272 <== mon.2 v2:172.21.15.89:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f8bc4051070 con 0x55bc0dee4bf0 2024-03-22T01:37:50.794 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.791+0000 7f8bb2ff5640 1 -- 172.21.15.72:0/2720809272 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f8bc40a4550 con 0x55bc0dee4bf0 2024-03-22T01:37:50.794 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.791+0000 7f8bd842b740 10 client.25724 fetch_fsmap learned FSMap version 21 2024-03-22T01:37:50.794 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.791+0000 7f8bd842b740 1 -- 172.21.15.72:0/2720809272 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({fsmap=21}) -- 0x55bc0df63840 con 0x55bc0dee4bf0 2024-03-22T01:37:50.794 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.791+0000 7f8bb2ff5640 1 -- 172.21.15.72:0/2720809272 <== mon.2 v2:172.21.15.89:3300/0 4 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f8bc404d3f0 con 0x55bc0dee4bf0 2024-03-22T01:37:50.795 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.791+0000 7f8bd842b740 10 client.25724 fetch_fsmap finished waiting for FSMap version 21 2024-03-22T01:37:50.795 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.791+0000 7f8bd842b740 10 client.25724 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T01:37:50.795 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.791+0000 7f8bd842b740 20 client.25724 populate_metadata read hostname 'smithi072' 2024-03-22T01:37:50.795 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.791+0000 7f8bd842b740 1 --2- 172.21.15.72:0/2720809272 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55bc0defd660 0x55bc0df65ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:37:50.795 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.791+0000 7f8bd842b740 4 client.25724 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T01:37:50.795 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.791+0000 7f8bd842b740 1 -- 172.21.15.72:0/2720809272 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55bc0de4f8d0 con 0x55bc0defd660 2024-03-22T01:37:50.795 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.792+0000 7f8bceffd640 1 --2- 172.21.15.72:0/2720809272 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55bc0defd660 0x55bc0df65ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:37:50.795 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.792+0000 7f8bb2ff5640 10 client.25724 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:37:50.795 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.792+0000 7f8bceffd640 1 --2- 172.21.15.72:0/2720809272 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55bc0defd660 0x55bc0df65ea0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=67a5d99ed44eb5e0 server_cookie=29c6f6cae3cde5d5 in_seq=0 out_seq=0 2024-03-22T01:37:50.796 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.795+0000 7f8bb2ff5640 1 -- 172.21.15.72:0/2720809272 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55bc0de4f8d0 con 0x55bc0defd660 2024-03-22T01:37:50.796 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.795+0000 7f8bb2ff5640 10 client.25724 handle_command_reply: tid=0 2024-03-22T01:37:50.796 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.795+0000 7f8bd842b740 1 client.25724 shutdown 2024-03-22T01:37:50.796 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.796+0000 7f8bd842b740 1 -- 172.21.15.72:0/2720809272 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bc0dee4bf0 msgr2=0x55bc0dee7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:50.797 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.796+0000 7f8bd842b740 1 --2- 172.21.15.72:0/2720809272 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bc0dee4bf0 0x55bc0dee7090 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8bc40a4710 tx=0x7f8bc4003820 comp rx=0 tx=0).stop 2024-03-22T01:37:50.797 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.796+0000 7f8bd842b740 1 -- 172.21.15.72:0/2720809272 shutdown_connections 2024-03-22T01:37:50.797 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.796+0000 7f8bd842b740 1 --2- 172.21.15.72:0/2720809272 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55bc0defd660 0x55bc0df65ea0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:50.797 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.796+0000 7f8bd842b740 1 --2- 172.21.15.72:0/2720809272 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bc0dee4bf0 0x55bc0dee7090 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:50.797 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.796+0000 7f8bd842b740 1 --2- 172.21.15.72:0/2720809272 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55bc0deed9c0 0x55bc0dee76c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:50.797 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.796+0000 7f8bd842b740 1 --2- 172.21.15.72:0/2720809272 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55bc0dee42d0 0x55bc0def5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:50.797 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.796+0000 7f8bd842b740 1 -- 172.21.15.72:0/2720809272 >> 172.21.15.72:0/2720809272 conn(0x55bc0dedebf0 msgr2=0x55bc0dedfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:50.797 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.796+0000 7f8bb2ff5640 0 client.25724 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:37:50.797 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.796+0000 7f8bd842b740 1 -- 172.21.15.72:0/2720809272 shutdown_connections 2024-03-22T01:37:50.797 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.796+0000 7f8bd842b740 1 -- 172.21.15.72:0/2720809272 wait complete. 2024-03-22T01:37:50.797 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.796+0000 7f8bd842b740 20 client.25724 trim_cache size 0 max 16384 2024-03-22T01:37:50.797 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout:{ 2024-03-22T01:37:50.798 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout: "message": "", 2024-03-22T01:37:50.798 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout: "return_code": 0 2024-03-22T01:37:50.798 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout:} 2024-03-22T01:37:50.799 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.797+0000 7f8bd5765640 1 -- 172.21.15.72:0/2746446490 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f8bb80a3220 msgr2=0x7f8bb80a56e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:50.799 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.797+0000 7f8bd5765640 1 --2- 172.21.15.72:0/2746446490 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f8bb80a3220 0x7f8bb80a56e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8bc8005e30 tx=0x7f8bc8005d80 comp rx=0 tx=0).stop 2024-03-22T01:37:50.799 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.797+0000 7f8bd5765640 1 -- 172.21.15.72:0/2746446490 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8bd01013f0 msgr2=0x7f8bd01986d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:37:50.799 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.797+0000 7f8bd5765640 1 --2- 172.21.15.72:0/2746446490 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8bd01013f0 0x7f8bd01986d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f8bc4005fd0 tx=0x7f8bc400dde0 comp rx=0 tx=0).stop 2024-03-22T01:37:50.799 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.797+0000 7f8bd5765640 1 -- 172.21.15.72:0/2746446490 shutdown_connections 2024-03-22T01:37:50.799 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.797+0000 7f8bd5765640 1 --2- 172.21.15.72:0/2746446490 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f8bb80a3220 0x7f8bb80a56e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:50.799 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.797+0000 7f8bd5765640 1 --2- 172.21.15.72:0/2746446490 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8bd01013f0 0x7f8bd01986d0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:50.799 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.797+0000 7f8bd5765640 1 --2- 172.21.15.72:0/2746446490 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bd0106a40 0x7f8bd019ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:50.799 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.797+0000 7f8bd5765640 1 --2- 172.21.15.72:0/2746446490 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8bd0100a30 0x7f8bd0198190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:37:50.799 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.797+0000 7f8bd5765640 1 -- 172.21.15.72:0/2746446490 >> 172.21.15.72:0/2746446490 conn(0x7f8bd0077b80 msgr2=0x7f8bd0103480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:37:50.799 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.797+0000 7f8bd5765640 1 -- 172.21.15.72:0/2746446490 shutdown_connections 2024-03-22T01:37:50.799 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:37:50.797+0000 7f8bd5765640 1 -- 172.21.15.72:0/2746446490 wait complete. 2024-03-22T01:37:52.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:37:51 smithi089 ceph-mon[35770]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-22T01:37:52.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:37:51 smithi072 ceph-mon[32414]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-22T01:37:52.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:37:51 smithi098 ceph-mon[36468]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-22T01:37:54.357 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:37:53 smithi089 ceph-mon[35770]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 2 op/s 2024-03-22T01:37:54.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:37:53 smithi072 ceph-mon[32414]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 2 op/s 2024-03-22T01:37:54.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:37:53 smithi098 ceph-mon[36468]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 2 op/s 2024-03-22T01:37:56.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:37:55 smithi089 ceph-mon[35770]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-22T01:37:56.371 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:37:55 smithi072 ceph-mon[32414]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-22T01:37:56.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:37:55 smithi098 ceph-mon[36468]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-22T01:37:58.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:37:57 smithi089 ceph-mon[35770]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-22T01:37:58.371 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:37:57 smithi072 ceph-mon[32414]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-22T01:37:58.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:37:57 smithi098 ceph-mon[36468]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-22T01:38:00.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:37:59 smithi089 ceph-mon[35770]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-22T01:38:00.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:37:59 smithi072 ceph-mon[32414]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-22T01:38:00.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:37:59 smithi098 ceph-mon[36468]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-22T01:38:02.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:38:01 smithi089 ceph-mon[35770]: pgmap v44: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-22T01:38:02.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:38:01 smithi072 ceph-mon[32414]: pgmap v44: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-22T01:38:02.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:38:01 smithi098 ceph-mon[36468]: pgmap v44: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-22T01:38:02.623 INFO:tasks.workunit.client.0.smithi072.stderr:Note: switching to 'c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e'. 2024-03-22T01:38:02.623 INFO:tasks.workunit.client.0.smithi072.stderr: 2024-03-22T01:38:02.623 INFO:tasks.workunit.client.0.smithi072.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-22T01:38:02.623 INFO:tasks.workunit.client.0.smithi072.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-22T01:38:02.623 INFO:tasks.workunit.client.0.smithi072.stderr:state without impacting any branches by switching back to a branch. 2024-03-22T01:38:02.623 INFO:tasks.workunit.client.0.smithi072.stderr: 2024-03-22T01:38:02.623 INFO:tasks.workunit.client.0.smithi072.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-22T01:38:02.624 INFO:tasks.workunit.client.0.smithi072.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-22T01:38:02.624 INFO:tasks.workunit.client.0.smithi072.stderr: 2024-03-22T01:38:02.624 INFO:tasks.workunit.client.0.smithi072.stderr: git switch -c 2024-03-22T01:38:02.624 INFO:tasks.workunit.client.0.smithi072.stderr: 2024-03-22T01:38:02.624 INFO:tasks.workunit.client.0.smithi072.stderr:Or undo this operation with: 2024-03-22T01:38:02.624 INFO:tasks.workunit.client.0.smithi072.stderr: 2024-03-22T01:38:02.624 INFO:tasks.workunit.client.0.smithi072.stderr: git switch - 2024-03-22T01:38:02.624 INFO:tasks.workunit.client.0.smithi072.stderr: 2024-03-22T01:38:02.624 INFO:tasks.workunit.client.0.smithi072.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-22T01:38:02.624 INFO:tasks.workunit.client.0.smithi072.stderr: 2024-03-22T01:38:02.624 INFO:tasks.workunit.client.0.smithi072.stderr:HEAD is now at c3a04c0e75f Merge PR #54581 into wip-batrick-testing-20240321.172515 2024-03-22T01:38:02.628 DEBUG:teuthology.orchestra.run.smithi072:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-22T01:38:02.828 INFO:tasks.workunit.client.0.smithi072.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-22T01:38:02.830 INFO:tasks.workunit.client.0.smithi072.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-22T01:38:02.830 INFO:tasks.workunit.client.0.smithi072.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-22T01:38:03.899 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.897+0000 7f5c5420e640 1 -- 172.21.15.72:0/2202894642 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c4c107c10 msgr2=0x7f5c4c107ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:03.899 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.897+0000 7f5c5420e640 1 --2- 172.21.15.72:0/2202894642 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c4c107c10 0x7f5c4c107ff0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f5c48009900 tx=0x7f5c4802bc30 comp rx=0 tx=0).stop 2024-03-22T01:38:03.899 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.897+0000 7f5c5420e640 1 -- 172.21.15.72:0/2202894642 shutdown_connections 2024-03-22T01:38:03.899 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.897+0000 7f5c5420e640 1 --2- 172.21.15.72:0/2202894642 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c4c069e10 0x7f5c4c10e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:03.899 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.897+0000 7f5c5420e640 1 --2- 172.21.15.72:0/2202894642 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c4c069450 0x7f5c4c0698d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:03.899 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.897+0000 7f5c5420e640 1 --2- 172.21.15.72:0/2202894642 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c4c107c10 0x7f5c4c107ff0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:03.899 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.897+0000 7f5c5420e640 1 -- 172.21.15.72:0/2202894642 >> 172.21.15.72:0/2202894642 conn(0x7f5c4c077b80 msgr2=0x7f5c4c077f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:38:03.900 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.898+0000 7f5c5420e640 1 -- 172.21.15.72:0/2202894642 shutdown_connections 2024-03-22T01:38:03.900 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.898+0000 7f5c5420e640 1 -- 172.21.15.72:0/2202894642 wait complete. 2024-03-22T01:38:03.900 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.898+0000 7f5c5420e640 1 Processor -- start 2024-03-22T01:38:03.900 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.898+0000 7f5c5420e640 1 -- start start 2024-03-22T01:38:03.900 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.898+0000 7f5c5420e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c4c069450 0x7f5c4c198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:03.900 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.898+0000 7f5c5420e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c4c069e10 0x7f5c4c198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:03.900 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.898+0000 7f5c5420e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c4c107c10 0x7f5c4c19cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:03.900 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.898+0000 7f5c5420e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f5c4c111f50 con 0x7f5c4c069450 2024-03-22T01:38:03.900 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.898+0000 7f5c5420e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5c4c111dd0 con 0x7f5c4c069e10 2024-03-22T01:38:03.900 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.898+0000 7f5c5420e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5c4c1120d0 con 0x7f5c4c107c10 2024-03-22T01:38:03.900 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.898+0000 7f5c51faa640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c4c069450 0x7f5c4c198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:03.900 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.899+0000 7f5c51faa640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c4c069450 0x7f5c4c198450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33472/0 (socket says 172.21.15.72:33472) 2024-03-22T01:38:03.901 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.899+0000 7f5c51faa640 1 -- 172.21.15.72:0/2617593020 learned_addr learned my addr 172.21.15.72:0/2617593020 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:38:03.901 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.899+0000 7f5c517a9640 1 --2- 172.21.15.72:0/2617593020 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c4c069e10 0x7f5c4c198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:03.901 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.899+0000 7f5c527ab640 1 --2- 172.21.15.72:0/2617593020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c4c107c10 0x7f5c4c19cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:03.901 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.899+0000 7f5c51faa640 1 -- 172.21.15.72:0/2617593020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c4c107c10 msgr2=0x7f5c4c19cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:03.901 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.899+0000 7f5c51faa640 1 --2- 172.21.15.72:0/2617593020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c4c107c10 0x7f5c4c19cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:03.901 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.899+0000 7f5c51faa640 1 -- 172.21.15.72:0/2617593020 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c4c069e10 msgr2=0x7f5c4c198990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:03.901 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.899+0000 7f5c51faa640 1 --2- 172.21.15.72:0/2617593020 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c4c069e10 0x7f5c4c198990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:03.901 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.899+0000 7f5c51faa640 1 -- 172.21.15.72:0/2617593020 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c4c19d440 con 0x7f5c4c069450 2024-03-22T01:38:03.901 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.899+0000 7f5c527ab640 1 --2- 172.21.15.72:0/2617593020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c4c107c10 0x7f5c4c19cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T01:38:03.901 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.899+0000 7f5c517a9640 1 --2- 172.21.15.72:0/2617593020 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c4c069e10 0x7f5c4c198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:38:03.901 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.899+0000 7f5c51faa640 1 --2- 172.21.15.72:0/2617593020 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c4c069450 0x7f5c4c198450 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f5c48005940 tx=0x7f5c480042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:38:03.901 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.899+0000 7f5c42ffd640 1 -- 172.21.15.72:0/2617593020 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c48041070 con 0x7f5c4c069450 2024-03-22T01:38:03.902 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.899+0000 7f5c42ffd640 1 -- 172.21.15.72:0/2617593020 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c4802ccc0 con 0x7f5c4c069450 2024-03-22T01:38:03.902 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.899+0000 7f5c5420e640 1 -- 172.21.15.72:0/2617593020 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c4c19d6d0 con 0x7f5c4c069450 2024-03-22T01:38:03.902 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.899+0000 7f5c42ffd640 1 -- 172.21.15.72:0/2617593020 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c48003c70 con 0x7f5c4c069450 2024-03-22T01:38:03.902 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.899+0000 7f5c5420e640 1 -- 172.21.15.72:0/2617593020 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c4c1a4fc0 con 0x7f5c4c069450 2024-03-22T01:38:03.909 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.901+0000 7f5c42ffd640 1 -- 172.21.15.72:0/2617593020 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 24) ==== 138087+0+0 (secure 0 0 0) 0x7f5c48044070 con 0x7f5c4c069450 2024-03-22T01:38:03.909 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.902+0000 7f5c56df3740 1 Processor -- start 2024-03-22T01:38:03.909 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.902+0000 7f5c56df3740 1 -- start start 2024-03-22T01:38:03.909 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.902+0000 7f5c42ffd640 1 --2- 172.21.15.72:0/2617593020 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f5c380a31a0 0x7f5c380a5660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:03.910 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.902+0000 7f5c56df3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561cf0fa12d0 0x561cf0fa16b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:03.910 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.902+0000 7f5c56df3740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561cf0fa1bf0 0x561cf0faa480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:03.910 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.902+0000 7f5c56df3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561cf0faa9c0 0x561cf0facdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:03.910 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.902+0000 7f5c56df3740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x561cf0f05040 con 0x561cf0fa1bf0 2024-03-22T01:38:03.910 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.902+0000 7f5c56df3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x561cf0ead9e0 con 0x561cf0faa9c0 2024-03-22T01:38:03.910 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.902+0000 7f5c56df3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561cf0f00ff0 con 0x561cf0fa12d0 2024-03-22T01:38:03.910 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.902+0000 7f5c527ab640 1 --2- 172.21.15.72:0/2617593020 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f5c380a31a0 0x7f5c380a5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:03.910 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.902+0000 7f5c42ffd640 1 -- 172.21.15.72:0/2617593020 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f5c480f1870 con 0x7f5c4c069450 2024-03-22T01:38:03.910 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.902+0000 7f5c517a9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561cf0fa1bf0 0x561cf0faa480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:03.910 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.903+0000 7f5c517a9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561cf0fa1bf0 0x561cf0faa480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33484/0 (socket says 172.21.15.72:33484) 2024-03-22T01:38:03.910 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.903+0000 7f5c517a9640 1 -- 172.21.15.72:0/169987448 learned_addr learned my addr 172.21.15.72:0/169987448 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:38:03.910 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.903+0000 7f5c527ab640 1 --2- 172.21.15.72:0/169987448 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561cf0faa9c0 0x561cf0facdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:03.911 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.903+0000 7f5c51faa640 1 --2- 172.21.15.72:0/169987448 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561cf0fa12d0 0x561cf0fa16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:03.911 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.903+0000 7f5c517a9640 1 -- 172.21.15.72:0/169987448 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561cf0fa12d0 msgr2=0x561cf0fa16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:03.911 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.903+0000 7f5c517a9640 1 --2- 172.21.15.72:0/169987448 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561cf0fa12d0 0x561cf0fa16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:03.911 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.903+0000 7f5c517a9640 1 -- 172.21.15.72:0/169987448 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561cf0faa9c0 msgr2=0x561cf0facdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:03.911 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.903+0000 7f5c517a9640 1 --2- 172.21.15.72:0/169987448 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561cf0faa9c0 0x561cf0facdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:03.911 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.903+0000 7f5c517a9640 1 -- 172.21.15.72:0/169987448 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561cf0c19210 con 0x561cf0fa1bf0 2024-03-22T01:38:03.911 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.903+0000 7f5c517a9640 1 --2- 172.21.15.72:0/169987448 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561cf0fa1bf0 0x561cf0faa480 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c009e10 tx=0x7f5c3c00e5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26f8aac4034453ee server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:38:03.911 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.903+0000 7f5c527ab640 1 --2- 172.21.15.72:0/2617593020 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f5c380a31a0 0x7f5c380a5660 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5c44005fd0 tx=0x7f5c44009e70 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:38:03.911 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.903+0000 7f5c37fff640 1 -- 172.21.15.72:0/169987448 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c3c017070 con 0x561cf0fa1bf0 2024-03-22T01:38:03.911 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.903+0000 7f5c37fff640 1 -- 172.21.15.72:0/169987448 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c3c013950 con 0x561cf0fa1bf0 2024-03-22T01:38:03.911 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.903+0000 7f5c37fff640 1 -- 172.21.15.72:0/169987448 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c3c0129b0 con 0x561cf0fa1bf0 2024-03-22T01:38:03.911 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.904+0000 7f5c56df3740 1 -- 172.21.15.72:0/169987448 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561cf0fa1bf0 msgr2=0x561cf0faa480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:03.912 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.904+0000 7f5c56df3740 1 --2- 172.21.15.72:0/169987448 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561cf0fa1bf0 0x561cf0faa480 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c009e10 tx=0x7f5c3c00e5a0 comp rx=0 tx=0).stop 2024-03-22T01:38:03.912 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.904+0000 7f5c56df3740 1 -- 172.21.15.72:0/169987448 shutdown_connections 2024-03-22T01:38:03.912 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.904+0000 7f5c56df3740 1 --2- 172.21.15.72:0/169987448 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561cf0faa9c0 0x561cf0facdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:03.912 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.904+0000 7f5c56df3740 1 --2- 172.21.15.72:0/169987448 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561cf0fa1bf0 0x561cf0faa480 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:03.912 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.904+0000 7f5c56df3740 1 --2- 172.21.15.72:0/169987448 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561cf0fa12d0 0x561cf0fa16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:03.912 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.904+0000 7f5c56df3740 1 -- 172.21.15.72:0/169987448 >> 172.21.15.72:0/169987448 conn(0x561cf0f9bbf0 msgr2=0x561cf0f9bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:38:03.912 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.904+0000 7f5c56df3740 1 -- 172.21.15.72:0/169987448 shutdown_connections 2024-03-22T01:38:03.912 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.904+0000 7f5c56df3740 1 -- 172.21.15.72:0/169987448 wait complete. 2024-03-22T01:38:03.912 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.904+0000 7f5c56df3740 1 -- start start 2024-03-22T01:38:03.912 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.904+0000 7f5c56df3740 1 Processor -- start 2024-03-22T01:38:03.912 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.905+0000 7f5c56df3740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561cf0fa12d0 0x561cf0faa290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:03.912 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.905+0000 7f5c56df3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561cf0fa1bf0 0x561cf0fa7620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:03.913 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.905+0000 7f5c56df3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561cf0faa9c0 0x561cf0fa7b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:03.913 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.905+0000 7f5c56df3740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x561cf0fad5f0 con 0x561cf0fa12d0 2024-03-22T01:38:03.913 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.905+0000 7f5c56df3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x561cf0ec0db0 con 0x561cf0faa9c0 2024-03-22T01:38:03.913 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.905+0000 7f5c56df3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561cf0fad770 con 0x561cf0fa1bf0 2024-03-22T01:38:03.913 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.905+0000 7f5c56df3740 10 client.0 _ll_register_callbacks cb 0x561cf0c18a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:38:03.913 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.905+0000 7f5c51faa640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561cf0fa12d0 0x561cf0faa290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:03.913 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.905+0000 7f5c51faa640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561cf0fa12d0 0x561cf0faa290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33500/0 (socket says 172.21.15.72:33500) 2024-03-22T01:38:03.913 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.905+0000 7f5c51faa640 1 -- 172.21.15.72:0/3419400147 learned_addr learned my addr 172.21.15.72:0/3419400147 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:38:03.913 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.905+0000 7f5c527ab640 1 --2- 172.21.15.72:0/3419400147 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561cf0faa9c0 0x561cf0fa7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:03.913 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.905+0000 7f5c517a9640 1 --2- 172.21.15.72:0/3419400147 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561cf0fa1bf0 0x561cf0fa7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:03.913 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.905+0000 7f5c51faa640 1 -- 172.21.15.72:0/3419400147 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561cf0fa1bf0 msgr2=0x561cf0fa7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:03.913 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.905+0000 7f5c51faa640 1 --2- 172.21.15.72:0/3419400147 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561cf0fa1bf0 0x561cf0fa7620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:03.913 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.905+0000 7f5c51faa640 1 -- 172.21.15.72:0/3419400147 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561cf0faa9c0 msgr2=0x561cf0fa7b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:03.914 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.905+0000 7f5c51faa640 1 --2- 172.21.15.72:0/3419400147 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561cf0faa9c0 0x561cf0fa7b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:03.914 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.905+0000 7f5c51faa640 1 -- 172.21.15.72:0/3419400147 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561cf0ce5620 con 0x561cf0fa12d0 2024-03-22T01:38:03.914 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.905+0000 7f5c517a9640 1 --2- 172.21.15.72:0/3419400147 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561cf0fa1bf0 0x561cf0fa7620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:38:03.914 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.906+0000 7f5c51faa640 1 --2- 172.21.15.72:0/3419400147 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561cf0fa12d0 0x561cf0faa290 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f5c4802c850 tx=0x7f5c480bd540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16fe00ba9fce8f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:38:03.914 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.906+0000 7f5c357fa640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:38:03.914 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.906+0000 7f5c357fa640 1 -- 172.21.15.72:0/3419400147 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c4804b040 con 0x561cf0fa12d0 2024-03-22T01:38:03.914 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.906+0000 7f5c357fa640 1 -- 172.21.15.72:0/3419400147 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f5c48005570 con 0x561cf0fa12d0 2024-03-22T01:38:03.914 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.906+0000 7f5c56df3740 1 -- 172.21.15.72:0/3419400147 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5c3c013950 con 0x561cf0fa12d0 2024-03-22T01:38:03.914 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.906+0000 7f5c357fa640 1 -- 172.21.15.72:0/3419400147 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f5c48004c80 con 0x561cf0fa12d0 2024-03-22T01:38:03.914 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.906+0000 7f5c56df3740 10 client.15891 fetch_fsmap learned FSMap version 21 2024-03-22T01:38:03.914 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.906+0000 7f5c56df3740 1 -- 172.21.15.72:0/3419400147 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x561cf0fb94d0 con 0x561cf0fa12d0 2024-03-22T01:38:03.914 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.906+0000 7f5c357fa640 1 -- 172.21.15.72:0/3419400147 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c3c013950 con 0x561cf0fa12d0 2024-03-22T01:38:03.915 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.906+0000 7f5c357fa640 1 -- 172.21.15.72:0/3419400147 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f5c480b9de0 con 0x561cf0fa12d0 2024-03-22T01:38:03.915 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.906+0000 7f5c56df3740 10 client.15891 fetch_fsmap finished waiting for FSMap version 21 2024-03-22T01:38:03.915 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.906+0000 7f5c56df3740 10 client.15891 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T01:38:03.915 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.906+0000 7f5c56df3740 20 client.15891 populate_metadata read hostname 'smithi072' 2024-03-22T01:38:03.915 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.906+0000 7f5c56df3740 1 --2- 172.21.15.72:0/3419400147 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x561cf0fba930 0x561cf0fbcdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:03.915 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.906+0000 7f5c56df3740 4 client.15891 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T01:38:03.915 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.906+0000 7f5c56df3740 1 -- 172.21.15.72:0/3419400147 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561cf0f12580 con 0x561cf0fba930 2024-03-22T01:38:03.915 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.906+0000 7f5c517a9640 1 --2- 172.21.15.72:0/3419400147 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x561cf0fba930 0x561cf0fbcdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:03.915 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.907+0000 7f5c517a9640 1 --2- 172.21.15.72:0/3419400147 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x561cf0fba930 0x561cf0fbcdc0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=abca89304cc1f547 server_cookie=9e6718558b4378e3 in_seq=0 out_seq=0 2024-03-22T01:38:03.915 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.907+0000 7f5c357fa640 10 client.15891 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:38:03.915 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.908+0000 7f5c357fa640 1 -- 172.21.15.72:0/3419400147 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561cf0f12580 con 0x561cf0fba930 2024-03-22T01:38:03.915 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.908+0000 7f5c357fa640 10 client.15891 handle_command_reply: tid=0 2024-03-22T01:38:03.916 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.908+0000 7f5c56df3740 1 client.15891 shutdown 2024-03-22T01:38:03.916 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.908+0000 7f5c56df3740 1 -- 172.21.15.72:0/3419400147 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561cf0fa12d0 msgr2=0x561cf0faa290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:03.916 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.908+0000 7f5c56df3740 1 --2- 172.21.15.72:0/3419400147 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561cf0fa12d0 0x561cf0faa290 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f5c4802c850 tx=0x7f5c480bd540 comp rx=0 tx=0).stop 2024-03-22T01:38:03.916 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.908+0000 7f5c56df3740 1 -- 172.21.15.72:0/3419400147 shutdown_connections 2024-03-22T01:38:03.916 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.908+0000 7f5c56df3740 1 --2- 172.21.15.72:0/3419400147 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x561cf0fba930 0x561cf0fbcdc0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:03.916 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.908+0000 7f5c56df3740 1 --2- 172.21.15.72:0/3419400147 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561cf0faa9c0 0x561cf0fa7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:03.916 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.908+0000 7f5c56df3740 1 --2- 172.21.15.72:0/3419400147 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561cf0fa1bf0 0x561cf0fa7620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:03.916 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.908+0000 7f5c357fa640 0 client.15891 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:38:03.916 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.908+0000 7f5c56df3740 1 --2- 172.21.15.72:0/3419400147 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561cf0fa12d0 0x561cf0faa290 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:03.916 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.908+0000 7f5c56df3740 1 -- 172.21.15.72:0/3419400147 >> 172.21.15.72:0/3419400147 conn(0x561cf0f9bbf0 msgr2=0x561cf0f9d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:38:03.916 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.908+0000 7f5c56df3740 1 -- 172.21.15.72:0/3419400147 shutdown_connections 2024-03-22T01:38:03.916 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.908+0000 7f5c56df3740 1 -- 172.21.15.72:0/3419400147 wait complete. 2024-03-22T01:38:03.916 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.908+0000 7f5c56df3740 20 client.15891 trim_cache size 0 max 16384 2024-03-22T01:38:03.927 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.923+0000 7f5c56df3740 1 Processor -- start 2024-03-22T01:38:03.927 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.923+0000 7f5c56df3740 1 -- start start 2024-03-22T01:38:03.927 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.923+0000 7f5c56df3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561cf0fa12d0 0x561cf0fb6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:03.927 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.923+0000 7f5c56df3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561cf0fa1bf0 0x561cf0f9aad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:03.927 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.923+0000 7f5c56df3740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561cf0faa9c0 0x561cf0fc2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:03.927 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.923+0000 7f5c56df3740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f5c4c1ad090 con 0x561cf0faa9c0 2024-03-22T01:38:03.927 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.923+0000 7f5c56df3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5c4c1af910 con 0x561cf0fa1bf0 2024-03-22T01:38:03.927 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.923+0000 7f5c56df3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5c4c1ad700 con 0x561cf0fa12d0 2024-03-22T01:38:03.928 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.923+0000 7f5c527ab640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561cf0faa9c0 0x561cf0fc2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:03.928 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.923+0000 7f5c527ab640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561cf0faa9c0 0x561cf0fc2670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33508/0 (socket says 172.21.15.72:33508) 2024-03-22T01:38:03.928 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.923+0000 7f5c527ab640 1 -- 172.21.15.72:0/4149897856 learned_addr learned my addr 172.21.15.72:0/4149897856 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:38:03.928 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.923+0000 7f5c517a9640 1 --2- 172.21.15.72:0/4149897856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561cf0fa1bf0 0x561cf0f9aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:03.928 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.923+0000 7f5c51faa640 1 --2- 172.21.15.72:0/4149897856 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561cf0fa12d0 0x561cf0fb6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:03.928 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.923+0000 7f5c527ab640 1 -- 172.21.15.72:0/4149897856 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561cf0fa12d0 msgr2=0x561cf0fb6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:03.928 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.923+0000 7f5c527ab640 1 --2- 172.21.15.72:0/4149897856 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561cf0fa12d0 0x561cf0fb6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:03.928 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.923+0000 7f5c527ab640 1 -- 172.21.15.72:0/4149897856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561cf0fa1bf0 msgr2=0x561cf0f9aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:03.928 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.923+0000 7f5c527ab640 1 --2- 172.21.15.72:0/4149897856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561cf0fa1bf0 0x561cf0f9aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:03.928 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.923+0000 7f5c527ab640 1 -- 172.21.15.72:0/4149897856 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561cf0f9ba20 con 0x561cf0faa9c0 2024-03-22T01:38:03.928 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.924+0000 7f5c527ab640 1 --2- 172.21.15.72:0/4149897856 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561cf0faa9c0 0x561cf0fc2670 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f5c44011d10 tx=0x7f5c4401e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43a793dbcf1b5bf1 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:38:03.928 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.924+0000 7f5c377fe640 1 -- 172.21.15.72:0/4149897856 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c44014110 con 0x561cf0faa9c0 2024-03-22T01:38:03.928 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.924+0000 7f5c377fe640 1 -- 172.21.15.72:0/4149897856 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c44004530 con 0x561cf0faa9c0 2024-03-22T01:38:03.928 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.924+0000 7f5c377fe640 1 -- 172.21.15.72:0/4149897856 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c44021d60 con 0x561cf0faa9c0 2024-03-22T01:38:03.929 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.924+0000 7f5c56df3740 1 -- 172.21.15.72:0/4149897856 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561cf0faa9c0 msgr2=0x561cf0fc2670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:03.929 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.924+0000 7f5c56df3740 1 --2- 172.21.15.72:0/4149897856 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561cf0faa9c0 0x561cf0fc2670 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f5c44011d10 tx=0x7f5c4401e000 comp rx=0 tx=0).stop 2024-03-22T01:38:03.929 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.924+0000 7f5c56df3740 1 -- 172.21.15.72:0/4149897856 shutdown_connections 2024-03-22T01:38:03.929 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.924+0000 7f5c56df3740 1 --2- 172.21.15.72:0/4149897856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561cf0fa1bf0 0x561cf0f9aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:03.929 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.924+0000 7f5c56df3740 1 --2- 172.21.15.72:0/4149897856 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561cf0faa9c0 0x561cf0fc2670 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:03.929 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.924+0000 7f5c56df3740 1 --2- 172.21.15.72:0/4149897856 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561cf0fa12d0 0x561cf0fb6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:03.929 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.924+0000 7f5c56df3740 1 -- 172.21.15.72:0/4149897856 >> 172.21.15.72:0/4149897856 conn(0x561cf0f9bbf0 msgr2=0x561cf0f9a3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:38:03.929 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.924+0000 7f5c56df3740 1 -- 172.21.15.72:0/4149897856 shutdown_connections 2024-03-22T01:38:03.929 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.924+0000 7f5c56df3740 1 -- 172.21.15.72:0/4149897856 wait complete. 2024-03-22T01:38:03.929 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.925+0000 7f5c56df3740 1 -- start start 2024-03-22T01:38:03.929 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.925+0000 7f5c56df3740 1 Processor -- start 2024-03-22T01:38:03.929 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.925+0000 7f5c56df3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561cf0fa12d0 0x561cf0fb2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:03.929 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.925+0000 7f5c56df3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561cf0fa1bf0 0x561cf0fadf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:03.929 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.925+0000 7f5c56df3740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561cf0faa9c0 0x561cf0fae460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:03.929 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.925+0000 7f5c56df3740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f5c4c1addd0 con 0x561cf0faa9c0 2024-03-22T01:38:03.930 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.925+0000 7f5c56df3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5c4c1af240 con 0x561cf0fa12d0 2024-03-22T01:38:03.930 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.925+0000 7f5c56df3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5c4c1ae4a0 con 0x561cf0fa1bf0 2024-03-22T01:38:03.930 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.925+0000 7f5c56df3740 10 client.0 _ll_register_callbacks cb 0x7f5c48009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:38:03.930 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.925+0000 7f5c527ab640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561cf0faa9c0 0x561cf0fae460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:03.930 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.925+0000 7f5c51faa640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561cf0fa12d0 0x561cf0fb2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:03.930 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.925+0000 7f5c527ab640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561cf0faa9c0 0x561cf0fae460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33510/0 (socket says 172.21.15.72:33510) 2024-03-22T01:38:03.930 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.925+0000 7f5c527ab640 1 -- 172.21.15.72:0/3156097560 learned_addr learned my addr 172.21.15.72:0/3156097560 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:38:03.930 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.925+0000 7f5c517a9640 1 --2- 172.21.15.72:0/3156097560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561cf0fa1bf0 0x561cf0fadf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:03.930 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.926+0000 7f5c527ab640 1 -- 172.21.15.72:0/3156097560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561cf0fa1bf0 msgr2=0x561cf0fadf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:03.930 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.926+0000 7f5c527ab640 1 --2- 172.21.15.72:0/3156097560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561cf0fa1bf0 0x561cf0fadf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:03.930 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.926+0000 7f5c527ab640 1 -- 172.21.15.72:0/3156097560 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561cf0fa12d0 msgr2=0x561cf0fb2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:03.930 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.926+0000 7f5c527ab640 1 --2- 172.21.15.72:0/3156097560 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561cf0fa12d0 0x561cf0fb2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:03.931 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.926+0000 7f5c527ab640 1 -- 172.21.15.72:0/3156097560 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561cf0faead0 con 0x561cf0faa9c0 2024-03-22T01:38:03.931 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.926+0000 7f5c51faa640 1 --2- 172.21.15.72:0/3156097560 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561cf0fa12d0 0x561cf0fb2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:38:03.931 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.926+0000 7f5c527ab640 1 --2- 172.21.15.72:0/3156097560 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561cf0faa9c0 0x561cf0fae460 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f5c44009db0 tx=0x7f5c44009ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ced76bdc067a2640 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:38:03.931 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.926+0000 7f5c357fa640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:38:03.931 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.926+0000 7f5c357fa640 1 -- 172.21.15.72:0/3156097560 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c440046e0 con 0x561cf0faa9c0 2024-03-22T01:38:03.931 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.926+0000 7f5c357fa640 1 -- 172.21.15.72:0/3156097560 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f5c44021ae0 con 0x561cf0faa9c0 2024-03-22T01:38:03.931 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.926+0000 7f5c56df3740 1 -- 172.21.15.72:0/3156097560 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5c44004530 con 0x561cf0faa9c0 2024-03-22T01:38:03.931 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.926+0000 7f5c357fa640 1 -- 172.21.15.72:0/3156097560 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f5c44021ce0 con 0x561cf0faa9c0 2024-03-22T01:38:03.931 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.926+0000 7f5c56df3740 10 client.15897 fetch_fsmap learned FSMap version 21 2024-03-22T01:38:03.931 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.926+0000 7f5c56df3740 1 -- 172.21.15.72:0/3156097560 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x561cf1020b60 con 0x561cf0faa9c0 2024-03-22T01:38:03.931 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.926+0000 7f5c357fa640 1 -- 172.21.15.72:0/3156097560 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c44004530 con 0x561cf0faa9c0 2024-03-22T01:38:03.931 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.926+0000 7f5c357fa640 1 -- 172.21.15.72:0/3156097560 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f5c4402a840 con 0x561cf0faa9c0 2024-03-22T01:38:03.931 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.927+0000 7f5c56df3740 10 client.15897 fetch_fsmap finished waiting for FSMap version 21 2024-03-22T01:38:03.931 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.927+0000 7f5c56df3740 10 client.15897 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T01:38:03.932 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.927+0000 7f5c56df3740 20 client.15897 populate_metadata read hostname 'smithi072' 2024-03-22T01:38:03.932 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.927+0000 7f5c56df3740 1 --2- 172.21.15.72:0/3156097560 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x561cf1021280 0x561cf10236a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:03.932 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.927+0000 7f5c56df3740 4 client.15897 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T01:38:03.932 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.927+0000 7f5c56df3740 1 -- 172.21.15.72:0/3156097560 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561cf0fa7ef0 con 0x561cf1021280 2024-03-22T01:38:03.932 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.927+0000 7f5c51faa640 1 --2- 172.21.15.72:0/3156097560 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x561cf1021280 0x561cf10236a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:03.932 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.927+0000 7f5c357fa640 10 client.15897 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:38:03.932 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.927+0000 7f5c51faa640 1 --2- 172.21.15.72:0/3156097560 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x561cf1021280 0x561cf10236a0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf9674acff35a66e server_cookie=396ae56bcc4d536a in_seq=0 out_seq=0 2024-03-22T01:38:03.933 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.932+0000 7f5c357fa640 1 -- 172.21.15.72:0/3156097560 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561cf0fa7ef0 con 0x561cf1021280 2024-03-22T01:38:03.933 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.932+0000 7f5c357fa640 10 client.15897 handle_command_reply: tid=0 2024-03-22T01:38:03.933 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.932+0000 7f5c56df3740 1 client.15897 shutdown 2024-03-22T01:38:03.933 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.932+0000 7f5c56df3740 1 -- 172.21.15.72:0/3156097560 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561cf0faa9c0 msgr2=0x561cf0fae460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:03.934 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.932+0000 7f5c56df3740 1 --2- 172.21.15.72:0/3156097560 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561cf0faa9c0 0x561cf0fae460 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f5c44009db0 tx=0x7f5c44009ee0 comp rx=0 tx=0).stop 2024-03-22T01:38:03.934 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.932+0000 7f5c56df3740 1 -- 172.21.15.72:0/3156097560 shutdown_connections 2024-03-22T01:38:03.934 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.932+0000 7f5c56df3740 1 --2- 172.21.15.72:0/3156097560 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x561cf1021280 0x561cf10236a0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:03.934 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.932+0000 7f5c56df3740 1 --2- 172.21.15.72:0/3156097560 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x561cf0faa9c0 0x561cf0fae460 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:03.934 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.932+0000 7f5c357fa640 0 client.15897 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:38:03.934 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.932+0000 7f5c56df3740 1 --2- 172.21.15.72:0/3156097560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561cf0fa1bf0 0x561cf0fadf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:03.934 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.932+0000 7f5c56df3740 1 --2- 172.21.15.72:0/3156097560 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561cf0fa12d0 0x561cf0fb2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:03.934 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.932+0000 7f5c56df3740 1 -- 172.21.15.72:0/3156097560 >> 172.21.15.72:0/3156097560 conn(0x561cf0f9bbf0 msgr2=0x561cf0f9ac50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:38:03.934 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.932+0000 7f5c56df3740 1 -- 172.21.15.72:0/3156097560 shutdown_connections 2024-03-22T01:38:03.934 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.932+0000 7f5c56df3740 1 -- 172.21.15.72:0/3156097560 wait complete. 2024-03-22T01:38:03.934 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.933+0000 7f5c56df3740 20 client.15897 trim_cache size 0 max 16384 2024-03-22T01:38:03.937 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout:{ 2024-03-22T01:38:03.938 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout: "message": "", 2024-03-22T01:38:03.938 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout: "return_code": 0 2024-03-22T01:38:03.938 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout:} 2024-03-22T01:38:03.939 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.933+0000 7f5c5420e640 1 -- 172.21.15.72:0/2617593020 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f5c380a31a0 msgr2=0x7f5c380a5660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:03.939 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.933+0000 7f5c5420e640 1 --2- 172.21.15.72:0/2617593020 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f5c380a31a0 0x7f5c380a5660 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5c44005fd0 tx=0x7f5c44009e70 comp rx=0 tx=0).stop 2024-03-22T01:38:03.939 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.933+0000 7f5c5420e640 1 -- 172.21.15.72:0/2617593020 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c4c069450 msgr2=0x7f5c4c198450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:03.939 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.933+0000 7f5c5420e640 1 --2- 172.21.15.72:0/2617593020 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c4c069450 0x7f5c4c198450 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f5c48005940 tx=0x7f5c480042f0 comp rx=0 tx=0).stop 2024-03-22T01:38:03.939 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.933+0000 7f5c5420e640 1 -- 172.21.15.72:0/2617593020 shutdown_connections 2024-03-22T01:38:03.939 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.933+0000 7f5c5420e640 1 --2- 172.21.15.72:0/2617593020 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f5c380a31a0 0x7f5c380a5660 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:03.939 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.933+0000 7f5c5420e640 1 --2- 172.21.15.72:0/2617593020 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c4c069e10 0x7f5c4c198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:03.939 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.933+0000 7f5c5420e640 1 --2- 172.21.15.72:0/2617593020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c4c107c10 0x7f5c4c19cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:03.939 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.933+0000 7f5c5420e640 1 --2- 172.21.15.72:0/2617593020 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c4c069450 0x7f5c4c198450 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:03.939 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.933+0000 7f5c5420e640 1 -- 172.21.15.72:0/2617593020 >> 172.21.15.72:0/2617593020 conn(0x7f5c4c077b80 msgr2=0x7f5c4c0ff6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:38:03.939 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.933+0000 7f5c5420e640 1 -- 172.21.15.72:0/2617593020 shutdown_connections 2024-03-22T01:38:03.939 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:03.933+0000 7f5c5420e640 1 -- 172.21.15.72:0/2617593020 wait complete. 2024-03-22T01:38:04.005 INFO:tasks.workunit.client.0.smithi072.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-22T01:38:04.099 INFO:tasks.workunit.client.0.smithi072.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-22T01:38:04.138 INFO:tasks.workunit.client.0.smithi072.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-22T01:38:04.142 INFO:tasks.workunit.client.0.smithi072.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-22T01:38:04.142 INFO:tasks.workunit.client.0.smithi072.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-22T01:38:04.178 INFO:tasks.workunit.client.0.smithi072.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-22T01:38:04.181 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-22T01:38:04.181 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-22T01:38:04.197 INFO:tasks.workunit:Running workunits matching suites/ffsb.sh on client.0... 2024-03-22T01:38:04.199 INFO:tasks.workunit:Running workunit suites/ffsb.sh... 2024-03-22T01:38:04.200 DEBUG:teuthology.orchestra.run.smithi072:workunit test suites/ffsb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2024-03-22T01:38:04.259 INFO:tasks.workunit.client.0.smithi072.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2024-03-22T01:38:04.260 INFO:tasks.workunit.client.0.smithi072.stderr:+ mydir=/home/ubuntu/cephtest/clone.client.0/qa/workunits/suites 2024-03-22T01:38:04.260 INFO:tasks.workunit.client.0.smithi072.stderr:+ trap -- retry EXIT 2024-03-22T01:38:04.260 INFO:tasks.workunit.client.0.smithi072.stderr:+ rm -rf ffsb 2024-03-22T01:38:04.260 INFO:tasks.workunit.client.0.smithi072.stderr:+ timeout 1800 git clone https://git.ceph.com/ffsb.git --depth 1 2024-03-22T01:38:04.263 INFO:tasks.workunit.client.0.smithi072.stderr:Cloning into 'ffsb'... 2024-03-22T01:38:04.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:38:03 smithi089 ceph-mon[35770]: pgmap v45: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-22T01:38:04.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:38:03 smithi072 ceph-mon[32414]: pgmap v45: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-22T01:38:04.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:38:03 smithi098 ceph-mon[36468]: pgmap v45: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-22T01:38:05.315 INFO:tasks.workunit.client.0.smithi072.stderr:+ trap - EXIT 2024-03-22T01:38:05.315 INFO:tasks.workunit.client.0.smithi072.stderr:+ cd ffsb 2024-03-22T01:38:05.315 INFO:tasks.workunit.client.0.smithi072.stderr:+ ./configure 2024-03-22T01:38:05.452 INFO:tasks.workunit.client.0.smithi072.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-03-22T01:38:06.357 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:38:05 smithi089 ceph-mon[35770]: pgmap v46: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 KiB/s wr, 18 op/s 2024-03-22T01:38:06.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:38:05 smithi072 ceph-mon[32414]: pgmap v46: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 KiB/s wr, 18 op/s 2024-03-22T01:38:06.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:38:05 smithi098 ceph-mon[36468]: pgmap v46: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 KiB/s wr, 18 op/s 2024-03-22T01:38:06.460 INFO:tasks.workunit.client.0.smithi072.stdout:checking whether build environment is sane... yes 2024-03-22T01:38:06.470 INFO:tasks.workunit.client.0.smithi072.stdout:checking for a thread-safe mkdir -p... /usr/bin/mkdir -p 2024-03-22T01:38:06.470 INFO:tasks.workunit.client.0.smithi072.stdout:checking for gawk... gawk 2024-03-22T01:38:06.485 INFO:tasks.workunit.client.0.smithi072.stdout:checking whether make sets $(MAKE)... yes 2024-03-22T01:38:06.518 INFO:tasks.workunit.client.0.smithi072.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-03-22T01:38:06.518 INFO:tasks.workunit.client.0.smithi072.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-03-22T01:38:06.528 INFO:tasks.workunit.client.0.smithi072.stdout:checking for style of include used by make... GNU 2024-03-22T01:38:06.533 INFO:tasks.workunit.client.0.smithi072.stdout:checking for gcc... gcc 2024-03-22T01:38:06.586 INFO:tasks.workunit.client.0.smithi072.stdout:checking for C compiler default output file name... a.out 2024-03-22T01:38:06.587 INFO:tasks.workunit.client.0.smithi072.stdout:checking whether the C compiler works... yes 2024-03-22T01:38:06.591 INFO:tasks.workunit.client.0.smithi072.stdout:checking whether we are cross compiling... no 2024-03-22T01:38:06.630 INFO:tasks.workunit.client.0.smithi072.stdout:checking for suffix of executables... 2024-03-22T01:38:06.665 INFO:tasks.workunit.client.0.smithi072.stdout:checking for suffix of object files... o 2024-03-22T01:38:06.695 INFO:tasks.workunit.client.0.smithi072.stdout:checking whether we are using the GNU C compiler... yes 2024-03-22T01:38:06.725 INFO:tasks.workunit.client.0.smithi072.stdout:checking whether gcc accepts -g... yes 2024-03-22T01:38:06.799 INFO:tasks.workunit.client.0.smithi072.stdout:checking for gcc option to accept ISO C89... none needed 2024-03-22T01:38:06.912 INFO:tasks.workunit.client.0.smithi072.stdout:checking dependency style of gcc... gcc3 2024-03-22T01:38:06.986 INFO:tasks.workunit.client.0.smithi072.stdout:checking how to run the C preprocessor... gcc -E 2024-03-22T01:38:07.027 INFO:tasks.workunit.client.0.smithi072.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-03-22T01:38:07.028 INFO:tasks.workunit.client.0.smithi072.stdout:checking for egrep... /usr/bin/grep -E 2024-03-22T01:38:07.037 INFO:tasks.workunit.client.0.smithi072.stdout:checking for AIX... no 2024-03-22T01:38:07.068 INFO:tasks.workunit.client.0.smithi072.stdout:checking for gcc... (cached) gcc 2024-03-22T01:38:07.076 INFO:tasks.workunit.client.0.smithi072.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-03-22T01:38:07.077 INFO:tasks.workunit.client.0.smithi072.stdout:checking whether gcc accepts -g... (cached) yes 2024-03-22T01:38:07.077 INFO:tasks.workunit.client.0.smithi072.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-03-22T01:38:07.077 INFO:tasks.workunit.client.0.smithi072.stdout:checking dependency style of gcc... (cached) gcc3 2024-03-22T01:38:07.139 INFO:tasks.workunit.client.0.smithi072.stdout:checking for main in -lm... yes 2024-03-22T01:38:07.188 INFO:tasks.workunit.client.0.smithi072.stdout:checking for main in -lpthread... yes 2024-03-22T01:38:07.348 INFO:tasks.workunit.client.0.smithi072.stdout:checking for ANSI C header files... yes 2024-03-22T01:38:07.418 INFO:tasks.workunit.client.0.smithi072.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2024-03-22T01:38:07.465 INFO:tasks.workunit.client.0.smithi072.stdout:checking for sys/types.h... yes 2024-03-22T01:38:07.514 INFO:tasks.workunit.client.0.smithi072.stdout:checking for sys/stat.h... yes 2024-03-22T01:38:07.560 INFO:tasks.workunit.client.0.smithi072.stdout:checking for stdlib.h... yes 2024-03-22T01:38:07.609 INFO:tasks.workunit.client.0.smithi072.stdout:checking for string.h... yes 2024-03-22T01:38:07.668 INFO:tasks.workunit.client.0.smithi072.stdout:checking for memory.h... yes 2024-03-22T01:38:07.714 INFO:tasks.workunit.client.0.smithi072.stdout:checking for strings.h... yes 2024-03-22T01:38:07.765 INFO:tasks.workunit.client.0.smithi072.stdout:checking for inttypes.h... yes 2024-03-22T01:38:07.815 INFO:tasks.workunit.client.0.smithi072.stdout:checking for stdint.h... yes 2024-03-22T01:38:07.863 INFO:tasks.workunit.client.0.smithi072.stdout:checking for unistd.h... yes 2024-03-22T01:38:07.921 INFO:tasks.workunit.client.0.smithi072.stdout:checking pthread.h usability... yes 2024-03-22T01:38:07.943 INFO:tasks.workunit.client.0.smithi072.stdout:checking pthread.h presence... yes 2024-03-22T01:38:07.944 INFO:tasks.workunit.client.0.smithi072.stdout:checking for pthread.h... yes 2024-03-22T01:38:07.994 INFO:tasks.workunit.client.0.smithi072.stdout:checking fcntl.h usability... yes 2024-03-22T01:38:08.015 INFO:tasks.workunit.client.0.smithi072.stdout:checking fcntl.h presence... yes 2024-03-22T01:38:08.015 INFO:tasks.workunit.client.0.smithi072.stdout:checking for fcntl.h... yes 2024-03-22T01:38:08.065 INFO:tasks.workunit.client.0.smithi072.stdout:checking limits.h usability... yes 2024-03-22T01:38:08.085 INFO:tasks.workunit.client.0.smithi072.stdout:checking limits.h presence... yes 2024-03-22T01:38:08.085 INFO:tasks.workunit.client.0.smithi072.stdout:checking for limits.h... yes 2024-03-22T01:38:08.099 INFO:tasks.workunit.client.0.smithi072.stdout:checking for stdint.h... (cached) yes 2024-03-22T01:38:08.147 INFO:tasks.workunit.client.0.smithi072.stdout:checking sys/time.h usability... yes 2024-03-22T01:38:08.168 INFO:tasks.workunit.client.0.smithi072.stdout:checking sys/time.h presence... yes 2024-03-22T01:38:08.168 INFO:tasks.workunit.client.0.smithi072.stdout:checking for sys/time.h... yes 2024-03-22T01:38:08.183 INFO:tasks.workunit.client.0.smithi072.stdout:checking for unistd.h... (cached) yes 2024-03-22T01:38:08.235 INFO:tasks.workunit.client.0.smithi072.stdout:checking sys/vfs.h usability... yes 2024-03-22T01:38:08.255 INFO:tasks.workunit.client.0.smithi072.stdout:checking sys/vfs.h presence... yes 2024-03-22T01:38:08.255 INFO:tasks.workunit.client.0.smithi072.stdout:checking for sys/vfs.h... yes 2024-03-22T01:38:08.301 INFO:tasks.workunit.client.0.smithi072.stdout:checking sys/limits.h usability... no 2024-03-22T01:38:08.322 INFO:tasks.workunit.client.0.smithi072.stdout:checking sys/limits.h presence... no 2024-03-22T01:38:08.322 INFO:tasks.workunit.client.0.smithi072.stdout:checking for sys/limits.h... no 2024-03-22T01:38:08.354 INFO:tasks.workunit.client.0.smithi072.stdout:checking for an ANSI C-conforming const... yes 2024-03-22T01:38:08.357 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:38:07 smithi089 ceph-mon[35770]: pgmap v47: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 23 KiB/s wr, 17 op/s 2024-03-22T01:38:08.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:38:07 smithi072 ceph-mon[32414]: pgmap v47: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 23 KiB/s wr, 17 op/s 2024-03-22T01:38:08.387 INFO:tasks.workunit.client.0.smithi072.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-03-22T01:38:08.388 INFO:tasks.workunit.client.0.smithi072.stdout:checking for function prototypes... yes 2024-03-22T01:38:08.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:38:07 smithi098 ceph-mon[36468]: pgmap v47: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 23 KiB/s wr, 17 op/s 2024-03-22T01:38:08.424 INFO:tasks.workunit.client.0.smithi072.stdout:checking whether setvbuf arguments are reversed... no 2024-03-22T01:38:08.520 INFO:tasks.workunit.client.0.smithi072.stdout:checking for system... yes 2024-03-22T01:38:08.587 INFO:tasks.workunit.client.0.smithi072.stdout:checking for gettimeofday... yes 2024-03-22T01:38:08.647 INFO:tasks.workunit.client.0.smithi072.stdout:checking for mkdir... yes 2024-03-22T01:38:08.708 INFO:tasks.workunit.client.0.smithi072.stdout:checking for strerror... yes 2024-03-22T01:38:08.772 INFO:tasks.workunit.client.0.smithi072.stdout:checking for open64... yes 2024-03-22T01:38:08.832 INFO:tasks.workunit.client.0.smithi072.stdout:checking for stat64... yes 2024-03-22T01:38:08.893 INFO:tasks.workunit.client.0.smithi072.stdout:checking for fseeko64... yes 2024-03-22T01:38:08.957 INFO:tasks.workunit.client.0.smithi072.stdout:checking for lrand48_r... yes 2024-03-22T01:38:09.025 INFO:tasks.workunit.client.0.smithi072.stdout:checking for srand48_r... yes 2024-03-22T01:38:09.048 INFO:tasks.workunit.client.0.smithi072.stdout:configure: creating ./config.status 2024-03-22T01:38:09.149 INFO:tasks.workunit.client.0.smithi072.stdout:config.status: creating Makefile 2024-03-22T01:38:09.163 INFO:tasks.workunit.client.0.smithi072.stdout:config.status: creating config.h 2024-03-22T01:38:09.182 INFO:tasks.workunit.client.0.smithi072.stdout:config.status: executing depfiles commands 2024-03-22T01:38:09.229 INFO:tasks.workunit.client.0.smithi072.stdout:config.status: executing default commands 2024-03-22T01:38:09.279 INFO:tasks.workunit.client.0.smithi072.stderr:+ make 2024-03-22T01:38:09.285 INFO:tasks.workunit.client.0.smithi072.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run aclocal-1.10 2024-03-22T01:38:09.287 INFO:tasks.workunit.client.0.smithi072.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing: line 46: aclocal-1.10: command not found 2024-03-22T01:38:09.288 INFO:tasks.workunit.client.0.smithi072.stderr:WARNING: `aclocal-1.10' is missing on your system. You should only need it if 2024-03-22T01:38:09.288 INFO:tasks.workunit.client.0.smithi072.stderr: you modified `acinclude.m4' or `configure.in'. You might want 2024-03-22T01:38:09.288 INFO:tasks.workunit.client.0.smithi072.stderr: to install the `Automake' and `Perl' packages. Grab them from 2024-03-22T01:38:09.288 INFO:tasks.workunit.client.0.smithi072.stderr: any GNU archive site. 2024-03-22T01:38:09.290 INFO:tasks.workunit.client.0.smithi072.stdout: cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run automake-1.10 --gnu 2024-03-22T01:38:09.291 INFO:tasks.workunit.client.0.smithi072.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing: line 46: automake-1.10: command not found 2024-03-22T01:38:09.292 INFO:tasks.workunit.client.0.smithi072.stderr:WARNING: `automake-1.10' is missing on your system. You should only need it if 2024-03-22T01:38:09.292 INFO:tasks.workunit.client.0.smithi072.stderr: you modified `Makefile.am', `acinclude.m4' or `configure.in'. 2024-03-22T01:38:09.292 INFO:tasks.workunit.client.0.smithi072.stderr: You might want to install the `Automake' and `Perl' packages. 2024-03-22T01:38:09.292 INFO:tasks.workunit.client.0.smithi072.stderr: Grab them from any GNU archive site. 2024-03-22T01:38:09.343 INFO:tasks.workunit.client.0.smithi072.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run autoconf 2024-03-22T01:38:09.755 INFO:tasks.workunit.client.0.smithi072.stderr:aclocal.m4:16: warning: this file was generated for autoconf 2.61. 2024-03-22T01:38:09.755 INFO:tasks.workunit.client.0.smithi072.stderr:You have another version of autoconf. It may work, but is not guaranteed to. 2024-03-22T01:38:09.755 INFO:tasks.workunit.client.0.smithi072.stderr:If you have problems, you may need to regenerate the build system entirely. 2024-03-22T01:38:09.755 INFO:tasks.workunit.client.0.smithi072.stderr:To do so, use the procedure documented by the package, typically `autoreconf'. 2024-03-22T01:38:09.926 INFO:tasks.workunit.client.0.smithi072.stdout:/bin/sh ./config.status --recheck 2024-03-22T01:38:09.979 INFO:tasks.workunit.client.0.smithi072.stdout:running CONFIG_SHELL=/bin/sh /bin/sh ./configure --no-create --no-recursion 2024-03-22T01:38:10.097 INFO:tasks.workunit.client.0.smithi072.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-03-22T01:38:10.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:38:09 smithi089 ceph-mon[35770]: pgmap v48: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 23 KiB/s wr, 17 op/s 2024-03-22T01:38:10.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:38:09 smithi089 ceph-mon[35770]: mgrmap e25: x(active, since 90s), standbys: y, z 2024-03-22T01:38:10.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:38:09 smithi072 ceph-mon[32414]: pgmap v48: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 23 KiB/s wr, 17 op/s 2024-03-22T01:38:10.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:38:09 smithi072 ceph-mon[32414]: mgrmap e25: x(active, since 90s), standbys: y, z 2024-03-22T01:38:10.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:38:09 smithi098 ceph-mon[36468]: pgmap v48: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 23 KiB/s wr, 17 op/s 2024-03-22T01:38:10.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:38:09 smithi098 ceph-mon[36468]: mgrmap e25: x(active, since 90s), standbys: y, z 2024-03-22T01:38:11.104 INFO:tasks.workunit.client.0.smithi072.stdout:checking whether build environment is sane... yes 2024-03-22T01:38:11.108 INFO:tasks.workunit.client.0.smithi072.stdout:checking for a thread-safe mkdir -p... /usr/bin/mkdir -p 2024-03-22T01:38:11.108 INFO:tasks.workunit.client.0.smithi072.stdout:checking for gawk... gawk 2024-03-22T01:38:11.143 INFO:tasks.workunit.client.0.smithi072.stdout:checking whether make sets $(MAKE)... yes 2024-03-22T01:38:11.173 INFO:tasks.workunit.client.0.smithi072.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-03-22T01:38:11.173 INFO:tasks.workunit.client.0.smithi072.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-03-22T01:38:11.187 INFO:tasks.workunit.client.0.smithi072.stdout:checking for style of include used by make... GNU 2024-03-22T01:38:11.191 INFO:tasks.workunit.client.0.smithi072.stdout:checking for gcc... gcc 2024-03-22T01:38:11.275 INFO:tasks.workunit.client.0.smithi072.stdout:checking whether the C compiler works... yes 2024-03-22T01:38:11.275 INFO:tasks.workunit.client.0.smithi072.stdout:checking for C compiler default output file name... a.out 2024-03-22T01:38:11.325 INFO:tasks.workunit.client.0.smithi072.stdout:checking for suffix of executables... 2024-03-22T01:38:11.362 INFO:tasks.workunit.client.0.smithi072.stdout:checking whether we are cross compiling... no 2024-03-22T01:38:11.397 INFO:tasks.workunit.client.0.smithi072.stdout:checking for suffix of object files... o 2024-03-22T01:38:11.421 INFO:tasks.workunit.client.0.smithi072.stdout:checking whether we are using the GNU C compiler... yes 2024-03-22T01:38:11.446 INFO:tasks.workunit.client.0.smithi072.stdout:checking whether gcc accepts -g... yes 2024-03-22T01:38:11.495 INFO:tasks.workunit.client.0.smithi072.stdout:checking for gcc option to accept ISO C89... none needed 2024-03-22T01:38:11.596 INFO:tasks.workunit.client.0.smithi072.stdout:checking dependency style of gcc... gcc3 2024-03-22T01:38:11.642 INFO:tasks.workunit.client.0.smithi072.stdout:checking how to run the C preprocessor... gcc -E 2024-03-22T01:38:11.696 INFO:tasks.workunit.client.0.smithi072.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-03-22T01:38:11.697 INFO:tasks.workunit.client.0.smithi072.stdout:checking for egrep... /usr/bin/grep -E 2024-03-22T01:38:11.841 INFO:tasks.workunit.client.0.smithi072.stdout:checking for ANSI C header files... yes 2024-03-22T01:38:11.881 INFO:tasks.workunit.client.0.smithi072.stdout:checking for sys/types.h... yes 2024-03-22T01:38:11.924 INFO:tasks.workunit.client.0.smithi072.stdout:checking for sys/stat.h... yes 2024-03-22T01:38:11.970 INFO:tasks.workunit.client.0.smithi072.stdout:checking for stdlib.h... yes 2024-03-22T01:38:12.016 INFO:tasks.workunit.client.0.smithi072.stdout:checking for string.h... yes 2024-03-22T01:38:12.055 INFO:tasks.workunit.client.0.smithi072.stdout:checking for memory.h... yes 2024-03-22T01:38:12.098 INFO:tasks.workunit.client.0.smithi072.stdout:checking for strings.h... yes 2024-03-22T01:38:12.141 INFO:tasks.workunit.client.0.smithi072.stdout:checking for inttypes.h... yes 2024-03-22T01:38:12.184 INFO:tasks.workunit.client.0.smithi072.stdout:checking for stdint.h... yes 2024-03-22T01:38:12.227 INFO:tasks.workunit.client.0.smithi072.stdout:checking for unistd.h... yes 2024-03-22T01:38:12.275 INFO:tasks.workunit.client.0.smithi072.stdout:checking minix/config.h usability... no 2024-03-22T01:38:12.307 INFO:tasks.workunit.client.0.smithi072.stdout:checking minix/config.h presence... no 2024-03-22T01:38:12.308 INFO:tasks.workunit.client.0.smithi072.stdout:checking for minix/config.h... no 2024-03-22T01:38:12.347 INFO:tasks.workunit.client.0.smithi072.stdout:checking whether it is safe to define __EXTENSIONS__... yes 2024-03-22T01:38:12.347 INFO:tasks.workunit.client.0.smithi072.stdout:checking for gcc... (cached) gcc 2024-03-22T01:38:12.357 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:38:12 smithi089 ceph-mon[35770]: pgmap v49: 129 pgs: 129 active+clean; 4.2 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 275 KiB/s wr, 28 op/s 2024-03-22T01:38:12.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:38:12 smithi072 ceph-mon[32414]: pgmap v49: 129 pgs: 129 active+clean; 4.2 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 275 KiB/s wr, 28 op/s 2024-03-22T01:38:12.386 INFO:tasks.workunit.client.0.smithi072.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-03-22T01:38:12.386 INFO:tasks.workunit.client.0.smithi072.stdout:checking whether gcc accepts -g... (cached) yes 2024-03-22T01:38:12.387 INFO:tasks.workunit.client.0.smithi072.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-03-22T01:38:12.387 INFO:tasks.workunit.client.0.smithi072.stdout:checking dependency style of gcc... (cached) gcc3 2024-03-22T01:38:12.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:38:12 smithi098 ceph-mon[36468]: pgmap v49: 129 pgs: 129 active+clean; 4.2 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 275 KiB/s wr, 28 op/s 2024-03-22T01:38:12.429 INFO:tasks.workunit.client.0.smithi072.stdout:checking for main in -lm... yes 2024-03-22T01:38:12.475 INFO:tasks.workunit.client.0.smithi072.stdout:checking for main in -lpthread... yes 2024-03-22T01:38:12.477 INFO:tasks.workunit.client.0.smithi072.stdout:checking for ANSI C header files... (cached) yes 2024-03-22T01:38:12.523 INFO:tasks.workunit.client.0.smithi072.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2024-03-22T01:38:12.601 INFO:tasks.workunit.client.0.smithi072.stdout:checking pthread.h usability... yes 2024-03-22T01:38:12.624 INFO:tasks.workunit.client.0.smithi072.stdout:checking pthread.h presence... yes 2024-03-22T01:38:12.624 INFO:tasks.workunit.client.0.smithi072.stdout:checking for pthread.h... yes 2024-03-22T01:38:12.675 INFO:tasks.workunit.client.0.smithi072.stdout:checking fcntl.h usability... yes 2024-03-22T01:38:12.696 INFO:tasks.workunit.client.0.smithi072.stdout:checking fcntl.h presence... yes 2024-03-22T01:38:12.696 INFO:tasks.workunit.client.0.smithi072.stdout:checking for fcntl.h... yes 2024-03-22T01:38:12.761 INFO:tasks.workunit.client.0.smithi072.stdout:checking limits.h usability... yes 2024-03-22T01:38:12.783 INFO:tasks.workunit.client.0.smithi072.stdout:checking limits.h presence... yes 2024-03-22T01:38:12.783 INFO:tasks.workunit.client.0.smithi072.stdout:checking for limits.h... yes 2024-03-22T01:38:12.797 INFO:tasks.workunit.client.0.smithi072.stdout:checking for stdint.h... (cached) yes 2024-03-22T01:38:12.837 INFO:tasks.workunit.client.0.smithi072.stdout:checking sys/time.h usability... yes 2024-03-22T01:38:12.858 INFO:tasks.workunit.client.0.smithi072.stdout:checking sys/time.h presence... yes 2024-03-22T01:38:12.858 INFO:tasks.workunit.client.0.smithi072.stdout:checking for sys/time.h... yes 2024-03-22T01:38:12.872 INFO:tasks.workunit.client.0.smithi072.stdout:checking for unistd.h... (cached) yes 2024-03-22T01:38:12.912 INFO:tasks.workunit.client.0.smithi072.stdout:checking sys/vfs.h usability... yes 2024-03-22T01:38:12.933 INFO:tasks.workunit.client.0.smithi072.stdout:checking sys/vfs.h presence... yes 2024-03-22T01:38:12.933 INFO:tasks.workunit.client.0.smithi072.stdout:checking for sys/vfs.h... yes 2024-03-22T01:38:12.986 INFO:tasks.workunit.client.0.smithi072.stdout:checking sys/limits.h usability... no 2024-03-22T01:38:13.017 INFO:tasks.workunit.client.0.smithi072.stdout:checking sys/limits.h presence... no 2024-03-22T01:38:13.017 INFO:tasks.workunit.client.0.smithi072.stdout:checking for sys/limits.h... no 2024-03-22T01:38:13.047 INFO:tasks.workunit.client.0.smithi072.stdout:checking for an ANSI C-conforming const... yes 2024-03-22T01:38:13.080 INFO:tasks.workunit.client.0.smithi072.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-03-22T01:38:13.141 INFO:tasks.workunit.client.0.smithi072.stdout:checking for system... yes 2024-03-22T01:38:13.200 INFO:tasks.workunit.client.0.smithi072.stdout:checking for gettimeofday... yes 2024-03-22T01:38:13.260 INFO:tasks.workunit.client.0.smithi072.stdout:checking for mkdir... yes 2024-03-22T01:38:13.317 INFO:tasks.workunit.client.0.smithi072.stdout:checking for strerror... yes 2024-03-22T01:38:13.374 INFO:tasks.workunit.client.0.smithi072.stdout:checking for open64... yes 2024-03-22T01:38:13.431 INFO:tasks.workunit.client.0.smithi072.stdout:checking for stat64... yes 2024-03-22T01:38:13.492 INFO:tasks.workunit.client.0.smithi072.stdout:checking for fseeko64... yes 2024-03-22T01:38:13.551 INFO:tasks.workunit.client.0.smithi072.stdout:checking for lrand48_r... yes 2024-03-22T01:38:13.607 INFO:tasks.workunit.client.0.smithi072.stdout:checking for srand48_r... yes 2024-03-22T01:38:13.633 INFO:tasks.workunit.client.0.smithi072.stdout:configure: creating ./config.status 2024-03-22T01:38:13.697 INFO:tasks.workunit.client.0.smithi072.stdout: /bin/sh ./config.status 2024-03-22T01:38:13.751 INFO:tasks.workunit.client.0.smithi072.stdout:config.status: creating Makefile 2024-03-22T01:38:13.764 INFO:tasks.workunit.client.0.smithi072.stdout:config.status: creating config.h 2024-03-22T01:38:13.787 INFO:tasks.workunit.client.0.smithi072.stdout:config.status: executing depfiles commands 2024-03-22T01:38:13.798 INFO:tasks.workunit.client.0.smithi072.stdout:config.status: executing default commands 2024-03-22T01:38:13.828 INFO:tasks.workunit.client.0.smithi072.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run autoheader 2024-03-22T01:38:13.973 INFO:tasks.workunit.client.0.smithi072.stdout:rm -f stamp-h1 2024-03-22T01:38:13.975 INFO:tasks.workunit.client.0.smithi072.stdout:touch config.h.in 2024-03-22T01:38:13.976 INFO:tasks.workunit.client.0.smithi072.stdout:cd . && /bin/sh ./config.status config.h 2024-03-22T01:38:14.019 INFO:tasks.workunit.client.0.smithi072.stdout:config.status: creating config.h 2024-03-22T01:38:14.048 INFO:tasks.workunit.client.0.smithi072.stdout:make all-am 2024-03-22T01:38:14.055 INFO:tasks.workunit.client.0.smithi072.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb' 2024-03-22T01:38:14.055 INFO:tasks.workunit.client.0.smithi072.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fileops.o -MD -MP -MF .deps/fileops.Tpo -c -o fileops.o fileops.c 2024-03-22T01:38:14.221 INFO:tasks.workunit.client.0.smithi072.stdout:mv -f .deps/fileops.Tpo .deps/fileops.Po 2024-03-22T01:38:14.224 INFO:tasks.workunit.client.0.smithi072.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rand.o -MD -MP -MF .deps/rand.Tpo -c -o rand.o rand.c 2024-03-22T01:38:14.327 INFO:tasks.workunit.client.0.smithi072.stdout:mv -f .deps/rand.Tpo .deps/rand.Po 2024-03-22T01:38:14.330 INFO:tasks.workunit.client.0.smithi072.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT main.o -MD -MP -MF .deps/main.Tpo -c -o main.o main.c 2024-03-22T01:38:14.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:38:14 smithi089 ceph-mon[35770]: pgmap v50: 129 pgs: 129 active+clean; 4.2 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 275 KiB/s wr, 27 op/s 2024-03-22T01:38:14.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:38:14 smithi072 ceph-mon[32414]: pgmap v50: 129 pgs: 129 active+clean; 4.2 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 275 KiB/s wr, 27 op/s 2024-03-22T01:38:14.421 INFO:tasks.workunit.client.0.smithi072.stdout:mv -f .deps/main.Tpo .deps/main.Po 2024-03-22T01:38:14.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:38:14 smithi098 ceph-mon[36468]: pgmap v50: 129 pgs: 129 active+clean; 4.2 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 275 KiB/s wr, 27 op/s 2024-03-22T01:38:14.424 INFO:tasks.workunit.client.0.smithi072.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fh.o -MD -MP -MF .deps/fh.Tpo -c -o fh.o fh.c 2024-03-22T01:38:14.511 INFO:tasks.workunit.client.0.smithi072.stdout:mv -f .deps/fh.Tpo .deps/fh.Po 2024-03-22T01:38:14.514 INFO:tasks.workunit.client.0.smithi072.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT filelist.o -MD -MP -MF .deps/filelist.Tpo -c -o filelist.o filelist.c 2024-03-22T01:38:14.637 INFO:tasks.workunit.client.0.smithi072.stdout:mv -f .deps/filelist.Tpo .deps/filelist.Po 2024-03-22T01:38:14.640 INFO:tasks.workunit.client.0.smithi072.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT metaops.o -MD -MP -MF .deps/metaops.Tpo -c -o metaops.o metaops.c 2024-03-22T01:38:14.687 INFO:tasks.workunit.client.0.smithi072.stdout:mv -f .deps/metaops.Tpo .deps/metaops.Po 2024-03-22T01:38:14.689 INFO:tasks.workunit.client.0.smithi072.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rwlock.o -MD -MP -MF .deps/rwlock.Tpo -c -o rwlock.o rwlock.c 2024-03-22T01:38:14.730 INFO:tasks.workunit.client.0.smithi072.stdout:mv -f .deps/rwlock.Tpo .deps/rwlock.Po 2024-03-22T01:38:14.733 INFO:tasks.workunit.client.0.smithi072.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT cirlist.o -MD -MP -MF .deps/cirlist.Tpo -c -o cirlist.o cirlist.c 2024-03-22T01:38:14.778 INFO:tasks.workunit.client.0.smithi072.stdout:mv -f .deps/cirlist.Tpo .deps/cirlist.Po 2024-03-22T01:38:14.781 INFO:tasks.workunit.client.0.smithi072.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rbt.o -MD -MP -MF .deps/rbt.Tpo -c -o rbt.o rbt.c 2024-03-22T01:38:14.978 INFO:tasks.workunit.client.0.smithi072.stdout:mv -f .deps/rbt.Tpo .deps/rbt.Po 2024-03-22T01:38:14.981 INFO:tasks.workunit.client.0.smithi072.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_tg.o -MD -MP -MF .deps/ffsb_tg.Tpo -c -o ffsb_tg.o ffsb_tg.c 2024-03-22T01:38:15.108 INFO:tasks.workunit.client.0.smithi072.stdout:mv -f .deps/ffsb_tg.Tpo .deps/ffsb_tg.Po 2024-03-22T01:38:15.110 INFO:tasks.workunit.client.0.smithi072.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fs.o -MD -MP -MF .deps/ffsb_fs.Tpo -c -o ffsb_fs.o ffsb_fs.c 2024-03-22T01:38:15.255 INFO:tasks.workunit.client.0.smithi072.stdout:mv -f .deps/ffsb_fs.Tpo .deps/ffsb_fs.Po 2024-03-22T01:38:15.258 INFO:tasks.workunit.client.0.smithi072.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_thread.o -MD -MP -MF .deps/ffsb_thread.Tpo -c -o ffsb_thread.o ffsb_thread.c 2024-03-22T01:38:15.324 INFO:tasks.workunit.client.0.smithi072.stdout:mv -f .deps/ffsb_thread.Tpo .deps/ffsb_thread.Po 2024-03-22T01:38:15.327 INFO:tasks.workunit.client.0.smithi072.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_op.o -MD -MP -MF .deps/ffsb_op.Tpo -c -o ffsb_op.o ffsb_op.c 2024-03-22T01:38:15.394 INFO:tasks.workunit.client.0.smithi072.stdout:mv -f .deps/ffsb_op.Tpo .deps/ffsb_op.Po 2024-03-22T01:38:15.397 INFO:tasks.workunit.client.0.smithi072.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT util.o -MD -MP -MF .deps/util.Tpo -c -o util.o util.c 2024-03-22T01:38:15.497 INFO:tasks.workunit.client.0.smithi072.stdout:mv -f .deps/util.Tpo .deps/util.Po 2024-03-22T01:38:15.501 INFO:tasks.workunit.client.0.smithi072.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT parser.o -MD -MP -MF .deps/parser.Tpo -c -o parser.o parser.c 2024-03-22T01:38:15.528 INFO:tasks.workunit.client.0.smithi072.stderr:In file included from parser.h:22, 2024-03-22T01:38:15.528 INFO:tasks.workunit.client.0.smithi072.stderr: from parser.c:25: 2024-03-22T01:38:15.528 INFO:tasks.workunit.client.0.smithi072.stderr:parser.c: In function ‘init_filesys’: 2024-03-22T01:38:15.528 INFO:tasks.workunit.client.0.smithi072.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-22T01:38:15.528 INFO:tasks.workunit.client.0.smithi072.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-22T01:38:15.529 INFO:tasks.workunit.client.0.smithi072.stderr: | ^ 2024-03-22T01:38:15.529 INFO:tasks.workunit.client.0.smithi072.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-22T01:38:15.529 INFO:tasks.workunit.client.0.smithi072.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-22T01:38:15.529 INFO:tasks.workunit.client.0.smithi072.stderr: | ^~~~~~~~ 2024-03-22T01:38:15.529 INFO:tasks.workunit.client.0.smithi072.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-22T01:38:15.529 INFO:tasks.workunit.client.0.smithi072.stderr: 36 | container_of(ptr, type, member) 2024-03-22T01:38:15.529 INFO:tasks.workunit.client.0.smithi072.stderr: | ^~~~~~~~~~~~ 2024-03-22T01:38:15.529 INFO:tasks.workunit.client.0.smithi072.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-03-22T01:38:15.529 INFO:tasks.workunit.client.0.smithi072.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-03-22T01:38:15.529 INFO:tasks.workunit.client.0.smithi072.stderr: | ^~~~~~~~~~ 2024-03-22T01:38:15.529 INFO:tasks.workunit.client.0.smithi072.stderr:parser.c:934:17: note: in expansion of macro ‘list_for_each_entry’ 2024-03-22T01:38:15.529 INFO:tasks.workunit.client.0.smithi072.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2024-03-22T01:38:15.529 INFO:tasks.workunit.client.0.smithi072.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-22T01:38:15.529 INFO:tasks.workunit.client.0.smithi072.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-22T01:38:15.530 INFO:tasks.workunit.client.0.smithi072.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-22T01:38:15.530 INFO:tasks.workunit.client.0.smithi072.stderr: | ^ 2024-03-22T01:38:15.530 INFO:tasks.workunit.client.0.smithi072.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-22T01:38:15.530 INFO:tasks.workunit.client.0.smithi072.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-22T01:38:15.530 INFO:tasks.workunit.client.0.smithi072.stderr: | ^~~~~~~~ 2024-03-22T01:38:15.530 INFO:tasks.workunit.client.0.smithi072.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-22T01:38:15.530 INFO:tasks.workunit.client.0.smithi072.stderr: 36 | container_of(ptr, type, member) 2024-03-22T01:38:15.530 INFO:tasks.workunit.client.0.smithi072.stderr: | ^~~~~~~~~~~~ 2024-03-22T01:38:15.530 INFO:tasks.workunit.client.0.smithi072.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-03-22T01:38:15.530 INFO:tasks.workunit.client.0.smithi072.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-03-22T01:38:15.530 INFO:tasks.workunit.client.0.smithi072.stderr: | ^~~~~~~~~~ 2024-03-22T01:38:15.530 INFO:tasks.workunit.client.0.smithi072.stderr:parser.c:934:17: note: in expansion of macro ‘list_for_each_entry’ 2024-03-22T01:38:15.531 INFO:tasks.workunit.client.0.smithi072.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2024-03-22T01:38:15.531 INFO:tasks.workunit.client.0.smithi072.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-22T01:38:15.531 INFO:tasks.workunit.client.0.smithi072.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-22T01:38:15.531 INFO:tasks.workunit.client.0.smithi072.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-22T01:38:15.531 INFO:tasks.workunit.client.0.smithi072.stderr: | ^ 2024-03-22T01:38:15.531 INFO:tasks.workunit.client.0.smithi072.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-22T01:38:15.531 INFO:tasks.workunit.client.0.smithi072.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-22T01:38:15.531 INFO:tasks.workunit.client.0.smithi072.stderr: | ^~~~~~~~ 2024-03-22T01:38:15.531 INFO:tasks.workunit.client.0.smithi072.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-22T01:38:15.531 INFO:tasks.workunit.client.0.smithi072.stderr: 36 | container_of(ptr, type, member) 2024-03-22T01:38:15.531 INFO:tasks.workunit.client.0.smithi072.stderr: | ^~~~~~~~~~~~ 2024-03-22T01:38:15.532 INFO:tasks.workunit.client.0.smithi072.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-03-22T01:38:15.532 INFO:tasks.workunit.client.0.smithi072.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-03-22T01:38:15.532 INFO:tasks.workunit.client.0.smithi072.stderr: | ^~~~~~~~~~ 2024-03-22T01:38:15.532 INFO:tasks.workunit.client.0.smithi072.stderr:parser.c:941:17: note: in expansion of macro ‘list_for_each_entry’ 2024-03-22T01:38:15.532 INFO:tasks.workunit.client.0.smithi072.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2024-03-22T01:38:15.532 INFO:tasks.workunit.client.0.smithi072.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-22T01:38:15.532 INFO:tasks.workunit.client.0.smithi072.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-22T01:38:15.532 INFO:tasks.workunit.client.0.smithi072.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-22T01:38:15.532 INFO:tasks.workunit.client.0.smithi072.stderr: | ^ 2024-03-22T01:38:15.532 INFO:tasks.workunit.client.0.smithi072.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-22T01:38:15.532 INFO:tasks.workunit.client.0.smithi072.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-22T01:38:15.532 INFO:tasks.workunit.client.0.smithi072.stderr: | ^~~~~~~~ 2024-03-22T01:38:15.532 INFO:tasks.workunit.client.0.smithi072.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-22T01:38:15.533 INFO:tasks.workunit.client.0.smithi072.stderr: 36 | container_of(ptr, type, member) 2024-03-22T01:38:15.533 INFO:tasks.workunit.client.0.smithi072.stderr: | ^~~~~~~~~~~~ 2024-03-22T01:38:15.533 INFO:tasks.workunit.client.0.smithi072.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-03-22T01:38:15.533 INFO:tasks.workunit.client.0.smithi072.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-03-22T01:38:15.533 INFO:tasks.workunit.client.0.smithi072.stderr: | ^~~~~~~~~~ 2024-03-22T01:38:15.533 INFO:tasks.workunit.client.0.smithi072.stderr:parser.c:941:17: note: in expansion of macro ‘list_for_each_entry’ 2024-03-22T01:38:15.533 INFO:tasks.workunit.client.0.smithi072.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2024-03-22T01:38:15.533 INFO:tasks.workunit.client.0.smithi072.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-22T01:38:15.533 INFO:tasks.workunit.client.0.smithi072.stderr:parser.c: In function ‘init_tg_stats’: 2024-03-22T01:38:15.533 INFO:tasks.workunit.client.0.smithi072.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-22T01:38:15.533 INFO:tasks.workunit.client.0.smithi072.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-22T01:38:15.533 INFO:tasks.workunit.client.0.smithi072.stderr: | ^ 2024-03-22T01:38:15.533 INFO:tasks.workunit.client.0.smithi072.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-22T01:38:15.533 INFO:tasks.workunit.client.0.smithi072.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-22T01:38:15.533 INFO:tasks.workunit.client.0.smithi072.stderr: | ^~~~~~~~ 2024-03-22T01:38:15.534 INFO:tasks.workunit.client.0.smithi072.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-22T01:38:15.534 INFO:tasks.workunit.client.0.smithi072.stderr: 36 | container_of(ptr, type, member) 2024-03-22T01:38:15.534 INFO:tasks.workunit.client.0.smithi072.stderr: | ^~~~~~~~~~~~ 2024-03-22T01:38:15.534 INFO:tasks.workunit.client.0.smithi072.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-03-22T01:38:15.534 INFO:tasks.workunit.client.0.smithi072.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-03-22T01:38:15.534 INFO:tasks.workunit.client.0.smithi072.stderr: | ^~~~~~~~~~ 2024-03-22T01:38:15.534 INFO:tasks.workunit.client.0.smithi072.stderr:parser.c:971:41: note: in expansion of macro ‘list_for_each_entry’ 2024-03-22T01:38:15.534 INFO:tasks.workunit.client.0.smithi072.stderr: 971 | list_for_each_entry(tmp_list, 2024-03-22T01:38:15.534 INFO:tasks.workunit.client.0.smithi072.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-22T01:38:15.534 INFO:tasks.workunit.client.0.smithi072.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-22T01:38:15.534 INFO:tasks.workunit.client.0.smithi072.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-22T01:38:15.534 INFO:tasks.workunit.client.0.smithi072.stderr: | ^ 2024-03-22T01:38:15.534 INFO:tasks.workunit.client.0.smithi072.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-22T01:38:15.534 INFO:tasks.workunit.client.0.smithi072.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-22T01:38:15.535 INFO:tasks.workunit.client.0.smithi072.stderr: | ^~~~~~~~ 2024-03-22T01:38:15.535 INFO:tasks.workunit.client.0.smithi072.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-22T01:38:15.535 INFO:tasks.workunit.client.0.smithi072.stderr: 36 | container_of(ptr, type, member) 2024-03-22T01:38:15.535 INFO:tasks.workunit.client.0.smithi072.stderr: | ^~~~~~~~~~~~ 2024-03-22T01:38:15.535 INFO:tasks.workunit.client.0.smithi072.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-03-22T01:38:15.535 INFO:tasks.workunit.client.0.smithi072.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-03-22T01:38:15.535 INFO:tasks.workunit.client.0.smithi072.stderr: | ^~~~~~~~~~ 2024-03-22T01:38:15.535 INFO:tasks.workunit.client.0.smithi072.stderr:parser.c:971:41: note: in expansion of macro ‘list_for_each_entry’ 2024-03-22T01:38:15.535 INFO:tasks.workunit.client.0.smithi072.stderr: 971 | list_for_each_entry(tmp_list, 2024-03-22T01:38:15.535 INFO:tasks.workunit.client.0.smithi072.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-22T01:38:15.535 INFO:tasks.workunit.client.0.smithi072.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-22T01:38:15.535 INFO:tasks.workunit.client.0.smithi072.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-22T01:38:15.535 INFO:tasks.workunit.client.0.smithi072.stderr: | ^ 2024-03-22T01:38:15.535 INFO:tasks.workunit.client.0.smithi072.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-22T01:38:15.535 INFO:tasks.workunit.client.0.smithi072.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-22T01:38:15.536 INFO:tasks.workunit.client.0.smithi072.stderr: | ^~~~~~~~ 2024-03-22T01:38:15.536 INFO:tasks.workunit.client.0.smithi072.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-22T01:38:15.536 INFO:tasks.workunit.client.0.smithi072.stderr: 36 | container_of(ptr, type, member) 2024-03-22T01:38:15.536 INFO:tasks.workunit.client.0.smithi072.stderr: | ^~~~~~~~~~~~ 2024-03-22T01:38:15.536 INFO:tasks.workunit.client.0.smithi072.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-03-22T01:38:15.536 INFO:tasks.workunit.client.0.smithi072.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-03-22T01:38:15.536 INFO:tasks.workunit.client.0.smithi072.stderr: | ^~~~~~~~~~ 2024-03-22T01:38:15.536 INFO:tasks.workunit.client.0.smithi072.stderr:parser.c:980:41: note: in expansion of macro ‘list_for_each_entry’ 2024-03-22T01:38:15.536 INFO:tasks.workunit.client.0.smithi072.stderr: 980 | list_for_each_entry(tmp_list, 2024-03-22T01:38:15.536 INFO:tasks.workunit.client.0.smithi072.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-22T01:38:15.536 INFO:tasks.workunit.client.0.smithi072.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-22T01:38:15.536 INFO:tasks.workunit.client.0.smithi072.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-22T01:38:15.536 INFO:tasks.workunit.client.0.smithi072.stderr: | ^ 2024-03-22T01:38:15.536 INFO:tasks.workunit.client.0.smithi072.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-22T01:38:15.537 INFO:tasks.workunit.client.0.smithi072.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-22T01:38:15.537 INFO:tasks.workunit.client.0.smithi072.stderr: | ^~~~~~~~ 2024-03-22T01:38:15.537 INFO:tasks.workunit.client.0.smithi072.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-22T01:38:15.537 INFO:tasks.workunit.client.0.smithi072.stderr: 36 | container_of(ptr, type, member) 2024-03-22T01:38:15.537 INFO:tasks.workunit.client.0.smithi072.stderr: | ^~~~~~~~~~~~ 2024-03-22T01:38:15.537 INFO:tasks.workunit.client.0.smithi072.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-03-22T01:38:15.537 INFO:tasks.workunit.client.0.smithi072.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-03-22T01:38:15.537 INFO:tasks.workunit.client.0.smithi072.stderr: | ^~~~~~~~~~ 2024-03-22T01:38:15.537 INFO:tasks.workunit.client.0.smithi072.stderr:parser.c:980:41: note: in expansion of macro ‘list_for_each_entry’ 2024-03-22T01:38:15.537 INFO:tasks.workunit.client.0.smithi072.stderr: 980 | list_for_each_entry(tmp_list, 2024-03-22T01:38:15.537 INFO:tasks.workunit.client.0.smithi072.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-22T01:38:15.573 INFO:tasks.workunit.client.0.smithi072.stderr:parser.c: In function ‘get_optstr’: 2024-03-22T01:38:15.573 INFO:tasks.workunit.client.0.smithi072.stderr:parser.c:206:23: warning: ‘strnlen’ specified bound 4095 exceeds source size 1024 [-Wstringop-overread] 2024-03-22T01:38:15.573 INFO:tasks.workunit.client.0.smithi072.stderr: 206 | len = strnlen(temp, 4095) + 1; 2024-03-22T01:38:15.573 INFO:tasks.workunit.client.0.smithi072.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-22T01:38:15.573 INFO:tasks.workunit.client.0.smithi072.stderr:parser.c:200:14: note: source object declared here 2024-03-22T01:38:15.573 INFO:tasks.workunit.client.0.smithi072.stderr: 200 | char temp[BUFSIZE]; 2024-03-22T01:38:15.573 INFO:tasks.workunit.client.0.smithi072.stderr: | ^~~~ 2024-03-22T01:38:15.580 INFO:tasks.workunit.client.0.smithi072.stderr:parser.c: In function ‘size64_convert’: 2024-03-22T01:38:15.580 INFO:tasks.workunit.client.0.smithi072.stderr:parser.c:119:9: warning: ‘memcpy’ reading 3 bytes from a region of size 2 [-Wstringop-overread] 2024-03-22T01:38:15.580 INFO:tasks.workunit.client.0.smithi072.stderr: 119 | memcpy(unit, "\0", 3); 2024-03-22T01:38:15.580 INFO:tasks.workunit.client.0.smithi072.stderr: | ^~~~~~~~~~~~~~~~~~~~~ 2024-03-22T01:38:15.877 INFO:tasks.workunit.client.0.smithi072.stdout:mv -f .deps/parser.Tpo .deps/parser.Po 2024-03-22T01:38:15.879 INFO:tasks.workunit.client.0.smithi072.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fc.o -MD -MP -MF .deps/ffsb_fc.Tpo -c -o ffsb_fc.o ffsb_fc.c 2024-03-22T01:38:15.935 INFO:tasks.workunit.client.0.smithi072.stdout:mv -f .deps/ffsb_fc.Tpo .deps/ffsb_fc.Po 2024-03-22T01:38:15.938 INFO:tasks.workunit.client.0.smithi072.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_stats.o -MD -MP -MF .deps/ffsb_stats.Tpo -c -o ffsb_stats.o ffsb_stats.c 2024-03-22T01:38:16.026 INFO:tasks.workunit.client.0.smithi072.stdout:mv -f .deps/ffsb_stats.Tpo .deps/ffsb_stats.Po 2024-03-22T01:38:16.029 INFO:tasks.workunit.client.0.smithi072.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT list.o -MD -MP -MF .deps/list.Tpo -c -o list.o list.c 2024-03-22T01:38:16.058 INFO:tasks.workunit.client.0.smithi072.stdout:mv -f .deps/list.Tpo .deps/list.Po 2024-03-22T01:38:16.061 INFO:tasks.workunit.client.0.smithi072.stdout:gcc -D_GNU_SOURCE -g -O2 -D_REENTRANT -o ffsb fileops.o rand.o main.o fh.o filelist.o metaops.o rwlock.o cirlist.o rbt.o ffsb_tg.o ffsb_fs.o ffsb_thread.o ffsb_op.o util.o parser.o ffsb_fc.o ffsb_stats.o list.o -lpthread -lm 2024-03-22T01:38:16.092 INFO:tasks.workunit.client.0.smithi072.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb' 2024-03-22T01:38:16.092 INFO:tasks.workunit.client.0.smithi072.stderr:+ cd .. 2024-03-22T01:38:16.092 INFO:tasks.workunit.client.0.smithi072.stderr:+ mkdir tmp 2024-03-22T01:38:16.094 INFO:tasks.workunit.client.0.smithi072.stderr:+ cd tmp 2024-03-22T01:38:16.094 INFO:tasks.workunit.client.0.smithi072.stderr:+ for f in $mydir/*.ffsb 2024-03-22T01:38:16.094 INFO:tasks.workunit.client.0.smithi072.stderr:+ ../ffsb/ffsb /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/random_write.32.ffsb 2024-03-22T01:38:16.095 INFO:tasks.workunit.client.0.smithi072.stdout:FFSB version 6.0-RC2 started 2024-03-22T01:38:16.095 INFO:tasks.workunit.client.0.smithi072.stdout: 2024-03-22T01:38:16.097 INFO:tasks.workunit.client.0.smithi072.stdout:benchmark time = 300 2024-03-22T01:38:16.097 INFO:tasks.workunit.client.0.smithi072.stdout:ThreadGroup 0 2024-03-22T01:38:16.097 INFO:tasks.workunit.client.0.smithi072.stdout:================ 2024-03-22T01:38:16.098 INFO:tasks.workunit.client.0.smithi072.stdout: num_threads = 32 2024-03-22T01:38:16.098 INFO:tasks.workunit.client.0.smithi072.stdout: 2024-03-22T01:38:16.098 INFO:tasks.workunit.client.0.smithi072.stdout: read_random = off 2024-03-22T01:38:16.098 INFO:tasks.workunit.client.0.smithi072.stdout: read_size = 0 (0B) 2024-03-22T01:38:16.098 INFO:tasks.workunit.client.0.smithi072.stdout: read_blocksize = 0 (0B) 2024-03-22T01:38:16.098 INFO:tasks.workunit.client.0.smithi072.stdout: read_skip = off 2024-03-22T01:38:16.098 INFO:tasks.workunit.client.0.smithi072.stdout: read_skipsize = 0 (0B) 2024-03-22T01:38:16.098 INFO:tasks.workunit.client.0.smithi072.stdout: 2024-03-22T01:38:16.098 INFO:tasks.workunit.client.0.smithi072.stdout: write_random = on 2024-03-22T01:38:16.098 INFO:tasks.workunit.client.0.smithi072.stdout: write_size = 5242880 (5MB) 2024-03-22T01:38:16.098 INFO:tasks.workunit.client.0.smithi072.stdout: fsync_file = 0 2024-03-22T01:38:16.098 INFO:tasks.workunit.client.0.smithi072.stdout: write_blocksize = 4096 (4KB) 2024-03-22T01:38:16.098 INFO:tasks.workunit.client.0.smithi072.stdout: wait time = 0 2024-03-22T01:38:16.098 INFO:tasks.workunit.client.0.smithi072.stdout: 2024-03-22T01:38:16.099 INFO:tasks.workunit.client.0.smithi072.stdout: op weights 2024-03-22T01:38:16.099 INFO:tasks.workunit.client.0.smithi072.stdout: read = 0 (0.00%) 2024-03-22T01:38:16.099 INFO:tasks.workunit.client.0.smithi072.stdout: readall = 0 (0.00%) 2024-03-22T01:38:16.099 INFO:tasks.workunit.client.0.smithi072.stdout: write = 1 (100.00%) 2024-03-22T01:38:16.099 INFO:tasks.workunit.client.0.smithi072.stdout: create = 0 (0.00%) 2024-03-22T01:38:16.099 INFO:tasks.workunit.client.0.smithi072.stdout: append = 0 (0.00%) 2024-03-22T01:38:16.099 INFO:tasks.workunit.client.0.smithi072.stdout: delete = 0 (0.00%) 2024-03-22T01:38:16.099 INFO:tasks.workunit.client.0.smithi072.stdout: metaop = 0 (0.00%) 2024-03-22T01:38:16.099 INFO:tasks.workunit.client.0.smithi072.stdout: createdir = 0 (0.00%) 2024-03-22T01:38:16.099 INFO:tasks.workunit.client.0.smithi072.stdout: stat = 0 (0.00%) 2024-03-22T01:38:16.099 INFO:tasks.workunit.client.0.smithi072.stdout: writeall = 0 (0.00%) 2024-03-22T01:38:16.099 INFO:tasks.workunit.client.0.smithi072.stdout: writeall_fsync = 0 (0.00%) 2024-03-22T01:38:16.099 INFO:tasks.workunit.client.0.smithi072.stdout: open_close = 0 (0.00%) 2024-03-22T01:38:16.099 INFO:tasks.workunit.client.0.smithi072.stdout: write_fsync = 0 (0.00%) 2024-03-22T01:38:16.100 INFO:tasks.workunit.client.0.smithi072.stdout: create_fsync = 0 (0.00%) 2024-03-22T01:38:16.100 INFO:tasks.workunit.client.0.smithi072.stdout: append_fsync = 0 (0.00%) 2024-03-22T01:38:16.100 INFO:tasks.workunit.client.0.smithi072.stdout: 2024-03-22T01:38:16.100 INFO:tasks.workunit.client.0.smithi072.stdout:FileSystem . 2024-03-22T01:38:16.100 INFO:tasks.workunit.client.0.smithi072.stdout:========== 2024-03-22T01:38:16.100 INFO:tasks.workunit.client.0.smithi072.stdout: num_dirs = 0 2024-03-22T01:38:16.100 INFO:tasks.workunit.client.0.smithi072.stdout: starting files = 128 2024-03-22T01:38:16.100 INFO:tasks.workunit.client.0.smithi072.stdout: 2024-03-22T01:38:16.100 INFO:tasks.workunit.client.0.smithi072.stdout: min file size = 104857600 (100MB) 2024-03-22T01:38:16.100 INFO:tasks.workunit.client.0.smithi072.stdout: max file size = 104857600 (100MB) 2024-03-22T01:38:16.100 INFO:tasks.workunit.client.0.smithi072.stdout: directio = off 2024-03-22T01:38:16.100 INFO:tasks.workunit.client.0.smithi072.stdout: alignedio = on 2024-03-22T01:38:16.100 INFO:tasks.workunit.client.0.smithi072.stdout: bufferedio = off 2024-03-22T01:38:16.100 INFO:tasks.workunit.client.0.smithi072.stdout: 2024-03-22T01:38:16.100 INFO:tasks.workunit.client.0.smithi072.stdout: aging is off 2024-03-22T01:38:16.101 INFO:tasks.workunit.client.0.smithi072.stdout: current utilization = 0.04% 2024-03-22T01:38:16.101 INFO:tasks.workunit.client.0.smithi072.stdout: 2024-03-22T01:38:16.101 INFO:tasks.workunit.client.0.smithi072.stdout:checking existing fs: . 2024-03-22T01:38:16.101 INFO:tasks.workunit.client.0.smithi072.stdout:recreating new fileset 2024-03-22T01:38:16.101 INFO:tasks.workunit.client.0.smithi072.stderr:opendir: No such file or directory 2024-03-22T01:38:16.357 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:38:16 smithi089 ceph-mon[35770]: pgmap v51: 129 pgs: 129 active+clean; 7.1 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 606 KiB/s wr, 74 op/s 2024-03-22T01:38:16.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:38:16 smithi072 ceph-mon[32414]: pgmap v51: 129 pgs: 129 active+clean; 7.1 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 606 KiB/s wr, 74 op/s 2024-03-22T01:38:16.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:38:16 smithi098 ceph-mon[36468]: pgmap v51: 129 pgs: 129 active+clean; 7.1 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 606 KiB/s wr, 74 op/s 2024-03-22T01:38:17.024 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.024+0000 7f9028fd0640 1 -- 172.21.15.72:0/3665719614 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9024102780 msgr2=0x7f90241067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:17.024 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.024+0000 7f9028fd0640 1 --2- 172.21.15.72:0/3665719614 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9024102780 0x7f90241067c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f9014009900 tx=0x7f901402bc30 comp rx=0 tx=0).stop 2024-03-22T01:38:17.024 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.024+0000 7f9028fd0640 1 -- 172.21.15.72:0/3665719614 shutdown_connections 2024-03-22T01:38:17.024 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.024+0000 7f9028fd0640 1 --2- 172.21.15.72:0/3665719614 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9024102780 0x7f90241067c0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:17.024 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.024+0000 7f9028fd0640 1 --2- 172.21.15.72:0/3665719614 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9024106ea0 0x7f902410ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:17.024 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.024+0000 7f9028fd0640 1 --2- 172.21.15.72:0/3665719614 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9024101dd0 0x7f90241021b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:17.024 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.024+0000 7f9028fd0640 1 -- 172.21.15.72:0/3665719614 >> 172.21.15.72:0/3665719614 conn(0x7f90240fd630 msgr2=0x7f90240ffa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:38:17.024 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.024+0000 7f9028fd0640 1 -- 172.21.15.72:0/3665719614 shutdown_connections 2024-03-22T01:38:17.024 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.024+0000 7f9028fd0640 1 -- 172.21.15.72:0/3665719614 wait complete. 2024-03-22T01:38:17.025 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.025+0000 7f9028fd0640 1 Processor -- start 2024-03-22T01:38:17.025 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.025+0000 7f9028fd0640 1 -- start start 2024-03-22T01:38:17.027 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.025+0000 7f9028fd0640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9024101dd0 0x7f9024198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:17.027 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.025+0000 7f9028fd0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9024106ea0 0x7f9024198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:17.027 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.025+0000 7f9028fd0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f902419cdd0 0x7f902419d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:17.027 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.025+0000 7f9028fd0640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f9024111f50 con 0x7f9024101dd0 2024-03-22T01:38:17.027 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.025+0000 7f9028fd0640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9024111dd0 con 0x7f9024106ea0 2024-03-22T01:38:17.027 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.025+0000 7f9028fd0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f90241120d0 con 0x7f902419cdd0 2024-03-22T01:38:17.027 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.025+0000 7f9021d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9024106ea0 0x7f9024198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:17.027 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.025+0000 7f9022d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f902419cdd0 0x7f902419d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:17.027 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.025+0000 7f902259c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9024101dd0 0x7f9024198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:17.027 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.025+0000 7f9021d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9024106ea0 0x7f9024198aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.72:46778/0 (socket says 172.21.15.72:46778) 2024-03-22T01:38:17.027 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.025+0000 7f9022d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f902419cdd0 0x7f902419d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.72:36208/0 (socket says 172.21.15.72:36208) 2024-03-22T01:38:17.028 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.025+0000 7f9021d9b640 1 -- 172.21.15.72:0/2127515378 learned_addr learned my addr 172.21.15.72:0/2127515378 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:38:17.028 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.026+0000 7f9022d9d640 1 -- 172.21.15.72:0/2127515378 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9024106ea0 msgr2=0x7f9024198aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:17.028 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.026+0000 7f9022d9d640 1 --2- 172.21.15.72:0/2127515378 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9024106ea0 0x7f9024198aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:17.028 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.026+0000 7f9022d9d640 1 -- 172.21.15.72:0/2127515378 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9024101dd0 msgr2=0x7f9024198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:17.028 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.026+0000 7f9022d9d640 1 --2- 172.21.15.72:0/2127515378 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9024101dd0 0x7f9024198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:17.028 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.026+0000 7f9022d9d640 1 -- 172.21.15.72:0/2127515378 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f902419d8c0 con 0x7f902419cdd0 2024-03-22T01:38:17.028 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.026+0000 7f9021d9b640 1 --2- 172.21.15.72:0/2127515378 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9024106ea0 0x7f9024198aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:38:17.028 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.026+0000 7f902259c640 1 --2- 172.21.15.72:0/2127515378 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9024101dd0 0x7f9024198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:38:17.028 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.026+0000 7f9022d9d640 1 --2- 172.21.15.72:0/2127515378 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f902419cdd0 0x7f902419d280 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f901c009fd0 tx=0x7f901c00ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:38:17.028 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.026+0000 7f90137fe640 1 -- 172.21.15.72:0/2127515378 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f901c00ce20 con 0x7f902419cdd0 2024-03-22T01:38:17.028 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.026+0000 7f90137fe640 1 -- 172.21.15.72:0/2127515378 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f901c005020 con 0x7f902419cdd0 2024-03-22T01:38:17.028 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.026+0000 7f9028fd0640 1 -- 172.21.15.72:0/2127515378 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f902419dbb0 con 0x7f902419cdd0 2024-03-22T01:38:17.029 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.026+0000 7f90137fe640 1 -- 172.21.15.72:0/2127515378 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f901c010720 con 0x7f902419cdd0 2024-03-22T01:38:17.029 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.026+0000 7f9028fd0640 1 -- 172.21.15.72:0/2127515378 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90241a53a0 con 0x7f902419cdd0 2024-03-22T01:38:17.036 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.029+0000 7f90137fe640 1 -- 172.21.15.72:0/2127515378 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 137985+0+0 (secure 0 0 0) 0x7f901c005540 con 0x7f902419cdd0 2024-03-22T01:38:17.036 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.029+0000 7f902bbb8740 1 Processor -- start 2024-03-22T01:38:17.036 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.029+0000 7f902bbb8740 1 -- start start 2024-03-22T01:38:17.036 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.030+0000 7f902bbb8740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564faf75ec90 0x564faf75f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:17.037 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.030+0000 7f902bbb8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564faf75f5b0 0x564faf76c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:17.037 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.030+0000 7f902bbb8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564faf76cdd0 0x564faf76f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:17.037 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.030+0000 7f902bbb8740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x564faf6c3040 con 0x564faf75ec90 2024-03-22T01:38:17.037 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.030+0000 7f902bbb8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x564faf66b9e0 con 0x564faf76cdd0 2024-03-22T01:38:17.037 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.030+0000 7f902bbb8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x564faf6beff0 con 0x564faf75f5b0 2024-03-22T01:38:17.037 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.030+0000 7f902259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564faf75f5b0 0x564faf76c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:17.037 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.030+0000 7f9021d9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564faf75ec90 0x564faf75f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:17.037 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.030+0000 7f902259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564faf75f5b0 0x564faf76c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.72:36224/0 (socket says 172.21.15.72:36224) 2024-03-22T01:38:17.037 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.030+0000 7f902259c640 1 -- 172.21.15.72:0/3559384555 learned_addr learned my addr 172.21.15.72:0/3559384555 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:38:17.037 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.030+0000 7f9021d9b640 1 --2- 172.21.15.72:0/3559384555 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564faf76cdd0 0x564faf76f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:17.037 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.030+0000 7f90137fe640 1 --2- 172.21.15.72:0/2127515378 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f900c0a3080 0x7f900c0a5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:17.038 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.030+0000 7f902259c640 1 -- 172.21.15.72:0/3559384555 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564faf76cdd0 msgr2=0x564faf76f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:17.038 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.030+0000 7f902259c640 1 --2- 172.21.15.72:0/3559384555 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564faf76cdd0 0x564faf76f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:17.038 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.030+0000 7f902259c640 1 -- 172.21.15.72:0/3559384555 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564faf75ec90 msgr2=0x564faf75f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:17.038 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.030+0000 7f902259c640 1 --2- 172.21.15.72:0/3559384555 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564faf75ec90 0x564faf75f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:17.038 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.030+0000 7f902259c640 1 -- 172.21.15.72:0/3559384555 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564faf3d7210 con 0x564faf75f5b0 2024-03-22T01:38:17.038 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.030+0000 7f9022d9d640 1 --2- 172.21.15.72:0/2127515378 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f900c0a3080 0x7f900c0a5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:17.038 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.030+0000 7f9021d9b640 1 --2- 172.21.15.72:0/3559384555 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564faf75ec90 0x564faf75f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:38:17.038 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.030+0000 7f90137fe640 1 -- 172.21.15.72:0/2127515378 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f901c0d0150 con 0x7f902419cdd0 2024-03-22T01:38:17.038 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.030+0000 7f902259c640 1 --2- 172.21.15.72:0/3559384555 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564faf75f5b0 0x564faf76c890 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9018011540 tx=0x7f9018011a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=46339f24355cbb03 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:38:17.038 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.031+0000 7f9022d9d640 1 --2- 172.21.15.72:0/2127515378 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f900c0a3080 0x7f900c0a5540 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f901c098b60 tx=0x7f901c0c6e50 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:38:17.038 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.031+0000 7f9010ff9640 1 -- 172.21.15.72:0/3559384555 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f901800f640 con 0x564faf75f5b0 2024-03-22T01:38:17.039 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.031+0000 7f9010ff9640 1 -- 172.21.15.72:0/3559384555 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f901800fcc0 con 0x564faf75f5b0 2024-03-22T01:38:17.039 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.031+0000 7f9010ff9640 1 -- 172.21.15.72:0/3559384555 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9018019d30 con 0x564faf75f5b0 2024-03-22T01:38:17.039 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.031+0000 7f902bbb8740 1 -- 172.21.15.72:0/3559384555 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564faf75f5b0 msgr2=0x564faf76c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:17.039 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.031+0000 7f902bbb8740 1 --2- 172.21.15.72:0/3559384555 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564faf75f5b0 0x564faf76c890 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9018011540 tx=0x7f9018011a00 comp rx=0 tx=0).stop 2024-03-22T01:38:17.039 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.031+0000 7f902bbb8740 1 -- 172.21.15.72:0/3559384555 shutdown_connections 2024-03-22T01:38:17.039 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.031+0000 7f902bbb8740 1 --2- 172.21.15.72:0/3559384555 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564faf76cdd0 0x564faf76f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:17.039 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.031+0000 7f902bbb8740 1 --2- 172.21.15.72:0/3559384555 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564faf75f5b0 0x564faf76c890 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:17.039 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.031+0000 7f902bbb8740 1 --2- 172.21.15.72:0/3559384555 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564faf75ec90 0x564faf75f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:17.039 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.031+0000 7f902bbb8740 1 -- 172.21.15.72:0/3559384555 >> 172.21.15.72:0/3559384555 conn(0x564faf759bf0 msgr2=0x564faf759fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:38:17.039 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.032+0000 7f902bbb8740 1 -- 172.21.15.72:0/3559384555 shutdown_connections 2024-03-22T01:38:17.039 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.032+0000 7f902bbb8740 1 -- 172.21.15.72:0/3559384555 wait complete. 2024-03-22T01:38:17.039 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.032+0000 7f902bbb8740 1 -- start start 2024-03-22T01:38:17.039 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.032+0000 7f902bbb8740 1 Processor -- start 2024-03-22T01:38:17.040 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.032+0000 7f902bbb8740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564faf75ec90 0x564faf76c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:17.040 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.032+0000 7f902bbb8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564faf75f5b0 0x564faf769b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:17.040 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.032+0000 7f902bbb8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564faf76cdd0 0x564faf76a180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:17.040 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.032+0000 7f902bbb8740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x564faf76fa00 con 0x564faf75ec90 2024-03-22T01:38:17.040 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.032+0000 7f902bbb8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x564faf67edb0 con 0x564faf75f5b0 2024-03-22T01:38:17.040 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.032+0000 7f902bbb8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x564faf76fb80 con 0x564faf76cdd0 2024-03-22T01:38:17.040 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.032+0000 7f902bbb8740 10 client.0 _ll_register_callbacks cb 0x564faf3d6a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:38:17.040 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.032+0000 7f9021d9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564faf75ec90 0x564faf76c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:17.040 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.032+0000 7f9021d9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564faf75ec90 0x564faf76c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:40996/0 (socket says 172.21.15.72:40996) 2024-03-22T01:38:17.040 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.032+0000 7f9021d9b640 1 -- 172.21.15.72:0/2593221189 learned_addr learned my addr 172.21.15.72:0/2593221189 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:38:17.040 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.032+0000 7f902259c640 1 --2- 172.21.15.72:0/2593221189 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564faf75f5b0 0x564faf769b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:17.040 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.032+0000 7f9021d9b640 1 --2- 172.21.15.72:0/2593221189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564faf76cdd0 0x564faf76a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:17.041 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.033+0000 7f9021d9b640 1 -- 172.21.15.72:0/2593221189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564faf76cdd0 msgr2=0x564faf76a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:17.041 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.033+0000 7f9021d9b640 1 --2- 172.21.15.72:0/2593221189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564faf76cdd0 0x564faf76a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:17.041 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.033+0000 7f9021d9b640 1 -- 172.21.15.72:0/2593221189 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564faf75f5b0 msgr2=0x564faf769b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:17.041 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.033+0000 7f9021d9b640 1 --2- 172.21.15.72:0/2593221189 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564faf75f5b0 0x564faf769b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:17.041 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.033+0000 7f9021d9b640 1 -- 172.21.15.72:0/2593221189 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564faf4a3620 con 0x564faf75ec90 2024-03-22T01:38:17.041 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.033+0000 7f902259c640 1 --2- 172.21.15.72:0/2593221189 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564faf75f5b0 0x564faf769b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:38:17.041 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.033+0000 7f9021d9b640 1 --2- 172.21.15.72:0/2593221189 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564faf75ec90 0x564faf76c7e0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f9014000c00 tx=0x7f901402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d60b30cb995735 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:38:17.041 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.033+0000 7f9009ffb640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:38:17.041 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.033+0000 7f9009ffb640 1 -- 172.21.15.72:0/2593221189 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9014005120 con 0x564faf75ec90 2024-03-22T01:38:17.041 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.033+0000 7f9009ffb640 1 -- 172.21.15.72:0/2593221189 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f90140052c0 con 0x564faf75ec90 2024-03-22T01:38:17.041 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.033+0000 7f902bbb8740 1 -- 172.21.15.72:0/2593221189 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f901800fcc0 con 0x564faf75ec90 2024-03-22T01:38:17.041 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.033+0000 7f9009ffb640 1 -- 172.21.15.72:0/2593221189 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f9014055050 con 0x564faf75ec90 2024-03-22T01:38:17.042 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.033+0000 7f902bbb8740 10 client.15909 fetch_fsmap learned FSMap version 21 2024-03-22T01:38:17.042 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.033+0000 7f902bbb8740 1 -- 172.21.15.72:0/2593221189 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x564faf77b9f0 con 0x564faf75ec90 2024-03-22T01:38:17.042 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.033+0000 7f9009ffb640 1 -- 172.21.15.72:0/2593221189 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f901800fcc0 con 0x564faf75ec90 2024-03-22T01:38:17.042 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.034+0000 7f9009ffb640 1 -- 172.21.15.72:0/2593221189 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f9014034440 con 0x564faf75ec90 2024-03-22T01:38:17.042 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.034+0000 7f902bbb8740 10 client.15909 fetch_fsmap finished waiting for FSMap version 21 2024-03-22T01:38:17.042 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.034+0000 7f902bbb8740 10 client.15909 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T01:38:17.042 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.034+0000 7f902bbb8740 20 client.15909 populate_metadata read hostname 'smithi072' 2024-03-22T01:38:17.042 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.034+0000 7f902bbb8740 1 --2- 172.21.15.72:0/2593221189 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x564faf77ce60 0x564faf77f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:17.042 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.034+0000 7f902bbb8740 4 client.15909 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T01:38:17.042 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.034+0000 7f902bbb8740 1 -- 172.21.15.72:0/2593221189 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564faf6d0580 con 0x564faf77ce60 2024-03-22T01:38:17.042 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.034+0000 7f902259c640 1 --2- 172.21.15.72:0/2593221189 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x564faf77ce60 0x564faf77f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:17.043 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.034+0000 7f9009ffb640 10 client.15909 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:38:17.043 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.034+0000 7f902259c640 1 --2- 172.21.15.72:0/2593221189 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x564faf77ce60 0x564faf77f250 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=88768a731b1f35c3 server_cookie=8a0f45cfb925b5a3 in_seq=0 out_seq=0 2024-03-22T01:38:17.043 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.035+0000 7f9009ffb640 1 -- 172.21.15.72:0/2593221189 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564faf6d0580 con 0x564faf77ce60 2024-03-22T01:38:17.043 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.035+0000 7f9009ffb640 10 client.15909 handle_command_reply: tid=0 2024-03-22T01:38:17.043 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.035+0000 7f902bbb8740 1 client.15909 shutdown 2024-03-22T01:38:17.043 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.035+0000 7f902bbb8740 1 -- 172.21.15.72:0/2593221189 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564faf75ec90 msgr2=0x564faf76c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:17.043 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.035+0000 7f902bbb8740 1 --2- 172.21.15.72:0/2593221189 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564faf75ec90 0x564faf76c7e0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f9014000c00 tx=0x7f901402bc30 comp rx=0 tx=0).stop 2024-03-22T01:38:17.043 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.035+0000 7f902bbb8740 1 -- 172.21.15.72:0/2593221189 shutdown_connections 2024-03-22T01:38:17.043 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.035+0000 7f902bbb8740 1 --2- 172.21.15.72:0/2593221189 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x564faf77ce60 0x564faf77f250 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:17.043 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.035+0000 7f902bbb8740 1 --2- 172.21.15.72:0/2593221189 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564faf75f5b0 0x564faf769b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:17.043 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.035+0000 7f9009ffb640 0 client.15909 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:38:17.043 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.035+0000 7f902bbb8740 1 --2- 172.21.15.72:0/2593221189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564faf76cdd0 0x564faf76a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:17.044 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.035+0000 7f902bbb8740 1 --2- 172.21.15.72:0/2593221189 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564faf75ec90 0x564faf76c7e0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:17.044 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.035+0000 7f902bbb8740 1 -- 172.21.15.72:0/2593221189 >> 172.21.15.72:0/2593221189 conn(0x564faf759bf0 msgr2=0x564faf75b940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:38:17.044 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.035+0000 7f902bbb8740 1 -- 172.21.15.72:0/2593221189 shutdown_connections 2024-03-22T01:38:17.044 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.035+0000 7f902bbb8740 1 -- 172.21.15.72:0/2593221189 wait complete. 2024-03-22T01:38:17.044 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.035+0000 7f902bbb8740 20 client.15909 trim_cache size 0 max 16384 2024-03-22T01:38:17.055 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.050+0000 7f902bbb8740 1 Processor -- start 2024-03-22T01:38:17.055 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.050+0000 7f902bbb8740 1 -- start start 2024-03-22T01:38:17.055 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.050+0000 7f902bbb8740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564faf75ec90 0x564faf7787a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:17.055 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.050+0000 7f902bbb8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564faf75f5b0 0x564faf784b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:17.055 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.050+0000 7f902bbb8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564faf76cdd0 0x564faf76c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:17.055 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.050+0000 7f902bbb8740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f90241aafa0 con 0x564faf75ec90 2024-03-22T01:38:17.055 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.050+0000 7f902bbb8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f90241ad820 con 0x564faf76cdd0 2024-03-22T01:38:17.055 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.050+0000 7f902bbb8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f90241ab610 con 0x564faf75f5b0 2024-03-22T01:38:17.056 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.050+0000 7f9021d9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564faf75ec90 0x564faf7787a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:17.056 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.050+0000 7f9021d9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564faf75ec90 0x564faf7787a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:40998/0 (socket says 172.21.15.72:40998) 2024-03-22T01:38:17.056 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.050+0000 7f9021d9b640 1 -- 172.21.15.72:0/3132872946 learned_addr learned my addr 172.21.15.72:0/3132872946 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:38:17.056 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.050+0000 7f902259c640 1 --2- 172.21.15.72:0/3132872946 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564faf75f5b0 0x564faf784b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:17.056 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.050+0000 7f9021d9b640 1 --2- 172.21.15.72:0/3132872946 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564faf76cdd0 0x564faf76c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:17.056 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.051+0000 7f902259c640 1 -- 172.21.15.72:0/3132872946 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564faf76cdd0 msgr2=0x564faf76c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:17.056 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.051+0000 7f902259c640 1 --2- 172.21.15.72:0/3132872946 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564faf76cdd0 0x564faf76c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:17.056 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.051+0000 7f902259c640 1 -- 172.21.15.72:0/3132872946 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564faf75ec90 msgr2=0x564faf7787a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:17.056 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.051+0000 7f902259c640 1 --2- 172.21.15.72:0/3132872946 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564faf75ec90 0x564faf7787a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:17.056 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.051+0000 7f902259c640 1 -- 172.21.15.72:0/3132872946 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564faf75e5b0 con 0x564faf75f5b0 2024-03-22T01:38:17.056 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.051+0000 7f902259c640 1 --2- 172.21.15.72:0/3132872946 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564faf75f5b0 0x564faf784b10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9018009d00 tx=0x7f9018009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=212d52aac867ca1d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:38:17.056 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.051+0000 7f900bfff640 1 -- 172.21.15.72:0/3132872946 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90180050d0 con 0x564faf75f5b0 2024-03-22T01:38:17.057 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.051+0000 7f900bfff640 1 -- 172.21.15.72:0/3132872946 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f901800f640 con 0x564faf75f5b0 2024-03-22T01:38:17.057 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.051+0000 7f900bfff640 1 -- 172.21.15.72:0/3132872946 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90180035f0 con 0x564faf75f5b0 2024-03-22T01:38:17.057 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.052+0000 7f902bbb8740 1 -- 172.21.15.72:0/3132872946 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564faf75f5b0 msgr2=0x564faf784b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:17.057 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.052+0000 7f902bbb8740 1 --2- 172.21.15.72:0/3132872946 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564faf75f5b0 0x564faf784b10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9018009d00 tx=0x7f9018009310 comp rx=0 tx=0).stop 2024-03-22T01:38:17.057 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.052+0000 7f902bbb8740 1 -- 172.21.15.72:0/3132872946 shutdown_connections 2024-03-22T01:38:17.057 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.052+0000 7f902bbb8740 1 --2- 172.21.15.72:0/3132872946 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564faf76cdd0 0x564faf76c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:17.057 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.052+0000 7f902bbb8740 1 --2- 172.21.15.72:0/3132872946 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564faf75f5b0 0x564faf784b10 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:17.057 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.052+0000 7f902bbb8740 1 --2- 172.21.15.72:0/3132872946 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564faf75ec90 0x564faf7787a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:17.057 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.052+0000 7f902bbb8740 1 -- 172.21.15.72:0/3132872946 >> 172.21.15.72:0/3132872946 conn(0x564faf759bf0 msgr2=0x564faf758c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:38:17.057 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.052+0000 7f902bbb8740 1 -- 172.21.15.72:0/3132872946 shutdown_connections 2024-03-22T01:38:17.057 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.052+0000 7f902bbb8740 1 -- 172.21.15.72:0/3132872946 wait complete. 2024-03-22T01:38:17.057 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.052+0000 7f902bbb8740 1 -- start start 2024-03-22T01:38:17.058 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.052+0000 7f902bbb8740 1 Processor -- start 2024-03-22T01:38:17.058 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.052+0000 7f902bbb8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564faf75ec90 0x564faf770630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:17.058 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.052+0000 7f902bbb8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564faf75f5b0 0x564faf770b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:17.058 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.052+0000 7f902bbb8740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564faf76cdd0 0x564faf7710b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:17.058 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.052+0000 7f902bbb8740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f90241abce0 con 0x564faf76cdd0 2024-03-22T01:38:17.058 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.052+0000 7f902bbb8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f90241ad150 con 0x564faf75ec90 2024-03-22T01:38:17.058 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.052+0000 7f902bbb8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f90241ac3b0 con 0x564faf75f5b0 2024-03-22T01:38:17.059 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.052+0000 7f902bbb8740 10 client.0 _ll_register_callbacks cb 0x7f901402cd90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:38:17.059 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.053+0000 7f9021d9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564faf76cdd0 0x564faf7710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:17.059 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.053+0000 7f9021d9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564faf76cdd0 0x564faf7710b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:41002/0 (socket says 172.21.15.72:41002) 2024-03-22T01:38:17.059 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.053+0000 7f9021d9b640 1 -- 172.21.15.72:0/959440704 learned_addr learned my addr 172.21.15.72:0/959440704 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:38:17.059 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.053+0000 7f902259c640 1 --2- 172.21.15.72:0/959440704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564faf75f5b0 0x564faf770b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:17.059 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.053+0000 7f9021d9b640 1 --2- 172.21.15.72:0/959440704 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564faf75ec90 0x564faf770630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:17.059 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.053+0000 7f9021d9b640 1 -- 172.21.15.72:0/959440704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564faf75f5b0 msgr2=0x564faf770b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:17.059 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.053+0000 7f9021d9b640 1 --2- 172.21.15.72:0/959440704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564faf75f5b0 0x564faf770b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:17.059 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.053+0000 7f9021d9b640 1 -- 172.21.15.72:0/959440704 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564faf75ec90 msgr2=0x564faf770630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:17.059 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.053+0000 7f9021d9b640 1 --2- 172.21.15.72:0/959440704 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564faf75ec90 0x564faf770630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:17.059 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.053+0000 7f9021d9b640 1 -- 172.21.15.72:0/959440704 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564faf7e2140 con 0x564faf76cdd0 2024-03-22T01:38:17.059 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.053+0000 7f9021d9b640 1 --2- 172.21.15.72:0/959440704 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564faf76cdd0 0x564faf7710b0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f90140349b0 tx=0x7f9014005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=880a6554a4dddf1 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:38:17.059 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.053+0000 7f9009ffb640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:38:17.059 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.053+0000 7f9009ffb640 1 -- 172.21.15.72:0/959440704 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9014034c30 con 0x564faf76cdd0 2024-03-22T01:38:17.060 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.053+0000 7f9009ffb640 1 -- 172.21.15.72:0/959440704 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f9014004f20 con 0x564faf76cdd0 2024-03-22T01:38:17.060 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.053+0000 7f902bbb8740 1 -- 172.21.15.72:0/959440704 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f901800f640 con 0x564faf76cdd0 2024-03-22T01:38:17.060 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.053+0000 7f9009ffb640 1 -- 172.21.15.72:0/959440704 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f9014005120 con 0x564faf76cdd0 2024-03-22T01:38:17.060 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.053+0000 7f902bbb8740 10 client.15915 fetch_fsmap learned FSMap version 21 2024-03-22T01:38:17.060 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.053+0000 7f902bbb8740 1 -- 172.21.15.72:0/959440704 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x564faf7e3350 con 0x564faf76cdd0 2024-03-22T01:38:17.060 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.054+0000 7f9009ffb640 1 -- 172.21.15.72:0/959440704 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f901800f640 con 0x564faf76cdd0 2024-03-22T01:38:17.060 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.054+0000 7f9009ffb640 1 -- 172.21.15.72:0/959440704 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f9014003c70 con 0x564faf76cdd0 2024-03-22T01:38:17.060 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.054+0000 7f902bbb8740 10 client.15915 fetch_fsmap finished waiting for FSMap version 21 2024-03-22T01:38:17.060 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.054+0000 7f902bbb8740 10 client.15915 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T01:38:17.060 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.054+0000 7f902bbb8740 20 client.15915 populate_metadata read hostname 'smithi072' 2024-03-22T01:38:17.060 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.054+0000 7f902bbb8740 1 --2- 172.21.15.72:0/959440704 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x564faf7e3a40 0x564faf7e5e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:17.060 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.054+0000 7f902bbb8740 4 client.15915 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T01:38:17.060 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.054+0000 7f902bbb8740 1 -- 172.21.15.72:0/959440704 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564faf6ca8d0 con 0x564faf7e3a40 2024-03-22T01:38:17.060 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.054+0000 7f902259c640 1 --2- 172.21.15.72:0/959440704 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x564faf7e3a40 0x564faf7e5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:17.061 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.054+0000 7f9009ffb640 10 client.15915 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:38:17.061 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.054+0000 7f902259c640 1 --2- 172.21.15.72:0/959440704 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x564faf7e3a40 0x564faf7e5e60 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd1751de97a69d35 server_cookie=7c02847aa2870401 in_seq=0 out_seq=0 2024-03-22T01:38:17.235 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.235+0000 7f9009ffb640 1 -- 172.21.15.72:0/959440704 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564faf6ca8d0 con 0x564faf7e3a40 2024-03-22T01:38:17.235 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.235+0000 7f9009ffb640 10 client.15915 handle_command_reply: tid=0 2024-03-22T01:38:17.235 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.235+0000 7f902bbb8740 1 client.15915 shutdown 2024-03-22T01:38:17.235 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.235+0000 7f902bbb8740 1 -- 172.21.15.72:0/959440704 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564faf76cdd0 msgr2=0x564faf7710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:17.235 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.235+0000 7f902bbb8740 1 --2- 172.21.15.72:0/959440704 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564faf76cdd0 0x564faf7710b0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f90140349b0 tx=0x7f9014005fd0 comp rx=0 tx=0).stop 2024-03-22T01:38:17.235 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.235+0000 7f902bbb8740 1 -- 172.21.15.72:0/959440704 shutdown_connections 2024-03-22T01:38:17.236 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.235+0000 7f902bbb8740 1 --2- 172.21.15.72:0/959440704 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x564faf7e3a40 0x564faf7e5e60 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:17.236 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.235+0000 7f902bbb8740 1 --2- 172.21.15.72:0/959440704 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564faf76cdd0 0x564faf7710b0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:17.236 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.235+0000 7f902bbb8740 1 --2- 172.21.15.72:0/959440704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564faf75f5b0 0x564faf770b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:17.236 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.235+0000 7f9009ffb640 0 client.15915 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:38:17.236 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.235+0000 7f902bbb8740 1 --2- 172.21.15.72:0/959440704 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564faf75ec90 0x564faf770630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:17.236 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.235+0000 7f902bbb8740 1 -- 172.21.15.72:0/959440704 >> 172.21.15.72:0/959440704 conn(0x564faf759bf0 msgr2=0x564faf764830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:38:17.236 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.236+0000 7f902bbb8740 1 -- 172.21.15.72:0/959440704 shutdown_connections 2024-03-22T01:38:17.236 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.236+0000 7f902bbb8740 1 -- 172.21.15.72:0/959440704 wait complete. 2024-03-22T01:38:17.236 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.236+0000 7f902bbb8740 20 client.15915 trim_cache size 0 max 16384 2024-03-22T01:38:17.236 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout:{ 2024-03-22T01:38:17.236 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout: "message": "", 2024-03-22T01:38:17.237 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout: "return_code": 0 2024-03-22T01:38:17.237 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout:} 2024-03-22T01:38:17.237 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.236+0000 7f9028fd0640 1 -- 172.21.15.72:0/2127515378 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f900c0a3080 msgr2=0x7f900c0a5540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:17.237 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.236+0000 7f9028fd0640 1 --2- 172.21.15.72:0/2127515378 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f900c0a3080 0x7f900c0a5540 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f901c098b60 tx=0x7f901c0c6e50 comp rx=0 tx=0).stop 2024-03-22T01:38:17.238 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.236+0000 7f9028fd0640 1 -- 172.21.15.72:0/2127515378 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f902419cdd0 msgr2=0x7f902419d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:17.238 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.236+0000 7f9028fd0640 1 --2- 172.21.15.72:0/2127515378 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f902419cdd0 0x7f902419d280 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f901c009fd0 tx=0x7f901c00ef10 comp rx=0 tx=0).stop 2024-03-22T01:38:17.238 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.236+0000 7f9028fd0640 1 -- 172.21.15.72:0/2127515378 shutdown_connections 2024-03-22T01:38:17.238 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.236+0000 7f9028fd0640 1 --2- 172.21.15.72:0/2127515378 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f900c0a3080 0x7f900c0a5540 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:17.238 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.236+0000 7f9028fd0640 1 --2- 172.21.15.72:0/2127515378 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9024106ea0 0x7f9024198aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:17.238 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.236+0000 7f9028fd0640 1 --2- 172.21.15.72:0/2127515378 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f902419cdd0 0x7f902419d280 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:17.238 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.236+0000 7f9028fd0640 1 --2- 172.21.15.72:0/2127515378 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9024101dd0 0x7f9024198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:17.238 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.236+0000 7f9028fd0640 1 -- 172.21.15.72:0/2127515378 >> 172.21.15.72:0/2127515378 conn(0x7f90240fd630 msgr2=0x7f90240ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:38:17.238 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.236+0000 7f9028fd0640 1 -- 172.21.15.72:0/2127515378 shutdown_connections 2024-03-22T01:38:17.238 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:17.236+0000 7f9028fd0640 1 -- 172.21.15.72:0/2127515378 wait complete. 2024-03-22T01:38:17.796 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T01:38:17.796 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T01:38:18.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.048+0000 7f83c359e640 1 -- 172.21.15.72:0/4079514557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83bc09be10 msgr2=0x7f83bc09c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:18.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.048+0000 7f83c359e640 1 --2- 172.21.15.72:0/4079514557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83bc09be10 0x7f83bc09c1f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f83b4009930 tx=0x7f83b402bc30 comp rx=0 tx=0).stop 2024-03-22T01:38:18.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.048+0000 7f83c359e640 1 -- 172.21.15.72:0/4079514557 shutdown_connections 2024-03-22T01:38:18.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.048+0000 7f83c359e640 1 --2- 172.21.15.72:0/4079514557 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83bc004cb0 0x7f83bc005130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:18.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.048+0000 7f83c359e640 1 --2- 172.21.15.72:0/4079514557 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83bc005670 0x7f83bc0a2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:18.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.048+0000 7f83c359e640 1 --2- 172.21.15.72:0/4079514557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83bc09be10 0x7f83bc09c1f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:18.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.048+0000 7f83c359e640 1 -- 172.21.15.72:0/4079514557 >> 172.21.15.72:0/4079514557 conn(0x7f83bc00a740 msgr2=0x7f83bc00ab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:38:18.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.049+0000 7f83c359e640 1 -- 172.21.15.72:0/4079514557 shutdown_connections 2024-03-22T01:38:18.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.049+0000 7f83c359e640 1 -- 172.21.15.72:0/4079514557 wait complete. 2024-03-22T01:38:18.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.051+0000 7f83c359e640 1 Processor -- start 2024-03-22T01:38:18.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.051+0000 7f83c359e640 1 -- start start 2024-03-22T01:38:18.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.051+0000 7f83c359e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83bc004cb0 0x7f83bc12a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:18.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.051+0000 7f83c359e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83bc005670 0x7f83bc12a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:18.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.051+0000 7f83c259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83bc004cb0 0x7f83bc12a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:18.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.051+0000 7f83c259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83bc004cb0 0x7f83bc12a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.72:46838/0 (socket says 172.21.15.72:46838) 2024-03-22T01:38:18.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.052+0000 7f83c259c640 1 -- 172.21.15.72:0/2617550542 learned_addr learned my addr 172.21.15.72:0/2617550542 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:38:18.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.052+0000 7f83c359e640 1 --2- 172.21.15.72:0/2617550542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83bc09be10 0x7f83bc12ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:18.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.052+0000 7f83c359e640 1 -- 172.21.15.72:0/2617550542 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f83bc0a3f30 con 0x7f83bc005670 2024-03-22T01:38:18.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.052+0000 7f83c359e640 1 -- 172.21.15.72:0/2617550542 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f83bc0a3db0 con 0x7f83bc004cb0 2024-03-22T01:38:18.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.052+0000 7f83c359e640 1 -- 172.21.15.72:0/2617550542 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f83bc0a40b0 con 0x7f83bc09be10 2024-03-22T01:38:18.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.052+0000 7f83c1d9b640 1 --2- 172.21.15.72:0/2617550542 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83bc005670 0x7f83bc12a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:18.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.052+0000 7f83c2d9d640 1 --2- 172.21.15.72:0/2617550542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83bc09be10 0x7f83bc12ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:18.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.052+0000 7f83c1d9b640 1 -- 172.21.15.72:0/2617550542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83bc09be10 msgr2=0x7f83bc12ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:18.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.052+0000 7f83c1d9b640 1 --2- 172.21.15.72:0/2617550542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83bc09be10 0x7f83bc12ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:18.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.052+0000 7f83c1d9b640 1 -- 172.21.15.72:0/2617550542 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83bc004cb0 msgr2=0x7f83bc12a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:18.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.052+0000 7f83c1d9b640 1 --2- 172.21.15.72:0/2617550542 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83bc004cb0 0x7f83bc12a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:18.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.052+0000 7f83c1d9b640 1 -- 172.21.15.72:0/2617550542 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83bc12f400 con 0x7f83bc005670 2024-03-22T01:38:18.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.052+0000 7f83c259c640 1 --2- 172.21.15.72:0/2617550542 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83bc004cb0 0x7f83bc12a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:38:18.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.052+0000 7f83c1d9b640 1 --2- 172.21.15.72:0/2617550542 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83bc005670 0x7f83bc12a950 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f83b800e9f0 tx=0x7f83b800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:38:18.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.053+0000 7f83b37fe640 1 -- 172.21.15.72:0/2617550542 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83b800ccc0 con 0x7f83bc005670 2024-03-22T01:38:18.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.053+0000 7f83c359e640 1 -- 172.21.15.72:0/2617550542 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83bc12f6f0 con 0x7f83bc005670 2024-03-22T01:38:18.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.053+0000 7f83b37fe640 1 -- 172.21.15.72:0/2617550542 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83b8010910 con 0x7f83bc005670 2024-03-22T01:38:18.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.053+0000 7f83b37fe640 1 -- 172.21.15.72:0/2617550542 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83b8018960 con 0x7f83bc005670 2024-03-22T01:38:18.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.053+0000 7f83c359e640 1 -- 172.21.15.72:0/2617550542 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83bc136f80 con 0x7f83bc005670 2024-03-22T01:38:18.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.057+0000 7f83b37fe640 1 -- 172.21.15.72:0/2617550542 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 25) ==== 137985+0+0 (secure 0 0 0) 0x7f83b8018b00 con 0x7f83bc005670 2024-03-22T01:38:18.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.057+0000 7f83cbbf9740 1 Processor -- start 2024-03-22T01:38:18.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.058+0000 7f83cbbf9740 1 -- start start 2024-03-22T01:38:18.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.058+0000 7f83cbbf9740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f14abb7600 0x55f14abb79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:18.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.058+0000 7f83cbbf9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f14abb7f20 0x55f14abc05c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:18.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.058+0000 7f83b37fe640 1 --2- 172.21.15.72:0/2617550542 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f83ac0a7560 0x7f83ac0a9a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:18.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.058+0000 7f83cbbf9740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f14abc0b00 0x55f14abc2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:18.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.058+0000 7f83cbbf9740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55f14ab1b8c0 con 0x55f14abc0b00 2024-03-22T01:38:18.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.058+0000 7f83cbbf9740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55f14aac3c20 con 0x55f14abb7600 2024-03-22T01:38:18.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.058+0000 7f83cbbf9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f14ab17870 con 0x55f14abb7f20 2024-03-22T01:38:18.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.059+0000 7f83c2d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f14abb7600 0x55f14abb79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:18.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.059+0000 7f83c2d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f14abb7600 0x55f14abb79e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.72:46846/0 (socket says 172.21.15.72:46846) 2024-03-22T01:38:18.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.059+0000 7f83c2d9d640 1 -- 172.21.15.72:0/2342140014 learned_addr learned my addr 172.21.15.72:0/2342140014 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:38:18.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.059+0000 7f83c259c640 1 --2- 172.21.15.72:0/2617550542 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f83ac0a7560 0x7f83ac0a9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:18.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.059+0000 7f83c2d9d640 1 --2- 172.21.15.72:0/2342140014 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f14abc0b00 0x55f14abc2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:18.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.059+0000 7f83b37fe640 1 -- 172.21.15.72:0/2617550542 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f83b80d0ee0 con 0x7f83bc005670 2024-03-22T01:38:18.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.059+0000 7f83c1d9b640 1 --2- 172.21.15.72:0/2342140014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f14abb7f20 0x55f14abc05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:18.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.059+0000 7f83c2d9d640 1 -- 172.21.15.72:0/2342140014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f14abb7f20 msgr2=0x55f14abc05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:18.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.059+0000 7f83c2d9d640 1 --2- 172.21.15.72:0/2342140014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f14abb7f20 0x55f14abc05c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:18.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.059+0000 7f83c2d9d640 1 -- 172.21.15.72:0/2342140014 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f14abc0b00 msgr2=0x55f14abc2ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:18.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.059+0000 7f83c2d9d640 1 --2- 172.21.15.72:0/2342140014 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f14abc0b00 0x55f14abc2ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:18.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.059+0000 7f83c2d9d640 1 -- 172.21.15.72:0/2342140014 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f14aae8150 con 0x55f14abb7600 2024-03-22T01:38:18.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.059+0000 7f83c259c640 1 --2- 172.21.15.72:0/2617550542 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f83ac0a7560 0x7f83ac0a9a20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f83b4006010 tx=0x7f83b402df90 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:38:18.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.060+0000 7f83c2d9d640 1 --2- 172.21.15.72:0/2342140014 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f14abb7600 0x55f14abb79e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f83c40511f0 tx=0x7f83c4067aa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d73db4624b27f93f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:38:18.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.060+0000 7f83b0ff9640 1 -- 172.21.15.72:0/2342140014 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c407eb90 con 0x55f14abb7600 2024-03-22T01:38:18.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.060+0000 7f83b0ff9640 1 -- 172.21.15.72:0/2342140014 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83c4069040 con 0x55f14abb7600 2024-03-22T01:38:18.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.060+0000 7f83b0ff9640 1 -- 172.21.15.72:0/2342140014 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c4083420 con 0x55f14abb7600 2024-03-22T01:38:18.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.061+0000 7f83cbbf9740 1 -- 172.21.15.72:0/2342140014 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f14abb7600 msgr2=0x55f14abb79e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:18.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.061+0000 7f83cbbf9740 1 --2- 172.21.15.72:0/2342140014 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f14abb7600 0x55f14abb79e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f83c40511f0 tx=0x7f83c4067aa0 comp rx=0 tx=0).stop 2024-03-22T01:38:18.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.061+0000 7f83cbbf9740 1 -- 172.21.15.72:0/2342140014 shutdown_connections 2024-03-22T01:38:18.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.061+0000 7f83cbbf9740 1 --2- 172.21.15.72:0/2342140014 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f14abc0b00 0x55f14abc2ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:18.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.061+0000 7f83cbbf9740 1 --2- 172.21.15.72:0/2342140014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f14abb7f20 0x55f14abc05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:18.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.061+0000 7f83cbbf9740 1 --2- 172.21.15.72:0/2342140014 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f14abb7600 0x55f14abb79e0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:18.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.061+0000 7f83cbbf9740 1 -- 172.21.15.72:0/2342140014 >> 172.21.15.72:0/2342140014 conn(0x55f14abb1d30 msgr2=0x55f14abb2110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:38:18.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.061+0000 7f83cbbf9740 1 -- 172.21.15.72:0/2342140014 shutdown_connections 2024-03-22T01:38:18.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.062+0000 7f83cbbf9740 1 -- 172.21.15.72:0/2342140014 wait complete. 2024-03-22T01:38:18.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.062+0000 7f83cbbf9740 1 -- start start 2024-03-22T01:38:18.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.062+0000 7f83cbbf9740 1 Processor -- start 2024-03-22T01:38:18.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.062+0000 7f83cbbf9740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f14abb7600 0x55f14abbda80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:18.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.062+0000 7f83cbbf9740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f14abb7f20 0x55f14abbdfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:18.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.062+0000 7f83cbbf9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f14abc0b00 0x55f14abbe500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:18.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.062+0000 7f83cbbf9740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55f14abc38f0 con 0x55f14abb7600 2024-03-22T01:38:18.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.062+0000 7f83cbbf9740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55f14aad7130 con 0x55f14abb7f20 2024-03-22T01:38:18.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.063+0000 7f83cbbf9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f14abc3a70 con 0x55f14abc0b00 2024-03-22T01:38:18.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.063+0000 7f83cbbf9740 10 client.0 _ll_register_callbacks cb 0x55f14aad6b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:38:18.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.063+0000 7f83c259c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f14abb7600 0x55f14abbda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:18.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.063+0000 7f83c259c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f14abb7600 0x55f14abbda80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:41036/0 (socket says 172.21.15.72:41036) 2024-03-22T01:38:18.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.063+0000 7f83c259c640 1 -- 172.21.15.72:0/2847295151 learned_addr learned my addr 172.21.15.72:0/2847295151 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:38:18.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.064+0000 7f83c1d9b640 1 --2- 172.21.15.72:0/2847295151 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f14abb7f20 0x55f14abbdfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:18.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.064+0000 7f83c2d9d640 1 --2- 172.21.15.72:0/2847295151 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f14abc0b00 0x55f14abbe500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:18.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.064+0000 7f83c259c640 1 -- 172.21.15.72:0/2847295151 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f14abc0b00 msgr2=0x55f14abbe500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:18.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.064+0000 7f83c259c640 1 --2- 172.21.15.72:0/2847295151 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f14abc0b00 0x55f14abbe500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:18.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.064+0000 7f83c259c640 1 -- 172.21.15.72:0/2847295151 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f14abb7f20 msgr2=0x55f14abbdfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:18.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.064+0000 7f83c259c640 1 --2- 172.21.15.72:0/2847295151 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f14abb7f20 0x55f14abbdfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:18.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.064+0000 7f83c259c640 1 -- 172.21.15.72:0/2847295151 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f14a8f97d0 con 0x55f14abb7600 2024-03-22T01:38:18.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.064+0000 7f83c1d9b640 1 --2- 172.21.15.72:0/2847295151 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f14abb7f20 0x55f14abbdfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T01:38:18.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.065+0000 7f83c259c640 1 --2- 172.21.15.72:0/2847295151 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f14abb7600 0x55f14abbda80 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f83b402cb00 tx=0x7f83b4009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82edef519a4033e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:38:18.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.065+0000 7f83a9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:38:18.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.065+0000 7f83a9ffb640 1 -- 172.21.15.72:0/2847295151 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83b4050070 con 0x55f14abb7600 2024-03-22T01:38:18.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.066+0000 7f83a9ffb640 1 -- 172.21.15.72:0/2847295151 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f83b4003800 con 0x55f14abb7600 2024-03-22T01:38:18.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.066+0000 7f83cbbf9740 1 -- 172.21.15.72:0/2847295151 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f83c4069040 con 0x55f14abb7600 2024-03-22T01:38:18.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.066+0000 7f83a9ffb640 1 -- 172.21.15.72:0/2847295151 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f83b405c050 con 0x55f14abb7600 2024-03-22T01:38:18.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.066+0000 7f83cbbf9740 10 client.15927 fetch_fsmap learned FSMap version 21 2024-03-22T01:38:18.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.066+0000 7f83cbbf9740 1 -- 172.21.15.72:0/2847295151 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55f14a82f3f0 con 0x55f14abb7600 2024-03-22T01:38:18.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.067+0000 7f83a9ffb640 1 -- 172.21.15.72:0/2847295151 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83c4069040 con 0x55f14abb7600 2024-03-22T01:38:18.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.067+0000 7f83a9ffb640 1 -- 172.21.15.72:0/2847295151 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f83b404b070 con 0x55f14abb7600 2024-03-22T01:38:18.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.067+0000 7f83cbbf9740 10 client.15927 fetch_fsmap finished waiting for FSMap version 21 2024-03-22T01:38:18.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.067+0000 7f83cbbf9740 10 client.15927 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T01:38:18.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.067+0000 7f83cbbf9740 20 client.15927 populate_metadata read hostname 'smithi072' 2024-03-22T01:38:18.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.067+0000 7f83cbbf9740 1 --2- 172.21.15.72:0/2847295151 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55f14abd9610 0x55f14abdba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:18.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.067+0000 7f83cbbf9740 4 client.15927 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T01:38:18.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.067+0000 7f83cbbf9740 1 -- 172.21.15.72:0/2847295151 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f14ab28e00 con 0x55f14abd9610 2024-03-22T01:38:18.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.068+0000 7f83c1d9b640 1 --2- 172.21.15.72:0/2847295151 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55f14abd9610 0x55f14abdba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:18.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.068+0000 7f83a9ffb640 10 client.15927 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:38:18.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.068+0000 7f83c1d9b640 1 --2- 172.21.15.72:0/2847295151 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55f14abd9610 0x55f14abdba40 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ca27f866e3d1c97 server_cookie=3a13ff5d057526c1 in_seq=0 out_seq=0 2024-03-22T01:38:18.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.070+0000 7f83a9ffb640 1 -- 172.21.15.72:0/2847295151 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f14ab28e00 con 0x55f14abd9610 2024-03-22T01:38:18.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.070+0000 7f83a9ffb640 10 client.15927 handle_command_reply: tid=0 2024-03-22T01:38:18.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.071+0000 7f83cbbf9740 1 client.15927 shutdown 2024-03-22T01:38:18.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.071+0000 7f83cbbf9740 1 -- 172.21.15.72:0/2847295151 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f14abb7600 msgr2=0x55f14abbda80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:18.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.071+0000 7f83cbbf9740 1 --2- 172.21.15.72:0/2847295151 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f14abb7600 0x55f14abbda80 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f83b402cb00 tx=0x7f83b4009260 comp rx=0 tx=0).stop 2024-03-22T01:38:18.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.071+0000 7f83cbbf9740 1 -- 172.21.15.72:0/2847295151 shutdown_connections 2024-03-22T01:38:18.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.071+0000 7f83cbbf9740 1 --2- 172.21.15.72:0/2847295151 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55f14abd9610 0x55f14abdba40 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:18.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.071+0000 7f83cbbf9740 1 --2- 172.21.15.72:0/2847295151 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f14abb7f20 0x55f14abbdfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:18.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.071+0000 7f83cbbf9740 1 --2- 172.21.15.72:0/2847295151 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f14abc0b00 0x55f14abbe500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:18.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.071+0000 7f83cbbf9740 1 --2- 172.21.15.72:0/2847295151 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f14abb7600 0x55f14abbda80 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:18.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.071+0000 7f83cbbf9740 1 -- 172.21.15.72:0/2847295151 >> 172.21.15.72:0/2847295151 conn(0x55f14abb1d30 msgr2=0x55f14abb3a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:38:18.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.071+0000 7f83a9ffb640 0 client.15927 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:38:18.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.071+0000 7f83cbbf9740 1 -- 172.21.15.72:0/2847295151 shutdown_connections 2024-03-22T01:38:18.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.071+0000 7f83cbbf9740 1 -- 172.21.15.72:0/2847295151 wait complete. 2024-03-22T01:38:18.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.071+0000 7f83cbbf9740 20 client.15927 trim_cache size 0 max 16384 2024-03-22T01:38:18.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.105+0000 7f83cbbf9740 1 Processor -- start 2024-03-22T01:38:18.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.106+0000 7f83cbbf9740 1 -- start start 2024-03-22T01:38:18.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.106+0000 7f83cbbf9740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f14abb7600 0x55f14abc01b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:18.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.106+0000 7f83cbbf9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f14abb7f20 0x55f14abcc690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:18.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.106+0000 7f83cbbf9740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f14abc0b00 0x55f14abb0e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:18.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.107+0000 7f83cbbf9740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f83bc13f150 con 0x55f14abc0b00 2024-03-22T01:38:18.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.107+0000 7f83cbbf9740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f83bc1419d0 con 0x55f14abb7600 2024-03-22T01:38:18.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.107+0000 7f83cbbf9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f83bc13f7c0 con 0x55f14abb7f20 2024-03-22T01:38:18.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.107+0000 7f83c2d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f14abc0b00 0x55f14abb0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:18.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.107+0000 7f83c2d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f14abc0b00 0x55f14abb0e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:41038/0 (socket says 172.21.15.72:41038) 2024-03-22T01:38:18.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.107+0000 7f83c2d9d640 1 -- 172.21.15.72:0/3578724667 learned_addr learned my addr 172.21.15.72:0/3578724667 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:38:18.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.107+0000 7f83c259c640 1 --2- 172.21.15.72:0/3578724667 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f14abb7600 0x55f14abc01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:18.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.107+0000 7f83c1d9b640 1 --2- 172.21.15.72:0/3578724667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f14abb7f20 0x55f14abcc690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:18.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.107+0000 7f83c2d9d640 1 -- 172.21.15.72:0/3578724667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f14abb7f20 msgr2=0x55f14abcc690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:18.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.107+0000 7f83c2d9d640 1 --2- 172.21.15.72:0/3578724667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f14abb7f20 0x55f14abcc690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:18.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.107+0000 7f83c2d9d640 1 -- 172.21.15.72:0/3578724667 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f14abb7600 msgr2=0x55f14abc01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:18.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.107+0000 7f83c2d9d640 1 --2- 172.21.15.72:0/3578724667 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f14abb7600 0x55f14abc01b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:18.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.107+0000 7f83c2d9d640 1 -- 172.21.15.72:0/3578724667 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f14abbca30 con 0x55f14abc0b00 2024-03-22T01:38:18.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.108+0000 7f83c2d9d640 1 --2- 172.21.15.72:0/3578724667 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f14abc0b00 0x55f14abb0e10 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f83c40672c0 tx=0x7f83c4067580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7244bc96b4614475 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:38:18.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.108+0000 7f83abfff640 1 -- 172.21.15.72:0/3578724667 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c407eb90 con 0x55f14abc0b00 2024-03-22T01:38:18.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.108+0000 7f83abfff640 1 -- 172.21.15.72:0/3578724667 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83c4069050 con 0x55f14abc0b00 2024-03-22T01:38:18.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.108+0000 7f83abfff640 1 -- 172.21.15.72:0/3578724667 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c4086d90 con 0x55f14abc0b00 2024-03-22T01:38:18.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.109+0000 7f83cbbf9740 1 -- 172.21.15.72:0/3578724667 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f14abc0b00 msgr2=0x55f14abb0e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:18.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.109+0000 7f83cbbf9740 1 --2- 172.21.15.72:0/3578724667 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f14abc0b00 0x55f14abb0e10 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f83c40672c0 tx=0x7f83c4067580 comp rx=0 tx=0).stop 2024-03-22T01:38:18.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.109+0000 7f83cbbf9740 1 -- 172.21.15.72:0/3578724667 shutdown_connections 2024-03-22T01:38:18.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.109+0000 7f83cbbf9740 1 --2- 172.21.15.72:0/3578724667 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f14abc0b00 0x55f14abb0e10 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:18.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.110+0000 7f83cbbf9740 1 --2- 172.21.15.72:0/3578724667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f14abb7f20 0x55f14abcc690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:18.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.110+0000 7f83cbbf9740 1 --2- 172.21.15.72:0/3578724667 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f14abb7600 0x55f14abc01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:18.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.110+0000 7f83cbbf9740 1 -- 172.21.15.72:0/3578724667 >> 172.21.15.72:0/3578724667 conn(0x55f14abb1d30 msgr2=0x55f14abb2110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:38:18.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.110+0000 7f83cbbf9740 1 -- 172.21.15.72:0/3578724667 shutdown_connections 2024-03-22T01:38:18.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.110+0000 7f83cbbf9740 1 -- 172.21.15.72:0/3578724667 wait complete. 2024-03-22T01:38:18.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.110+0000 7f83cbbf9740 1 -- start start 2024-03-22T01:38:18.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.111+0000 7f83cbbf9740 1 Processor -- start 2024-03-22T01:38:18.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.111+0000 7f83cbbf9740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f14abb7600 0x55f14abc4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:18.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.111+0000 7f83cbbf9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f14abb7f20 0x55f14abc5210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:18.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.111+0000 7f83cbbf9740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f14abc0b00 0x55f14ac3e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:18.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.111+0000 7f83cbbf9740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f83bc13fe90 con 0x55f14abc0b00 2024-03-22T01:38:18.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.111+0000 7f83cbbf9740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f83bc141300 con 0x55f14abb7600 2024-03-22T01:38:18.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.111+0000 7f83cbbf9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f83bc140560 con 0x55f14abb7f20 2024-03-22T01:38:18.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.111+0000 7f83cbbf9740 10 client.0 _ll_register_callbacks cb 0x7f83b4004550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:38:18.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.111+0000 7f83c1d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f14abb7f20 0x55f14abc5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:18.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.111+0000 7f83c1d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f14abb7f20 0x55f14abc5210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.72:36306/0 (socket says 172.21.15.72:36306) 2024-03-22T01:38:18.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.111+0000 7f83c1d9b640 1 -- 172.21.15.72:0/2074818905 learned_addr learned my addr 172.21.15.72:0/2074818905 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:38:18.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.112+0000 7f83c2d9d640 1 --2- 172.21.15.72:0/2074818905 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f14abc0b00 0x55f14ac3e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:18.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.112+0000 7f83c259c640 1 --2- 172.21.15.72:0/2074818905 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f14abb7600 0x55f14abc4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:18.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.112+0000 7f83c2d9d640 1 -- 172.21.15.72:0/2074818905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f14abb7f20 msgr2=0x55f14abc5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:18.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.112+0000 7f83c2d9d640 1 --2- 172.21.15.72:0/2074818905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f14abb7f20 0x55f14abc5210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:18.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.112+0000 7f83c2d9d640 1 -- 172.21.15.72:0/2074818905 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f14abb7600 msgr2=0x55f14abc4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:18.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.112+0000 7f83c2d9d640 1 --2- 172.21.15.72:0/2074818905 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f14abb7600 0x55f14abc4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:18.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.112+0000 7f83c2d9d640 1 -- 172.21.15.72:0/2074818905 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f14a7ef0f0 con 0x55f14abc0b00 2024-03-22T01:38:18.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.112+0000 7f83c2d9d640 1 --2- 172.21.15.72:0/2074818905 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f14abc0b00 0x55f14ac3e980 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x55f14abace70 tx=0x7f83c407ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ee8c94d6e4f3709 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:38:18.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.112+0000 7f83a9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:38:18.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.113+0000 7f83a9ffb640 1 -- 172.21.15.72:0/2074818905 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c409a070 con 0x55f14abc0b00 2024-03-22T01:38:18.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.113+0000 7f83a9ffb640 1 -- 172.21.15.72:0/2074818905 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f83c4092660 con 0x55f14abc0b00 2024-03-22T01:38:18.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.113+0000 7f83cbbf9740 1 -- 172.21.15.72:0/2074818905 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f83c4069050 con 0x55f14abc0b00 2024-03-22T01:38:18.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.113+0000 7f83a9ffb640 1 -- 172.21.15.72:0/2074818905 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f83c408a070 con 0x55f14abc0b00 2024-03-22T01:38:18.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.113+0000 7f83cbbf9740 10 client.15933 fetch_fsmap learned FSMap version 21 2024-03-22T01:38:18.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.113+0000 7f83cbbf9740 1 -- 172.21.15.72:0/2074818905 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55f14ac3ffa0 con 0x55f14abc0b00 2024-03-22T01:38:18.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.113+0000 7f83a9ffb640 1 -- 172.21.15.72:0/2074818905 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83c4069050 con 0x55f14abc0b00 2024-03-22T01:38:18.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.113+0000 7f83a9ffb640 1 -- 172.21.15.72:0/2074818905 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f83c409d020 con 0x55f14abc0b00 2024-03-22T01:38:18.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.114+0000 7f83cbbf9740 10 client.15933 fetch_fsmap finished waiting for FSMap version 21 2024-03-22T01:38:18.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.114+0000 7f83cbbf9740 10 client.15933 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T01:38:18.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.114+0000 7f83cbbf9740 20 client.15933 populate_metadata read hostname 'smithi072' 2024-03-22T01:38:18.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.114+0000 7f83cbbf9740 1 --2- 172.21.15.72:0/2074818905 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55f14ac40510 0x55f14ac42900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:18.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.114+0000 7f83cbbf9740 4 client.15933 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T01:38:18.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.114+0000 7f83cbbf9740 1 -- 172.21.15.72:0/2074818905 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f14ab23150 con 0x55f14ac40510 2024-03-22T01:38:18.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.114+0000 7f83c259c640 1 --2- 172.21.15.72:0/2074818905 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55f14ac40510 0x55f14ac42900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:18.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.117+0000 7f83c259c640 1 --2- 172.21.15.72:0/2074818905 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55f14ac40510 0x55f14ac42900 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cbe4c50b80f93975 server_cookie=629af5364c4730ff in_seq=0 out_seq=0 2024-03-22T01:38:18.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.121+0000 7f83a9ffb640 10 client.15933 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:38:18.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:38:18 smithi072 ceph-mon[32414]: pgmap v52: 129 pgs: 129 active+clean; 7.1 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 583 KiB/s wr, 58 op/s 2024-03-22T01:38:18.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.123+0000 7f83a9ffb640 1 -- 172.21.15.72:0/2074818905 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55f14ab23150 con 0x55f14ac40510 2024-03-22T01:38:18.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.123+0000 7f83a9ffb640 10 client.15933 handle_command_reply: tid=0 2024-03-22T01:38:18.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.123+0000 7f83cbbf9740 1 client.15933 shutdown 2024-03-22T01:38:18.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.123+0000 7f83cbbf9740 1 -- 172.21.15.72:0/2074818905 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f14abc0b00 msgr2=0x55f14ac3e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:18.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.123+0000 7f83cbbf9740 1 --2- 172.21.15.72:0/2074818905 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f14abc0b00 0x55f14ac3e980 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x55f14abace70 tx=0x7f83c407ee90 comp rx=0 tx=0).stop 2024-03-22T01:38:18.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.124+0000 7f83cbbf9740 1 -- 172.21.15.72:0/2074818905 shutdown_connections 2024-03-22T01:38:18.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.124+0000 7f83cbbf9740 1 --2- 172.21.15.72:0/2074818905 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55f14ac40510 0x55f14ac42900 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:18.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.124+0000 7f83cbbf9740 1 --2- 172.21.15.72:0/2074818905 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f14abc0b00 0x55f14ac3e980 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:18.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.124+0000 7f83cbbf9740 1 --2- 172.21.15.72:0/2074818905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f14abb7f20 0x55f14abc5210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:18.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.124+0000 7f83a9ffb640 0 client.15933 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:38:18.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.124+0000 7f83cbbf9740 1 --2- 172.21.15.72:0/2074818905 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f14abb7600 0x55f14abc4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:18.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.124+0000 7f83cbbf9740 1 -- 172.21.15.72:0/2074818905 >> 172.21.15.72:0/2074818905 conn(0x55f14abb1d30 msgr2=0x55f14abb38c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:38:18.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.124+0000 7f83cbbf9740 1 -- 172.21.15.72:0/2074818905 shutdown_connections 2024-03-22T01:38:18.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.125+0000 7f83cbbf9740 1 -- 172.21.15.72:0/2074818905 wait complete. 2024-03-22T01:38:18.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.125+0000 7f83cbbf9740 20 client.15933 trim_cache size 0 max 16384 2024-03-22T01:38:18.126 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-22T01:38:18.126 INFO:teuthology.orchestra.run.smithi072.stdout: "status": "no active scrubs running", 2024-03-22T01:38:18.126 INFO:teuthology.orchestra.run.smithi072.stdout: "scrubs": {} 2024-03-22T01:38:18.126 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-22T01:38:18.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.128+0000 7f83c359e640 1 -- 172.21.15.72:0/2617550542 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f83ac0a7560 msgr2=0x7f83ac0a9a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:18.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.128+0000 7f83c359e640 1 --2- 172.21.15.72:0/2617550542 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f83ac0a7560 0x7f83ac0a9a20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f83b4006010 tx=0x7f83b402df90 comp rx=0 tx=0).stop 2024-03-22T01:38:18.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.128+0000 7f83c359e640 1 -- 172.21.15.72:0/2617550542 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83bc005670 msgr2=0x7f83bc12a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:18.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.128+0000 7f83c359e640 1 --2- 172.21.15.72:0/2617550542 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83bc005670 0x7f83bc12a950 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f83b800e9f0 tx=0x7f83b800eeb0 comp rx=0 tx=0).stop 2024-03-22T01:38:18.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.128+0000 7f83c359e640 1 -- 172.21.15.72:0/2617550542 shutdown_connections 2024-03-22T01:38:18.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.128+0000 7f83c359e640 1 --2- 172.21.15.72:0/2617550542 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f83ac0a7560 0x7f83ac0a9a20 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:18.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.128+0000 7f83c359e640 1 --2- 172.21.15.72:0/2617550542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83bc09be10 0x7f83bc12ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:18.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.128+0000 7f83c359e640 1 --2- 172.21.15.72:0/2617550542 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83bc005670 0x7f83bc12a950 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:18.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.128+0000 7f83c359e640 1 --2- 172.21.15.72:0/2617550542 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83bc004cb0 0x7f83bc12a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:18.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.128+0000 7f83c359e640 1 -- 172.21.15.72:0/2617550542 >> 172.21.15.72:0/2617550542 conn(0x7f83bc00a740 msgr2=0x7f83bc093aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:38:18.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.128+0000 7f83c359e640 1 -- 172.21.15.72:0/2617550542 shutdown_connections 2024-03-22T01:38:18.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.128+0000 7f83c359e640 1 -- 172.21.15.72:0/2617550542 wait complete. 2024-03-22T01:38:18.149 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-22T01:38:18.149 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T01:38:18.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.291+0000 7fbfc221c640 1 -- 172.21.15.72:0/1916523613 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbfbc106b70 msgr2=0x7fbfbc106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:18.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.291+0000 7fbfc221c640 1 --2- 172.21.15.72:0/1916523613 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbfbc106b70 0x7fbfbc106f50 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fbfac009930 tx=0x7fbfac02bc30 comp rx=0 tx=0).stop 2024-03-22T01:38:18.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.292+0000 7fbfc221c640 1 -- 172.21.15.72:0/1916523613 shutdown_connections 2024-03-22T01:38:18.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.293+0000 7fbfc221c640 1 --2- 172.21.15.72:0/1916523613 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbfbc100a50 0x7fbfbc10ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:18.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.293+0000 7fbfc221c640 1 --2- 172.21.15.72:0/1916523613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbfbc100090 0x7fbfbc100510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:18.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.293+0000 7fbfc221c640 1 --2- 172.21.15.72:0/1916523613 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbfbc106b70 0x7fbfbc106f50 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:18.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.293+0000 7fbfc221c640 1 -- 172.21.15.72:0/1916523613 >> 172.21.15.72:0/1916523613 conn(0x7fbfbc076fb0 msgr2=0x7fbfbc0fe3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:38:18.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.293+0000 7fbfc221c640 1 -- 172.21.15.72:0/1916523613 shutdown_connections 2024-03-22T01:38:18.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.293+0000 7fbfc221c640 1 -- 172.21.15.72:0/1916523613 wait complete. 2024-03-22T01:38:18.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.294+0000 7fbfc221c640 1 Processor -- start 2024-03-22T01:38:18.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.294+0000 7fbfc221c640 1 -- start start 2024-03-22T01:38:18.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.295+0000 7fbfc221c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbfbc100090 0x7fbfbc198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:18.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.295+0000 7fbfc221c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbfbc100a50 0x7fbfbc198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:18.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.295+0000 7fbfc221c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbfbc106b70 0x7fbfbc19cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:18.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.295+0000 7fbfc221c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fbfbc112080 con 0x7fbfbc100090 2024-03-22T01:38:18.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.295+0000 7fbfc221c640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fbfbc111f00 con 0x7fbfbc106b70 2024-03-22T01:38:18.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.295+0000 7fbfbaffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbfbc100a50 0x7fbfbc198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:18.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.295+0000 7fbfbb7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbfbc100090 0x7fbfbc198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:18.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.295+0000 7fbfbaffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbfbc100a50 0x7fbfbc198a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.72:36326/0 (socket says 172.21.15.72:36326) 2024-03-22T01:38:18.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.295+0000 7fbfbaffd640 1 -- 172.21.15.72:0/3018346145 learned_addr learned my addr 172.21.15.72:0/3018346145 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:38:18.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.295+0000 7fbfc221c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbfbc112200 con 0x7fbfbc100a50 2024-03-22T01:38:18.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.295+0000 7fbfbbfff640 1 --2- 172.21.15.72:0/3018346145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbfbc106b70 0x7fbfbc19cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:18.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.296+0000 7fbfbaffd640 1 -- 172.21.15.72:0/3018346145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbfbc106b70 msgr2=0x7fbfbc19cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:18.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.296+0000 7fbfbaffd640 1 --2- 172.21.15.72:0/3018346145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbfbc106b70 0x7fbfbc19cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:18.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.296+0000 7fbfbaffd640 1 -- 172.21.15.72:0/3018346145 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbfbc100090 msgr2=0x7fbfbc198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:18.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.296+0000 7fbfbaffd640 1 --2- 172.21.15.72:0/3018346145 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbfbc100090 0x7fbfbc198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:18.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.296+0000 7fbfbaffd640 1 -- 172.21.15.72:0/3018346145 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfbc19d4a0 con 0x7fbfbc100a50 2024-03-22T01:38:18.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.296+0000 7fbfbaffd640 1 --2- 172.21.15.72:0/3018346145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbfbc100a50 0x7fbfbc198a40 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbfa400e9f0 tx=0x7fbfa400edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:38:18.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.296+0000 7fbfb8ff9640 1 -- 172.21.15.72:0/3018346145 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfa400ccc0 con 0x7fbfbc100a50 2024-03-22T01:38:18.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.296+0000 7fbfb8ff9640 1 -- 172.21.15.72:0/3018346145 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbfa400ce60 con 0x7fbfbc100a50 2024-03-22T01:38:18.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.296+0000 7fbfb8ff9640 1 -- 172.21.15.72:0/3018346145 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfa4010450 con 0x7fbfbc100a50 2024-03-22T01:38:18.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.296+0000 7fbfc221c640 1 -- 172.21.15.72:0/3018346145 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfbc19d790 con 0x7fbfbc100a50 2024-03-22T01:38:18.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.297+0000 7fbfc221c640 1 -- 172.21.15.72:0/3018346145 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfbc1a5020 con 0x7fbfbc100a50 2024-03-22T01:38:18.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.298+0000 7fbfc221c640 1 -- 172.21.15.72:0/3018346145 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfbc068180 con 0x7fbfbc100a50 2024-03-22T01:38:18.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.300+0000 7fbfb8ff9640 1 -- 172.21.15.72:0/3018346145 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 137985+0+0 (secure 0 0 0) 0x7fbfa4005020 con 0x7fbfbc100a50 2024-03-22T01:38:18.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.302+0000 7fbfb8ff9640 1 --2- 172.21.15.72:0/3018346145 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fbfa80a3110 0x7fbfa80a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:18.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.302+0000 7fbfbb7fe640 1 --2- 172.21.15.72:0/3018346145 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fbfa80a3110 0x7fbfa80a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:18.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.302+0000 7fbfb8ff9640 1 -- 172.21.15.72:0/3018346145 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fbfa40cffd0 con 0x7fbfbc100a50 2024-03-22T01:38:18.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.303+0000 7fbfbb7fe640 1 --2- 172.21.15.72:0/3018346145 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fbfa80a3110 0x7fbfa80a55d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fbfac006010 tx=0x7fbfac000bc0 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:38:18.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.306+0000 7fbfb8ff9640 1 -- 172.21.15.72:0/3018346145 <== mon.1 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbfa409c710 con 0x7fbfbc100a50 2024-03-22T01:38:18.357 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:38:18 smithi089 ceph-mon[35770]: pgmap v52: 129 pgs: 129 active+clean; 7.1 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 583 KiB/s wr, 58 op/s 2024-03-22T01:38:18.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:38:18 smithi098 ceph-mon[36468]: pgmap v52: 129 pgs: 129 active+clean; 7.1 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 583 KiB/s wr, 58 op/s 2024-03-22T01:38:18.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.832+0000 7fbfc221c640 1 -- 172.21.15.72:0/3018346145 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbfbc10a050 con 0x7fbfbc100a50 2024-03-22T01:38:18.837 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:38:18.837 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6835/3077216308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3077216308},{"type":"v1","addr":"172.21.15.72:6835","nonce":3077216308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2460254143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":2460254143},{"type":"v1","addr":"172.21.15.72:6837","nonce":2460254143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6839/2026432819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6838","nonce":2026432819},{"type":"v1","addr":"172.21.15.72:6839","nonce":2026432819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6841/400094545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6840","nonce":400094545},{"type":"v1","addr":"172.21.15.72:6841","nonce":400094545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6833/1578701171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":1578701171},{"type":"v1","addr":"172.21.15.89:6833","nonce":1578701171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/2703047213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2703047213},{"type":"v1","addr":"172.21.15.98:6833","nonce":2703047213}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2589468787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2589468787},{"type":"v1","addr":"172.21.15.89:6835","nonce":2589468787}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/92657636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":92657636},{"type":"v1","addr":"172.21.15.98:6835","nonce":92657636}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1662941782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1662941782},{"type":"v1","addr":"172.21.15.89:6837","nonce":1662941782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24490,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/781860371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":781860371},{"type":"v1","addr":"172.21.15.98:6837","nonce":781860371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-22T01:34:14.815677+0000","modified":"2024-03-22T01:34:39.842959+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24524},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3192173650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3192173650},{"type":"v1","addr":"172.21.15.98:6839","nonce":3192173650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.89:6839/149200280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":149200280},{"type":"v1","addr":"172.21.15.89:6839","nonce":149200280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24524]},"id":1}]} 2024-03-22T01:38:18.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.837+0000 7fbfb8ff9640 1 -- 172.21.15.72:0/3018346145 <== mon.1 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11785 (secure 0 0 0) 0x7fbfa409c010 con 0x7fbfbc100a50 2024-03-22T01:38:18.837 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 21 2024-03-22T01:38:18.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.842+0000 7fbfc221c640 1 -- 172.21.15.72:0/3018346145 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fbfa80a3110 msgr2=0x7fbfa80a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:18.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.842+0000 7fbfc221c640 1 --2- 172.21.15.72:0/3018346145 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fbfa80a3110 0x7fbfa80a55d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fbfac006010 tx=0x7fbfac000bc0 comp rx=0 tx=0).stop 2024-03-22T01:38:18.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.842+0000 7fbfc221c640 1 -- 172.21.15.72:0/3018346145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbfbc100a50 msgr2=0x7fbfbc198a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:18.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.842+0000 7fbfc221c640 1 --2- 172.21.15.72:0/3018346145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbfbc100a50 0x7fbfbc198a40 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbfa400e9f0 tx=0x7fbfa400edf0 comp rx=0 tx=0).stop 2024-03-22T01:38:18.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.843+0000 7fbfc221c640 1 -- 172.21.15.72:0/3018346145 shutdown_connections 2024-03-22T01:38:18.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.843+0000 7fbfc221c640 1 --2- 172.21.15.72:0/3018346145 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fbfa80a3110 0x7fbfa80a55d0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:18.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.843+0000 7fbfc221c640 1 --2- 172.21.15.72:0/3018346145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbfbc106b70 0x7fbfbc19cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:18.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.843+0000 7fbfc221c640 1 --2- 172.21.15.72:0/3018346145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbfbc100a50 0x7fbfbc198a40 secure :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbfa400e9f0 tx=0x7fbfa400edf0 comp rx=0 tx=0).stop 2024-03-22T01:38:18.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.843+0000 7fbfc221c640 1 --2- 172.21.15.72:0/3018346145 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbfbc100090 0x7fbfbc198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:18.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.843+0000 7fbfc221c640 1 -- 172.21.15.72:0/3018346145 >> 172.21.15.72:0/3018346145 conn(0x7fbfbc076fb0 msgr2=0x7fbfbc104c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:38:18.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.845+0000 7fbfc221c640 1 -- 172.21.15.72:0/3018346145 shutdown_connections 2024-03-22T01:38:18.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:18.846+0000 7fbfc221c640 1 -- 172.21.15.72:0/3018346145 wait complete. 2024-03-22T01:38:18.874 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-22T01:38:19.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.200+0000 7f51be1c8640 1 -- 172.21.15.72:0/4112870641 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51b8101970 msgr2=0x7f51b8101df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:19.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.200+0000 7f51be1c8640 1 --2- 172.21.15.72:0/4112870641 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51b8101970 0x7f51b8101df0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f51a8009930 tx=0x7f51a802bc30 comp rx=0 tx=0).stop 2024-03-22T01:38:19.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.201+0000 7f51be1c8640 1 -- 172.21.15.72:0/4112870641 shutdown_connections 2024-03-22T01:38:19.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.201+0000 7f51be1c8640 1 --2- 172.21.15.72:0/4112870641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51b8102330 0x7f51b810e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:19.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.201+0000 7f51be1c8640 1 --2- 172.21.15.72:0/4112870641 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51b8101970 0x7f51b8101df0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:19.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.201+0000 7f51be1c8640 1 --2- 172.21.15.72:0/4112870641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51b8107980 0x7f51b8107d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:19.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.201+0000 7f51be1c8640 1 -- 172.21.15.72:0/4112870641 >> 172.21.15.72:0/4112870641 conn(0x7f51b80fd480 msgr2=0x7f51b80ff8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:38:19.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.201+0000 7f51be1c8640 1 -- 172.21.15.72:0/4112870641 shutdown_connections 2024-03-22T01:38:19.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.201+0000 7f51be1c8640 1 -- 172.21.15.72:0/4112870641 wait complete. 2024-03-22T01:38:19.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.201+0000 7f51be1c8640 1 Processor -- start 2024-03-22T01:38:19.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.201+0000 7f51be1c8640 1 -- start start 2024-03-22T01:38:19.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.202+0000 7f51be1c8640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51b8101970 0x7f51b8198370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:19.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.202+0000 7f51be1c8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51b8102330 0x7f51b81988b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:19.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.202+0000 7f51be1c8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51b8107980 0x7f51b819cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:19.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.202+0000 7f51be1c8640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f51b8111e10 con 0x7f51b8107980 2024-03-22T01:38:19.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.202+0000 7f51be1c8640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f51b8111c90 con 0x7f51b8101970 2024-03-22T01:38:19.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.202+0000 7f51be1c8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f51b8111f90 con 0x7f51b8102330 2024-03-22T01:38:19.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.202+0000 7f51bd9c7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51b8107980 0x7f51b819cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:19.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.202+0000 7f51bd9c7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51b8107980 0x7f51b819cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:41096/0 (socket says 172.21.15.72:41096) 2024-03-22T01:38:19.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.202+0000 7f51bd9c7640 1 -- 172.21.15.72:0/3853563326 learned_addr learned my addr 172.21.15.72:0/3853563326 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:38:19.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.202+0000 7f51bd1c6640 1 --2- 172.21.15.72:0/3853563326 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51b8101970 0x7f51b8198370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:19.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.202+0000 7f51bc9c5640 1 --2- 172.21.15.72:0/3853563326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51b8102330 0x7f51b81988b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:19.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.202+0000 7f51bd9c7640 1 -- 172.21.15.72:0/3853563326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51b8102330 msgr2=0x7f51b81988b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:19.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.202+0000 7f51bd9c7640 1 --2- 172.21.15.72:0/3853563326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51b8102330 0x7f51b81988b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:19.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.202+0000 7f51bd9c7640 1 -- 172.21.15.72:0/3853563326 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51b8101970 msgr2=0x7f51b8198370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:19.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.202+0000 7f51bd9c7640 1 --2- 172.21.15.72:0/3853563326 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51b8101970 0x7f51b8198370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:19.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.202+0000 7f51bd9c7640 1 -- 172.21.15.72:0/3853563326 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51b819d360 con 0x7f51b8107980 2024-03-22T01:38:19.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.202+0000 7f51bd1c6640 1 --2- 172.21.15.72:0/3853563326 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51b8101970 0x7f51b8198370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:38:19.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.202+0000 7f51bd9c7640 1 --2- 172.21.15.72:0/3853563326 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51b8107980 0x7f51b819cc30 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f51b000b970 tx=0x7f51b000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:38:19.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.203+0000 7f51ae7fc640 1 -- 172.21.15.72:0/3853563326 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b0019070 con 0x7f51b8107980 2024-03-22T01:38:19.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.203+0000 7f51ae7fc640 1 -- 172.21.15.72:0/3853563326 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51b0014440 con 0x7f51b8107980 2024-03-22T01:38:19.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.203+0000 7f51ae7fc640 1 -- 172.21.15.72:0/3853563326 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b00075d0 con 0x7f51b8107980 2024-03-22T01:38:19.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.203+0000 7f51be1c8640 1 -- 172.21.15.72:0/3853563326 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51b819d650 con 0x7f51b8107980 2024-03-22T01:38:19.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.203+0000 7f51be1c8640 1 -- 172.21.15.72:0/3853563326 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51b8073dc0 con 0x7f51b8107980 2024-03-22T01:38:19.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.205+0000 7f51ae7fc640 1 -- 172.21.15.72:0/3853563326 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 25) ==== 137985+0+0 (secure 0 0 0) 0x7f51b001c070 con 0x7f51b8107980 2024-03-22T01:38:19.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.206+0000 7f51c2010740 1 Processor -- start 2024-03-22T01:38:19.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.206+0000 7f51ae7fc640 1 --2- 172.21.15.72:0/3853563326 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f51a40a3110 0x7f51a40a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:19.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.206+0000 7f51c2010740 1 -- start start 2024-03-22T01:38:19.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.206+0000 7f51bc9c5640 1 --2- 172.21.15.72:0/3853563326 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f51a40a3110 0x7f51a40a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:19.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.206+0000 7f51ae7fc640 1 -- 172.21.15.72:0/3853563326 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f51b00cf010 con 0x7f51b8107980 2024-03-22T01:38:19.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.206+0000 7f51c2010740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564b81ec4200 0x564b81ec45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:19.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.206+0000 7f51c2010740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b81ec4b20 0x564b81ecd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:19.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.206+0000 7f51c2010740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b81ecd700 0x564b81ecfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:19.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.206+0000 7f51c2010740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x564b81e1f8d0 con 0x564b81ec4200 2024-03-22T01:38:19.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.206+0000 7f51c2010740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x564b81dc7e00 con 0x564b81ecd700 2024-03-22T01:38:19.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.206+0000 7f51c2010740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x564b81e1b880 con 0x564b81ec4b20 2024-03-22T01:38:19.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.207+0000 7f51bd1c6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564b81ec4200 0x564b81ec45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:19.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.207+0000 7f51bd1c6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564b81ec4200 0x564b81ec45e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:41112/0 (socket says 172.21.15.72:41112) 2024-03-22T01:38:19.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.207+0000 7f51bd1c6640 1 -- 172.21.15.72:0/2167452428 learned_addr learned my addr 172.21.15.72:0/2167452428 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:38:19.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.207+0000 7f51bd9c7640 1 --2- 172.21.15.72:0/2167452428 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b81ecd700 0x564b81ecfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:19.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.207+0000 7f51bc9c5640 1 --2- 172.21.15.72:0/2167452428 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b81ec4b20 0x564b81ecd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:19.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.207+0000 7f51bc9c5640 1 --2- 172.21.15.72:0/3853563326 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f51a40a3110 0x7f51a40a55d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f51a8006010 tx=0x7f51a8004f10 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:38:19.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.207+0000 7f51bd1c6640 1 -- 172.21.15.72:0/2167452428 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b81ec4b20 msgr2=0x564b81ecd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:19.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.207+0000 7f51bd1c6640 1 --2- 172.21.15.72:0/2167452428 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b81ec4b20 0x564b81ecd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:19.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.207+0000 7f51bd1c6640 1 -- 172.21.15.72:0/2167452428 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b81ecd700 msgr2=0x564b81ecfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:19.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.207+0000 7f51bd1c6640 1 --2- 172.21.15.72:0/2167452428 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b81ecd700 0x564b81ecfaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:19.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.207+0000 7f51bd1c6640 1 -- 172.21.15.72:0/2167452428 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564b81b333f0 con 0x564b81ec4200 2024-03-22T01:38:19.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.207+0000 7f51bd1c6640 1 --2- 172.21.15.72:0/2167452428 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564b81ec4200 0x564b81ec45e0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f51b4009e70 tx=0x7f51b400e5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1605b7dfcabf1efe server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:38:19.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.207+0000 7f51a37fe640 1 -- 172.21.15.72:0/2167452428 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b4017070 con 0x564b81ec4200 2024-03-22T01:38:19.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.208+0000 7f51a37fe640 1 -- 172.21.15.72:0/2167452428 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51b4013950 con 0x564b81ec4200 2024-03-22T01:38:19.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.208+0000 7f51a37fe640 1 -- 172.21.15.72:0/2167452428 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b4012900 con 0x564b81ec4200 2024-03-22T01:38:19.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.208+0000 7f51c2010740 1 -- 172.21.15.72:0/2167452428 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564b81ec4200 msgr2=0x564b81ec45e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:19.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.208+0000 7f51c2010740 1 --2- 172.21.15.72:0/2167452428 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564b81ec4200 0x564b81ec45e0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f51b4009e70 tx=0x7f51b400e5a0 comp rx=0 tx=0).stop 2024-03-22T01:38:19.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.209+0000 7f51c2010740 1 -- 172.21.15.72:0/2167452428 shutdown_connections 2024-03-22T01:38:19.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.209+0000 7f51c2010740 1 --2- 172.21.15.72:0/2167452428 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b81ecd700 0x564b81ecfaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:19.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.209+0000 7f51c2010740 1 --2- 172.21.15.72:0/2167452428 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b81ec4b20 0x564b81ecd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:19.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.209+0000 7f51c2010740 1 --2- 172.21.15.72:0/2167452428 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564b81ec4200 0x564b81ec45e0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:19.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.209+0000 7f51c2010740 1 -- 172.21.15.72:0/2167452428 >> 172.21.15.72:0/2167452428 conn(0x564b81ebe930 msgr2=0x564b81ebed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:38:19.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.209+0000 7f51c2010740 1 -- 172.21.15.72:0/2167452428 shutdown_connections 2024-03-22T01:38:19.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.209+0000 7f51c2010740 1 -- 172.21.15.72:0/2167452428 wait complete. 2024-03-22T01:38:19.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.209+0000 7f51c2010740 1 -- start start 2024-03-22T01:38:19.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.209+0000 7f51c2010740 1 Processor -- start 2024-03-22T01:38:19.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.210+0000 7f51c2010740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b81ec4200 0x564b81eca720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:19.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.210+0000 7f51c2010740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b81ec4b20 0x564b81ecac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:19.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.210+0000 7f51c2010740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564b81ecd700 0x564b81ecb1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:19.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.210+0000 7f51c2010740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x564b81ed04f0 con 0x564b81ecd700 2024-03-22T01:38:19.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.210+0000 7f51c2010740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x564b81ddb300 con 0x564b81ec4200 2024-03-22T01:38:19.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.210+0000 7f51c2010740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x564b81ed0670 con 0x564b81ec4b20 2024-03-22T01:38:19.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.210+0000 7f51c2010740 10 client.0 _ll_register_callbacks cb 0x564b81e16e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:38:19.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.210+0000 7f51bd1c6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b81ec4200 0x564b81eca720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:19.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.210+0000 7f51bd9c7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564b81ecd700 0x564b81ecb1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:19.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.210+0000 7f51bd9c7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564b81ecd700 0x564b81ecb1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:41124/0 (socket says 172.21.15.72:41124) 2024-03-22T01:38:19.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.210+0000 7f51bd9c7640 1 -- 172.21.15.72:0/2531723816 learned_addr learned my addr 172.21.15.72:0/2531723816 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:38:19.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.211+0000 7f51bc9c5640 1 --2- 172.21.15.72:0/2531723816 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b81ec4b20 0x564b81ecac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:19.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.211+0000 7f51bd9c7640 1 -- 172.21.15.72:0/2531723816 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b81ec4b20 msgr2=0x564b81ecac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:19.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.211+0000 7f51bd9c7640 1 --2- 172.21.15.72:0/2531723816 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b81ec4b20 0x564b81ecac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:19.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.211+0000 7f51bd9c7640 1 -- 172.21.15.72:0/2531723816 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b81ec4200 msgr2=0x564b81eca720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:19.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.211+0000 7f51bd9c7640 1 --2- 172.21.15.72:0/2531723816 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b81ec4200 0x564b81eca720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:19.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.211+0000 7f51bd9c7640 1 -- 172.21.15.72:0/2531723816 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564b81bfd7d0 con 0x564b81ecd700 2024-03-22T01:38:19.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.211+0000 7f51bd1c6640 1 --2- 172.21.15.72:0/2531723816 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b81ec4200 0x564b81eca720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:38:19.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.211+0000 7f51bd9c7640 1 --2- 172.21.15.72:0/2531723816 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564b81ecd700 0x564b81ecb1a0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f51b00975c0 tx=0x7f51b009f2b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8c2eccc2788e9eb server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:38:19.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.211+0000 7f51a0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:38:19.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.211+0000 7f51a0ff9640 1 -- 172.21.15.72:0/2531723816 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b00b4060 con 0x564b81ecd700 2024-03-22T01:38:19.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.211+0000 7f51c2010740 1 -- 172.21.15.72:0/2531723816 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51b4013950 con 0x564b81ecd700 2024-03-22T01:38:19.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.211+0000 7f51a0ff9640 1 -- 172.21.15.72:0/2531723816 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f51b00145f0 con 0x564b81ecd700 2024-03-22T01:38:19.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.211+0000 7f51a0ff9640 1 -- 172.21.15.72:0/2531723816 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f51b0014a60 con 0x564b81ecd700 2024-03-22T01:38:19.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.211+0000 7f51c2010740 10 client.15951 fetch_fsmap learned FSMap version 21 2024-03-22T01:38:19.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.211+0000 7f51c2010740 1 -- 172.21.15.72:0/2531723816 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x564b81edc820 con 0x564b81ecd700 2024-03-22T01:38:19.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.212+0000 7f51a0ff9640 1 -- 172.21.15.72:0/2531723816 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51b4013950 con 0x564b81ecd700 2024-03-22T01:38:19.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.212+0000 7f51a0ff9640 1 -- 172.21.15.72:0/2531723816 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f51b00bb1d0 con 0x564b81ecd700 2024-03-22T01:38:19.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.212+0000 7f51c2010740 10 client.15951 fetch_fsmap finished waiting for FSMap version 21 2024-03-22T01:38:19.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.212+0000 7f51c2010740 10 client.15951 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T01:38:19.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.212+0000 7f51c2010740 20 client.15951 populate_metadata read hostname 'smithi072' 2024-03-22T01:38:19.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.212+0000 7f51c2010740 1 --2- 172.21.15.72:0/2531723816 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x564b81eddf70 0x564b81ee0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:19.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.212+0000 7f51c2010740 4 client.15951 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T01:38:19.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.212+0000 7f51c2010740 1 -- 172.21.15.72:0/2531723816 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564b81e2ce10 con 0x564b81eddf70 2024-03-22T01:38:19.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.212+0000 7f51bd1c6640 1 --2- 172.21.15.72:0/2531723816 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x564b81eddf70 0x564b81ee0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:19.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.213+0000 7f51a0ff9640 10 client.15951 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:38:19.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.213+0000 7f51bd1c6640 1 --2- 172.21.15.72:0/2531723816 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x564b81eddf70 0x564b81ee0400 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b522428063f1c1a2 server_cookie=3801097fb829fbbf in_seq=0 out_seq=0 2024-03-22T01:38:19.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.214+0000 7f51a0ff9640 1 -- 172.21.15.72:0/2531723816 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564b81e2ce10 con 0x564b81eddf70 2024-03-22T01:38:19.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.214+0000 7f51a0ff9640 10 client.15951 handle_command_reply: tid=0 2024-03-22T01:38:19.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.214+0000 7f51c2010740 1 client.15951 shutdown 2024-03-22T01:38:19.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.215+0000 7f51c2010740 1 -- 172.21.15.72:0/2531723816 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564b81ecd700 msgr2=0x564b81ecb1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:19.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.215+0000 7f51c2010740 1 --2- 172.21.15.72:0/2531723816 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564b81ecd700 0x564b81ecb1a0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f51b00975c0 tx=0x7f51b009f2b0 comp rx=0 tx=0).stop 2024-03-22T01:38:19.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.215+0000 7f51c2010740 1 -- 172.21.15.72:0/2531723816 shutdown_connections 2024-03-22T01:38:19.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.215+0000 7f51c2010740 1 --2- 172.21.15.72:0/2531723816 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x564b81eddf70 0x564b81ee0400 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:19.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.215+0000 7f51c2010740 1 --2- 172.21.15.72:0/2531723816 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564b81ecd700 0x564b81ecb1a0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:19.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.215+0000 7f51a0ff9640 0 client.15951 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:38:19.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.215+0000 7f51c2010740 1 --2- 172.21.15.72:0/2531723816 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b81ec4b20 0x564b81ecac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:19.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.215+0000 7f51c2010740 1 --2- 172.21.15.72:0/2531723816 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b81ec4200 0x564b81eca720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:19.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.215+0000 7f51c2010740 1 -- 172.21.15.72:0/2531723816 >> 172.21.15.72:0/2531723816 conn(0x564b81ebe930 msgr2=0x564b81ec0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:38:19.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.215+0000 7f51c2010740 1 -- 172.21.15.72:0/2531723816 shutdown_connections 2024-03-22T01:38:19.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.216+0000 7f51c2010740 1 -- 172.21.15.72:0/2531723816 wait complete. 2024-03-22T01:38:19.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.216+0000 7f51c2010740 20 client.15951 trim_cache size 0 max 16384 2024-03-22T01:38:19.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.229+0000 7f51c2010740 1 Processor -- start 2024-03-22T01:38:19.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.229+0000 7f51c2010740 1 -- start start 2024-03-22T01:38:19.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.230+0000 7f51c2010740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b81ec4200 0x564b81ecce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:19.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.230+0000 7f51c2010740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b81ec4b20 0x564b81ee5a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:19.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.230+0000 7f51c2010740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564b81ecd700 0x564b81ec7980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:19.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.230+0000 7f51c2010740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f51b81aad80 con 0x564b81ecd700 2024-03-22T01:38:19.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.230+0000 7f51c2010740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f51b81ad600 con 0x564b81ec4200 2024-03-22T01:38:19.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.230+0000 7f51c2010740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f51b81ab3f0 con 0x564b81ec4b20 2024-03-22T01:38:19.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.230+0000 7f51bd1c6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b81ec4200 0x564b81ecce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:19.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.230+0000 7f51bd9c7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564b81ecd700 0x564b81ec7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:19.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.230+0000 7f51bd1c6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b81ec4200 0x564b81ecce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.72:46936/0 (socket says 172.21.15.72:46936) 2024-03-22T01:38:19.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.230+0000 7f51bd1c6640 1 -- 172.21.15.72:0/2457390450 learned_addr learned my addr 172.21.15.72:0/2457390450 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:38:19.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.230+0000 7f51bc9c5640 1 --2- 172.21.15.72:0/2457390450 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b81ec4b20 0x564b81ee5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:19.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.230+0000 7f51bd1c6640 1 -- 172.21.15.72:0/2457390450 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b81ec4b20 msgr2=0x564b81ee5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:19.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.230+0000 7f51bd1c6640 1 --2- 172.21.15.72:0/2457390450 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b81ec4b20 0x564b81ee5a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:19.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.230+0000 7f51bd1c6640 1 -- 172.21.15.72:0/2457390450 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564b81ecd700 msgr2=0x564b81ec7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:19.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.230+0000 7f51bd1c6640 1 --2- 172.21.15.72:0/2457390450 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564b81ecd700 0x564b81ec7980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:19.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.230+0000 7f51bd1c6640 1 -- 172.21.15.72:0/2457390450 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564b81ec8c70 con 0x564b81ec4200 2024-03-22T01:38:19.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.230+0000 7f51bc9c5640 1 --2- 172.21.15.72:0/2457390450 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b81ec4b20 0x564b81ee5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:38:19.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.230+0000 7f51bd9c7640 1 --2- 172.21.15.72:0/2457390450 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564b81ecd700 0x564b81ec7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:38:19.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.230+0000 7f51bd1c6640 1 --2- 172.21.15.72:0/2457390450 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b81ec4200 0x564b81ecce50 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f51b4000c00 tx=0x7f51b401e040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=242cfe447b1f35b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:38:19.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.231+0000 7f51a2ffd640 1 -- 172.21.15.72:0/2457390450 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b4017040 con 0x564b81ec4200 2024-03-22T01:38:19.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.231+0000 7f51a2ffd640 1 -- 172.21.15.72:0/2457390450 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51b4013d60 con 0x564b81ec4200 2024-03-22T01:38:19.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.231+0000 7f51a2ffd640 1 -- 172.21.15.72:0/2457390450 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b4012550 con 0x564b81ec4200 2024-03-22T01:38:19.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.231+0000 7f51c2010740 1 -- 172.21.15.72:0/2457390450 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b81ec4200 msgr2=0x564b81ecce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:19.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.231+0000 7f51c2010740 1 --2- 172.21.15.72:0/2457390450 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b81ec4200 0x564b81ecce50 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f51b4000c00 tx=0x7f51b401e040 comp rx=0 tx=0).stop 2024-03-22T01:38:19.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.231+0000 7f51c2010740 1 -- 172.21.15.72:0/2457390450 shutdown_connections 2024-03-22T01:38:19.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.231+0000 7f51c2010740 1 --2- 172.21.15.72:0/2457390450 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564b81ecd700 0x564b81ec7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:19.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.231+0000 7f51c2010740 1 --2- 172.21.15.72:0/2457390450 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b81ec4b20 0x564b81ee5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:19.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.231+0000 7f51c2010740 1 --2- 172.21.15.72:0/2457390450 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b81ec4200 0x564b81ecce50 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:19.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.231+0000 7f51c2010740 1 -- 172.21.15.72:0/2457390450 >> 172.21.15.72:0/2457390450 conn(0x564b81ebe930 msgr2=0x564b81eb5220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:38:19.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.231+0000 7f51c2010740 1 -- 172.21.15.72:0/2457390450 shutdown_connections 2024-03-22T01:38:19.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.231+0000 7f51c2010740 1 -- 172.21.15.72:0/2457390450 wait complete. 2024-03-22T01:38:19.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.232+0000 7f51c2010740 1 -- start start 2024-03-22T01:38:19.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.232+0000 7f51c2010740 1 Processor -- start 2024-03-22T01:38:19.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.232+0000 7f51c2010740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564b81ec4200 0x564b81ee32f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:19.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.232+0000 7f51c2010740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b81ec4b20 0x564b81ee3830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:19.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.232+0000 7f51c2010740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b81ecd700 0x564b81ee3d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:19.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.232+0000 7f51c2010740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f51b81abac0 con 0x564b81ec4200 2024-03-22T01:38:19.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.232+0000 7f51c2010740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f51b81acf30 con 0x564b81ec4b20 2024-03-22T01:38:19.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.232+0000 7f51c2010740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f51b81ac190 con 0x564b81ecd700 2024-03-22T01:38:19.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.232+0000 7f51bd1c6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564b81ec4200 0x564b81ee32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:19.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.232+0000 7f51bc9c5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b81ec4b20 0x564b81ee3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:19.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.232+0000 7f51bd1c6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564b81ec4200 0x564b81ee32f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:41150/0 (socket says 172.21.15.72:41150) 2024-03-22T01:38:19.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.232+0000 7f51bd1c6640 1 -- 172.21.15.72:0/3153063289 learned_addr learned my addr 172.21.15.72:0/3153063289 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:38:19.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.232+0000 7f51bc9c5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b81ec4b20 0x564b81ee3830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.72:46950/0 (socket says 172.21.15.72:46950) 2024-03-22T01:38:19.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.232+0000 7f51c2010740 10 client.0 _ll_register_callbacks cb 0x7f51b009aac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:38:19.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.232+0000 7f51bd9c7640 1 --2- 172.21.15.72:0/3153063289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b81ecd700 0x564b81ee3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:19.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.233+0000 7f51bc9c5640 1 -- 172.21.15.72:0/3153063289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b81ecd700 msgr2=0x564b81ee3d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:19.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.233+0000 7f51bc9c5640 1 --2- 172.21.15.72:0/3153063289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b81ecd700 0x564b81ee3d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:19.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.233+0000 7f51bc9c5640 1 -- 172.21.15.72:0/3153063289 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564b81ec4200 msgr2=0x564b81ee32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:19.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.233+0000 7f51bc9c5640 1 --2- 172.21.15.72:0/3153063289 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564b81ec4200 0x564b81ee32f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:19.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.233+0000 7f51bc9c5640 1 -- 172.21.15.72:0/3153063289 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564b81f430f0 con 0x564b81ec4b20 2024-03-22T01:38:19.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.233+0000 7f51bd1c6640 1 --2- 172.21.15.72:0/3153063289 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564b81ec4200 0x564b81ee32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:38:19.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.233+0000 7f51bc9c5640 1 --2- 172.21.15.72:0/3153063289 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b81ec4b20 0x564b81ee3830 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f51a8009230 tx=0x7f51a802bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b3ab8c2ab88405 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:38:19.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.233+0000 7f51a0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-03-22T01:38:19.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.233+0000 7f51a0ff9640 1 -- 172.21.15.72:0/3153063289 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51a80048c0 con 0x564b81ec4b20 2024-03-22T01:38:19.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.233+0000 7f51c2010740 1 -- 172.21.15.72:0/3153063289 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51b4013d60 con 0x564b81ec4b20 2024-03-22T01:38:19.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.233+0000 7f51a0ff9640 1 -- 172.21.15.72:0/3153063289 <== mon.2 v2:172.21.15.89:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f51a8004a60 con 0x564b81ec4b20 2024-03-22T01:38:19.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.233+0000 7f51a0ff9640 1 -- 172.21.15.72:0/3153063289 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f51a805b050 con 0x564b81ec4b20 2024-03-22T01:38:19.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.233+0000 7f51c2010740 10 client.25796 fetch_fsmap learned FSMap version 21 2024-03-22T01:38:19.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.233+0000 7f51c2010740 1 -- 172.21.15.72:0/3153063289 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x564b81f44310 con 0x564b81ec4b20 2024-03-22T01:38:19.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.233+0000 7f51a0ff9640 1 -- 172.21.15.72:0/3153063289 <== mon.2 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51b4013d60 con 0x564b81ec4b20 2024-03-22T01:38:19.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.234+0000 7f51a0ff9640 1 -- 172.21.15.72:0/3153063289 <== mon.2 v2:172.21.15.89:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f51a8040450 con 0x564b81ec4b20 2024-03-22T01:38:19.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.234+0000 7f51c2010740 10 client.25796 fetch_fsmap finished waiting for FSMap version 21 2024-03-22T01:38:19.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.234+0000 7f51c2010740 10 client.25796 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T01:38:19.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.234+0000 7f51c2010740 20 client.25796 populate_metadata read hostname 'smithi072' 2024-03-22T01:38:19.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.234+0000 7f51c2010740 1 --2- 172.21.15.72:0/3153063289 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x564b81eddf70 0x564b81f46b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:19.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.234+0000 7f51c2010740 4 client.25796 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T01:38:19.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.234+0000 7f51c2010740 1 -- 172.21.15.72:0/3153063289 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564b81e27160 con 0x564b81eddf70 2024-03-22T01:38:19.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.234+0000 7f51bd1c6640 1 --2- 172.21.15.72:0/3153063289 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x564b81eddf70 0x564b81f46b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:19.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.234+0000 7f51a0ff9640 10 client.25796 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:38:19.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.234+0000 7f51bd1c6640 1 --2- 172.21.15.72:0/3153063289 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x564b81eddf70 0x564b81f46b50 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f4f309a92c4c62b server_cookie=21d9583d80fba04c in_seq=0 out_seq=0 2024-03-22T01:38:19.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.234+0000 7f51a0ff9640 1 -- 172.21.15.72:0/3153063289 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564b81e27160 con 0x564b81eddf70 2024-03-22T01:38:19.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.234+0000 7f51a0ff9640 10 client.25796 handle_command_reply: tid=0 2024-03-22T01:38:19.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.234+0000 7f51c2010740 1 client.25796 shutdown 2024-03-22T01:38:19.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.235+0000 7f51c2010740 1 -- 172.21.15.72:0/3153063289 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b81ec4b20 msgr2=0x564b81ee3830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:19.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.235+0000 7f51c2010740 1 --2- 172.21.15.72:0/3153063289 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b81ec4b20 0x564b81ee3830 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f51a8009230 tx=0x7f51a802bc30 comp rx=0 tx=0).stop 2024-03-22T01:38:19.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.235+0000 7f51c2010740 1 -- 172.21.15.72:0/3153063289 shutdown_connections 2024-03-22T01:38:19.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.235+0000 7f51c2010740 1 --2- 172.21.15.72:0/3153063289 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x564b81eddf70 0x564b81f46b50 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:19.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.235+0000 7f51c2010740 1 --2- 172.21.15.72:0/3153063289 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b81ec4b20 0x564b81ee3830 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:19.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.235+0000 7f51a0ff9640 0 client.25796 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:38:19.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.235+0000 7f51c2010740 1 --2- 172.21.15.72:0/3153063289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564b81ecd700 0x564b81ee3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:19.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.235+0000 7f51c2010740 1 --2- 172.21.15.72:0/3153063289 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564b81ec4200 0x564b81ee32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:19.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.235+0000 7f51c2010740 1 -- 172.21.15.72:0/3153063289 >> 172.21.15.72:0/3153063289 conn(0x564b81ebe930 msgr2=0x564b81eb4990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:38:19.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.235+0000 7f51c2010740 1 -- 172.21.15.72:0/3153063289 shutdown_connections 2024-03-22T01:38:19.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.235+0000 7f51c2010740 1 -- 172.21.15.72:0/3153063289 wait complete. 2024-03-22T01:38:19.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.235+0000 7f51c2010740 20 client.25796 trim_cache size 0 max 16384 2024-03-22T01:38:19.245 INFO:teuthology.orchestra.run.smithi072.stdout:[] 2024-03-22T01:38:19.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.235+0000 7f51be1c8640 1 -- 172.21.15.72:0/3853563326 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f51a40a3110 msgr2=0x7f51a40a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:19.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.235+0000 7f51be1c8640 1 --2- 172.21.15.72:0/3853563326 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f51a40a3110 0x7f51a40a55d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f51a8006010 tx=0x7f51a8004f10 comp rx=0 tx=0).stop 2024-03-22T01:38:19.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.235+0000 7f51be1c8640 1 -- 172.21.15.72:0/3853563326 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51b8107980 msgr2=0x7f51b819cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:19.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.235+0000 7f51be1c8640 1 --2- 172.21.15.72:0/3853563326 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51b8107980 0x7f51b819cc30 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f51b000b970 tx=0x7f51b000bd70 comp rx=0 tx=0).stop 2024-03-22T01:38:19.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.236+0000 7f51be1c8640 1 -- 172.21.15.72:0/3853563326 shutdown_connections 2024-03-22T01:38:19.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.236+0000 7f51be1c8640 1 --2- 172.21.15.72:0/3853563326 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f51a40a3110 0x7f51a40a55d0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:19.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.236+0000 7f51be1c8640 1 --2- 172.21.15.72:0/3853563326 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51b8107980 0x7f51b819cc30 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:19.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.236+0000 7f51be1c8640 1 --2- 172.21.15.72:0/3853563326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51b8102330 0x7f51b81988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:19.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.236+0000 7f51be1c8640 1 --2- 172.21.15.72:0/3853563326 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51b8101970 0x7f51b8198370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:19.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.236+0000 7f51be1c8640 1 -- 172.21.15.72:0/3853563326 >> 172.21.15.72:0/3853563326 conn(0x7f51b80fd480 msgr2=0x7f51b80fdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:38:19.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.236+0000 7f51be1c8640 1 -- 172.21.15.72:0/3853563326 shutdown_connections 2024-03-22T01:38:19.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:19.236+0000 7f51be1c8640 1 -- 172.21.15.72:0/3853563326 wait complete. 2024-03-22T01:38:19.247 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:38:19 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/3018346145' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:38:19.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:38:19 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/3018346145' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:38:19.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:38:19 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/3018346145' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:38:20.248 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-22T01:38:20.248 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T01:38:20.249 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-22T01:38:20.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:38:20 smithi098 ceph-mon[36468]: pgmap v53: 129 pgs: 129 active+clean; 7.1 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 583 KiB/s wr, 58 op/s 2024-03-22T01:38:20.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.502+0000 7f93a951e640 1 -- 172.21.15.72:0/3922441965 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93a4102140 msgr2=0x7f93a4105670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:20.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.502+0000 7f93a951e640 1 --2- 172.21.15.72:0/3922441965 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93a4102140 0x7f93a4105670 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f9398009900 tx=0x7f939802bc30 comp rx=0 tx=0).stop 2024-03-22T01:38:20.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.502+0000 7f93a951e640 1 -- 172.21.15.72:0/3922441965 shutdown_connections 2024-03-22T01:38:20.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.502+0000 7f93a951e640 1 --2- 172.21.15.72:0/3922441965 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93a4102140 0x7f93a4105670 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.502+0000 7f93a951e640 1 --2- 172.21.15.72:0/3922441965 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f93a4105d50 0x7f93a410d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.502+0000 7f93a951e640 1 --2- 172.21.15.72:0/3922441965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93a4101790 0x7f93a4101b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.502+0000 7f93a951e640 1 -- 172.21.15.72:0/3922441965 >> 172.21.15.72:0/3922441965 conn(0x7f93a4075cf0 msgr2=0x7f93a4076100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:38:20.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.502+0000 7f93a951e640 1 -- 172.21.15.72:0/3922441965 shutdown_connections 2024-03-22T01:38:20.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.502+0000 7f93a951e640 1 -- 172.21.15.72:0/3922441965 wait complete. 2024-03-22T01:38:20.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.503+0000 7f93a951e640 1 Processor -- start 2024-03-22T01:38:20.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.504+0000 7f93a951e640 1 -- start start 2024-03-22T01:38:20.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.504+0000 7f93a951e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f93a4101790 0x7f93a4199300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:20.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.504+0000 7f93a951e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93a4105d50 0x7f93a4199840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:20.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.504+0000 7f93a951e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93a4199dd0 0x7f93a41a4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:20.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.504+0000 7f93a951e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f93a4110e40 con 0x7f93a4101790 2024-03-22T01:38:20.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.504+0000 7f93a951e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f93a4110cc0 con 0x7f93a4199dd0 2024-03-22T01:38:20.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.504+0000 7f93a951e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f93a4110fc0 con 0x7f93a4105d50 2024-03-22T01:38:20.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.504+0000 7f93a27fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93a4105d50 0x7f93a4199840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:20.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.504+0000 7f93a27fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93a4105d50 0x7f93a4199840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.72:36406/0 (socket says 172.21.15.72:36406) 2024-03-22T01:38:20.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.504+0000 7f93a27fc640 1 -- 172.21.15.72:0/3051712601 learned_addr learned my addr 172.21.15.72:0/3051712601 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:38:20.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.504+0000 7f93a37fe640 1 --2- 172.21.15.72:0/3051712601 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93a4199dd0 0x7f93a41a4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:20.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.504+0000 7f93a2ffd640 1 --2- 172.21.15.72:0/3051712601 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f93a4101790 0x7f93a4199300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:20.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.505+0000 7f93a27fc640 1 -- 172.21.15.72:0/3051712601 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93a4199dd0 msgr2=0x7f93a41a4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:20.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.505+0000 7f93a27fc640 1 --2- 172.21.15.72:0/3051712601 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93a4199dd0 0x7f93a41a4380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.505+0000 7f93a27fc640 1 -- 172.21.15.72:0/3051712601 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f93a4101790 msgr2=0x7f93a4199300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:20.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.505+0000 7f93a27fc640 1 --2- 172.21.15.72:0/3051712601 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f93a4101790 0x7f93a4199300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.505+0000 7f93a27fc640 1 -- 172.21.15.72:0/3051712601 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93a41a4b50 con 0x7f93a4105d50 2024-03-22T01:38:20.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.505+0000 7f93a27fc640 1 --2- 172.21.15.72:0/3051712601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93a4105d50 0x7f93a4199840 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9398036040 tx=0x7f939802cbf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:38:20.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.505+0000 7f938bfff640 1 -- 172.21.15.72:0/3051712601 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9398041070 con 0x7f93a4105d50 2024-03-22T01:38:20.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.505+0000 7f938bfff640 1 -- 172.21.15.72:0/3051712601 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9398004450 con 0x7f93a4105d50 2024-03-22T01:38:20.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.505+0000 7f93a951e640 1 -- 172.21.15.72:0/3051712601 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93a41a4de0 con 0x7f93a4105d50 2024-03-22T01:38:20.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.505+0000 7f938bfff640 1 -- 172.21.15.72:0/3051712601 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9398003dc0 con 0x7f93a4105d50 2024-03-22T01:38:20.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.505+0000 7f93a951e640 1 -- 172.21.15.72:0/3051712601 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93a41a5250 con 0x7f93a4105d50 2024-03-22T01:38:20.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.507+0000 7f938bfff640 1 -- 172.21.15.72:0/3051712601 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 137985+0+0 (secure 0 0 0) 0x7f9398044070 con 0x7f93a4105d50 2024-03-22T01:38:20.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.508+0000 7f938bfff640 1 --2- 172.21.15.72:0/3051712601 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f938c0a3110 0x7f938c0a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:20.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.508+0000 7f938bfff640 1 -- 172.21.15.72:0/3051712601 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f93980f1520 con 0x7f93a4105d50 2024-03-22T01:38:20.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.508+0000 7f93a2ffd640 1 --2- 172.21.15.72:0/3051712601 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f938c0a3110 0x7f938c0a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:20.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.509+0000 7f93ac1e6740 1 Processor -- start 2024-03-22T01:38:20.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.509+0000 7f93ac1e6740 1 -- start start 2024-03-22T01:38:20.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.509+0000 7f93a2ffd640 1 --2- 172.21.15.72:0/3051712601 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f938c0a3110 0x7f938c0a55d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9394009990 tx=0x7f9394009110 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:38:20.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.509+0000 7f93ac1e6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56290aaf8ba0 0x56290aaf8f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:20.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.509+0000 7f93ac1e6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56290aaf94c0 0x56290ab01b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:20.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.509+0000 7f93ac1e6740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56290ab020a0 0x56290ab04490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:20.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.509+0000 7f93ac1e6740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x56290aa5c570 con 0x56290ab020a0 2024-03-22T01:38:20.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.509+0000 7f93ac1e6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56290aa04e10 con 0x56290aaf8ba0 2024-03-22T01:38:20.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.509+0000 7f93ac1e6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56290aa58520 con 0x56290aaf94c0 2024-03-22T01:38:20.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.509+0000 7f93a37fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56290ab020a0 0x56290ab04490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:20.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.509+0000 7f93a37fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56290ab020a0 0x56290ab04490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:41178/0 (socket says 172.21.15.72:41178) 2024-03-22T01:38:20.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.510+0000 7f93a37fe640 1 -- 172.21.15.72:0/650864409 learned_addr learned my addr 172.21.15.72:0/650864409 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:38:20.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.510+0000 7f93a2ffd640 1 --2- 172.21.15.72:0/650864409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56290aaf8ba0 0x56290aaf8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:20.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.510+0000 7f93a27fc640 1 --2- 172.21.15.72:0/650864409 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56290aaf94c0 0x56290ab01b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:20.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.510+0000 7f93a2ffd640 1 -- 172.21.15.72:0/650864409 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56290aaf94c0 msgr2=0x56290ab01b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:20.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.510+0000 7f93a2ffd640 1 --2- 172.21.15.72:0/650864409 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56290aaf94c0 0x56290ab01b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.510+0000 7f93a2ffd640 1 -- 172.21.15.72:0/650864409 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56290ab020a0 msgr2=0x56290ab04490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:20.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.510+0000 7f93a2ffd640 1 --2- 172.21.15.72:0/650864409 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56290ab020a0 0x56290ab04490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.510+0000 7f93a2ffd640 1 -- 172.21.15.72:0/650864409 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56290a8387e0 con 0x56290aaf8ba0 2024-03-22T01:38:20.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.510+0000 7f93a2ffd640 1 --2- 172.21.15.72:0/650864409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56290aaf8ba0 0x56290aaf8f80 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f939400eac0 tx=0x7f9394011000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b639b6c0353e106 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:38:20.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.510+0000 7f93897fa640 1 -- 172.21.15.72:0/650864409 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f939401b040 con 0x56290aaf8ba0 2024-03-22T01:38:20.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.510+0000 7f93897fa640 1 -- 172.21.15.72:0/650864409 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9394024070 con 0x56290aaf8ba0 2024-03-22T01:38:20.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.510+0000 7f93897fa640 1 -- 172.21.15.72:0/650864409 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f939401f3f0 con 0x56290aaf8ba0 2024-03-22T01:38:20.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.511+0000 7f93ac1e6740 1 -- 172.21.15.72:0/650864409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56290aaf8ba0 msgr2=0x56290aaf8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:20.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.511+0000 7f93ac1e6740 1 --2- 172.21.15.72:0/650864409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56290aaf8ba0 0x56290aaf8f80 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f939400eac0 tx=0x7f9394011000 comp rx=0 tx=0).stop 2024-03-22T01:38:20.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.512+0000 7f93ac1e6740 1 -- 172.21.15.72:0/650864409 shutdown_connections 2024-03-22T01:38:20.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.512+0000 7f93ac1e6740 1 --2- 172.21.15.72:0/650864409 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56290ab020a0 0x56290ab04490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.512+0000 7f93ac1e6740 1 --2- 172.21.15.72:0/650864409 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56290aaf94c0 0x56290ab01b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.512+0000 7f93ac1e6740 1 --2- 172.21.15.72:0/650864409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56290aaf8ba0 0x56290aaf8f80 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.512+0000 7f93ac1e6740 1 -- 172.21.15.72:0/650864409 >> 172.21.15.72:0/650864409 conn(0x56290aaf32d0 msgr2=0x56290aaf36b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:38:20.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.512+0000 7f93ac1e6740 1 -- 172.21.15.72:0/650864409 shutdown_connections 2024-03-22T01:38:20.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.512+0000 7f93ac1e6740 1 -- 172.21.15.72:0/650864409 wait complete. 2024-03-22T01:38:20.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.513+0000 7f93ac1e6740 1 -- start start 2024-03-22T01:38:20.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.513+0000 7f93ac1e6740 1 Processor -- start 2024-03-22T01:38:20.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.513+0000 7f93ac1e6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56290aaf8ba0 0x56290ab01850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:20.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.513+0000 7f93ac1e6740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56290aaf94c0 0x56290aafee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:20.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.513+0000 7f93ac1e6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56290ab020a0 0x56290aaff380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:20.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.514+0000 7f93ac1e6740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x56290ab04e90 con 0x56290aaf94c0 2024-03-22T01:38:20.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.514+0000 7f93ac1e6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56290aa18110 con 0x56290aaf8ba0 2024-03-22T01:38:20.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.514+0000 7f93ac1e6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56290ab05010 con 0x56290ab020a0 2024-03-22T01:38:20.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.514+0000 7f93a37fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56290ab020a0 0x56290aaff380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:20.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.514+0000 7f93a2ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56290aaf8ba0 0x56290ab01850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:20.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.514+0000 7f93a37fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56290ab020a0 0x56290aaff380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.72:36424/0 (socket says 172.21.15.72:36424) 2024-03-22T01:38:20.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.514+0000 7f93a37fe640 1 -- 172.21.15.72:0/2054098695 learned_addr learned my addr 172.21.15.72:0/2054098695 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:38:20.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.514+0000 7f93a2ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56290aaf8ba0 0x56290ab01850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.72:46986/0 (socket says 172.21.15.72:46986) 2024-03-22T01:38:20.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.514+0000 7f93ac1e6740 10 client.0 _ll_register_callbacks cb 0x56290aa54be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:38:20.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.514+0000 7f93a37fe640 1 -- 172.21.15.72:0/2054098695 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56290aaf8ba0 msgr2=0x56290ab01850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:20.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.514+0000 7f93a37fe640 1 --2- 172.21.15.72:0/2054098695 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56290aaf8ba0 0x56290ab01850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.514+0000 7f93a37fe640 1 -- 172.21.15.72:0/2054098695 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56290aaf94c0 msgr2=0x56290aafee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:20.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.514+0000 7f93a37fe640 1 --2- 172.21.15.72:0/2054098695 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56290aaf94c0 0x56290aafee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.514+0000 7f93a37fe640 1 -- 172.21.15.72:0/2054098695 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56290a898f70 con 0x56290ab020a0 2024-03-22T01:38:20.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.514+0000 7f93a2ffd640 1 --2- 172.21.15.72:0/2054098695 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56290aaf8ba0 0x56290ab01850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:38:20.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.514+0000 7f93a37fe640 1 --2- 172.21.15.72:0/2054098695 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56290ab020a0 0x56290aaff380 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f939c011d10 tx=0x7f939c00f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf6aac5f8b1c33 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:38:20.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.514+0000 7f9386ff5640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-03-22T01:38:20.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.515+0000 7f9386ff5640 1 -- 172.21.15.72:0/2054098695 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f939c003970 con 0x56290ab020a0 2024-03-22T01:38:20.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.515+0000 7f9386ff5640 1 -- 172.21.15.72:0/2054098695 <== mon.1 v2:172.21.15.98:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f939c017050 con 0x56290ab020a0 2024-03-22T01:38:20.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.515+0000 7f93ac1e6740 1 -- 172.21.15.72:0/2054098695 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9394024070 con 0x56290ab020a0 2024-03-22T01:38:20.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.515+0000 7f9386ff5640 1 -- 172.21.15.72:0/2054098695 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f939c003b10 con 0x56290ab020a0 2024-03-22T01:38:20.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.515+0000 7f93ac1e6740 10 client.25468 fetch_fsmap learned FSMap version 21 2024-03-22T01:38:20.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.515+0000 7f93ac1e6740 1 -- 172.21.15.72:0/2054098695 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x56290a730170 con 0x56290ab020a0 2024-03-22T01:38:20.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.515+0000 7f9386ff5640 1 -- 172.21.15.72:0/2054098695 <== mon.1 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f939c003cb0 con 0x56290ab020a0 2024-03-22T01:38:20.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.515+0000 7f9386ff5640 1 -- 172.21.15.72:0/2054098695 <== mon.1 v2:172.21.15.98:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f939c019070 con 0x56290ab020a0 2024-03-22T01:38:20.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.515+0000 7f93ac1e6740 10 client.25468 fetch_fsmap finished waiting for FSMap version 21 2024-03-22T01:38:20.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.515+0000 7f93ac1e6740 10 client.25468 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T01:38:20.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.515+0000 7f93ac1e6740 20 client.25468 populate_metadata read hostname 'smithi072' 2024-03-22T01:38:20.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.515+0000 7f93ac1e6740 1 --2- 172.21.15.72:0/2054098695 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x56290ab1aa20 0x56290ab1ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:20.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.515+0000 7f93ac1e6740 4 client.25468 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T01:38:20.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.515+0000 7f93ac1e6740 1 -- 172.21.15.72:0/2054098695 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f9394024070 con 0x56290ab1aa20 2024-03-22T01:38:20.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.516+0000 7f93a2ffd640 1 --2- 172.21.15.72:0/2054098695 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x56290ab1aa20 0x56290ab1ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:20.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.516+0000 7f9386ff5640 10 client.25468 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:38:20.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.516+0000 7f93a2ffd640 1 --2- 172.21.15.72:0/2054098695 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x56290ab1aa20 0x56290ab1ce70 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=123a5bfc21f0df31 server_cookie=16c6df9a59e90b2d in_seq=0 out_seq=0 2024-03-22T01:38:20.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.517+0000 7f9386ff5640 1 -- 172.21.15.72:0/2054098695 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f9394024070 con 0x56290ab1aa20 2024-03-22T01:38:20.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.517+0000 7f9386ff5640 10 client.25468 handle_command_reply: tid=0 2024-03-22T01:38:20.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.517+0000 7f93ac1e6740 1 client.25468 shutdown 2024-03-22T01:38:20.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.517+0000 7f93ac1e6740 1 -- 172.21.15.72:0/2054098695 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56290ab020a0 msgr2=0x56290aaff380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:20.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.517+0000 7f93ac1e6740 1 --2- 172.21.15.72:0/2054098695 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56290ab020a0 0x56290aaff380 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f939c011d10 tx=0x7f939c00f790 comp rx=0 tx=0).stop 2024-03-22T01:38:20.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.518+0000 7f93ac1e6740 1 -- 172.21.15.72:0/2054098695 shutdown_connections 2024-03-22T01:38:20.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.518+0000 7f93ac1e6740 1 --2- 172.21.15.72:0/2054098695 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x56290ab1aa20 0x56290ab1ce70 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.518+0000 7f93ac1e6740 1 --2- 172.21.15.72:0/2054098695 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56290ab020a0 0x56290aaff380 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.518+0000 7f93ac1e6740 1 --2- 172.21.15.72:0/2054098695 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56290aaf94c0 0x56290aafee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.518+0000 7f93ac1e6740 1 --2- 172.21.15.72:0/2054098695 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56290aaf8ba0 0x56290ab01850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.518+0000 7f93ac1e6740 1 -- 172.21.15.72:0/2054098695 >> 172.21.15.72:0/2054098695 conn(0x56290aaf32d0 msgr2=0x56290aaf5020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:38:20.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.518+0000 7f9386ff5640 0 client.25468 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:38:20.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.518+0000 7f93ac1e6740 1 -- 172.21.15.72:0/2054098695 shutdown_connections 2024-03-22T01:38:20.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.518+0000 7f93ac1e6740 1 -- 172.21.15.72:0/2054098695 wait complete. 2024-03-22T01:38:20.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.521+0000 7f93ac1e6740 20 client.25468 trim_cache size 0 max 16384 2024-03-22T01:38:20.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.550+0000 7f93ac1e6740 1 Processor -- start 2024-03-22T01:38:20.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.550+0000 7f93ac1e6740 1 -- start start 2024-03-22T01:38:20.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.550+0000 7f93ac1e6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56290aaf8ba0 0x56290ab08240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:20.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.550+0000 7f93ac1e6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56290aaf94c0 0x56290ab01570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:20.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.550+0000 7f93ac1e6740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56290ab020a0 0x56290ab22740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:20.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.550+0000 7f93ac1e6740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f93a41aaeb0 con 0x56290ab020a0 2024-03-22T01:38:20.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.550+0000 7f93ac1e6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f93a41ad730 con 0x56290aaf8ba0 2024-03-22T01:38:20.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.550+0000 7f93ac1e6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f93a41ab520 con 0x56290aaf94c0 2024-03-22T01:38:20.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.550+0000 7f93a27fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56290aaf94c0 0x56290ab01570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:20.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.550+0000 7f93a27fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56290aaf94c0 0x56290ab01570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.72:36440/0 (socket says 172.21.15.72:36440) 2024-03-22T01:38:20.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.550+0000 7f93a37fe640 1 --2- 172.21.15.72:0/2958634001 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56290ab020a0 0x56290ab22740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:20.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.550+0000 7f93a27fc640 1 -- 172.21.15.72:0/2958634001 learned_addr learned my addr 172.21.15.72:0/2958634001 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:38:20.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.551+0000 7f93a2ffd640 1 --2- 172.21.15.72:0/2958634001 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56290aaf8ba0 0x56290ab08240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:20.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.551+0000 7f93a37fe640 1 -- 172.21.15.72:0/2958634001 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56290aaf94c0 msgr2=0x56290ab01570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:20.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.551+0000 7f93a37fe640 1 --2- 172.21.15.72:0/2958634001 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56290aaf94c0 0x56290ab01570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.551+0000 7f93a37fe640 1 -- 172.21.15.72:0/2958634001 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56290aaf8ba0 msgr2=0x56290ab08240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:20.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.551+0000 7f93a37fe640 1 --2- 172.21.15.72:0/2958634001 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56290aaf8ba0 0x56290ab08240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.551+0000 7f93a37fe640 1 -- 172.21.15.72:0/2958634001 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56290aaeec10 con 0x56290ab020a0 2024-03-22T01:38:20.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.551+0000 7f93a27fc640 1 --2- 172.21.15.72:0/2958634001 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56290aaf94c0 0x56290ab01570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:38:20.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.551+0000 7f93a37fe640 1 --2- 172.21.15.72:0/2958634001 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56290ab020a0 0x56290ab22740 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f939c01aa40 tx=0x7f939c011900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b86736721eb51ef1 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:38:20.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.552+0000 7f9388ff9640 1 -- 172.21.15.72:0/2958634001 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f939c02e070 con 0x56290ab020a0 2024-03-22T01:38:20.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.552+0000 7f9388ff9640 1 -- 172.21.15.72:0/2958634001 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f939c01a3e0 con 0x56290ab020a0 2024-03-22T01:38:20.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.552+0000 7f9388ff9640 1 -- 172.21.15.72:0/2958634001 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f939c029e30 con 0x56290ab020a0 2024-03-22T01:38:20.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.553+0000 7f93ac1e6740 1 -- 172.21.15.72:0/2958634001 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56290ab020a0 msgr2=0x56290ab22740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:20.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.553+0000 7f93ac1e6740 1 --2- 172.21.15.72:0/2958634001 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56290ab020a0 0x56290ab22740 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f939c01aa40 tx=0x7f939c011900 comp rx=0 tx=0).stop 2024-03-22T01:38:20.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.553+0000 7f93ac1e6740 1 -- 172.21.15.72:0/2958634001 shutdown_connections 2024-03-22T01:38:20.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.553+0000 7f93ac1e6740 1 --2- 172.21.15.72:0/2958634001 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56290ab020a0 0x56290ab22740 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.553+0000 7f93ac1e6740 1 --2- 172.21.15.72:0/2958634001 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56290aaf94c0 0x56290ab01570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.553+0000 7f93ac1e6740 1 --2- 172.21.15.72:0/2958634001 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56290aaf8ba0 0x56290ab08240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.553+0000 7f93ac1e6740 1 -- 172.21.15.72:0/2958634001 >> 172.21.15.72:0/2958634001 conn(0x56290aaf32d0 msgr2=0x56290aaf23f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:38:20.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.555+0000 7f93ac1e6740 1 -- 172.21.15.72:0/2958634001 shutdown_connections 2024-03-22T01:38:20.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.556+0000 7f93ac1e6740 1 -- 172.21.15.72:0/2958634001 wait complete. 2024-03-22T01:38:20.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.557+0000 7f93ac1e6740 1 -- start start 2024-03-22T01:38:20.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.557+0000 7f93ac1e6740 1 Processor -- start 2024-03-22T01:38:20.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.558+0000 7f93ac1e6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56290aaf8ba0 0x56290ab17680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:20.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.558+0000 7f93ac1e6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56290aaf94c0 0x56290ab129f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:20.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.558+0000 7f93ac1e6740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56290ab020a0 0x56290ab12f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:20.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.558+0000 7f93ac1e6740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f93a41abbf0 con 0x56290ab020a0 2024-03-22T01:38:20.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.558+0000 7f93ac1e6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f93a41ad060 con 0x56290aaf8ba0 2024-03-22T01:38:20.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.558+0000 7f93ac1e6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f93a41ac2c0 con 0x56290aaf94c0 2024-03-22T01:38:20.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.558+0000 7f93a37fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56290ab020a0 0x56290ab12f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:20.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.558+0000 7f93a37fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56290ab020a0 0x56290ab12f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:41200/0 (socket says 172.21.15.72:41200) 2024-03-22T01:38:20.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.558+0000 7f93a37fe640 1 -- 172.21.15.72:0/534090841 learned_addr learned my addr 172.21.15.72:0/534090841 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:38:20.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.558+0000 7f93a2ffd640 1 --2- 172.21.15.72:0/534090841 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56290aaf8ba0 0x56290ab17680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:20.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.558+0000 7f93a27fc640 1 --2- 172.21.15.72:0/534090841 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56290aaf94c0 0x56290ab129f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:20.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.558+0000 7f93ac1e6740 10 client.0 _ll_register_callbacks cb 0x7f939c009cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:38:20.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.559+0000 7f93a2ffd640 1 -- 172.21.15.72:0/534090841 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56290aaf94c0 msgr2=0x56290ab129f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:20.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.559+0000 7f93a2ffd640 1 --2- 172.21.15.72:0/534090841 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56290aaf94c0 0x56290ab129f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.559+0000 7f93a2ffd640 1 -- 172.21.15.72:0/534090841 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56290ab020a0 msgr2=0x56290ab12f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:20.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.559+0000 7f93a2ffd640 1 --2- 172.21.15.72:0/534090841 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56290ab020a0 0x56290ab12f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.559+0000 7f93a2ffd640 1 -- 172.21.15.72:0/534090841 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56290ab135d0 con 0x56290aaf8ba0 2024-03-22T01:38:20.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.559+0000 7f93a37fe640 1 --2- 172.21.15.72:0/534090841 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56290ab020a0 0x56290ab12f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:38:20.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.559+0000 7f93a27fc640 1 --2- 172.21.15.72:0/534090841 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56290aaf94c0 0x56290ab129f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:38:20.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.560+0000 7f93a2ffd640 1 --2- 172.21.15.72:0/534090841 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56290aaf8ba0 0x56290ab17680 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f939400e5a0 tx=0x7f9394013af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9d5a04008c4e4e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:38:20.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.560+0000 7f9386ff5640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-03-22T01:38:20.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.560+0000 7f9386ff5640 1 -- 172.21.15.72:0/534090841 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9394039060 con 0x56290aaf8ba0 2024-03-22T01:38:20.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.560+0000 7f9386ff5640 1 -- 172.21.15.72:0/534090841 <== mon.2 v2:172.21.15.89:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f9394034ab0 con 0x56290aaf8ba0 2024-03-22T01:38:20.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.560+0000 7f93ac1e6740 1 -- 172.21.15.72:0/534090841 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f939c01a3e0 con 0x56290aaf8ba0 2024-03-22T01:38:20.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.561+0000 7f9386ff5640 1 -- 172.21.15.72:0/534090841 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f9394034cb0 con 0x56290aaf8ba0 2024-03-22T01:38:20.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.563+0000 7f93ac1e6740 10 client.25814 fetch_fsmap learned FSMap version 21 2024-03-22T01:38:20.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.563+0000 7f93ac1e6740 1 -- 172.21.15.72:0/534090841 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x56290ab80c80 con 0x56290aaf8ba0 2024-03-22T01:38:20.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.564+0000 7f9386ff5640 1 -- 172.21.15.72:0/534090841 <== mon.2 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f939c01a3e0 con 0x56290aaf8ba0 2024-03-22T01:38:20.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.565+0000 7f9386ff5640 1 -- 172.21.15.72:0/534090841 <== mon.2 v2:172.21.15.89:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f939401e030 con 0x56290aaf8ba0 2024-03-22T01:38:20.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.565+0000 7f93ac1e6740 10 client.25814 fetch_fsmap finished waiting for FSMap version 21 2024-03-22T01:38:20.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.565+0000 7f93ac1e6740 10 client.25814 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T01:38:20.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.565+0000 7f93ac1e6740 20 client.25814 populate_metadata read hostname 'smithi072' 2024-03-22T01:38:20.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.565+0000 7f93ac1e6740 1 --2- 172.21.15.72:0/534090841 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x56290ab81080 0x56290ab834a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:20.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.565+0000 7f93ac1e6740 4 client.25814 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-22T01:38:20.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.565+0000 7f93ac1e6740 1 -- 172.21.15.72:0/534090841 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56290aa69ab0 con 0x56290ab81080 2024-03-22T01:38:20.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.567+0000 7f93a27fc640 1 --2- 172.21.15.72:0/534090841 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x56290ab81080 0x56290ab834a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:20.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.569+0000 7f93a27fc640 1 --2- 172.21.15.72:0/534090841 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x56290ab81080 0x56290ab834a0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e5be9732707a49e5 server_cookie=b5d2852749007b9d in_seq=0 out_seq=0 2024-03-22T01:38:20.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.569+0000 7f9386ff5640 10 client.25814 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:38:20.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.571+0000 7f9386ff5640 1 -- 172.21.15.72:0/534090841 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56290aa69ab0 con 0x56290ab81080 2024-03-22T01:38:20.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.571+0000 7f9386ff5640 10 client.25814 handle_command_reply: tid=0 2024-03-22T01:38:20.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.571+0000 7f93ac1e6740 1 client.25814 shutdown 2024-03-22T01:38:20.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.572+0000 7f93ac1e6740 1 -- 172.21.15.72:0/534090841 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56290aaf8ba0 msgr2=0x56290ab17680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:20.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.572+0000 7f93ac1e6740 1 --2- 172.21.15.72:0/534090841 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56290aaf8ba0 0x56290ab17680 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f939400e5a0 tx=0x7f9394013af0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.572+0000 7f93ac1e6740 1 -- 172.21.15.72:0/534090841 shutdown_connections 2024-03-22T01:38:20.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.572+0000 7f93ac1e6740 1 --2- 172.21.15.72:0/534090841 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x56290ab81080 0x56290ab834a0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.572+0000 7f93ac1e6740 1 --2- 172.21.15.72:0/534090841 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56290ab020a0 0x56290ab12f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.572+0000 7f93ac1e6740 1 --2- 172.21.15.72:0/534090841 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56290aaf94c0 0x56290ab129f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.572+0000 7f93ac1e6740 1 --2- 172.21.15.72:0/534090841 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56290aaf8ba0 0x56290ab17680 secure :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f939400e5a0 tx=0x7f9394013af0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.572+0000 7f9386ff5640 0 client.25814 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:38:20.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.572+0000 7f93ac1e6740 1 -- 172.21.15.72:0/534090841 >> 172.21.15.72:0/534090841 conn(0x56290aaf32d0 msgr2=0x56290aaf4560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:38:20.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.572+0000 7f93ac1e6740 1 -- 172.21.15.72:0/534090841 shutdown_connections 2024-03-22T01:38:20.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.572+0000 7f93ac1e6740 1 -- 172.21.15.72:0/534090841 wait complete. 2024-03-22T01:38:20.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.572+0000 7f93ac1e6740 20 client.25814 trim_cache size 0 max 16384 2024-03-22T01:38:20.572 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-22T01:38:20.572 INFO:teuthology.orchestra.run.smithi072.stdout: "return_code": 0, 2024-03-22T01:38:20.572 INFO:teuthology.orchestra.run.smithi072.stdout: "scrub_tag": "e4b71388-fc76-406f-ab22-6d1ede692217", 2024-03-22T01:38:20.572 INFO:teuthology.orchestra.run.smithi072.stdout: "mode": "asynchronous" 2024-03-22T01:38:20.572 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-22T01:38:20.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.573+0000 7f93a951e640 1 -- 172.21.15.72:0/3051712601 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f938c0a3110 msgr2=0x7f938c0a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:20.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.573+0000 7f93a951e640 1 --2- 172.21.15.72:0/3051712601 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f938c0a3110 0x7f938c0a55d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9394009990 tx=0x7f9394009110 comp rx=0 tx=0).stop 2024-03-22T01:38:20.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.573+0000 7f93a951e640 1 -- 172.21.15.72:0/3051712601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93a4105d50 msgr2=0x7f93a4199840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:20.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.573+0000 7f93a951e640 1 --2- 172.21.15.72:0/3051712601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93a4105d50 0x7f93a4199840 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9398036040 tx=0x7f939802cbf0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.573+0000 7f93a951e640 1 -- 172.21.15.72:0/3051712601 shutdown_connections 2024-03-22T01:38:20.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.573+0000 7f93a951e640 1 --2- 172.21.15.72:0/3051712601 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f938c0a3110 0x7f938c0a55d0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.573+0000 7f93a951e640 1 --2- 172.21.15.72:0/3051712601 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93a4199dd0 0x7f93a41a4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.573+0000 7f93a951e640 1 --2- 172.21.15.72:0/3051712601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93a4105d50 0x7f93a4199840 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.573+0000 7f93a951e640 1 --2- 172.21.15.72:0/3051712601 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f93a4101790 0x7f93a4199300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.573+0000 7f93a951e640 1 -- 172.21.15.72:0/3051712601 >> 172.21.15.72:0/3051712601 conn(0x7f93a4075cf0 msgr2=0x7f93a40fe6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:38:20.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.574+0000 7f93a951e640 1 -- 172.21.15.72:0/3051712601 shutdown_connections 2024-03-22T01:38:20.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.574+0000 7f93a951e640 1 -- 172.21.15.72:0/3051712601 wait complete. 2024-03-22T01:38:20.599 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:38:20 smithi072 ceph-mon[32414]: pgmap v53: 129 pgs: 129 active+clean; 7.1 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 583 KiB/s wr, 58 op/s 2024-03-22T01:38:20.601 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T01:38:20.601 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T01:38:20.607 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:38:20 smithi089 ceph-mon[35770]: pgmap v53: 129 pgs: 129 active+clean; 7.1 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 583 KiB/s wr, 58 op/s 2024-03-22T01:38:20.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.827+0000 7f7c5604d640 1 -- 172.21.15.72:0/627770519 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7c48096910 msgr2=0x7f7c4809a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:20.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.827+0000 7f7c5604d640 1 --2- 172.21.15.72:0/627770519 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7c48096910 0x7f7c4809a950 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f7c40009900 tx=0x7f7c4002bc30 comp rx=0 tx=0).stop 2024-03-22T01:38:20.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.827+0000 7f7c5604d640 1 -- 172.21.15.72:0/627770519 shutdown_connections 2024-03-22T01:38:20.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.827+0000 7f7c5604d640 1 --2- 172.21.15.72:0/627770519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c4809b030 0x7f7c480a2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.827+0000 7f7c5604d640 1 --2- 172.21.15.72:0/627770519 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7c48096910 0x7f7c4809a950 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.827+0000 7f7c5604d640 1 --2- 172.21.15.72:0/627770519 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c48095f60 0x7f7c48096340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.827+0000 7f7c5604d640 1 -- 172.21.15.72:0/627770519 >> 172.21.15.72:0/627770519 conn(0x7f7c4800ceb0 msgr2=0x7f7c4800d2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:38:20.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.827+0000 7f7c5604d640 1 -- 172.21.15.72:0/627770519 shutdown_connections 2024-03-22T01:38:20.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.827+0000 7f7c5604d640 1 -- 172.21.15.72:0/627770519 wait complete. 2024-03-22T01:38:20.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.828+0000 7f7c5604d640 1 Processor -- start 2024-03-22T01:38:20.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.828+0000 7f7c5604d640 1 -- start start 2024-03-22T01:38:20.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.828+0000 7f7c5604d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c48095f60 0x7f7c481343e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:20.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.828+0000 7f7c5604d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c4809b030 0x7f7c48134920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:20.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.828+0000 7f7c5604d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7c4812e590 0x7f7c4812ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:20.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.828+0000 7f7c5604d640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f7c480a8370 con 0x7f7c4812e590 2024-03-22T01:38:20.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.828+0000 7f7c5604d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f7c480a81f0 con 0x7f7c48095f60 2024-03-22T01:38:20.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.828+0000 7f7c5604d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7c480a84f0 con 0x7f7c4809b030 2024-03-22T01:38:20.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.828+0000 7f7c5584c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7c4812e590 0x7f7c4812ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:20.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.828+0000 7f7c5484a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c4809b030 0x7f7c48134920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:20.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.828+0000 7f7c5484a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c4809b030 0x7f7c48134920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.72:39660/0 (socket says 172.21.15.72:39660) 2024-03-22T01:38:20.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.828+0000 7f7c5484a640 1 -- 172.21.15.72:0/3640121618 learned_addr learned my addr 172.21.15.72:0/3640121618 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:38:20.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.829+0000 7f7c5504b640 1 --2- 172.21.15.72:0/3640121618 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c48095f60 0x7f7c481343e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:20.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.829+0000 7f7c5584c640 1 -- 172.21.15.72:0/3640121618 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c4809b030 msgr2=0x7f7c48134920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:20.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.829+0000 7f7c5584c640 1 --2- 172.21.15.72:0/3640121618 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c4809b030 0x7f7c48134920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.829+0000 7f7c5584c640 1 -- 172.21.15.72:0/3640121618 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c48095f60 msgr2=0x7f7c481343e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:20.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.829+0000 7f7c5584c640 1 --2- 172.21.15.72:0/3640121618 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c48095f60 0x7f7c481343e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.829+0000 7f7c5584c640 1 -- 172.21.15.72:0/3640121618 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c4812f0b0 con 0x7f7c4812e590 2024-03-22T01:38:20.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.829+0000 7f7c5584c640 1 --2- 172.21.15.72:0/3640121618 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7c4812e590 0x7f7c4812ea40 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f7c500679d0 tx=0x7f7c50068550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:38:20.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.829+0000 7f7c467fc640 1 -- 172.21.15.72:0/3640121618 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c50086070 con 0x7f7c4812e590 2024-03-22T01:38:20.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.830+0000 7f7c467fc640 1 -- 172.21.15.72:0/3640121618 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c50069040 con 0x7f7c4812e590 2024-03-22T01:38:20.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.830+0000 7f7c467fc640 1 -- 172.21.15.72:0/3640121618 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c50080620 con 0x7f7c4812e590 2024-03-22T01:38:20.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.830+0000 7f7c5604d640 1 -- 172.21.15.72:0/3640121618 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c4812f3a0 con 0x7f7c4812e590 2024-03-22T01:38:20.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.830+0000 7f7c5604d640 1 -- 172.21.15.72:0/3640121618 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c4813b5a0 con 0x7f7c4812e590 2024-03-22T01:38:20.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.834+0000 7f7c467fc640 1 -- 172.21.15.72:0/3640121618 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 25) ==== 137985+0+0 (secure 0 0 0) 0x7f7c5007f070 con 0x7f7c4812e590 2024-03-22T01:38:20.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.834+0000 7f7c59f73740 1 Processor -- start 2024-03-22T01:38:20.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.834+0000 7f7c59f73740 1 -- start start 2024-03-22T01:38:20.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.834+0000 7f7c59f73740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560236a64620 0x560236a64a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:20.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.834+0000 7f7c59f73740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560236a64f40 0x560236a6d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:20.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.834+0000 7f7c59f73740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560236a6db20 0x560236a6ff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:20.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.834+0000 7f7c59f73740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x5602369c88c0 con 0x560236a64620 2024-03-22T01:38:20.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.834+0000 7f7c59f73740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x560236970c20 con 0x560236a64f40 2024-03-22T01:38:20.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.834+0000 7f7c59f73740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5602369c4870 con 0x560236a6db20 2024-03-22T01:38:20.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.835+0000 7f7c5504b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560236a64f40 0x560236a6d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:20.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.835+0000 7f7c5504b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560236a64f40 0x560236a6d5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.72:47590/0 (socket says 172.21.15.72:47590) 2024-03-22T01:38:20.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.835+0000 7f7c5504b640 1 -- 172.21.15.72:0/4055540242 learned_addr learned my addr 172.21.15.72:0/4055540242 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:38:20.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.835+0000 7f7c5504b640 1 -- 172.21.15.72:0/4055540242 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560236a6db20 msgr2=0x560236a6ff10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-22T01:38:20.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.835+0000 7f7c5504b640 1 --2- 172.21.15.72:0/4055540242 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560236a6db20 0x560236a6ff10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.835+0000 7f7c5504b640 1 -- 172.21.15.72:0/4055540242 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560236a64620 msgr2=0x560236a64a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:20.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.835+0000 7f7c5504b640 1 --2- 172.21.15.72:0/4055540242 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560236a64620 0x560236a64a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.835+0000 7f7c5504b640 1 -- 172.21.15.72:0/4055540242 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560236995150 con 0x560236a64f40 2024-03-22T01:38:20.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.836+0000 7f7c467fc640 1 --2- 172.21.15.72:0/3640121618 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f7c3c0a3190 0x7f7c3c0a5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:20.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.837+0000 7f7c467fc640 1 -- 172.21.15.72:0/3640121618 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f7c5013ab30 con 0x7f7c4812e590 2024-03-22T01:38:20.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.838+0000 7f7c5484a640 1 --2- 172.21.15.72:0/3640121618 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f7c3c0a3190 0x7f7c3c0a5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:20.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.838+0000 7f7c5504b640 1 --2- 172.21.15.72:0/4055540242 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560236a64f40 0x560236a6d5e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c00bcb0 tx=0x7f7c4c00f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a41c3472df00761f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:38:20.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.839+0000 7f7c3b7fe640 1 -- 172.21.15.72:0/4055540242 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c4c015070 con 0x560236a64f40 2024-03-22T01:38:20.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.839+0000 7f7c5484a640 1 --2- 172.21.15.72:0/3640121618 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f7c3c0a3190 0x7f7c3c0a5650 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7c48093350 tx=0x7f7c40005dd0 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:38:20.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.839+0000 7f7c3b7fe640 1 -- 172.21.15.72:0/4055540242 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c4c01a940 con 0x560236a64f40 2024-03-22T01:38:20.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.839+0000 7f7c3b7fe640 1 -- 172.21.15.72:0/4055540242 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c4c011aa0 con 0x560236a64f40 2024-03-22T01:38:20.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.840+0000 7f7c59f73740 1 -- 172.21.15.72:0/4055540242 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560236a64f40 msgr2=0x560236a6d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:20.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.840+0000 7f7c59f73740 1 --2- 172.21.15.72:0/4055540242 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560236a64f40 0x560236a6d5e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c00bcb0 tx=0x7f7c4c00f730 comp rx=0 tx=0).stop 2024-03-22T01:38:20.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.840+0000 7f7c59f73740 1 -- 172.21.15.72:0/4055540242 shutdown_connections 2024-03-22T01:38:20.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.840+0000 7f7c59f73740 1 --2- 172.21.15.72:0/4055540242 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560236a64f40 0x560236a6d5e0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.840+0000 7f7c59f73740 1 --2- 172.21.15.72:0/4055540242 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560236a6db20 0x560236a6ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.840+0000 7f7c59f73740 1 --2- 172.21.15.72:0/4055540242 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560236a64620 0x560236a64a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.840+0000 7f7c59f73740 1 -- 172.21.15.72:0/4055540242 >> 172.21.15.72:0/4055540242 conn(0x560236a5ed50 msgr2=0x560236a5f130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:38:20.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.840+0000 7f7c59f73740 1 -- 172.21.15.72:0/4055540242 shutdown_connections 2024-03-22T01:38:20.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.840+0000 7f7c59f73740 1 -- 172.21.15.72:0/4055540242 wait complete. 2024-03-22T01:38:20.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.841+0000 7f7c59f73740 1 -- start start 2024-03-22T01:38:20.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.841+0000 7f7c59f73740 1 Processor -- start 2024-03-22T01:38:20.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.841+0000 7f7c59f73740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560236a64620 0x560236a6aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:20.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.841+0000 7f7c59f73740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560236a64f40 0x560236a6afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:20.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.841+0000 7f7c59f73740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560236a6db20 0x560236a6b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:20.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.841+0000 7f7c59f73740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x560236a70960 con 0x560236a64620 2024-03-22T01:38:20.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.841+0000 7f7c59f73740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x560236984130 con 0x560236a64f40 2024-03-22T01:38:20.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.841+0000 7f7c59f73740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560236a70ae0 con 0x560236a6db20 2024-03-22T01:38:20.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.841+0000 7f7c59f73740 10 client.0 _ll_register_callbacks cb 0x560236983b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:38:20.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.842+0000 7f7c5584c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560236a6db20 0x560236a6b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:20.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.842+0000 7f7c5584c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560236a6db20 0x560236a6b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.72:39676/0 (socket says 172.21.15.72:39676) 2024-03-22T01:38:20.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.842+0000 7f7c5584c640 1 -- 172.21.15.72:0/1183292195 learned_addr learned my addr 172.21.15.72:0/1183292195 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:38:20.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.842+0000 7f7c5504b640 1 --2- 172.21.15.72:0/1183292195 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560236a64620 0x560236a6aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:20.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.842+0000 7f7c5584c640 1 -- 172.21.15.72:0/1183292195 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560236a64f40 msgr2=0x560236a6afc0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-22T01:38:20.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.842+0000 7f7c5584c640 1 --2- 172.21.15.72:0/1183292195 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560236a64f40 0x560236a6afc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.842+0000 7f7c5584c640 1 -- 172.21.15.72:0/1183292195 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560236a64620 msgr2=0x560236a6aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:20.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.842+0000 7f7c5584c640 1 --2- 172.21.15.72:0/1183292195 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560236a64620 0x560236a6aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.842+0000 7f7c5584c640 1 -- 172.21.15.72:0/1183292195 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602367a67d0 con 0x560236a6db20 2024-03-22T01:38:20.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.842+0000 7f7c5504b640 1 --2- 172.21.15.72:0/1183292195 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560236a64620 0x560236a6aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:38:20.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.843+0000 7f7c5584c640 1 --2- 172.21.15.72:0/1183292195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560236a6db20 0x560236a6b500 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7c50102f80 tx=0x7f7c50106540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=506966152db59a26 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:38:20.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.843+0000 7f7c38ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-03-22T01:38:20.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.843+0000 7f7c38ff9640 1 -- 172.21.15.72:0/1183292195 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c501382f0 con 0x560236a6db20 2024-03-22T01:38:20.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.843+0000 7f7c38ff9640 1 -- 172.21.15.72:0/1183292195 <== mon.1 v2:172.21.15.98:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f7c50105f70 con 0x560236a6db20 2024-03-22T01:38:20.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.843+0000 7f7c59f73740 1 -- 172.21.15.72:0/1183292195 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c4c01a940 con 0x560236a6db20 2024-03-22T01:38:20.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.843+0000 7f7c38ff9640 1 -- 172.21.15.72:0/1183292195 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f7c50106170 con 0x560236a6db20 2024-03-22T01:38:20.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.843+0000 7f7c59f73740 10 client.25483 fetch_fsmap learned FSMap version 21 2024-03-22T01:38:20.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.843+0000 7f7c59f73740 1 -- 172.21.15.72:0/1183292195 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5602366dc3f0 con 0x560236a6db20 2024-03-22T01:38:20.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.844+0000 7f7c38ff9640 1 -- 172.21.15.72:0/1183292195 <== mon.1 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c4c01a940 con 0x560236a6db20 2024-03-22T01:38:20.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.844+0000 7f7c38ff9640 1 -- 172.21.15.72:0/1183292195 <== mon.1 v2:172.21.15.98:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f7c50105b70 con 0x560236a6db20 2024-03-22T01:38:20.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.844+0000 7f7c59f73740 10 client.25483 fetch_fsmap finished waiting for FSMap version 21 2024-03-22T01:38:20.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.844+0000 7f7c59f73740 10 client.25483 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T01:38:20.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.844+0000 7f7c59f73740 20 client.25483 populate_metadata read hostname 'smithi072' 2024-03-22T01:38:20.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.844+0000 7f7c59f73740 1 --2- 172.21.15.72:0/1183292195 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x560236a7de40 0x560236a80300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:20.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.844+0000 7f7c59f73740 4 client.25483 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T01:38:20.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.844+0000 7f7c59f73740 1 -- 172.21.15.72:0/1183292195 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5602369d5e00 con 0x560236a7de40 2024-03-22T01:38:20.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.845+0000 7f7c5504b640 1 --2- 172.21.15.72:0/1183292195 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x560236a7de40 0x560236a80300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:20.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.847+0000 7f7c5504b640 1 --2- 172.21.15.72:0/1183292195 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x560236a7de40 0x560236a80300 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=71eef1acb4bede79 server_cookie=3e4bae97bc390323 in_seq=0 out_seq=0 2024-03-22T01:38:20.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.847+0000 7f7c38ff9640 10 client.25483 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:38:20.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.848+0000 7f7c38ff9640 1 -- 172.21.15.72:0/1183292195 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5602369d5e00 con 0x560236a7de40 2024-03-22T01:38:20.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.848+0000 7f7c38ff9640 10 client.25483 handle_command_reply: tid=0 2024-03-22T01:38:20.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.848+0000 7f7c59f73740 1 client.25483 shutdown 2024-03-22T01:38:20.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.849+0000 7f7c59f73740 1 -- 172.21.15.72:0/1183292195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560236a6db20 msgr2=0x560236a6b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:20.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.849+0000 7f7c59f73740 1 --2- 172.21.15.72:0/1183292195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560236a6db20 0x560236a6b500 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7c50102f80 tx=0x7f7c50106540 comp rx=0 tx=0).stop 2024-03-22T01:38:20.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.849+0000 7f7c59f73740 1 -- 172.21.15.72:0/1183292195 shutdown_connections 2024-03-22T01:38:20.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.849+0000 7f7c59f73740 1 --2- 172.21.15.72:0/1183292195 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x560236a7de40 0x560236a80300 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.849+0000 7f7c59f73740 1 --2- 172.21.15.72:0/1183292195 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560236a64f40 0x560236a6afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.849+0000 7f7c59f73740 1 --2- 172.21.15.72:0/1183292195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560236a6db20 0x560236a6b500 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.849+0000 7f7c59f73740 1 --2- 172.21.15.72:0/1183292195 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560236a64620 0x560236a6aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.849+0000 7f7c59f73740 1 -- 172.21.15.72:0/1183292195 >> 172.21.15.72:0/1183292195 conn(0x560236a5ed50 msgr2=0x560236a60aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:38:20.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.849+0000 7f7c38ff9640 0 client.25483 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:38:20.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.849+0000 7f7c59f73740 1 -- 172.21.15.72:0/1183292195 shutdown_connections 2024-03-22T01:38:20.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.849+0000 7f7c59f73740 1 -- 172.21.15.72:0/1183292195 wait complete. 2024-03-22T01:38:20.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.849+0000 7f7c59f73740 20 client.25483 trim_cache size 0 max 16384 2024-03-22T01:38:20.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.893+0000 7f7c59f73740 1 Processor -- start 2024-03-22T01:38:20.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.894+0000 7f7c59f73740 1 -- start start 2024-03-22T01:38:20.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.894+0000 7f7c59f73740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560236a64620 0x560236a6d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:20.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.894+0000 7f7c59f73740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560236a64f40 0x560236a79780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:20.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.894+0000 7f7c59f73740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560236a6db20 0x560236a69be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:20.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.894+0000 7f7c59f73740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f7c48143510 con 0x560236a6db20 2024-03-22T01:38:20.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.894+0000 7f7c59f73740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f7c48145d90 con 0x560236a64620 2024-03-22T01:38:20.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.894+0000 7f7c59f73740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7c48143b80 con 0x560236a64f40 2024-03-22T01:38:20.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.894+0000 7f7c5504b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560236a64620 0x560236a6d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:20.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.894+0000 7f7c5504b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560236a64620 0x560236a6d1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.72:47614/0 (socket says 172.21.15.72:47614) 2024-03-22T01:38:20.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.894+0000 7f7c5504b640 1 -- 172.21.15.72:0/2119624600 learned_addr learned my addr 172.21.15.72:0/2119624600 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:38:20.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.894+0000 7f7c5584c640 1 --2- 172.21.15.72:0/2119624600 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560236a6db20 0x560236a69be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:20.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.898+0000 7f7c5504b640 1 -- 172.21.15.72:0/2119624600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560236a64f40 msgr2=0x560236a79780 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-22T01:38:20.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.898+0000 7f7c5504b640 1 --2- 172.21.15.72:0/2119624600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560236a64f40 0x560236a79780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.898+0000 7f7c5504b640 1 -- 172.21.15.72:0/2119624600 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560236a6db20 msgr2=0x560236a69be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:20.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.898+0000 7f7c5504b640 1 --2- 172.21.15.72:0/2119624600 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560236a6db20 0x560236a69be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.898+0000 7f7c5504b640 1 -- 172.21.15.72:0/2119624600 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56023669c0f0 con 0x560236a64620 2024-03-22T01:38:20.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.899+0000 7f7c5584c640 1 --2- 172.21.15.72:0/2119624600 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560236a6db20 0x560236a69be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:38:20.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.899+0000 7f7c5504b640 1 --2- 172.21.15.72:0/2119624600 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560236a64620 0x560236a6d1b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c0058c0 tx=0x7f7c4c00b560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a388358178c370de server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:38:20.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.900+0000 7f7c3affd640 1 -- 172.21.15.72:0/2119624600 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c4c015040 con 0x560236a64620 2024-03-22T01:38:20.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.900+0000 7f7c3affd640 1 -- 172.21.15.72:0/2119624600 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c4c005520 con 0x560236a64620 2024-03-22T01:38:20.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.902+0000 7f7c59f73740 1 -- 172.21.15.72:0/2119624600 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560236a64620 msgr2=0x560236a6d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:20.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.902+0000 7f7c59f73740 1 --2- 172.21.15.72:0/2119624600 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560236a64620 0x560236a6d1b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c0058c0 tx=0x7f7c4c00b560 comp rx=0 tx=0).stop 2024-03-22T01:38:20.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.904+0000 7f7c59f73740 1 -- 172.21.15.72:0/2119624600 shutdown_connections 2024-03-22T01:38:20.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.904+0000 7f7c59f73740 1 --2- 172.21.15.72:0/2119624600 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560236a6db20 0x560236a69be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.904+0000 7f7c59f73740 1 --2- 172.21.15.72:0/2119624600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560236a64f40 0x560236a79780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.904+0000 7f7c59f73740 1 --2- 172.21.15.72:0/2119624600 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560236a64620 0x560236a6d1b0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.904+0000 7f7c59f73740 1 -- 172.21.15.72:0/2119624600 >> 172.21.15.72:0/2119624600 conn(0x560236a5ed50 msgr2=0x560236a60180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:38:20.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.913+0000 7f7c59f73740 1 -- 172.21.15.72:0/2119624600 shutdown_connections 2024-03-22T01:38:20.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.913+0000 7f7c59f73740 1 -- 172.21.15.72:0/2119624600 wait complete. 2024-03-22T01:38:20.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.914+0000 7f7c59f73740 1 -- start start 2024-03-22T01:38:20.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.914+0000 7f7c59f73740 1 Processor -- start 2024-03-22T01:38:20.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.915+0000 7f7c59f73740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560236a64620 0x560236a76210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:20.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.915+0000 7f7c59f73740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560236a6db20 0x560236a714e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:20.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.915+0000 7f7c59f73740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560236a71b80 0x560236ae3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:20.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.915+0000 7f7c59f73740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f7c48144250 con 0x560236a64620 2024-03-22T01:38:20.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.915+0000 7f7c59f73740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f7c481456c0 con 0x560236a6db20 2024-03-22T01:38:20.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.915+0000 7f7c59f73740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7c48144920 con 0x560236a71b80 2024-03-22T01:38:20.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.915+0000 7f7c59f73740 10 client.0 _ll_register_callbacks cb 0x7f7c50106660 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:38:20.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.924+0000 7f7c5584c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560236a71b80 0x560236ae3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:20.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.924+0000 7f7c5484a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560236a6db20 0x560236a714e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:20.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.925+0000 7f7c5504b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560236a64620 0x560236a76210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:20.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.925+0000 7f7c5504b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560236a64620 0x560236a76210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33364/0 (socket says 172.21.15.72:33364) 2024-03-22T01:38:20.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.925+0000 7f7c5504b640 1 -- 172.21.15.72:0/3325813217 learned_addr learned my addr 172.21.15.72:0/3325813217 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:38:20.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.925+0000 7f7c5504b640 1 -- 172.21.15.72:0/3325813217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560236a71b80 msgr2=0x560236ae3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:20.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.925+0000 7f7c5504b640 1 --2- 172.21.15.72:0/3325813217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560236a71b80 0x560236ae3200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.925+0000 7f7c5504b640 1 -- 172.21.15.72:0/3325813217 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560236a6db20 msgr2=0x560236a714e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:20.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.925+0000 7f7c5504b640 1 --2- 172.21.15.72:0/3325813217 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560236a6db20 0x560236a714e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.925+0000 7f7c5504b640 1 -- 172.21.15.72:0/3325813217 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560236a722c0 con 0x560236a64620 2024-03-22T01:38:20.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.926+0000 7f7c5504b640 1 --2- 172.21.15.72:0/3325813217 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560236a64620 0x560236a76210 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c005bc0 tx=0x7f7c4c027170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81ae5ce9b5f4a88f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:38:20.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.928+0000 7f7c38ff9640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:38:20.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.928+0000 7f7c38ff9640 1 -- 172.21.15.72:0/3325813217 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c4c057060 con 0x560236a64620 2024-03-22T01:38:20.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.928+0000 7f7c38ff9640 1 -- 172.21.15.72:0/3325813217 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f7c4c011a30 con 0x560236a64620 2024-03-22T01:38:20.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.928+0000 7f7c59f73740 1 -- 172.21.15.72:0/3325813217 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c4c005520 con 0x560236a64620 2024-03-22T01:38:20.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.931+0000 7f7c38ff9640 1 -- 172.21.15.72:0/3325813217 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f7c4c015050 con 0x560236a64620 2024-03-22T01:38:20.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.931+0000 7f7c59f73740 10 client.15987 fetch_fsmap learned FSMap version 21 2024-03-22T01:38:20.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.931+0000 7f7c59f73740 1 -- 172.21.15.72:0/3325813217 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x560236ae47c0 con 0x560236a64620 2024-03-22T01:38:20.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.933+0000 7f7c38ff9640 1 -- 172.21.15.72:0/3325813217 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c4c005520 con 0x560236a64620 2024-03-22T01:38:20.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.933+0000 7f7c38ff9640 1 -- 172.21.15.72:0/3325813217 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f7c4c01ac20 con 0x560236a64620 2024-03-22T01:38:20.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.934+0000 7f7c59f73740 10 client.15987 fetch_fsmap finished waiting for FSMap version 21 2024-03-22T01:38:20.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.934+0000 7f7c59f73740 10 client.15987 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T01:38:20.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.934+0000 7f7c59f73740 20 client.15987 populate_metadata read hostname 'smithi072' 2024-03-22T01:38:20.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.934+0000 7f7c59f73740 1 --2- 172.21.15.72:0/3325813217 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x560236ae4b40 0x560236ae6f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:20.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.934+0000 7f7c59f73740 4 client.15987 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T01:38:20.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.934+0000 7f7c59f73740 1 -- 172.21.15.72:0/3325813217 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5602369d0150 con 0x560236ae4b40 2024-03-22T01:38:20.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.934+0000 7f7c5484a640 1 --2- 172.21.15.72:0/3325813217 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x560236ae4b40 0x560236ae6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:20.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.937+0000 7f7c38ff9640 10 client.15987 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:38:20.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.937+0000 7f7c5484a640 1 --2- 172.21.15.72:0/3325813217 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x560236ae4b40 0x560236ae6f30 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17be747d52aa3ec9 server_cookie=45e6f2f4e316a679 in_seq=0 out_seq=0 2024-03-22T01:38:20.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.941+0000 7f7c38ff9640 1 -- 172.21.15.72:0/3325813217 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5602369d0150 con 0x560236ae4b40 2024-03-22T01:38:20.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.941+0000 7f7c38ff9640 10 client.15987 handle_command_reply: tid=0 2024-03-22T01:38:20.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.941+0000 7f7c59f73740 1 client.15987 shutdown 2024-03-22T01:38:20.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.944+0000 7f7c59f73740 1 -- 172.21.15.72:0/3325813217 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560236a64620 msgr2=0x560236a76210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:20.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.944+0000 7f7c59f73740 1 --2- 172.21.15.72:0/3325813217 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560236a64620 0x560236a76210 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c005bc0 tx=0x7f7c4c027170 comp rx=0 tx=0).stop 2024-03-22T01:38:20.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.947+0000 7f7c59f73740 1 -- 172.21.15.72:0/3325813217 shutdown_connections 2024-03-22T01:38:20.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.947+0000 7f7c59f73740 1 --2- 172.21.15.72:0/3325813217 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x560236ae4b40 0x560236ae6f30 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.947+0000 7f7c59f73740 1 --2- 172.21.15.72:0/3325813217 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560236a6db20 0x560236a714e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.947+0000 7f7c59f73740 1 --2- 172.21.15.72:0/3325813217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560236a71b80 0x560236ae3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.947+0000 7f7c59f73740 1 --2- 172.21.15.72:0/3325813217 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560236a64620 0x560236a76210 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.947+0000 7f7c59f73740 1 -- 172.21.15.72:0/3325813217 >> 172.21.15.72:0/3325813217 conn(0x560236a5ed50 msgr2=0x560236a60a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:38:20.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.947+0000 7f7c38ff9640 0 client.15987 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:38:20.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.951+0000 7f7c59f73740 1 -- 172.21.15.72:0/3325813217 shutdown_connections 2024-03-22T01:38:20.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.951+0000 7f7c59f73740 1 -- 172.21.15.72:0/3325813217 wait complete. 2024-03-22T01:38:20.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.951+0000 7f7c59f73740 20 client.15987 trim_cache size 0 max 16384 2024-03-22T01:38:20.960 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-22T01:38:20.960 INFO:teuthology.orchestra.run.smithi072.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-22T01:38:20.960 INFO:teuthology.orchestra.run.smithi072.stdout: "scrubs": { 2024-03-22T01:38:20.960 INFO:teuthology.orchestra.run.smithi072.stdout: "e4b71388-fc76-406f-ab22-6d1ede692217": { 2024-03-22T01:38:20.960 INFO:teuthology.orchestra.run.smithi072.stdout: "path": "/", 2024-03-22T01:38:20.960 INFO:teuthology.orchestra.run.smithi072.stdout: "tag": "e4b71388-fc76-406f-ab22-6d1ede692217", 2024-03-22T01:38:20.960 INFO:teuthology.orchestra.run.smithi072.stdout: "options": "recursive,force" 2024-03-22T01:38:20.960 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:38:20.961 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:38:20.961 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-22T01:38:20.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.960+0000 7f7c5604d640 1 -- 172.21.15.72:0/3640121618 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f7c3c0a3190 msgr2=0x7f7c3c0a5650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:20.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.960+0000 7f7c5604d640 1 --2- 172.21.15.72:0/3640121618 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f7c3c0a3190 0x7f7c3c0a5650 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7c48093350 tx=0x7f7c40005dd0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.960+0000 7f7c5604d640 1 -- 172.21.15.72:0/3640121618 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7c4812e590 msgr2=0x7f7c4812ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:20.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.960+0000 7f7c5604d640 1 --2- 172.21.15.72:0/3640121618 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7c4812e590 0x7f7c4812ea40 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f7c500679d0 tx=0x7f7c50068550 comp rx=0 tx=0).stop 2024-03-22T01:38:20.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.960+0000 7f7c5604d640 1 -- 172.21.15.72:0/3640121618 shutdown_connections 2024-03-22T01:38:20.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.960+0000 7f7c5604d640 1 --2- 172.21.15.72:0/3640121618 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f7c3c0a3190 0x7f7c3c0a5650 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.960+0000 7f7c5604d640 1 --2- 172.21.15.72:0/3640121618 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7c4812e590 0x7f7c4812ea40 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.960+0000 7f7c5604d640 1 --2- 172.21.15.72:0/3640121618 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c4809b030 0x7f7c48134920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.960+0000 7f7c5604d640 1 --2- 172.21.15.72:0/3640121618 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c48095f60 0x7f7c481343e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:20.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.960+0000 7f7c5604d640 1 -- 172.21.15.72:0/3640121618 >> 172.21.15.72:0/3640121618 conn(0x7f7c4800ceb0 msgr2=0x7f7c48093c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:38:20.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.961+0000 7f7c5604d640 1 -- 172.21.15.72:0/3640121618 shutdown_connections 2024-03-22T01:38:20.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:20.961+0000 7f7c5604d640 1 -- 172.21.15.72:0/3640121618 wait complete. 2024-03-22T01:38:20.990 INFO:tasks.cephfs.filesystem:scrub status for tag:e4b71388-fc76-406f-ab22-6d1ede692217 - {'path': '/', 'tag': 'e4b71388-fc76-406f-ab22-6d1ede692217', 'options': 'recursive,force'} 2024-03-22T01:38:21.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:38:21 smithi072 ceph-mon[32414]: pgmap v54: 129 pgs: 129 active+clean; 1.1 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 94 MiB/s wr, 299 op/s 2024-03-22T01:38:21.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:38:21 smithi072 ceph-mon[32414]: scrub queued for path: / 2024-03-22T01:38:21.373 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:38:21 smithi072 ceph-mon[32414]: scrub summary: idle+waiting paths [/] 2024-03-22T01:38:21.373 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:38:21 smithi072 ceph-mon[32414]: scrub summary: active paths [/] 2024-03-22T01:38:21.373 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:38:21 smithi072 ceph-mon[32414]: scrub summary: idle+waiting paths [/] 2024-03-22T01:38:21.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:38:21 smithi098 ceph-mon[36468]: pgmap v54: 129 pgs: 129 active+clean; 1.1 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 94 MiB/s wr, 299 op/s 2024-03-22T01:38:21.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:38:21 smithi098 ceph-mon[36468]: scrub queued for path: / 2024-03-22T01:38:21.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:38:21 smithi098 ceph-mon[36468]: scrub summary: idle+waiting paths [/] 2024-03-22T01:38:21.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:38:21 smithi098 ceph-mon[36468]: scrub summary: active paths [/] 2024-03-22T01:38:21.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:38:21 smithi098 ceph-mon[36468]: scrub summary: idle+waiting paths [/] 2024-03-22T01:38:21.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:38:21 smithi089 ceph-mon[35770]: pgmap v54: 129 pgs: 129 active+clean; 1.1 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 94 MiB/s wr, 299 op/s 2024-03-22T01:38:21.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:38:21 smithi089 ceph-mon[35770]: scrub queued for path: / 2024-03-22T01:38:21.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:38:21 smithi089 ceph-mon[35770]: scrub summary: idle+waiting paths [/] 2024-03-22T01:38:21.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:38:21 smithi089 ceph-mon[35770]: scrub summary: active paths [/] 2024-03-22T01:38:21.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:38:21 smithi089 ceph-mon[35770]: scrub summary: idle+waiting paths [/] 2024-03-22T01:38:23.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:38:23 smithi089 ceph-mon[35770]: pgmap v55: 129 pgs: 129 active+clean; 1.1 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 94 MiB/s wr, 288 op/s 2024-03-22T01:38:23.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:38:23 smithi072 ceph-mon[32414]: pgmap v55: 129 pgs: 129 active+clean; 1.1 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 94 MiB/s wr, 288 op/s 2024-03-22T01:38:23.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:38:23 smithi098 ceph-mon[36468]: pgmap v55: 129 pgs: 129 active+clean; 1.1 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 94 MiB/s wr, 288 op/s 2024-03-22T01:38:24.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:38:24 smithi089 ceph-mon[35770]: scrub summary: idle 2024-03-22T01:38:24.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:38:24 smithi072 ceph-mon[32414]: scrub summary: idle 2024-03-22T01:38:24.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:38:24 smithi098 ceph-mon[36468]: scrub summary: idle 2024-03-22T01:38:25.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:38:25 smithi089 ceph-mon[35770]: pgmap v56: 129 pgs: 129 active+clean; 4.4 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 378 MiB/s wr, 418 op/s 2024-03-22T01:38:25.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:38:25 smithi072 ceph-mon[32414]: pgmap v56: 129 pgs: 129 active+clean; 4.4 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 378 MiB/s wr, 418 op/s 2024-03-22T01:38:25.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:38:25 smithi098 ceph-mon[36468]: pgmap v56: 129 pgs: 129 active+clean; 4.4 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 378 MiB/s wr, 418 op/s 2024-03-22T01:38:27.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:38:27 smithi089 ceph-mon[35770]: pgmap v57: 129 pgs: 129 active+clean; 5.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 433 MiB/s wr, 389 op/s 2024-03-22T01:38:27.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:38:27 smithi072 ceph-mon[32414]: pgmap v57: 129 pgs: 129 active+clean; 5.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 433 MiB/s wr, 389 op/s 2024-03-22T01:38:27.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:38:27 smithi098 ceph-mon[36468]: pgmap v57: 129 pgs: 129 active+clean; 5.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 433 MiB/s wr, 389 op/s 2024-03-22T01:38:29.861 INFO:tasks.workunit.client.0.smithi072.stdout:fs setup took 13 secs 2024-03-22T01:38:29.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:38:29 smithi098 ceph-mon[36468]: pgmap v58: 129 pgs: 129 active+clean; 5.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 433 MiB/s wr, 389 op/s 2024-03-22T01:38:30.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:38:29 smithi089 ceph-mon[35770]: pgmap v58: 129 pgs: 129 active+clean; 5.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 433 MiB/s wr, 389 op/s 2024-03-22T01:38:30.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:38:29 smithi072 ceph-mon[32414]: pgmap v58: 129 pgs: 129 active+clean; 5.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 433 MiB/s wr, 389 op/s 2024-03-22T01:38:30.326 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.324+0000 7f62efa9a640 1 -- 172.21.15.72:0/2062449039 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f62e806a410 msgr2=0x7f62e806a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:30.326 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.324+0000 7f62efa9a640 1 --2- 172.21.15.72:0/2062449039 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f62e806a410 0x7f62e806a7f0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f62e4009900 tx=0x7f62e402bc30 comp rx=0 tx=0).stop 2024-03-22T01:38:30.326 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.324+0000 7f62efa9a640 1 -- 172.21.15.72:0/2062449039 shutdown_connections 2024-03-22T01:38:30.326 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.324+0000 7f62efa9a640 1 --2- 172.21.15.72:0/2062449039 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62e8106f50 0x7f62e810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:30.326 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.324+0000 7f62efa9a640 1 --2- 172.21.15.72:0/2062449039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62e806adc0 0x7f62e8106820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:30.326 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.324+0000 7f62efa9a640 1 --2- 172.21.15.72:0/2062449039 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f62e806a410 0x7f62e806a7f0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:30.326 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.324+0000 7f62efa9a640 1 -- 172.21.15.72:0/2062449039 >> 172.21.15.72:0/2062449039 conn(0x7f62e8077850 msgr2=0x7f62e8077c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:38:30.326 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.324+0000 7f62efa9a640 1 -- 172.21.15.72:0/2062449039 shutdown_connections 2024-03-22T01:38:30.327 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.324+0000 7f62efa9a640 1 -- 172.21.15.72:0/2062449039 wait complete. 2024-03-22T01:38:30.327 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.325+0000 7f62efa9a640 1 Processor -- start 2024-03-22T01:38:30.327 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.325+0000 7f62efa9a640 1 -- start start 2024-03-22T01:38:30.327 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.325+0000 7f62efa9a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62e806a410 0x7f62e8100db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:30.327 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.325+0000 7f62efa9a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f62e8106f50 0x7f62e81012f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:30.327 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.325+0000 7f62efa9a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62e8101830 0x7f62e81a6600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:30.327 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.325+0000 7f62efa9a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f62e8111f70 con 0x7f62e8106f50 2024-03-22T01:38:30.327 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.325+0000 7f62efa9a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f62e8111df0 con 0x7f62e806a410 2024-03-22T01:38:30.327 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.325+0000 7f62efa9a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f62e81120f0 con 0x7f62e8101830 2024-03-22T01:38:30.328 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.325+0000 7f62ed035640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f62e8106f50 0x7f62e81012f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:30.328 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.325+0000 7f62ed035640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f62e8106f50 0x7f62e81012f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33376/0 (socket says 172.21.15.72:33376) 2024-03-22T01:38:30.328 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.325+0000 7f62ed836640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62e806a410 0x7f62e8100db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:30.328 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.325+0000 7f62ed035640 1 -- 172.21.15.72:0/2246175855 learned_addr learned my addr 172.21.15.72:0/2246175855 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:38:30.328 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.325+0000 7f62ee037640 1 --2- 172.21.15.72:0/2246175855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62e8101830 0x7f62e81a6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:30.328 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.325+0000 7f62ed035640 1 -- 172.21.15.72:0/2246175855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62e8101830 msgr2=0x7f62e81a6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:30.328 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.325+0000 7f62ed035640 1 --2- 172.21.15.72:0/2246175855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62e8101830 0x7f62e81a6600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:30.328 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.325+0000 7f62ed035640 1 -- 172.21.15.72:0/2246175855 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62e806a410 msgr2=0x7f62e8100db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:30.328 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.325+0000 7f62ed035640 1 --2- 172.21.15.72:0/2246175855 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62e806a410 0x7f62e8100db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:30.328 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.325+0000 7f62ed035640 1 -- 172.21.15.72:0/2246175855 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62e81a6d00 con 0x7f62e8106f50 2024-03-22T01:38:30.328 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.325+0000 7f62ed836640 1 --2- 172.21.15.72:0/2246175855 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62e806a410 0x7f62e8100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:38:30.328 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.325+0000 7f62ed035640 1 --2- 172.21.15.72:0/2246175855 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f62e8106f50 0x7f62e81012f0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f62d8005fd0 tx=0x7f62d800dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:38:30.328 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.325+0000 7f62deffd640 1 -- 172.21.15.72:0/2246175855 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d8017070 con 0x7f62e8106f50 2024-03-22T01:38:30.328 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.326+0000 7f62deffd640 1 -- 172.21.15.72:0/2246175855 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62d8009c30 con 0x7f62e8106f50 2024-03-22T01:38:30.329 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.326+0000 7f62deffd640 1 -- 172.21.15.72:0/2246175855 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d8003d70 con 0x7f62e8106f50 2024-03-22T01:38:30.329 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.326+0000 7f62efa9a640 1 -- 172.21.15.72:0/2246175855 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62e81a6ff0 con 0x7f62e8106f50 2024-03-22T01:38:30.329 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.326+0000 7f62efa9a640 1 -- 172.21.15.72:0/2246175855 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62e81a74b0 con 0x7f62e8106f50 2024-03-22T01:38:30.336 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.328+0000 7f62deffd640 1 -- 172.21.15.72:0/2246175855 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 25) ==== 137985+0+0 (secure 0 0 0) 0x7f62d8003f10 con 0x7f62e8106f50 2024-03-22T01:38:30.337 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.328+0000 7f62f2684740 1 Processor -- start 2024-03-22T01:38:30.337 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.328+0000 7f62f2684740 1 -- start start 2024-03-22T01:38:30.337 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.328+0000 7f62deffd640 1 --2- 172.21.15.72:0/2246175855 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f62d40a30b0 0x7f62d40a5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:30.337 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.328+0000 7f62f2684740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a02ddfe2d0 0x55a02ddfe6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:30.337 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.328+0000 7f62f2684740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a02ddfebf0 0x55a02de07480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:30.337 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.328+0000 7f62f2684740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a02de079c0 0x55a02de09db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:30.337 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.328+0000 7f62f2684740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55a02dd62040 con 0x55a02ddfe2d0 2024-03-22T01:38:30.337 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.328+0000 7f62f2684740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55a02dd0a9e0 con 0x55a02ddfebf0 2024-03-22T01:38:30.337 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.328+0000 7f62f2684740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a02dd5dff0 con 0x55a02de079c0 2024-03-22T01:38:30.337 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.329+0000 7f62ee037640 1 --2- 172.21.15.72:0/2246175855 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f62d40a30b0 0x7f62d40a5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:30.337 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.329+0000 7f62deffd640 1 -- 172.21.15.72:0/2246175855 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f62d80cf450 con 0x7f62e8106f50 2024-03-22T01:38:30.337 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.329+0000 7f62ed836640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a02ddfe2d0 0x55a02ddfe6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:30.338 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.329+0000 7f62ed836640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a02ddfe2d0 0x55a02ddfe6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33380/0 (socket says 172.21.15.72:33380) 2024-03-22T01:38:30.338 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.329+0000 7f62ed836640 1 -- 172.21.15.72:0/3518732741 learned_addr learned my addr 172.21.15.72:0/3518732741 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:38:30.338 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.329+0000 7f62ed035640 1 --2- 172.21.15.72:0/3518732741 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a02ddfebf0 0x55a02de07480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:30.338 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.329+0000 7f62ee037640 1 --2- 172.21.15.72:0/3518732741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a02de079c0 0x55a02de09db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:30.338 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.329+0000 7f62ed836640 1 -- 172.21.15.72:0/3518732741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a02de079c0 msgr2=0x55a02de09db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:30.338 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.329+0000 7f62ed836640 1 --2- 172.21.15.72:0/3518732741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a02de079c0 0x55a02de09db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:30.338 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.329+0000 7f62ed836640 1 -- 172.21.15.72:0/3518732741 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a02ddfebf0 msgr2=0x55a02de07480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:30.338 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.329+0000 7f62ed836640 1 --2- 172.21.15.72:0/3518732741 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a02ddfebf0 0x55a02de07480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:30.338 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.329+0000 7f62ed836640 1 -- 172.21.15.72:0/3518732741 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a02da76210 con 0x55a02ddfe2d0 2024-03-22T01:38:30.338 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.329+0000 7f62ee037640 1 --2- 172.21.15.72:0/2246175855 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f62d40a30b0 0x7f62d40a5570 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f62e0005c40 tx=0x7f62e0012a90 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:38:30.338 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.329+0000 7f62ed836640 1 --2- 172.21.15.72:0/3518732741 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a02ddfe2d0 0x55a02ddfe6b0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f62e402c170 tx=0x7f62e4009610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b5f726c2125ab21 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:38:30.339 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.329+0000 7f62d3fff640 1 -- 172.21.15.72:0/3518732741 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62e4042070 con 0x55a02ddfe2d0 2024-03-22T01:38:30.339 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.329+0000 7f62d3fff640 1 -- 172.21.15.72:0/3518732741 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62e4004c70 con 0x55a02ddfe2d0 2024-03-22T01:38:30.339 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.329+0000 7f62d3fff640 1 -- 172.21.15.72:0/3518732741 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62e4005180 con 0x55a02ddfe2d0 2024-03-22T01:38:30.339 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.330+0000 7f62f2684740 1 -- 172.21.15.72:0/3518732741 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a02ddfe2d0 msgr2=0x55a02ddfe6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:30.339 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.330+0000 7f62f2684740 1 --2- 172.21.15.72:0/3518732741 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a02ddfe2d0 0x55a02ddfe6b0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f62e402c170 tx=0x7f62e4009610 comp rx=0 tx=0).stop 2024-03-22T01:38:30.339 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.330+0000 7f62f2684740 1 -- 172.21.15.72:0/3518732741 shutdown_connections 2024-03-22T01:38:30.339 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.330+0000 7f62f2684740 1 --2- 172.21.15.72:0/3518732741 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a02ddfebf0 0x55a02de07480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:30.339 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.330+0000 7f62f2684740 1 --2- 172.21.15.72:0/3518732741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a02de079c0 0x55a02de09db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:30.339 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.330+0000 7f62f2684740 1 --2- 172.21.15.72:0/3518732741 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a02ddfe2d0 0x55a02ddfe6b0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:30.339 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.330+0000 7f62f2684740 1 -- 172.21.15.72:0/3518732741 >> 172.21.15.72:0/3518732741 conn(0x55a02ddf8bf0 msgr2=0x55a02ddf8fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:38:30.339 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.330+0000 7f62f2684740 1 -- 172.21.15.72:0/3518732741 shutdown_connections 2024-03-22T01:38:30.339 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.330+0000 7f62f2684740 1 -- 172.21.15.72:0/3518732741 wait complete. 2024-03-22T01:38:30.339 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.330+0000 7f62f2684740 1 -- start start 2024-03-22T01:38:30.339 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.331+0000 7f62f2684740 1 Processor -- start 2024-03-22T01:38:30.340 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.331+0000 7f62f2684740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a02ddfe2d0 0x55a02de072b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:30.340 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.331+0000 7f62f2684740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a02ddfebf0 0x55a02de04640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:30.340 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.331+0000 7f62f2684740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a02de079c0 0x55a02de04b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:30.340 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.331+0000 7f62f2684740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55a02de0a590 con 0x55a02ddfebf0 2024-03-22T01:38:30.340 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.331+0000 7f62f2684740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55a02dd1ddb0 con 0x55a02ddfe2d0 2024-03-22T01:38:30.340 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.331+0000 7f62f2684740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a02de0a710 con 0x55a02de079c0 2024-03-22T01:38:30.340 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.331+0000 7f62f2684740 10 client.0 _ll_register_callbacks cb 0x55a02da75a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:38:30.340 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.331+0000 7f62ed035640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a02ddfebf0 0x55a02de04640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:30.340 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.331+0000 7f62ed035640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a02ddfebf0 0x55a02de04640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33386/0 (socket says 172.21.15.72:33386) 2024-03-22T01:38:30.340 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.331+0000 7f62ed035640 1 -- 172.21.15.72:0/985901415 learned_addr learned my addr 172.21.15.72:0/985901415 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:38:30.340 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.331+0000 7f62ed836640 1 --2- 172.21.15.72:0/985901415 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a02ddfe2d0 0x55a02de072b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:30.340 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.331+0000 7f62ed035640 1 -- 172.21.15.72:0/985901415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a02de079c0 msgr2=0x55a02de04b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:30.340 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.331+0000 7f62ee037640 1 --2- 172.21.15.72:0/985901415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a02de079c0 0x55a02de04b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:30.341 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.331+0000 7f62ed035640 1 --2- 172.21.15.72:0/985901415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a02de079c0 0x55a02de04b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:30.341 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.331+0000 7f62ed035640 1 -- 172.21.15.72:0/985901415 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a02ddfe2d0 msgr2=0x55a02de072b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:30.341 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.331+0000 7f62ed035640 1 --2- 172.21.15.72:0/985901415 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a02ddfe2d0 0x55a02de072b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:30.341 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.331+0000 7f62ed035640 1 -- 172.21.15.72:0/985901415 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a02db42620 con 0x55a02ddfebf0 2024-03-22T01:38:30.341 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.331+0000 7f62ee037640 1 --2- 172.21.15.72:0/985901415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a02de079c0 0x55a02de04b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T01:38:30.341 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.332+0000 7f62ed035640 1 --2- 172.21.15.72:0/985901415 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a02ddfebf0 0x55a02de04640 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f62d80058c0 tx=0x7f62d809aeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4c9ee3997490fc1 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:38:30.341 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.332+0000 7f62d17fa640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:38:30.341 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.332+0000 7f62d17fa640 1 -- 172.21.15.72:0/985901415 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d8020040 con 0x55a02ddfebf0 2024-03-22T01:38:30.341 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.332+0000 7f62d17fa640 1 -- 172.21.15.72:0/985901415 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f62d800f840 con 0x55a02ddfebf0 2024-03-22T01:38:30.341 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.332+0000 7f62f2684740 1 -- 172.21.15.72:0/985901415 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a02dd6f580 con 0x55a02ddfebf0 2024-03-22T01:38:30.341 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.332+0000 7f62d17fa640 1 -- 172.21.15.72:0/985901415 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f62d809bb90 con 0x55a02ddfebf0 2024-03-22T01:38:30.342 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.332+0000 7f62f2684740 10 client.15999 fetch_fsmap learned FSMap version 21 2024-03-22T01:38:30.342 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.332+0000 7f62f2684740 1 -- 172.21.15.72:0/985901415 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55a02de16460 con 0x55a02ddfebf0 2024-03-22T01:38:30.342 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.332+0000 7f62d17fa640 1 -- 172.21.15.72:0/985901415 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55a02dd6f580 con 0x55a02ddfebf0 2024-03-22T01:38:30.342 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.332+0000 7f62d17fa640 1 -- 172.21.15.72:0/985901415 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f62d80cca70 con 0x55a02ddfebf0 2024-03-22T01:38:30.342 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.332+0000 7f62f2684740 10 client.15999 fetch_fsmap finished waiting for FSMap version 21 2024-03-22T01:38:30.342 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.332+0000 7f62f2684740 10 client.15999 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T01:38:30.342 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.332+0000 7f62f2684740 20 client.15999 populate_metadata read hostname 'smithi072' 2024-03-22T01:38:30.343 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.332+0000 7f62f2684740 1 --2- 172.21.15.72:0/985901415 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55a02de17940 0x55a02de19d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:30.343 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.332+0000 7f62f2684740 4 client.15999 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T01:38:30.343 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.332+0000 7f62f2684740 1 -- 172.21.15.72:0/985901415 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a02dd698d0 con 0x55a02de17940 2024-03-22T01:38:30.343 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.332+0000 7f62ed836640 1 --2- 172.21.15.72:0/985901415 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55a02de17940 0x55a02de19d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:30.345 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.333+0000 7f62d17fa640 10 client.15999 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:38:30.345 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.333+0000 7f62ed836640 1 --2- 172.21.15.72:0/985901415 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55a02de17940 0x55a02de19d50 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3117fc447706aef5 server_cookie=ff0895b81d1c8892 in_seq=0 out_seq=0 2024-03-22T01:38:30.345 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.333+0000 7f62d17fa640 1 -- 172.21.15.72:0/985901415 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a02dd698d0 con 0x55a02de17940 2024-03-22T01:38:30.346 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.333+0000 7f62d17fa640 10 client.15999 handle_command_reply: tid=0 2024-03-22T01:38:30.346 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.333+0000 7f62f2684740 1 client.15999 shutdown 2024-03-22T01:38:30.346 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.333+0000 7f62f2684740 1 -- 172.21.15.72:0/985901415 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a02ddfebf0 msgr2=0x55a02de04640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:30.346 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.333+0000 7f62f2684740 1 --2- 172.21.15.72:0/985901415 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a02ddfebf0 0x55a02de04640 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f62d80058c0 tx=0x7f62d809aeb0 comp rx=0 tx=0).stop 2024-03-22T01:38:30.346 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.334+0000 7f62f2684740 1 -- 172.21.15.72:0/985901415 shutdown_connections 2024-03-22T01:38:30.347 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.334+0000 7f62f2684740 1 --2- 172.21.15.72:0/985901415 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55a02de17940 0x55a02de19d50 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:30.347 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.334+0000 7f62f2684740 1 --2- 172.21.15.72:0/985901415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a02de079c0 0x55a02de04b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:30.347 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.334+0000 7f62f2684740 1 --2- 172.21.15.72:0/985901415 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a02ddfebf0 0x55a02de04640 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:30.347 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.334+0000 7f62f2684740 1 --2- 172.21.15.72:0/985901415 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a02ddfe2d0 0x55a02de072b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:30.347 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.334+0000 7f62d17fa640 0 client.15999 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:38:30.347 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.334+0000 7f62f2684740 1 -- 172.21.15.72:0/985901415 >> 172.21.15.72:0/985901415 conn(0x55a02ddf8bf0 msgr2=0x55a02ddfa940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:38:30.348 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.334+0000 7f62f2684740 1 -- 172.21.15.72:0/985901415 shutdown_connections 2024-03-22T01:38:30.348 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.334+0000 7f62f2684740 1 -- 172.21.15.72:0/985901415 wait complete. 2024-03-22T01:38:30.348 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.334+0000 7f62f2684740 20 client.15999 trim_cache size 0 max 16384 2024-03-22T01:38:30.357 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.348+0000 7f62f2684740 1 Processor -- start 2024-03-22T01:38:30.357 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.348+0000 7f62f2684740 1 -- start start 2024-03-22T01:38:30.357 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.348+0000 7f62f2684740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a02ddf8bf0 0x55a02de13330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:30.357 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.348+0000 7f62f2684740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a02ddfe2d0 0x55a02ddfa750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:30.357 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.348+0000 7f62f2684740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a02ddfebf0 0x55a02de1f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:30.357 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.348+0000 7f62f2684740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f62e81af280 con 0x55a02ddf8bf0 2024-03-22T01:38:30.357 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.348+0000 7f62f2684740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f62e81b1b00 con 0x55a02ddfe2d0 2024-03-22T01:38:30.357 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.348+0000 7f62f2684740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f62e81af8f0 con 0x55a02ddfebf0 2024-03-22T01:38:30.358 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.348+0000 7f62ed836640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a02ddf8bf0 0x55a02de13330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:30.358 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.348+0000 7f62ed836640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a02ddf8bf0 0x55a02de13330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33388/0 (socket says 172.21.15.72:33388) 2024-03-22T01:38:30.358 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.348+0000 7f62ed836640 1 -- 172.21.15.72:0/2146483635 learned_addr learned my addr 172.21.15.72:0/2146483635 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:38:30.358 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.348+0000 7f62ed035640 1 --2- 172.21.15.72:0/2146483635 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a02ddfe2d0 0x55a02ddfa750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:30.358 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.348+0000 7f62ee037640 1 --2- 172.21.15.72:0/2146483635 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a02ddfebf0 0x55a02de1f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:30.358 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.349+0000 7f62ed836640 1 -- 172.21.15.72:0/2146483635 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a02ddfebf0 msgr2=0x55a02de1f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:30.358 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.349+0000 7f62ed836640 1 --2- 172.21.15.72:0/2146483635 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a02ddfebf0 0x55a02de1f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:30.358 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.349+0000 7f62ed836640 1 -- 172.21.15.72:0/2146483635 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a02ddfe2d0 msgr2=0x55a02ddfa750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:30.358 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.349+0000 7f62ed836640 1 --2- 172.21.15.72:0/2146483635 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a02ddfe2d0 0x55a02ddfa750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:30.358 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.349+0000 7f62ed836640 1 -- 172.21.15.72:0/2146483635 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a02de02bb0 con 0x55a02ddf8bf0 2024-03-22T01:38:30.359 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.349+0000 7f62ed035640 1 --2- 172.21.15.72:0/2146483635 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a02ddfe2d0 0x55a02ddfa750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:38:30.359 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.349+0000 7f62ee037640 1 --2- 172.21.15.72:0/2146483635 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a02ddfebf0 0x55a02de1f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T01:38:30.359 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.349+0000 7f62ed836640 1 --2- 172.21.15.72:0/2146483635 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a02ddf8bf0 0x55a02de13330 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f62e4005c40 tx=0x7f62e4000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5f1257dbde7bc5d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:38:30.359 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.349+0000 7f62d37fe640 1 -- 172.21.15.72:0/2146483635 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62e4042040 con 0x55a02ddf8bf0 2024-03-22T01:38:30.359 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.349+0000 7f62d37fe640 1 -- 172.21.15.72:0/2146483635 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62e402ca00 con 0x55a02ddf8bf0 2024-03-22T01:38:30.359 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.349+0000 7f62d37fe640 1 -- 172.21.15.72:0/2146483635 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62e4005180 con 0x55a02ddf8bf0 2024-03-22T01:38:30.359 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.349+0000 7f62f2684740 1 -- 172.21.15.72:0/2146483635 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a02ddf8bf0 msgr2=0x55a02de13330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:30.359 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.349+0000 7f62f2684740 1 --2- 172.21.15.72:0/2146483635 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a02ddf8bf0 0x55a02de13330 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f62e4005c40 tx=0x7f62e4000bc0 comp rx=0 tx=0).stop 2024-03-22T01:38:30.359 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.349+0000 7f62f2684740 1 -- 172.21.15.72:0/2146483635 shutdown_connections 2024-03-22T01:38:30.359 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.349+0000 7f62f2684740 1 --2- 172.21.15.72:0/2146483635 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a02ddfe2d0 0x55a02ddfa750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:30.359 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.349+0000 7f62f2684740 1 --2- 172.21.15.72:0/2146483635 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a02ddfebf0 0x55a02de1f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:30.359 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.349+0000 7f62f2684740 1 --2- 172.21.15.72:0/2146483635 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a02ddf8bf0 0x55a02de13330 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:30.359 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.349+0000 7f62f2684740 1 -- 172.21.15.72:0/2146483635 >> 172.21.15.72:0/2146483635 conn(0x55a02de04ce0 msgr2=0x55a02ddfa040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:38:30.359 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.349+0000 7f62f2684740 1 -- 172.21.15.72:0/2146483635 shutdown_connections 2024-03-22T01:38:30.360 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.350+0000 7f62f2684740 1 -- 172.21.15.72:0/2146483635 wait complete. 2024-03-22T01:38:30.360 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.350+0000 7f62f2684740 1 -- start start 2024-03-22T01:38:30.360 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.350+0000 7f62f2684740 1 Processor -- start 2024-03-22T01:38:30.360 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.350+0000 7f62f2684740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a02ddf8bf0 0x55a02de01230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:30.360 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.350+0000 7f62f2684740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a02ddfe2d0 0x55a02de01770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:30.360 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.350+0000 7f62f2684740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a02ddfebf0 0x55a02de01cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:30.360 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.350+0000 7f62f2684740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f62e81affc0 con 0x55a02ddfe2d0 2024-03-22T01:38:30.360 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.350+0000 7f62f2684740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f62e81b1430 con 0x55a02ddf8bf0 2024-03-22T01:38:30.360 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.350+0000 7f62f2684740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f62e81b0690 con 0x55a02ddfebf0 2024-03-22T01:38:30.360 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.350+0000 7f62f2684740 10 client.0 _ll_register_callbacks cb 0x7f62d80977f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:38:30.360 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.350+0000 7f62ed035640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a02ddfe2d0 0x55a02de01770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:30.360 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.350+0000 7f62ed035640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a02ddfe2d0 0x55a02de01770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33390/0 (socket says 172.21.15.72:33390) 2024-03-22T01:38:30.360 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.350+0000 7f62ed035640 1 -- 172.21.15.72:0/2885627761 learned_addr learned my addr 172.21.15.72:0/2885627761 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:38:30.361 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.350+0000 7f62ee037640 1 --2- 172.21.15.72:0/2885627761 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a02ddfebf0 0x55a02de01cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:30.361 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.350+0000 7f62ed836640 1 --2- 172.21.15.72:0/2885627761 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a02ddf8bf0 0x55a02de01230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:30.361 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.351+0000 7f62ed035640 1 -- 172.21.15.72:0/2885627761 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a02ddfebf0 msgr2=0x55a02de01cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:30.361 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.351+0000 7f62ed035640 1 --2- 172.21.15.72:0/2885627761 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a02ddfebf0 0x55a02de01cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:30.361 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.351+0000 7f62ed035640 1 -- 172.21.15.72:0/2885627761 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a02ddf8bf0 msgr2=0x55a02de01230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:30.361 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.351+0000 7f62ed035640 1 --2- 172.21.15.72:0/2885627761 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a02ddf8bf0 0x55a02de01230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:30.361 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.351+0000 7f62ed035640 1 -- 172.21.15.72:0/2885627761 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a02de7cc90 con 0x55a02ddfe2d0 2024-03-22T01:38:30.361 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.351+0000 7f62ed035640 1 --2- 172.21.15.72:0/2885627761 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a02ddfe2d0 0x55a02de01770 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f62d8097e60 tx=0x7f62d8097dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa7935f0eda7bdb9 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:38:30.361 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.351+0000 7f62d17fa640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:38:30.361 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.351+0000 7f62d17fa640 1 -- 172.21.15.72:0/2885627761 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d8020070 con 0x55a02ddfe2d0 2024-03-22T01:38:30.361 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.351+0000 7f62d17fa640 1 -- 172.21.15.72:0/2885627761 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f62d8097b00 con 0x55a02ddfe2d0 2024-03-22T01:38:30.361 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.351+0000 7f62f2684740 1 -- 172.21.15.72:0/2885627761 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62e402ca00 con 0x55a02ddfe2d0 2024-03-22T01:38:30.361 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.351+0000 7f62d17fa640 1 -- 172.21.15.72:0/2885627761 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f62d800fe90 con 0x55a02ddfe2d0 2024-03-22T01:38:30.361 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.351+0000 7f62f2684740 10 client.16005 fetch_fsmap learned FSMap version 21 2024-03-22T01:38:30.361 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.351+0000 7f62f2684740 1 -- 172.21.15.72:0/2885627761 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55a02de7ded0 con 0x55a02ddfe2d0 2024-03-22T01:38:30.361 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.351+0000 7f62d17fa640 1 -- 172.21.15.72:0/2885627761 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62e402ca00 con 0x55a02ddfe2d0 2024-03-22T01:38:30.362 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.351+0000 7f62d17fa640 1 -- 172.21.15.72:0/2885627761 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f62d809f200 con 0x55a02ddfe2d0 2024-03-22T01:38:30.362 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.351+0000 7f62f2684740 10 client.16005 fetch_fsmap finished waiting for FSMap version 21 2024-03-22T01:38:30.362 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.351+0000 7f62f2684740 10 client.16005 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T01:38:30.362 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.351+0000 7f62f2684740 20 client.16005 populate_metadata read hostname 'smithi072' 2024-03-22T01:38:30.362 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.351+0000 7f62f2684740 1 --2- 172.21.15.72:0/2885627761 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55a02de079c0 0x55a02de80500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:30.362 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.351+0000 7f62f2684740 4 client.16005 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T01:38:30.362 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.351+0000 7f62f2684740 1 -- 172.21.15.72:0/2885627761 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a02dce7770 con 0x55a02de079c0 2024-03-22T01:38:30.362 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.352+0000 7f62ed836640 1 --2- 172.21.15.72:0/2885627761 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55a02de079c0 0x55a02de80500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:30.362 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.352+0000 7f62ed836640 1 --2- 172.21.15.72:0/2885627761 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55a02de079c0 0x55a02de80500 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e2411180b2c78adc server_cookie=8ea47aefe676b73c in_seq=0 out_seq=0 2024-03-22T01:38:30.362 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.352+0000 7f62d17fa640 10 client.16005 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:38:30.408 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.408+0000 7f62d17fa640 1 -- 172.21.15.72:0/2885627761 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a02dce7770 con 0x55a02de079c0 2024-03-22T01:38:30.408 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.408+0000 7f62d17fa640 10 client.16005 handle_command_reply: tid=0 2024-03-22T01:38:30.408 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.408+0000 7f62f2684740 1 client.16005 shutdown 2024-03-22T01:38:30.408 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.408+0000 7f62f2684740 1 -- 172.21.15.72:0/2885627761 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a02ddfe2d0 msgr2=0x55a02de01770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:30.408 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.408+0000 7f62f2684740 1 --2- 172.21.15.72:0/2885627761 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a02ddfe2d0 0x55a02de01770 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f62d8097e60 tx=0x7f62d8097dd0 comp rx=0 tx=0).stop 2024-03-22T01:38:30.409 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.409+0000 7f62f2684740 1 -- 172.21.15.72:0/2885627761 shutdown_connections 2024-03-22T01:38:30.409 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.409+0000 7f62f2684740 1 --2- 172.21.15.72:0/2885627761 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55a02de079c0 0x55a02de80500 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:30.409 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.409+0000 7f62f2684740 1 --2- 172.21.15.72:0/2885627761 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a02ddfebf0 0x55a02de01cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:30.409 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.409+0000 7f62d17fa640 0 client.16005 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:38:30.409 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.409+0000 7f62f2684740 1 --2- 172.21.15.72:0/2885627761 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a02ddfe2d0 0x55a02de01770 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:30.409 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.409+0000 7f62f2684740 1 --2- 172.21.15.72:0/2885627761 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a02ddf8bf0 0x55a02de01230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:30.409 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.409+0000 7f62f2684740 1 -- 172.21.15.72:0/2885627761 >> 172.21.15.72:0/2885627761 conn(0x55a02de04ce0 msgr2=0x55a02ddfa8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:38:30.409 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.409+0000 7f62f2684740 1 -- 172.21.15.72:0/2885627761 shutdown_connections 2024-03-22T01:38:30.409 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.409+0000 7f62f2684740 1 -- 172.21.15.72:0/2885627761 wait complete. 2024-03-22T01:38:30.409 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.409+0000 7f62f2684740 20 client.16005 trim_cache size 0 max 16384 2024-03-22T01:38:30.409 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout:{ 2024-03-22T01:38:30.410 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout: "message": "", 2024-03-22T01:38:30.410 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout: "return_code": 0 2024-03-22T01:38:30.410 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout:} 2024-03-22T01:38:30.411 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.409+0000 7f62efa9a640 1 -- 172.21.15.72:0/2246175855 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f62d40a30b0 msgr2=0x7f62d40a5570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:30.411 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.409+0000 7f62efa9a640 1 --2- 172.21.15.72:0/2246175855 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f62d40a30b0 0x7f62d40a5570 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f62e0005c40 tx=0x7f62e0012a90 comp rx=0 tx=0).stop 2024-03-22T01:38:30.411 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.409+0000 7f62efa9a640 1 -- 172.21.15.72:0/2246175855 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f62e8106f50 msgr2=0x7f62e81012f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:30.411 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.409+0000 7f62efa9a640 1 --2- 172.21.15.72:0/2246175855 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f62e8106f50 0x7f62e81012f0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f62d8005fd0 tx=0x7f62d800dde0 comp rx=0 tx=0).stop 2024-03-22T01:38:30.411 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.409+0000 7f62efa9a640 1 -- 172.21.15.72:0/2246175855 shutdown_connections 2024-03-22T01:38:30.411 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.409+0000 7f62efa9a640 1 --2- 172.21.15.72:0/2246175855 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f62d40a30b0 0x7f62d40a5570 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:30.411 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.409+0000 7f62efa9a640 1 --2- 172.21.15.72:0/2246175855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62e8101830 0x7f62e81a6600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:30.411 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.409+0000 7f62efa9a640 1 --2- 172.21.15.72:0/2246175855 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f62e8106f50 0x7f62e81012f0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:30.411 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.409+0000 7f62efa9a640 1 --2- 172.21.15.72:0/2246175855 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62e806a410 0x7f62e8100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:30.411 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.409+0000 7f62efa9a640 1 -- 172.21.15.72:0/2246175855 >> 172.21.15.72:0/2246175855 conn(0x7f62e8077850 msgr2=0x7f62e80feeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:38:30.411 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.409+0000 7f62efa9a640 1 -- 172.21.15.72:0/2246175855 shutdown_connections 2024-03-22T01:38:30.411 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:30.409+0000 7f62efa9a640 1 -- 172.21.15.72:0/2246175855 wait complete. 2024-03-22T01:38:31.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:38:31 smithi098 ceph-mon[36468]: pgmap v59: 129 pgs: 129 active+clean; 9.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 780 MiB/s wr, 518 op/s 2024-03-22T01:38:32.107 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:38:31 smithi089 ceph-mon[35770]: pgmap v59: 129 pgs: 129 active+clean; 9.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 780 MiB/s wr, 518 op/s 2024-03-22T01:38:32.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:38:31 smithi072 ceph-mon[32414]: pgmap v59: 129 pgs: 129 active+clean; 9.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 780 MiB/s wr, 518 op/s 2024-03-22T01:38:33.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:38:33 smithi098 ceph-mon[36468]: pgmap v60: 129 pgs: 129 active+clean; 9.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 687 MiB/s wr, 276 op/s 2024-03-22T01:38:34.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:38:33 smithi089 ceph-mon[35770]: pgmap v60: 129 pgs: 129 active+clean; 9.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 687 MiB/s wr, 276 op/s 2024-03-22T01:38:34.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:38:33 smithi072 ceph-mon[32414]: pgmap v60: 129 pgs: 129 active+clean; 9.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 687 MiB/s wr, 276 op/s 2024-03-22T01:38:34.478 INFO:tasks.workunit.client.0.smithi072.stdout:Syncing()...4 sec 2024-03-22T01:38:34.479 INFO:tasks.workunit.client.0.smithi072.stdout:Starting Actual Benchmark At: Fri Mar 22 01:38:34 2024 2024-03-22T01:38:34.479 INFO:tasks.workunit.client.0.smithi072.stdout: 2024-03-22T01:38:35.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:38:35 smithi098 ceph-mon[36468]: pgmap v61: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 829 MiB/s wr, 345 op/s 2024-03-22T01:38:36.107 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:38:35 smithi089 ceph-mon[35770]: pgmap v61: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 829 MiB/s wr, 345 op/s 2024-03-22T01:38:36.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:38:35 smithi072 ceph-mon[32414]: pgmap v61: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 829 MiB/s wr, 345 op/s 2024-03-22T01:38:38.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:38:37 smithi089 ceph-mon[35770]: pgmap v62: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 680 MiB/s wr, 259 op/s 2024-03-22T01:38:38.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:38:37 smithi072 ceph-mon[32414]: pgmap v62: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 680 MiB/s wr, 259 op/s 2024-03-22T01:38:38.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:38:37 smithi098 ceph-mon[36468]: pgmap v62: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 680 MiB/s wr, 259 op/s 2024-03-22T01:38:40.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:38:39 smithi089 ceph-mon[35770]: pgmap v63: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 624 MiB/s wr, 241 op/s 2024-03-22T01:38:40.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:38:39 smithi072 ceph-mon[32414]: pgmap v63: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 624 MiB/s wr, 241 op/s 2024-03-22T01:38:40.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:38:39 smithi098 ceph-mon[36468]: pgmap v63: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 624 MiB/s wr, 241 op/s 2024-03-22T01:38:42.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:38:41 smithi089 ceph-mon[35770]: pgmap v64: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 662 MiB/s wr, 5.11k op/s 2024-03-22T01:38:42.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:38:41 smithi072 ceph-mon[32414]: pgmap v64: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 662 MiB/s wr, 5.11k op/s 2024-03-22T01:38:42.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:38:41 smithi098 ceph-mon[36468]: pgmap v64: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 662 MiB/s wr, 5.11k op/s 2024-03-22T01:38:43.535 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.533+0000 7f3e3b7ac640 1 -- 172.21.15.72:0/432303222 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e34107c10 msgr2=0x7f3e34107ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:43.536 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.533+0000 7f3e3b7ac640 1 --2- 172.21.15.72:0/432303222 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e34107c10 0x7f3e34107ff0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f3e30009900 tx=0x7f3e3002bc30 comp rx=0 tx=0).stop 2024-03-22T01:38:43.536 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.533+0000 7f3e3b7ac640 1 -- 172.21.15.72:0/432303222 shutdown_connections 2024-03-22T01:38:43.536 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.533+0000 7f3e3b7ac640 1 --2- 172.21.15.72:0/432303222 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e34069e10 0x7f3e3410e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:43.536 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.533+0000 7f3e3b7ac640 1 --2- 172.21.15.72:0/432303222 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e34069450 0x7f3e340698d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:43.536 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.533+0000 7f3e3b7ac640 1 --2- 172.21.15.72:0/432303222 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e34107c10 0x7f3e34107ff0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:43.536 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.533+0000 7f3e3b7ac640 1 -- 172.21.15.72:0/432303222 >> 172.21.15.72:0/432303222 conn(0x7f3e34077b80 msgr2=0x7f3e34077f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:38:43.536 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.533+0000 7f3e3b7ac640 1 -- 172.21.15.72:0/432303222 shutdown_connections 2024-03-22T01:38:43.536 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.533+0000 7f3e3b7ac640 1 -- 172.21.15.72:0/432303222 wait complete. 2024-03-22T01:38:43.536 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.534+0000 7f3e3b7ac640 1 Processor -- start 2024-03-22T01:38:43.536 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.534+0000 7f3e3b7ac640 1 -- start start 2024-03-22T01:38:43.536 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.534+0000 7f3e3b7ac640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e34069450 0x7f3e3410bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:43.536 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.534+0000 7f3e3b7ac640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e34069e10 0x7f3e3410c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:43.537 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.534+0000 7f3e3b7ac640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e34107c10 0x7f3e3410c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:43.537 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.534+0000 7f3e3b7ac640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f3e34102640 con 0x7f3e34069e10 2024-03-22T01:38:43.537 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.534+0000 7f3e3b7ac640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3e341024c0 con 0x7f3e34107c10 2024-03-22T01:38:43.537 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.534+0000 7f3e3b7ac640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3e341027c0 con 0x7f3e34069450 2024-03-22T01:38:43.537 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.534+0000 7f3e38d47640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e34069e10 0x7f3e3410c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:43.537 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.534+0000 7f3e38d47640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e34069e10 0x7f3e3410c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:41734/0 (socket says 172.21.15.72:41734) 2024-03-22T01:38:43.537 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.534+0000 7f3e38d47640 1 -- 172.21.15.72:0/1386165290 learned_addr learned my addr 172.21.15.72:0/1386165290 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:38:43.537 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.534+0000 7f3e39548640 1 --2- 172.21.15.72:0/1386165290 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e34069450 0x7f3e3410bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:43.537 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.535+0000 7f3e38d47640 1 -- 172.21.15.72:0/1386165290 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e34069450 msgr2=0x7f3e3410bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:43.537 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.535+0000 7f3e38d47640 1 --2- 172.21.15.72:0/1386165290 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e34069450 0x7f3e3410bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:43.537 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.535+0000 7f3e38d47640 1 -- 172.21.15.72:0/1386165290 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e34107c10 msgr2=0x7f3e3410c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:43.537 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.535+0000 7f3e38d47640 1 --2- 172.21.15.72:0/1386165290 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e34107c10 0x7f3e3410c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:43.537 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.535+0000 7f3e38d47640 1 -- 172.21.15.72:0/1386165290 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e341aac20 con 0x7f3e34069e10 2024-03-22T01:38:43.538 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.535+0000 7f3e39548640 1 --2- 172.21.15.72:0/1386165290 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e34069450 0x7f3e3410bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:38:43.538 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.535+0000 7f3e38d47640 1 --2- 172.21.15.72:0/1386165290 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e34069e10 0x7f3e3410c090 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f3e2400d7a0 tx=0x7f3e2400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:38:43.538 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.535+0000 7f3e2a7fc640 1 -- 172.21.15.72:0/1386165290 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e24017070 con 0x7f3e34069e10 2024-03-22T01:38:43.538 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.535+0000 7f3e3b7ac640 1 -- 172.21.15.72:0/1386165290 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e341aaf10 con 0x7f3e34069e10 2024-03-22T01:38:43.538 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.535+0000 7f3e2a7fc640 1 -- 172.21.15.72:0/1386165290 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e24005500 con 0x7f3e34069e10 2024-03-22T01:38:43.538 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.535+0000 7f3e2a7fc640 1 -- 172.21.15.72:0/1386165290 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e24004e00 con 0x7f3e34069e10 2024-03-22T01:38:43.538 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.535+0000 7f3e3b7ac640 1 -- 172.21.15.72:0/1386165290 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e341ab3d0 con 0x7f3e34069e10 2024-03-22T01:38:43.540 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.537+0000 7f3e2a7fc640 1 -- 172.21.15.72:0/1386165290 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 25) ==== 137985+0+0 (secure 0 0 0) 0x7f3e24009c30 con 0x7f3e34069e10 2024-03-22T01:38:43.540 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.538+0000 7f3e3e38d740 1 Processor -- start 2024-03-22T01:38:43.540 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.538+0000 7f3e3e38d740 1 -- start start 2024-03-22T01:38:43.540 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.538+0000 7f3e3e38d740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55af9cd592d0 0x55af9cd596b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:43.540 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.538+0000 7f3e3e38d740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55af9cd59bf0 0x55af9cd62480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:43.540 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.538+0000 7f3e3e38d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af9cd629c0 0x55af9cd64db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:43.540 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.538+0000 7f3e3e38d740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55af9ccbd040 con 0x55af9cd592d0 2024-03-22T01:38:43.541 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.538+0000 7f3e3e38d740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55af9cc659e0 con 0x55af9cd59bf0 2024-03-22T01:38:43.541 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.538+0000 7f3e3e38d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55af9ccb8ff0 con 0x55af9cd629c0 2024-03-22T01:38:43.541 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.538+0000 7f3e39d49640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55af9cd592d0 0x55af9cd596b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:43.541 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.538+0000 7f3e2a7fc640 1 --2- 172.21.15.72:0/1386165290 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f3e200a3180 0x7f3e200a5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:43.541 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.538+0000 7f3e39d49640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55af9cd592d0 0x55af9cd596b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:41742/0 (socket says 172.21.15.72:41742) 2024-03-22T01:38:43.541 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.538+0000 7f3e39d49640 1 -- 172.21.15.72:0/639517176 learned_addr learned my addr 172.21.15.72:0/639517176 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:38:43.541 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.538+0000 7f3e39548640 1 --2- 172.21.15.72:0/639517176 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55af9cd59bf0 0x55af9cd62480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:43.541 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.539+0000 7f3e2a7fc640 1 -- 172.21.15.72:0/1386165290 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f3e240cf690 con 0x7f3e34069e10 2024-03-22T01:38:43.541 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.539+0000 7f3e39d49640 1 --2- 172.21.15.72:0/1386165290 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f3e200a3180 0x7f3e200a5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:43.541 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.539+0000 7f3e39d49640 1 -- 172.21.15.72:0/639517176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af9cd629c0 msgr2=0x55af9cd64db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:43.541 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.539+0000 7f3e39d49640 1 --2- 172.21.15.72:0/639517176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af9cd629c0 0x55af9cd64db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:43.541 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.539+0000 7f3e39d49640 1 -- 172.21.15.72:0/639517176 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55af9cd59bf0 msgr2=0x55af9cd62480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:43.541 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.539+0000 7f3e39d49640 1 --2- 172.21.15.72:0/639517176 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55af9cd59bf0 0x55af9cd62480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:43.541 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.539+0000 7f3e39d49640 1 -- 172.21.15.72:0/639517176 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55af9c9d1210 con 0x55af9cd592d0 2024-03-22T01:38:43.542 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.539+0000 7f3e39d49640 1 --2- 172.21.15.72:0/639517176 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55af9cd592d0 0x55af9cd596b0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c015900 tx=0x7f3e2c015dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df2158540fda3b4b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:38:43.542 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.540+0000 7f3e39d49640 1 --2- 172.21.15.72:0/1386165290 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f3e200a3180 0x7f3e200a5640 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c01f000 tx=0x7f3e2c01b000 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:38:43.542 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.540+0000 7f3e1f7fe640 1 -- 172.21.15.72:0/639517176 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e2c022070 con 0x55af9cd592d0 2024-03-22T01:38:43.542 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.540+0000 7f3e1f7fe640 1 -- 172.21.15.72:0/639517176 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e2c0152f0 con 0x55af9cd592d0 2024-03-22T01:38:43.542 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.540+0000 7f3e1f7fe640 1 -- 172.21.15.72:0/639517176 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e2c005340 con 0x55af9cd592d0 2024-03-22T01:38:43.544 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.541+0000 7f3e3e38d740 1 -- 172.21.15.72:0/639517176 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55af9cd592d0 msgr2=0x55af9cd596b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:43.544 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.541+0000 7f3e3e38d740 1 --2- 172.21.15.72:0/639517176 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55af9cd592d0 0x55af9cd596b0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c015900 tx=0x7f3e2c015dc0 comp rx=0 tx=0).stop 2024-03-22T01:38:43.544 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.541+0000 7f3e3e38d740 1 -- 172.21.15.72:0/639517176 shutdown_connections 2024-03-22T01:38:43.545 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.541+0000 7f3e3e38d740 1 --2- 172.21.15.72:0/639517176 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55af9cd59bf0 0x55af9cd62480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:43.545 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.541+0000 7f3e3e38d740 1 --2- 172.21.15.72:0/639517176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af9cd629c0 0x55af9cd64db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:43.545 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.541+0000 7f3e3e38d740 1 --2- 172.21.15.72:0/639517176 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55af9cd592d0 0x55af9cd596b0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:43.545 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.541+0000 7f3e3e38d740 1 -- 172.21.15.72:0/639517176 >> 172.21.15.72:0/639517176 conn(0x55af9cd53bf0 msgr2=0x55af9cd53fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:38:43.545 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.541+0000 7f3e3e38d740 1 -- 172.21.15.72:0/639517176 shutdown_connections 2024-03-22T01:38:43.545 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.541+0000 7f3e3e38d740 1 -- 172.21.15.72:0/639517176 wait complete. 2024-03-22T01:38:43.545 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.541+0000 7f3e3e38d740 1 -- start start 2024-03-22T01:38:43.545 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.541+0000 7f3e3e38d740 1 Processor -- start 2024-03-22T01:38:43.545 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.542+0000 7f3e3e38d740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55af9cd592d0 0x55af9cd623b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:43.545 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.542+0000 7f3e3e38d740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55af9cd59bf0 0x55af9cd5f740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:43.545 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.542+0000 7f3e3e38d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af9cd629c0 0x55af9cd5fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:43.546 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.542+0000 7f3e3e38d740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55af9cd655f0 con 0x55af9cd59bf0 2024-03-22T01:38:43.546 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.542+0000 7f3e3e38d740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55af9cc78db0 con 0x55af9cd592d0 2024-03-22T01:38:43.546 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.542+0000 7f3e3e38d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55af9cd65770 con 0x55af9cd629c0 2024-03-22T01:38:43.546 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.542+0000 7f3e3e38d740 10 client.0 _ll_register_callbacks cb 0x55af9c9d0a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:38:43.546 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.542+0000 7f3e39548640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55af9cd592d0 0x55af9cd623b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:43.546 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.543+0000 7f3e39548640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55af9cd592d0 0x55af9cd623b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.72:42204/0 (socket says 172.21.15.72:42204) 2024-03-22T01:38:43.546 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.543+0000 7f3e39548640 1 -- 172.21.15.72:0/3192573622 learned_addr learned my addr 172.21.15.72:0/3192573622 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:38:43.546 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.543+0000 7f3e39d49640 1 --2- 172.21.15.72:0/3192573622 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af9cd629c0 0x55af9cd5fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:43.546 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.543+0000 7f3e38d47640 1 --2- 172.21.15.72:0/3192573622 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55af9cd59bf0 0x55af9cd5f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:43.546 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.543+0000 7f3e39d49640 1 -- 172.21.15.72:0/3192573622 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55af9cd592d0 msgr2=0x55af9cd623b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:43.546 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.543+0000 7f3e39d49640 1 --2- 172.21.15.72:0/3192573622 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55af9cd592d0 0x55af9cd623b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:43.546 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.543+0000 7f3e39d49640 1 -- 172.21.15.72:0/3192573622 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55af9cd59bf0 msgr2=0x55af9cd5f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:43.546 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.543+0000 7f3e39d49640 1 --2- 172.21.15.72:0/3192573622 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55af9cd59bf0 0x55af9cd5f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:43.547 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.543+0000 7f3e39d49640 1 -- 172.21.15.72:0/3192573622 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55af9ca9d620 con 0x55af9cd629c0 2024-03-22T01:38:43.547 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.543+0000 7f3e38d47640 1 --2- 172.21.15.72:0/3192573622 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55af9cd59bf0 0x55af9cd5f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:38:43.547 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.543+0000 7f3e39548640 1 --2- 172.21.15.72:0/3192573622 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55af9cd592d0 0x55af9cd623b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:38:43.547 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.544+0000 7f3e39d49640 1 --2- 172.21.15.72:0/3192573622 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af9cd629c0 0x55af9cd5fd50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c012760 tx=0x7f3e2c01dc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=43d7b312fcff52a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:38:43.547 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.544+0000 7f3e1cff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-03-22T01:38:43.547 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.544+0000 7f3e1cff9640 1 -- 172.21.15.72:0/3192573622 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e2c022070 con 0x55af9cd629c0 2024-03-22T01:38:43.547 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.544+0000 7f3e3e38d740 1 -- 172.21.15.72:0/3192573622 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e2c0152f0 con 0x55af9cd629c0 2024-03-22T01:38:43.547 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.544+0000 7f3e1cff9640 1 -- 172.21.15.72:0/3192573622 <== mon.1 v2:172.21.15.98:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f3e2c004b20 con 0x55af9cd629c0 2024-03-22T01:38:43.548 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.545+0000 7f3e1cff9640 1 -- 172.21.15.72:0/3192573622 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f3e2c004d20 con 0x55af9cd629c0 2024-03-22T01:38:43.548 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.545+0000 7f3e3e38d740 10 client.25504 fetch_fsmap learned FSMap version 21 2024-03-22T01:38:43.549 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.545+0000 7f3e3e38d740 1 -- 172.21.15.72:0/3192573622 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55af9cd715f0 con 0x55af9cd629c0 2024-03-22T01:38:43.549 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.545+0000 7f3e1cff9640 1 -- 172.21.15.72:0/3192573622 <== mon.1 v2:172.21.15.98:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e2c0152f0 con 0x55af9cd629c0 2024-03-22T01:38:43.549 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.546+0000 7f3e1cff9640 1 -- 172.21.15.72:0/3192573622 <== mon.1 v2:172.21.15.98:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f3e2c005910 con 0x55af9cd629c0 2024-03-22T01:38:43.549 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.546+0000 7f3e3e38d740 10 client.25504 fetch_fsmap finished waiting for FSMap version 21 2024-03-22T01:38:43.549 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.547+0000 7f3e3e38d740 10 client.25504 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T01:38:43.549 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.547+0000 7f3e3e38d740 20 client.25504 populate_metadata read hostname 'smithi072' 2024-03-22T01:38:43.549 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.547+0000 7f3e3e38d740 1 --2- 172.21.15.72:0/3192573622 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55af9cd72ab0 0x55af9cd74ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:43.549 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.547+0000 7f3e3e38d740 4 client.25504 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T01:38:43.549 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.547+0000 7f3e3e38d740 1 -- 172.21.15.72:0/3192573622 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55af9ccca580 con 0x55af9cd72ab0 2024-03-22T01:38:43.549 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.548+0000 7f3e39548640 1 --2- 172.21.15.72:0/3192573622 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55af9cd72ab0 0x55af9cd74ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:43.550 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.550+0000 7f3e1cff9640 10 client.25504 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:38:43.550 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.550+0000 7f3e39548640 1 --2- 172.21.15.72:0/3192573622 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55af9cd72ab0 0x55af9cd74ea0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=45ede8fcf500a1d server_cookie=8be05e613f29460f in_seq=0 out_seq=0 2024-03-22T01:38:43.551 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.551+0000 7f3e1cff9640 1 -- 172.21.15.72:0/3192573622 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55af9ccca580 con 0x55af9cd72ab0 2024-03-22T01:38:43.551 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.551+0000 7f3e1cff9640 10 client.25504 handle_command_reply: tid=0 2024-03-22T01:38:43.551 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.551+0000 7f3e3e38d740 1 client.25504 shutdown 2024-03-22T01:38:43.551 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.551+0000 7f3e3e38d740 1 -- 172.21.15.72:0/3192573622 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af9cd629c0 msgr2=0x55af9cd5fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:43.552 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.551+0000 7f3e3e38d740 1 --2- 172.21.15.72:0/3192573622 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af9cd629c0 0x55af9cd5fd50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c012760 tx=0x7f3e2c01dc30 comp rx=0 tx=0).stop 2024-03-22T01:38:43.552 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.551+0000 7f3e3e38d740 1 -- 172.21.15.72:0/3192573622 shutdown_connections 2024-03-22T01:38:43.552 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.551+0000 7f3e3e38d740 1 --2- 172.21.15.72:0/3192573622 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55af9cd72ab0 0x55af9cd74ea0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:43.552 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.551+0000 7f3e3e38d740 1 --2- 172.21.15.72:0/3192573622 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af9cd629c0 0x55af9cd5fd50 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:43.552 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.551+0000 7f3e1cff9640 0 client.25504 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:38:43.552 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.551+0000 7f3e3e38d740 1 --2- 172.21.15.72:0/3192573622 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55af9cd59bf0 0x55af9cd5f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:43.552 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.551+0000 7f3e3e38d740 1 --2- 172.21.15.72:0/3192573622 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55af9cd592d0 0x55af9cd623b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:43.552 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.551+0000 7f3e3e38d740 1 -- 172.21.15.72:0/3192573622 >> 172.21.15.72:0/3192573622 conn(0x55af9cd53bf0 msgr2=0x55af9cd55940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:38:43.554 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.553+0000 7f3e3e38d740 1 -- 172.21.15.72:0/3192573622 shutdown_connections 2024-03-22T01:38:43.554 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.554+0000 7f3e3e38d740 1 -- 172.21.15.72:0/3192573622 wait complete. 2024-03-22T01:38:43.554 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.554+0000 7f3e3e38d740 20 client.25504 trim_cache size 0 max 16384 2024-03-22T01:38:43.575 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.575+0000 7f3e3e38d740 1 Processor -- start 2024-03-22T01:38:43.575 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.575+0000 7f3e3e38d740 1 -- start start 2024-03-22T01:38:43.577 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.576+0000 7f3e3e38d740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55af9cd592d0 0x55af9cd6e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:43.577 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.576+0000 7f3e3e38d740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55af9cd59bf0 0x55af9cd7a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:43.578 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.576+0000 7f3e3e38d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af9cd629c0 0x55af9cd61e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:43.578 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.576+0000 7f3e3e38d740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f3e341b3600 con 0x55af9cd592d0 2024-03-22T01:38:43.578 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.576+0000 7f3e3e38d740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3e341b5e80 con 0x55af9cd59bf0 2024-03-22T01:38:43.578 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.576+0000 7f3e3e38d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3e341b3c70 con 0x55af9cd629c0 2024-03-22T01:38:43.578 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.576+0000 7f3e39548640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55af9cd592d0 0x55af9cd6e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:43.578 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.576+0000 7f3e39548640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55af9cd592d0 0x55af9cd6e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:41760/0 (socket says 172.21.15.72:41760) 2024-03-22T01:38:43.578 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.576+0000 7f3e39548640 1 -- 172.21.15.72:0/2610073867 learned_addr learned my addr 172.21.15.72:0/2610073867 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:38:43.578 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.576+0000 7f3e38d47640 1 --2- 172.21.15.72:0/2610073867 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55af9cd59bf0 0x55af9cd7a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:43.578 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.576+0000 7f3e39d49640 1 --2- 172.21.15.72:0/2610073867 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af9cd629c0 0x55af9cd61e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:43.578 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.576+0000 7f3e39548640 1 -- 172.21.15.72:0/2610073867 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af9cd629c0 msgr2=0x55af9cd61e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:43.578 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.576+0000 7f3e39548640 1 --2- 172.21.15.72:0/2610073867 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af9cd629c0 0x55af9cd61e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:43.578 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.576+0000 7f3e39548640 1 -- 172.21.15.72:0/2610073867 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55af9cd59bf0 msgr2=0x55af9cd7a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:43.579 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.576+0000 7f3e39548640 1 --2- 172.21.15.72:0/2610073867 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55af9cd59bf0 0x55af9cd7a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:43.579 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.576+0000 7f3e39548640 1 -- 172.21.15.72:0/2610073867 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55af9cd59a20 con 0x55af9cd592d0 2024-03-22T01:38:43.579 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.577+0000 7f3e39548640 1 --2- 172.21.15.72:0/2610073867 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55af9cd592d0 0x55af9cd6e390 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f3e30004650 tx=0x7f3e3002e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abf97d14e122a4aa server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:38:43.579 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.577+0000 7f3e1effd640 1 -- 172.21.15.72:0/2610073867 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e3004b020 con 0x55af9cd592d0 2024-03-22T01:38:43.579 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.577+0000 7f3e1effd640 1 -- 172.21.15.72:0/2610073867 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e30047ca0 con 0x55af9cd592d0 2024-03-22T01:38:43.579 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.577+0000 7f3e1effd640 1 -- 172.21.15.72:0/2610073867 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e30005040 con 0x55af9cd592d0 2024-03-22T01:38:43.582 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.578+0000 7f3e3e38d740 1 -- 172.21.15.72:0/2610073867 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55af9cd592d0 msgr2=0x55af9cd6e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:43.583 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.578+0000 7f3e3e38d740 1 --2- 172.21.15.72:0/2610073867 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55af9cd592d0 0x55af9cd6e390 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f3e30004650 tx=0x7f3e3002e040 comp rx=0 tx=0).stop 2024-03-22T01:38:43.583 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.578+0000 7f3e3e38d740 1 -- 172.21.15.72:0/2610073867 shutdown_connections 2024-03-22T01:38:43.583 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.578+0000 7f3e3e38d740 1 --2- 172.21.15.72:0/2610073867 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55af9cd59bf0 0x55af9cd7a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:43.583 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.578+0000 7f3e3e38d740 1 --2- 172.21.15.72:0/2610073867 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af9cd629c0 0x55af9cd61e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:43.583 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.578+0000 7f3e3e38d740 1 --2- 172.21.15.72:0/2610073867 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55af9cd592d0 0x55af9cd6e390 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:43.583 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.578+0000 7f3e3e38d740 1 -- 172.21.15.72:0/2610073867 >> 172.21.15.72:0/2610073867 conn(0x55af9cd53bf0 msgr2=0x55af9cd52c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:38:43.583 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.578+0000 7f3e3e38d740 1 -- 172.21.15.72:0/2610073867 shutdown_connections 2024-03-22T01:38:43.583 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.578+0000 7f3e3e38d740 1 -- 172.21.15.72:0/2610073867 wait complete. 2024-03-22T01:38:43.583 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.578+0000 7f3e3e38d740 1 -- start start 2024-03-22T01:38:43.583 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.579+0000 7f3e3e38d740 1 Processor -- start 2024-03-22T01:38:43.583 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.579+0000 7f3e3e38d740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55af9cd592d0 0x55af9cd6a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:43.583 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.579+0000 7f3e3e38d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af9cd59bf0 0x55af9cd5b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:43.583 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.579+0000 7f3e3e38d740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55af9cd629c0 0x55af9cd5bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:43.583 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.579+0000 7f3e3e38d740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f3e341b4340 con 0x55af9cd592d0 2024-03-22T01:38:43.583 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.579+0000 7f3e3e38d740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3e341b57b0 con 0x55af9cd629c0 2024-03-22T01:38:43.584 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.579+0000 7f3e3e38d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3e341b4a10 con 0x55af9cd59bf0 2024-03-22T01:38:43.584 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.579+0000 7f3e3e38d740 10 client.0 _ll_register_callbacks cb 0x7f3e2c020070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:38:43.584 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.579+0000 7f3e39548640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55af9cd592d0 0x55af9cd6a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:43.584 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.579+0000 7f3e39d49640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55af9cd629c0 0x55af9cd5bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:43.584 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.579+0000 7f3e39548640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55af9cd592d0 0x55af9cd6a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:41768/0 (socket says 172.21.15.72:41768) 2024-03-22T01:38:43.584 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.579+0000 7f3e39548640 1 -- 172.21.15.72:0/2014627542 learned_addr learned my addr 172.21.15.72:0/2014627542 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:38:43.584 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.579+0000 7f3e38d47640 1 --2- 172.21.15.72:0/2014627542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af9cd59bf0 0x55af9cd5b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:43.584 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.579+0000 7f3e39548640 1 -- 172.21.15.72:0/2014627542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af9cd59bf0 msgr2=0x55af9cd5b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:43.584 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.579+0000 7f3e39548640 1 --2- 172.21.15.72:0/2014627542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af9cd59bf0 0x55af9cd5b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:43.584 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.579+0000 7f3e39548640 1 -- 172.21.15.72:0/2014627542 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55af9cd629c0 msgr2=0x55af9cd5bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:43.584 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.579+0000 7f3e39548640 1 --2- 172.21.15.72:0/2014627542 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55af9cd629c0 0x55af9cd5bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:43.584 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.579+0000 7f3e39548640 1 -- 172.21.15.72:0/2014627542 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55af9cd6afb0 con 0x55af9cd592d0 2024-03-22T01:38:43.584 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.580+0000 7f3e39548640 1 --2- 172.21.15.72:0/2014627542 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55af9cd592d0 0x55af9cd6a9c0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f3e30005bb0 tx=0x7f3e30047340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9faece841c37ef24 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:38:43.584 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.580+0000 7f3e1cff9640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:38:43.585 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.580+0000 7f3e1cff9640 1 -- 172.21.15.72:0/2014627542 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e3004b030 con 0x55af9cd592d0 2024-03-22T01:38:43.585 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.580+0000 7f3e1cff9640 1 -- 172.21.15.72:0/2014627542 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f3e3002c8b0 con 0x55af9cd592d0 2024-03-22T01:38:43.585 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.580+0000 7f3e3e38d740 1 -- 172.21.15.72:0/2014627542 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e30047ca0 con 0x55af9cd592d0 2024-03-22T01:38:43.585 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.580+0000 7f3e1cff9640 1 -- 172.21.15.72:0/2014627542 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f3e30060050 con 0x55af9cd592d0 2024-03-22T01:38:43.585 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.580+0000 7f3e3e38d740 10 client.16023 fetch_fsmap learned FSMap version 21 2024-03-22T01:38:43.585 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.580+0000 7f3e3e38d740 1 -- 172.21.15.72:0/2014627542 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55af9cdd8520 con 0x55af9cd592d0 2024-03-22T01:38:43.585 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.580+0000 7f3e1cff9640 1 -- 172.21.15.72:0/2014627542 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e30047ca0 con 0x55af9cd592d0 2024-03-22T01:38:43.585 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.580+0000 7f3e1cff9640 1 -- 172.21.15.72:0/2014627542 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f3e30004990 con 0x55af9cd592d0 2024-03-22T01:38:43.585 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.580+0000 7f3e3e38d740 10 client.16023 fetch_fsmap finished waiting for FSMap version 21 2024-03-22T01:38:43.585 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.580+0000 7f3e3e38d740 10 client.16023 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T01:38:43.585 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.580+0000 7f3e3e38d740 20 client.16023 populate_metadata read hostname 'smithi072' 2024-03-22T01:38:43.585 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.580+0000 7f3e3e38d740 1 --2- 172.21.15.72:0/2014627542 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55af9cd72ab0 0x55af9cddad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:43.585 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.580+0000 7f3e3e38d740 4 client.16023 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T01:38:43.585 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.580+0000 7f3e3e38d740 1 -- 172.21.15.72:0/2014627542 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55af9ccc48d0 con 0x55af9cd72ab0 2024-03-22T01:38:43.586 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.580+0000 7f3e38d47640 1 --2- 172.21.15.72:0/2014627542 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55af9cd72ab0 0x55af9cddad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:43.586 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.581+0000 7f3e1cff9640 10 client.16023 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:38:43.586 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:38:43.581+0000 7f3e38d47640 1 --2- 172.21.15.72:0/2014627542 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55af9cd72ab0 0x55af9cddad00 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=28a4c7d711e57623 server_cookie=242407a562544d10 in_seq=0 out_seq=0 2024-03-22T01:38:44.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:38:43 smithi089 ceph-mon[35770]: pgmap v65: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 315 MiB/s wr, 4.98k op/s 2024-03-22T01:38:44.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:38:43 smithi072 ceph-mon[32414]: pgmap v65: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 315 MiB/s wr, 4.98k op/s 2024-03-22T01:38:44.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:38:43 smithi098 ceph-mon[36468]: pgmap v65: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 315 MiB/s wr, 4.98k op/s 2024-03-22T01:38:45.023 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:38:44 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:38:45.107 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:38:44 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:38:45.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:38:44 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:38:46.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:38:45 smithi089 ceph-mon[35770]: pgmap v66: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 315 MiB/s wr, 4.98k op/s 2024-03-22T01:38:46.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:38:45 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:38:46.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:38:45 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:38:46.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:38:45 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:38:46.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:38:45 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:38:46.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:38:45 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:38:46.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:38:45 smithi072 ceph-mon[32414]: pgmap v66: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 315 MiB/s wr, 4.98k op/s 2024-03-22T01:38:46.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:38:45 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:38:46.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:38:45 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:38:46.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:38:45 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:38:46.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:38:45 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:38:46.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:38:45 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:38:46.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:38:45 smithi098 ceph-mon[36468]: pgmap v66: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 315 MiB/s wr, 4.98k op/s 2024-03-22T01:38:46.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:38:45 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:38:46.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:38:45 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:38:46.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:38:45 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:38:46.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:38:45 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:38:46.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:38:45 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:38:48.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:38:47 smithi089 ceph-mon[35770]: pgmap v67: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s rd, 200 MiB/s wr, 9.34k op/s 2024-03-22T01:38:48.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:38:47 smithi072 ceph-mon[32414]: pgmap v67: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s rd, 200 MiB/s wr, 9.34k op/s 2024-03-22T01:38:48.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:38:47 smithi098 ceph-mon[36468]: pgmap v67: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s rd, 200 MiB/s wr, 9.34k op/s 2024-03-22T01:38:50.357 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:38:49 smithi089 ceph-mon[35770]: pgmap v68: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s rd, 65 MiB/s wr, 9.29k op/s 2024-03-22T01:38:50.371 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:38:49 smithi072 ceph-mon[32414]: pgmap v68: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s rd, 65 MiB/s wr, 9.29k op/s 2024-03-22T01:38:50.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:38:49 smithi098 ceph-mon[36468]: pgmap v68: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s rd, 65 MiB/s wr, 9.29k op/s 2024-03-22T01:38:50.991 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T01:38:50.991 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T01:38:51.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.114+0000 7fd9950fa640 1 -- 172.21.15.72:0/168043800 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd990106a40 msgr2=0x7fd990106e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:51.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.114+0000 7fd9950fa640 1 --2- 172.21.15.72:0/168043800 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd990106a40 0x7fd990106e20 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd980009900 tx=0x7fd98002bc30 comp rx=0 tx=0).stop 2024-03-22T01:38:51.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.115+0000 7fd9950fa640 1 -- 172.21.15.72:0/168043800 shutdown_connections 2024-03-22T01:38:51.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.115+0000 7fd9950fa640 1 --2- 172.21.15.72:0/168043800 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9901013f0 0x7fd99010d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:51.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.115+0000 7fd9950fa640 1 --2- 172.21.15.72:0/168043800 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd990100a30 0x7fd990100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:51.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.115+0000 7fd9950fa640 1 --2- 172.21.15.72:0/168043800 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd990106a40 0x7fd990106e20 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:51.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.115+0000 7fd9950fa640 1 -- 172.21.15.72:0/168043800 >> 172.21.15.72:0/168043800 conn(0x7fd990077c40 msgr2=0x7fd990078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:38:51.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.115+0000 7fd9950fa640 1 -- 172.21.15.72:0/168043800 shutdown_connections 2024-03-22T01:38:51.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.115+0000 7fd9950fa640 1 -- 172.21.15.72:0/168043800 wait complete. 2024-03-22T01:38:51.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.115+0000 7fd9950fa640 1 Processor -- start 2024-03-22T01:38:51.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.115+0000 7fd9950fa640 1 -- start start 2024-03-22T01:38:51.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.116+0000 7fd9950fa640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd990100a30 0x7fd9900fd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:51.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.116+0000 7fd9950fa640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9901013f0 0x7fd9900fdf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:51.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.116+0000 7fd9950fa640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd990106a40 0x7fd9900fe460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:51.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.116+0000 7fd9950fa640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fd990110e20 con 0x7fd9901013f0 2024-03-22T01:38:51.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.116+0000 7fd9950fa640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd990110ca0 con 0x7fd990106a40 2024-03-22T01:38:51.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.116+0000 7fd9950fa640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd990110fa0 con 0x7fd990100a30 2024-03-22T01:38:51.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.116+0000 7fd98ed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd990100a30 0x7fd9900fd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:51.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.116+0000 7fd98f59e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd990106a40 0x7fd9900fe460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:51.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.116+0000 7fd987fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9901013f0 0x7fd9900fdf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:51.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.116+0000 7fd98ed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd990100a30 0x7fd9900fd9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.72:44438/0 (socket says 172.21.15.72:44438) 2024-03-22T01:38:51.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.116+0000 7fd98f59e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd990106a40 0x7fd9900fe460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.72:39768/0 (socket says 172.21.15.72:39768) 2024-03-22T01:38:51.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.116+0000 7fd98ed9d640 1 -- 172.21.15.72:0/1855106146 learned_addr learned my addr 172.21.15.72:0/1855106146 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:38:51.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.116+0000 7fd987fff640 1 -- 172.21.15.72:0/1855106146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd990100a30 msgr2=0x7fd9900fd9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:51.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.116+0000 7fd987fff640 1 --2- 172.21.15.72:0/1855106146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd990100a30 0x7fd9900fd9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:51.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.116+0000 7fd987fff640 1 -- 172.21.15.72:0/1855106146 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd990106a40 msgr2=0x7fd9900fe460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:51.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.116+0000 7fd987fff640 1 --2- 172.21.15.72:0/1855106146 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd990106a40 0x7fd9900fe460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:51.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.116+0000 7fd987fff640 1 -- 172.21.15.72:0/1855106146 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9901a68d0 con 0x7fd9901013f0 2024-03-22T01:38:51.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.116+0000 7fd98f59e640 1 --2- 172.21.15.72:0/1855106146 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd990106a40 0x7fd9900fe460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:38:51.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.116+0000 7fd987fff640 1 --2- 172.21.15.72:0/1855106146 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9901013f0 0x7fd9900fdf20 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fd97800e9f0 tx=0x7fd97800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:38:51.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.116+0000 7fd98cd99640 1 -- 172.21.15.72:0/1855106146 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd97800ccc0 con 0x7fd9901013f0 2024-03-22T01:38:51.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.116+0000 7fd98cd99640 1 -- 172.21.15.72:0/1855106146 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd978010910 con 0x7fd9901013f0 2024-03-22T01:38:51.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.116+0000 7fd9950fa640 1 -- 172.21.15.72:0/1855106146 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9901a6bc0 con 0x7fd9901013f0 2024-03-22T01:38:51.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.116+0000 7fd98cd99640 1 -- 172.21.15.72:0/1855106146 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd978018960 con 0x7fd9901013f0 2024-03-22T01:38:51.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.117+0000 7fd9950fa640 1 -- 172.21.15.72:0/1855106146 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9901a7110 con 0x7fd9901013f0 2024-03-22T01:38:51.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.119+0000 7fd98cd99640 1 -- 172.21.15.72:0/1855106146 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 25) ==== 137985+0+0 (secure 0 0 0) 0x7fd978010ac0 con 0x7fd9901013f0 2024-03-22T01:38:51.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.119+0000 7fd98cd99640 1 --2- 172.21.15.72:0/1855106146 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fd97c0a3090 0x7fd97c0a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:51.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.119+0000 7fd997db6740 1 Processor -- start 2024-03-22T01:38:51.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.119+0000 7fd997db6740 1 -- start start 2024-03-22T01:38:51.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.119+0000 7fd98ed9d640 1 --2- 172.21.15.72:0/1855106146 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fd97c0a3090 0x7fd97c0a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:51.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.119+0000 7fd98cd99640 1 -- 172.21.15.72:0/1855106146 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fd9780cff20 con 0x7fd9901013f0 2024-03-22T01:38:51.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.120+0000 7fd997db6740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55682e211410 0x55682e2117f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:51.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.120+0000 7fd997db6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55682e211d30 0x55682e21a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:51.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.120+0000 7fd997db6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55682e21a910 0x55682e21cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:51.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.120+0000 7fd997db6740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55682e1758c0 con 0x55682e211410 2024-03-22T01:38:51.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.120+0000 7fd997db6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55682e11dc20 con 0x55682e21a910 2024-03-22T01:38:51.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.120+0000 7fd997db6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55682e171870 con 0x55682e211d30 2024-03-22T01:38:51.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.120+0000 7fd98ed9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55682e211410 0x55682e2117f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:51.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.120+0000 7fd98ed9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55682e211410 0x55682e2117f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:52426/0 (socket says 172.21.15.72:52426) 2024-03-22T01:38:51.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.120+0000 7fd98ed9d640 1 -- 172.21.15.72:0/1697062143 learned_addr learned my addr 172.21.15.72:0/1697062143 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:38:51.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.120+0000 7fd98f59e640 1 --2- 172.21.15.72:0/1697062143 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55682e21a910 0x55682e21cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:51.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.120+0000 7fd987fff640 1 --2- 172.21.15.72:0/1697062143 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55682e211d30 0x55682e21a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:51.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.120+0000 7fd98ed9d640 1 --2- 172.21.15.72:0/1855106146 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fd97c0a3090 0x7fd97c0a5550 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd980000c00 tx=0x7fd98002c6f0 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:38:51.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.120+0000 7fd98ed9d640 1 -- 172.21.15.72:0/1697062143 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55682e211d30 msgr2=0x55682e21a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:51.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.120+0000 7fd98ed9d640 1 --2- 172.21.15.72:0/1697062143 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55682e211d30 0x55682e21a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:51.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.120+0000 7fd98ed9d640 1 -- 172.21.15.72:0/1697062143 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55682e21a910 msgr2=0x55682e21cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:51.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.120+0000 7fd98ed9d640 1 --2- 172.21.15.72:0/1697062143 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55682e21a910 0x55682e21cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:51.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.120+0000 7fd98ed9d640 1 -- 172.21.15.72:0/1697062143 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55682e142150 con 0x55682e211410 2024-03-22T01:38:51.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.120+0000 7fd98f59e640 1 --2- 172.21.15.72:0/1697062143 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55682e21a910 0x55682e21cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:38:51.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.120+0000 7fd98ed9d640 1 --2- 172.21.15.72:0/1697062143 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55682e211410 0x55682e2117f0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fd980036040 tx=0x7fd980047000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdedd77a1c07cf84 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:38:51.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.121+0000 7fd9857fa640 1 -- 172.21.15.72:0/1697062143 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd980050070 con 0x55682e211410 2024-03-22T01:38:51.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.121+0000 7fd9857fa640 1 -- 172.21.15.72:0/1697062143 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd980004ba0 con 0x55682e211410 2024-03-22T01:38:51.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.121+0000 7fd9857fa640 1 -- 172.21.15.72:0/1697062143 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9800035f0 con 0x55682e211410 2024-03-22T01:38:51.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.121+0000 7fd997db6740 1 -- 172.21.15.72:0/1697062143 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55682e211410 msgr2=0x55682e2117f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:38:51.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.121+0000 7fd997db6740 1 --2- 172.21.15.72:0/1697062143 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55682e211410 0x55682e2117f0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fd980036040 tx=0x7fd980047000 comp rx=0 tx=0).stop 2024-03-22T01:38:51.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.121+0000 7fd997db6740 1 -- 172.21.15.72:0/1697062143 shutdown_connections 2024-03-22T01:38:51.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.121+0000 7fd997db6740 1 --2- 172.21.15.72:0/1697062143 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55682e21a910 0x55682e21cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:51.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.121+0000 7fd997db6740 1 --2- 172.21.15.72:0/1697062143 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55682e211d30 0x55682e21a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:51.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.121+0000 7fd997db6740 1 --2- 172.21.15.72:0/1697062143 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55682e211410 0x55682e2117f0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:51.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.121+0000 7fd997db6740 1 -- 172.21.15.72:0/1697062143 >> 172.21.15.72:0/1697062143 conn(0x55682e20bb40 msgr2=0x55682e20bf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:38:51.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.121+0000 7fd997db6740 1 -- 172.21.15.72:0/1697062143 shutdown_connections 2024-03-22T01:38:51.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.121+0000 7fd997db6740 1 -- 172.21.15.72:0/1697062143 wait complete. 2024-03-22T01:38:51.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.122+0000 7fd997db6740 1 -- start start 2024-03-22T01:38:51.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.122+0000 7fd997db6740 1 Processor -- start 2024-03-22T01:38:51.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.122+0000 7fd997db6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55682e211410 0x55682e2178e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:51.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.122+0000 7fd997db6740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55682e211d30 0x55682e217e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:51.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.122+0000 7fd997db6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55682e21a910 0x55682e218360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:51.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.122+0000 7fd997db6740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55682e21d6a0 con 0x55682e211d30 2024-03-22T01:38:51.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.122+0000 7fd997db6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55682e131130 con 0x55682e21a910 2024-03-22T01:38:51.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.122+0000 7fd997db6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55682e21d820 con 0x55682e211410 2024-03-22T01:38:51.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.122+0000 7fd987fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55682e211d30 0x55682e217e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:51.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.122+0000 7fd997db6740 10 client.0 _ll_register_callbacks cb 0x55682e130b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:38:51.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.122+0000 7fd98ed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55682e211410 0x55682e2178e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:51.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.122+0000 7fd987fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55682e211d30 0x55682e217e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:52432/0 (socket says 172.21.15.72:52432) 2024-03-22T01:38:51.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.122+0000 7fd987fff640 1 -- 172.21.15.72:0/1183142685 learned_addr learned my addr 172.21.15.72:0/1183142685 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:38:51.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.122+0000 7fd98f59e640 1 --2- 172.21.15.72:0/1183142685 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55682e21a910 0x55682e218360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:51.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.123+0000 7fd987fff640 1 -- 172.21.15.72:0/1183142685 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55682e211410 msgr2=0x55682e2178e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:51.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.123+0000 7fd987fff640 1 --2- 172.21.15.72:0/1183142685 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55682e211410 0x55682e2178e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:51.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.123+0000 7fd987fff640 1 -- 172.21.15.72:0/1183142685 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55682e21a910 msgr2=0x55682e218360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:38:51.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.123+0000 7fd987fff640 1 --2- 172.21.15.72:0/1183142685 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55682e21a910 0x55682e218360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:38:51.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.123+0000 7fd987fff640 1 -- 172.21.15.72:0/1183142685 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55682df537d0 con 0x55682e211d30 2024-03-22T01:38:51.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.123+0000 7fd98ed9d640 1 --2- 172.21.15.72:0/1183142685 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55682e211410 0x55682e2178e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:38:51.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.123+0000 7fd98f59e640 1 --2- 172.21.15.72:0/1183142685 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55682e21a910 0x55682e218360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:38:51.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.123+0000 7fd987fff640 1 --2- 172.21.15.72:0/1183142685 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55682e211d30 0x55682e217e20 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fd97809bfa0 tx=0x7fd9780a00b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78b4d78cee9539d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:38:51.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.123+0000 7fd9767fc640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:38:51.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.123+0000 7fd9767fc640 1 -- 172.21.15.72:0/1183142685 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd978096d80 con 0x55682e211d30 2024-03-22T01:38:51.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.123+0000 7fd9767fc640 1 -- 172.21.15.72:0/1183142685 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fd97809b460 con 0x55682e211d30 2024-03-22T01:38:51.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.123+0000 7fd997db6740 1 -- 172.21.15.72:0/1183142685 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd980004ba0 con 0x55682e211d30 2024-03-22T01:38:51.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.124+0000 7fd9767fc640 1 -- 172.21.15.72:0/1183142685 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fd9780bc170 con 0x55682e211d30 2024-03-22T01:38:51.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.124+0000 7fd997db6740 10 client.16035 fetch_fsmap learned FSMap version 21 2024-03-22T01:38:51.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.124+0000 7fd997db6740 1 -- 172.21.15.72:0/1183142685 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55682de893f0 con 0x55682e211d30 2024-03-22T01:38:51.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.124+0000 7fd9767fc640 1 -- 172.21.15.72:0/1183142685 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd97809b660 con 0x55682e211d30 2024-03-22T01:38:51.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.124+0000 7fd9767fc640 1 -- 172.21.15.72:0/1183142685 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fd978097df0 con 0x55682e211d30 2024-03-22T01:38:51.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.124+0000 7fd997db6740 10 client.16035 fetch_fsmap finished waiting for FSMap version 21 2024-03-22T01:38:51.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.124+0000 7fd997db6740 10 client.16035 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T01:38:51.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.124+0000 7fd997db6740 20 client.16035 populate_metadata read hostname 'smithi072' 2024-03-22T01:38:51.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.124+0000 7fd997db6740 1 --2- 172.21.15.72:0/1183142685 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55682e22ab80 0x55682e22cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:38:51.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.124+0000 7fd997db6740 4 client.16035 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T01:38:51.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.124+0000 7fd997db6740 1 -- 172.21.15.72:0/1183142685 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fd980004ba0 con 0x55682e22ab80 2024-03-22T01:38:51.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.124+0000 7fd98ed9d640 1 --2- 172.21.15.72:0/1183142685 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55682e22ab80 0x55682e22cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:38:51.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.125+0000 7fd9767fc640 10 client.16035 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:38:51.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:38:51.125+0000 7fd98ed9d640 1 --2- 172.21.15.72:0/1183142685 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55682e22ab80 0x55682e22cf70 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c0717d1b33d650b1 server_cookie=695ca288dbf719d in_seq=0 out_seq=0 2024-03-22T01:38:52.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:38:51 smithi089 ceph-mon[35770]: pgmap v69: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s rd, 74 MiB/s wr, 10.39k op/s 2024-03-22T01:38:52.809 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:38:51 smithi098 ceph-mon[36468]: pgmap v69: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s rd, 74 MiB/s wr, 10.39k op/s 2024-03-22T01:38:52.809 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:38:51 smithi072 ceph-mon[32414]: pgmap v69: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s rd, 74 MiB/s wr, 10.39k op/s 2024-03-22T01:38:54.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:38:53 smithi089 ceph-mon[35770]: pgmap v70: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 5.59k op/s 2024-03-22T01:38:54.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:38:53 smithi072 ceph-mon[32414]: pgmap v70: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 5.59k op/s 2024-03-22T01:38:54.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:38:53 smithi098 ceph-mon[36468]: pgmap v70: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 5.59k op/s 2024-03-22T01:38:56.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:38:55 smithi089 ceph-mon[35770]: pgmap v71: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 37 MiB/s wr, 5.71k op/s 2024-03-22T01:38:56.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:38:55 smithi072 ceph-mon[32414]: pgmap v71: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 37 MiB/s wr, 5.71k op/s 2024-03-22T01:38:56.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:38:55 smithi098 ceph-mon[36468]: pgmap v71: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 37 MiB/s wr, 5.71k op/s 2024-03-22T01:38:58.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:38:57 smithi089 ceph-mon[35770]: pgmap v72: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 46 MiB/s wr, 7.01k op/s 2024-03-22T01:38:58.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:38:57 smithi072 ceph-mon[32414]: pgmap v72: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 46 MiB/s wr, 7.01k op/s 2024-03-22T01:38:58.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:38:57 smithi098 ceph-mon[36468]: pgmap v72: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 46 MiB/s wr, 7.01k op/s 2024-03-22T01:39:00.223 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:38:59 smithi098 ceph-mon[36468]: pgmap v73: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s wr, 2.58k op/s 2024-03-22T01:39:00.357 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:38:59 smithi089 ceph-mon[35770]: pgmap v73: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s wr, 2.58k op/s 2024-03-22T01:39:00.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:38:59 smithi072 ceph-mon[32414]: pgmap v73: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s wr, 2.58k op/s 2024-03-22T01:39:02.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:39:01 smithi089 ceph-mon[35770]: pgmap v74: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 32 MiB/s wr, 4.44k op/s 2024-03-22T01:39:02.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:39:01 smithi072 ceph-mon[32414]: pgmap v74: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 32 MiB/s wr, 4.44k op/s 2024-03-22T01:39:02.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:39:01 smithi098 ceph-mon[36468]: pgmap v74: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 32 MiB/s wr, 4.44k op/s 2024-03-22T01:39:04.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:39:03 smithi089 ceph-mon[35770]: pgmap v75: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s wr, 3.51k op/s 2024-03-22T01:39:04.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:39:03 smithi072 ceph-mon[32414]: pgmap v75: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s wr, 3.51k op/s 2024-03-22T01:39:04.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:39:03 smithi098 ceph-mon[36468]: pgmap v75: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s wr, 3.51k op/s 2024-03-22T01:39:06.357 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:39:05 smithi089 ceph-mon[35770]: pgmap v76: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s wr, 3.59k op/s 2024-03-22T01:39:06.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:39:05 smithi072 ceph-mon[32414]: pgmap v76: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s wr, 3.59k op/s 2024-03-22T01:39:06.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:39:05 smithi098 ceph-mon[36468]: pgmap v76: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s wr, 3.59k op/s 2024-03-22T01:39:08.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:39:07 smithi089 ceph-mon[35770]: pgmap v77: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 35 MiB/s wr, 5.13k op/s 2024-03-22T01:39:08.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:39:07 smithi072 ceph-mon[32414]: pgmap v77: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 35 MiB/s wr, 5.13k op/s 2024-03-22T01:39:08.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:39:07 smithi098 ceph-mon[36468]: pgmap v77: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 35 MiB/s wr, 5.13k op/s 2024-03-22T01:39:10.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:39:09 smithi089 ceph-mon[35770]: pgmap v78: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 28 MiB/s wr, 4.01k op/s 2024-03-22T01:39:10.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:39:09 smithi072 ceph-mon[32414]: pgmap v78: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 28 MiB/s wr, 4.01k op/s 2024-03-22T01:39:10.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:39:09 smithi098 ceph-mon[36468]: pgmap v78: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 28 MiB/s wr, 4.01k op/s 2024-03-22T01:39:12.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:39:11 smithi089 ceph-mon[35770]: pgmap v79: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 38 MiB/s wr, 5.56k op/s 2024-03-22T01:39:12.371 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:39:11 smithi072 ceph-mon[32414]: pgmap v79: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 38 MiB/s wr, 5.56k op/s 2024-03-22T01:39:12.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:39:11 smithi098 ceph-mon[36468]: pgmap v79: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 38 MiB/s wr, 5.56k op/s 2024-03-22T01:39:14.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:39:13 smithi089 ceph-mon[35770]: pgmap v80: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 26 MiB/s wr, 3.89k op/s 2024-03-22T01:39:14.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:39:13 smithi072 ceph-mon[32414]: pgmap v80: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 26 MiB/s wr, 3.89k op/s 2024-03-22T01:39:14.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:39:13 smithi098 ceph-mon[36468]: pgmap v80: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 26 MiB/s wr, 3.89k op/s 2024-03-22T01:39:16.357 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:39:15 smithi089 ceph-mon[35770]: pgmap v81: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 26 MiB/s wr, 3.79k op/s 2024-03-22T01:39:16.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:39:15 smithi072 ceph-mon[32414]: pgmap v81: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 26 MiB/s wr, 3.79k op/s 2024-03-22T01:39:16.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:39:15 smithi098 ceph-mon[36468]: pgmap v81: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 26 MiB/s wr, 3.79k op/s 2024-03-22T01:39:18.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:39:17 smithi089 ceph-mon[35770]: pgmap v82: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 36 MiB/s wr, 5.26k op/s 2024-03-22T01:39:18.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:39:17 smithi072 ceph-mon[32414]: pgmap v82: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 36 MiB/s wr, 5.26k op/s 2024-03-22T01:39:18.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:39:17 smithi098 ceph-mon[36468]: pgmap v82: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 36 MiB/s wr, 5.26k op/s 2024-03-22T01:39:20.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:39:19 smithi089 ceph-mon[35770]: pgmap v83: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 28 MiB/s wr, 4.05k op/s 2024-03-22T01:39:20.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:39:19 smithi072 ceph-mon[32414]: pgmap v83: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 28 MiB/s wr, 4.05k op/s 2024-03-22T01:39:20.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:39:19 smithi098 ceph-mon[36468]: pgmap v83: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 28 MiB/s wr, 4.05k op/s 2024-03-22T01:39:22.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:39:21 smithi089 ceph-mon[35770]: pgmap v84: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 5.29k op/s 2024-03-22T01:39:22.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:39:21 smithi072 ceph-mon[32414]: pgmap v84: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 5.29k op/s 2024-03-22T01:39:22.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:39:21 smithi098 ceph-mon[36468]: pgmap v84: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 5.29k op/s 2024-03-22T01:39:24.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:39:23 smithi089 ceph-mon[35770]: pgmap v85: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s wr, 3.93k op/s 2024-03-22T01:39:24.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:39:23 smithi072 ceph-mon[32414]: pgmap v85: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s wr, 3.93k op/s 2024-03-22T01:39:24.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:39:23 smithi098 ceph-mon[36468]: pgmap v85: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s wr, 3.93k op/s 2024-03-22T01:39:26.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:39:25 smithi089 ceph-mon[35770]: pgmap v86: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 28 MiB/s wr, 4.10k op/s 2024-03-22T01:39:26.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:39:25 smithi072 ceph-mon[32414]: pgmap v86: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 28 MiB/s wr, 4.10k op/s 2024-03-22T01:39:26.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:39:25 smithi098 ceph-mon[36468]: pgmap v86: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 28 MiB/s wr, 4.10k op/s 2024-03-22T01:39:28.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:39:27 smithi089 ceph-mon[35770]: pgmap v87: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 38 MiB/s wr, 5.51k op/s 2024-03-22T01:39:28.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:39:27 smithi072 ceph-mon[32414]: pgmap v87: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 38 MiB/s wr, 5.51k op/s 2024-03-22T01:39:28.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:39:27 smithi098 ceph-mon[36468]: pgmap v87: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 38 MiB/s wr, 5.51k op/s 2024-03-22T01:39:30.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:39:29 smithi089 ceph-mon[35770]: pgmap v88: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 31 MiB/s wr, 4.35k op/s 2024-03-22T01:39:30.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:39:29 smithi072 ceph-mon[32414]: pgmap v88: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 31 MiB/s wr, 4.35k op/s 2024-03-22T01:39:30.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:39:29 smithi098 ceph-mon[36468]: pgmap v88: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 31 MiB/s wr, 4.35k op/s 2024-03-22T01:39:32.357 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:39:31 smithi089 ceph-mon[35770]: pgmap v89: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 34 MiB/s wr, 4.93k op/s 2024-03-22T01:39:32.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:39:31 smithi072 ceph-mon[32414]: pgmap v89: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 34 MiB/s wr, 4.93k op/s 2024-03-22T01:39:32.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:39:31 smithi098 ceph-mon[36468]: pgmap v89: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 34 MiB/s wr, 4.93k op/s 2024-03-22T01:39:34.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:39:33 smithi089 ceph-mon[35770]: pgmap v90: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s wr, 3.52k op/s 2024-03-22T01:39:34.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:39:33 smithi072 ceph-mon[32414]: pgmap v90: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s wr, 3.52k op/s 2024-03-22T01:39:34.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:39:33 smithi098 ceph-mon[36468]: pgmap v90: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s wr, 3.52k op/s 2024-03-22T01:39:36.357 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:39:35 smithi089 ceph-mon[35770]: pgmap v91: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s wr, 3.98k op/s 2024-03-22T01:39:36.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:39:35 smithi072 ceph-mon[32414]: pgmap v91: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s wr, 3.98k op/s 2024-03-22T01:39:36.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:39:35 smithi098 ceph-mon[36468]: pgmap v91: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s wr, 3.98k op/s 2024-03-22T01:39:38.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:39:37 smithi089 ceph-mon[35770]: pgmap v92: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 34 MiB/s wr, 4.90k op/s 2024-03-22T01:39:38.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:39:37 smithi072 ceph-mon[32414]: pgmap v92: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 34 MiB/s wr, 4.90k op/s 2024-03-22T01:39:38.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:39:37 smithi098 ceph-mon[36468]: pgmap v92: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 34 MiB/s wr, 4.90k op/s 2024-03-22T01:39:40.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:39:40 smithi089 ceph-mon[35770]: pgmap v93: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s wr, 3.81k op/s 2024-03-22T01:39:40.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:39:40 smithi072 ceph-mon[32414]: pgmap v93: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s wr, 3.81k op/s 2024-03-22T01:39:40.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:39:40 smithi098 ceph-mon[36468]: pgmap v93: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s wr, 3.81k op/s 2024-03-22T01:39:42.357 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:39:42 smithi089 ceph-mon[35770]: pgmap v94: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 32 MiB/s wr, 4.75k op/s 2024-03-22T01:39:42.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:39:42 smithi072 ceph-mon[32414]: pgmap v94: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 32 MiB/s wr, 4.75k op/s 2024-03-22T01:39:42.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:39:42 smithi098 ceph-mon[36468]: pgmap v94: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 32 MiB/s wr, 4.75k op/s 2024-03-22T01:39:44.357 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:39:44 smithi089 ceph-mon[35770]: pgmap v95: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s wr, 3.71k op/s 2024-03-22T01:39:44.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:39:44 smithi072 ceph-mon[32414]: pgmap v95: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s wr, 3.71k op/s 2024-03-22T01:39:44.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:39:44 smithi098 ceph-mon[36468]: pgmap v95: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s wr, 3.71k op/s 2024-03-22T01:39:46.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:39:46 smithi089 ceph-mon[35770]: pgmap v96: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 29 MiB/s wr, 4.13k op/s 2024-03-22T01:39:46.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:39:46 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:39:46.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:39:46 smithi072 ceph-mon[32414]: pgmap v96: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 29 MiB/s wr, 4.13k op/s 2024-03-22T01:39:46.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:39:46 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:39:46.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:39:46 smithi098 ceph-mon[36468]: pgmap v96: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 29 MiB/s wr, 4.13k op/s 2024-03-22T01:39:46.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:39:46 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:39:47.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:39:47 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:39:47.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:39:47 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:39:47.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:39:47 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:39:47.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:39:47 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:39:47.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:39:47 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:39:47.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:39:47 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:39:47.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:39:47 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:39:47.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:39:47 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:39:47.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:39:47 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:39:48.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:39:48 smithi089 ceph-mon[35770]: pgmap v97: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 29 MiB/s wr, 4.31k op/s 2024-03-22T01:39:48.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:39:48 smithi072 ceph-mon[32414]: pgmap v97: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 29 MiB/s wr, 4.31k op/s 2024-03-22T01:39:48.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:39:48 smithi098 ceph-mon[36468]: pgmap v97: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 29 MiB/s wr, 4.31k op/s 2024-03-22T01:39:50.357 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:39:50 smithi089 ceph-mon[35770]: pgmap v98: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s wr, 3.27k op/s 2024-03-22T01:39:50.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:39:50 smithi072 ceph-mon[32414]: pgmap v98: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s wr, 3.27k op/s 2024-03-22T01:39:50.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:39:50 smithi098 ceph-mon[36468]: pgmap v98: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s wr, 3.27k op/s 2024-03-22T01:39:52.357 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:39:52 smithi089 ceph-mon[35770]: pgmap v99: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 MiB/s wr, 4.58k op/s 2024-03-22T01:39:52.371 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:39:52 smithi072 ceph-mon[32414]: pgmap v99: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 MiB/s wr, 4.58k op/s 2024-03-22T01:39:52.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:39:52 smithi098 ceph-mon[36468]: pgmap v99: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 MiB/s wr, 4.58k op/s 2024-03-22T01:39:54.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:39:54 smithi089 ceph-mon[35770]: pgmap v100: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s wr, 3.17k op/s 2024-03-22T01:39:54.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:39:54 smithi072 ceph-mon[32414]: pgmap v100: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s wr, 3.17k op/s 2024-03-22T01:39:54.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:39:54 smithi098 ceph-mon[36468]: pgmap v100: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s wr, 3.17k op/s 2024-03-22T01:39:56.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:39:56 smithi089 ceph-mon[35770]: pgmap v101: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s wr, 3.66k op/s 2024-03-22T01:39:56.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:39:56 smithi072 ceph-mon[32414]: pgmap v101: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s wr, 3.66k op/s 2024-03-22T01:39:56.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:39:56 smithi098 ceph-mon[36468]: pgmap v101: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s wr, 3.66k op/s 2024-03-22T01:39:58.357 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:39:58 smithi089 ceph-mon[35770]: pgmap v102: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s wr, 4.02k op/s 2024-03-22T01:39:58.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:39:58 smithi072 ceph-mon[32414]: pgmap v102: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s wr, 4.02k op/s 2024-03-22T01:39:58.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:39:58 smithi098 ceph-mon[36468]: pgmap v102: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s wr, 4.02k op/s 2024-03-22T01:40:00.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:40:00 smithi089 ceph-mon[35770]: pgmap v103: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s wr, 3.19k op/s 2024-03-22T01:40:00.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:40:00 smithi089 ceph-mon[35770]: overall HEALTH_OK 2024-03-22T01:40:00.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:40:00 smithi072 ceph-mon[32414]: pgmap v103: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s wr, 3.19k op/s 2024-03-22T01:40:00.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:40:00 smithi072 ceph-mon[32414]: overall HEALTH_OK 2024-03-22T01:40:00.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:40:00 smithi098 ceph-mon[36468]: pgmap v103: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s wr, 3.19k op/s 2024-03-22T01:40:00.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:40:00 smithi098 ceph-mon[36468]: overall HEALTH_OK 2024-03-22T01:40:02.357 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:40:02 smithi089 ceph-mon[35770]: pgmap v104: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s wr, 4.33k op/s 2024-03-22T01:40:02.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:40:02 smithi072 ceph-mon[32414]: pgmap v104: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s wr, 4.33k op/s 2024-03-22T01:40:02.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:40:02 smithi098 ceph-mon[36468]: pgmap v104: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s wr, 4.33k op/s 2024-03-22T01:40:04.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:40:04 smithi089 ceph-mon[35770]: pgmap v105: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s wr, 2.63k op/s 2024-03-22T01:40:04.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:40:04 smithi072 ceph-mon[32414]: pgmap v105: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s wr, 2.63k op/s 2024-03-22T01:40:04.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:40:04 smithi098 ceph-mon[36468]: pgmap v105: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s wr, 2.63k op/s 2024-03-22T01:40:06.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:40:06 smithi089 ceph-mon[35770]: pgmap v106: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s wr, 2.63k op/s 2024-03-22T01:40:06.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:40:06 smithi072 ceph-mon[32414]: pgmap v106: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s wr, 2.63k op/s 2024-03-22T01:40:06.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:40:06 smithi098 ceph-mon[36468]: pgmap v106: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s wr, 2.63k op/s 2024-03-22T01:40:08.357 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:40:08 smithi089 ceph-mon[35770]: pgmap v107: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s wr, 3.59k op/s 2024-03-22T01:40:08.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:40:08 smithi072 ceph-mon[32414]: pgmap v107: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s wr, 3.59k op/s 2024-03-22T01:40:08.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:40:08 smithi098 ceph-mon[36468]: pgmap v107: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s wr, 3.59k op/s 2024-03-22T01:40:10.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:40:10 smithi089 ceph-mon[35770]: pgmap v108: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s wr, 2.81k op/s 2024-03-22T01:40:10.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:40:10 smithi072 ceph-mon[32414]: pgmap v108: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s wr, 2.81k op/s 2024-03-22T01:40:10.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:40:10 smithi098 ceph-mon[36468]: pgmap v108: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s wr, 2.81k op/s 2024-03-22T01:40:12.357 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:40:12 smithi089 ceph-mon[35770]: pgmap v109: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s wr, 3.83k op/s 2024-03-22T01:40:12.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:40:12 smithi072 ceph-mon[32414]: pgmap v109: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s wr, 3.83k op/s 2024-03-22T01:40:12.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:40:12 smithi098 ceph-mon[36468]: pgmap v109: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s wr, 3.83k op/s 2024-03-22T01:40:14.357 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:40:14 smithi089 ceph-mon[35770]: pgmap v110: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s wr, 2.59k op/s 2024-03-22T01:40:14.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:40:14 smithi072 ceph-mon[32414]: pgmap v110: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s wr, 2.59k op/s 2024-03-22T01:40:14.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:40:14 smithi098 ceph-mon[36468]: pgmap v110: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s wr, 2.59k op/s 2024-03-22T01:40:16.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:40:16 smithi089 ceph-mon[35770]: pgmap v111: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s wr, 2.59k op/s 2024-03-22T01:40:16.371 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:40:16 smithi072 ceph-mon[32414]: pgmap v111: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s wr, 2.59k op/s 2024-03-22T01:40:16.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:40:16 smithi098 ceph-mon[36468]: pgmap v111: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s wr, 2.59k op/s 2024-03-22T01:40:18.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:40:18 smithi089 ceph-mon[35770]: pgmap v112: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s wr, 3.36k op/s 2024-03-22T01:40:18.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:40:18 smithi072 ceph-mon[32414]: pgmap v112: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s wr, 3.36k op/s 2024-03-22T01:40:18.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:40:18 smithi098 ceph-mon[36468]: pgmap v112: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s wr, 3.36k op/s 2024-03-22T01:40:20.357 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:40:20 smithi089 ceph-mon[35770]: pgmap v113: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s wr, 2.14k op/s 2024-03-22T01:40:20.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:40:20 smithi072 ceph-mon[32414]: pgmap v113: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s wr, 2.14k op/s 2024-03-22T01:40:20.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:40:20 smithi098 ceph-mon[36468]: pgmap v113: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s wr, 2.14k op/s 2024-03-22T01:40:22.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:40:22 smithi089 ceph-mon[35770]: pgmap v114: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s wr, 3.63k op/s 2024-03-22T01:40:22.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:40:22 smithi072 ceph-mon[32414]: pgmap v114: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s wr, 3.63k op/s 2024-03-22T01:40:22.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:40:22 smithi098 ceph-mon[36468]: pgmap v114: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s wr, 3.63k op/s 2024-03-22T01:40:24.357 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:40:24 smithi089 ceph-mon[35770]: pgmap v115: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 2.60k op/s 2024-03-22T01:40:24.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:40:24 smithi072 ceph-mon[32414]: pgmap v115: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 2.60k op/s 2024-03-22T01:40:24.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:40:24 smithi098 ceph-mon[36468]: pgmap v115: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 2.60k op/s 2024-03-22T01:40:26.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:40:26 smithi089 ceph-mon[35770]: pgmap v116: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s wr, 2.61k op/s 2024-03-22T01:40:26.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:40:26 smithi072 ceph-mon[32414]: pgmap v116: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s wr, 2.61k op/s 2024-03-22T01:40:26.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:40:26 smithi098 ceph-mon[36468]: pgmap v116: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s wr, 2.61k op/s 2024-03-22T01:40:28.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:40:28 smithi089 ceph-mon[35770]: pgmap v117: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s wr, 3.38k op/s 2024-03-22T01:40:28.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:40:28 smithi072 ceph-mon[32414]: pgmap v117: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s wr, 3.38k op/s 2024-03-22T01:40:28.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:40:28 smithi098 ceph-mon[36468]: pgmap v117: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s wr, 3.38k op/s 2024-03-22T01:40:30.357 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:40:30 smithi089 ceph-mon[35770]: pgmap v118: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 2.69k op/s 2024-03-22T01:40:30.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:40:30 smithi072 ceph-mon[32414]: pgmap v118: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 2.69k op/s 2024-03-22T01:40:30.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:40:30 smithi098 ceph-mon[36468]: pgmap v118: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 2.69k op/s 2024-03-22T01:40:32.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:40:32 smithi089 ceph-mon[35770]: pgmap v119: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s wr, 3.42k op/s 2024-03-22T01:40:32.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:40:32 smithi072 ceph-mon[32414]: pgmap v119: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s wr, 3.42k op/s 2024-03-22T01:40:32.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:40:32 smithi098 ceph-mon[36468]: pgmap v119: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s wr, 3.42k op/s 2024-03-22T01:40:34.357 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:40:34 smithi089 ceph-mon[35770]: pgmap v120: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s wr, 2.20k op/s 2024-03-22T01:40:34.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:40:34 smithi072 ceph-mon[32414]: pgmap v120: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s wr, 2.20k op/s 2024-03-22T01:40:34.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:40:34 smithi098 ceph-mon[36468]: pgmap v120: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s wr, 2.20k op/s 2024-03-22T01:40:36.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:40:36 smithi089 ceph-mon[35770]: pgmap v121: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s wr, 2.42k op/s 2024-03-22T01:40:36.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:40:36 smithi072 ceph-mon[32414]: pgmap v121: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s wr, 2.42k op/s 2024-03-22T01:40:36.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:40:36 smithi098 ceph-mon[36468]: pgmap v121: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s wr, 2.42k op/s 2024-03-22T01:40:38.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:40:38 smithi089 ceph-mon[35770]: pgmap v122: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s wr, 3.48k op/s 2024-03-22T01:40:38.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:40:38 smithi072 ceph-mon[32414]: pgmap v122: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s wr, 3.48k op/s 2024-03-22T01:40:38.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:40:38 smithi098 ceph-mon[36468]: pgmap v122: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s wr, 3.48k op/s 2024-03-22T01:40:40.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:40:40 smithi089 ceph-mon[35770]: pgmap v123: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s wr, 2.89k op/s 2024-03-22T01:40:40.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:40:40 smithi072 ceph-mon[32414]: pgmap v123: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s wr, 2.89k op/s 2024-03-22T01:40:40.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:40:40 smithi098 ceph-mon[36468]: pgmap v123: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s wr, 2.89k op/s 2024-03-22T01:40:42.357 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:40:42 smithi089 ceph-mon[35770]: pgmap v124: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 MiB/s wr, 3.74k op/s 2024-03-22T01:40:42.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:40:42 smithi072 ceph-mon[32414]: pgmap v124: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 MiB/s wr, 3.74k op/s 2024-03-22T01:40:42.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:40:42 smithi098 ceph-mon[36468]: pgmap v124: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 MiB/s wr, 3.74k op/s 2024-03-22T01:40:44.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:40:44 smithi089 ceph-mon[35770]: pgmap v125: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s wr, 2.82k op/s 2024-03-22T01:40:44.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:40:44 smithi072 ceph-mon[32414]: pgmap v125: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s wr, 2.82k op/s 2024-03-22T01:40:44.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:40:44 smithi098 ceph-mon[36468]: pgmap v125: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s wr, 2.82k op/s 2024-03-22T01:40:46.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:40:46 smithi089 ceph-mon[35770]: pgmap v126: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s wr, 2.78k op/s 2024-03-22T01:40:46.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:40:46 smithi072 ceph-mon[32414]: pgmap v126: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s wr, 2.78k op/s 2024-03-22T01:40:46.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:40:46 smithi098 ceph-mon[36468]: pgmap v126: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s wr, 2.78k op/s 2024-03-22T01:40:47.357 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:40:47 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:40:47.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:40:47 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:40:47.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:40:47 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:40:48.357 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:40:48 smithi089 ceph-mon[35770]: pgmap v127: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s wr, 3.79k op/s 2024-03-22T01:40:48.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:40:48 smithi072 ceph-mon[32414]: pgmap v127: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s wr, 3.79k op/s 2024-03-22T01:40:48.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:40:48 smithi098 ceph-mon[36468]: pgmap v127: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s wr, 3.79k op/s 2024-03-22T01:40:50.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:40:50 smithi089 ceph-mon[35770]: pgmap v128: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s wr, 2.77k op/s 2024-03-22T01:40:50.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:40:50 smithi072 ceph-mon[32414]: pgmap v128: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s wr, 2.77k op/s 2024-03-22T01:40:50.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:40:50 smithi098 ceph-mon[36468]: pgmap v128: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s wr, 2.77k op/s 2024-03-22T01:40:52.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:40:52 smithi089 ceph-mon[35770]: pgmap v129: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s wr, 3.44k op/s 2024-03-22T01:40:52.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:40:52 smithi072 ceph-mon[32414]: pgmap v129: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s wr, 3.44k op/s 2024-03-22T01:40:52.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:40:52 smithi098 ceph-mon[36468]: pgmap v129: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s wr, 3.44k op/s 2024-03-22T01:40:54.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:40:54 smithi089 ceph-mon[35770]: pgmap v130: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s wr, 2.60k op/s 2024-03-22T01:40:54.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:40:54 smithi072 ceph-mon[32414]: pgmap v130: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s wr, 2.60k op/s 2024-03-22T01:40:54.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:40:54 smithi098 ceph-mon[36468]: pgmap v130: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s wr, 2.60k op/s 2024-03-22T01:40:56.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:40:56 smithi089 ceph-mon[35770]: pgmap v131: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s wr, 2.76k op/s 2024-03-22T01:40:56.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:40:56 smithi072 ceph-mon[32414]: pgmap v131: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s wr, 2.76k op/s 2024-03-22T01:40:56.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:40:56 smithi098 ceph-mon[36468]: pgmap v131: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s wr, 2.76k op/s 2024-03-22T01:40:58.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:40:58 smithi089 ceph-mon[35770]: pgmap v132: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s wr, 3.56k op/s 2024-03-22T01:40:58.371 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:40:58 smithi072 ceph-mon[32414]: pgmap v132: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s wr, 3.56k op/s 2024-03-22T01:40:58.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:40:58 smithi098 ceph-mon[36468]: pgmap v132: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s wr, 3.56k op/s 2024-03-22T01:41:00.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:41:00 smithi072 ceph-mon[32414]: pgmap v133: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s wr, 2.57k op/s 2024-03-22T01:41:00.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:41:00 smithi098 ceph-mon[36468]: pgmap v133: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s wr, 2.57k op/s 2024-03-22T01:41:00.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:41:00 smithi089 ceph-mon[35770]: pgmap v133: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s wr, 2.57k op/s 2024-03-22T01:41:02.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:41:02 smithi072 ceph-mon[32414]: pgmap v134: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s wr, 3.15k op/s 2024-03-22T01:41:02.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:41:02 smithi098 ceph-mon[36468]: pgmap v134: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s wr, 3.15k op/s 2024-03-22T01:41:02.607 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:41:02 smithi089 ceph-mon[35770]: pgmap v134: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s wr, 3.15k op/s 2024-03-22T01:41:04.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:41:04 smithi072 ceph-mon[32414]: pgmap v135: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s wr, 2.36k op/s 2024-03-22T01:41:04.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:41:04 smithi098 ceph-mon[36468]: pgmap v135: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s wr, 2.36k op/s 2024-03-22T01:41:04.607 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:41:04 smithi089 ceph-mon[35770]: pgmap v135: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s wr, 2.36k op/s 2024-03-22T01:41:06.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:41:06 smithi072 ceph-mon[32414]: pgmap v136: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s wr, 2.47k op/s 2024-03-22T01:41:06.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:41:06 smithi098 ceph-mon[36468]: pgmap v136: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s wr, 2.47k op/s 2024-03-22T01:41:06.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:41:06 smithi089 ceph-mon[35770]: pgmap v136: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s wr, 2.47k op/s 2024-03-22T01:41:08.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:41:08 smithi072 ceph-mon[32414]: pgmap v137: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s wr, 3.19k op/s 2024-03-22T01:41:08.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:41:08 smithi098 ceph-mon[36468]: pgmap v137: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s wr, 3.19k op/s 2024-03-22T01:41:08.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:41:08 smithi089 ceph-mon[35770]: pgmap v137: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s wr, 3.19k op/s 2024-03-22T01:41:10.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:41:10 smithi072 ceph-mon[32414]: pgmap v138: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s wr, 2.17k op/s 2024-03-22T01:41:10.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:41:10 smithi098 ceph-mon[36468]: pgmap v138: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s wr, 2.17k op/s 2024-03-22T01:41:10.607 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:41:10 smithi089 ceph-mon[35770]: pgmap v138: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s wr, 2.17k op/s 2024-03-22T01:41:12.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:41:12 smithi098 ceph-mon[36468]: pgmap v139: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s wr, 3.05k op/s 2024-03-22T01:41:12.607 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:41:12 smithi089 ceph-mon[35770]: pgmap v139: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s wr, 3.05k op/s 2024-03-22T01:41:12.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:41:12 smithi072 ceph-mon[32414]: pgmap v139: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s wr, 3.05k op/s 2024-03-22T01:41:14.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:41:14 smithi098 ceph-mon[36468]: pgmap v140: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s wr, 2.50k op/s 2024-03-22T01:41:14.607 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:41:14 smithi089 ceph-mon[35770]: pgmap v140: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s wr, 2.50k op/s 2024-03-22T01:41:14.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:41:14 smithi072 ceph-mon[32414]: pgmap v140: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s wr, 2.50k op/s 2024-03-22T01:41:16.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:41:16 smithi098 ceph-mon[36468]: pgmap v141: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s wr, 2.52k op/s 2024-03-22T01:41:16.423 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:41:16 smithi072 ceph-mon[32414]: pgmap v141: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s wr, 2.52k op/s 2024-03-22T01:41:16.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:41:16 smithi089 ceph-mon[35770]: pgmap v141: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s wr, 2.52k op/s 2024-03-22T01:41:18.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:41:18 smithi098 ceph-mon[36468]: pgmap v142: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s wr, 2.99k op/s 2024-03-22T01:41:18.607 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:41:18 smithi089 ceph-mon[35770]: pgmap v142: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s wr, 2.99k op/s 2024-03-22T01:41:18.621 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:41:18 smithi072 ceph-mon[32414]: pgmap v142: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s wr, 2.99k op/s 2024-03-22T01:41:20.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:41:20 smithi098 ceph-mon[36468]: pgmap v143: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s wr, 2.08k op/s 2024-03-22T01:41:20.607 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:41:20 smithi089 ceph-mon[35770]: pgmap v143: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s wr, 2.08k op/s 2024-03-22T01:41:20.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:41:20 smithi072 ceph-mon[32414]: pgmap v143: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s wr, 2.08k op/s 2024-03-22T01:41:22.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:41:22 smithi098 ceph-mon[36468]: pgmap v144: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s wr, 3.03k op/s 2024-03-22T01:41:22.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:41:22 smithi089 ceph-mon[35770]: pgmap v144: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s wr, 3.03k op/s 2024-03-22T01:41:22.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:41:22 smithi072 ceph-mon[32414]: pgmap v144: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s wr, 3.03k op/s 2024-03-22T01:41:24.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:41:24 smithi098 ceph-mon[36468]: pgmap v145: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s wr, 2.02k op/s 2024-03-22T01:41:24.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:41:24 smithi089 ceph-mon[35770]: pgmap v145: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s wr, 2.02k op/s 2024-03-22T01:41:24.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:41:24 smithi072 ceph-mon[32414]: pgmap v145: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s wr, 2.02k op/s 2024-03-22T01:41:26.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:41:26 smithi098 ceph-mon[36468]: pgmap v146: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s wr, 1.96k op/s 2024-03-22T01:41:26.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:41:26 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:41:26.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:41:26 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:41:26.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:41:26 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:41:26.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:41:26 smithi089 ceph-mon[35770]: pgmap v146: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s wr, 1.96k op/s 2024-03-22T01:41:26.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:41:26 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:41:26.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:41:26 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:41:26.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:41:26 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:41:26.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:41:26 smithi072 ceph-mon[32414]: pgmap v146: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s wr, 1.96k op/s 2024-03-22T01:41:26.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:41:26 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:41:26.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:41:26 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:41:26.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:41:26 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:41:28.410 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:41:28 smithi089 ceph-mon[35770]: pgmap v147: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s wr, 2.98k op/s 2024-03-22T01:41:28.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:41:28 smithi098 ceph-mon[36468]: pgmap v147: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s wr, 2.98k op/s 2024-03-22T01:41:28.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:41:28 smithi072 ceph-mon[32414]: pgmap v147: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s wr, 2.98k op/s 2024-03-22T01:41:30.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:41:30 smithi098 ceph-mon[36468]: pgmap v148: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 2.41k op/s 2024-03-22T01:41:30.607 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:41:30 smithi089 ceph-mon[35770]: pgmap v148: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 2.41k op/s 2024-03-22T01:41:30.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:41:30 smithi072 ceph-mon[32414]: pgmap v148: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 2.41k op/s 2024-03-22T01:41:32.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:41:32 smithi098 ceph-mon[36468]: pgmap v149: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s wr, 3.46k op/s 2024-03-22T01:41:32.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:41:32 smithi089 ceph-mon[35770]: pgmap v149: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s wr, 3.46k op/s 2024-03-22T01:41:32.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:41:32 smithi072 ceph-mon[32414]: pgmap v149: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s wr, 3.46k op/s 2024-03-22T01:41:34.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:41:34 smithi098 ceph-mon[36468]: pgmap v150: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s wr, 2.66k op/s 2024-03-22T01:41:34.607 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:41:34 smithi089 ceph-mon[35770]: pgmap v150: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s wr, 2.66k op/s 2024-03-22T01:41:34.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:41:34 smithi072 ceph-mon[32414]: pgmap v150: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s wr, 2.66k op/s 2024-03-22T01:41:36.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:41:36 smithi098 ceph-mon[36468]: pgmap v151: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s wr, 2.56k op/s 2024-03-22T01:41:36.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:41:36 smithi089 ceph-mon[35770]: pgmap v151: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s wr, 2.56k op/s 2024-03-22T01:41:36.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:41:36 smithi072 ceph-mon[32414]: pgmap v151: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s wr, 2.56k op/s 2024-03-22T01:41:37.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:41:37 smithi089 ceph-mon[35770]: pgmap v152: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s wr, 3.35k op/s 2024-03-22T01:41:37.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:41:37 smithi089 ceph-mon[35770]: Health check failed: 1 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2024-03-22T01:41:37.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:41:37 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:41:37.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:41:37 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:41:37.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:41:37 smithi072 ceph-mon[32414]: pgmap v152: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s wr, 3.35k op/s 2024-03-22T01:41:37.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:41:37 smithi072 ceph-mon[32414]: Health check failed: 1 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2024-03-22T01:41:37.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:41:37 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:41:37.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:41:37 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:41:37.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:41:37 smithi098 ceph-mon[36468]: pgmap v152: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s wr, 3.35k op/s 2024-03-22T01:41:37.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:41:37 smithi098 ceph-mon[36468]: Health check failed: 1 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2024-03-22T01:41:37.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:41:37 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:41:37.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:41:37 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:41:39.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:41:39 smithi089 ceph-mon[35770]: pgmap v153: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s wr, 2.38k op/s 2024-03-22T01:41:39.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:41:39 smithi072 ceph-mon[32414]: pgmap v153: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s wr, 2.38k op/s 2024-03-22T01:41:39.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:41:39 smithi098 ceph-mon[36468]: pgmap v153: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s wr, 2.38k op/s 2024-03-22T01:41:41.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:41:41 smithi089 ceph-mon[35770]: pgmap v154: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s wr, 3.30k op/s 2024-03-22T01:41:41.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:41:41 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:41:41.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:41:41 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:41:41.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:41:41 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:41:41.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:41:41 smithi072 ceph-mon[32414]: pgmap v154: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s wr, 3.30k op/s 2024-03-22T01:41:41.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:41:41 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:41:41.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:41:41 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:41:41.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:41:41 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:41:41.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:41:41 smithi098 ceph-mon[36468]: pgmap v154: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s wr, 3.30k op/s 2024-03-22T01:41:41.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:41:41 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:41:41.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:41:41 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:41:41.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:41:41 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:41:42.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:41:42 smithi098 ceph-mon[36468]: Health check failed: 0 slow ops, oldest one blocked for 184 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:41:42.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:41:42 smithi089 ceph-mon[35770]: Health check failed: 0 slow ops, oldest one blocked for 184 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:41:42.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:41:42 smithi072 ceph-mon[32414]: Health check failed: 0 slow ops, oldest one blocked for 184 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:41:43.550 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:41:43.550+0000 7f3e297fa640 1 -- 172.21.15.72:0/1386165290 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f3e0c0034f0 con 0x7f3e34069e10 2024-03-22T01:41:43.550 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:41:43.551+0000 7f3e2a7fc640 1 -- 172.21.15.72:0/1386165290 <== mon.0 v2:172.21.15.72:3300/0 6 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f3e0c0034f0 con 0x7f3e34069e10 2024-03-22T01:41:43.593 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:41:43.595+0000 7f3e137fe640 1 -- 172.21.15.72:0/2014627542 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f3e040034c0 con 0x55af9cd592d0 2024-03-22T01:41:43.594 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:41:43.595+0000 7f3e1cff9640 1 -- 172.21.15.72:0/2014627542 <== mon.0 v2:172.21.15.72:3300/0 6 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f3e040034c0 con 0x55af9cd592d0 2024-03-22T01:41:43.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:41:43 smithi098 ceph-mon[36468]: pgmap v155: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s wr, 2.67k op/s 2024-03-22T01:41:43.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:41:43 smithi089 ceph-mon[35770]: pgmap v155: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s wr, 2.67k op/s 2024-03-22T01:41:43.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:41:43 smithi072 ceph-mon[32414]: pgmap v155: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s wr, 2.67k op/s 2024-03-22T01:41:44.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:41:44 smithi098 ceph-mon[36468]: Health check update: 2 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2024-03-22T01:41:44.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:41:44 smithi089 ceph-mon[35770]: Health check update: 2 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2024-03-22T01:41:44.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:41:44 smithi072 ceph-mon[32414]: Health check update: 2 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2024-03-22T01:41:45.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:41:45 smithi098 ceph-mon[36468]: pgmap v156: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s wr, 2.51k op/s 2024-03-22T01:41:45.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:41:45 smithi089 ceph-mon[35770]: pgmap v156: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s wr, 2.51k op/s 2024-03-22T01:41:45.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:41:45 smithi072 ceph-mon[32414]: pgmap v156: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s wr, 2.51k op/s 2024-03-22T01:41:47.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:41:47 smithi098 ceph-mon[36468]: pgmap v157: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 31 MiB/s wr, 4.21k op/s 2024-03-22T01:41:47.674 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:41:47 smithi072 ceph-mon[32414]: pgmap v157: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 31 MiB/s wr, 4.21k op/s 2024-03-22T01:41:47.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:41:47 smithi089 ceph-mon[35770]: pgmap v157: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 31 MiB/s wr, 4.21k op/s 2024-03-22T01:41:49.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:41:49 smithi089 ceph-mon[35770]: pgmap v158: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s wr, 3.53k op/s 2024-03-22T01:41:49.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:41:49 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:41:49.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:41:49 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:41:49.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:41:49 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:41:49.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:41:49 smithi089 ceph-mon[35770]: Health check update: 0 slow ops, oldest one blocked for 189 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:41:49.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:41:49 smithi072 ceph-mon[32414]: pgmap v158: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s wr, 3.53k op/s 2024-03-22T01:41:49.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:41:49 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:41:49.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:41:49 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:41:49.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:41:49 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:41:49.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:41:49 smithi072 ceph-mon[32414]: Health check update: 0 slow ops, oldest one blocked for 189 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:41:49.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:41:49 smithi098 ceph-mon[36468]: pgmap v158: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s wr, 3.53k op/s 2024-03-22T01:41:49.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:41:49 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:41:49.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:41:49 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:41:49.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:41:49 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:41:49.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:41:49 smithi098 ceph-mon[36468]: Health check update: 0 slow ops, oldest one blocked for 189 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:41:51.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:41:51.130+0000 7fd986ffd640 1 -- 172.21.15.72:0/1855106146 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fd9640034c0 con 0x7fd9901013f0 2024-03-22T01:41:51.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:41:51.130+0000 7fd98cd99640 1 -- 172.21.15.72:0/1855106146 <== mon.0 v2:172.21.15.72:3300/0 6 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fd97807e070 con 0x7fd9901013f0 2024-03-22T01:41:51.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:41:51.135+0000 7fd9757fa640 1 -- 172.21.15.72:0/1183142685 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fd9680034c0 con 0x55682e211d30 2024-03-22T01:41:51.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:41:51.135+0000 7fd9767fc640 1 -- 172.21.15.72:0/1183142685 <== mon.0 v2:172.21.15.72:3300/0 6 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fd9680034c0 con 0x55682e211d30 2024-03-22T01:41:51.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:41:51 smithi089 ceph-mon[35770]: pgmap v159: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 35 MiB/s wr, 5.07k op/s 2024-03-22T01:41:51.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:41:51 smithi072 ceph-mon[32414]: pgmap v159: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 35 MiB/s wr, 5.07k op/s 2024-03-22T01:41:51.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:41:51 smithi098 ceph-mon[36468]: pgmap v159: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 35 MiB/s wr, 5.07k op/s 2024-03-22T01:41:53.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:41:53 smithi089 ceph-mon[35770]: pgmap v160: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 29 MiB/s wr, 4.34k op/s 2024-03-22T01:41:53.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:41:53 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:41:53.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:41:53 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:41:53.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:41:53 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:41:53.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:41:53 smithi072 ceph-mon[32414]: pgmap v160: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 29 MiB/s wr, 4.34k op/s 2024-03-22T01:41:53.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:41:53 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:41:53.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:41:53 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:41:53.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:41:53 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:41:53.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:41:53 smithi098 ceph-mon[36468]: pgmap v160: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 29 MiB/s wr, 4.34k op/s 2024-03-22T01:41:53.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:41:53 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:41:53.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:41:53 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:41:53.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:41:53 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:41:54.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:41:54 smithi089 ceph-mon[35770]: Health check update: 0 slow ops, oldest one blocked for 195 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:41:54.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:41:54 smithi072 ceph-mon[32414]: Health check update: 0 slow ops, oldest one blocked for 195 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:41:54.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:41:54 smithi098 ceph-mon[36468]: Health check update: 0 slow ops, oldest one blocked for 195 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:41:55.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:41:55 smithi089 ceph-mon[35770]: pgmap v161: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s wr, 3.91k op/s 2024-03-22T01:41:55.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:41:55 smithi072 ceph-mon[32414]: pgmap v161: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s wr, 3.91k op/s 2024-03-22T01:41:55.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:41:55 smithi098 ceph-mon[36468]: pgmap v161: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s wr, 3.91k op/s 2024-03-22T01:41:57.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:41:57 smithi089 ceph-mon[35770]: pgmap v162: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 32 MiB/s wr, 5.06k op/s 2024-03-22T01:41:57.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:41:57 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:41:57.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:41:57 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:41:57.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:41:57 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:41:57.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:41:57 smithi072 ceph-mon[32414]: pgmap v162: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 32 MiB/s wr, 5.06k op/s 2024-03-22T01:41:57.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:41:57 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:41:57.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:41:57 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:41:57.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:41:57 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:41:57.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:41:57 smithi098 ceph-mon[36468]: pgmap v162: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 32 MiB/s wr, 5.06k op/s 2024-03-22T01:41:57.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:41:57 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:41:57.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:41:57 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:41:57.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:41:57 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:41:59.779 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:41:59 smithi072 ceph-mon[32414]: pgmap v163: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s wr, 3.76k op/s 2024-03-22T01:41:59.857 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:41:59 smithi089 ceph-mon[35770]: pgmap v163: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s wr, 3.76k op/s 2024-03-22T01:41:59.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:41:59 smithi089 ceph-mon[35770]: Health check update: 0 slow ops, oldest one blocked for 200 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:41:59.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:41:59 smithi098 ceph-mon[36468]: pgmap v163: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s wr, 3.76k op/s 2024-03-22T01:41:59.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:41:59 smithi098 ceph-mon[36468]: Health check update: 0 slow ops, oldest one blocked for 200 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:42:00.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:41:59 smithi072 ceph-mon[32414]: Health check update: 0 slow ops, oldest one blocked for 200 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:42:01.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:01 smithi089 ceph-mon[35770]: pgmap v164: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s wr, 4.10k op/s 2024-03-22T01:42:01.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:01 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:42:01.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:01 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:42:01.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:01 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:01.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:01 smithi072 ceph-mon[32414]: pgmap v164: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s wr, 4.10k op/s 2024-03-22T01:42:01.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:01 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:42:01.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:01 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:42:01.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:01 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:01.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:01 smithi098 ceph-mon[36468]: pgmap v164: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s wr, 4.10k op/s 2024-03-22T01:42:01.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:01 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:42:01.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:01 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:42:01.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:01 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:03.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:03 smithi089 ceph-mon[35770]: pgmap v165: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 2.71k op/s 2024-03-22T01:42:03.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:03 smithi072 ceph-mon[32414]: pgmap v165: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 2.71k op/s 2024-03-22T01:42:03.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:03 smithi098 ceph-mon[36468]: pgmap v165: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 2.71k op/s 2024-03-22T01:42:04.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:04 smithi089 ceph-mon[35770]: 1 slow requests, 1 included below; oldest blocked for > 183.079551 secs 2024-03-22T01:42:04.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:04 smithi089 ceph-mon[35770]: slow request 183.079550 seconds old, received at 2024-03-22T01:39:00.828862+0000: client_request(client.25502:16 mksnap #0x1//scheduled-2024-03-22-01_39_00_UTC 2024-03-22T01:39:00.828832+0000 caller_uid=0, caller_gid=0{}) currently acquired locks 2024-03-22T01:42:04.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:04 smithi072 ceph-mon[32414]: 1 slow requests, 1 included below; oldest blocked for > 183.079551 secs 2024-03-22T01:42:04.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:04 smithi072 ceph-mon[32414]: slow request 183.079550 seconds old, received at 2024-03-22T01:39:00.828862+0000: client_request(client.25502:16 mksnap #0x1//scheduled-2024-03-22-01_39_00_UTC 2024-03-22T01:39:00.828832+0000 caller_uid=0, caller_gid=0{}) currently acquired locks 2024-03-22T01:42:04.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:04 smithi098 ceph-mon[36468]: 1 slow requests, 1 included below; oldest blocked for > 183.079551 secs 2024-03-22T01:42:04.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:04 smithi098 ceph-mon[36468]: slow request 183.079550 seconds old, received at 2024-03-22T01:39:00.828862+0000: client_request(client.25502:16 mksnap #0x1//scheduled-2024-03-22-01_39_00_UTC 2024-03-22T01:39:00.828832+0000 caller_uid=0, caller_gid=0{}) currently acquired locks 2024-03-22T01:42:05.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:05 smithi089 ceph-mon[35770]: pgmap v166: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s wr, 2.53k op/s 2024-03-22T01:42:05.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:05 smithi089 ceph-mon[35770]: Health check update: 0 slow ops, oldest one blocked for 206 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:42:05.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:05 smithi072 ceph-mon[32414]: pgmap v166: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s wr, 2.53k op/s 2024-03-22T01:42:05.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:05 smithi072 ceph-mon[32414]: Health check update: 0 slow ops, oldest one blocked for 206 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:42:05.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:05 smithi098 ceph-mon[36468]: pgmap v166: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s wr, 2.53k op/s 2024-03-22T01:42:05.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:05 smithi098 ceph-mon[36468]: Health check update: 0 slow ops, oldest one blocked for 206 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:42:07.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:07 smithi089 ceph-mon[35770]: pgmap v167: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s wr, 3.46k op/s 2024-03-22T01:42:07.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:07 smithi072 ceph-mon[32414]: pgmap v167: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s wr, 3.46k op/s 2024-03-22T01:42:07.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:07 smithi098 ceph-mon[36468]: pgmap v167: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s wr, 3.46k op/s 2024-03-22T01:42:08.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:08 smithi089 ceph-mon[35770]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-03-22T01:42:08.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:08 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:42:08.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:08 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:42:08.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:08 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:08.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:08 smithi072 ceph-mon[32414]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-03-22T01:42:08.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:08 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:42:08.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:08 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:42:08.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:08 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:08.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:08 smithi098 ceph-mon[36468]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-03-22T01:42:08.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:08 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:42:08.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:08 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:42:08.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:08 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:09.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:09 smithi089 ceph-mon[35770]: pgmap v168: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s wr, 2.40k op/s 2024-03-22T01:42:09.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:09 smithi089 ceph-mon[35770]: 1 slow requests, 0 included below; oldest blocked for > 188.079637 secs 2024-03-22T01:42:09.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:09 smithi072 ceph-mon[32414]: pgmap v168: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s wr, 2.40k op/s 2024-03-22T01:42:09.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:09 smithi072 ceph-mon[32414]: 1 slow requests, 0 included below; oldest blocked for > 188.079637 secs 2024-03-22T01:42:09.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:09 smithi098 ceph-mon[36468]: pgmap v168: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s wr, 2.40k op/s 2024-03-22T01:42:09.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:09 smithi098 ceph-mon[36468]: 1 slow requests, 0 included below; oldest blocked for > 188.079637 secs 2024-03-22T01:42:10.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:10 smithi089 ceph-mon[35770]: Health check update: 0 slow ops, oldest one blocked for 212 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:42:10.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:10 smithi072 ceph-mon[32414]: Health check update: 0 slow ops, oldest one blocked for 212 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:42:10.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:10 smithi098 ceph-mon[36468]: Health check update: 0 slow ops, oldest one blocked for 212 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:42:11.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:11 smithi089 ceph-mon[35770]: pgmap v169: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s wr, 2.85k op/s 2024-03-22T01:42:11.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:11 smithi072 ceph-mon[32414]: pgmap v169: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s wr, 2.85k op/s 2024-03-22T01:42:11.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:11 smithi098 ceph-mon[36468]: pgmap v169: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s wr, 2.85k op/s 2024-03-22T01:42:12.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:12 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:42:12.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:12 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:12.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:12 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:42:12.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:12 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:13.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:12 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:42:13.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:12 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:13.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:13 smithi072 ceph-mon[32414]: pgmap v170: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 2.53k op/s 2024-03-22T01:42:13.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:13 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:42:13.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:13 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:13.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:13 smithi098 ceph-mon[36468]: pgmap v170: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 2.53k op/s 2024-03-22T01:42:13.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:13 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:42:13.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:13 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:14.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:13 smithi089 ceph-mon[35770]: pgmap v170: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 2.53k op/s 2024-03-22T01:42:14.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:13 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:42:14.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:13 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:14.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:14 smithi072 ceph-mon[32414]: 1 slow requests, 0 included below; oldest blocked for > 193.079727 secs 2024-03-22T01:42:14.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:14 smithi098 ceph-mon[36468]: 1 slow requests, 0 included below; oldest blocked for > 193.079727 secs 2024-03-22T01:42:15.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:14 smithi089 ceph-mon[35770]: 1 slow requests, 0 included below; oldest blocked for > 193.079727 secs 2024-03-22T01:42:15.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:15 smithi072 ceph-mon[32414]: pgmap v171: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s wr, 2.23k op/s 2024-03-22T01:42:15.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:15 smithi098 ceph-mon[36468]: pgmap v171: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s wr, 2.23k op/s 2024-03-22T01:42:16.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:15 smithi089 ceph-mon[35770]: pgmap v171: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s wr, 2.23k op/s 2024-03-22T01:42:16.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:16 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:42:16.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:16 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:17.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:16 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:42:17.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:16 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:17.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:16 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:42:17.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:16 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:17.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:17 smithi098 ceph-mon[36468]: pgmap v172: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s wr, 4.03k op/s 2024-03-22T01:42:17.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:17 smithi098 ceph-mon[36468]: Health check update: 0 slow ops, oldest one blocked for 219 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:42:17.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:17 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:42:17.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:17 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:18.023 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:17 smithi072 ceph-mon[32414]: pgmap v172: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s wr, 4.03k op/s 2024-03-22T01:42:18.023 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:17 smithi072 ceph-mon[32414]: Health check update: 0 slow ops, oldest one blocked for 219 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:42:18.023 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:17 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:42:18.023 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:17 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:18.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:17 smithi089 ceph-mon[35770]: pgmap v172: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s wr, 4.03k op/s 2024-03-22T01:42:18.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:17 smithi089 ceph-mon[35770]: Health check update: 0 slow ops, oldest one blocked for 219 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:42:18.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:17 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:42:18.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:17 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:19.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:19 smithi098 ceph-mon[36468]: pgmap v173: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s wr, 3.24k op/s 2024-03-22T01:42:19.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:19 smithi098 ceph-mon[36468]: 1 slow requests, 0 included below; oldest blocked for > 198.079792 secs 2024-03-22T01:42:20.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:19 smithi089 ceph-mon[35770]: pgmap v173: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s wr, 3.24k op/s 2024-03-22T01:42:20.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:19 smithi089 ceph-mon[35770]: 1 slow requests, 0 included below; oldest blocked for > 198.079792 secs 2024-03-22T01:42:20.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:19 smithi072 ceph-mon[32414]: pgmap v173: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s wr, 3.24k op/s 2024-03-22T01:42:20.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:19 smithi072 ceph-mon[32414]: 1 slow requests, 0 included below; oldest blocked for > 198.079792 secs 2024-03-22T01:42:20.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:20 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:42:20.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:20 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:21.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:20 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:42:21.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:20 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:21.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:20 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:42:21.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:20 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:21.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:21 smithi098 ceph-mon[36468]: pgmap v174: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 32 MiB/s wr, 4.22k op/s 2024-03-22T01:42:21.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:21 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:42:21.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:21 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:22.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:21 smithi089 ceph-mon[35770]: pgmap v174: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 32 MiB/s wr, 4.22k op/s 2024-03-22T01:42:22.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:21 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:42:22.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:21 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:22.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:21 smithi072 ceph-mon[32414]: pgmap v174: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 32 MiB/s wr, 4.22k op/s 2024-03-22T01:42:22.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:21 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:42:22.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:21 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:23.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:22 smithi089 ceph-mon[35770]: Health check update: 0 slow ops, oldest one blocked for 224 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:42:23.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:22 smithi072 ceph-mon[32414]: Health check update: 0 slow ops, oldest one blocked for 224 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:42:23.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:22 smithi098 ceph-mon[36468]: Health check update: 0 slow ops, oldest one blocked for 224 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:42:24.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:23 smithi089 ceph-mon[35770]: pgmap v175: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 33 MiB/s wr, 4.03k op/s 2024-03-22T01:42:24.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:23 smithi072 ceph-mon[32414]: pgmap v175: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 33 MiB/s wr, 4.03k op/s 2024-03-22T01:42:24.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:23 smithi098 ceph-mon[36468]: pgmap v175: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 33 MiB/s wr, 4.03k op/s 2024-03-22T01:42:25.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:24 smithi089 ceph-mon[35770]: 1 slow requests, 0 included below; oldest blocked for > 203.079855 secs 2024-03-22T01:42:25.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:24 smithi072 ceph-mon[32414]: 1 slow requests, 0 included below; oldest blocked for > 203.079855 secs 2024-03-22T01:42:25.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:24 smithi098 ceph-mon[36468]: 1 slow requests, 0 included below; oldest blocked for > 203.079855 secs 2024-03-22T01:42:26.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:25 smithi089 ceph-mon[35770]: pgmap v176: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 31 MiB/s wr, 3.77k op/s 2024-03-22T01:42:26.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:25 smithi072 ceph-mon[32414]: pgmap v176: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 31 MiB/s wr, 3.77k op/s 2024-03-22T01:42:26.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:25 smithi098 ceph-mon[36468]: pgmap v176: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 31 MiB/s wr, 3.77k op/s 2024-03-22T01:42:27.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:26 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:42:27.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:26 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:42:27.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:26 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:42:28.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:27 smithi089 ceph-mon[35770]: pgmap v177: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 41 MiB/s wr, 5.10k op/s 2024-03-22T01:42:28.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:27 smithi072 ceph-mon[32414]: pgmap v177: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 41 MiB/s wr, 5.10k op/s 2024-03-22T01:42:28.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:27 smithi098 ceph-mon[36468]: pgmap v177: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 41 MiB/s wr, 5.10k op/s 2024-03-22T01:42:29.008 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:28 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:42:29.008 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:28 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:29.008 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:28 smithi089 ceph-mon[35770]: Health check update: 0 slow ops, oldest one blocked for 229 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:42:29.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:28 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:42:29.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:28 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:29.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:28 smithi072 ceph-mon[32414]: Health check update: 0 slow ops, oldest one blocked for 229 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:42:29.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:28 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:42:29.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:28 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:29.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:28 smithi098 ceph-mon[36468]: Health check update: 0 slow ops, oldest one blocked for 229 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:42:30.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:29 smithi089 ceph-mon[35770]: pgmap v178: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 28 MiB/s wr, 3.47k op/s 2024-03-22T01:42:30.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:29 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:42:30.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:29 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:30.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:29 smithi089 ceph-mon[35770]: 1 slow requests, 0 included below; oldest blocked for > 208.079942 secs 2024-03-22T01:42:30.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:29 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:42:30.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:29 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:42:30.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:29 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:42:30.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:29 smithi072 ceph-mon[32414]: pgmap v178: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 28 MiB/s wr, 3.47k op/s 2024-03-22T01:42:30.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:29 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:42:30.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:29 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:30.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:29 smithi072 ceph-mon[32414]: 1 slow requests, 0 included below; oldest blocked for > 208.079942 secs 2024-03-22T01:42:30.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:29 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:42:30.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:29 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:42:30.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:29 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:42:30.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:29 smithi098 ceph-mon[36468]: pgmap v178: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 28 MiB/s wr, 3.47k op/s 2024-03-22T01:42:30.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:29 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:42:30.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:29 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:30.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:29 smithi098 ceph-mon[36468]: 1 slow requests, 0 included below; oldest blocked for > 208.079942 secs 2024-03-22T01:42:30.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:29 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:42:30.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:29 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:42:30.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:29 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:42:32.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:31 smithi089 ceph-mon[35770]: pgmap v179: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 35 MiB/s wr, 4.27k op/s 2024-03-22T01:42:32.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:31 smithi072 ceph-mon[32414]: pgmap v179: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 35 MiB/s wr, 4.27k op/s 2024-03-22T01:42:32.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:31 smithi098 ceph-mon[36468]: pgmap v179: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 35 MiB/s wr, 4.27k op/s 2024-03-22T01:42:33.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:32 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:42:33.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:32 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:33.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:32 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:42:33.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:32 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:33.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:32 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:42:33.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:32 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:34.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:33 smithi089 ceph-mon[35770]: pgmap v180: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 34 MiB/s wr, 4.25k op/s 2024-03-22T01:42:34.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:33 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:42:34.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:33 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:34.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:33 smithi089 ceph-mon[35770]: Health check update: 0 slow ops, oldest one blocked for 234 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:42:34.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:33 smithi072 ceph-mon[32414]: pgmap v180: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 34 MiB/s wr, 4.25k op/s 2024-03-22T01:42:34.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:33 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:42:34.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:33 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:34.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:33 smithi072 ceph-mon[32414]: Health check update: 0 slow ops, oldest one blocked for 234 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:42:34.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:33 smithi098 ceph-mon[36468]: pgmap v180: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 34 MiB/s wr, 4.25k op/s 2024-03-22T01:42:34.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:33 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:42:34.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:33 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:34.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:33 smithi098 ceph-mon[36468]: Health check update: 0 slow ops, oldest one blocked for 234 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:42:35.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:34 smithi089 ceph-mon[35770]: 1 slow requests, 0 included below; oldest blocked for > 213.080024 secs 2024-03-22T01:42:35.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:34 smithi072 ceph-mon[32414]: 1 slow requests, 0 included below; oldest blocked for > 213.080024 secs 2024-03-22T01:42:35.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:34 smithi098 ceph-mon[36468]: 1 slow requests, 0 included below; oldest blocked for > 213.080024 secs 2024-03-22T01:42:36.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:35 smithi089 ceph-mon[35770]: pgmap v181: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 29 MiB/s wr, 3.58k op/s 2024-03-22T01:42:36.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:35 smithi072 ceph-mon[32414]: pgmap v181: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 29 MiB/s wr, 3.58k op/s 2024-03-22T01:42:36.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:35 smithi098 ceph-mon[36468]: pgmap v181: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 29 MiB/s wr, 3.58k op/s 2024-03-22T01:42:37.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:36 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:42:37.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:36 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:37.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:36 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:42:37.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:36 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:37.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:36 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:42:37.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:36 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:38.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:37 smithi089 ceph-mon[35770]: pgmap v182: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 37 MiB/s wr, 4.64k op/s 2024-03-22T01:42:38.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:37 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:42:38.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:37 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:38.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:37 smithi072 ceph-mon[32414]: pgmap v182: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 37 MiB/s wr, 4.64k op/s 2024-03-22T01:42:38.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:37 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:42:38.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:37 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:38.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:37 smithi098 ceph-mon[36468]: pgmap v182: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 37 MiB/s wr, 4.64k op/s 2024-03-22T01:42:38.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:37 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:42:38.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:37 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:39.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:38 smithi089 ceph-mon[35770]: Health check update: 0 slow ops, oldest one blocked for 239 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:42:39.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:38 smithi072 ceph-mon[32414]: Health check update: 0 slow ops, oldest one blocked for 239 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:42:39.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:38 smithi098 ceph-mon[36468]: Health check update: 0 slow ops, oldest one blocked for 239 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:42:40.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:39 smithi089 ceph-mon[35770]: pgmap v183: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 30 MiB/s wr, 3.78k op/s 2024-03-22T01:42:40.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:39 smithi089 ceph-mon[35770]: 1 slow requests, 0 included below; oldest blocked for > 218.080111 secs 2024-03-22T01:42:40.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:39 smithi072 ceph-mon[32414]: pgmap v183: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 30 MiB/s wr, 3.78k op/s 2024-03-22T01:42:40.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:39 smithi072 ceph-mon[32414]: 1 slow requests, 0 included below; oldest blocked for > 218.080111 secs 2024-03-22T01:42:40.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:39 smithi098 ceph-mon[36468]: pgmap v183: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 30 MiB/s wr, 3.78k op/s 2024-03-22T01:42:40.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:39 smithi098 ceph-mon[36468]: 1 slow requests, 0 included below; oldest blocked for > 218.080111 secs 2024-03-22T01:42:41.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:40 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:42:41.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:40 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:41.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:40 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:42:41.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:40 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:41.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:40 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:42:41.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:40 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:42.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:41 smithi089 ceph-mon[35770]: pgmap v184: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 30 MiB/s wr, 3.81k op/s 2024-03-22T01:42:42.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:41 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:42:42.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:41 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:42.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:41 smithi072 ceph-mon[32414]: pgmap v184: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 30 MiB/s wr, 3.81k op/s 2024-03-22T01:42:42.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:41 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:42:42.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:41 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:42.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:41 smithi098 ceph-mon[36468]: pgmap v184: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 30 MiB/s wr, 3.81k op/s 2024-03-22T01:42:42.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:41 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:42:42.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:41 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:44.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:43 smithi089 ceph-mon[35770]: pgmap v185: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s wr, 3.54k op/s 2024-03-22T01:42:44.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:43 smithi072 ceph-mon[32414]: pgmap v185: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s wr, 3.54k op/s 2024-03-22T01:42:44.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:43 smithi098 ceph-mon[36468]: pgmap v185: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s wr, 3.54k op/s 2024-03-22T01:42:45.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:44 smithi089 ceph-mon[35770]: 1 slow requests, 0 included below; oldest blocked for > 223.080125 secs 2024-03-22T01:42:45.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:44 smithi072 ceph-mon[32414]: 1 slow requests, 0 included below; oldest blocked for > 223.080125 secs 2024-03-22T01:42:45.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:44 smithi098 ceph-mon[36468]: 1 slow requests, 0 included below; oldest blocked for > 223.080125 secs 2024-03-22T01:42:46.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:45 smithi089 ceph-mon[35770]: pgmap v186: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s wr, 2.50k op/s 2024-03-22T01:42:46.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:45 smithi072 ceph-mon[32414]: pgmap v186: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s wr, 2.50k op/s 2024-03-22T01:42:46.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:45 smithi098 ceph-mon[36468]: pgmap v186: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s wr, 2.50k op/s 2024-03-22T01:42:48.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:47 smithi089 ceph-mon[35770]: pgmap v187: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 33 MiB/s wr, 4.17k op/s 2024-03-22T01:42:48.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:47 smithi072 ceph-mon[32414]: pgmap v187: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 33 MiB/s wr, 4.17k op/s 2024-03-22T01:42:48.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:47 smithi098 ceph-mon[36468]: pgmap v187: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 33 MiB/s wr, 4.17k op/s 2024-03-22T01:42:49.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:48 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:42:49.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:48 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:49.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:48 smithi089 ceph-mon[35770]: Health check update: 0 slow ops, oldest one blocked for 245 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:42:49.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:48 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:42:49.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:48 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:49.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:48 smithi072 ceph-mon[32414]: Health check update: 0 slow ops, oldest one blocked for 245 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:42:49.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:48 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:42:49.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:48 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:49.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:48 smithi098 ceph-mon[36468]: Health check update: 0 slow ops, oldest one blocked for 245 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:42:50.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:49 smithi072 ceph-mon[32414]: pgmap v188: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s wr, 3.48k op/s 2024-03-22T01:42:50.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:49 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:42:50.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:49 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:50.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:49 smithi072 ceph-mon[32414]: 1 slow requests, 0 included below; oldest blocked for > 228.080168 secs 2024-03-22T01:42:50.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:49 smithi098 ceph-mon[36468]: pgmap v188: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s wr, 3.48k op/s 2024-03-22T01:42:50.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:49 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:42:50.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:49 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:50.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:49 smithi098 ceph-mon[36468]: 1 slow requests, 0 included below; oldest blocked for > 228.080168 secs 2024-03-22T01:42:50.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:49 smithi089 ceph-mon[35770]: pgmap v188: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s wr, 3.48k op/s 2024-03-22T01:42:50.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:49 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:42:50.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:49 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:50.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:49 smithi089 ceph-mon[35770]: 1 slow requests, 0 included below; oldest blocked for > 228.080168 secs 2024-03-22T01:42:52.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:51 smithi072 ceph-mon[32414]: pgmap v189: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 31 MiB/s wr, 4.11k op/s 2024-03-22T01:42:52.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:51 smithi098 ceph-mon[36468]: pgmap v189: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 31 MiB/s wr, 4.11k op/s 2024-03-22T01:42:52.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:51 smithi089 ceph-mon[35770]: pgmap v189: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 31 MiB/s wr, 4.11k op/s 2024-03-22T01:42:53.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:52 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:42:53.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:52 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:53.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:52 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:42:53.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:52 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:53.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:52 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:42:53.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:52 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:54.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:53 smithi098 ceph-mon[36468]: pgmap v190: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 31 MiB/s wr, 4.26k op/s 2024-03-22T01:42:54.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:53 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:42:54.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:53 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:54.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:53 smithi098 ceph-mon[36468]: Health check update: 0 slow ops, oldest one blocked for 250 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:42:54.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:53 smithi089 ceph-mon[35770]: pgmap v190: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 31 MiB/s wr, 4.26k op/s 2024-03-22T01:42:54.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:53 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:42:54.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:53 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:54.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:53 smithi089 ceph-mon[35770]: Health check update: 0 slow ops, oldest one blocked for 250 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:42:54.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:53 smithi072 ceph-mon[32414]: pgmap v190: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 31 MiB/s wr, 4.26k op/s 2024-03-22T01:42:54.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:53 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:42:54.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:53 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:54.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:53 smithi072 ceph-mon[32414]: Health check update: 0 slow ops, oldest one blocked for 250 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:42:55.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:54 smithi098 ceph-mon[36468]: 1 slow requests, 0 included below; oldest blocked for > 233.080229 secs 2024-03-22T01:42:55.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:54 smithi089 ceph-mon[35770]: 1 slow requests, 0 included below; oldest blocked for > 233.080229 secs 2024-03-22T01:42:55.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:54 smithi072 ceph-mon[32414]: 1 slow requests, 0 included below; oldest blocked for > 233.080229 secs 2024-03-22T01:42:56.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:55 smithi098 ceph-mon[36468]: pgmap v191: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s wr, 3.67k op/s 2024-03-22T01:42:56.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:55 smithi089 ceph-mon[35770]: pgmap v191: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s wr, 3.67k op/s 2024-03-22T01:42:56.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:55 smithi072 ceph-mon[32414]: pgmap v191: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s wr, 3.67k op/s 2024-03-22T01:42:57.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:56 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:42:57.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:56 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:57.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:56 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:42:57.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:56 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:57.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:56 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:42:57.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:56 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:58.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:57 smithi098 ceph-mon[36468]: pgmap v192: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 MiB/s wr, 4.13k op/s 2024-03-22T01:42:58.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:57 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:42:58.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:57 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:58.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:57 smithi089 ceph-mon[35770]: pgmap v192: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 MiB/s wr, 4.13k op/s 2024-03-22T01:42:58.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:57 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:42:58.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:57 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:58.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:57 smithi072 ceph-mon[32414]: pgmap v192: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 MiB/s wr, 4.13k op/s 2024-03-22T01:42:58.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:57 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:42:58.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:57 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:42:59.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:58 smithi098 ceph-mon[36468]: Health check update: 0 slow ops, oldest one blocked for 256 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:42:59.258 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:58 smithi089 ceph-mon[35770]: Health check update: 0 slow ops, oldest one blocked for 256 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:42:59.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:58 smithi072 ceph-mon[32414]: Health check update: 0 slow ops, oldest one blocked for 256 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:43:00.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:59 smithi098 ceph-mon[36468]: pgmap v193: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s wr, 2.56k op/s 2024-03-22T01:43:00.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:42:59 smithi098 ceph-mon[36468]: 1 slow requests, 0 included below; oldest blocked for > 238.080425 secs 2024-03-22T01:43:00.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:59 smithi089 ceph-mon[35770]: pgmap v193: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s wr, 2.56k op/s 2024-03-22T01:43:00.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:42:59 smithi089 ceph-mon[35770]: 1 slow requests, 0 included below; oldest blocked for > 238.080425 secs 2024-03-22T01:43:00.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:59 smithi072 ceph-mon[32414]: pgmap v193: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s wr, 2.56k op/s 2024-03-22T01:43:00.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:42:59 smithi072 ceph-mon[32414]: 1 slow requests, 0 included below; oldest blocked for > 238.080425 secs 2024-03-22T01:43:01.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:00 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:43:01.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:00 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:01.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:00 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:43:01.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:00 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:01.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:00 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:43:01.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:00 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:02.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:01 smithi089 ceph-mon[35770]: pgmap v194: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s wr, 2.84k op/s 2024-03-22T01:43:02.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:01 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:43:02.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:01 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:02.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:01 smithi072 ceph-mon[32414]: pgmap v194: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s wr, 2.84k op/s 2024-03-22T01:43:02.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:01 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:43:02.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:01 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:02.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:01 smithi098 ceph-mon[36468]: pgmap v194: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s wr, 2.84k op/s 2024-03-22T01:43:02.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:01 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:43:02.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:01 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:04.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:03 smithi089 ceph-mon[35770]: pgmap v195: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s wr, 2.00k op/s 2024-03-22T01:43:04.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:03 smithi072 ceph-mon[32414]: pgmap v195: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s wr, 2.00k op/s 2024-03-22T01:43:04.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:03 smithi098 ceph-mon[36468]: pgmap v195: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s wr, 2.00k op/s 2024-03-22T01:43:05.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:04 smithi089 ceph-mon[35770]: 1 slow requests, 0 included below; oldest blocked for > 243.080493 secs 2024-03-22T01:43:05.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:04 smithi072 ceph-mon[32414]: 1 slow requests, 0 included below; oldest blocked for > 243.080493 secs 2024-03-22T01:43:05.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:04 smithi098 ceph-mon[36468]: 1 slow requests, 0 included below; oldest blocked for > 243.080493 secs 2024-03-22T01:43:06.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:05 smithi089 ceph-mon[35770]: pgmap v196: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s wr, 1.67k op/s 2024-03-22T01:43:06.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:05 smithi089 ceph-mon[35770]: Health check update: 0 slow ops, oldest one blocked for 262 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:43:06.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:05 smithi072 ceph-mon[32414]: pgmap v196: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s wr, 1.67k op/s 2024-03-22T01:43:06.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:05 smithi072 ceph-mon[32414]: Health check update: 0 slow ops, oldest one blocked for 262 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:43:06.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:05 smithi098 ceph-mon[36468]: pgmap v196: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s wr, 1.67k op/s 2024-03-22T01:43:06.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:05 smithi098 ceph-mon[36468]: Health check update: 0 slow ops, oldest one blocked for 262 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:43:08.357 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:07 smithi089 ceph-mon[35770]: pgmap v197: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s wr, 2.12k op/s 2024-03-22T01:43:08.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:07 smithi072 ceph-mon[32414]: pgmap v197: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s wr, 2.12k op/s 2024-03-22T01:43:08.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:07 smithi098 ceph-mon[36468]: pgmap v197: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s wr, 2.12k op/s 2024-03-22T01:43:09.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:08 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:43:09.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:08 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:09.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:08 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:43:09.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:08 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:09.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:08 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:43:09.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:08 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:10.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:09 smithi089 ceph-mon[35770]: pgmap v198: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s wr, 1.92k op/s 2024-03-22T01:43:10.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:09 smithi089 ceph-mon[35770]: 1 slow requests, 0 included below; oldest blocked for > 248.080529 secs 2024-03-22T01:43:10.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:09 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:43:10.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:09 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:10.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:09 smithi072 ceph-mon[32414]: pgmap v198: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s wr, 1.92k op/s 2024-03-22T01:43:10.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:09 smithi072 ceph-mon[32414]: 1 slow requests, 0 included below; oldest blocked for > 248.080529 secs 2024-03-22T01:43:10.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:09 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:43:10.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:09 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:10.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:09 smithi098 ceph-mon[36468]: pgmap v198: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s wr, 1.92k op/s 2024-03-22T01:43:10.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:09 smithi098 ceph-mon[36468]: 1 slow requests, 0 included below; oldest blocked for > 248.080529 secs 2024-03-22T01:43:10.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:09 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:43:10.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:09 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:12.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:11 smithi089 ceph-mon[35770]: pgmap v199: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s wr, 2.58k op/s 2024-03-22T01:43:12.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:11 smithi072 ceph-mon[32414]: pgmap v199: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s wr, 2.58k op/s 2024-03-22T01:43:12.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:11 smithi098 ceph-mon[36468]: pgmap v199: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s wr, 2.58k op/s 2024-03-22T01:43:13.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:12 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:43:13.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:12 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:13.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:12 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:43:13.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:12 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:13.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:12 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:43:13.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:12 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:14.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:13 smithi089 ceph-mon[35770]: pgmap v200: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s wr, 2.21k op/s 2024-03-22T01:43:14.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:13 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:43:14.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:13 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:14.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:13 smithi089 ceph-mon[35770]: Health check update: 0 slow ops, oldest one blocked for 271 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:43:14.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:13 smithi089 ceph-mon[35770]: 1 slow requests, 0 included below; oldest blocked for > 253.080618 secs 2024-03-22T01:43:14.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:13 smithi072 ceph-mon[32414]: pgmap v200: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s wr, 2.21k op/s 2024-03-22T01:43:14.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:13 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:43:14.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:13 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:14.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:13 smithi072 ceph-mon[32414]: Health check update: 0 slow ops, oldest one blocked for 271 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:43:14.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:13 smithi072 ceph-mon[32414]: 1 slow requests, 0 included below; oldest blocked for > 253.080618 secs 2024-03-22T01:43:14.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:13 smithi098 ceph-mon[36468]: pgmap v200: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s wr, 2.21k op/s 2024-03-22T01:43:14.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:13 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:43:14.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:13 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:14.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:13 smithi098 ceph-mon[36468]: Health check update: 0 slow ops, oldest one blocked for 271 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:43:14.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:13 smithi098 ceph-mon[36468]: 1 slow requests, 0 included below; oldest blocked for > 253.080618 secs 2024-03-22T01:43:16.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:15 smithi089 ceph-mon[35770]: pgmap v201: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s wr, 1.95k op/s 2024-03-22T01:43:16.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:15 smithi072 ceph-mon[32414]: pgmap v201: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s wr, 1.95k op/s 2024-03-22T01:43:16.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:15 smithi098 ceph-mon[36468]: pgmap v201: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s wr, 1.95k op/s 2024-03-22T01:43:17.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:17 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:43:17.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:17 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:17.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:17 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:43:17.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:17 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:17.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:17 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:43:17.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:17 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:18.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:18 smithi089 ceph-mon[35770]: pgmap v202: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s wr, 2.61k op/s 2024-03-22T01:43:18.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:18 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:43:18.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:18 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:18.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:18 smithi072 ceph-mon[32414]: pgmap v202: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s wr, 2.61k op/s 2024-03-22T01:43:18.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:18 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:43:18.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:18 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:18.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:18 smithi098 ceph-mon[36468]: pgmap v202: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s wr, 2.61k op/s 2024-03-22T01:43:18.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:18 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:43:18.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:18 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:19.273 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:19 smithi072 ceph-mon[32414]: Health check update: 0 slow ops, oldest one blocked for 276 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:43:19.274 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:19 smithi072 ceph-mon[32414]: 1 slow requests, 0 included below; oldest blocked for > 258.080681 secs 2024-03-22T01:43:19.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:19 smithi089 ceph-mon[35770]: Health check update: 0 slow ops, oldest one blocked for 276 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:43:19.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:19 smithi089 ceph-mon[35770]: 1 slow requests, 0 included below; oldest blocked for > 258.080681 secs 2024-03-22T01:43:19.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:19 smithi098 ceph-mon[36468]: Health check update: 0 slow ops, oldest one blocked for 276 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:43:19.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:19 smithi098 ceph-mon[36468]: 1 slow requests, 0 included below; oldest blocked for > 258.080681 secs 2024-03-22T01:43:20.357 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:20 smithi089 ceph-mon[35770]: pgmap v203: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s wr, 2.24k op/s 2024-03-22T01:43:20.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:20 smithi072 ceph-mon[32414]: pgmap v203: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s wr, 2.24k op/s 2024-03-22T01:43:20.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:20 smithi098 ceph-mon[36468]: pgmap v203: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s wr, 2.24k op/s 2024-03-22T01:43:21.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:21 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:43:21.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:21 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:21.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:21 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:43:21.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:21 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:21.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:21 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:43:21.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:21 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:22.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:22 smithi089 ceph-mon[35770]: pgmap v204: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s wr, 2.67k op/s 2024-03-22T01:43:22.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:22 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:43:22.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:22 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:22.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:22 smithi072 ceph-mon[32414]: pgmap v204: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s wr, 2.67k op/s 2024-03-22T01:43:22.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:22 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:43:22.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:22 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:22.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:22 smithi098 ceph-mon[36468]: pgmap v204: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s wr, 2.67k op/s 2024-03-22T01:43:22.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:22 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:43:22.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:22 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:24.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:24 smithi089 ceph-mon[35770]: pgmap v205: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 2.43k op/s 2024-03-22T01:43:24.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:24 smithi089 ceph-mon[35770]: Health check update: 0 slow ops, oldest one blocked for 283 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:43:24.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:24 smithi089 ceph-mon[35770]: 1 slow requests, 0 included below; oldest blocked for > 263.082310 secs 2024-03-22T01:43:24.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:24 smithi072 ceph-mon[32414]: pgmap v205: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 2.43k op/s 2024-03-22T01:43:24.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:24 smithi072 ceph-mon[32414]: Health check update: 0 slow ops, oldest one blocked for 283 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:43:24.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:24 smithi072 ceph-mon[32414]: 1 slow requests, 0 included below; oldest blocked for > 263.082310 secs 2024-03-22T01:43:24.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:24 smithi098 ceph-mon[36468]: pgmap v205: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 2.43k op/s 2024-03-22T01:43:24.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:24 smithi098 ceph-mon[36468]: Health check update: 0 slow ops, oldest one blocked for 283 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:43:24.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:24 smithi098 ceph-mon[36468]: 1 slow requests, 0 included below; oldest blocked for > 263.082310 secs 2024-03-22T01:43:26.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:26 smithi089 ceph-mon[35770]: pgmap v206: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s wr, 2.15k op/s 2024-03-22T01:43:26.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:26 smithi072 ceph-mon[32414]: pgmap v206: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s wr, 2.15k op/s 2024-03-22T01:43:26.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:26 smithi098 ceph-mon[36468]: pgmap v206: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s wr, 2.15k op/s 2024-03-22T01:43:28.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:28 smithi089 ceph-mon[35770]: pgmap v207: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s wr, 3.19k op/s 2024-03-22T01:43:28.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:28 smithi072 ceph-mon[32414]: pgmap v207: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s wr, 3.19k op/s 2024-03-22T01:43:28.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:28 smithi098 ceph-mon[36468]: pgmap v207: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s wr, 3.19k op/s 2024-03-22T01:43:29.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:29 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:43:29.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:29 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:29.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:29 smithi089 ceph-mon[35770]: 1 slow requests, 0 included below; oldest blocked for > 268.082372 secs 2024-03-22T01:43:29.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:29 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:43:29.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:29 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:29.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:29 smithi072 ceph-mon[32414]: 1 slow requests, 0 included below; oldest blocked for > 268.082372 secs 2024-03-22T01:43:29.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:29 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:43:29.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:29 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:29.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:29 smithi098 ceph-mon[36468]: 1 slow requests, 0 included below; oldest blocked for > 268.082372 secs 2024-03-22T01:43:30.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:30 smithi089 ceph-mon[35770]: pgmap v208: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s wr, 2.65k op/s 2024-03-22T01:43:30.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:30 smithi089 ceph-mon[35770]: Health check update: 0 slow ops, oldest one blocked for 289 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:43:30.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:30 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:43:30.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:30 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:30.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:30 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:43:30.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:30 smithi072 ceph-mon[32414]: pgmap v208: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s wr, 2.65k op/s 2024-03-22T01:43:30.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:30 smithi072 ceph-mon[32414]: Health check update: 0 slow ops, oldest one blocked for 289 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:43:30.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:30 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:43:30.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:30 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:30.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:30 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:43:30.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:30 smithi098 ceph-mon[36468]: pgmap v208: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s wr, 2.65k op/s 2024-03-22T01:43:30.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:30 smithi098 ceph-mon[36468]: Health check update: 0 slow ops, oldest one blocked for 289 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:43:30.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:30 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:43:30.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:30 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:30.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:30 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:43:31.322 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:31 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:43:31.323 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:31 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:43:31.323 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:31 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:43:31.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:31 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:43:31.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:31 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:43:31.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:31 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:43:31.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:31 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:43:31.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:31 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:43:31.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:31 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:43:32.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:32 smithi089 ceph-mon[35770]: pgmap v209: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s wr, 3.08k op/s 2024-03-22T01:43:32.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:32 smithi072 ceph-mon[32414]: pgmap v209: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s wr, 3.08k op/s 2024-03-22T01:43:32.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:32 smithi098 ceph-mon[36468]: pgmap v209: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s wr, 3.08k op/s 2024-03-22T01:43:33.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:33 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:43:33.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:33 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:33.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:33 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:43:33.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:33 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:33.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:33 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:43:33.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:33 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:34.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:34 smithi089 ceph-mon[35770]: pgmap v210: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 2.61k op/s 2024-03-22T01:43:34.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:34 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:43:34.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:34 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:34.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:34 smithi089 ceph-mon[35770]: 1 slow requests, 0 included below; oldest blocked for > 273.082453 secs 2024-03-22T01:43:34.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:34 smithi072 ceph-mon[32414]: pgmap v210: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 2.61k op/s 2024-03-22T01:43:34.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:34 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:43:34.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:34 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:34.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:34 smithi072 ceph-mon[32414]: 1 slow requests, 0 included below; oldest blocked for > 273.082453 secs 2024-03-22T01:43:34.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:34 smithi098 ceph-mon[36468]: pgmap v210: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 2.61k op/s 2024-03-22T01:43:34.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:34 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:43:34.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:34 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:34.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:34 smithi098 ceph-mon[36468]: 1 slow requests, 0 included below; oldest blocked for > 273.082453 secs 2024-03-22T01:43:36.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:36 smithi089 ceph-mon[35770]: pgmap v211: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s wr, 2.08k op/s 2024-03-22T01:43:36.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:36 smithi072 ceph-mon[32414]: pgmap v211: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s wr, 2.08k op/s 2024-03-22T01:43:36.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:36 smithi098 ceph-mon[36468]: pgmap v211: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s wr, 2.08k op/s 2024-03-22T01:43:37.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:37 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:43:37.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:37 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:37.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:37 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:43:37.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:37 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:37.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:37 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:43:37.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:37 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:38.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:38 smithi089 ceph-mon[35770]: pgmap v212: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s wr, 3.32k op/s 2024-03-22T01:43:38.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:38 smithi089 ceph-mon[35770]: Health check update: 0 slow ops, oldest one blocked for 296 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:43:38.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:38 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:43:38.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:38 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:38.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:38 smithi072 ceph-mon[32414]: pgmap v212: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s wr, 3.32k op/s 2024-03-22T01:43:38.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:38 smithi072 ceph-mon[32414]: Health check update: 0 slow ops, oldest one blocked for 296 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:43:38.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:38 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:43:38.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:38 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:38.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:38 smithi098 ceph-mon[36468]: pgmap v212: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s wr, 3.32k op/s 2024-03-22T01:43:38.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:38 smithi098 ceph-mon[36468]: Health check update: 0 slow ops, oldest one blocked for 296 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:43:38.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:38 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:43:38.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:38 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:39.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:39 smithi089 ceph-mon[35770]: 1 slow requests, 0 included below; oldest blocked for > 278.082515 secs 2024-03-22T01:43:39.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:39 smithi072 ceph-mon[32414]: 1 slow requests, 0 included below; oldest blocked for > 278.082515 secs 2024-03-22T01:43:39.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:39 smithi098 ceph-mon[36468]: 1 slow requests, 0 included below; oldest blocked for > 278.082515 secs 2024-03-22T01:43:40.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:40 smithi072 ceph-mon[32414]: pgmap v213: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 2.45k op/s 2024-03-22T01:43:40.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:40 smithi098 ceph-mon[36468]: pgmap v213: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 2.45k op/s 2024-03-22T01:43:40.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:40 smithi089 ceph-mon[35770]: pgmap v213: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 2.45k op/s 2024-03-22T01:43:41.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:41 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:43:41.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:41 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:41.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:41 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:43:41.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:41 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:41.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:41 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:43:41.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:41 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:42.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:42 smithi072 ceph-mon[32414]: pgmap v214: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s wr, 2.85k op/s 2024-03-22T01:43:42.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:42 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:43:42.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:42 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:42.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:42 smithi098 ceph-mon[36468]: pgmap v214: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s wr, 2.85k op/s 2024-03-22T01:43:42.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:42 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:43:42.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:42 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:42.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:42 smithi089 ceph-mon[35770]: pgmap v214: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s wr, 2.85k op/s 2024-03-22T01:43:42.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:42 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:43:42.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:42 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:44.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:44 smithi098 ceph-mon[36468]: pgmap v215: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s wr, 2.72k op/s 2024-03-22T01:43:44.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:44 smithi098 ceph-mon[36468]: 1 slow requests, 0 included below; oldest blocked for > 283.082600 secs 2024-03-22T01:43:44.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:44 smithi089 ceph-mon[35770]: pgmap v215: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s wr, 2.72k op/s 2024-03-22T01:43:44.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:44 smithi089 ceph-mon[35770]: 1 slow requests, 0 included below; oldest blocked for > 283.082600 secs 2024-03-22T01:43:44.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:44 smithi072 ceph-mon[32414]: pgmap v215: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s wr, 2.72k op/s 2024-03-22T01:43:44.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:44 smithi072 ceph-mon[32414]: 1 slow requests, 0 included below; oldest blocked for > 283.082600 secs 2024-03-22T01:43:45.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:45 smithi089 ceph-mon[35770]: Health check update: 0 slow ops, oldest one blocked for 304 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:43:45.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:45 smithi072 ceph-mon[32414]: Health check update: 0 slow ops, oldest one blocked for 304 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:43:45.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:45 smithi098 ceph-mon[36468]: Health check update: 0 slow ops, oldest one blocked for 304 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:43:46.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:46 smithi089 ceph-mon[35770]: pgmap v216: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 2.57k op/s 2024-03-22T01:43:46.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:46 smithi072 ceph-mon[32414]: pgmap v216: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 2.57k op/s 2024-03-22T01:43:46.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:46 smithi098 ceph-mon[36468]: pgmap v216: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 2.57k op/s 2024-03-22T01:43:48.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:48 smithi089 ceph-mon[35770]: pgmap v217: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s wr, 3.25k op/s 2024-03-22T01:43:48.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:48 smithi072 ceph-mon[32414]: pgmap v217: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s wr, 3.25k op/s 2024-03-22T01:43:48.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:48 smithi098 ceph-mon[36468]: pgmap v217: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s wr, 3.25k op/s 2024-03-22T01:43:49.523 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:49 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:43:49.523 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:49 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:49.523 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:49 smithi072 ceph-mon[32414]: pgmap v218: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s wr, 2.14k op/s 2024-03-22T01:43:49.523 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:49 smithi072 ceph-mon[32414]: 1 slow requests, 0 included below; oldest blocked for > 288.082661 secs 2024-03-22T01:43:49.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:49 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:43:49.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:49 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:49.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:49 smithi089 ceph-mon[35770]: pgmap v218: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s wr, 2.14k op/s 2024-03-22T01:43:49.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:49 smithi089 ceph-mon[35770]: 1 slow requests, 0 included below; oldest blocked for > 288.082661 secs 2024-03-22T01:43:49.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:49 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:43:49.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:49 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:49.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:49 smithi098 ceph-mon[36468]: pgmap v218: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s wr, 2.14k op/s 2024-03-22T01:43:49.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:49 smithi098 ceph-mon[36468]: 1 slow requests, 0 included below; oldest blocked for > 288.082661 secs 2024-03-22T01:43:50.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:50 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:43:50.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:50 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:50.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:50 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:43:50.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:50 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:50.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:50 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:43:50.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:50 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:51.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:51 smithi089 ceph-mon[35770]: pgmap v219: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s wr, 2.62k op/s 2024-03-22T01:43:51.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:51 smithi072 ceph-mon[32414]: pgmap v219: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s wr, 2.62k op/s 2024-03-22T01:43:51.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:51 smithi098 ceph-mon[36468]: pgmap v219: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s wr, 2.62k op/s 2024-03-22T01:43:52.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:52 smithi089 ceph-mon[35770]: Health check update: 0 slow ops, oldest one blocked for 311 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:43:52.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:52 smithi072 ceph-mon[32414]: Health check update: 0 slow ops, oldest one blocked for 311 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:43:52.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:52 smithi098 ceph-mon[36468]: Health check update: 0 slow ops, oldest one blocked for 311 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:43:53.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:53 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:43:53.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:53 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:53.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:53 smithi089 ceph-mon[35770]: pgmap v220: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 2.62k op/s 2024-03-22T01:43:53.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:53 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:43:53.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:53 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:53.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:53 smithi072 ceph-mon[32414]: pgmap v220: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 2.62k op/s 2024-03-22T01:43:53.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:53 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:43:53.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:53 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:53.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:53 smithi098 ceph-mon[36468]: pgmap v220: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 2.62k op/s 2024-03-22T01:43:54.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:54 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:43:54.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:54 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:54.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:54 smithi089 ceph-mon[35770]: 1 slow requests, 0 included below; oldest blocked for > 293.082784 secs 2024-03-22T01:43:54.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:54 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:43:54.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:54 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:54.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:54 smithi072 ceph-mon[32414]: 1 slow requests, 0 included below; oldest blocked for > 293.082784 secs 2024-03-22T01:43:54.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:54 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:43:54.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:54 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:54.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:54 smithi098 ceph-mon[36468]: 1 slow requests, 0 included below; oldest blocked for > 293.082784 secs 2024-03-22T01:43:55.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:55 smithi089 ceph-mon[35770]: pgmap v221: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s wr, 2.18k op/s 2024-03-22T01:43:55.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:55 smithi072 ceph-mon[32414]: pgmap v221: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s wr, 2.18k op/s 2024-03-22T01:43:55.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:55 smithi098 ceph-mon[36468]: pgmap v221: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s wr, 2.18k op/s 2024-03-22T01:43:57.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:57 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:43:57.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:57 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:57.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:57 smithi089 ceph-mon[35770]: pgmap v222: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s wr, 3.19k op/s 2024-03-22T01:43:57.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:57 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:43:57.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:57 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:57.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:57 smithi072 ceph-mon[32414]: pgmap v222: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s wr, 3.19k op/s 2024-03-22T01:43:57.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:57 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:43:57.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:57 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:57.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:57 smithi098 ceph-mon[36468]: pgmap v222: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s wr, 3.19k op/s 2024-03-22T01:43:58.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:58 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:43:58.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:58 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:58.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:58 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:43:58.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:58 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:58.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:58 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:43:58.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:58 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:43:59.509 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:59 smithi089 ceph-mon[35770]: pgmap v223: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 2.72k op/s 2024-03-22T01:43:59.509 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:43:59 smithi089 ceph-mon[35770]: 1 slow requests, 0 included below; oldest blocked for > 298.082862 secs 2024-03-22T01:43:59.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:59 smithi072 ceph-mon[32414]: pgmap v223: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 2.72k op/s 2024-03-22T01:43:59.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:43:59 smithi072 ceph-mon[32414]: 1 slow requests, 0 included below; oldest blocked for > 298.082862 secs 2024-03-22T01:43:59.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:59 smithi098 ceph-mon[36468]: pgmap v223: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 2.72k op/s 2024-03-22T01:43:59.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:43:59 smithi098 ceph-mon[36468]: 1 slow requests, 0 included below; oldest blocked for > 298.082862 secs 2024-03-22T01:44:00.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:00 smithi089 ceph-mon[35770]: Health check update: 0 slow ops, oldest one blocked for 317 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:44:00.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:00 smithi072 ceph-mon[32414]: Health check update: 0 slow ops, oldest one blocked for 317 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:44:00.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:00 smithi098 ceph-mon[36468]: Health check update: 0 slow ops, oldest one blocked for 317 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:44:01.572 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:01 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:44:01.572 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:01 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:44:01.572 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:01 smithi098 ceph-mon[36468]: pgmap v224: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s wr, 3.07k op/s 2024-03-22T01:44:01.607 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:01 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:44:01.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:01 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:44:01.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:01 smithi089 ceph-mon[35770]: pgmap v224: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s wr, 3.07k op/s 2024-03-22T01:44:01.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:01 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:44:01.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:01 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:44:01.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:01 smithi072 ceph-mon[32414]: pgmap v224: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s wr, 3.07k op/s 2024-03-22T01:44:02.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:02 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:44:02.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:02 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:44:02.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:02 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:44:02.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:02 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:44:02.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:02 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:44:02.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:02 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:44:03.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:03 smithi089 ceph-mon[35770]: pgmap v225: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 2.87k op/s 2024-03-22T01:44:03.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:03 smithi072 ceph-mon[32414]: pgmap v225: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 2.87k op/s 2024-03-22T01:44:03.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:03 smithi098 ceph-mon[36468]: pgmap v225: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 2.87k op/s 2024-03-22T01:44:04.607 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:04 smithi089 ceph-mon[35770]: 1 slow requests, 0 included below; oldest blocked for > 303.082947 secs 2024-03-22T01:44:04.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:04 smithi072 ceph-mon[32414]: 1 slow requests, 0 included below; oldest blocked for > 303.082947 secs 2024-03-22T01:44:04.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:04 smithi098 ceph-mon[36468]: 1 slow requests, 0 included below; oldest blocked for > 303.082947 secs 2024-03-22T01:44:05.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:05 smithi089 ceph-mon[35770]: pgmap v226: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s wr, 2.31k op/s 2024-03-22T01:44:05.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:05 smithi072 ceph-mon[32414]: pgmap v226: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s wr, 2.31k op/s 2024-03-22T01:44:05.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:05 smithi098 ceph-mon[36468]: pgmap v226: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s wr, 2.31k op/s 2024-03-22T01:44:06.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:06 smithi089 ceph-mon[35770]: Health check update: 0 slow ops, oldest one blocked for 325 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:44:06.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:06 smithi072 ceph-mon[32414]: Health check update: 0 slow ops, oldest one blocked for 325 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:44:06.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:06 smithi098 ceph-mon[36468]: Health check update: 0 slow ops, oldest one blocked for 325 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:44:07.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:07 smithi089 ceph-mon[35770]: pgmap v227: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s wr, 3.04k op/s 2024-03-22T01:44:07.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:07 smithi072 ceph-mon[32414]: pgmap v227: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s wr, 3.04k op/s 2024-03-22T01:44:07.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:07 smithi098 ceph-mon[36468]: pgmap v227: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s wr, 3.04k op/s 2024-03-22T01:44:09.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:09 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:44:09.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:09 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:44:09.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:09 smithi089 ceph-mon[35770]: pgmap v228: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s wr, 2.12k op/s 2024-03-22T01:44:09.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:09 smithi089 ceph-mon[35770]: 1 slow requests, 0 included below; oldest blocked for > 308.083030 secs 2024-03-22T01:44:09.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:09 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:44:09.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:09 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:44:09.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:09 smithi072 ceph-mon[32414]: pgmap v228: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s wr, 2.12k op/s 2024-03-22T01:44:09.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:09 smithi072 ceph-mon[32414]: 1 slow requests, 0 included below; oldest blocked for > 308.083030 secs 2024-03-22T01:44:09.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:09 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:44:09.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:09 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:44:09.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:09 smithi098 ceph-mon[36468]: pgmap v228: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s wr, 2.12k op/s 2024-03-22T01:44:09.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:09 smithi098 ceph-mon[36468]: 1 slow requests, 0 included below; oldest blocked for > 308.083030 secs 2024-03-22T01:44:10.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:10 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:44:10.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:10 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:44:10.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:10 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:44:10.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:10 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:44:10.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:10 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:44:10.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:10 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:44:11.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:11 smithi089 ceph-mon[35770]: pgmap v229: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s wr, 2.55k op/s 2024-03-22T01:44:11.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:11 smithi072 ceph-mon[32414]: pgmap v229: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s wr, 2.55k op/s 2024-03-22T01:44:11.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:11 smithi098 ceph-mon[36468]: pgmap v229: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s wr, 2.55k op/s 2024-03-22T01:44:12.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:12 smithi089 ceph-mon[35770]: Health check update: 0 slow ops, oldest one blocked for 331 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:44:12.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:12 smithi072 ceph-mon[32414]: Health check update: 0 slow ops, oldest one blocked for 331 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:44:12.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:12 smithi098 ceph-mon[36468]: Health check update: 0 slow ops, oldest one blocked for 331 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:44:13.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:13 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:44:13.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:13 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:44:13.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:13 smithi089 ceph-mon[35770]: pgmap v230: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s wr, 2.45k op/s 2024-03-22T01:44:13.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:13 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:44:13.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:13 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:44:13.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:13 smithi072 ceph-mon[32414]: pgmap v230: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s wr, 2.45k op/s 2024-03-22T01:44:13.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:13 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:44:13.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:13 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:44:13.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:13 smithi098 ceph-mon[36468]: pgmap v230: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s wr, 2.45k op/s 2024-03-22T01:44:14.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:14 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:44:14.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:14 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:44:14.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:14 smithi089 ceph-mon[35770]: 1 slow requests, 0 included below; oldest blocked for > 313.083092 secs 2024-03-22T01:44:14.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:14 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:44:14.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:14 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:44:14.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:14 smithi072 ceph-mon[32414]: 1 slow requests, 0 included below; oldest blocked for > 313.083092 secs 2024-03-22T01:44:14.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:14 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:44:14.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:14 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:44:14.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:14 smithi098 ceph-mon[36468]: 1 slow requests, 0 included below; oldest blocked for > 313.083092 secs 2024-03-22T01:44:15.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:15 smithi089 ceph-mon[35770]: pgmap v231: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s wr, 2.22k op/s 2024-03-22T01:44:15.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:15 smithi072 ceph-mon[32414]: pgmap v231: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s wr, 2.22k op/s 2024-03-22T01:44:15.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:15 smithi098 ceph-mon[36468]: pgmap v231: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s wr, 2.22k op/s 2024-03-22T01:44:17.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:17 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:44:17.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:17 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:44:17.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:17 smithi089 ceph-mon[35770]: pgmap v232: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 2.90k op/s 2024-03-22T01:44:17.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:17 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:44:17.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:17 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:44:17.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:17 smithi072 ceph-mon[32414]: pgmap v232: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 2.90k op/s 2024-03-22T01:44:17.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:17 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:44:17.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:17 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:44:17.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:17 smithi098 ceph-mon[36468]: pgmap v232: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 2.90k op/s 2024-03-22T01:44:18.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:18 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:44:18.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:18 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:44:18.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:18 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:44:18.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:18 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:44:18.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:18 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:44:18.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:18 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:44:19.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:19 smithi089 ceph-mon[35770]: pgmap v233: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s wr, 2.25k op/s 2024-03-22T01:44:19.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:19 smithi089 ceph-mon[35770]: Health check update: 0 slow ops, oldest one blocked for 337 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:44:19.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:19 smithi089 ceph-mon[35770]: 1 slow requests, 0 included below; oldest blocked for > 318.083176 secs 2024-03-22T01:44:19.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:19 smithi072 ceph-mon[32414]: pgmap v233: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s wr, 2.25k op/s 2024-03-22T01:44:19.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:19 smithi072 ceph-mon[32414]: Health check update: 0 slow ops, oldest one blocked for 337 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:44:19.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:19 smithi072 ceph-mon[32414]: 1 slow requests, 0 included below; oldest blocked for > 318.083176 secs 2024-03-22T01:44:19.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:19 smithi098 ceph-mon[36468]: pgmap v233: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s wr, 2.25k op/s 2024-03-22T01:44:19.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:19 smithi098 ceph-mon[36468]: Health check update: 0 slow ops, oldest one blocked for 337 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:44:19.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:19 smithi098 ceph-mon[36468]: 1 slow requests, 0 included below; oldest blocked for > 318.083176 secs 2024-03-22T01:44:21.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:21 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:44:21.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:21 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:44:21.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:21 smithi089 ceph-mon[35770]: pgmap v234: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s wr, 2.64k op/s 2024-03-22T01:44:21.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:21 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:44:21.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:21 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:44:21.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:21 smithi072 ceph-mon[32414]: pgmap v234: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s wr, 2.64k op/s 2024-03-22T01:44:21.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:21 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:44:21.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:21 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:44:21.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:21 smithi098 ceph-mon[36468]: pgmap v234: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s wr, 2.64k op/s 2024-03-22T01:44:22.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:22 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:44:22.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:22 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:44:22.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:22 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:44:22.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:22 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:44:22.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:22 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:44:22.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:22 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:44:23.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:23 smithi089 ceph-mon[35770]: pgmap v235: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s wr, 2.47k op/s 2024-03-22T01:44:23.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:23 smithi072 ceph-mon[32414]: pgmap v235: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s wr, 2.47k op/s 2024-03-22T01:44:23.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:23 smithi098 ceph-mon[36468]: pgmap v235: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s wr, 2.47k op/s 2024-03-22T01:44:24.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:24 smithi089 ceph-mon[35770]: Health check update: 0 slow ops, oldest one blocked for 343 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:44:24.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:24 smithi089 ceph-mon[35770]: 1 slow requests, 0 included below; oldest blocked for > 323.083246 secs 2024-03-22T01:44:24.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:24 smithi072 ceph-mon[32414]: Health check update: 0 slow ops, oldest one blocked for 343 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:44:24.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:24 smithi072 ceph-mon[32414]: 1 slow requests, 0 included below; oldest blocked for > 323.083246 secs 2024-03-22T01:44:24.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:24 smithi098 ceph-mon[36468]: Health check update: 0 slow ops, oldest one blocked for 343 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:44:24.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:24 smithi098 ceph-mon[36468]: 1 slow requests, 0 included below; oldest blocked for > 323.083246 secs 2024-03-22T01:44:25.607 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:25 smithi089 ceph-mon[35770]: pgmap v236: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s wr, 2.18k op/s 2024-03-22T01:44:25.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:25 smithi072 ceph-mon[32414]: pgmap v236: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s wr, 2.18k op/s 2024-03-22T01:44:25.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:25 smithi098 ceph-mon[36468]: pgmap v236: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s wr, 2.18k op/s 2024-03-22T01:44:27.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:27 smithi089 ceph-mon[35770]: pgmap v237: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s wr, 2.96k op/s 2024-03-22T01:44:27.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:27 smithi072 ceph-mon[32414]: pgmap v237: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s wr, 2.96k op/s 2024-03-22T01:44:27.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:27 smithi098 ceph-mon[36468]: pgmap v237: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s wr, 2.96k op/s 2024-03-22T01:44:29.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:29 smithi089 ceph-mon[35770]: pgmap v238: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s wr, 2.42k op/s 2024-03-22T01:44:29.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:29 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:44:29.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:29 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:44:29.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:29 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:44:29.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:29 smithi089 ceph-mon[35770]: Health check update: 0 slow ops, oldest one blocked for 348 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:44:29.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:29 smithi089 ceph-mon[35770]: 1 slow requests, 0 included below; oldest blocked for > 328.083308 secs 2024-03-22T01:44:29.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:29 smithi072 ceph-mon[32414]: pgmap v238: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s wr, 2.42k op/s 2024-03-22T01:44:29.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:29 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:44:29.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:29 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:44:29.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:29 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:44:29.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:29 smithi072 ceph-mon[32414]: Health check update: 0 slow ops, oldest one blocked for 348 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:44:29.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:29 smithi072 ceph-mon[32414]: 1 slow requests, 0 included below; oldest blocked for > 328.083308 secs 2024-03-22T01:44:29.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:29 smithi098 ceph-mon[36468]: pgmap v238: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s wr, 2.42k op/s 2024-03-22T01:44:29.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:29 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:44:29.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:29 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:44:29.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:29 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:44:29.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:29 smithi098 ceph-mon[36468]: Health check update: 0 slow ops, oldest one blocked for 348 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:44:29.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:29 smithi098 ceph-mon[36468]: 1 slow requests, 0 included below; oldest blocked for > 328.083308 secs 2024-03-22T01:44:31.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:31 smithi089 ceph-mon[35770]: pgmap v239: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 2.75k op/s 2024-03-22T01:44:31.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:31 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:44:31.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:31 smithi072 ceph-mon[32414]: pgmap v239: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 2.75k op/s 2024-03-22T01:44:31.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:31 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:44:31.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:31 smithi098 ceph-mon[36468]: pgmap v239: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 2.75k op/s 2024-03-22T01:44:31.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:31 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:44:32.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:32 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:44:32.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:32 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:44:32.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:32 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:44:32.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:32 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:44:32.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:32 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:44:32.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:32 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:44:32.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:32 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:44:32.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:32 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:44:32.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:32 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:44:33.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:33 smithi072 ceph-mon[32414]: pgmap v240: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s wr, 2.91k op/s 2024-03-22T01:44:33.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:33 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:44:33.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:33 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:44:33.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:33 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:44:33.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:33 smithi098 ceph-mon[36468]: pgmap v240: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s wr, 2.91k op/s 2024-03-22T01:44:33.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:33 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:44:33.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:33 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:44:33.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:33 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:44:33.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:33 smithi089 ceph-mon[35770]: pgmap v240: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s wr, 2.91k op/s 2024-03-22T01:44:33.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:33 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:44:33.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:33 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:44:33.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:33 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:44:34.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:34 smithi072 ceph-mon[32414]: Health check update: 0 slow ops, oldest one blocked for 355 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:44:34.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:34 smithi072 ceph-mon[32414]: 1 slow requests, 0 included below; oldest blocked for > 333.083323 secs 2024-03-22T01:44:34.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:34 smithi098 ceph-mon[36468]: Health check update: 0 slow ops, oldest one blocked for 355 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:44:34.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:34 smithi098 ceph-mon[36468]: 1 slow requests, 0 included below; oldest blocked for > 333.083323 secs 2024-03-22T01:44:34.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:34 smithi089 ceph-mon[35770]: Health check update: 0 slow ops, oldest one blocked for 355 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:44:34.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:34 smithi089 ceph-mon[35770]: 1 slow requests, 0 included below; oldest blocked for > 333.083323 secs 2024-03-22T01:44:35.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:35 smithi098 ceph-mon[36468]: pgmap v241: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s wr, 2.54k op/s 2024-03-22T01:44:35.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:35 smithi089 ceph-mon[35770]: pgmap v241: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s wr, 2.54k op/s 2024-03-22T01:44:35.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:35 smithi072 ceph-mon[32414]: pgmap v241: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s wr, 2.54k op/s 2024-03-22T01:44:37.674 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:37 smithi098 ceph-mon[36468]: pgmap v242: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s wr, 3.17k op/s 2024-03-22T01:44:37.674 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:37 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:44:37.674 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:37 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:44:37.674 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:37 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:44:37.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:37 smithi089 ceph-mon[35770]: pgmap v242: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s wr, 3.17k op/s 2024-03-22T01:44:37.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:37 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:44:37.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:37 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:44:37.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:37 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:44:37.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:37 smithi072 ceph-mon[32414]: pgmap v242: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s wr, 3.17k op/s 2024-03-22T01:44:37.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:37 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:44:37.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:37 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:44:37.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:37 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:44:39.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:39 smithi098 ceph-mon[36468]: pgmap v243: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s wr, 2.46k op/s 2024-03-22T01:44:39.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:39 smithi098 ceph-mon[36468]: Health check update: 0 slow ops, oldest one blocked for 361 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:44:39.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:39 smithi098 ceph-mon[36468]: 1 slow requests, 0 included below; oldest blocked for > 338.083387 secs 2024-03-22T01:44:39.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:39 smithi089 ceph-mon[35770]: pgmap v243: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s wr, 2.46k op/s 2024-03-22T01:44:39.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:39 smithi089 ceph-mon[35770]: Health check update: 0 slow ops, oldest one blocked for 361 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:44:39.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:39 smithi089 ceph-mon[35770]: 1 slow requests, 0 included below; oldest blocked for > 338.083387 secs 2024-03-22T01:44:39.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:39 smithi072 ceph-mon[32414]: pgmap v243: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s wr, 2.46k op/s 2024-03-22T01:44:39.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:39 smithi072 ceph-mon[32414]: Health check update: 0 slow ops, oldest one blocked for 361 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:44:39.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:39 smithi072 ceph-mon[32414]: 1 slow requests, 0 included below; oldest blocked for > 338.083387 secs 2024-03-22T01:44:41.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:41 smithi098 ceph-mon[36468]: pgmap v244: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 2.66k op/s 2024-03-22T01:44:41.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:41 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:44:41.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:41 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:44:41.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:41 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:44:41.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:41 smithi089 ceph-mon[35770]: pgmap v244: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 2.66k op/s 2024-03-22T01:44:41.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:41 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:44:41.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:41 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:44:41.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:41 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:44:41.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:41 smithi072 ceph-mon[32414]: pgmap v244: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 2.66k op/s 2024-03-22T01:44:41.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:41 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:44:41.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:41 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:44:41.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:41 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:44:43.567 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:44:43.568+0000 7f3e297fa640 1 -- 172.21.15.72:0/1386165290 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f3e0c005850 con 0x7f3e34069e10 2024-03-22T01:44:43.567 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:44:43.568+0000 7f3e2a7fc640 1 -- 172.21.15.72:0/1386165290 <== mon.0 v2:172.21.15.72:3300/0 7 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f3e0c005850 con 0x7f3e34069e10 2024-03-22T01:44:43.611 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:44:43.612+0000 7f3e137fe640 1 -- 172.21.15.72:0/2014627542 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f3e04005850 con 0x55af9cd592d0 2024-03-22T01:44:43.611 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:44:43.612+0000 7f3e1cff9640 1 -- 172.21.15.72:0/2014627542 <== mon.0 v2:172.21.15.72:3300/0 7 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f3e04005850 con 0x55af9cd592d0 2024-03-22T01:44:43.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:43 smithi098 ceph-mon[36468]: pgmap v245: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 2.69k op/s 2024-03-22T01:44:43.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:43 smithi089 ceph-mon[35770]: pgmap v245: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 2.69k op/s 2024-03-22T01:44:43.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:43 smithi072 ceph-mon[32414]: pgmap v245: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 2.69k op/s 2024-03-22T01:44:45.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:44 smithi089 ceph-mon[35770]: 1 slow requests, 0 included below; oldest blocked for > 343.083454 secs 2024-03-22T01:44:45.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:44 smithi072 ceph-mon[32414]: 1 slow requests, 0 included below; oldest blocked for > 343.083454 secs 2024-03-22T01:44:45.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:44 smithi098 ceph-mon[36468]: 1 slow requests, 0 included below; oldest blocked for > 343.083454 secs 2024-03-22T01:44:46.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:45 smithi089 ceph-mon[35770]: pgmap v246: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s wr, 2.16k op/s 2024-03-22T01:44:46.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:45 smithi072 ceph-mon[32414]: pgmap v246: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s wr, 2.16k op/s 2024-03-22T01:44:46.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:45 smithi098 ceph-mon[36468]: pgmap v246: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s wr, 2.16k op/s 2024-03-22T01:44:47.110 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:46 smithi089 ceph-mon[35770]: Health check update: 0 slow ops, oldest one blocked for 366 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:44:47.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:46 smithi072 ceph-mon[32414]: Health check update: 0 slow ops, oldest one blocked for 366 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:44:47.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:46 smithi098 ceph-mon[36468]: Health check update: 0 slow ops, oldest one blocked for 366 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:44:48.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:47 smithi089 ceph-mon[35770]: pgmap v247: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s wr, 3.24k op/s 2024-03-22T01:44:48.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:47 smithi072 ceph-mon[32414]: pgmap v247: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s wr, 3.24k op/s 2024-03-22T01:44:48.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:47 smithi098 ceph-mon[36468]: pgmap v247: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s wr, 3.24k op/s 2024-03-22T01:44:49.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:48 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:44:49.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:48 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:44:49.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:48 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:44:49.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:48 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:44:49.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:48 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:44:49.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:48 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:44:50.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:49 smithi089 ceph-mon[35770]: pgmap v248: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s wr, 2.60k op/s 2024-03-22T01:44:50.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:49 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:44:50.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:49 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:44:50.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:49 smithi089 ceph-mon[35770]: 1 slow requests, 0 included below; oldest blocked for > 348.083518 secs 2024-03-22T01:44:50.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:49 smithi072 ceph-mon[32414]: pgmap v248: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s wr, 2.60k op/s 2024-03-22T01:44:50.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:49 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:44:50.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:49 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:44:50.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:49 smithi072 ceph-mon[32414]: 1 slow requests, 0 included below; oldest blocked for > 348.083518 secs 2024-03-22T01:44:50.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:49 smithi098 ceph-mon[36468]: pgmap v248: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s wr, 2.60k op/s 2024-03-22T01:44:50.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:49 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:44:50.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:49 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:44:50.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:49 smithi098 ceph-mon[36468]: 1 slow requests, 0 included below; oldest blocked for > 348.083518 secs 2024-03-22T01:44:51.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:44:51.147+0000 7fd986ffd640 1 -- 172.21.15.72:0/1855106146 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fd964005850 con 0x7fd9901013f0 2024-03-22T01:44:51.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:44:51.148+0000 7fd98cd99640 1 -- 172.21.15.72:0/1855106146 <== mon.0 v2:172.21.15.72:3300/0 7 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fd964005850 con 0x7fd9901013f0 2024-03-22T01:44:51.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:44:51.151+0000 7fd9757fa640 1 -- 172.21.15.72:0/1183142685 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fd968005820 con 0x55682e211d30 2024-03-22T01:44:51.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:44:51.152+0000 7fd9767fc640 1 -- 172.21.15.72:0/1183142685 <== mon.0 v2:172.21.15.72:3300/0 7 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fd968005820 con 0x55682e211d30 2024-03-22T01:44:52.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:51 smithi089 ceph-mon[35770]: pgmap v249: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s wr, 2.41k op/s 2024-03-22T01:44:52.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:51 smithi072 ceph-mon[32414]: pgmap v249: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s wr, 2.41k op/s 2024-03-22T01:44:52.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:51 smithi098 ceph-mon[36468]: pgmap v249: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s wr, 2.41k op/s 2024-03-22T01:44:53.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:52 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:44:53.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:52 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:44:53.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:52 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:44:53.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:52 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:44:53.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:52 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:44:53.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:52 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:44:54.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:53 smithi089 ceph-mon[35770]: pgmap v250: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s wr, 2.17k op/s 2024-03-22T01:44:54.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:53 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:44:54.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:53 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:44:54.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:53 smithi072 ceph-mon[32414]: pgmap v250: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s wr, 2.17k op/s 2024-03-22T01:44:54.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:53 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:44:54.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:53 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:44:54.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:53 smithi098 ceph-mon[36468]: pgmap v250: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s wr, 2.17k op/s 2024-03-22T01:44:54.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:53 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:44:54.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:53 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:44:55.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:54 smithi089 ceph-mon[35770]: 1 slow requests, 0 included below; oldest blocked for > 353.083577 secs 2024-03-22T01:44:55.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:54 smithi072 ceph-mon[32414]: 1 slow requests, 0 included below; oldest blocked for > 353.083577 secs 2024-03-22T01:44:55.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:54 smithi098 ceph-mon[36468]: 1 slow requests, 0 included below; oldest blocked for > 353.083577 secs 2024-03-22T01:44:56.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:55 smithi089 ceph-mon[35770]: pgmap v251: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s wr, 2.15k op/s 2024-03-22T01:44:56.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:55 smithi089 ceph-mon[35770]: Health check update: 0 slow ops, oldest one blocked for 374 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:44:56.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:55 smithi072 ceph-mon[32414]: pgmap v251: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s wr, 2.15k op/s 2024-03-22T01:44:56.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:55 smithi072 ceph-mon[32414]: Health check update: 0 slow ops, oldest one blocked for 374 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:44:56.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:55 smithi098 ceph-mon[36468]: pgmap v251: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s wr, 2.15k op/s 2024-03-22T01:44:56.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:55 smithi098 ceph-mon[36468]: Health check update: 0 slow ops, oldest one blocked for 374 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:44:57.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:56 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:44:57.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:56 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:44:57.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:56 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:44:57.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:56 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:44:57.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:56 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:44:57.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:56 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:44:58.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:57 smithi089 ceph-mon[35770]: pgmap v252: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s wr, 2.62k op/s 2024-03-22T01:44:58.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:57 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:44:58.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:57 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:44:58.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:57 smithi072 ceph-mon[32414]: pgmap v252: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s wr, 2.62k op/s 2024-03-22T01:44:58.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:57 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:44:58.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:57 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:44:58.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:57 smithi098 ceph-mon[36468]: pgmap v252: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s wr, 2.62k op/s 2024-03-22T01:44:58.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:57 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:44:58.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:57 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:45:00.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:59 smithi089 ceph-mon[35770]: pgmap v253: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s wr, 1.75k op/s 2024-03-22T01:45:00.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:44:59 smithi089 ceph-mon[35770]: 1 slow requests, 0 included below; oldest blocked for > 358.083636 secs 2024-03-22T01:45:00.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:59 smithi072 ceph-mon[32414]: pgmap v253: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s wr, 1.75k op/s 2024-03-22T01:45:00.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:44:59 smithi072 ceph-mon[32414]: 1 slow requests, 0 included below; oldest blocked for > 358.083636 secs 2024-03-22T01:45:00.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:59 smithi098 ceph-mon[36468]: pgmap v253: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s wr, 1.75k op/s 2024-03-22T01:45:00.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:44:59 smithi098 ceph-mon[36468]: 1 slow requests, 0 included below; oldest blocked for > 358.083636 secs 2024-03-22T01:45:01.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:00 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:45:01.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:00 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:45:01.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:00 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:45:01.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:00 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:45:01.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:00 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:45:01.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:00 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:45:02.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:01 smithi089 ceph-mon[35770]: pgmap v254: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s wr, 2.15k op/s 2024-03-22T01:45:02.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:01 smithi089 ceph-mon[35770]: Health check update: 0 slow ops, oldest one blocked for 381 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:45:02.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:01 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:45:02.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:01 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:45:02.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:01 smithi072 ceph-mon[32414]: pgmap v254: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s wr, 2.15k op/s 2024-03-22T01:45:02.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:01 smithi072 ceph-mon[32414]: Health check update: 0 slow ops, oldest one blocked for 381 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:45:02.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:01 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:45:02.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:01 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:45:02.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:01 smithi098 ceph-mon[36468]: pgmap v254: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s wr, 2.15k op/s 2024-03-22T01:45:02.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:01 smithi098 ceph-mon[36468]: Health check update: 0 slow ops, oldest one blocked for 381 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:45:02.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:01 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:45:02.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:01 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:45:04.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:03 smithi089 ceph-mon[35770]: pgmap v255: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s wr, 2.26k op/s 2024-03-22T01:45:04.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:03 smithi072 ceph-mon[32414]: pgmap v255: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s wr, 2.26k op/s 2024-03-22T01:45:04.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:03 smithi098 ceph-mon[36468]: pgmap v255: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s wr, 2.26k op/s 2024-03-22T01:45:05.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:04 smithi089 ceph-mon[35770]: 1 slow requests, 1 included below; oldest blocked for > 363.083741 secs 2024-03-22T01:45:05.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:04 smithi089 ceph-mon[35770]: slow request 363.083741 seconds old, received at 2024-03-22T01:39:00.828862+0000: client_request(client.25502:16 mksnap #0x1//scheduled-2024-03-22-01_39_00_UTC 2024-03-22T01:39:00.828832+0000 caller_uid=0, caller_gid=0{}) currently acquired locks 2024-03-22T01:45:05.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:04 smithi072 ceph-mon[32414]: 1 slow requests, 1 included below; oldest blocked for > 363.083741 secs 2024-03-22T01:45:05.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:04 smithi072 ceph-mon[32414]: slow request 363.083741 seconds old, received at 2024-03-22T01:39:00.828862+0000: client_request(client.25502:16 mksnap #0x1//scheduled-2024-03-22-01_39_00_UTC 2024-03-22T01:39:00.828832+0000 caller_uid=0, caller_gid=0{}) currently acquired locks 2024-03-22T01:45:05.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:04 smithi098 ceph-mon[36468]: 1 slow requests, 1 included below; oldest blocked for > 363.083741 secs 2024-03-22T01:45:05.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:04 smithi098 ceph-mon[36468]: slow request 363.083741 seconds old, received at 2024-03-22T01:39:00.828862+0000: client_request(client.25502:16 mksnap #0x1//scheduled-2024-03-22-01_39_00_UTC 2024-03-22T01:39:00.828832+0000 caller_uid=0, caller_gid=0{}) currently acquired locks 2024-03-22T01:45:06.109 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:05 smithi089 ceph-mon[35770]: pgmap v256: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s wr, 2.14k op/s 2024-03-22T01:45:06.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:05 smithi072 ceph-mon[32414]: pgmap v256: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s wr, 2.14k op/s 2024-03-22T01:45:06.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:05 smithi098 ceph-mon[36468]: pgmap v256: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s wr, 2.14k op/s 2024-03-22T01:45:07.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:06 smithi098 ceph-mon[36468]: Health check update: 0 slow ops, oldest one blocked for 388 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:45:07.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:06 smithi089 ceph-mon[35770]: Health check update: 0 slow ops, oldest one blocked for 388 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:45:07.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:06 smithi072 ceph-mon[32414]: Health check update: 0 slow ops, oldest one blocked for 388 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:45:08.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:07 smithi098 ceph-mon[36468]: pgmap v257: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s wr, 2.34k op/s 2024-03-22T01:45:08.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:07 smithi089 ceph-mon[35770]: pgmap v257: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s wr, 2.34k op/s 2024-03-22T01:45:08.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:07 smithi072 ceph-mon[32414]: pgmap v257: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s wr, 2.34k op/s 2024-03-22T01:45:09.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:08 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:45:09.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:08 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:45:09.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:08 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:45:09.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:08 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:45:09.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:08 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:45:09.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:08 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:45:10.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:09 smithi098 ceph-mon[36468]: pgmap v258: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s wr, 1.70k op/s 2024-03-22T01:45:10.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:09 smithi098 ceph-mon[36468]: 1 slow requests, 0 included below; oldest blocked for > 368.083806 secs 2024-03-22T01:45:10.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:09 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:45:10.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:09 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:45:10.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:09 smithi089 ceph-mon[35770]: pgmap v258: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s wr, 1.70k op/s 2024-03-22T01:45:10.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:09 smithi089 ceph-mon[35770]: 1 slow requests, 0 included below; oldest blocked for > 368.083806 secs 2024-03-22T01:45:10.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:09 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:45:10.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:09 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:45:10.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:09 smithi072 ceph-mon[32414]: pgmap v258: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s wr, 1.70k op/s 2024-03-22T01:45:10.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:09 smithi072 ceph-mon[32414]: 1 slow requests, 0 included below; oldest blocked for > 368.083806 secs 2024-03-22T01:45:10.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:09 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:45:10.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:09 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:45:12.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:11 smithi089 ceph-mon[35770]: pgmap v259: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s wr, 1.98k op/s 2024-03-22T01:45:12.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:11 smithi072 ceph-mon[32414]: pgmap v259: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s wr, 1.98k op/s 2024-03-22T01:45:12.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:11 smithi098 ceph-mon[36468]: pgmap v259: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s wr, 1.98k op/s 2024-03-22T01:45:13.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:12 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:45:13.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:12 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:45:13.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:12 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:45:13.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:12 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:45:13.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:12 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:45:13.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:12 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:45:14.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:13 smithi089 ceph-mon[35770]: pgmap v260: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s wr, 1.51k op/s 2024-03-22T01:45:14.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:13 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:45:14.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:13 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:45:14.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:13 smithi089 ceph-mon[35770]: Health check update: 0 slow ops, oldest one blocked for 393 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:45:14.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:13 smithi072 ceph-mon[32414]: pgmap v260: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s wr, 1.51k op/s 2024-03-22T01:45:14.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:13 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:45:14.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:13 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:45:14.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:13 smithi072 ceph-mon[32414]: Health check update: 0 slow ops, oldest one blocked for 393 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:45:14.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:13 smithi098 ceph-mon[36468]: pgmap v260: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s wr, 1.51k op/s 2024-03-22T01:45:14.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:13 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:45:14.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:13 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:45:14.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:13 smithi098 ceph-mon[36468]: Health check update: 0 slow ops, oldest one blocked for 393 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:45:15.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:14 smithi089 ceph-mon[35770]: 1 slow requests, 0 included below; oldest blocked for > 373.083867 secs 2024-03-22T01:45:15.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:14 smithi072 ceph-mon[32414]: 1 slow requests, 0 included below; oldest blocked for > 373.083867 secs 2024-03-22T01:45:15.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:14 smithi098 ceph-mon[36468]: 1 slow requests, 0 included below; oldest blocked for > 373.083867 secs 2024-03-22T01:45:16.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:15 smithi089 ceph-mon[35770]: pgmap v261: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s wr, 1.30k op/s 2024-03-22T01:45:16.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:15 smithi072 ceph-mon[32414]: pgmap v261: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s wr, 1.30k op/s 2024-03-22T01:45:16.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:15 smithi098 ceph-mon[36468]: pgmap v261: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s wr, 1.30k op/s 2024-03-22T01:45:17.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:16 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:45:17.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:16 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:45:17.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:16 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:45:17.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:16 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:45:17.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:16 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:45:17.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:16 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:45:18.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:17 smithi089 ceph-mon[35770]: pgmap v262: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s wr, 1.91k op/s 2024-03-22T01:45:18.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:17 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:45:18.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:17 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:45:18.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:17 smithi072 ceph-mon[32414]: pgmap v262: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s wr, 1.91k op/s 2024-03-22T01:45:18.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:17 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:45:18.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:17 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:45:18.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:17 smithi098 ceph-mon[36468]: pgmap v262: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s wr, 1.91k op/s 2024-03-22T01:45:18.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:17 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:45:18.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:17 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:45:19.273 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:18 smithi072 ceph-mon[32414]: Health check update: 0 slow ops, oldest one blocked for 398 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:45:19.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:18 smithi089 ceph-mon[35770]: Health check update: 0 slow ops, oldest one blocked for 398 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:45:19.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:18 smithi098 ceph-mon[36468]: Health check update: 0 slow ops, oldest one blocked for 398 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-22T01:45:20.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:19 smithi089 ceph-mon[35770]: pgmap v263: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s wr, 1.32k op/s 2024-03-22T01:45:20.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:19 smithi089 ceph-mon[35770]: 1 slow requests, 0 included below; oldest blocked for > 378.083954 secs 2024-03-22T01:45:20.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:19 smithi072 ceph-mon[32414]: pgmap v263: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s wr, 1.32k op/s 2024-03-22T01:45:20.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:19 smithi072 ceph-mon[32414]: 1 slow requests, 0 included below; oldest blocked for > 378.083954 secs 2024-03-22T01:45:20.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:19 smithi098 ceph-mon[36468]: pgmap v263: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s wr, 1.32k op/s 2024-03-22T01:45:20.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:19 smithi098 ceph-mon[36468]: 1 slow requests, 0 included below; oldest blocked for > 378.083954 secs 2024-03-22T01:45:21.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:20 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:45:21.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:20 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:45:21.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:20 smithi089 ceph-mon[35770]: MDS health message cleared (mds.?): 1 slow metadata IOs are blocked > 180 secs, oldest blocked for 400 secs 2024-03-22T01:45:21.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:20 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:45:21.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:20 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:45:21.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:20 smithi072 ceph-mon[32414]: MDS health message cleared (mds.?): 1 slow metadata IOs are blocked > 180 secs, oldest blocked for 400 secs 2024-03-22T01:45:21.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:20 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:45:21.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:20 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:45:21.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:20 smithi098 ceph-mon[36468]: MDS health message cleared (mds.?): 1 slow metadata IOs are blocked > 180 secs, oldest blocked for 400 secs 2024-03-22T01:45:22.006 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:22.007+0000 7f3e1cff9640 1 -- 172.21.15.72:0/2014627542 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55af9ccc48d0 con 0x55af9cd72ab0 2024-03-22T01:45:22.006 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:22.007+0000 7f3e1cff9640 10 client.16023 handle_command_reply: tid=0 2024-03-22T01:45:22.006 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:22.007+0000 7f3e3e38d740 1 client.16023 shutdown 2024-03-22T01:45:22.007 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout:{ 2024-03-22T01:45:22.007 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout: "message": "", 2024-03-22T01:45:22.007 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout: "return_code": 0 2024-03-22T01:45:22.007 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout:} 2024-03-22T01:45:22.008 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:22.007+0000 7f3e3e38d740 1 -- 172.21.15.72:0/2014627542 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55af9cd592d0 msgr2=0x55af9cd6a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:45:22.008 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:22.007+0000 7f3e3e38d740 1 --2- 172.21.15.72:0/2014627542 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55af9cd592d0 0x55af9cd6a9c0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f3e30005bb0 tx=0x7f3e30047340 comp rx=0 tx=0).stop 2024-03-22T01:45:22.008 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:22.007+0000 7f3e3e38d740 1 -- 172.21.15.72:0/2014627542 shutdown_connections 2024-03-22T01:45:22.008 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:22.007+0000 7f3e3e38d740 1 --2- 172.21.15.72:0/2014627542 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55af9cd72ab0 0x55af9cddad00 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:22.008 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:22.007+0000 7f3e3e38d740 1 --2- 172.21.15.72:0/2014627542 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55af9cd629c0 0x55af9cd5bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:22.008 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:22.007+0000 7f3e3e38d740 1 --2- 172.21.15.72:0/2014627542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af9cd59bf0 0x55af9cd5b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:22.008 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:22.007+0000 7f3e3e38d740 1 --2- 172.21.15.72:0/2014627542 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55af9cd592d0 0x55af9cd6a9c0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:22.008 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:22.007+0000 7f3e3e38d740 1 -- 172.21.15.72:0/2014627542 >> 172.21.15.72:0/2014627542 conn(0x55af9cd53bf0 msgr2=0x55af9cd54ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:45:22.008 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:22.007+0000 7f3e1cff9640 0 client.16023 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:45:22.008 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:22.007+0000 7f3e3e38d740 1 -- 172.21.15.72:0/2014627542 shutdown_connections 2024-03-22T01:45:22.008 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:22.007+0000 7f3e3e38d740 1 -- 172.21.15.72:0/2014627542 wait complete. 2024-03-22T01:45:22.008 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:22.007+0000 7f3e3e38d740 20 client.16023 trim_cache size 0 max 16384 2024-03-22T01:45:22.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.008+0000 7fd9767fc640 1 -- 172.21.15.72:0/1183142685 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fd980004ba0 con 0x55682e22ab80 2024-03-22T01:45:22.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.008+0000 7fd9767fc640 10 client.16035 handle_command_reply: tid=0 2024-03-22T01:45:22.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.008+0000 7fd997db6740 1 client.16035 shutdown 2024-03-22T01:45:22.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.008+0000 7fd997db6740 1 -- 172.21.15.72:0/1183142685 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55682e211d30 msgr2=0x55682e217e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:45:22.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.008+0000 7fd997db6740 1 --2- 172.21.15.72:0/1183142685 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55682e211d30 0x55682e217e20 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fd97809bfa0 tx=0x7fd9780a00b0 comp rx=0 tx=0).stop 2024-03-22T01:45:22.011 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:22.008+0000 7f3e3b7ac640 1 -- 172.21.15.72:0/1386165290 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f3e200a3180 msgr2=0x7f3e200a5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:45:22.011 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:22.008+0000 7f3e3b7ac640 1 --2- 172.21.15.72:0/1386165290 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f3e200a3180 0x7f3e200a5640 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c01f000 tx=0x7f3e2c01b000 comp rx=0 tx=0).stop 2024-03-22T01:45:22.011 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:22.008+0000 7f3e3b7ac640 1 -- 172.21.15.72:0/1386165290 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e34069e10 msgr2=0x7f3e3410c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:45:22.012 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:22.009+0000 7f3e3b7ac640 1 --2- 172.21.15.72:0/1386165290 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e34069e10 0x7f3e3410c090 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f3e2400d7a0 tx=0x7f3e2400dc60 comp rx=0 tx=0).stop 2024-03-22T01:45:22.012 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:22.009+0000 7f3e3b7ac640 1 -- 172.21.15.72:0/1386165290 shutdown_connections 2024-03-22T01:45:22.012 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:22.009+0000 7f3e3b7ac640 1 --2- 172.21.15.72:0/1386165290 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f3e200a3180 0x7f3e200a5640 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:22.012 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:22.009+0000 7f3e3b7ac640 1 --2- 172.21.15.72:0/1386165290 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e34107c10 0x7f3e3410c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:22.012 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:22.009+0000 7f3e3b7ac640 1 --2- 172.21.15.72:0/1386165290 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e34069e10 0x7f3e3410c090 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:22.012 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:22.009+0000 7f3e3b7ac640 1 --2- 172.21.15.72:0/1386165290 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e34069450 0x7f3e3410bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:22.012 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:22.009+0000 7f3e3b7ac640 1 -- 172.21.15.72:0/1386165290 >> 172.21.15.72:0/1386165290 conn(0x7f3e34077b80 msgr2=0x7f3e340ff4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:45:22.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.008+0000 7fd997db6740 1 -- 172.21.15.72:0/1183142685 shutdown_connections 2024-03-22T01:45:22.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.009+0000 7fd997db6740 1 --2- 172.21.15.72:0/1183142685 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55682e22ab80 0x55682e22cf70 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:22.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.009+0000 7fd997db6740 1 --2- 172.21.15.72:0/1183142685 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55682e21a910 0x55682e218360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:22.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.009+0000 7fd997db6740 1 --2- 172.21.15.72:0/1183142685 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55682e211d30 0x55682e217e20 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:22.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.009+0000 7fd997db6740 1 --2- 172.21.15.72:0/1183142685 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55682e211410 0x55682e2178e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:22.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.009+0000 7fd997db6740 1 -- 172.21.15.72:0/1183142685 >> 172.21.15.72:0/1183142685 conn(0x55682e20bb40 msgr2=0x55682e20d890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:45:22.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.009+0000 7fd9767fc640 0 client.16035 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:45:22.014 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:22.009+0000 7f3e3b7ac640 1 -- 172.21.15.72:0/1386165290 shutdown_connections 2024-03-22T01:45:22.014 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:22.009+0000 7f3e3b7ac640 1 -- 172.21.15.72:0/1386165290 wait complete. 2024-03-22T01:45:22.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.009+0000 7fd997db6740 1 -- 172.21.15.72:0/1183142685 shutdown_connections 2024-03-22T01:45:22.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.009+0000 7fd997db6740 1 -- 172.21.15.72:0/1183142685 wait complete. 2024-03-22T01:45:22.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.009+0000 7fd997db6740 20 client.16035 trim_cache size 0 max 16384 2024-03-22T01:45:22.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.029+0000 7fd997db6740 1 Processor -- start 2024-03-22T01:45:22.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.029+0000 7fd997db6740 1 -- start start 2024-03-22T01:45:22.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.030+0000 7fd997db6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55682e211410 0x55682e21a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:22.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.030+0000 7fd997db6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55682e211d30 0x55682e2264a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:22.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.030+0000 7fd997db6740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55682e21a910 0x55682e232820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:22.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.030+0000 7fd997db6740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fd9901af440 con 0x55682e21a910 2024-03-22T01:45:22.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.030+0000 7fd997db6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd9901b1cc0 con 0x55682e211d30 2024-03-22T01:45:22.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.030+0000 7fd997db6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd9901afab0 con 0x55682e211410 2024-03-22T01:45:22.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.030+0000 7fd98f59e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55682e21a910 0x55682e232820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:22.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.030+0000 7fd98f59e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55682e21a910 0x55682e232820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46722/0 (socket says 172.21.15.72:46722) 2024-03-22T01:45:22.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.030+0000 7fd98f59e640 1 -- 172.21.15.72:0/4180112115 learned_addr learned my addr 172.21.15.72:0/4180112115 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:45:22.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.030+0000 7fd98ed9d640 1 --2- 172.21.15.72:0/4180112115 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55682e211410 0x55682e21a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:22.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.030+0000 7fd987fff640 1 --2- 172.21.15.72:0/4180112115 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55682e211d30 0x55682e2264a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:22.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.030+0000 7fd98f59e640 1 -- 172.21.15.72:0/4180112115 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55682e211410 msgr2=0x55682e21a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:45:22.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.030+0000 7fd98f59e640 1 --2- 172.21.15.72:0/4180112115 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55682e211410 0x55682e21a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:22.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.030+0000 7fd98f59e640 1 -- 172.21.15.72:0/4180112115 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55682e211d30 msgr2=0x55682e2264a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:45:22.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.030+0000 7fd98f59e640 1 --2- 172.21.15.72:0/4180112115 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55682e211d30 0x55682e2264a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:22.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.031+0000 7fd98f59e640 1 -- 172.21.15.72:0/4180112115 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55682de490f0 con 0x55682e21a910 2024-03-22T01:45:22.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.031+0000 7fd98f59e640 1 --2- 172.21.15.72:0/4180112115 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55682e21a910 0x55682e232820 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fd988010ba0 tx=0x7fd988007470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=558ac0a03aaf2e3d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:45:22.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.031+0000 7fd984ff9640 1 -- 172.21.15.72:0/4180112115 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd988020070 con 0x55682e21a910 2024-03-22T01:45:22.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.031+0000 7fd984ff9640 1 -- 172.21.15.72:0/4180112115 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd988003b70 con 0x55682e21a910 2024-03-22T01:45:22.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.031+0000 7fd984ff9640 1 -- 172.21.15.72:0/4180112115 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd98801b700 con 0x55682e21a910 2024-03-22T01:45:22.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.032+0000 7fd997db6740 1 -- 172.21.15.72:0/4180112115 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55682e21a910 msgr2=0x55682e232820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:45:22.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.032+0000 7fd997db6740 1 --2- 172.21.15.72:0/4180112115 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55682e21a910 0x55682e232820 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fd988010ba0 tx=0x7fd988007470 comp rx=0 tx=0).stop 2024-03-22T01:45:22.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.032+0000 7fd997db6740 1 -- 172.21.15.72:0/4180112115 shutdown_connections 2024-03-22T01:45:22.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.032+0000 7fd997db6740 1 --2- 172.21.15.72:0/4180112115 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55682e211d30 0x55682e2264a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:22.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.032+0000 7fd997db6740 1 --2- 172.21.15.72:0/4180112115 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55682e21a910 0x55682e232820 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:22.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.032+0000 7fd997db6740 1 --2- 172.21.15.72:0/4180112115 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55682e211410 0x55682e21a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:22.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.032+0000 7fd997db6740 1 -- 172.21.15.72:0/4180112115 >> 172.21.15.72:0/4180112115 conn(0x55682e20bb40 msgr2=0x55682e20a370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:45:22.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.032+0000 7fd997db6740 1 -- 172.21.15.72:0/4180112115 shutdown_connections 2024-03-22T01:45:22.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.032+0000 7fd997db6740 1 -- 172.21.15.72:0/4180112115 wait complete. 2024-03-22T01:45:22.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.032+0000 7fd997db6740 1 -- start start 2024-03-22T01:45:22.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.032+0000 7fd997db6740 1 Processor -- start 2024-03-22T01:45:22.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.033+0000 7fd997db6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55682e211410 0x55682e2226c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:22.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.033+0000 7fd997db6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55682e211d30 0x55682e222c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:22.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.033+0000 7fd997db6740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55682e21a910 0x55682e21db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:22.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.033+0000 7fd997db6740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fd9901b0180 con 0x55682e21a910 2024-03-22T01:45:22.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.033+0000 7fd997db6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd9901b15f0 con 0x55682e211410 2024-03-22T01:45:22.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.033+0000 7fd997db6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd9901b0850 con 0x55682e211d30 2024-03-22T01:45:22.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.033+0000 7fd98ed9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55682e211410 0x55682e2226c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:22.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.033+0000 7fd987fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55682e211d30 0x55682e222c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:22.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.033+0000 7fd98ed9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55682e211410 0x55682e2226c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.72:39960/0 (socket says 172.21.15.72:39960) 2024-03-22T01:45:22.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.033+0000 7fd98ed9d640 1 -- 172.21.15.72:0/3003627225 learned_addr learned my addr 172.21.15.72:0/3003627225 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:45:22.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.033+0000 7fd997db6740 10 client.0 _ll_register_callbacks cb 0x7fd97809ba40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:45:22.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.033+0000 7fd98f59e640 1 --2- 172.21.15.72:0/3003627225 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55682e21a910 0x55682e21db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:22.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.033+0000 7fd98ed9d640 1 -- 172.21.15.72:0/3003627225 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55682e211d30 msgr2=0x55682e222c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:45:22.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.033+0000 7fd98ed9d640 1 --2- 172.21.15.72:0/3003627225 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55682e211d30 0x55682e222c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:22.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.033+0000 7fd98ed9d640 1 -- 172.21.15.72:0/3003627225 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55682e21a910 msgr2=0x55682e21db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:45:22.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.033+0000 7fd98ed9d640 1 --2- 172.21.15.72:0/3003627225 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55682e21a910 0x55682e21db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:22.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.033+0000 7fd98ed9d640 1 -- 172.21.15.72:0/3003627225 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55682e21e2c0 con 0x55682e211410 2024-03-22T01:45:22.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.033+0000 7fd98ed9d640 1 --2- 172.21.15.72:0/3003627225 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55682e211410 0x55682e2226c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd980009430 tx=0x7fd9800053f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=903ab5cf3c03da52 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:45:22.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.033+0000 7fd9767fc640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-03-22T01:45:22.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.034+0000 7fd9767fc640 1 -- 172.21.15.72:0/3003627225 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd980050040 con 0x55682e211410 2024-03-22T01:45:22.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.034+0000 7fd997db6740 1 -- 172.21.15.72:0/3003627225 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd988003b70 con 0x55682e211410 2024-03-22T01:45:22.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.034+0000 7fd9767fc640 1 -- 172.21.15.72:0/3003627225 <== mon.2 v2:172.21.15.89:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fd98002ca70 con 0x55682e211410 2024-03-22T01:45:22.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.034+0000 7fd9767fc640 1 -- 172.21.15.72:0/3003627225 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=102) ==== 24+0+0 (secure 0 0 0) 0x7fd980068050 con 0x55682e211410 2024-03-22T01:45:22.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.034+0000 7fd997db6740 10 client.25877 fetch_fsmap learned FSMap version 102 2024-03-22T01:45:22.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.034+0000 7fd997db6740 1 -- 172.21.15.72:0/3003627225 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=102,monmap=4+}) -- 0x55682e290880 con 0x55682e211410 2024-03-22T01:45:22.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.034+0000 7fd9767fc640 1 -- 172.21.15.72:0/3003627225 <== mon.2 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd988003b70 con 0x55682e211410 2024-03-22T01:45:22.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.034+0000 7fd9767fc640 1 -- 172.21.15.72:0/3003627225 <== mon.2 v2:172.21.15.89:3300/0 5 ==== fsmap(e 102) ==== 8330+0+0 (secure 0 0 0) 0x7fd98005a540 con 0x55682e211410 2024-03-22T01:45:22.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.034+0000 7fd997db6740 10 client.25877 fetch_fsmap finished waiting for FSMap version 102 2024-03-22T01:45:22.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.034+0000 7fd997db6740 10 client.25877 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T01:45:22.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.034+0000 7fd997db6740 20 client.25877 populate_metadata read hostname 'smithi072' 2024-03-22T01:45:22.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.034+0000 7fd997db6740 1 --2- 172.21.15.72:0/3003627225 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55682e290c10 0x55682e293000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:22.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.034+0000 7fd997db6740 4 client.25877 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T01:45:22.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.034+0000 7fd997db6740 1 -- 172.21.15.72:0/3003627225 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55682e182e00 con 0x55682e290c10 2024-03-22T01:45:22.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.035+0000 7fd987fff640 1 --2- 172.21.15.72:0/3003627225 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55682e290c10 0x55682e293000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:22.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.035+0000 7fd9767fc640 10 client.25877 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:45:22.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.035+0000 7fd987fff640 1 --2- 172.21.15.72:0/3003627225 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55682e290c10 0x55682e293000 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b77cbaaedd984cfd server_cookie=e114b36427cbbfeb in_seq=0 out_seq=0 2024-03-22T01:45:22.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.035+0000 7fd9767fc640 1 -- 172.21.15.72:0/3003627225 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55682e182e00 con 0x55682e290c10 2024-03-22T01:45:22.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.035+0000 7fd9767fc640 10 client.25877 handle_command_reply: tid=0 2024-03-22T01:45:22.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.035+0000 7fd997db6740 1 client.25877 shutdown 2024-03-22T01:45:22.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.035+0000 7fd997db6740 1 -- 172.21.15.72:0/3003627225 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55682e211410 msgr2=0x55682e2226c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:45:22.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.035+0000 7fd997db6740 1 --2- 172.21.15.72:0/3003627225 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55682e211410 0x55682e2226c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd980009430 tx=0x7fd9800053f0 comp rx=0 tx=0).stop 2024-03-22T01:45:22.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.035+0000 7fd997db6740 1 -- 172.21.15.72:0/3003627225 shutdown_connections 2024-03-22T01:45:22.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.035+0000 7fd997db6740 1 --2- 172.21.15.72:0/3003627225 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55682e290c10 0x55682e293000 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:22.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.035+0000 7fd997db6740 1 --2- 172.21.15.72:0/3003627225 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55682e21a910 0x55682e21db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:22.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.035+0000 7fd997db6740 1 --2- 172.21.15.72:0/3003627225 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55682e211d30 0x55682e222c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:22.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.035+0000 7fd997db6740 1 --2- 172.21.15.72:0/3003627225 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55682e211410 0x55682e2226c0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:22.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.035+0000 7fd9767fc640 0 client.25877 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:45:22.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.035+0000 7fd997db6740 1 -- 172.21.15.72:0/3003627225 >> 172.21.15.72:0/3003627225 conn(0x55682e20bb40 msgr2=0x55682e218130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:45:22.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.035+0000 7fd997db6740 1 -- 172.21.15.72:0/3003627225 shutdown_connections 2024-03-22T01:45:22.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.035+0000 7fd997db6740 1 -- 172.21.15.72:0/3003627225 wait complete. 2024-03-22T01:45:22.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.036+0000 7fd997db6740 20 client.25877 trim_cache size 0 max 16384 2024-03-22T01:45:22.041 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-22T01:45:22.042 INFO:teuthology.orchestra.run.smithi072.stdout: "status": "no active scrubs running", 2024-03-22T01:45:22.042 INFO:teuthology.orchestra.run.smithi072.stdout: "scrubs": {} 2024-03-22T01:45:22.042 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-22T01:45:22.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.036+0000 7fd9950fa640 1 -- 172.21.15.72:0/1855106146 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fd97c0a3090 msgr2=0x7fd97c0a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:45:22.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.036+0000 7fd9950fa640 1 --2- 172.21.15.72:0/1855106146 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fd97c0a3090 0x7fd97c0a5550 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd980000c00 tx=0x7fd98002c6f0 comp rx=0 tx=0).stop 2024-03-22T01:45:22.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.036+0000 7fd9950fa640 1 -- 172.21.15.72:0/1855106146 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9901013f0 msgr2=0x7fd9900fdf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:45:22.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.036+0000 7fd9950fa640 1 --2- 172.21.15.72:0/1855106146 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9901013f0 0x7fd9900fdf20 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fd97800e9f0 tx=0x7fd97800eeb0 comp rx=0 tx=0).stop 2024-03-22T01:45:22.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.036+0000 7fd9950fa640 1 -- 172.21.15.72:0/1855106146 shutdown_connections 2024-03-22T01:45:22.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.036+0000 7fd9950fa640 1 --2- 172.21.15.72:0/1855106146 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fd97c0a3090 0x7fd97c0a5550 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:22.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.036+0000 7fd9950fa640 1 --2- 172.21.15.72:0/1855106146 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd990106a40 0x7fd9900fe460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:22.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.036+0000 7fd9950fa640 1 --2- 172.21.15.72:0/1855106146 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9901013f0 0x7fd9900fdf20 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:22.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.036+0000 7fd9950fa640 1 --2- 172.21.15.72:0/1855106146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd990100a30 0x7fd9900fd9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:22.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.036+0000 7fd9950fa640 1 -- 172.21.15.72:0/1855106146 >> 172.21.15.72:0/1855106146 conn(0x7fd990077c40 msgr2=0x7fd990069140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:45:22.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.036+0000 7fd9950fa640 1 -- 172.21.15.72:0/1855106146 shutdown_connections 2024-03-22T01:45:22.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:22.036+0000 7fd9950fa640 1 -- 172.21.15.72:0/1855106146 wait complete. 2024-03-22T01:45:22.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:21 smithi089 ceph-mon[35770]: pgmap v264: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 8.6 MiB/s wr, 1.81k op/s 2024-03-22T01:45:22.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:21 smithi089 ceph-mon[35770]: Health check cleared: SLOW_OPS (was: 0 slow ops, oldest one blocked for 398 sec, osd.9 has slow ops) 2024-03-22T01:45:22.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:21 smithi089 ceph-mon[35770]: Health check update: 1 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2024-03-22T01:45:22.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:21 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:45:22.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:21 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:45:22.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:21 smithi072 ceph-mon[32414]: pgmap v264: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 8.6 MiB/s wr, 1.81k op/s 2024-03-22T01:45:22.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:21 smithi072 ceph-mon[32414]: Health check cleared: SLOW_OPS (was: 0 slow ops, oldest one blocked for 398 sec, osd.9 has slow ops) 2024-03-22T01:45:22.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:21 smithi072 ceph-mon[32414]: Health check update: 1 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2024-03-22T01:45:22.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:21 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:45:22.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:21 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:45:22.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:21 smithi098 ceph-mon[36468]: pgmap v264: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 8.6 MiB/s wr, 1.81k op/s 2024-03-22T01:45:22.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:21 smithi098 ceph-mon[36468]: Health check cleared: SLOW_OPS (was: 0 slow ops, oldest one blocked for 398 sec, osd.9 has slow ops) 2024-03-22T01:45:22.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:21 smithi098 ceph-mon[36468]: Health check update: 1 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2024-03-22T01:45:22.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:21 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.98:6838/3192173650,v1:172.21.15.98:6839/3192173650] up:standby-replay 2024-03-22T01:45:22.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:21 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:45:23.651 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-22T01:45:23.651 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T01:45:24.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:23 smithi089 ceph-mon[35770]: pgmap v265: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 954 KiB/s rd, 6.3 MiB/s wr, 1.32k op/s 2024-03-22T01:45:24.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:23 smithi072 ceph-mon[32414]: pgmap v265: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 954 KiB/s rd, 6.3 MiB/s wr, 1.32k op/s 2024-03-22T01:45:24.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:23 smithi098 ceph-mon[36468]: pgmap v265: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 954 KiB/s rd, 6.3 MiB/s wr, 1.32k op/s 2024-03-22T01:45:26.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:25 smithi089 ceph-mon[35770]: pgmap v266: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 954 KiB/s rd, 6.4 MiB/s wr, 1.34k op/s 2024-03-22T01:45:26.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:25 smithi072 ceph-mon[32414]: pgmap v266: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 954 KiB/s rd, 6.4 MiB/s wr, 1.34k op/s 2024-03-22T01:45:26.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:25 smithi098 ceph-mon[36468]: pgmap v266: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 954 KiB/s rd, 6.4 MiB/s wr, 1.34k op/s 2024-03-22T01:45:28.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:27 smithi089 ceph-mon[35770]: pgmap v267: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.4 MiB/s wr, 1.38k op/s 2024-03-22T01:45:28.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:27 smithi089 ceph-mon[35770]: MDS health message cleared (mds.?): 1 slow requests are blocked > 180 secs 2024-03-22T01:45:28.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:27 smithi089 ceph-mon[35770]: MDS health message cleared (mds.?): 100+ slow metadata IOs are blocked > 180 secs, oldest blocked for 402 secs 2024-03-22T01:45:28.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:27 smithi072 ceph-mon[32414]: pgmap v267: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.4 MiB/s wr, 1.38k op/s 2024-03-22T01:45:28.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:27 smithi072 ceph-mon[32414]: MDS health message cleared (mds.?): 1 slow requests are blocked > 180 secs 2024-03-22T01:45:28.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:27 smithi072 ceph-mon[32414]: MDS health message cleared (mds.?): 100+ slow metadata IOs are blocked > 180 secs, oldest blocked for 402 secs 2024-03-22T01:45:28.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:27 smithi098 ceph-mon[36468]: pgmap v267: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.4 MiB/s wr, 1.38k op/s 2024-03-22T01:45:28.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:27 smithi098 ceph-mon[36468]: MDS health message cleared (mds.?): 1 slow requests are blocked > 180 secs 2024-03-22T01:45:28.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:27 smithi098 ceph-mon[36468]: MDS health message cleared (mds.?): 100+ slow metadata IOs are blocked > 180 secs, oldest blocked for 402 secs 2024-03-22T01:45:28.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:28.951+0000 7f5d6bf8a640 1 -- 172.21.15.72:0/2305954233 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5d64101290 msgr2=0x7f5d6410df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:45:28.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:28.951+0000 7f5d6bf8a640 1 --2- 172.21.15.72:0/2305954233 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5d64101290 0x7f5d6410df50 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f5d54009900 tx=0x7f5d5402bc30 comp rx=0 tx=0).stop 2024-03-22T01:45:28.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:28.951+0000 7f5d6bf8a640 1 -- 172.21.15.72:0/2305954233 shutdown_connections 2024-03-22T01:45:28.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:28.951+0000 7f5d6bf8a640 1 --2- 172.21.15.72:0/2305954233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d6410e490 0x7f5d64110880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:28.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:28.951+0000 7f5d6bf8a640 1 --2- 172.21.15.72:0/2305954233 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5d64101290 0x7f5d6410df50 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:28.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:28.951+0000 7f5d6bf8a640 1 --2- 172.21.15.72:0/2305954233 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d641008e0 0x7f5d64100cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:28.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:28.951+0000 7f5d6bf8a640 1 -- 172.21.15.72:0/2305954233 >> 172.21.15.72:0/2305954233 conn(0x7f5d640fc520 msgr2=0x7f5d640fe940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:45:28.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:28.951+0000 7f5d6bf8a640 1 -- 172.21.15.72:0/2305954233 shutdown_connections 2024-03-22T01:45:28.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:28.951+0000 7f5d6bf8a640 1 -- 172.21.15.72:0/2305954233 wait complete. 2024-03-22T01:45:28.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:28.952+0000 7f5d6bf8a640 1 Processor -- start 2024-03-22T01:45:28.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:28.952+0000 7f5d6bf8a640 1 -- start start 2024-03-22T01:45:28.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:28.952+0000 7f5d6bf8a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d641008e0 0x7f5d6419a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:28.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:28.952+0000 7f5d6bf8a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d64101290 0x7f5d6419abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:28.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:28.952+0000 7f5d6bf8a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5d6410e490 0x7f5d6419ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:28.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:28.952+0000 7f5d6bf8a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f5d64114170 con 0x7f5d6410e490 2024-03-22T01:45:28.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:28.952+0000 7f5d6bf8a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5d64113ff0 con 0x7f5d64101290 2024-03-22T01:45:28.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:28.952+0000 7f5d6bf8a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5d641142f0 con 0x7f5d641008e0 2024-03-22T01:45:28.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:28.952+0000 7f5d6a527640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5d6410e490 0x7f5d6419ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:28.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:28.952+0000 7f5d6a527640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5d6410e490 0x7f5d6419ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46734/0 (socket says 172.21.15.72:46734) 2024-03-22T01:45:28.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:28.952+0000 7f5d6a527640 1 -- 172.21.15.72:0/4088225754 learned_addr learned my addr 172.21.15.72:0/4088225754 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:45:28.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:28.952+0000 7f5d69525640 1 --2- 172.21.15.72:0/4088225754 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d64101290 0x7f5d6419abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:28.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:28.952+0000 7f5d69d26640 1 --2- 172.21.15.72:0/4088225754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d641008e0 0x7f5d6419a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:28.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:28.953+0000 7f5d69525640 1 -- 172.21.15.72:0/4088225754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d641008e0 msgr2=0x7f5d6419a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:45:28.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:28.953+0000 7f5d69525640 1 --2- 172.21.15.72:0/4088225754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d641008e0 0x7f5d6419a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:28.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:28.953+0000 7f5d69525640 1 -- 172.21.15.72:0/4088225754 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5d6410e490 msgr2=0x7f5d6419ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:45:28.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:28.953+0000 7f5d69525640 1 --2- 172.21.15.72:0/4088225754 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5d6410e490 0x7f5d6419ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:28.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:28.953+0000 7f5d69525640 1 -- 172.21.15.72:0/4088225754 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d6419f660 con 0x7f5d64101290 2024-03-22T01:45:28.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:28.953+0000 7f5d6a527640 1 --2- 172.21.15.72:0/4088225754 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5d6410e490 0x7f5d6419ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:45:28.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:28.953+0000 7f5d69525640 1 --2- 172.21.15.72:0/4088225754 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d64101290 0x7f5d6419abb0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5d5402c170 tx=0x7f5d54004360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:45:28.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:28.953+0000 7f5d5affd640 1 -- 172.21.15.72:0/4088225754 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d54043020 con 0x7f5d64101290 2024-03-22T01:45:28.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:28.953+0000 7f5d5affd640 1 -- 172.21.15.72:0/4088225754 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d540054e0 con 0x7f5d64101290 2024-03-22T01:45:28.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:28.953+0000 7f5d6bf8a640 1 -- 172.21.15.72:0/4088225754 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d6419f8f0 con 0x7f5d64101290 2024-03-22T01:45:28.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:28.953+0000 7f5d5affd640 1 -- 172.21.15.72:0/4088225754 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d54003b80 con 0x7f5d64101290 2024-03-22T01:45:28.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:28.953+0000 7f5d6bf8a640 1 -- 172.21.15.72:0/4088225754 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d641a71e0 con 0x7f5d64101290 2024-03-22T01:45:28.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:28.955+0000 7f5d5affd640 1 -- 172.21.15.72:0/4088225754 <== mon.2 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 25) ==== 137985+0+0 (secure 0 0 0) 0x7f5d54046020 con 0x7f5d64101290 2024-03-22T01:45:28.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:28.956+0000 7f5d5affd640 1 --2- 172.21.15.72:0/4088225754 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f5d500a3110 0x7f5d500a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:28.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:28.956+0000 7f5d69d26640 1 --2- 172.21.15.72:0/4088225754 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f5d500a3110 0x7f5d500a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:28.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:28.956+0000 7f5d5affd640 1 -- 172.21.15.72:0/4088225754 <== mon.2 v2:172.21.15.89:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f5d540f1a20 con 0x7f5d64101290 2024-03-22T01:45:29.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:28.954+0000 7f5d6bf8a640 1 -- 172.21.15.72:0/4088225754 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d6406ab50 con 0x7f5d64101290 2024-03-22T01:45:29.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.060+0000 7f5d69d26640 1 --2- 172.21.15.72:0/4088225754 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f5d500a3110 0x7f5d500a55d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5d600095f0 tx=0x7f5d60005f90 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:45:29.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.065+0000 7f5d5affd640 1 -- 172.21.15.72:0/4088225754 <== mon.2 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5d5404d020 con 0x7f5d64101290 2024-03-22T01:45:29.259 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:28 smithi089 ceph-mon[35770]: Health check cleared: MDS_SLOW_METADATA_IO (was: 1 MDSs report slow metadata IOs) 2024-03-22T01:45:29.259 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:28 smithi089 ceph-mon[35770]: Health check cleared: MDS_SLOW_REQUEST (was: 1 MDSs report slow requests) 2024-03-22T01:45:29.259 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:28 smithi089 ceph-mon[35770]: Cluster is now healthy 2024-03-22T01:45:29.259 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:28 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:45:29.259 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:28 smithi089 ceph-mon[35770]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:45:29.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.282+0000 7f5d6bf8a640 1 -- 172.21.15.72:0/4088225754 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5d64068e60 con 0x7f5d64101290 2024-03-22T01:45:29.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.282+0000 7f5d5affd640 1 -- 172.21.15.72:0/4088225754 <== mon.2 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) ==== 95+0+11791 (secure 0 0 0) 0x7f5d540be210 con 0x7f5d64101290 2024-03-22T01:45:29.282 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:45:29.282 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":103,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6835/3077216308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3077216308},{"type":"v1","addr":"172.21.15.72:6835","nonce":3077216308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2460254143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":2460254143},{"type":"v1","addr":"172.21.15.72:6837","nonce":2460254143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6839/2026432819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6838","nonce":2026432819},{"type":"v1","addr":"172.21.15.72:6839","nonce":2026432819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6841/400094545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6840","nonce":400094545},{"type":"v1","addr":"172.21.15.72:6841","nonce":400094545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6833/1578701171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":1578701171},{"type":"v1","addr":"172.21.15.89:6833","nonce":1578701171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/2703047213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2703047213},{"type":"v1","addr":"172.21.15.98:6833","nonce":2703047213}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2589468787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2589468787},{"type":"v1","addr":"172.21.15.89:6835","nonce":2589468787}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/92657636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":92657636},{"type":"v1","addr":"172.21.15.98:6835","nonce":92657636}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1662941782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1662941782},{"type":"v1","addr":"172.21.15.89:6837","nonce":1662941782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24490,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/781860371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":781860371},{"type":"v1","addr":"172.21.15.98:6837","nonce":781860371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":103,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-22T01:34:14.815677+0000","modified":"2024-03-22T01:45:27.515143+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24524},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":171,"addr":"172.21.15.98:6839/3192173650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3192173650},{"type":"v1","addr":"172.21.15.98:6839","nonce":3192173650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":171,"addr":"172.21.15.89:6839/149200280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":149200280},{"type":"v1","addr":"172.21.15.89:6839","nonce":149200280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24524]},"id":1}]} 2024-03-22T01:45:29.282 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 103 2024-03-22T01:45:29.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.284+0000 7f5d6bf8a640 1 -- 172.21.15.72:0/4088225754 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f5d500a3110 msgr2=0x7f5d500a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:45:29.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.284+0000 7f5d6bf8a640 1 --2- 172.21.15.72:0/4088225754 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f5d500a3110 0x7f5d500a55d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5d600095f0 tx=0x7f5d60005f90 comp rx=0 tx=0).stop 2024-03-22T01:45:29.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.284+0000 7f5d6bf8a640 1 -- 172.21.15.72:0/4088225754 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d64101290 msgr2=0x7f5d6419abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:45:29.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.284+0000 7f5d6bf8a640 1 --2- 172.21.15.72:0/4088225754 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d64101290 0x7f5d6419abb0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5d5402c170 tx=0x7f5d54004360 comp rx=0 tx=0).stop 2024-03-22T01:45:29.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.284+0000 7f5d6bf8a640 1 -- 172.21.15.72:0/4088225754 shutdown_connections 2024-03-22T01:45:29.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.284+0000 7f5d6bf8a640 1 --2- 172.21.15.72:0/4088225754 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f5d500a3110 0x7f5d500a55d0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:29.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.284+0000 7f5d6bf8a640 1 --2- 172.21.15.72:0/4088225754 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d64101290 0x7f5d6419abb0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:29.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.284+0000 7f5d6bf8a640 1 --2- 172.21.15.72:0/4088225754 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5d6410e490 0x7f5d6419ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:29.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.284+0000 7f5d6bf8a640 1 --2- 172.21.15.72:0/4088225754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d641008e0 0x7f5d6419a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:29.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.284+0000 7f5d6bf8a640 1 -- 172.21.15.72:0/4088225754 >> 172.21.15.72:0/4088225754 conn(0x7f5d640fc520 msgr2=0x7f5d6410f740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:45:29.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.284+0000 7f5d6bf8a640 1 -- 172.21.15.72:0/4088225754 shutdown_connections 2024-03-22T01:45:29.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.284+0000 7f5d6bf8a640 1 -- 172.21.15.72:0/4088225754 wait complete. 2024-03-22T01:45:29.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:28 smithi072 ceph-mon[32414]: Health check cleared: MDS_SLOW_METADATA_IO (was: 1 MDSs report slow metadata IOs) 2024-03-22T01:45:29.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:28 smithi072 ceph-mon[32414]: Health check cleared: MDS_SLOW_REQUEST (was: 1 MDSs report slow requests) 2024-03-22T01:45:29.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:28 smithi072 ceph-mon[32414]: Cluster is now healthy 2024-03-22T01:45:29.373 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:28 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:45:29.373 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:28 smithi072 ceph-mon[32414]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:45:29.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:28 smithi098 ceph-mon[36468]: Health check cleared: MDS_SLOW_METADATA_IO (was: 1 MDSs report slow metadata IOs) 2024-03-22T01:45:29.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:28 smithi098 ceph-mon[36468]: Health check cleared: MDS_SLOW_REQUEST (was: 1 MDSs report slow requests) 2024-03-22T01:45:29.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:28 smithi098 ceph-mon[36468]: Cluster is now healthy 2024-03-22T01:45:29.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:28 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] up:active 2024-03-22T01:45:29.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:28 smithi098 ceph-mon[36468]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-22T01:45:29.429 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-22T01:45:29.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.534+0000 7eff98c8a640 1 -- 172.21.15.72:0/3618548722 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff94100a20 msgr2=0x7eff94100ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:45:29.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.534+0000 7eff98c8a640 1 --2- 172.21.15.72:0/3618548722 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff94100a20 0x7eff94100ea0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7eff7c009930 tx=0x7eff7c02bc30 comp rx=0 tx=0).stop 2024-03-22T01:45:29.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.534+0000 7eff98c8a640 1 -- 172.21.15.72:0/3618548722 shutdown_connections 2024-03-22T01:45:29.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.534+0000 7eff98c8a640 1 --2- 172.21.15.72:0/3618548722 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff941013e0 0x7eff9410d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:29.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.534+0000 7eff98c8a640 1 --2- 172.21.15.72:0/3618548722 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff94100a20 0x7eff94100ea0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:29.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.534+0000 7eff98c8a640 1 --2- 172.21.15.72:0/3618548722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff94106a30 0x7eff94106e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:29.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.534+0000 7eff98c8a640 1 -- 172.21.15.72:0/3618548722 >> 172.21.15.72:0/3618548722 conn(0x7eff94077bb0 msgr2=0x7eff94077fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:45:29.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.534+0000 7eff98c8a640 1 -- 172.21.15.72:0/3618548722 shutdown_connections 2024-03-22T01:45:29.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.534+0000 7eff98c8a640 1 -- 172.21.15.72:0/3618548722 wait complete. 2024-03-22T01:45:29.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.535+0000 7eff98c8a640 1 Processor -- start 2024-03-22T01:45:29.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.535+0000 7eff98c8a640 1 -- start start 2024-03-22T01:45:29.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.535+0000 7eff98c8a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff94100a20 0x7eff941984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:29.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.535+0000 7eff98c8a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff941013e0 0x7eff94198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:29.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.535+0000 7eff98c8a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff94106a30 0x7eff9419cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:29.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.535+0000 7eff98c8a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7eff94110e40 con 0x7eff94100a20 2024-03-22T01:45:29.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.535+0000 7eff98c8a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7eff94110cc0 con 0x7eff941013e0 2024-03-22T01:45:29.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.535+0000 7eff98c8a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7eff94110fc0 con 0x7eff94106a30 2024-03-22T01:45:29.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.535+0000 7eff9259c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff94100a20 0x7eff941984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:29.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.535+0000 7eff9259c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff94100a20 0x7eff941984d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46752/0 (socket says 172.21.15.72:46752) 2024-03-22T01:45:29.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.535+0000 7eff9259c640 1 -- 172.21.15.72:0/4172162229 learned_addr learned my addr 172.21.15.72:0/4172162229 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:45:29.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.535+0000 7eff8bfff640 1 --2- 172.21.15.72:0/4172162229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff941013e0 0x7eff94198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:29.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.535+0000 7eff92d9d640 1 --2- 172.21.15.72:0/4172162229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff94106a30 0x7eff9419cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:29.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.535+0000 7eff9259c640 1 -- 172.21.15.72:0/4172162229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff94106a30 msgr2=0x7eff9419cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:45:29.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.535+0000 7eff9259c640 1 --2- 172.21.15.72:0/4172162229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff94106a30 0x7eff9419cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:29.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.535+0000 7eff9259c640 1 -- 172.21.15.72:0/4172162229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff941013e0 msgr2=0x7eff94198a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:45:29.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.535+0000 7eff9259c640 1 --2- 172.21.15.72:0/4172162229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff941013e0 0x7eff94198a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:29.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.535+0000 7eff9259c640 1 -- 172.21.15.72:0/4172162229 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff9419d470 con 0x7eff94100a20 2024-03-22T01:45:29.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.535+0000 7eff9259c640 1 --2- 172.21.15.72:0/4172162229 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff94100a20 0x7eff941984d0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7eff8400c970 tx=0x7eff8400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:45:29.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.536+0000 7eff8b7fe640 1 -- 172.21.15.72:0/4172162229 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff84007bb0 con 0x7eff94100a20 2024-03-22T01:45:29.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.536+0000 7eff8b7fe640 1 -- 172.21.15.72:0/4172162229 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff84007d50 con 0x7eff94100a20 2024-03-22T01:45:29.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.536+0000 7eff8b7fe640 1 -- 172.21.15.72:0/4172162229 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff8400f450 con 0x7eff94100a20 2024-03-22T01:45:29.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.536+0000 7eff98c8a640 1 -- 172.21.15.72:0/4172162229 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff9419d760 con 0x7eff94100a20 2024-03-22T01:45:29.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.536+0000 7eff98c8a640 1 -- 172.21.15.72:0/4172162229 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff941a4ff0 con 0x7eff94100a20 2024-03-22T01:45:29.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.538+0000 7eff8b7fe640 1 -- 172.21.15.72:0/4172162229 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 25) ==== 137985+0+0 (secure 0 0 0) 0x7eff84005020 con 0x7eff94100a20 2024-03-22T01:45:29.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.538+0000 7eff8b7fe640 1 --2- 172.21.15.72:0/4172162229 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7eff800a3090 0x7eff800a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:29.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.538+0000 7eff8bfff640 1 --2- 172.21.15.72:0/4172162229 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7eff800a3090 0x7eff800a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:29.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.538+0000 7eff8b7fe640 1 -- 172.21.15.72:0/4172162229 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7eff840cf940 con 0x7eff94100a20 2024-03-22T01:45:29.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.539+0000 7eff8bfff640 1 --2- 172.21.15.72:0/4172162229 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7eff800a3090 0x7eff800a5550 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7eff7c005ab0 tx=0x7eff7c005fd0 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:45:29.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.982+0000 7eff9b86a740 1 Processor -- start 2024-03-22T01:45:29.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.982+0000 7eff9b86a740 1 -- start start 2024-03-22T01:45:29.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.982+0000 7eff9b86a740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603959f0800 0x5603959f0be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:29.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.983+0000 7eff9b86a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603959f1120 0x5603959f97c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:29.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.983+0000 7eff9b86a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603959f9d00 0x5603959fc0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:29.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.983+0000 7eff9b86a740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x5603959548d0 con 0x5603959f0800 2024-03-22T01:45:29.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.983+0000 7eff9b86a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5603958fce00 con 0x5603959f9d00 2024-03-22T01:45:29.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.983+0000 7eff9b86a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560395950880 con 0x5603959f1120 2024-03-22T01:45:29.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.983+0000 7eff9259c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603959f0800 0x5603959f0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:29.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.983+0000 7eff92d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603959f9d00 0x5603959fc0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:29.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.983+0000 7eff9259c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603959f0800 0x5603959f0be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46758/0 (socket says 172.21.15.72:46758) 2024-03-22T01:45:29.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.983+0000 7eff9259c640 1 -- 172.21.15.72:0/572643345 learned_addr learned my addr 172.21.15.72:0/572643345 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:45:29.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.983+0000 7eff92d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603959f9d00 0x5603959fc0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.72:40008/0 (socket says 172.21.15.72:40008) 2024-03-22T01:45:29.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.983+0000 7eff8bfff640 1 --2- 172.21.15.72:0/572643345 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603959f1120 0x5603959f97c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:29.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.983+0000 7eff92d9d640 1 -- 172.21.15.72:0/572643345 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603959f1120 msgr2=0x5603959f97c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:45:29.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.983+0000 7eff92d9d640 1 --2- 172.21.15.72:0/572643345 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603959f1120 0x5603959f97c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:29.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.983+0000 7eff92d9d640 1 -- 172.21.15.72:0/572643345 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603959f0800 msgr2=0x5603959f0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:45:29.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.983+0000 7eff92d9d640 1 --2- 172.21.15.72:0/572643345 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603959f0800 0x5603959f0be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:29.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.983+0000 7eff92d9d640 1 -- 172.21.15.72:0/572643345 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5603956683f0 con 0x5603959f9d00 2024-03-22T01:45:29.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.983+0000 7eff92d9d640 1 --2- 172.21.15.72:0/572643345 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603959f9d00 0x5603959fc0f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7eff8c009ed0 tx=0x7eff8c0107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aeb2d37b81ec1496 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:45:29.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.984+0000 7eff88ff9640 1 -- 172.21.15.72:0/572643345 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff8c003b70 con 0x5603959f9d00 2024-03-22T01:45:29.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.984+0000 7eff88ff9640 1 -- 172.21.15.72:0/572643345 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff8c015070 con 0x5603959f9d00 2024-03-22T01:45:29.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.984+0000 7eff88ff9640 1 -- 172.21.15.72:0/572643345 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff8c011870 con 0x5603959f9d00 2024-03-22T01:45:29.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.984+0000 7eff9b86a740 1 -- 172.21.15.72:0/572643345 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603959f9d00 msgr2=0x5603959fc0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:45:29.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.984+0000 7eff9b86a740 1 --2- 172.21.15.72:0/572643345 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603959f9d00 0x5603959fc0f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7eff8c009ed0 tx=0x7eff8c0107a0 comp rx=0 tx=0).stop 2024-03-22T01:45:29.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.984+0000 7eff9b86a740 1 -- 172.21.15.72:0/572643345 shutdown_connections 2024-03-22T01:45:29.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.984+0000 7eff9b86a740 1 --2- 172.21.15.72:0/572643345 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603959f9d00 0x5603959fc0f0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:29.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.984+0000 7eff9b86a740 1 --2- 172.21.15.72:0/572643345 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603959f1120 0x5603959f97c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:29.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.984+0000 7eff9b86a740 1 --2- 172.21.15.72:0/572643345 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603959f0800 0x5603959f0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:29.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.984+0000 7eff9b86a740 1 -- 172.21.15.72:0/572643345 >> 172.21.15.72:0/572643345 conn(0x5603959eaf30 msgr2=0x5603959eb310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:45:29.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.985+0000 7eff9b86a740 1 -- 172.21.15.72:0/572643345 shutdown_connections 2024-03-22T01:45:29.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.985+0000 7eff9b86a740 1 -- 172.21.15.72:0/572643345 wait complete. 2024-03-22T01:45:29.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.985+0000 7eff9b86a740 1 -- start start 2024-03-22T01:45:29.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.985+0000 7eff9b86a740 1 Processor -- start 2024-03-22T01:45:29.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.985+0000 7eff9b86a740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603959f0800 0x5603959f6da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:29.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.985+0000 7eff9b86a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603959f1120 0x5603959f72e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:29.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.985+0000 7eff9b86a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603959f9d00 0x5603959f7820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:29.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.985+0000 7eff9b86a740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x5603959fcaf0 con 0x5603959f0800 2024-03-22T01:45:29.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.985+0000 7eff9b86a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x560395910300 con 0x5603959f9d00 2024-03-22T01:45:29.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.985+0000 7eff9b86a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5603959fcc70 con 0x5603959f1120 2024-03-22T01:45:29.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.985+0000 7eff9b86a740 10 client.0 _ll_register_callbacks cb 0x56039594be30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:45:29.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.986+0000 7eff9259c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603959f0800 0x5603959f6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:29.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.986+0000 7eff92d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603959f9d00 0x5603959f7820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:29.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.986+0000 7eff9259c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603959f0800 0x5603959f6da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46762/0 (socket says 172.21.15.72:46762) 2024-03-22T01:45:29.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.986+0000 7eff92d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603959f9d00 0x5603959f7820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.72:40014/0 (socket says 172.21.15.72:40014) 2024-03-22T01:45:29.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.986+0000 7eff9259c640 1 -- 172.21.15.72:0/3485563669 learned_addr learned my addr 172.21.15.72:0/3485563669 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:45:29.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.986+0000 7eff8bfff640 1 --2- 172.21.15.72:0/3485563669 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603959f1120 0x5603959f72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:29.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.986+0000 7eff92d9d640 1 -- 172.21.15.72:0/3485563669 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603959f1120 msgr2=0x5603959f72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:45:29.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.986+0000 7eff92d9d640 1 --2- 172.21.15.72:0/3485563669 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603959f1120 0x5603959f72e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:29.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.986+0000 7eff92d9d640 1 -- 172.21.15.72:0/3485563669 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603959f0800 msgr2=0x5603959f6da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:45:29.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.986+0000 7eff92d9d640 1 --2- 172.21.15.72:0/3485563669 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603959f0800 0x5603959f6da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:29.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.986+0000 7eff92d9d640 1 -- 172.21.15.72:0/3485563669 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603957327d0 con 0x5603959f9d00 2024-03-22T01:45:29.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.986+0000 7eff9259c640 1 --2- 172.21.15.72:0/3485563669 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603959f0800 0x5603959f6da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:45:29.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.986+0000 7eff92d9d640 1 --2- 172.21.15.72:0/3485563669 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603959f9d00 0x5603959f7820 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7eff8c011010 tx=0x7eff8c004230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=110d47db399c2972 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:45:29.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.986+0000 7eff79ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-03-22T01:45:29.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.986+0000 7eff79ffb640 1 -- 172.21.15.72:0/3485563669 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff8c02c060 con 0x5603959f9d00 2024-03-22T01:45:29.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.986+0000 7eff79ffb640 1 -- 172.21.15.72:0/3485563669 <== mon.2 v2:172.21.15.89:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7eff8c003970 con 0x5603959f9d00 2024-03-22T01:45:29.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.986+0000 7eff9b86a740 1 -- 172.21.15.72:0/3485563669 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7eff8c015070 con 0x5603959f9d00 2024-03-22T01:45:29.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.987+0000 7eff79ffb640 1 -- 172.21.15.72:0/3485563669 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7eff8c033050 con 0x5603959f9d00 2024-03-22T01:45:29.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.987+0000 7eff9b86a740 10 client.25895 fetch_fsmap learned FSMap version 103 2024-03-22T01:45:29.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.987+0000 7eff9b86a740 1 -- 172.21.15.72:0/3485563669 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=4+}) -- 0x560395a08e20 con 0x5603959f9d00 2024-03-22T01:45:29.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.987+0000 7eff79ffb640 1 -- 172.21.15.72:0/3485563669 <== mon.2 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff8c015070 con 0x5603959f9d00 2024-03-22T01:45:29.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.987+0000 7eff79ffb640 1 -- 172.21.15.72:0/3485563669 <== mon.2 v2:172.21.15.89:3300/0 5 ==== fsmap(e 103) ==== 8330+0+0 (secure 0 0 0) 0x7eff8c011c10 con 0x5603959f9d00 2024-03-22T01:45:29.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.987+0000 7eff9b86a740 10 client.25895 fetch_fsmap finished waiting for FSMap version 103 2024-03-22T01:45:29.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.987+0000 7eff9b86a740 10 client.25895 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T01:45:29.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.987+0000 7eff9b86a740 20 client.25895 populate_metadata read hostname 'smithi072' 2024-03-22T01:45:29.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.987+0000 7eff9b86a740 1 --2- 172.21.15.72:0/3485563669 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x560395a0a280 0x560395a0c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:29.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.987+0000 7eff9b86a740 4 client.25895 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T01:45:29.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.987+0000 7eff9b86a740 1 -- 172.21.15.72:0/3485563669 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560395961e10 con 0x560395a0a280 2024-03-22T01:45:29.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.987+0000 7eff9259c640 1 --2- 172.21.15.72:0/3485563669 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x560395a0a280 0x560395a0c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:29.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.988+0000 7eff9259c640 1 --2- 172.21.15.72:0/3485563669 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x560395a0a280 0x560395a0c670 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b05c76fa4b9f5601 server_cookie=2d93f88664f35c6e in_seq=0 out_seq=0 2024-03-22T01:45:29.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.988+0000 7eff79ffb640 10 client.25895 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:45:29.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.988+0000 7eff79ffb640 1 -- 172.21.15.72:0/3485563669 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560395961e10 con 0x560395a0a280 2024-03-22T01:45:29.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.988+0000 7eff79ffb640 10 client.25895 handle_command_reply: tid=0 2024-03-22T01:45:29.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.988+0000 7eff9b86a740 1 client.25895 shutdown 2024-03-22T01:45:29.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.989+0000 7eff9b86a740 1 -- 172.21.15.72:0/3485563669 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603959f9d00 msgr2=0x5603959f7820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:45:29.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.989+0000 7eff9b86a740 1 --2- 172.21.15.72:0/3485563669 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603959f9d00 0x5603959f7820 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7eff8c011010 tx=0x7eff8c004230 comp rx=0 tx=0).stop 2024-03-22T01:45:29.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.989+0000 7eff9b86a740 1 -- 172.21.15.72:0/3485563669 shutdown_connections 2024-03-22T01:45:29.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.989+0000 7eff9b86a740 1 --2- 172.21.15.72:0/3485563669 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x560395a0a280 0x560395a0c670 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:29.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.989+0000 7eff9b86a740 1 --2- 172.21.15.72:0/3485563669 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603959f9d00 0x5603959f7820 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:29.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.989+0000 7eff79ffb640 0 client.25895 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:45:29.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.989+0000 7eff9b86a740 1 --2- 172.21.15.72:0/3485563669 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603959f1120 0x5603959f72e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:29.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.989+0000 7eff9b86a740 1 --2- 172.21.15.72:0/3485563669 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603959f0800 0x5603959f6da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:29.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.989+0000 7eff9b86a740 1 -- 172.21.15.72:0/3485563669 >> 172.21.15.72:0/3485563669 conn(0x5603959eaf30 msgr2=0x5603959ecc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:45:29.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.989+0000 7eff9b86a740 1 -- 172.21.15.72:0/3485563669 shutdown_connections 2024-03-22T01:45:29.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.989+0000 7eff9b86a740 1 -- 172.21.15.72:0/3485563669 wait complete. 2024-03-22T01:45:29.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:29.989+0000 7eff9b86a740 20 client.25895 trim_cache size 0 max 16384 2024-03-22T01:45:30.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.003+0000 7eff9b86a740 1 Processor -- start 2024-03-22T01:45:30.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.003+0000 7eff9b86a740 1 -- start start 2024-03-22T01:45:30.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.003+0000 7eff9b86a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603959f0800 0x5603959f94d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:30.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.003+0000 7eff9b86a740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603959f1120 0x560395a058c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:30.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.003+0000 7eff9b86a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603959f9d00 0x560395a11c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:30.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.003+0000 7eff9b86a740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7eff941ad170 con 0x5603959f1120 2024-03-22T01:45:30.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.004+0000 7eff9b86a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7eff941af9f0 con 0x5603959f9d00 2024-03-22T01:45:30.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.004+0000 7eff9b86a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7eff941ad7e0 con 0x5603959f0800 2024-03-22T01:45:30.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.004+0000 7eff8bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603959f1120 0x560395a058c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:30.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.004+0000 7eff8bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603959f1120 0x560395a058c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46778/0 (socket says 172.21.15.72:46778) 2024-03-22T01:45:30.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.004+0000 7eff8bfff640 1 -- 172.21.15.72:0/1743686539 learned_addr learned my addr 172.21.15.72:0/1743686539 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:45:30.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.004+0000 7eff92d9d640 1 --2- 172.21.15.72:0/1743686539 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603959f9d00 0x560395a11c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:30.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.004+0000 7eff9259c640 1 --2- 172.21.15.72:0/1743686539 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603959f0800 0x5603959f94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:30.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.004+0000 7eff8bfff640 1 -- 172.21.15.72:0/1743686539 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603959f0800 msgr2=0x5603959f94d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:45:30.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.004+0000 7eff8bfff640 1 --2- 172.21.15.72:0/1743686539 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603959f0800 0x5603959f94d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:30.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.004+0000 7eff8bfff640 1 -- 172.21.15.72:0/1743686539 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603959f9d00 msgr2=0x560395a11c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:45:30.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.004+0000 7eff8bfff640 1 --2- 172.21.15.72:0/1743686539 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603959f9d00 0x560395a11c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:30.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.004+0000 7eff8bfff640 1 -- 172.21.15.72:0/1743686539 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5603959effa0 con 0x5603959f1120 2024-03-22T01:45:30.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.004+0000 7eff92d9d640 1 --2- 172.21.15.72:0/1743686539 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603959f9d00 0x560395a11c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:45:30.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.004+0000 7eff8bfff640 1 --2- 172.21.15.72:0/1743686539 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603959f1120 0x560395a058c0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7eff7c009930 tx=0x7eff7c03f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e2414a9ed01ba2e server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:45:30.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.004+0000 7eff7bfff640 1 -- 172.21.15.72:0/1743686539 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff7c0042c0 con 0x5603959f1120 2024-03-22T01:45:30.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.004+0000 7eff7bfff640 1 -- 172.21.15.72:0/1743686539 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff7c004460 con 0x5603959f1120 2024-03-22T01:45:30.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.005+0000 7eff7bfff640 1 -- 172.21.15.72:0/1743686539 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff7c0035f0 con 0x5603959f1120 2024-03-22T01:45:30.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.005+0000 7eff9b86a740 1 -- 172.21.15.72:0/1743686539 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603959f1120 msgr2=0x560395a058c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:45:30.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.005+0000 7eff9b86a740 1 --2- 172.21.15.72:0/1743686539 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603959f1120 0x560395a058c0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7eff7c009930 tx=0x7eff7c03f000 comp rx=0 tx=0).stop 2024-03-22T01:45:30.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.005+0000 7eff9b86a740 1 -- 172.21.15.72:0/1743686539 shutdown_connections 2024-03-22T01:45:30.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.005+0000 7eff9b86a740 1 --2- 172.21.15.72:0/1743686539 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603959f9d00 0x560395a11c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:30.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.005+0000 7eff9b86a740 1 --2- 172.21.15.72:0/1743686539 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603959f1120 0x560395a058c0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:30.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.005+0000 7eff9b86a740 1 --2- 172.21.15.72:0/1743686539 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603959f0800 0x5603959f94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:30.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.005+0000 7eff9b86a740 1 -- 172.21.15.72:0/1743686539 >> 172.21.15.72:0/1743686539 conn(0x5603959eaf30 msgr2=0x5603959f6d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:45:30.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.005+0000 7eff9b86a740 1 -- 172.21.15.72:0/1743686539 shutdown_connections 2024-03-22T01:45:30.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.005+0000 7eff9b86a740 1 -- 172.21.15.72:0/1743686539 wait complete. 2024-03-22T01:45:30.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.006+0000 7eff9b86a740 1 -- start start 2024-03-22T01:45:30.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.006+0000 7eff9b86a740 1 Processor -- start 2024-03-22T01:45:30.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.006+0000 7eff9b86a740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603959f0800 0x560395a02140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:30.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.006+0000 7eff9b86a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603959f1120 0x5603959fd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:30.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.006+0000 7eff9b86a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603959f9d00 0x5603959fd9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:30.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.006+0000 7eff9b86a740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7eff941adeb0 con 0x5603959f0800 2024-03-22T01:45:30.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.006+0000 7eff9b86a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7eff941af320 con 0x5603959f9d00 2024-03-22T01:45:30.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.006+0000 7eff9b86a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7eff941ae580 con 0x5603959f1120 2024-03-22T01:45:30.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.006+0000 7eff9b86a740 10 client.0 _ll_register_callbacks cb 0x7eff84007f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:45:30.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.006+0000 7eff9259c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603959f0800 0x560395a02140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:30.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.006+0000 7eff9259c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603959f0800 0x560395a02140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46788/0 (socket says 172.21.15.72:46788) 2024-03-22T01:45:30.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.006+0000 7eff9259c640 1 -- 172.21.15.72:0/3381439435 learned_addr learned my addr 172.21.15.72:0/3381439435 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:45:30.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.006+0000 7eff92d9d640 1 --2- 172.21.15.72:0/3381439435 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603959f9d00 0x5603959fd9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:30.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.006+0000 7eff8bfff640 1 --2- 172.21.15.72:0/3381439435 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603959f1120 0x5603959fd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:30.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.007+0000 7eff9259c640 1 -- 172.21.15.72:0/3381439435 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603959f1120 msgr2=0x5603959fd4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:45:30.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.007+0000 7eff9259c640 1 --2- 172.21.15.72:0/3381439435 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603959f1120 0x5603959fd4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:30.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.007+0000 7eff9259c640 1 -- 172.21.15.72:0/3381439435 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603959f9d00 msgr2=0x5603959fd9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:45:30.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.007+0000 7eff9259c640 1 --2- 172.21.15.72:0/3381439435 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603959f9d00 0x5603959fd9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:30.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.007+0000 7eff9259c640 1 -- 172.21.15.72:0/3381439435 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603959fe220 con 0x5603959f0800 2024-03-22T01:45:30.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.007+0000 7eff9259c640 1 --2- 172.21.15.72:0/3381439435 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603959f0800 0x560395a02140 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7eff8409b450 tx=0x7eff840cf040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c6c8eb324693453 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:45:30.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.007+0000 7eff79ffb640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:45:30.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.007+0000 7eff79ffb640 1 -- 172.21.15.72:0/3381439435 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff840b9060 con 0x5603959f0800 2024-03-22T01:45:30.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.007+0000 7eff79ffb640 1 -- 172.21.15.72:0/3381439435 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7eff84097ff0 con 0x5603959f0800 2024-03-22T01:45:30.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.007+0000 7eff9b86a740 1 -- 172.21.15.72:0/3381439435 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7eff7c004460 con 0x5603959f0800 2024-03-22T01:45:30.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.007+0000 7eff79ffb640 1 -- 172.21.15.72:0/3381439435 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7eff8401d040 con 0x5603959f0800 2024-03-22T01:45:30.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.007+0000 7eff9b86a740 10 client.16065 fetch_fsmap learned FSMap version 103 2024-03-22T01:45:30.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.007+0000 7eff9b86a740 1 -- 172.21.15.72:0/3381439435 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=4+}) -- 0x560395a70270 con 0x5603959f0800 2024-03-22T01:45:30.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.007+0000 7eff79ffb640 1 -- 172.21.15.72:0/3381439435 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff7c004460 con 0x5603959f0800 2024-03-22T01:45:30.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.008+0000 7eff79ffb640 1 -- 172.21.15.72:0/3381439435 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 103) ==== 8330+0+0 (secure 0 0 0) 0x7eff840860b0 con 0x5603959f0800 2024-03-22T01:45:30.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.008+0000 7eff9b86a740 10 client.16065 fetch_fsmap finished waiting for FSMap version 103 2024-03-22T01:45:30.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.008+0000 7eff9b86a740 10 client.16065 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T01:45:30.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.008+0000 7eff9b86a740 20 client.16065 populate_metadata read hostname 'smithi072' 2024-03-22T01:45:30.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.008+0000 7eff9b86a740 1 --2- 172.21.15.72:0/3381439435 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x560395a708f0 0x560395a72ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:30.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.008+0000 7eff9b86a740 4 client.16065 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T01:45:30.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.008+0000 7eff9b86a740 1 -- 172.21.15.72:0/3381439435 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56039595c160 con 0x560395a708f0 2024-03-22T01:45:30.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.008+0000 7eff8bfff640 1 --2- 172.21.15.72:0/3381439435 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x560395a708f0 0x560395a72ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:30.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.008+0000 7eff8bfff640 1 --2- 172.21.15.72:0/3381439435 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x560395a708f0 0x560395a72ce0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8366fc00aebb6bf9 server_cookie=e8b59653b06def79 in_seq=0 out_seq=0 2024-03-22T01:45:30.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.008+0000 7eff79ffb640 10 client.16065 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:45:30.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.008+0000 7eff79ffb640 1 -- 172.21.15.72:0/3381439435 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56039595c160 con 0x560395a708f0 2024-03-22T01:45:30.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.008+0000 7eff79ffb640 10 client.16065 handle_command_reply: tid=0 2024-03-22T01:45:30.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.008+0000 7eff9b86a740 1 client.16065 shutdown 2024-03-22T01:45:30.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.008+0000 7eff9b86a740 1 -- 172.21.15.72:0/3381439435 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603959f0800 msgr2=0x560395a02140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:45:30.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.008+0000 7eff9b86a740 1 --2- 172.21.15.72:0/3381439435 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603959f0800 0x560395a02140 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7eff8409b450 tx=0x7eff840cf040 comp rx=0 tx=0).stop 2024-03-22T01:45:30.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.009+0000 7eff9b86a740 1 -- 172.21.15.72:0/3381439435 shutdown_connections 2024-03-22T01:45:30.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.009+0000 7eff9b86a740 1 --2- 172.21.15.72:0/3381439435 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x560395a708f0 0x560395a72ce0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:30.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.009+0000 7eff9b86a740 1 --2- 172.21.15.72:0/3381439435 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603959f9d00 0x5603959fd9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:30.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.009+0000 7eff9b86a740 1 --2- 172.21.15.72:0/3381439435 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603959f1120 0x5603959fd4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:30.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.009+0000 7eff79ffb640 0 client.16065 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:45:30.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.009+0000 7eff9b86a740 1 --2- 172.21.15.72:0/3381439435 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603959f0800 0x560395a02140 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:30.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.009+0000 7eff9b86a740 1 -- 172.21.15.72:0/3381439435 >> 172.21.15.72:0/3381439435 conn(0x5603959eaf30 msgr2=0x5603959f75f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:45:30.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.009+0000 7eff9b86a740 1 -- 172.21.15.72:0/3381439435 shutdown_connections 2024-03-22T01:45:30.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.009+0000 7eff9b86a740 1 -- 172.21.15.72:0/3381439435 wait complete. 2024-03-22T01:45:30.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.009+0000 7eff9b86a740 20 client.16065 trim_cache size 0 max 16384 2024-03-22T01:45:30.016 INFO:teuthology.orchestra.run.smithi072.stdout:[] 2024-03-22T01:45:30.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.009+0000 7eff98c8a640 1 -- 172.21.15.72:0/4172162229 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7eff800a3090 msgr2=0x7eff800a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:45:30.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.009+0000 7eff98c8a640 1 --2- 172.21.15.72:0/4172162229 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7eff800a3090 0x7eff800a5550 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7eff7c005ab0 tx=0x7eff7c005fd0 comp rx=0 tx=0).stop 2024-03-22T01:45:30.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.009+0000 7eff98c8a640 1 -- 172.21.15.72:0/4172162229 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff94100a20 msgr2=0x7eff941984d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:45:30.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.009+0000 7eff98c8a640 1 --2- 172.21.15.72:0/4172162229 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff94100a20 0x7eff941984d0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7eff8400c970 tx=0x7eff8400ce30 comp rx=0 tx=0).stop 2024-03-22T01:45:30.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.009+0000 7eff98c8a640 1 -- 172.21.15.72:0/4172162229 shutdown_connections 2024-03-22T01:45:30.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.009+0000 7eff98c8a640 1 --2- 172.21.15.72:0/4172162229 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7eff800a3090 0x7eff800a5550 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:30.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.009+0000 7eff98c8a640 1 --2- 172.21.15.72:0/4172162229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff941013e0 0x7eff94198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:30.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.009+0000 7eff98c8a640 1 --2- 172.21.15.72:0/4172162229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff94106a30 0x7eff9419cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:30.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.009+0000 7eff98c8a640 1 --2- 172.21.15.72:0/4172162229 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff94100a20 0x7eff941984d0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:30.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.009+0000 7eff98c8a640 1 -- 172.21.15.72:0/4172162229 >> 172.21.15.72:0/4172162229 conn(0x7eff94077bb0 msgr2=0x7eff940ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:45:30.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.010+0000 7eff98c8a640 1 -- 172.21.15.72:0/4172162229 shutdown_connections 2024-03-22T01:45:30.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:30.010+0000 7eff98c8a640 1 -- 172.21.15.72:0/4172162229 wait complete. 2024-03-22T01:45:30.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:29 smithi089 ceph-mon[35770]: pgmap v268: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.2 MiB/s wr, 681 op/s 2024-03-22T01:45:30.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:29 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/4088225754' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:45:30.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:29 smithi072 ceph-mon[32414]: pgmap v268: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.2 MiB/s wr, 681 op/s 2024-03-22T01:45:30.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:29 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/4088225754' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:45:30.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:29 smithi098 ceph-mon[36468]: pgmap v268: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.2 MiB/s wr, 681 op/s 2024-03-22T01:45:30.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:29 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/4088225754' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:45:31.022 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-22T01:45:31.022 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T01:45:31.022 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-22T01:45:31.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.140+0000 7f534e71e640 1 -- 172.21.15.72:0/4184846216 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5348108290 msgr2=0x7f534810ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:45:31.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.140+0000 7f534e71e640 1 --2- 172.21.15.72:0/4184846216 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5348108290 0x7f534810ec30 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f5340009cd0 tx=0x7f534002c0f0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.140+0000 7f534e71e640 1 -- 172.21.15.72:0/4184846216 shutdown_connections 2024-03-22T01:45:31.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.140+0000 7f534e71e640 1 --2- 172.21.15.72:0/4184846216 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5348103750 0x7f5348107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.140+0000 7f534e71e640 1 --2- 172.21.15.72:0/4184846216 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5348108290 0x7f534810ec30 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.140+0000 7f534e71e640 1 --2- 172.21.15.72:0/4184846216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5348102da0 0x7f5348103180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.140+0000 7f534e71e640 1 -- 172.21.15.72:0/4184846216 >> 172.21.15.72:0/4184846216 conn(0x7f5348077c70 msgr2=0x7f5348078080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:45:31.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.141+0000 7f534e71e640 1 -- 172.21.15.72:0/4184846216 shutdown_connections 2024-03-22T01:45:31.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.141+0000 7f534e71e640 1 -- 172.21.15.72:0/4184846216 wait complete. 2024-03-22T01:45:31.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.141+0000 7f534e71e640 1 Processor -- start 2024-03-22T01:45:31.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.141+0000 7f534e71e640 1 -- start start 2024-03-22T01:45:31.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.141+0000 7f534e71e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5348102da0 0x7f5348198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:31.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.142+0000 7f534e71e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5348108290 0x7f5348198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:31.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.142+0000 7f534e71e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f534819cde0 0x7f534819d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:31.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.142+0000 7f534e71e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f5348112090 con 0x7f5348108290 2024-03-22T01:45:31.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.142+0000 7f534e71e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5348111f10 con 0x7f534819cde0 2024-03-22T01:45:31.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.142+0000 7f534e71e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5348112210 con 0x7f5348102da0 2024-03-22T01:45:31.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.142+0000 7f53477fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5348108290 0x7f5348198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:31.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.142+0000 7f53477fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5348108290 0x7f5348198ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48978/0 (socket says 172.21.15.72:48978) 2024-03-22T01:45:31.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.142+0000 7f53477fe640 1 -- 172.21.15.72:0/617746134 learned_addr learned my addr 172.21.15.72:0/617746134 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:45:31.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.142+0000 7f5347fff640 1 --2- 172.21.15.72:0/617746134 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5348102da0 0x7f5348198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:31.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.142+0000 7f534ccbb640 1 --2- 172.21.15.72:0/617746134 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f534819cde0 0x7f534819d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:31.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.142+0000 7f534ccbb640 1 -- 172.21.15.72:0/617746134 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5348102da0 msgr2=0x7f5348198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:45:31.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.142+0000 7f534ccbb640 1 --2- 172.21.15.72:0/617746134 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5348102da0 0x7f5348198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.142+0000 7f534ccbb640 1 -- 172.21.15.72:0/617746134 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5348108290 msgr2=0x7f5348198ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:45:31.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.142+0000 7f534ccbb640 1 --2- 172.21.15.72:0/617746134 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5348108290 0x7f5348198ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.142+0000 7f534ccbb640 1 -- 172.21.15.72:0/617746134 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f534819d8d0 con 0x7f534819cde0 2024-03-22T01:45:31.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.142+0000 7f53477fe640 1 --2- 172.21.15.72:0/617746134 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5348108290 0x7f5348198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:45:31.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.142+0000 7f534ccbb640 1 --2- 172.21.15.72:0/617746134 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f534819cde0 0x7f534819d290 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f534002c630 tx=0x7f5340032f20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:45:31.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.142+0000 7f53457fa640 1 -- 172.21.15.72:0/617746134 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5340041020 con 0x7f534819cde0 2024-03-22T01:45:31.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.142+0000 7f53457fa640 1 -- 172.21.15.72:0/617746134 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5340034cd0 con 0x7f534819cde0 2024-03-22T01:45:31.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.142+0000 7f534e71e640 1 -- 172.21.15.72:0/617746134 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f534819db60 con 0x7f534819cde0 2024-03-22T01:45:31.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.142+0000 7f53457fa640 1 -- 172.21.15.72:0/617746134 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5340039040 con 0x7f534819cde0 2024-03-22T01:45:31.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.143+0000 7f534e71e640 1 -- 172.21.15.72:0/617746134 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53481a52c0 con 0x7f534819cde0 2024-03-22T01:45:31.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.144+0000 7f53457fa640 1 -- 172.21.15.72:0/617746134 <== mon.2 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 25) ==== 137985+0+0 (secure 0 0 0) 0x7f5340044020 con 0x7f534819cde0 2024-03-22T01:45:31.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.145+0000 7f53513e6740 1 Processor -- start 2024-03-22T01:45:31.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.145+0000 7f53513e6740 1 -- start start 2024-03-22T01:45:31.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.145+0000 7f53457fa640 1 --2- 172.21.15.72:0/617746134 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f53300a3090 0x7f53300a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:31.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.145+0000 7f53513e6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5586284a6ba0 0x5586284a6f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:31.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.145+0000 7f53513e6740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5586284a74c0 0x5586284afb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:31.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.145+0000 7f53513e6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5586284b00a0 0x5586284b2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:31.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.145+0000 7f53513e6740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55862840a570 con 0x5586284a74c0 2024-03-22T01:45:31.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.145+0000 7f53513e6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5586283b2e10 con 0x5586284b00a0 2024-03-22T01:45:31.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.145+0000 7f53513e6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558628406520 con 0x5586284a6ba0 2024-03-22T01:45:31.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.145+0000 7f53477fe640 1 --2- 172.21.15.72:0/617746134 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f53300a3090 0x7f53300a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:31.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.145+0000 7f53457fa640 1 -- 172.21.15.72:0/617746134 <== mon.2 v2:172.21.15.89:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f53400f1740 con 0x7f534819cde0 2024-03-22T01:45:31.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.145+0000 7f53477fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5586284a74c0 0x5586284afb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:31.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.145+0000 7f53477fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5586284a74c0 0x5586284afb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48992/0 (socket says 172.21.15.72:48992) 2024-03-22T01:45:31.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.145+0000 7f53477fe640 1 -- 172.21.15.72:0/1865930198 learned_addr learned my addr 172.21.15.72:0/1865930198 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:45:31.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.145+0000 7f5347fff640 1 --2- 172.21.15.72:0/1865930198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5586284a6ba0 0x5586284a6f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:31.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.145+0000 7f534ccbb640 1 --2- 172.21.15.72:0/1865930198 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5586284b00a0 0x5586284b2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:31.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.146+0000 7f53477fe640 1 -- 172.21.15.72:0/1865930198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5586284a6ba0 msgr2=0x5586284a6f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:45:31.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.146+0000 7f53477fe640 1 --2- 172.21.15.72:0/1865930198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5586284a6ba0 0x5586284a6f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.146+0000 7f53477fe640 1 -- 172.21.15.72:0/1865930198 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5586284b00a0 msgr2=0x5586284b2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:45:31.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.146+0000 7f53477fe640 1 --2- 172.21.15.72:0/1865930198 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5586284b00a0 0x5586284b2490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.146+0000 7f53477fe640 1 -- 172.21.15.72:0/1865930198 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5586281e67e0 con 0x5586284a74c0 2024-03-22T01:45:31.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.146+0000 7f53477fe640 1 --2- 172.21.15.72:0/1865930198 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5586284a74c0 0x5586284afb60 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f5338010ec0 tx=0x7f533800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3019a068847e8c58 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:45:31.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.146+0000 7f53477fe640 1 --2- 172.21.15.72:0/617746134 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f53300a3090 0x7f53300a5550 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f533800fc90 tx=0x7f5338018000 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:45:31.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.146+0000 7f532e7fc640 1 -- 172.21.15.72:0/1865930198 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5338022070 con 0x5586284a74c0 2024-03-22T01:45:31.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.146+0000 7f532e7fc640 1 -- 172.21.15.72:0/1865930198 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5338009de0 con 0x5586284a74c0 2024-03-22T01:45:31.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.146+0000 7f532e7fc640 1 -- 172.21.15.72:0/1865930198 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5338019040 con 0x5586284a74c0 2024-03-22T01:45:31.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.147+0000 7f53513e6740 1 -- 172.21.15.72:0/1865930198 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5586284a74c0 msgr2=0x5586284afb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:45:31.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.147+0000 7f53513e6740 1 --2- 172.21.15.72:0/1865930198 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5586284a74c0 0x5586284afb60 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f5338010ec0 tx=0x7f533800f790 comp rx=0 tx=0).stop 2024-03-22T01:45:31.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.147+0000 7f53513e6740 1 -- 172.21.15.72:0/1865930198 shutdown_connections 2024-03-22T01:45:31.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.147+0000 7f53513e6740 1 --2- 172.21.15.72:0/1865930198 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5586284b00a0 0x5586284b2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.147+0000 7f53513e6740 1 --2- 172.21.15.72:0/1865930198 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5586284a74c0 0x5586284afb60 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.147+0000 7f53513e6740 1 --2- 172.21.15.72:0/1865930198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5586284a6ba0 0x5586284a6f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.147+0000 7f53513e6740 1 -- 172.21.15.72:0/1865930198 >> 172.21.15.72:0/1865930198 conn(0x5586284a12d0 msgr2=0x5586284a16b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:45:31.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.147+0000 7f53513e6740 1 -- 172.21.15.72:0/1865930198 shutdown_connections 2024-03-22T01:45:31.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.147+0000 7f53513e6740 1 -- 172.21.15.72:0/1865930198 wait complete. 2024-03-22T01:45:31.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.147+0000 7f53513e6740 1 -- start start 2024-03-22T01:45:31.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.147+0000 7f53513e6740 1 Processor -- start 2024-03-22T01:45:31.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.148+0000 7f53513e6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5586284a6ba0 0x5586284afa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:31.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.148+0000 7f53513e6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5586284a74c0 0x5586284acd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:31.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.148+0000 7f53513e6740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5586284b00a0 0x5586284ad2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:31.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.148+0000 7f53513e6740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x5586284b2e30 con 0x5586284b00a0 2024-03-22T01:45:31.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.148+0000 7f53513e6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5586283c6110 con 0x5586284a74c0 2024-03-22T01:45:31.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.148+0000 7f53513e6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5586284b2fb0 con 0x5586284a6ba0 2024-03-22T01:45:31.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.148+0000 7f53513e6740 10 client.0 _ll_register_callbacks cb 0x558628402be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:45:31.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.148+0000 7f534ccbb640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5586284b00a0 0x5586284ad2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:31.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.148+0000 7f534ccbb640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5586284b00a0 0x5586284ad2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:49002/0 (socket says 172.21.15.72:49002) 2024-03-22T01:45:31.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.148+0000 7f53477fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5586284a74c0 0x5586284acd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:31.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.148+0000 7f534ccbb640 1 -- 172.21.15.72:0/4170459416 learned_addr learned my addr 172.21.15.72:0/4170459416 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:45:31.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.148+0000 7f5347fff640 1 --2- 172.21.15.72:0/4170459416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5586284a6ba0 0x5586284afa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:31.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.148+0000 7f534ccbb640 1 -- 172.21.15.72:0/4170459416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5586284a6ba0 msgr2=0x5586284afa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:45:31.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.148+0000 7f534ccbb640 1 --2- 172.21.15.72:0/4170459416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5586284a6ba0 0x5586284afa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.148+0000 7f534ccbb640 1 -- 172.21.15.72:0/4170459416 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5586284a74c0 msgr2=0x5586284acd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:45:31.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.148+0000 7f534ccbb640 1 --2- 172.21.15.72:0/4170459416 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5586284a74c0 0x5586284acd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.148+0000 7f534ccbb640 1 -- 172.21.15.72:0/4170459416 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558628246f70 con 0x5586284b00a0 2024-03-22T01:45:31.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.148+0000 7f53477fe640 1 --2- 172.21.15.72:0/4170459416 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5586284a74c0 0x5586284acd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:45:31.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.148+0000 7f534ccbb640 1 --2- 172.21.15.72:0/4170459416 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5586284b00a0 0x5586284ad2f0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f53400b9cf0 tx=0x7f53400bd490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4862a61f67bc067 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:45:31.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.148+0000 7f53277fe640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:45:31.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.148+0000 7f53277fe640 1 -- 172.21.15.72:0/4170459416 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53400bd890 con 0x5586284b00a0 2024-03-22T01:45:31.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.148+0000 7f53277fe640 1 -- 172.21.15.72:0/4170459416 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f53400a80d0 con 0x5586284b00a0 2024-03-22T01:45:31.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.148+0000 7f53513e6740 1 -- 172.21.15.72:0/4170459416 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5338009de0 con 0x5586284b00a0 2024-03-22T01:45:31.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.148+0000 7f53277fe640 1 -- 172.21.15.72:0/4170459416 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7f53400c19e0 con 0x5586284b00a0 2024-03-22T01:45:31.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.148+0000 7f53513e6740 10 client.16077 fetch_fsmap learned FSMap version 103 2024-03-22T01:45:31.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.149+0000 7f53513e6740 1 -- 172.21.15.72:0/4170459416 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=4+}) -- 0x5586280de170 con 0x5586284b00a0 2024-03-22T01:45:31.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.149+0000 7f53277fe640 1 -- 172.21.15.72:0/4170459416 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5338009de0 con 0x5586284b00a0 2024-03-22T01:45:31.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.149+0000 7f53277fe640 1 -- 172.21.15.72:0/4170459416 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 103) ==== 8330+0+0 (secure 0 0 0) 0x7f534003c030 con 0x5586284b00a0 2024-03-22T01:45:31.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.149+0000 7f53513e6740 10 client.16077 fetch_fsmap finished waiting for FSMap version 103 2024-03-22T01:45:31.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.149+0000 7f53513e6740 10 client.16077 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T01:45:31.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.149+0000 7f53513e6740 20 client.16077 populate_metadata read hostname 'smithi072' 2024-03-22T01:45:31.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.149+0000 7f53513e6740 1 --2- 172.21.15.72:0/4170459416 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x5586284bffa0 0x5586284c2390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:31.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.149+0000 7f53513e6740 4 client.16077 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T01:45:31.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.149+0000 7f53513e6740 1 -- 172.21.15.72:0/4170459416 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558628417ab0 con 0x5586284bffa0 2024-03-22T01:45:31.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.149+0000 7f5347fff640 1 --2- 172.21.15.72:0/4170459416 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x5586284bffa0 0x5586284c2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:31.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.149+0000 7f5347fff640 1 --2- 172.21.15.72:0/4170459416 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x5586284bffa0 0x5586284c2390 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a4be3538743a0202 server_cookie=4c655c0df3691b02 in_seq=0 out_seq=0 2024-03-22T01:45:31.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.149+0000 7f53277fe640 10 client.16077 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:45:31.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.150+0000 7f53277fe640 1 -- 172.21.15.72:0/4170459416 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558628417ab0 con 0x5586284bffa0 2024-03-22T01:45:31.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.150+0000 7f53277fe640 10 client.16077 handle_command_reply: tid=0 2024-03-22T01:45:31.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.150+0000 7f53513e6740 1 client.16077 shutdown 2024-03-22T01:45:31.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.150+0000 7f53513e6740 1 -- 172.21.15.72:0/4170459416 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5586284b00a0 msgr2=0x5586284ad2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:45:31.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.150+0000 7f53513e6740 1 --2- 172.21.15.72:0/4170459416 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5586284b00a0 0x5586284ad2f0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f53400b9cf0 tx=0x7f53400bd490 comp rx=0 tx=0).stop 2024-03-22T01:45:31.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.150+0000 7f53513e6740 1 -- 172.21.15.72:0/4170459416 shutdown_connections 2024-03-22T01:45:31.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.150+0000 7f53513e6740 1 --2- 172.21.15.72:0/4170459416 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x5586284bffa0 0x5586284c2390 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.150+0000 7f53513e6740 1 --2- 172.21.15.72:0/4170459416 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5586284a74c0 0x5586284acd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.150+0000 7f53513e6740 1 --2- 172.21.15.72:0/4170459416 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5586284b00a0 0x5586284ad2f0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.150+0000 7f53513e6740 1 --2- 172.21.15.72:0/4170459416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5586284a6ba0 0x5586284afa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.150+0000 7f53277fe640 0 client.16077 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:45:31.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.150+0000 7f53513e6740 1 -- 172.21.15.72:0/4170459416 >> 172.21.15.72:0/4170459416 conn(0x5586284a12d0 msgr2=0x5586284a3020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:45:31.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.150+0000 7f53513e6740 1 -- 172.21.15.72:0/4170459416 shutdown_connections 2024-03-22T01:45:31.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.150+0000 7f53513e6740 1 -- 172.21.15.72:0/4170459416 wait complete. 2024-03-22T01:45:31.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.150+0000 7f53513e6740 20 client.16077 trim_cache size 0 max 16384 2024-03-22T01:45:31.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.168+0000 7f53513e6740 1 Processor -- start 2024-03-22T01:45:31.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.168+0000 7f53513e6740 1 -- start start 2024-03-22T01:45:31.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.168+0000 7f53513e6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5586284a6ba0 0x5586284bbc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:31.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.168+0000 7f53513e6740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5586284a74c0 0x5586284c7c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:31.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.168+0000 7f53513e6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5586284b00a0 0x5586284af4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:31.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.168+0000 7f53513e6740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f53481acea0 con 0x5586284a74c0 2024-03-22T01:45:31.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.168+0000 7f53513e6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f53481af720 con 0x5586284a6ba0 2024-03-22T01:45:31.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.168+0000 7f53513e6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f53481ad510 con 0x5586284b00a0 2024-03-22T01:45:31.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.168+0000 7f53477fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5586284a74c0 0x5586284c7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:31.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.168+0000 7f53477fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5586284a74c0 0x5586284c7c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:49014/0 (socket says 172.21.15.72:49014) 2024-03-22T01:45:31.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.168+0000 7f53477fe640 1 -- 172.21.15.72:0/3137847586 learned_addr learned my addr 172.21.15.72:0/3137847586 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:45:31.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.168+0000 7f5347fff640 1 --2- 172.21.15.72:0/3137847586 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5586284a6ba0 0x5586284bbc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:31.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.168+0000 7f534ccbb640 1 --2- 172.21.15.72:0/3137847586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5586284b00a0 0x5586284af4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:31.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.168+0000 7f53477fe640 1 -- 172.21.15.72:0/3137847586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5586284b00a0 msgr2=0x5586284af4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:45:31.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.168+0000 7f53477fe640 1 --2- 172.21.15.72:0/3137847586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5586284b00a0 0x5586284af4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.168+0000 7f53477fe640 1 -- 172.21.15.72:0/3137847586 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5586284a6ba0 msgr2=0x5586284bbc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:45:31.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.168+0000 7f53477fe640 1 --2- 172.21.15.72:0/3137847586 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5586284a6ba0 0x5586284bbc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.168+0000 7f53477fe640 1 -- 172.21.15.72:0/3137847586 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5586284a72f0 con 0x5586284a74c0 2024-03-22T01:45:31.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.168+0000 7f534ccbb640 1 --2- 172.21.15.72:0/3137847586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5586284b00a0 0x5586284af4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T01:45:31.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.169+0000 7f53477fe640 1 --2- 172.21.15.72:0/3137847586 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5586284a74c0 0x5586284c7c80 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f5338005610 tx=0x7f5338020040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0f97445ae9ee57e server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:45:31.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.169+0000 7f532dffb640 1 -- 172.21.15.72:0/3137847586 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5338022040 con 0x5586284a74c0 2024-03-22T01:45:31.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.169+0000 7f532dffb640 1 -- 172.21.15.72:0/3137847586 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5338009de0 con 0x5586284a74c0 2024-03-22T01:45:31.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.169+0000 7f532dffb640 1 -- 172.21.15.72:0/3137847586 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53380306e0 con 0x5586284a74c0 2024-03-22T01:45:31.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.169+0000 7f53513e6740 1 -- 172.21.15.72:0/3137847586 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5586284a74c0 msgr2=0x5586284c7c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:45:31.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.169+0000 7f53513e6740 1 --2- 172.21.15.72:0/3137847586 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5586284a74c0 0x5586284c7c80 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f5338005610 tx=0x7f5338020040 comp rx=0 tx=0).stop 2024-03-22T01:45:31.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.169+0000 7f53513e6740 1 -- 172.21.15.72:0/3137847586 shutdown_connections 2024-03-22T01:45:31.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.169+0000 7f53513e6740 1 --2- 172.21.15.72:0/3137847586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5586284b00a0 0x5586284af4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.169+0000 7f53513e6740 1 --2- 172.21.15.72:0/3137847586 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5586284a74c0 0x5586284c7c80 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.169+0000 7f53513e6740 1 --2- 172.21.15.72:0/3137847586 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5586284a6ba0 0x5586284bbc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.169+0000 7f53513e6740 1 -- 172.21.15.72:0/3137847586 >> 172.21.15.72:0/3137847586 conn(0x5586284a12d0 msgr2=0x5586284a8390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:45:31.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.169+0000 7f53513e6740 1 -- 172.21.15.72:0/3137847586 shutdown_connections 2024-03-22T01:45:31.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.169+0000 7f53513e6740 1 -- 172.21.15.72:0/3137847586 wait complete. 2024-03-22T01:45:31.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.170+0000 7f53513e6740 1 -- start start 2024-03-22T01:45:31.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.170+0000 7f53513e6740 1 Processor -- start 2024-03-22T01:45:31.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.170+0000 7f53513e6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5586284a6ba0 0x5586284aa050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:31.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.170+0000 7f53513e6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5586284a74c0 0x5586284aa590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:31.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.170+0000 7f53513e6740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5586284b00a0 0x5586285252b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:31.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.170+0000 7f53513e6740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f53481adbe0 con 0x5586284b00a0 2024-03-22T01:45:31.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.170+0000 7f53513e6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f53481af050 con 0x5586284a74c0 2024-03-22T01:45:31.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.170+0000 7f53513e6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f53481ae2b0 con 0x5586284a6ba0 2024-03-22T01:45:31.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.170+0000 7f53513e6740 10 client.0 _ll_register_callbacks cb 0x7f53400bd660 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:45:31.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.170+0000 7f534ccbb640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5586284b00a0 0x5586285252b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:31.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.170+0000 7f534ccbb640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5586284b00a0 0x5586285252b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:49020/0 (socket says 172.21.15.72:49020) 2024-03-22T01:45:31.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.170+0000 7f534ccbb640 1 -- 172.21.15.72:0/1470608192 learned_addr learned my addr 172.21.15.72:0/1470608192 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:45:31.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.170+0000 7f53477fe640 1 --2- 172.21.15.72:0/1470608192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5586284a74c0 0x5586284aa590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:31.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.170+0000 7f5347fff640 1 --2- 172.21.15.72:0/1470608192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5586284a6ba0 0x5586284aa050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:31.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.170+0000 7f534ccbb640 1 -- 172.21.15.72:0/1470608192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5586284a6ba0 msgr2=0x5586284aa050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:45:31.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.170+0000 7f534ccbb640 1 --2- 172.21.15.72:0/1470608192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5586284a6ba0 0x5586284aa050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.170+0000 7f534ccbb640 1 -- 172.21.15.72:0/1470608192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5586284a74c0 msgr2=0x5586284aa590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:45:31.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.170+0000 7f534ccbb640 1 --2- 172.21.15.72:0/1470608192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5586284a74c0 0x5586284aa590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.170+0000 7f534ccbb640 1 -- 172.21.15.72:0/1470608192 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5586285257f0 con 0x5586284b00a0 2024-03-22T01:45:31.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.170+0000 7f53477fe640 1 --2- 172.21.15.72:0/1470608192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5586284a74c0 0x5586284aa590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:45:31.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.171+0000 7f534ccbb640 1 --2- 172.21.15.72:0/1470608192 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5586284b00a0 0x5586285252b0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f53400bd590 tx=0x7f53400bd500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=765eb5ecae7a9f2c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:45:31.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.171+0000 7f53277fe640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:45:31.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.171+0000 7f53277fe640 1 -- 172.21.15.72:0/1470608192 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53400d5070 con 0x5586284b00a0 2024-03-22T01:45:31.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.171+0000 7f53277fe640 1 -- 172.21.15.72:0/1470608192 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f53400a80d0 con 0x5586284b00a0 2024-03-22T01:45:31.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.171+0000 7f53513e6740 1 -- 172.21.15.72:0/1470608192 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5338009de0 con 0x5586284b00a0 2024-03-22T01:45:31.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.171+0000 7f53277fe640 1 -- 172.21.15.72:0/1470608192 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7f5340034920 con 0x5586284b00a0 2024-03-22T01:45:31.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.171+0000 7f53513e6740 10 client.16083 fetch_fsmap learned FSMap version 103 2024-03-22T01:45:31.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.171+0000 7f53513e6740 1 -- 172.21.15.72:0/1470608192 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=4+}) -- 0x558628526a50 con 0x5586284b00a0 2024-03-22T01:45:31.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.171+0000 7f53277fe640 1 -- 172.21.15.72:0/1470608192 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5338009de0 con 0x5586284b00a0 2024-03-22T01:45:31.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.171+0000 7f53277fe640 1 -- 172.21.15.72:0/1470608192 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 103) ==== 8330+0+0 (secure 0 0 0) 0x7f53400d6bc0 con 0x5586284b00a0 2024-03-22T01:45:31.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.171+0000 7f53513e6740 10 client.16083 fetch_fsmap finished waiting for FSMap version 103 2024-03-22T01:45:31.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.171+0000 7f53513e6740 10 client.16083 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T01:45:31.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.171+0000 7f53513e6740 20 client.16083 populate_metadata read hostname 'smithi072' 2024-03-22T01:45:31.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.171+0000 7f53513e6740 1 --2- 172.21.15.72:0/1470608192 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x558628526fe0 0x5586285293d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:31.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.171+0000 7f53513e6740 4 client.16083 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-22T01:45:31.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.171+0000 7f53513e6740 1 -- 172.21.15.72:0/1470608192 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x558628411e00 con 0x558628526fe0 2024-03-22T01:45:31.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.171+0000 7f5347fff640 1 --2- 172.21.15.72:0/1470608192 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x558628526fe0 0x5586285293d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:31.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.172+0000 7f53277fe640 10 client.16083 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:45:31.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.172+0000 7f5347fff640 1 --2- 172.21.15.72:0/1470608192 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x558628526fe0 0x5586285293d0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a48e3ec65b54b12 server_cookie=a4ef6eaab2d9f328 in_seq=0 out_seq=0 2024-03-22T01:45:31.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.173+0000 7f53277fe640 1 -- 172.21.15.72:0/1470608192 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x558628411e00 con 0x558628526fe0 2024-03-22T01:45:31.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.173+0000 7f53277fe640 10 client.16083 handle_command_reply: tid=0 2024-03-22T01:45:31.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.173+0000 7f53513e6740 1 client.16083 shutdown 2024-03-22T01:45:31.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.173+0000 7f53513e6740 1 -- 172.21.15.72:0/1470608192 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5586284b00a0 msgr2=0x5586285252b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:45:31.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.173+0000 7f53513e6740 1 --2- 172.21.15.72:0/1470608192 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5586284b00a0 0x5586285252b0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f53400bd590 tx=0x7f53400bd500 comp rx=0 tx=0).stop 2024-03-22T01:45:31.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.173+0000 7f53513e6740 1 -- 172.21.15.72:0/1470608192 shutdown_connections 2024-03-22T01:45:31.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.173+0000 7f53513e6740 1 --2- 172.21.15.72:0/1470608192 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x558628526fe0 0x5586285293d0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.173+0000 7f53513e6740 1 --2- 172.21.15.72:0/1470608192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5586284a74c0 0x5586284aa590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.173+0000 7f53513e6740 1 --2- 172.21.15.72:0/1470608192 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5586284b00a0 0x5586285252b0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.173+0000 7f53277fe640 0 client.16083 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:45:31.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.173+0000 7f53513e6740 1 --2- 172.21.15.72:0/1470608192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5586284a6ba0 0x5586284aa050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.173+0000 7f53513e6740 1 -- 172.21.15.72:0/1470608192 >> 172.21.15.72:0/1470608192 conn(0x5586284a12d0 msgr2=0x5586284a25b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:45:31.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.173+0000 7f53513e6740 1 -- 172.21.15.72:0/1470608192 shutdown_connections 2024-03-22T01:45:31.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.173+0000 7f53513e6740 1 -- 172.21.15.72:0/1470608192 wait complete. 2024-03-22T01:45:31.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.173+0000 7f53513e6740 20 client.16083 trim_cache size 0 max 16384 2024-03-22T01:45:31.179 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-22T01:45:31.179 INFO:teuthology.orchestra.run.smithi072.stdout: "return_code": 0, 2024-03-22T01:45:31.179 INFO:teuthology.orchestra.run.smithi072.stdout: "scrub_tag": "b92ea364-f747-42c0-973f-030d2e78c51b", 2024-03-22T01:45:31.179 INFO:teuthology.orchestra.run.smithi072.stdout: "mode": "asynchronous" 2024-03-22T01:45:31.179 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-22T01:45:31.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.174+0000 7f534e71e640 1 -- 172.21.15.72:0/617746134 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f53300a3090 msgr2=0x7f53300a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:45:31.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.174+0000 7f534e71e640 1 --2- 172.21.15.72:0/617746134 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f53300a3090 0x7f53300a5550 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f533800fc90 tx=0x7f5338018000 comp rx=0 tx=0).stop 2024-03-22T01:45:31.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.174+0000 7f534e71e640 1 -- 172.21.15.72:0/617746134 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f534819cde0 msgr2=0x7f534819d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:45:31.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.174+0000 7f534e71e640 1 --2- 172.21.15.72:0/617746134 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f534819cde0 0x7f534819d290 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f534002c630 tx=0x7f5340032f20 comp rx=0 tx=0).stop 2024-03-22T01:45:31.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.174+0000 7f534e71e640 1 -- 172.21.15.72:0/617746134 shutdown_connections 2024-03-22T01:45:31.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.174+0000 7f534e71e640 1 --2- 172.21.15.72:0/617746134 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f53300a3090 0x7f53300a5550 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.174+0000 7f534e71e640 1 --2- 172.21.15.72:0/617746134 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f534819cde0 0x7f534819d290 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.174+0000 7f534e71e640 1 --2- 172.21.15.72:0/617746134 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5348108290 0x7f5348198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.174+0000 7f534e71e640 1 --2- 172.21.15.72:0/617746134 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5348102da0 0x7f5348198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.174+0000 7f534e71e640 1 -- 172.21.15.72:0/617746134 >> 172.21.15.72:0/617746134 conn(0x7f5348077c70 msgr2=0x7f53480ffb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:45:31.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.174+0000 7f534e71e640 1 -- 172.21.15.72:0/617746134 shutdown_connections 2024-03-22T01:45:31.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.174+0000 7f534e71e640 1 -- 172.21.15.72:0/617746134 wait complete. 2024-03-22T01:45:31.184 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T01:45:31.185 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T01:45:31.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.292+0000 7f034c847640 1 -- 172.21.15.72:0/2659268631 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0348106a30 msgr2=0x7f0348106e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:45:31.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.292+0000 7f034c847640 1 --2- 172.21.15.72:0/2659268631 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0348106a30 0x7f0348106e10 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f0338009930 tx=0x7f033802bc30 comp rx=0 tx=0).stop 2024-03-22T01:45:31.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.292+0000 7f034c847640 1 -- 172.21.15.72:0/2659268631 shutdown_connections 2024-03-22T01:45:31.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.292+0000 7f034c847640 1 --2- 172.21.15.72:0/2659268631 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03481013e0 0x7f034810d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.292+0000 7f034c847640 1 --2- 172.21.15.72:0/2659268631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0348100a20 0x7f0348100ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.292+0000 7f034c847640 1 --2- 172.21.15.72:0/2659268631 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0348106a30 0x7f0348106e10 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.292+0000 7f034c847640 1 -- 172.21.15.72:0/2659268631 >> 172.21.15.72:0/2659268631 conn(0x7f0348077bb0 msgr2=0x7f0348077fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:45:31.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.292+0000 7f034c847640 1 -- 172.21.15.72:0/2659268631 shutdown_connections 2024-03-22T01:45:31.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.292+0000 7f034c847640 1 -- 172.21.15.72:0/2659268631 wait complete. 2024-03-22T01:45:31.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.293+0000 7f034c847640 1 Processor -- start 2024-03-22T01:45:31.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.293+0000 7f034c847640 1 -- start start 2024-03-22T01:45:31.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.293+0000 7f034c847640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0348100a20 0x7f0348197400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:31.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.293+0000 7f034c847640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03481013e0 0x7f0348197940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:31.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.293+0000 7f034c847640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0348106a30 0x7f034819bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:31.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.293+0000 7f034c847640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f0348110e40 con 0x7f0348106a30 2024-03-22T01:45:31.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.293+0000 7f034c847640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f0348110cc0 con 0x7f0348100a20 2024-03-22T01:45:31.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.293+0000 7f034c847640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0348110fc0 con 0x7f03481013e0 2024-03-22T01:45:31.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.293+0000 7f0346d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0348106a30 0x7f034819bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:31.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.293+0000 7f0346d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0348106a30 0x7f034819bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:49040/0 (socket says 172.21.15.72:49040) 2024-03-22T01:45:31.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.293+0000 7f0346d9d640 1 -- 172.21.15.72:0/3666782460 learned_addr learned my addr 172.21.15.72:0/3666782460 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:45:31.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.293+0000 7f034659c640 1 --2- 172.21.15.72:0/3666782460 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0348100a20 0x7f0348197400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:31.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.293+0000 7f033ffff640 1 --2- 172.21.15.72:0/3666782460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03481013e0 0x7f0348197940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:31.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.293+0000 7f0346d9d640 1 -- 172.21.15.72:0/3666782460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03481013e0 msgr2=0x7f0348197940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:45:31.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.293+0000 7f0346d9d640 1 --2- 172.21.15.72:0/3666782460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03481013e0 0x7f0348197940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.293+0000 7f0346d9d640 1 -- 172.21.15.72:0/3666782460 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0348100a20 msgr2=0x7f0348197400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:45:31.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.293+0000 7f0346d9d640 1 --2- 172.21.15.72:0/3666782460 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0348100a20 0x7f0348197400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.293+0000 7f0346d9d640 1 -- 172.21.15.72:0/3666782460 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f034819c3f0 con 0x7f0348106a30 2024-03-22T01:45:31.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.294+0000 7f0346d9d640 1 --2- 172.21.15.72:0/3666782460 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0348106a30 0x7f034819bcc0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f034000eca0 tx=0x7f034000b690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:45:31.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.294+0000 7f033f7fe640 1 -- 172.21.15.72:0/3666782460 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0340004b20 con 0x7f0348106a30 2024-03-22T01:45:31.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.294+0000 7f034c847640 1 -- 172.21.15.72:0/3666782460 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f034819c6e0 con 0x7f0348106a30 2024-03-22T01:45:31.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.294+0000 7f033f7fe640 1 -- 172.21.15.72:0/3666782460 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0340004cc0 con 0x7f0348106a30 2024-03-22T01:45:31.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.294+0000 7f033f7fe640 1 -- 172.21.15.72:0/3666782460 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0340017650 con 0x7f0348106a30 2024-03-22T01:45:31.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.294+0000 7f034c847640 1 -- 172.21.15.72:0/3666782460 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f034806a230 con 0x7f0348106a30 2024-03-22T01:45:31.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.296+0000 7f033f7fe640 1 -- 172.21.15.72:0/3666782460 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 25) ==== 137985+0+0 (secure 0 0 0) 0x7f0340005020 con 0x7f0348106a30 2024-03-22T01:45:31.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.296+0000 7f034f42f740 1 Processor -- start 2024-03-22T01:45:31.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.296+0000 7f034f42f740 1 -- start start 2024-03-22T01:45:31.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.296+0000 7f033f7fe640 1 --2- 172.21.15.72:0/3666782460 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f03340a3090 0x7f03340a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:31.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.297+0000 7f034f42f740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56032f13b410 0x56032f13b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:31.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.297+0000 7f034f42f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56032f13bd30 0x56032f1443d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:31.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.297+0000 7f034f42f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56032f144910 0x56032f146d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:31.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.297+0000 7f034f42f740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x56032f09f8c0 con 0x56032f13b410 2024-03-22T01:45:31.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.297+0000 7f034f42f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56032f047c20 con 0x56032f13bd30 2024-03-22T01:45:31.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.297+0000 7f034f42f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56032f09b870 con 0x56032f144910 2024-03-22T01:45:31.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.297+0000 7f033ffff640 1 --2- 172.21.15.72:0/3666782460 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f03340a3090 0x7f03340a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:31.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.297+0000 7f034659c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56032f13b410 0x56032f13b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:31.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.297+0000 7f034659c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56032f13b410 0x56032f13b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:49056/0 (socket says 172.21.15.72:49056) 2024-03-22T01:45:31.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.297+0000 7f034659c640 1 -- 172.21.15.72:0/1136950755 learned_addr learned my addr 172.21.15.72:0/1136950755 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:45:31.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.297+0000 7f033f7fe640 1 -- 172.21.15.72:0/3666782460 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f03400d15f0 con 0x7f0348106a30 2024-03-22T01:45:31.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.297+0000 7f0346d9d640 1 --2- 172.21.15.72:0/1136950755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56032f144910 0x56032f146d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:31.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.297+0000 7f033ffff640 1 --2- 172.21.15.72:0/1136950755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56032f13bd30 0x56032f1443d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:31.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.297+0000 7f034659c640 1 -- 172.21.15.72:0/1136950755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56032f144910 msgr2=0x56032f146d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:45:31.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.297+0000 7f034659c640 1 --2- 172.21.15.72:0/1136950755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56032f144910 0x56032f146d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.297+0000 7f034659c640 1 -- 172.21.15.72:0/1136950755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56032f13bd30 msgr2=0x56032f1443d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:45:31.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.297+0000 7f034659c640 1 --2- 172.21.15.72:0/1136950755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56032f13bd30 0x56032f1443d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.297+0000 7f034659c640 1 -- 172.21.15.72:0/1136950755 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56032f06c150 con 0x56032f13b410 2024-03-22T01:45:31.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.297+0000 7f033ffff640 1 --2- 172.21.15.72:0/3666782460 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f03340a3090 0x7f03340a5550 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0348105170 tx=0x7f033000aa50 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:45:31.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.297+0000 7f034659c640 1 --2- 172.21.15.72:0/1136950755 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56032f13b410 0x56032f13b7f0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f033802cfd0 tx=0x7f0338005e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b984987c7a7f266 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:45:31.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.297+0000 7f033cff9640 1 -- 172.21.15.72:0/1136950755 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0338042070 con 0x56032f13b410 2024-03-22T01:45:31.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.297+0000 7f033cff9640 1 -- 172.21.15.72:0/1136950755 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f033802ca80 con 0x56032f13b410 2024-03-22T01:45:31.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.297+0000 7f033cff9640 1 -- 172.21.15.72:0/1136950755 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0338004f50 con 0x56032f13b410 2024-03-22T01:45:31.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.298+0000 7f034f42f740 1 -- 172.21.15.72:0/1136950755 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56032f13b410 msgr2=0x56032f13b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:45:31.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.298+0000 7f034f42f740 1 --2- 172.21.15.72:0/1136950755 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56032f13b410 0x56032f13b7f0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f033802cfd0 tx=0x7f0338005e00 comp rx=0 tx=0).stop 2024-03-22T01:45:31.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.298+0000 7f034f42f740 1 -- 172.21.15.72:0/1136950755 shutdown_connections 2024-03-22T01:45:31.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.298+0000 7f034f42f740 1 --2- 172.21.15.72:0/1136950755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56032f13bd30 0x56032f1443d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.298+0000 7f034f42f740 1 --2- 172.21.15.72:0/1136950755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56032f144910 0x56032f146d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.298+0000 7f034f42f740 1 --2- 172.21.15.72:0/1136950755 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56032f13b410 0x56032f13b7f0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.298+0000 7f034f42f740 1 -- 172.21.15.72:0/1136950755 >> 172.21.15.72:0/1136950755 conn(0x56032f135b40 msgr2=0x56032f135f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:45:31.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.298+0000 7f034f42f740 1 -- 172.21.15.72:0/1136950755 shutdown_connections 2024-03-22T01:45:31.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.298+0000 7f034f42f740 1 -- 172.21.15.72:0/1136950755 wait complete. 2024-03-22T01:45:31.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.299+0000 7f034f42f740 1 -- start start 2024-03-22T01:45:31.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.299+0000 7f034f42f740 1 Processor -- start 2024-03-22T01:45:31.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.299+0000 7f034f42f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56032f13b410 0x56032f141930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:31.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.299+0000 7f034f42f740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56032f13bd30 0x56032f141e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:31.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.299+0000 7f034f42f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56032f144910 0x56032f1423b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:31.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.299+0000 7f034f42f740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x56032f1476a0 con 0x56032f13bd30 2024-03-22T01:45:31.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.299+0000 7f034f42f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56032f05b130 con 0x56032f144910 2024-03-22T01:45:31.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.299+0000 7f034f42f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56032f147820 con 0x56032f13b410 2024-03-22T01:45:31.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.299+0000 7f034f42f740 10 client.0 _ll_register_callbacks cb 0x56032f05ab00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:45:31.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.299+0000 7f033ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56032f13bd30 0x56032f141e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:31.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.299+0000 7f033ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56032f13bd30 0x56032f141e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:49066/0 (socket says 172.21.15.72:49066) 2024-03-22T01:45:31.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.299+0000 7f033ffff640 1 -- 172.21.15.72:0/515401970 learned_addr learned my addr 172.21.15.72:0/515401970 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:45:31.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.299+0000 7f034659c640 1 --2- 172.21.15.72:0/515401970 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56032f13b410 0x56032f141930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:31.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.299+0000 7f0346d9d640 1 --2- 172.21.15.72:0/515401970 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56032f144910 0x56032f1423b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:31.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.299+0000 7f033ffff640 1 -- 172.21.15.72:0/515401970 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56032f13b410 msgr2=0x56032f141930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:45:31.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.299+0000 7f033ffff640 1 --2- 172.21.15.72:0/515401970 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56032f13b410 0x56032f141930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.299+0000 7f033ffff640 1 -- 172.21.15.72:0/515401970 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56032f144910 msgr2=0x56032f1423b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:45:31.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.299+0000 7f033ffff640 1 --2- 172.21.15.72:0/515401970 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56032f144910 0x56032f1423b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.299+0000 7f033ffff640 1 -- 172.21.15.72:0/515401970 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56032ee7d7d0 con 0x56032f13bd30 2024-03-22T01:45:31.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.299+0000 7f033ffff640 1 --2- 172.21.15.72:0/515401970 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56032f13bd30 0x56032f141e70 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f0330010b70 tx=0x7f0330004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1ea8826f17b1a8e server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:45:31.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.299+0000 7f032dffb640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:45:31.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.300+0000 7f032dffb640 1 -- 172.21.15.72:0/515401970 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0330020450 con 0x56032f13bd30 2024-03-22T01:45:31.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.300+0000 7f032dffb640 1 -- 172.21.15.72:0/515401970 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f03300205f0 con 0x56032f13bd30 2024-03-22T01:45:31.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.300+0000 7f034f42f740 1 -- 172.21.15.72:0/515401970 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f033802ca80 con 0x56032f13bd30 2024-03-22T01:45:31.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.300+0000 7f032dffb640 1 -- 172.21.15.72:0/515401970 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7f03300207f0 con 0x56032f13bd30 2024-03-22T01:45:31.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.300+0000 7f034f42f740 10 client.16095 fetch_fsmap learned FSMap version 103 2024-03-22T01:45:31.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.300+0000 7f034f42f740 1 -- 172.21.15.72:0/515401970 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=4+}) -- 0x56032edb33f0 con 0x56032f13bd30 2024-03-22T01:45:31.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.300+0000 7f032dffb640 1 -- 172.21.15.72:0/515401970 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f033802ca80 con 0x56032f13bd30 2024-03-22T01:45:31.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.300+0000 7f032dffb640 1 -- 172.21.15.72:0/515401970 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 103) ==== 8330+0+0 (secure 0 0 0) 0x7f033001c040 con 0x56032f13bd30 2024-03-22T01:45:31.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.300+0000 7f034f42f740 10 client.16095 fetch_fsmap finished waiting for FSMap version 103 2024-03-22T01:45:31.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.300+0000 7f034f42f740 10 client.16095 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T01:45:31.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.300+0000 7f034f42f740 20 client.16095 populate_metadata read hostname 'smithi072' 2024-03-22T01:45:31.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.300+0000 7f034f42f740 1 --2- 172.21.15.72:0/515401970 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x56032f154cb0 0x56032f1570d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:31.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.300+0000 7f034f42f740 4 client.16095 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T01:45:31.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.300+0000 7f034f42f740 1 -- 172.21.15.72:0/515401970 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56032f0ace00 con 0x56032f154cb0 2024-03-22T01:45:31.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.300+0000 7f034659c640 1 --2- 172.21.15.72:0/515401970 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x56032f154cb0 0x56032f1570d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:31.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.301+0000 7f032dffb640 10 client.16095 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:45:31.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.301+0000 7f034659c640 1 --2- 172.21.15.72:0/515401970 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x56032f154cb0 0x56032f1570d0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96c9c6b4d70433f server_cookie=38afcff65a3dfe33 in_seq=0 out_seq=0 2024-03-22T01:45:31.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.301+0000 7f032dffb640 1 -- 172.21.15.72:0/515401970 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56032f0ace00 con 0x56032f154cb0 2024-03-22T01:45:31.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.301+0000 7f032dffb640 10 client.16095 handle_command_reply: tid=0 2024-03-22T01:45:31.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.301+0000 7f034f42f740 1 client.16095 shutdown 2024-03-22T01:45:31.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.301+0000 7f034f42f740 1 -- 172.21.15.72:0/515401970 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56032f13bd30 msgr2=0x56032f141e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:45:31.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.301+0000 7f034f42f740 1 --2- 172.21.15.72:0/515401970 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56032f13bd30 0x56032f141e70 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f0330010b70 tx=0x7f0330004730 comp rx=0 tx=0).stop 2024-03-22T01:45:31.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.301+0000 7f034f42f740 1 -- 172.21.15.72:0/515401970 shutdown_connections 2024-03-22T01:45:31.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.301+0000 7f034f42f740 1 --2- 172.21.15.72:0/515401970 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x56032f154cb0 0x56032f1570d0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.301+0000 7f034f42f740 1 --2- 172.21.15.72:0/515401970 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56032f144910 0x56032f1423b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.301+0000 7f032dffb640 0 client.16095 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:45:31.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.301+0000 7f034f42f740 1 --2- 172.21.15.72:0/515401970 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56032f13bd30 0x56032f141e70 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.301+0000 7f034f42f740 1 --2- 172.21.15.72:0/515401970 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56032f13b410 0x56032f141930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.301+0000 7f034f42f740 1 -- 172.21.15.72:0/515401970 >> 172.21.15.72:0/515401970 conn(0x56032f135b40 msgr2=0x56032f137890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:45:31.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.301+0000 7f034f42f740 1 -- 172.21.15.72:0/515401970 shutdown_connections 2024-03-22T01:45:31.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.301+0000 7f034f42f740 1 -- 172.21.15.72:0/515401970 wait complete. 2024-03-22T01:45:31.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.301+0000 7f034f42f740 20 client.16095 trim_cache size 0 max 16384 2024-03-22T01:45:31.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.318+0000 7f034f42f740 1 Processor -- start 2024-03-22T01:45:31.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.318+0000 7f034f42f740 1 -- start start 2024-03-22T01:45:31.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.319+0000 7f034f42f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56032f13b410 0x56032f144060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:31.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.319+0000 7f034f42f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56032f13bd30 0x56032f150440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:31.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.319+0000 7f034f42f740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56032f144910 0x56032f137680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:31.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.319+0000 7f034f42f740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f03481aae40 con 0x56032f144910 2024-03-22T01:45:31.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.319+0000 7f034f42f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f03481ad6c0 con 0x56032f13bd30 2024-03-22T01:45:31.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.319+0000 7f034f42f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f03481ab4b0 con 0x56032f13b410 2024-03-22T01:45:31.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.319+0000 7f0346d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56032f144910 0x56032f137680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:31.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.319+0000 7f0346d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56032f144910 0x56032f137680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:49082/0 (socket says 172.21.15.72:49082) 2024-03-22T01:45:31.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.319+0000 7f0346d9d640 1 -- 172.21.15.72:0/1720900952 learned_addr learned my addr 172.21.15.72:0/1720900952 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:45:31.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.319+0000 7f033ffff640 1 --2- 172.21.15.72:0/1720900952 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56032f13bd30 0x56032f150440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:31.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.319+0000 7f034659c640 1 --2- 172.21.15.72:0/1720900952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56032f13b410 0x56032f144060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:31.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.319+0000 7f0346d9d640 1 -- 172.21.15.72:0/1720900952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56032f13b410 msgr2=0x56032f144060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:45:31.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.319+0000 7f0346d9d640 1 --2- 172.21.15.72:0/1720900952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56032f13b410 0x56032f144060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.319+0000 7f0346d9d640 1 -- 172.21.15.72:0/1720900952 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56032f13bd30 msgr2=0x56032f150440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:45:31.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.319+0000 7f0346d9d640 1 --2- 172.21.15.72:0/1720900952 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56032f13bd30 0x56032f150440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.319+0000 7f0346d9d640 1 -- 172.21.15.72:0/1720900952 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56032ed730f0 con 0x56032f144910 2024-03-22T01:45:31.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.319+0000 7f0346d9d640 1 --2- 172.21.15.72:0/1720900952 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56032f144910 0x56032f137680 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f0340099a90 tx=0x7f03400099b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a8702d6e17b8612 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:45:31.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.319+0000 7f032ffff640 1 -- 172.21.15.72:0/1720900952 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03400693f0 con 0x56032f144910 2024-03-22T01:45:31.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.319+0000 7f032ffff640 1 -- 172.21.15.72:0/1720900952 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f034009d630 con 0x56032f144910 2024-03-22T01:45:31.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.319+0000 7f032ffff640 1 -- 172.21.15.72:0/1720900952 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f034008d920 con 0x56032f144910 2024-03-22T01:45:31.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.320+0000 7f034f42f740 1 -- 172.21.15.72:0/1720900952 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56032f144910 msgr2=0x56032f137680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:45:31.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.320+0000 7f034f42f740 1 --2- 172.21.15.72:0/1720900952 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56032f144910 0x56032f137680 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f0340099a90 tx=0x7f03400099b0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.320+0000 7f034f42f740 1 -- 172.21.15.72:0/1720900952 shutdown_connections 2024-03-22T01:45:31.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.320+0000 7f034f42f740 1 --2- 172.21.15.72:0/1720900952 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56032f13bd30 0x56032f150440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.320+0000 7f034f42f740 1 --2- 172.21.15.72:0/1720900952 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56032f144910 0x56032f137680 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.320+0000 7f034f42f740 1 --2- 172.21.15.72:0/1720900952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56032f13b410 0x56032f144060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.320+0000 7f034f42f740 1 -- 172.21.15.72:0/1720900952 >> 172.21.15.72:0/1720900952 conn(0x56032f135b40 msgr2=0x56032f136f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:45:31.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.320+0000 7f034f42f740 1 -- 172.21.15.72:0/1720900952 shutdown_connections 2024-03-22T01:45:31.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.320+0000 7f034f42f740 1 -- 172.21.15.72:0/1720900952 wait complete. 2024-03-22T01:45:31.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.320+0000 7f034f42f740 1 -- start start 2024-03-22T01:45:31.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.320+0000 7f034f42f740 1 Processor -- start 2024-03-22T01:45:31.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.321+0000 7f034f42f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56032f13b410 0x56032f13e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:31.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.321+0000 7f034f42f740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56032f13bd30 0x56032f13eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:31.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.321+0000 7f034f42f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56032f144910 0x56032f13efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:31.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.321+0000 7f034f42f740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f03481abb80 con 0x56032f13bd30 2024-03-22T01:45:31.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.321+0000 7f034f42f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f03481acff0 con 0x56032f13b410 2024-03-22T01:45:31.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.321+0000 7f034f42f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f03481ac250 con 0x56032f144910 2024-03-22T01:45:31.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.321+0000 7f034f42f740 10 client.0 _ll_register_callbacks cb 0x7f033000a460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:45:31.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.321+0000 7f033ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56032f13bd30 0x56032f13eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:31.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.321+0000 7f034659c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56032f13b410 0x56032f13e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:31.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.321+0000 7f033ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56032f13bd30 0x56032f13eaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:49088/0 (socket says 172.21.15.72:49088) 2024-03-22T01:45:31.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.321+0000 7f033ffff640 1 -- 172.21.15.72:0/2174245935 learned_addr learned my addr 172.21.15.72:0/2174245935 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:45:31.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.321+0000 7f0346d9d640 1 --2- 172.21.15.72:0/2174245935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56032f144910 0x56032f13efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:31.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.321+0000 7f033ffff640 1 -- 172.21.15.72:0/2174245935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56032f144910 msgr2=0x56032f13efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:45:31.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.321+0000 7f033ffff640 1 --2- 172.21.15.72:0/2174245935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56032f144910 0x56032f13efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.321+0000 7f033ffff640 1 -- 172.21.15.72:0/2174245935 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56032f13b410 msgr2=0x56032f13e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:45:31.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.321+0000 7f033ffff640 1 --2- 172.21.15.72:0/2174245935 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56032f13b410 0x56032f13e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.321+0000 7f033ffff640 1 -- 172.21.15.72:0/2174245935 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56032f1ba080 con 0x56032f13bd30 2024-03-22T01:45:31.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.321+0000 7f034659c640 1 --2- 172.21.15.72:0/2174245935 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56032f13b410 0x56032f13e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:45:31.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.321+0000 7f033ffff640 1 --2- 172.21.15.72:0/2174245935 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56032f13bd30 0x56032f13eaa0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x56032f142f00 tx=0x7f0330011830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8f47f2e93c56071 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:45:31.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.321+0000 7f032dffb640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:45:31.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.321+0000 7f032dffb640 1 -- 172.21.15.72:0/2174245935 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f033002a5f0 con 0x56032f13bd30 2024-03-22T01:45:31.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.321+0000 7f032dffb640 1 -- 172.21.15.72:0/2174245935 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f0330020990 con 0x56032f13bd30 2024-03-22T01:45:31.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.321+0000 7f034f42f740 1 -- 172.21.15.72:0/2174245935 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f034009d630 con 0x56032f13bd30 2024-03-22T01:45:31.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.322+0000 7f032dffb640 1 -- 172.21.15.72:0/2174245935 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7f033002a790 con 0x56032f13bd30 2024-03-22T01:45:31.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.322+0000 7f034f42f740 10 client.16101 fetch_fsmap learned FSMap version 103 2024-03-22T01:45:31.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.322+0000 7f034f42f740 1 -- 172.21.15.72:0/2174245935 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=4+}) -- 0x56032f1bb350 con 0x56032f13bd30 2024-03-22T01:45:31.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.322+0000 7f032dffb640 1 -- 172.21.15.72:0/2174245935 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f034009d630 con 0x56032f13bd30 2024-03-22T01:45:31.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.322+0000 7f032dffb640 1 -- 172.21.15.72:0/2174245935 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 103) ==== 8330+0+0 (secure 0 0 0) 0x7f033001c040 con 0x56032f13bd30 2024-03-22T01:45:31.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.322+0000 7f034f42f740 10 client.16101 fetch_fsmap finished waiting for FSMap version 103 2024-03-22T01:45:31.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.322+0000 7f034f42f740 10 client.16101 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T01:45:31.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.322+0000 7f034f42f740 20 client.16101 populate_metadata read hostname 'smithi072' 2024-03-22T01:45:31.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.322+0000 7f034f42f740 1 --2- 172.21.15.72:0/2174245935 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x56032f1bb960 0x56032f1bdd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:31.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.322+0000 7f034f42f740 4 client.16101 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T01:45:31.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.322+0000 7f034f42f740 1 -- 172.21.15.72:0/2174245935 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56032f0a7150 con 0x56032f1bb960 2024-03-22T01:45:31.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.322+0000 7f034659c640 1 --2- 172.21.15.72:0/2174245935 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x56032f1bb960 0x56032f1bdd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:31.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.322+0000 7f032dffb640 10 client.16101 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:45:31.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.322+0000 7f034659c640 1 --2- 172.21.15.72:0/2174245935 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x56032f1bb960 0x56032f1bdd80 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ad2504b9fe2d99b server_cookie=118346ac3d1d06fb in_seq=0 out_seq=0 2024-03-22T01:45:31.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.323+0000 7f032dffb640 1 -- 172.21.15.72:0/2174245935 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x56032f0a7150 con 0x56032f1bb960 2024-03-22T01:45:31.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.323+0000 7f032dffb640 10 client.16101 handle_command_reply: tid=0 2024-03-22T01:45:31.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.323+0000 7f034f42f740 1 client.16101 shutdown 2024-03-22T01:45:31.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.323+0000 7f034f42f740 1 -- 172.21.15.72:0/2174245935 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56032f13bd30 msgr2=0x56032f13eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:45:31.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.323+0000 7f034f42f740 1 --2- 172.21.15.72:0/2174245935 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56032f13bd30 0x56032f13eaa0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x56032f142f00 tx=0x7f0330011830 comp rx=0 tx=0).stop 2024-03-22T01:45:31.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.323+0000 7f034f42f740 1 -- 172.21.15.72:0/2174245935 shutdown_connections 2024-03-22T01:45:31.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.323+0000 7f034f42f740 1 --2- 172.21.15.72:0/2174245935 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x56032f1bb960 0x56032f1bdd80 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.323+0000 7f034f42f740 1 --2- 172.21.15.72:0/2174245935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56032f144910 0x56032f13efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.323+0000 7f032dffb640 0 client.16101 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:45:31.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.323+0000 7f034f42f740 1 --2- 172.21.15.72:0/2174245935 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56032f13bd30 0x56032f13eaa0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.323+0000 7f034f42f740 1 --2- 172.21.15.72:0/2174245935 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56032f13b410 0x56032f13e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.323+0000 7f034f42f740 1 -- 172.21.15.72:0/2174245935 >> 172.21.15.72:0/2174245935 conn(0x56032f135b40 msgr2=0x56032f137800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:45:31.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.323+0000 7f034f42f740 1 -- 172.21.15.72:0/2174245935 shutdown_connections 2024-03-22T01:45:31.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.323+0000 7f034f42f740 1 -- 172.21.15.72:0/2174245935 wait complete. 2024-03-22T01:45:31.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.323+0000 7f034f42f740 20 client.16101 trim_cache size 0 max 16384 2024-03-22T01:45:31.330 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-22T01:45:31.330 INFO:teuthology.orchestra.run.smithi072.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-22T01:45:31.330 INFO:teuthology.orchestra.run.smithi072.stdout: "scrubs": { 2024-03-22T01:45:31.330 INFO:teuthology.orchestra.run.smithi072.stdout: "b92ea364-f747-42c0-973f-030d2e78c51b": { 2024-03-22T01:45:31.330 INFO:teuthology.orchestra.run.smithi072.stdout: "path": "/", 2024-03-22T01:45:31.330 INFO:teuthology.orchestra.run.smithi072.stdout: "tag": "b92ea364-f747-42c0-973f-030d2e78c51b", 2024-03-22T01:45:31.330 INFO:teuthology.orchestra.run.smithi072.stdout: "options": "recursive,force" 2024-03-22T01:45:31.330 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:45:31.331 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:45:31.331 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-22T01:45:31.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.323+0000 7f034c847640 1 -- 172.21.15.72:0/3666782460 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f03340a3090 msgr2=0x7f03340a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:45:31.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.323+0000 7f034c847640 1 --2- 172.21.15.72:0/3666782460 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f03340a3090 0x7f03340a5550 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0348105170 tx=0x7f033000aa50 comp rx=0 tx=0).stop 2024-03-22T01:45:31.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.323+0000 7f034c847640 1 -- 172.21.15.72:0/3666782460 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0348106a30 msgr2=0x7f034819bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:45:31.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.323+0000 7f034c847640 1 --2- 172.21.15.72:0/3666782460 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0348106a30 0x7f034819bcc0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f034000eca0 tx=0x7f034000b690 comp rx=0 tx=0).stop 2024-03-22T01:45:31.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.323+0000 7f034c847640 1 -- 172.21.15.72:0/3666782460 shutdown_connections 2024-03-22T01:45:31.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.323+0000 7f034c847640 1 --2- 172.21.15.72:0/3666782460 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f03340a3090 0x7f03340a5550 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.323+0000 7f034c847640 1 --2- 172.21.15.72:0/3666782460 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0348106a30 0x7f034819bcc0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.323+0000 7f034c847640 1 --2- 172.21.15.72:0/3666782460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03481013e0 0x7f0348197940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.323+0000 7f034c847640 1 --2- 172.21.15.72:0/3666782460 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0348100a20 0x7f0348197400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:31.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.323+0000 7f034c847640 1 -- 172.21.15.72:0/3666782460 >> 172.21.15.72:0/3666782460 conn(0x7f0348077bb0 msgr2=0x7f0348103310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:45:31.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.323+0000 7f034c847640 1 -- 172.21.15.72:0/3666782460 shutdown_connections 2024-03-22T01:45:31.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:45:31.323+0000 7f034c847640 1 -- 172.21.15.72:0/3666782460 wait complete. 2024-03-22T01:45:31.333 INFO:tasks.cephfs.filesystem:scrub status for tag:b92ea364-f747-42c0-973f-030d2e78c51b - {'path': '/', 'tag': 'b92ea364-f747-42c0-973f-030d2e78c51b', 'options': 'recursive,force'} 2024-03-22T01:45:32.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:31 smithi089 ceph-mon[35770]: pgmap v269: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.2 MiB/s wr, 672 op/s 2024-03-22T01:45:32.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:31 smithi089 ceph-mon[35770]: scrub queued for path: / 2024-03-22T01:45:32.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:31 smithi089 ceph-mon[35770]: scrub summary: idle+waiting paths [/] 2024-03-22T01:45:32.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:31 smithi089 ceph-mon[35770]: scrub summary: active paths [/] 2024-03-22T01:45:32.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:31 smithi089 ceph-mon[35770]: scrub summary: idle+waiting paths [/] 2024-03-22T01:45:32.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:31 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:45:32.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:31 smithi072 ceph-mon[32414]: pgmap v269: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.2 MiB/s wr, 672 op/s 2024-03-22T01:45:32.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:31 smithi072 ceph-mon[32414]: scrub queued for path: / 2024-03-22T01:45:32.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:31 smithi072 ceph-mon[32414]: scrub summary: idle+waiting paths [/] 2024-03-22T01:45:32.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:31 smithi072 ceph-mon[32414]: scrub summary: active paths [/] 2024-03-22T01:45:32.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:31 smithi072 ceph-mon[32414]: scrub summary: idle+waiting paths [/] 2024-03-22T01:45:32.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:31 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:45:32.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:31 smithi098 ceph-mon[36468]: pgmap v269: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.2 MiB/s wr, 672 op/s 2024-03-22T01:45:32.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:31 smithi098 ceph-mon[36468]: scrub queued for path: / 2024-03-22T01:45:32.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:31 smithi098 ceph-mon[36468]: scrub summary: idle+waiting paths [/] 2024-03-22T01:45:32.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:31 smithi098 ceph-mon[36468]: scrub summary: active paths [/] 2024-03-22T01:45:32.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:31 smithi098 ceph-mon[36468]: scrub summary: idle+waiting paths [/] 2024-03-22T01:45:32.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:31 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:45:33.321 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:33 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:45:33.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:33 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:45:33.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:33 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:45:33.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:33 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:45:33.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:33 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:45:33.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:33 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:45:33.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:33 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:45:33.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:33 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:45:33.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:33 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:45:34.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:34 smithi089 ceph-mon[35770]: pgmap v270: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 546 KiB/s wr, 165 op/s 2024-03-22T01:45:34.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:34 smithi089 ceph-mon[35770]: scrub summary: idle 2024-03-22T01:45:34.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:34 smithi072 ceph-mon[32414]: pgmap v270: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 546 KiB/s wr, 165 op/s 2024-03-22T01:45:34.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:34 smithi072 ceph-mon[32414]: scrub summary: idle 2024-03-22T01:45:34.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:34 smithi098 ceph-mon[36468]: pgmap v270: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 546 KiB/s wr, 165 op/s 2024-03-22T01:45:34.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:34 smithi098 ceph-mon[36468]: scrub summary: idle 2024-03-22T01:45:36.174 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.173+0000 7f480009a640 1 -- 172.21.15.72:0/3795740526 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f47f8107cc0 msgr2=0x7f47f81080a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:45:36.174 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.173+0000 7f480009a640 1 --2- 172.21.15.72:0/3795740526 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f47f8107cc0 0x7f47f81080a0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f47f4009ca0 tx=0x7f47f402c140 comp rx=0 tx=0).stop 2024-03-22T01:45:36.174 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.173+0000 7f480009a640 1 -- 172.21.15.72:0/3795740526 shutdown_connections 2024-03-22T01:45:36.174 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.173+0000 7f480009a640 1 --2- 172.21.15.72:0/3795740526 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47f8100df0 0x7f47f810e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:36.174 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.173+0000 7f480009a640 1 --2- 172.21.15.72:0/3795740526 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47f8100430 0x7f47f81008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:36.174 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.173+0000 7f480009a640 1 --2- 172.21.15.72:0/3795740526 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f47f8107cc0 0x7f47f81080a0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:36.175 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.173+0000 7f480009a640 1 -- 172.21.15.72:0/3795740526 >> 172.21.15.72:0/3795740526 conn(0x7f47f8075db0 msgr2=0x7f47f80761c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:45:36.175 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.173+0000 7f480009a640 1 -- 172.21.15.72:0/3795740526 shutdown_connections 2024-03-22T01:45:36.175 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.173+0000 7f480009a640 1 -- 172.21.15.72:0/3795740526 wait complete. 2024-03-22T01:45:36.175 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.173+0000 7f480009a640 1 Processor -- start 2024-03-22T01:45:36.175 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.173+0000 7f480009a640 1 -- start start 2024-03-22T01:45:36.175 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.174+0000 7f480009a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f47f8100430 0x7f47f8101610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:36.175 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.174+0000 7f480009a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47f8100df0 0x7f47f8101b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:36.175 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.174+0000 7f480009a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47f8107cc0 0x7f47f8102090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:36.175 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.174+0000 7f480009a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f47f8111fe0 con 0x7f47f8100430 2024-03-22T01:45:36.175 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.174+0000 7f480009a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f47f8111e60 con 0x7f47f8100df0 2024-03-22T01:45:36.175 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.174+0000 7f480009a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f47f8112160 con 0x7f47f8107cc0 2024-03-22T01:45:36.175 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.174+0000 7f47fde36640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f47f8100430 0x7f47f8101610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:36.176 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.174+0000 7f47fde36640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f47f8100430 0x7f47f8101610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:49104/0 (socket says 172.21.15.72:49104) 2024-03-22T01:45:36.176 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.174+0000 7f47fde36640 1 -- 172.21.15.72:0/1156420786 learned_addr learned my addr 172.21.15.72:0/1156420786 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:45:36.176 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.174+0000 7f47fd635640 1 --2- 172.21.15.72:0/1156420786 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47f8100df0 0x7f47f8101b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:36.176 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.174+0000 7f47fe637640 1 --2- 172.21.15.72:0/1156420786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47f8107cc0 0x7f47f8102090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:36.176 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.174+0000 7f47fde36640 1 -- 172.21.15.72:0/1156420786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47f8107cc0 msgr2=0x7f47f8102090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:45:36.176 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.174+0000 7f47fde36640 1 --2- 172.21.15.72:0/1156420786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47f8107cc0 0x7f47f8102090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:36.176 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.174+0000 7f47fde36640 1 -- 172.21.15.72:0/1156420786 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47f8100df0 msgr2=0x7f47f8101b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:45:36.176 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.174+0000 7f47fde36640 1 --2- 172.21.15.72:0/1156420786 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47f8100df0 0x7f47f8101b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:36.176 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.174+0000 7f47fde36640 1 -- 172.21.15.72:0/1156420786 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47f81a69e0 con 0x7f47f8100430 2024-03-22T01:45:36.176 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.174+0000 7f47fde36640 1 --2- 172.21.15.72:0/1156420786 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f47f8100430 0x7f47f8101610 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f47f4005cc0 tx=0x7f47f40346f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:45:36.176 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.174+0000 7f47eeffd640 1 -- 172.21.15.72:0/1156420786 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f4039050 con 0x7f47f8100430 2024-03-22T01:45:36.176 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.174+0000 7f47eeffd640 1 -- 172.21.15.72:0/1156420786 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47f4045070 con 0x7f47f8100430 2024-03-22T01:45:36.176 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.174+0000 7f480009a640 1 -- 172.21.15.72:0/1156420786 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47f81a6c70 con 0x7f47f8100430 2024-03-22T01:45:36.176 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.174+0000 7f47eeffd640 1 -- 172.21.15.72:0/1156420786 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f4004680 con 0x7f47f8100430 2024-03-22T01:45:36.177 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.175+0000 7f480009a640 1 -- 172.21.15.72:0/1156420786 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47f81a71b0 con 0x7f47f8100430 2024-03-22T01:45:36.183 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.177+0000 7f47eeffd640 1 -- 172.21.15.72:0/1156420786 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 25) ==== 137985+0+0 (secure 0 0 0) 0x7f47f4003970 con 0x7f47f8100430 2024-03-22T01:45:36.183 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.177+0000 7f4802922740 1 Processor -- start 2024-03-22T01:45:36.183 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.177+0000 7f4802922740 1 -- start start 2024-03-22T01:45:36.183 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.177+0000 7f47eeffd640 1 --2- 172.21.15.72:0/1156420786 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f47e40a3130 0x7f47e40a55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:36.183 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.178+0000 7f47eeffd640 1 -- 172.21.15.72:0/1156420786 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f47f40f1ff0 con 0x7f47f8100430 2024-03-22T01:45:36.184 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.178+0000 7f4802922740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563518cf2d0 0x5563518cf6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:36.184 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.178+0000 7f47fe637640 1 --2- 172.21.15.72:0/1156420786 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f47e40a3130 0x7f47e40a55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:36.184 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.178+0000 7f4802922740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5563518cfbf0 0x5563518d8480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:36.184 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.178+0000 7f4802922740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5563518d89c0 0x5563518dadb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:36.184 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.178+0000 7f4802922740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x556351833040 con 0x5563518cfbf0 2024-03-22T01:45:36.184 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.178+0000 7f4802922740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5563517db9e0 con 0x5563518cf2d0 2024-03-22T01:45:36.184 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.178+0000 7f4802922740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55635182eff0 con 0x5563518d89c0 2024-03-22T01:45:36.184 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.178+0000 7f47fd635640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5563518cfbf0 0x5563518d8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:36.184 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.178+0000 7f47fde36640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563518cf2d0 0x5563518cf6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:36.184 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.178+0000 7f47fd635640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5563518cfbf0 0x5563518d8480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:49116/0 (socket says 172.21.15.72:49116) 2024-03-22T01:45:36.184 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.178+0000 7f47fde36640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563518cf2d0 0x5563518cf6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.72:46968/0 (socket says 172.21.15.72:46968) 2024-03-22T01:45:36.184 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.178+0000 7f47fd635640 1 -- 172.21.15.72:0/725884365 learned_addr learned my addr 172.21.15.72:0/725884365 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:45:36.184 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.178+0000 7f47fe637640 1 --2- 172.21.15.72:0/725884365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5563518d89c0 0x5563518dadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:36.184 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.178+0000 7f47fde36640 1 -- 172.21.15.72:0/725884365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5563518d89c0 msgr2=0x5563518dadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:45:36.185 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.178+0000 7f47fde36640 1 --2- 172.21.15.72:0/725884365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5563518d89c0 0x5563518dadb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:36.185 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.178+0000 7f47fde36640 1 -- 172.21.15.72:0/725884365 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5563518cfbf0 msgr2=0x5563518d8480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:45:36.185 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.178+0000 7f47fde36640 1 --2- 172.21.15.72:0/725884365 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5563518cfbf0 0x5563518d8480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:36.185 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.178+0000 7f47fde36640 1 -- 172.21.15.72:0/725884365 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556351547210 con 0x5563518cf2d0 2024-03-22T01:45:36.185 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.178+0000 7f47fe637640 1 --2- 172.21.15.72:0/1156420786 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f47e40a3130 0x7f47e40a55f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f47f0009b60 tx=0x7f47f0009f60 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:45:36.185 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.178+0000 7f47fde36640 1 --2- 172.21.15.72:0/725884365 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563518cf2d0 0x5563518cf6b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f47f40ef4a0 tx=0x7f47f40bdd40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c600896d027ddc19 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:45:36.185 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.178+0000 7f47e3fff640 1 -- 172.21.15.72:0/725884365 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f40be840 con 0x5563518cf2d0 2024-03-22T01:45:36.185 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.178+0000 7f47e3fff640 1 -- 172.21.15.72:0/725884365 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47f403a040 con 0x5563518cf2d0 2024-03-22T01:45:36.185 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.178+0000 7f47e3fff640 1 -- 172.21.15.72:0/725884365 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f40bc4d0 con 0x5563518cf2d0 2024-03-22T01:45:36.185 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.179+0000 7f4802922740 1 -- 172.21.15.72:0/725884365 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563518cf2d0 msgr2=0x5563518cf6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:45:36.185 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.179+0000 7f4802922740 1 --2- 172.21.15.72:0/725884365 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563518cf2d0 0x5563518cf6b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f47f40ef4a0 tx=0x7f47f40bdd40 comp rx=0 tx=0).stop 2024-03-22T01:45:36.185 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.179+0000 7f4802922740 1 -- 172.21.15.72:0/725884365 shutdown_connections 2024-03-22T01:45:36.185 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.179+0000 7f4802922740 1 --2- 172.21.15.72:0/725884365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5563518d89c0 0x5563518dadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:36.185 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.179+0000 7f4802922740 1 --2- 172.21.15.72:0/725884365 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5563518cfbf0 0x5563518d8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:36.186 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.179+0000 7f4802922740 1 --2- 172.21.15.72:0/725884365 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563518cf2d0 0x5563518cf6b0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:36.186 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.179+0000 7f4802922740 1 -- 172.21.15.72:0/725884365 >> 172.21.15.72:0/725884365 conn(0x5563518c9bf0 msgr2=0x5563518c9fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:45:36.186 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.179+0000 7f4802922740 1 -- 172.21.15.72:0/725884365 shutdown_connections 2024-03-22T01:45:36.186 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.179+0000 7f4802922740 1 -- 172.21.15.72:0/725884365 wait complete. 2024-03-22T01:45:36.186 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.180+0000 7f4802922740 1 -- start start 2024-03-22T01:45:36.186 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.180+0000 7f4802922740 1 Processor -- start 2024-03-22T01:45:36.186 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.180+0000 7f4802922740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5563518cf2d0 0x5563518d82b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:36.186 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.180+0000 7f4802922740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563518cfbf0 0x5563518d5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:36.186 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.180+0000 7f4802922740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5563518d89c0 0x5563518d5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:36.186 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.180+0000 7f4802922740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x5563518db3b0 con 0x5563518d89c0 2024-03-22T01:45:36.186 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.180+0000 7f4802922740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5563517eedb0 con 0x5563518cfbf0 2024-03-22T01:45:36.186 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.180+0000 7f4802922740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5563518db530 con 0x5563518cf2d0 2024-03-22T01:45:36.186 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.180+0000 7f4802922740 10 client.0 _ll_register_callbacks cb 0x556351546a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:45:36.187 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.180+0000 7f47fe637640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5563518d89c0 0x5563518d5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:36.187 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.180+0000 7f47fe637640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5563518d89c0 0x5563518d5b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:49126/0 (socket says 172.21.15.72:49126) 2024-03-22T01:45:36.187 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.180+0000 7f47fe637640 1 -- 172.21.15.72:0/2535112090 learned_addr learned my addr 172.21.15.72:0/2535112090 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:45:36.187 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.180+0000 7f47fd635640 1 --2- 172.21.15.72:0/2535112090 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563518cfbf0 0x5563518d5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:36.187 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.180+0000 7f47fde36640 1 --2- 172.21.15.72:0/2535112090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5563518cf2d0 0x5563518d82b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:36.187 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.181+0000 7f47fe637640 1 -- 172.21.15.72:0/2535112090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5563518cf2d0 msgr2=0x5563518d82b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:45:36.187 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.181+0000 7f47fe637640 1 --2- 172.21.15.72:0/2535112090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5563518cf2d0 0x5563518d82b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:36.187 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.181+0000 7f47fe637640 1 -- 172.21.15.72:0/2535112090 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563518cfbf0 msgr2=0x5563518d5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:45:36.187 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.181+0000 7f47fe637640 1 --2- 172.21.15.72:0/2535112090 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563518cfbf0 0x5563518d5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:36.187 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.181+0000 7f47fe637640 1 -- 172.21.15.72:0/2535112090 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556351613620 con 0x5563518d89c0 2024-03-22T01:45:36.187 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.181+0000 7f47fd635640 1 --2- 172.21.15.72:0/2535112090 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563518cfbf0 0x5563518d5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:45:36.187 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.181+0000 7f47fe637640 1 --2- 172.21.15.72:0/2535112090 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5563518d89c0 0x5563518d5b80 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f47f0003620 tx=0x7f47f0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e0880e2e86bc7da server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:45:36.187 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.181+0000 7f47e17fa640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:45:36.187 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.181+0000 7f47e17fa640 1 -- 172.21.15.72:0/2535112090 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f002a060 con 0x5563518d89c0 2024-03-22T01:45:36.188 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.181+0000 7f47e17fa640 1 -- 172.21.15.72:0/2535112090 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f47f001d780 con 0x5563518d89c0 2024-03-22T01:45:36.188 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.181+0000 7f4802922740 1 -- 172.21.15.72:0/2535112090 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f47f403a040 con 0x5563518d89c0 2024-03-22T01:45:36.188 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.181+0000 7f47e17fa640 1 -- 172.21.15.72:0/2535112090 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7f47f0031050 con 0x5563518d89c0 2024-03-22T01:45:36.188 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.181+0000 7f4802922740 10 client.16113 fetch_fsmap learned FSMap version 103 2024-03-22T01:45:36.188 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.181+0000 7f4802922740 1 -- 172.21.15.72:0/2535112090 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=4+}) -- 0x5563518e72a0 con 0x5563518d89c0 2024-03-22T01:45:36.188 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.181+0000 7f47e17fa640 1 -- 172.21.15.72:0/2535112090 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47f403a040 con 0x5563518d89c0 2024-03-22T01:45:36.188 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.182+0000 7f47e17fa640 1 -- 172.21.15.72:0/2535112090 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 103) ==== 8330+0+0 (secure 0 0 0) 0x7f47f0013830 con 0x5563518d89c0 2024-03-22T01:45:36.188 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.182+0000 7f4802922740 10 client.16113 fetch_fsmap finished waiting for FSMap version 103 2024-03-22T01:45:36.188 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.182+0000 7f4802922740 10 client.16113 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T01:45:36.188 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.182+0000 7f4802922740 20 client.16113 populate_metadata read hostname 'smithi072' 2024-03-22T01:45:36.188 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.182+0000 7f4802922740 1 --2- 172.21.15.72:0/2535112090 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x5563518e87b0 0x5563518eaba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:36.188 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.182+0000 7f4802922740 4 client.16113 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T01:45:36.188 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.182+0000 7f4802922740 1 -- 172.21.15.72:0/2535112090 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556351840580 con 0x5563518e87b0 2024-03-22T01:45:36.189 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.182+0000 7f47fde36640 1 --2- 172.21.15.72:0/2535112090 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x5563518e87b0 0x5563518eaba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:36.189 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.182+0000 7f47fde36640 1 --2- 172.21.15.72:0/2535112090 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x5563518e87b0 0x5563518eaba0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c17d2d8f7c82f0f1 server_cookie=bea9baf677beff84 in_seq=0 out_seq=0 2024-03-22T01:45:36.189 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.182+0000 7f47e17fa640 10 client.16113 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:45:36.189 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.183+0000 7f47e17fa640 1 -- 172.21.15.72:0/2535112090 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556351840580 con 0x5563518e87b0 2024-03-22T01:45:36.189 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.183+0000 7f47e17fa640 10 client.16113 handle_command_reply: tid=0 2024-03-22T01:45:36.189 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.183+0000 7f4802922740 1 client.16113 shutdown 2024-03-22T01:45:36.189 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.183+0000 7f4802922740 1 -- 172.21.15.72:0/2535112090 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5563518d89c0 msgr2=0x5563518d5b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:45:36.189 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.183+0000 7f4802922740 1 --2- 172.21.15.72:0/2535112090 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5563518d89c0 0x5563518d5b80 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f47f0003620 tx=0x7f47f0004730 comp rx=0 tx=0).stop 2024-03-22T01:45:36.190 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.183+0000 7f4802922740 1 -- 172.21.15.72:0/2535112090 shutdown_connections 2024-03-22T01:45:36.190 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.183+0000 7f4802922740 1 --2- 172.21.15.72:0/2535112090 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x5563518e87b0 0x5563518eaba0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:36.190 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.183+0000 7f4802922740 1 --2- 172.21.15.72:0/2535112090 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563518cfbf0 0x5563518d5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:36.190 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.183+0000 7f4802922740 1 --2- 172.21.15.72:0/2535112090 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5563518d89c0 0x5563518d5b80 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:36.190 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.183+0000 7f4802922740 1 --2- 172.21.15.72:0/2535112090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5563518cf2d0 0x5563518d82b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:36.190 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.183+0000 7f47e17fa640 0 client.16113 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:45:36.190 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.183+0000 7f4802922740 1 -- 172.21.15.72:0/2535112090 >> 172.21.15.72:0/2535112090 conn(0x5563518c9bf0 msgr2=0x5563518cb940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:45:36.190 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.183+0000 7f4802922740 1 -- 172.21.15.72:0/2535112090 shutdown_connections 2024-03-22T01:45:36.190 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.183+0000 7f4802922740 1 -- 172.21.15.72:0/2535112090 wait complete. 2024-03-22T01:45:36.190 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.183+0000 7f4802922740 20 client.16113 trim_cache size 0 max 16384 2024-03-22T01:45:36.203 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.198+0000 7f4802922740 1 Processor -- start 2024-03-22T01:45:36.203 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.198+0000 7f4802922740 1 -- start start 2024-03-22T01:45:36.203 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.198+0000 7f4802922740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5563518c9bf0 0x5563518e4030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:36.203 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.198+0000 7f4802922740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563518cf2d0 0x5563518c8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:36.203 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.198+0000 7f4802922740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5563518cfbf0 0x5563518f04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:36.203 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.198+0000 7f4802922740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f47f81af200 con 0x5563518cfbf0 2024-03-22T01:45:36.203 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.198+0000 7f4802922740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f47f81b1a80 con 0x5563518cf2d0 2024-03-22T01:45:36.203 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.198+0000 7f4802922740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f47f81af870 con 0x5563518c9bf0 2024-03-22T01:45:36.203 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.198+0000 7f47fe637640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5563518cfbf0 0x5563518f04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:36.203 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.198+0000 7f47fe637640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5563518cfbf0 0x5563518f04b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:49134/0 (socket says 172.21.15.72:49134) 2024-03-22T01:45:36.204 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.198+0000 7f47fe637640 1 -- 172.21.15.72:0/4205625281 learned_addr learned my addr 172.21.15.72:0/4205625281 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:45:36.204 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.198+0000 7f47fd635640 1 --2- 172.21.15.72:0/4205625281 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563518cf2d0 0x5563518c8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:36.204 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.198+0000 7f47fde36640 1 --2- 172.21.15.72:0/4205625281 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5563518c9bf0 0x5563518e4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:36.204 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.198+0000 7f47fe637640 1 -- 172.21.15.72:0/4205625281 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5563518c9bf0 msgr2=0x5563518e4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:45:36.204 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.198+0000 7f47fe637640 1 --2- 172.21.15.72:0/4205625281 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5563518c9bf0 0x5563518e4030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:36.204 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.198+0000 7f47fe637640 1 -- 172.21.15.72:0/4205625281 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563518cf2d0 msgr2=0x5563518c8ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:45:36.204 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.198+0000 7f47fe637640 1 --2- 172.21.15.72:0/4205625281 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563518cf2d0 0x5563518c8ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:36.204 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.198+0000 7f47fe637640 1 -- 172.21.15.72:0/4205625281 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563518d6180 con 0x5563518cfbf0 2024-03-22T01:45:36.204 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.198+0000 7f47fde36640 1 --2- 172.21.15.72:0/4205625281 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5563518c9bf0 0x5563518e4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T01:45:36.205 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.199+0000 7f47fe637640 1 --2- 172.21.15.72:0/4205625281 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5563518cfbf0 0x5563518f04b0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f47f00118f0 tx=0x7f47f00044a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ee9746c46118b00 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:45:36.205 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.199+0000 7f47e37fe640 1 -- 172.21.15.72:0/4205625281 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f003a070 con 0x5563518cfbf0 2024-03-22T01:45:36.205 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.199+0000 7f47e37fe640 1 -- 172.21.15.72:0/4205625281 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47f0019050 con 0x5563518cfbf0 2024-03-22T01:45:36.205 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.199+0000 7f47e37fe640 1 -- 172.21.15.72:0/4205625281 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f0017040 con 0x5563518cfbf0 2024-03-22T01:45:36.205 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.199+0000 7f4802922740 1 -- 172.21.15.72:0/4205625281 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5563518cfbf0 msgr2=0x5563518f04b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:45:36.205 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.199+0000 7f4802922740 1 --2- 172.21.15.72:0/4205625281 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5563518cfbf0 0x5563518f04b0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f47f00118f0 tx=0x7f47f00044a0 comp rx=0 tx=0).stop 2024-03-22T01:45:36.205 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.199+0000 7f4802922740 1 -- 172.21.15.72:0/4205625281 shutdown_connections 2024-03-22T01:45:36.205 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.199+0000 7f4802922740 1 --2- 172.21.15.72:0/4205625281 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563518cf2d0 0x5563518c8ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:36.205 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.199+0000 7f4802922740 1 --2- 172.21.15.72:0/4205625281 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5563518cfbf0 0x5563518f04b0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:36.205 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.199+0000 7f4802922740 1 --2- 172.21.15.72:0/4205625281 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5563518c9bf0 0x5563518e4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:36.205 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.199+0000 7f4802922740 1 -- 172.21.15.72:0/4205625281 >> 172.21.15.72:0/4205625281 conn(0x5563518d5ce0 msgr2=0x5563518c83c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:45:36.205 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.199+0000 7f4802922740 1 -- 172.21.15.72:0/4205625281 shutdown_connections 2024-03-22T01:45:36.205 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.199+0000 7f4802922740 1 -- 172.21.15.72:0/4205625281 wait complete. 2024-03-22T01:45:36.205 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.200+0000 7f4802922740 1 -- start start 2024-03-22T01:45:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.200+0000 7f4802922740 1 Processor -- start 2024-03-22T01:45:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.200+0000 7f4802922740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5563518c9bf0 0x5563518e0930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.200+0000 7f4802922740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5563518cf2d0 0x5563518d1f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.200+0000 7f4802922740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563518cfbf0 0x5563518d2470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.200+0000 7f4802922740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f47f81aff40 con 0x5563518cf2d0 2024-03-22T01:45:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.200+0000 7f4802922740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f47f81b13b0 con 0x5563518cfbf0 2024-03-22T01:45:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.200+0000 7f4802922740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f47f81b0610 con 0x5563518c9bf0 2024-03-22T01:45:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.200+0000 7f4802922740 10 client.0 _ll_register_callbacks cb 0x7f47f0004570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:45:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.200+0000 7f47fd635640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5563518cf2d0 0x5563518d1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.200+0000 7f47fd635640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5563518cf2d0 0x5563518d1f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:49140/0 (socket says 172.21.15.72:49140) 2024-03-22T01:45:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.200+0000 7f47fd635640 1 -- 172.21.15.72:0/3133524100 learned_addr learned my addr 172.21.15.72:0/3133524100 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:45:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.200+0000 7f47fe637640 1 --2- 172.21.15.72:0/3133524100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563518cfbf0 0x5563518d2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.200+0000 7f47fde36640 1 --2- 172.21.15.72:0/3133524100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5563518c9bf0 0x5563518e0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.201+0000 7f47fd635640 1 -- 172.21.15.72:0/3133524100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5563518c9bf0 msgr2=0x5563518e0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:45:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.201+0000 7f47fd635640 1 --2- 172.21.15.72:0/3133524100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5563518c9bf0 0x5563518e0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.201+0000 7f47fd635640 1 -- 172.21.15.72:0/3133524100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563518cfbf0 msgr2=0x5563518d2470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:45:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.201+0000 7f47fd635640 1 --2- 172.21.15.72:0/3133524100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563518cfbf0 0x5563518d2470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.201+0000 7f47fd635640 1 -- 172.21.15.72:0/3133524100 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5563518d2ca0 con 0x5563518cf2d0 2024-03-22T01:45:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.201+0000 7f47fe637640 1 --2- 172.21.15.72:0/3133524100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563518cfbf0 0x5563518d2470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:45:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.201+0000 7f47fd635640 1 --2- 172.21.15.72:0/3133524100 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5563518cf2d0 0x5563518d1f30 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f47e8005fd0 tx=0x7f47e8003a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b594c2589c763e87 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:45:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.201+0000 7f47e17fa640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:45:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.201+0000 7f47e17fa640 1 -- 172.21.15.72:0/3133524100 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47e8012860 con 0x5563518cf2d0 2024-03-22T01:45:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.201+0000 7f47e17fa640 1 -- 172.21.15.72:0/3133524100 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f47e8012a00 con 0x5563518cf2d0 2024-03-22T01:45:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.201+0000 7f4802922740 1 -- 172.21.15.72:0/3133524100 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f47f0019050 con 0x5563518cf2d0 2024-03-22T01:45:36.208 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.201+0000 7f47e17fa640 1 -- 172.21.15.72:0/3133524100 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7f47e802a050 con 0x5563518cf2d0 2024-03-22T01:45:36.208 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.201+0000 7f4802922740 10 client.16119 fetch_fsmap learned FSMap version 103 2024-03-22T01:45:36.208 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.201+0000 7f4802922740 1 -- 172.21.15.72:0/3133524100 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=4+}) -- 0x55635194eaf0 con 0x5563518cf2d0 2024-03-22T01:45:36.208 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.201+0000 7f47e17fa640 1 -- 172.21.15.72:0/3133524100 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47f0019050 con 0x5563518cf2d0 2024-03-22T01:45:36.208 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.201+0000 7f47e17fa640 1 -- 172.21.15.72:0/3133524100 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 103) ==== 8330+0+0 (secure 0 0 0) 0x7f47e8004ae0 con 0x5563518cf2d0 2024-03-22T01:45:36.208 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.201+0000 7f4802922740 10 client.16119 fetch_fsmap finished waiting for FSMap version 103 2024-03-22T01:45:36.208 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.201+0000 7f4802922740 10 client.16119 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T01:45:36.208 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.202+0000 7f4802922740 20 client.16119 populate_metadata read hostname 'smithi072' 2024-03-22T01:45:36.208 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.202+0000 7f4802922740 1 --2- 172.21.15.72:0/3133524100 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x5563518d89c0 0x556351951360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:36.208 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.202+0000 7f4802922740 4 client.16119 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T01:45:36.208 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.202+0000 7f4802922740 1 -- 172.21.15.72:0/3133524100 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55635183a8d0 con 0x5563518d89c0 2024-03-22T01:45:36.208 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.202+0000 7f47fde36640 1 --2- 172.21.15.72:0/3133524100 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x5563518d89c0 0x556351951360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:36.209 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.202+0000 7f47e17fa640 10 client.16119 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:45:36.209 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.202+0000 7f47fde36640 1 --2- 172.21.15.72:0/3133524100 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x5563518d89c0 0x556351951360 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4cfe45835f13b816 server_cookie=10a230b34dce411a in_seq=0 out_seq=0 2024-03-22T01:45:36.222 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.222+0000 7f47e17fa640 1 -- 172.21.15.72:0/3133524100 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55635183a8d0 con 0x5563518d89c0 2024-03-22T01:45:36.222 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.222+0000 7f47e17fa640 10 client.16119 handle_command_reply: tid=0 2024-03-22T01:45:36.222 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.222+0000 7f4802922740 1 client.16119 shutdown 2024-03-22T01:45:36.222 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.222+0000 7f4802922740 1 -- 172.21.15.72:0/3133524100 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5563518cf2d0 msgr2=0x5563518d1f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:45:36.223 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.222+0000 7f4802922740 1 --2- 172.21.15.72:0/3133524100 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5563518cf2d0 0x5563518d1f30 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f47e8005fd0 tx=0x7f47e8003a80 comp rx=0 tx=0).stop 2024-03-22T01:45:36.223 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.222+0000 7f4802922740 1 -- 172.21.15.72:0/3133524100 shutdown_connections 2024-03-22T01:45:36.223 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.222+0000 7f4802922740 1 --2- 172.21.15.72:0/3133524100 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x5563518d89c0 0x556351951360 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:36.223 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.222+0000 7f4802922740 1 --2- 172.21.15.72:0/3133524100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563518cfbf0 0x5563518d2470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:36.223 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.222+0000 7f4802922740 1 --2- 172.21.15.72:0/3133524100 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5563518cf2d0 0x5563518d1f30 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:36.223 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.222+0000 7f4802922740 1 --2- 172.21.15.72:0/3133524100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5563518c9bf0 0x5563518e0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:36.223 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.222+0000 7f47e17fa640 0 client.16119 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:45:36.224 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.222+0000 7f4802922740 1 -- 172.21.15.72:0/3133524100 >> 172.21.15.72:0/3133524100 conn(0x5563518d5ce0 msgr2=0x5563518cb8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:45:36.224 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.222+0000 7f4802922740 1 -- 172.21.15.72:0/3133524100 shutdown_connections 2024-03-22T01:45:36.224 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.222+0000 7f4802922740 1 -- 172.21.15.72:0/3133524100 wait complete. 2024-03-22T01:45:36.224 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.222+0000 7f4802922740 20 client.16119 trim_cache size 0 max 16384 2024-03-22T01:45:36.224 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout:{ 2024-03-22T01:45:36.225 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout: "message": "", 2024-03-22T01:45:36.225 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout: "return_code": 0 2024-03-22T01:45:36.225 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout:} 2024-03-22T01:45:36.227 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.223+0000 7f480009a640 1 -- 172.21.15.72:0/1156420786 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f47e40a3130 msgr2=0x7f47e40a55f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:45:36.227 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.223+0000 7f480009a640 1 --2- 172.21.15.72:0/1156420786 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f47e40a3130 0x7f47e40a55f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f47f0009b60 tx=0x7f47f0009f60 comp rx=0 tx=0).stop 2024-03-22T01:45:36.227 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.223+0000 7f480009a640 1 -- 172.21.15.72:0/1156420786 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f47f8100430 msgr2=0x7f47f8101610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:45:36.227 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.223+0000 7f480009a640 1 --2- 172.21.15.72:0/1156420786 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f47f8100430 0x7f47f8101610 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f47f4005cc0 tx=0x7f47f40346f0 comp rx=0 tx=0).stop 2024-03-22T01:45:36.227 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.223+0000 7f480009a640 1 -- 172.21.15.72:0/1156420786 shutdown_connections 2024-03-22T01:45:36.227 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.223+0000 7f480009a640 1 --2- 172.21.15.72:0/1156420786 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f47e40a3130 0x7f47e40a55f0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:36.228 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.223+0000 7f480009a640 1 --2- 172.21.15.72:0/1156420786 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47f8100df0 0x7f47f8101b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:36.228 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.223+0000 7f480009a640 1 --2- 172.21.15.72:0/1156420786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47f8107cc0 0x7f47f8102090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:36.228 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.223+0000 7f480009a640 1 --2- 172.21.15.72:0/1156420786 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f47f8100430 0x7f47f8101610 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:36.228 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.223+0000 7f480009a640 1 -- 172.21.15.72:0/1156420786 >> 172.21.15.72:0/1156420786 conn(0x7f47f8075db0 msgr2=0x7f47f80ff040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:45:36.228 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.223+0000 7f480009a640 1 -- 172.21.15.72:0/1156420786 shutdown_connections 2024-03-22T01:45:36.228 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:36.223+0000 7f480009a640 1 -- 172.21.15.72:0/1156420786 wait complete. 2024-03-22T01:45:36.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:36 smithi089 ceph-mon[35770]: pgmap v271: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 160 KiB/s wr, 73 op/s 2024-03-22T01:45:36.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:36 smithi072 ceph-mon[32414]: pgmap v271: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 160 KiB/s wr, 73 op/s 2024-03-22T01:45:36.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:36 smithi098 ceph-mon[36468]: pgmap v271: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 160 KiB/s wr, 73 op/s 2024-03-22T01:45:38.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:38 smithi089 ceph-mon[35770]: pgmap v272: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 80 KiB/s wr, 78 op/s 2024-03-22T01:45:38.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:38 smithi072 ceph-mon[32414]: pgmap v272: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 80 KiB/s wr, 78 op/s 2024-03-22T01:45:38.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:38 smithi098 ceph-mon[36468]: pgmap v272: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 80 KiB/s wr, 78 op/s 2024-03-22T01:45:40.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:40 smithi089 ceph-mon[35770]: pgmap v273: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 6.7 KiB/s wr, 39 op/s 2024-03-22T01:45:40.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:40 smithi072 ceph-mon[32414]: pgmap v273: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 6.7 KiB/s wr, 39 op/s 2024-03-22T01:45:40.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:40 smithi098 ceph-mon[36468]: pgmap v273: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 6.7 KiB/s wr, 39 op/s 2024-03-22T01:45:42.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:42 smithi089 ceph-mon[35770]: pgmap v274: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 9.5 KiB/s wr, 41 op/s 2024-03-22T01:45:42.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:42 smithi072 ceph-mon[32414]: pgmap v274: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 9.5 KiB/s wr, 41 op/s 2024-03-22T01:45:42.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:42 smithi098 ceph-mon[36468]: pgmap v274: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 9.5 KiB/s wr, 41 op/s 2024-03-22T01:45:44.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:44 smithi089 ceph-mon[35770]: pgmap v275: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 17 KiB/s wr, 43 op/s 2024-03-22T01:45:44.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:44 smithi072 ceph-mon[32414]: pgmap v275: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 17 KiB/s wr, 43 op/s 2024-03-22T01:45:44.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:44 smithi098 ceph-mon[36468]: pgmap v275: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 17 KiB/s wr, 43 op/s 2024-03-22T01:45:46.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:46 smithi089 ceph-mon[35770]: pgmap v276: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 11 KiB/s wr, 35 op/s 2024-03-22T01:45:46.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:46 smithi072 ceph-mon[32414]: pgmap v276: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 11 KiB/s wr, 35 op/s 2024-03-22T01:45:46.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:46 smithi098 ceph-mon[36468]: pgmap v276: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 11 KiB/s wr, 35 op/s 2024-03-22T01:45:48.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:48 smithi089 ceph-mon[35770]: pgmap v277: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 14 KiB/s wr, 34 op/s 2024-03-22T01:45:48.374 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:48 smithi072 ceph-mon[32414]: pgmap v277: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 14 KiB/s wr, 34 op/s 2024-03-22T01:45:48.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:48 smithi098 ceph-mon[36468]: pgmap v277: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 14 KiB/s wr, 34 op/s 2024-03-22T01:45:49.321 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.319+0000 7f9b53aeb640 1 -- 172.21.15.72:0/541950265 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b4c106ba0 msgr2=0x7f9b4c106f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:45:49.321 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.320+0000 7f9b53aeb640 1 --2- 172.21.15.72:0/541950265 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b4c106ba0 0x7f9b4c106f80 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f9b48009900 tx=0x7f9b4802bc30 comp rx=0 tx=0).stop 2024-03-22T01:45:49.321 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.320+0000 7f9b53aeb640 1 -- 172.21.15.72:0/541950265 shutdown_connections 2024-03-22T01:45:49.321 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.320+0000 7f9b53aeb640 1 --2- 172.21.15.72:0/541950265 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b4c100ce0 0x7f9b4c101160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:49.321 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.320+0000 7f9b53aeb640 1 --2- 172.21.15.72:0/541950265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b4c1016a0 0x7f9b4c10d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:49.322 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.320+0000 7f9b53aeb640 1 --2- 172.21.15.72:0/541950265 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b4c106ba0 0x7f9b4c106f80 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:49.322 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.320+0000 7f9b53aeb640 1 -- 172.21.15.72:0/541950265 >> 172.21.15.72:0/541950265 conn(0x7f9b4c0778e0 msgr2=0x7f9b4c077cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:45:49.322 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.320+0000 7f9b53aeb640 1 -- 172.21.15.72:0/541950265 shutdown_connections 2024-03-22T01:45:49.322 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.320+0000 7f9b53aeb640 1 -- 172.21.15.72:0/541950265 wait complete. 2024-03-22T01:45:49.322 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.320+0000 7f9b53aeb640 1 Processor -- start 2024-03-22T01:45:49.322 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.320+0000 7f9b53aeb640 1 -- start start 2024-03-22T01:45:49.322 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.320+0000 7f9b53aeb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b4c100ce0 0x7f9b4c10e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:49.322 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.320+0000 7f9b53aeb640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b4c1016a0 0x7f9b4c10e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:49.322 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.321+0000 7f9b53aeb640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b4c106ba0 0x7f9b4c10ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:49.322 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.321+0000 7f9b53aeb640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f9b4c111fc0 con 0x7f9b4c1016a0 2024-03-22T01:45:49.322 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.321+0000 7f9b53aeb640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9b4c111e40 con 0x7f9b4c106ba0 2024-03-22T01:45:49.322 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.321+0000 7f9b53aeb640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9b4c112140 con 0x7f9b4c100ce0 2024-03-22T01:45:49.323 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.321+0000 7f9b51086640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b4c1016a0 0x7f9b4c10e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:49.323 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.321+0000 7f9b51086640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b4c1016a0 0x7f9b4c10e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33026/0 (socket says 172.21.15.72:33026) 2024-03-22T01:45:49.323 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.321+0000 7f9b51086640 1 -- 172.21.15.72:0/1811194832 learned_addr learned my addr 172.21.15.72:0/1811194832 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:45:49.323 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.321+0000 7f9b52088640 1 --2- 172.21.15.72:0/1811194832 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b4c106ba0 0x7f9b4c10ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:49.323 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.321+0000 7f9b51887640 1 --2- 172.21.15.72:0/1811194832 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b4c100ce0 0x7f9b4c10e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:49.323 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.321+0000 7f9b51086640 1 -- 172.21.15.72:0/1811194832 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b4c100ce0 msgr2=0x7f9b4c10e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:45:49.323 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.321+0000 7f9b51086640 1 --2- 172.21.15.72:0/1811194832 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b4c100ce0 0x7f9b4c10e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:49.323 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.321+0000 7f9b51086640 1 -- 172.21.15.72:0/1811194832 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b4c106ba0 msgr2=0x7f9b4c10ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:45:49.323 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.321+0000 7f9b51086640 1 --2- 172.21.15.72:0/1811194832 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b4c106ba0 0x7f9b4c10ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:49.323 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.321+0000 7f9b51086640 1 -- 172.21.15.72:0/1811194832 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b4c1a68a0 con 0x7f9b4c1016a0 2024-03-22T01:45:49.323 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.321+0000 7f9b52088640 1 --2- 172.21.15.72:0/1811194832 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b4c106ba0 0x7f9b4c10ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:45:49.323 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.321+0000 7f9b51086640 1 --2- 172.21.15.72:0/1811194832 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b4c1016a0 0x7f9b4c10e8f0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c00e9f0 tx=0x7f9b3c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:45:49.323 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.321+0000 7f9b42ffd640 1 -- 172.21.15.72:0/1811194832 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c00cd80 con 0x7f9b4c1016a0 2024-03-22T01:45:49.324 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.321+0000 7f9b42ffd640 1 -- 172.21.15.72:0/1811194832 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b3c010910 con 0x7f9b4c1016a0 2024-03-22T01:45:49.324 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.321+0000 7f9b53aeb640 1 -- 172.21.15.72:0/1811194832 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b4c1a6b90 con 0x7f9b4c1016a0 2024-03-22T01:45:49.324 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.321+0000 7f9b42ffd640 1 -- 172.21.15.72:0/1811194832 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c018990 con 0x7f9b4c1016a0 2024-03-22T01:45:49.324 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.321+0000 7f9b53aeb640 1 -- 172.21.15.72:0/1811194832 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b4c1a70e0 con 0x7f9b4c1016a0 2024-03-22T01:45:49.330 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.324+0000 7f9b42ffd640 1 -- 172.21.15.72:0/1811194832 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 25) ==== 137985+0+0 (secure 0 0 0) 0x7f9b3c010ac0 con 0x7f9b4c1016a0 2024-03-22T01:45:49.331 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.325+0000 7f9b567ab740 1 Processor -- start 2024-03-22T01:45:49.331 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.325+0000 7f9b567ab740 1 -- start start 2024-03-22T01:45:49.331 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.325+0000 7f9b567ab740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575e5c762d0 0x5575e5c766b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:49.331 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.325+0000 7f9b567ab740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5575e5c76bf0 0x5575e5c7f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:49.331 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.325+0000 7f9b567ab740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5575e5c7f9c0 0x5575e5c81db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:49.331 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.325+0000 7f9b567ab740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x5575e5bda040 con 0x5575e5c76bf0 2024-03-22T01:45:49.331 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.325+0000 7f9b567ab740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5575e5b829e0 con 0x5575e5c7f9c0 2024-03-22T01:45:49.331 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.325+0000 7f9b567ab740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5575e5bd5ff0 con 0x5575e5c762d0 2024-03-22T01:45:49.331 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.325+0000 7f9b42ffd640 1 --2- 172.21.15.72:0/1811194832 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f9b380a3080 0x7f9b380a5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:49.331 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.325+0000 7f9b51887640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5575e5c76bf0 0x5575e5c7f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:49.331 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.325+0000 7f9b52088640 1 --2- 172.21.15.72:0/1811194832 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f9b380a3080 0x7f9b380a5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:49.331 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.325+0000 7f9b51887640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5575e5c76bf0 0x5575e5c7f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33040/0 (socket says 172.21.15.72:33040) 2024-03-22T01:45:49.331 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.325+0000 7f9b51887640 1 -- 172.21.15.72:0/2150276233 learned_addr learned my addr 172.21.15.72:0/2150276233 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:45:49.331 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.325+0000 7f9b51086640 1 --2- 172.21.15.72:0/2150276233 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5575e5c7f9c0 0x5575e5c81db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:49.332 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.326+0000 7f9b42ffd640 1 -- 172.21.15.72:0/1811194832 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f9b3c0cfeb0 con 0x7f9b4c1016a0 2024-03-22T01:45:49.332 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.326+0000 7f9b52088640 1 --2- 172.21.15.72:0/2150276233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575e5c762d0 0x5575e5c766b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:49.332 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.326+0000 7f9b51086640 1 -- 172.21.15.72:0/2150276233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575e5c762d0 msgr2=0x5575e5c766b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:45:49.332 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.326+0000 7f9b51086640 1 --2- 172.21.15.72:0/2150276233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575e5c762d0 0x5575e5c766b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:49.332 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.326+0000 7f9b51086640 1 -- 172.21.15.72:0/2150276233 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5575e5c76bf0 msgr2=0x5575e5c7f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:45:49.332 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.326+0000 7f9b51086640 1 --2- 172.21.15.72:0/2150276233 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5575e5c76bf0 0x5575e5c7f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:49.332 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.326+0000 7f9b51086640 1 -- 172.21.15.72:0/2150276233 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575e58ee210 con 0x5575e5c7f9c0 2024-03-22T01:45:49.332 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.326+0000 7f9b51887640 1 --2- 172.21.15.72:0/2150276233 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5575e5c76bf0 0x5575e5c7f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:45:49.332 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.326+0000 7f9b51086640 1 --2- 172.21.15.72:0/2150276233 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5575e5c7f9c0 0x5575e5c81db0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c0cd250 tx=0x7f9b3c013330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e29d790d167a5f58 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:45:49.332 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.326+0000 7f9b52088640 1 --2- 172.21.15.72:0/1811194832 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f9b380a3080 0x7f9b380a5540 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9b44005fd0 tx=0x7f9b44011f10 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:45:49.332 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.326+0000 7f9b37fff640 1 -- 172.21.15.72:0/2150276233 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c026040 con 0x5575e5c7f9c0 2024-03-22T01:45:49.332 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.326+0000 7f9b37fff640 1 -- 172.21.15.72:0/2150276233 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b3c09cfe0 con 0x5575e5c7f9c0 2024-03-22T01:45:49.332 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.326+0000 7f9b37fff640 1 -- 172.21.15.72:0/2150276233 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c09cd00 con 0x5575e5c7f9c0 2024-03-22T01:45:49.332 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.327+0000 7f9b567ab740 1 -- 172.21.15.72:0/2150276233 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5575e5c7f9c0 msgr2=0x5575e5c81db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:45:49.333 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.327+0000 7f9b567ab740 1 --2- 172.21.15.72:0/2150276233 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5575e5c7f9c0 0x5575e5c81db0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c0cd250 tx=0x7f9b3c013330 comp rx=0 tx=0).stop 2024-03-22T01:45:49.333 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.327+0000 7f9b567ab740 1 -- 172.21.15.72:0/2150276233 shutdown_connections 2024-03-22T01:45:49.333 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.327+0000 7f9b567ab740 1 --2- 172.21.15.72:0/2150276233 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5575e5c7f9c0 0x5575e5c81db0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:49.333 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.327+0000 7f9b567ab740 1 --2- 172.21.15.72:0/2150276233 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5575e5c76bf0 0x5575e5c7f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:49.333 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.327+0000 7f9b567ab740 1 --2- 172.21.15.72:0/2150276233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575e5c762d0 0x5575e5c766b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:49.333 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.327+0000 7f9b567ab740 1 -- 172.21.15.72:0/2150276233 >> 172.21.15.72:0/2150276233 conn(0x5575e5c70bf0 msgr2=0x5575e5c70fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:45:49.333 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.327+0000 7f9b567ab740 1 -- 172.21.15.72:0/2150276233 shutdown_connections 2024-03-22T01:45:49.333 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.327+0000 7f9b567ab740 1 -- 172.21.15.72:0/2150276233 wait complete. 2024-03-22T01:45:49.333 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.327+0000 7f9b567ab740 1 -- start start 2024-03-22T01:45:49.333 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.327+0000 7f9b567ab740 1 Processor -- start 2024-03-22T01:45:49.333 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.328+0000 7f9b567ab740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5575e5c762d0 0x5575e5c7f390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:49.333 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.328+0000 7f9b567ab740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575e5c76bf0 0x5575e5c7c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:49.333 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.328+0000 7f9b567ab740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5575e5c7f9c0 0x5575e5c7cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:49.333 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.328+0000 7f9b567ab740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x5575e5c82590 con 0x5575e5c762d0 2024-03-22T01:45:49.333 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.328+0000 7f9b567ab740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5575e5b95db0 con 0x5575e5c7f9c0 2024-03-22T01:45:49.334 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.328+0000 7f9b567ab740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5575e5c82710 con 0x5575e5c76bf0 2024-03-22T01:45:49.334 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.328+0000 7f9b567ab740 10 client.0 _ll_register_callbacks cb 0x5575e58eda00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:45:49.334 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.328+0000 7f9b51887640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5575e5c762d0 0x5575e5c7f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:49.334 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.328+0000 7f9b52088640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5575e5c7f9c0 0x5575e5c7cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:49.334 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.328+0000 7f9b52088640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5575e5c7f9c0 0x5575e5c7cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.72:49662/0 (socket says 172.21.15.72:49662) 2024-03-22T01:45:49.334 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.328+0000 7f9b51887640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5575e5c762d0 0x5575e5c7f390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33044/0 (socket says 172.21.15.72:33044) 2024-03-22T01:45:49.334 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.328+0000 7f9b52088640 1 -- 172.21.15.72:0/570828707 learned_addr learned my addr 172.21.15.72:0/570828707 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:45:49.334 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.328+0000 7f9b51086640 1 --2- 172.21.15.72:0/570828707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575e5c76bf0 0x5575e5c7c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:49.334 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.328+0000 7f9b52088640 1 -- 172.21.15.72:0/570828707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575e5c76bf0 msgr2=0x5575e5c7c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:45:49.334 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.328+0000 7f9b52088640 1 --2- 172.21.15.72:0/570828707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575e5c76bf0 0x5575e5c7c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:49.334 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.328+0000 7f9b52088640 1 -- 172.21.15.72:0/570828707 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5575e5c762d0 msgr2=0x5575e5c7f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:45:49.334 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.328+0000 7f9b52088640 1 --2- 172.21.15.72:0/570828707 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5575e5c762d0 0x5575e5c7f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:49.335 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.328+0000 7f9b52088640 1 -- 172.21.15.72:0/570828707 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5575e59ba620 con 0x5575e5c7f9c0 2024-03-22T01:45:49.335 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.328+0000 7f9b52088640 1 --2- 172.21.15.72:0/570828707 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5575e5c7f9c0 0x5575e5c7cd30 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f9b4400ff60 tx=0x7f9b44009500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c6756e9d7599ee6 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:45:49.335 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.328+0000 7f9b357fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-03-22T01:45:49.335 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.329+0000 7f9b357fa640 1 -- 172.21.15.72:0/570828707 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b4401d600 con 0x5575e5c7f9c0 2024-03-22T01:45:49.335 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.329+0000 7f9b567ab740 1 -- 172.21.15.72:0/570828707 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b3c09cfe0 con 0x5575e5c7f9c0 2024-03-22T01:45:49.335 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.329+0000 7f9b357fa640 1 -- 172.21.15.72:0/570828707 <== mon.2 v2:172.21.15.89:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f9b4401d7a0 con 0x5575e5c7f9c0 2024-03-22T01:45:49.335 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.329+0000 7f9b357fa640 1 -- 172.21.15.72:0/570828707 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7f9b44034050 con 0x5575e5c7f9c0 2024-03-22T01:45:49.335 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.329+0000 7f9b567ab740 10 client.25967 fetch_fsmap learned FSMap version 103 2024-03-22T01:45:49.335 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.329+0000 7f9b567ab740 1 -- 172.21.15.72:0/570828707 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=4+}) -- 0x5575e5c8e570 con 0x5575e5c7f9c0 2024-03-22T01:45:49.335 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.329+0000 7f9b357fa640 1 -- 172.21.15.72:0/570828707 <== mon.2 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b3c09cfe0 con 0x5575e5c7f9c0 2024-03-22T01:45:49.335 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.329+0000 7f9b357fa640 1 -- 172.21.15.72:0/570828707 <== mon.2 v2:172.21.15.89:3300/0 5 ==== fsmap(e 103) ==== 8330+0+0 (secure 0 0 0) 0x7f9b4401e050 con 0x5575e5c7f9c0 2024-03-22T01:45:49.335 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.329+0000 7f9b567ab740 10 client.25967 fetch_fsmap finished waiting for FSMap version 103 2024-03-22T01:45:49.335 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.329+0000 7f9b567ab740 10 client.25967 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T01:45:49.335 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.329+0000 7f9b567ab740 20 client.25967 populate_metadata read hostname 'smithi072' 2024-03-22T01:45:49.336 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.329+0000 7f9b567ab740 1 --2- 172.21.15.72:0/570828707 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x5575e5c8fad0 0x5575e5c91ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:49.336 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.329+0000 7f9b567ab740 4 client.25967 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T01:45:49.336 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.329+0000 7f9b567ab740 1 -- 172.21.15.72:0/570828707 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5575e5be7580 con 0x5575e5c8fad0 2024-03-22T01:45:49.336 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.329+0000 7f9b51887640 1 --2- 172.21.15.72:0/570828707 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x5575e5c8fad0 0x5575e5c91ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:49.336 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.330+0000 7f9b357fa640 10 client.25967 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:45:49.336 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.330+0000 7f9b51887640 1 --2- 172.21.15.72:0/570828707 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x5575e5c8fad0 0x5575e5c91ec0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84b92aa89e5ce1e4 server_cookie=af71bb6d69cbf74e in_seq=0 out_seq=0 2024-03-22T01:45:49.336 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.330+0000 7f9b357fa640 1 -- 172.21.15.72:0/570828707 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5575e5be7580 con 0x5575e5c8fad0 2024-03-22T01:45:49.336 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.330+0000 7f9b357fa640 10 client.25967 handle_command_reply: tid=0 2024-03-22T01:45:49.336 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.330+0000 7f9b567ab740 1 client.25967 shutdown 2024-03-22T01:45:49.336 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.331+0000 7f9b567ab740 1 -- 172.21.15.72:0/570828707 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5575e5c7f9c0 msgr2=0x5575e5c7cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:45:49.336 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.331+0000 7f9b567ab740 1 --2- 172.21.15.72:0/570828707 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5575e5c7f9c0 0x5575e5c7cd30 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f9b4400ff60 tx=0x7f9b44009500 comp rx=0 tx=0).stop 2024-03-22T01:45:49.336 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.331+0000 7f9b567ab740 1 -- 172.21.15.72:0/570828707 shutdown_connections 2024-03-22T01:45:49.336 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.331+0000 7f9b567ab740 1 --2- 172.21.15.72:0/570828707 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x5575e5c8fad0 0x5575e5c91ec0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:49.337 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.331+0000 7f9b567ab740 1 --2- 172.21.15.72:0/570828707 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5575e5c7f9c0 0x5575e5c7cd30 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:49.337 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.331+0000 7f9b567ab740 1 --2- 172.21.15.72:0/570828707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575e5c76bf0 0x5575e5c7c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:49.337 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.331+0000 7f9b357fa640 0 client.25967 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:45:49.337 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.331+0000 7f9b567ab740 1 --2- 172.21.15.72:0/570828707 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5575e5c762d0 0x5575e5c7f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:49.337 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.331+0000 7f9b567ab740 1 -- 172.21.15.72:0/570828707 >> 172.21.15.72:0/570828707 conn(0x5575e5c70bf0 msgr2=0x5575e5c72940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:45:49.337 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.331+0000 7f9b567ab740 1 -- 172.21.15.72:0/570828707 shutdown_connections 2024-03-22T01:45:49.337 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.331+0000 7f9b567ab740 1 -- 172.21.15.72:0/570828707 wait complete. 2024-03-22T01:45:49.337 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.331+0000 7f9b567ab740 20 client.25967 trim_cache size 0 max 16384 2024-03-22T01:45:49.349 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.345+0000 7f9b567ab740 1 Processor -- start 2024-03-22T01:45:49.350 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.345+0000 7f9b567ab740 1 -- start start 2024-03-22T01:45:49.350 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.346+0000 7f9b567ab740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5575e5c762d0 0x5575e5c8b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:49.350 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.346+0000 7f9b567ab740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5575e5c76bf0 0x5575e5c97760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:49.350 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.346+0000 7f9b567ab740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575e5c7f9c0 0x5575e5c7ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:49.350 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.346+0000 7f9b567ab740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f9b4c1af2d0 con 0x5575e5c76bf0 2024-03-22T01:45:49.350 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.346+0000 7f9b567ab740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9b4c1b1b50 con 0x5575e5c762d0 2024-03-22T01:45:49.350 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.346+0000 7f9b567ab740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9b4c1af940 con 0x5575e5c7f9c0 2024-03-22T01:45:49.350 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.346+0000 7f9b51086640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5575e5c76bf0 0x5575e5c97760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:49.350 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.346+0000 7f9b51086640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5575e5c76bf0 0x5575e5c97760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33054/0 (socket says 172.21.15.72:33054) 2024-03-22T01:45:49.350 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.346+0000 7f9b51086640 1 -- 172.21.15.72:0/2776444265 learned_addr learned my addr 172.21.15.72:0/2776444265 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:45:49.350 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.346+0000 7f9b51887640 1 --2- 172.21.15.72:0/2776444265 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5575e5c762d0 0x5575e5c8b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:49.350 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.346+0000 7f9b52088640 1 --2- 172.21.15.72:0/2776444265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575e5c7f9c0 0x5575e5c7ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:49.350 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.346+0000 7f9b51086640 1 -- 172.21.15.72:0/2776444265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575e5c7f9c0 msgr2=0x5575e5c7ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:45:49.351 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.346+0000 7f9b51086640 1 --2- 172.21.15.72:0/2776444265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575e5c7f9c0 0x5575e5c7ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:49.351 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.346+0000 7f9b51086640 1 -- 172.21.15.72:0/2776444265 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5575e5c762d0 msgr2=0x5575e5c8b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:45:49.351 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.346+0000 7f9b51086640 1 --2- 172.21.15.72:0/2776444265 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5575e5c762d0 0x5575e5c8b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:49.351 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.346+0000 7f9b51086640 1 -- 172.21.15.72:0/2776444265 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575e5c7d330 con 0x5575e5c76bf0 2024-03-22T01:45:49.351 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.346+0000 7f9b51086640 1 --2- 172.21.15.72:0/2776444265 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5575e5c76bf0 0x5575e5c97760 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c098400 tx=0x7f9b3c005760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfe3939a6c538c2e server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:45:49.351 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.346+0000 7f9b377fe640 1 -- 172.21.15.72:0/2776444265 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c0b5020 con 0x5575e5c76bf0 2024-03-22T01:45:49.351 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.346+0000 7f9b377fe640 1 -- 172.21.15.72:0/2776444265 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b3c09c200 con 0x5575e5c76bf0 2024-03-22T01:45:49.351 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.346+0000 7f9b377fe640 1 -- 172.21.15.72:0/2776444265 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c09c440 con 0x5575e5c76bf0 2024-03-22T01:45:49.351 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.347+0000 7f9b567ab740 1 -- 172.21.15.72:0/2776444265 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5575e5c76bf0 msgr2=0x5575e5c97760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:45:49.351 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.347+0000 7f9b567ab740 1 --2- 172.21.15.72:0/2776444265 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5575e5c76bf0 0x5575e5c97760 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c098400 tx=0x7f9b3c005760 comp rx=0 tx=0).stop 2024-03-22T01:45:49.351 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.347+0000 7f9b567ab740 1 -- 172.21.15.72:0/2776444265 shutdown_connections 2024-03-22T01:45:49.351 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.347+0000 7f9b567ab740 1 --2- 172.21.15.72:0/2776444265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575e5c7f9c0 0x5575e5c7ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:49.351 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.347+0000 7f9b567ab740 1 --2- 172.21.15.72:0/2776444265 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5575e5c76bf0 0x5575e5c97760 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:49.352 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.347+0000 7f9b567ab740 1 --2- 172.21.15.72:0/2776444265 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5575e5c762d0 0x5575e5c8b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:49.352 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.347+0000 7f9b567ab740 1 -- 172.21.15.72:0/2776444265 >> 172.21.15.72:0/2776444265 conn(0x5575e5c70bf0 msgr2=0x5575e5c72880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:45:49.352 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.347+0000 7f9b567ab740 1 -- 172.21.15.72:0/2776444265 shutdown_connections 2024-03-22T01:45:49.352 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.347+0000 7f9b567ab740 1 -- 172.21.15.72:0/2776444265 wait complete. 2024-03-22T01:45:49.352 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.347+0000 7f9b567ab740 1 -- start start 2024-03-22T01:45:49.352 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.347+0000 7f9b567ab740 1 Processor -- start 2024-03-22T01:45:49.352 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.348+0000 7f9b567ab740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5575e5c762d0 0x5575e5c87c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:49.352 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.348+0000 7f9b567ab740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5575e5c76bf0 0x5575e5c82fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:49.352 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.348+0000 7f9b567ab740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575e5c7f9c0 0x5575e5c834e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:49.352 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.348+0000 7f9b567ab740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f9b4c1b0010 con 0x5575e5c76bf0 2024-03-22T01:45:49.352 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.348+0000 7f9b567ab740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9b4c1b1480 con 0x5575e5c762d0 2024-03-22T01:45:49.352 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.348+0000 7f9b567ab740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9b4c1b06e0 con 0x5575e5c7f9c0 2024-03-22T01:45:49.352 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.348+0000 7f9b567ab740 10 client.0 _ll_register_callbacks cb 0x7f9b44009e40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:45:49.352 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.348+0000 7f9b51086640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5575e5c76bf0 0x5575e5c82fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:49.352 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.348+0000 7f9b51086640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5575e5c76bf0 0x5575e5c82fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33068/0 (socket says 172.21.15.72:33068) 2024-03-22T01:45:49.353 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.348+0000 7f9b51887640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5575e5c762d0 0x5575e5c87c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:49.353 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.348+0000 7f9b51086640 1 -- 172.21.15.72:0/351199515 learned_addr learned my addr 172.21.15.72:0/351199515 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:45:49.353 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.348+0000 7f9b52088640 1 --2- 172.21.15.72:0/351199515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575e5c7f9c0 0x5575e5c834e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:49.353 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.348+0000 7f9b51086640 1 -- 172.21.15.72:0/351199515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575e5c7f9c0 msgr2=0x5575e5c834e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:45:49.353 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.348+0000 7f9b51086640 1 --2- 172.21.15.72:0/351199515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575e5c7f9c0 0x5575e5c834e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:49.353 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.348+0000 7f9b51086640 1 -- 172.21.15.72:0/351199515 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5575e5c762d0 msgr2=0x5575e5c87c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:45:49.353 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.348+0000 7f9b51086640 1 --2- 172.21.15.72:0/351199515 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5575e5c762d0 0x5575e5c87c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:49.353 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.348+0000 7f9b51086640 1 -- 172.21.15.72:0/351199515 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5575e5c83ce0 con 0x5575e5c76bf0 2024-03-22T01:45:49.353 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.348+0000 7f9b51887640 1 --2- 172.21.15.72:0/351199515 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5575e5c762d0 0x5575e5c87c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:45:49.353 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.348+0000 7f9b52088640 1 --2- 172.21.15.72:0/351199515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575e5c7f9c0 0x5575e5c834e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:45:49.353 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.348+0000 7f9b51086640 1 --2- 172.21.15.72:0/351199515 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5575e5c76bf0 0x5575e5c82fa0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c09c3c0 tx=0x7f9b3c09b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab3b5bbfdf1e8255 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:45:49.354 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.348+0000 7f9b357fa640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:45:49.354 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.348+0000 7f9b357fa640 1 -- 172.21.15.72:0/351199515 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c0b5020 con 0x5575e5c76bf0 2024-03-22T01:45:49.354 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.348+0000 7f9b357fa640 1 -- 172.21.15.72:0/351199515 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f9b3c0cd4d0 con 0x5575e5c76bf0 2024-03-22T01:45:49.354 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.348+0000 7f9b567ab740 1 -- 172.21.15.72:0/351199515 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5575e5be18d0 con 0x5575e5c76bf0 2024-03-22T01:45:49.354 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.349+0000 7f9b357fa640 1 -- 172.21.15.72:0/351199515 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7f9b3c0a4900 con 0x5575e5c76bf0 2024-03-22T01:45:49.354 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.349+0000 7f9b567ab740 10 client.16137 fetch_fsmap learned FSMap version 103 2024-03-22T01:45:49.354 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.349+0000 7f9b567ab740 1 -- 172.21.15.72:0/351199515 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=4+}) -- 0x5575e5cf5dd0 con 0x5575e5c76bf0 2024-03-22T01:45:49.354 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.349+0000 7f9b357fa640 1 -- 172.21.15.72:0/351199515 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5575e5be18d0 con 0x5575e5c76bf0 2024-03-22T01:45:49.354 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.349+0000 7f9b357fa640 1 -- 172.21.15.72:0/351199515 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 103) ==== 8330+0+0 (secure 0 0 0) 0x7f9b3c098610 con 0x5575e5c76bf0 2024-03-22T01:45:49.354 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.349+0000 7f9b567ab740 10 client.16137 fetch_fsmap finished waiting for FSMap version 103 2024-03-22T01:45:49.354 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.349+0000 7f9b567ab740 10 client.16137 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T01:45:49.354 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.349+0000 7f9b567ab740 20 client.16137 populate_metadata read hostname 'smithi072' 2024-03-22T01:45:49.354 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.349+0000 7f9b567ab740 1 --2- 172.21.15.72:0/351199515 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x5575e5cf5ff0 0x5575e5cf83e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:45:49.354 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.349+0000 7f9b567ab740 4 client.16137 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T01:45:49.355 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.349+0000 7f9b567ab740 1 -- 172.21.15.72:0/351199515 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5575e5b5f770 con 0x5575e5cf5ff0 2024-03-22T01:45:49.355 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.349+0000 7f9b51887640 1 --2- 172.21.15.72:0/351199515 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x5575e5cf5ff0 0x5575e5cf83e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:45:49.355 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.349+0000 7f9b51887640 1 --2- 172.21.15.72:0/351199515 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x5575e5cf5ff0 0x5575e5cf83e0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1eb85037b254395a server_cookie=740bb34c8823b8e4 in_seq=0 out_seq=0 2024-03-22T01:45:49.355 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.349+0000 7f9b357fa640 10 client.16137 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:45:49.363 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.363+0000 7f9b357fa640 1 -- 172.21.15.72:0/351199515 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5575e5b5f770 con 0x5575e5cf5ff0 2024-03-22T01:45:49.363 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.363+0000 7f9b357fa640 10 client.16137 handle_command_reply: tid=0 2024-03-22T01:45:49.363 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.363+0000 7f9b567ab740 1 client.16137 shutdown 2024-03-22T01:45:49.363 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.363+0000 7f9b567ab740 1 -- 172.21.15.72:0/351199515 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5575e5c76bf0 msgr2=0x5575e5c82fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:45:49.363 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.363+0000 7f9b567ab740 1 --2- 172.21.15.72:0/351199515 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5575e5c76bf0 0x5575e5c82fa0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c09c3c0 tx=0x7f9b3c09b7c0 comp rx=0 tx=0).stop 2024-03-22T01:45:49.363 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.363+0000 7f9b567ab740 1 -- 172.21.15.72:0/351199515 shutdown_connections 2024-03-22T01:45:49.363 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.363+0000 7f9b567ab740 1 --2- 172.21.15.72:0/351199515 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x5575e5cf5ff0 0x5575e5cf83e0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:49.363 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.363+0000 7f9b567ab740 1 --2- 172.21.15.72:0/351199515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5575e5c7f9c0 0x5575e5c834e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:49.363 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.363+0000 7f9b357fa640 0 client.16137 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:45:49.364 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.363+0000 7f9b567ab740 1 --2- 172.21.15.72:0/351199515 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5575e5c76bf0 0x5575e5c82fa0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:49.364 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.363+0000 7f9b567ab740 1 --2- 172.21.15.72:0/351199515 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5575e5c762d0 0x5575e5c87c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:49.364 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.363+0000 7f9b567ab740 1 -- 172.21.15.72:0/351199515 >> 172.21.15.72:0/351199515 conn(0x5575e5c70bf0 msgr2=0x5575e5c6f3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:45:49.364 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.363+0000 7f9b567ab740 1 -- 172.21.15.72:0/351199515 shutdown_connections 2024-03-22T01:45:49.364 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.363+0000 7f9b567ab740 1 -- 172.21.15.72:0/351199515 wait complete. 2024-03-22T01:45:49.364 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.363+0000 7f9b567ab740 20 client.16137 trim_cache size 0 max 16384 2024-03-22T01:45:49.364 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout:{ 2024-03-22T01:45:49.364 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout: "message": "", 2024-03-22T01:45:49.364 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout: "return_code": 0 2024-03-22T01:45:49.364 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout:} 2024-03-22T01:45:49.365 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.364+0000 7f9b53aeb640 1 -- 172.21.15.72:0/1811194832 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f9b380a3080 msgr2=0x7f9b380a5540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:45:49.366 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.364+0000 7f9b53aeb640 1 --2- 172.21.15.72:0/1811194832 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f9b380a3080 0x7f9b380a5540 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9b44005fd0 tx=0x7f9b44011f10 comp rx=0 tx=0).stop 2024-03-22T01:45:49.366 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.364+0000 7f9b53aeb640 1 -- 172.21.15.72:0/1811194832 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b4c1016a0 msgr2=0x7f9b4c10e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:45:49.366 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.364+0000 7f9b53aeb640 1 --2- 172.21.15.72:0/1811194832 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b4c1016a0 0x7f9b4c10e8f0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c00e9f0 tx=0x7f9b3c00eeb0 comp rx=0 tx=0).stop 2024-03-22T01:45:49.366 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.364+0000 7f9b53aeb640 1 -- 172.21.15.72:0/1811194832 shutdown_connections 2024-03-22T01:45:49.366 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.364+0000 7f9b53aeb640 1 --2- 172.21.15.72:0/1811194832 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f9b380a3080 0x7f9b380a5540 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:49.366 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.364+0000 7f9b53aeb640 1 --2- 172.21.15.72:0/1811194832 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b4c106ba0 0x7f9b4c10ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:49.366 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.364+0000 7f9b53aeb640 1 --2- 172.21.15.72:0/1811194832 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b4c1016a0 0x7f9b4c10e8f0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:49.366 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.364+0000 7f9b53aeb640 1 --2- 172.21.15.72:0/1811194832 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b4c100ce0 0x7f9b4c10e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:45:49.366 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.364+0000 7f9b53aeb640 1 -- 172.21.15.72:0/1811194832 >> 172.21.15.72:0/1811194832 conn(0x7f9b4c0778e0 msgr2=0x7f9b4c0ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:45:49.366 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.364+0000 7f9b53aeb640 1 -- 172.21.15.72:0/1811194832 shutdown_connections 2024-03-22T01:45:49.366 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:45:49.364+0000 7f9b53aeb640 1 -- 172.21.15.72:0/1811194832 wait complete. 2024-03-22T01:45:50.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:50 smithi089 ceph-mon[35770]: pgmap v278: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 14 KiB/s wr, 8 op/s 2024-03-22T01:45:50.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:50 smithi072 ceph-mon[32414]: pgmap v278: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 14 KiB/s wr, 8 op/s 2024-03-22T01:45:50.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:50 smithi098 ceph-mon[36468]: pgmap v278: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 14 KiB/s wr, 8 op/s 2024-03-22T01:45:52.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:52 smithi089 ceph-mon[35770]: pgmap v279: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-03-22T01:45:52.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:52 smithi072 ceph-mon[32414]: pgmap v279: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-03-22T01:45:52.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:52 smithi098 ceph-mon[36468]: pgmap v279: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-03-22T01:45:54.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:54 smithi089 ceph-mon[35770]: pgmap v280: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 7 op/s 2024-03-22T01:45:54.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:54 smithi072 ceph-mon[32414]: pgmap v280: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 7 op/s 2024-03-22T01:45:54.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:54 smithi098 ceph-mon[36468]: pgmap v280: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 7 op/s 2024-03-22T01:45:56.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:56 smithi089 ceph-mon[35770]: pgmap v281: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 8.2 KiB/s wr, 5 op/s 2024-03-22T01:45:56.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:56 smithi072 ceph-mon[32414]: pgmap v281: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 8.2 KiB/s wr, 5 op/s 2024-03-22T01:45:56.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:56 smithi098 ceph-mon[36468]: pgmap v281: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 8.2 KiB/s wr, 5 op/s 2024-03-22T01:45:58.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:45:58 smithi089 ceph-mon[35770]: pgmap v282: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 8.2 KiB/s wr, 6 op/s 2024-03-22T01:45:58.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:45:58 smithi072 ceph-mon[32414]: pgmap v282: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 8.2 KiB/s wr, 6 op/s 2024-03-22T01:45:58.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:45:58 smithi098 ceph-mon[36468]: pgmap v282: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 8.2 KiB/s wr, 6 op/s 2024-03-22T01:46:00.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:46:00 smithi089 ceph-mon[35770]: pgmap v283: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 5.6 KiB/s wr, 4 op/s 2024-03-22T01:46:00.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:46:00 smithi072 ceph-mon[32414]: pgmap v283: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 5.6 KiB/s wr, 4 op/s 2024-03-22T01:46:00.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:46:00 smithi098 ceph-mon[36468]: pgmap v283: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 5.6 KiB/s wr, 4 op/s 2024-03-22T01:46:01.334 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T01:46:01.335 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T01:46:01.451 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.451+0000 7f3f41247640 1 -- 172.21.15.72:0/3433623073 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3f3c106a30 msgr2=0x7f3f3c106e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:01.451 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.451+0000 7f3f41247640 1 --2- 172.21.15.72:0/3433623073 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3f3c106a30 0x7f3f3c106e10 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c009930 tx=0x7f3f2c02bc30 comp rx=0 tx=0).stop 2024-03-22T01:46:01.451 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.451+0000 7f3f41247640 1 -- 172.21.15.72:0/3433623073 shutdown_connections 2024-03-22T01:46:01.451 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.451+0000 7f3f41247640 1 --2- 172.21.15.72:0/3433623073 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f3c100a20 0x7f3f3c100ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.451 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.451+0000 7f3f41247640 1 --2- 172.21.15.72:0/3433623073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f3c1013e0 0x7f3f3c10d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.451+0000 7f3f41247640 1 --2- 172.21.15.72:0/3433623073 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3f3c106a30 0x7f3f3c106e10 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.451+0000 7f3f41247640 1 -- 172.21.15.72:0/3433623073 >> 172.21.15.72:0/3433623073 conn(0x7f3f3c077bb0 msgr2=0x7f3f3c077fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:01.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.452+0000 7f3f41247640 1 -- 172.21.15.72:0/3433623073 shutdown_connections 2024-03-22T01:46:01.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.452+0000 7f3f41247640 1 -- 172.21.15.72:0/3433623073 wait complete. 2024-03-22T01:46:01.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.452+0000 7f3f41247640 1 Processor -- start 2024-03-22T01:46:01.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.452+0000 7f3f41247640 1 -- start start 2024-03-22T01:46:01.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.453+0000 7f3f41247640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f3c100a20 0x7f3f3c1973a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:01.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.453+0000 7f3f41247640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f3c1013e0 0x7f3f3c1978e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:01.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.453+0000 7f3f41247640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3f3c106a30 0x7f3f3c19bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:01.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.453+0000 7f3f41247640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f3f3c110e40 con 0x7f3f3c106a30 2024-03-22T01:46:01.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.453+0000 7f3f41247640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3f3c110cc0 con 0x7f3f3c1013e0 2024-03-22T01:46:01.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.453+0000 7f3f41247640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3f3c110fc0 con 0x7f3f3c100a20 2024-03-22T01:46:01.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.453+0000 7f3f3b59e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3f3c106a30 0x7f3f3c19bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:01.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.453+0000 7f3f3b59e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3f3c106a30 0x7f3f3c19bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38396/0 (socket says 172.21.15.72:38396) 2024-03-22T01:46:01.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.453+0000 7f3f3a59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f3c1013e0 0x7f3f3c1978e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:01.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.453+0000 7f3f3b59e640 1 -- 172.21.15.72:0/699269046 learned_addr learned my addr 172.21.15.72:0/699269046 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:01.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.453+0000 7f3f3ad9d640 1 --2- 172.21.15.72:0/699269046 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f3c100a20 0x7f3f3c1973a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:01.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.453+0000 7f3f3b59e640 1 -- 172.21.15.72:0/699269046 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f3c100a20 msgr2=0x7f3f3c1973a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:01.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.453+0000 7f3f3b59e640 1 --2- 172.21.15.72:0/699269046 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f3c100a20 0x7f3f3c1973a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.453+0000 7f3f3b59e640 1 -- 172.21.15.72:0/699269046 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f3c1013e0 msgr2=0x7f3f3c1978e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:01.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.453+0000 7f3f3b59e640 1 --2- 172.21.15.72:0/699269046 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f3c1013e0 0x7f3f3c1978e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.453+0000 7f3f3b59e640 1 -- 172.21.15.72:0/699269046 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f3c19c390 con 0x7f3f3c106a30 2024-03-22T01:46:01.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.453+0000 7f3f3a59c640 1 --2- 172.21.15.72:0/699269046 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f3c1013e0 0x7f3f3c1978e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:46:01.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.453+0000 7f3f3b59e640 1 --2- 172.21.15.72:0/699269046 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3f3c106a30 0x7f3f3c19bc60 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f3f3400eb10 tx=0x7f3f3400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:01.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.453+0000 7f3f23fff640 1 -- 172.21.15.72:0/699269046 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f34004b20 con 0x7f3f3c106a30 2024-03-22T01:46:01.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.453+0000 7f3f23fff640 1 -- 172.21.15.72:0/699269046 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f34004cc0 con 0x7f3f3c106a30 2024-03-22T01:46:01.456 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.453+0000 7f3f41247640 1 -- 172.21.15.72:0/699269046 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f3c19c680 con 0x7f3f3c106a30 2024-03-22T01:46:01.456 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.453+0000 7f3f23fff640 1 -- 172.21.15.72:0/699269046 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f34010760 con 0x7f3f3c106a30 2024-03-22T01:46:01.456 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.454+0000 7f3f41247640 1 -- 172.21.15.72:0/699269046 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f3c1a3f10 con 0x7f3f3c106a30 2024-03-22T01:46:01.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.456+0000 7f3f23fff640 1 -- 172.21.15.72:0/699269046 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 25) ==== 137985+0+0 (secure 0 0 0) 0x7f3f34005020 con 0x7f3f3c106a30 2024-03-22T01:46:01.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.456+0000 7f3f43e2b740 1 Processor -- start 2024-03-22T01:46:01.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.456+0000 7f3f43e2b740 1 -- start start 2024-03-22T01:46:01.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.456+0000 7f3f43e2b740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563b03987410 0x563b039877f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:01.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.456+0000 7f3f43e2b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b03987d30 0x563b039903d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:01.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.456+0000 7f3f23fff640 1 --2- 172.21.15.72:0/699269046 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f3f240a3090 0x7f3f240a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:01.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.456+0000 7f3f43e2b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b03990910 0x563b03992d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:01.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.456+0000 7f3f43e2b740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x563b038eb8c0 con 0x563b03987410 2024-03-22T01:46:01.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.456+0000 7f3f43e2b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x563b03893c20 con 0x563b03990910 2024-03-22T01:46:01.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.456+0000 7f3f43e2b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563b038e7870 con 0x563b03987d30 2024-03-22T01:46:01.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.456+0000 7f3f3a59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563b03987410 0x563b039877f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:01.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.457+0000 7f3f3b59e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b03990910 0x563b03992d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:01.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.457+0000 7f3f3a59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563b03987410 0x563b039877f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38400/0 (socket says 172.21.15.72:38400) 2024-03-22T01:46:01.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.457+0000 7f3f3a59c640 1 -- 172.21.15.72:0/2969522050 learned_addr learned my addr 172.21.15.72:0/2969522050 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:01.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.457+0000 7f3f23fff640 1 -- 172.21.15.72:0/699269046 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f3f340d01f0 con 0x7f3f3c106a30 2024-03-22T01:46:01.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.457+0000 7f3f3a59c640 1 --2- 172.21.15.72:0/699269046 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f3f240a3090 0x7f3f240a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:01.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.457+0000 7f3f3ad9d640 1 --2- 172.21.15.72:0/2969522050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b03987d30 0x563b039903d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:01.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.457+0000 7f3f3b59e640 1 -- 172.21.15.72:0/2969522050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b03987d30 msgr2=0x563b039903d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:01.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.457+0000 7f3f3b59e640 1 --2- 172.21.15.72:0/2969522050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b03987d30 0x563b039903d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.457+0000 7f3f3b59e640 1 -- 172.21.15.72:0/2969522050 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563b03987410 msgr2=0x563b039877f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:01.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.457+0000 7f3f3b59e640 1 --2- 172.21.15.72:0/2969522050 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563b03987410 0x563b039877f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.457+0000 7f3f3b59e640 1 -- 172.21.15.72:0/2969522050 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b038b8150 con 0x563b03990910 2024-03-22T01:46:01.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.457+0000 7f3f3b59e640 1 --2- 172.21.15.72:0/2969522050 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b03990910 0x563b03992d00 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3f340cd590 tx=0x7f3f3401bb30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5057282752bf2aed server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:01.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.457+0000 7f3f3a59c640 1 --2- 172.21.15.72:0/699269046 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f3f240a3090 0x7f3f240a5550 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3f30011e50 tx=0x7f3f30005f90 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:01.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.457+0000 7f3f217fa640 1 -- 172.21.15.72:0/2969522050 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f3409c000 con 0x563b03990910 2024-03-22T01:46:01.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.457+0000 7f3f217fa640 1 -- 172.21.15.72:0/2969522050 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f3409c930 con 0x563b03990910 2024-03-22T01:46:01.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.457+0000 7f3f217fa640 1 -- 172.21.15.72:0/2969522050 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f340bc720 con 0x563b03990910 2024-03-22T01:46:01.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.458+0000 7f3f43e2b740 1 -- 172.21.15.72:0/2969522050 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b03990910 msgr2=0x563b03992d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:01.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.458+0000 7f3f43e2b740 1 --2- 172.21.15.72:0/2969522050 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b03990910 0x563b03992d00 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3f340cd590 tx=0x7f3f3401bb30 comp rx=0 tx=0).stop 2024-03-22T01:46:01.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.458+0000 7f3f43e2b740 1 -- 172.21.15.72:0/2969522050 shutdown_connections 2024-03-22T01:46:01.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.458+0000 7f3f43e2b740 1 --2- 172.21.15.72:0/2969522050 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b03990910 0x563b03992d00 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.458+0000 7f3f43e2b740 1 --2- 172.21.15.72:0/2969522050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b03987d30 0x563b039903d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.458+0000 7f3f43e2b740 1 --2- 172.21.15.72:0/2969522050 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563b03987410 0x563b039877f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.458+0000 7f3f43e2b740 1 -- 172.21.15.72:0/2969522050 >> 172.21.15.72:0/2969522050 conn(0x563b03981b40 msgr2=0x563b03981f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:01.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.458+0000 7f3f43e2b740 1 -- 172.21.15.72:0/2969522050 shutdown_connections 2024-03-22T01:46:01.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.458+0000 7f3f43e2b740 1 -- 172.21.15.72:0/2969522050 wait complete. 2024-03-22T01:46:01.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.458+0000 7f3f43e2b740 1 -- start start 2024-03-22T01:46:01.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.458+0000 7f3f43e2b740 1 Processor -- start 2024-03-22T01:46:01.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.459+0000 7f3f43e2b740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563b03987410 0x563b03993b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:01.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.459+0000 7f3f43e2b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b03987d30 0x563b03994050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:01.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.459+0000 7f3f43e2b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b03990910 0x563b03994590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:01.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.459+0000 7f3f43e2b740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x563b03993700 con 0x563b03987410 2024-03-22T01:46:01.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.459+0000 7f3f43e2b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x563b038a7130 con 0x563b03987d30 2024-03-22T01:46:01.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.459+0000 7f3f43e2b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563b03993880 con 0x563b03990910 2024-03-22T01:46:01.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.459+0000 7f3f43e2b740 10 client.0 _ll_register_callbacks cb 0x563b038a6b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:46:01.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.459+0000 7f3f3ad9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563b03987410 0x563b03993b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:01.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.459+0000 7f3f3a59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b03987d30 0x563b03994050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:01.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.459+0000 7f3f3ad9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563b03987410 0x563b03993b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38408/0 (socket says 172.21.15.72:38408) 2024-03-22T01:46:01.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.459+0000 7f3f3ad9d640 1 -- 172.21.15.72:0/1716810855 learned_addr learned my addr 172.21.15.72:0/1716810855 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:01.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.459+0000 7f3f3b59e640 1 --2- 172.21.15.72:0/1716810855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b03990910 0x563b03994590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:01.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.459+0000 7f3f3ad9d640 1 -- 172.21.15.72:0/1716810855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b03990910 msgr2=0x563b03994590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:01.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.459+0000 7f3f3ad9d640 1 --2- 172.21.15.72:0/1716810855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b03990910 0x563b03994590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.459+0000 7f3f3ad9d640 1 -- 172.21.15.72:0/1716810855 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b03987d30 msgr2=0x563b03994050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:01.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.459+0000 7f3f3ad9d640 1 --2- 172.21.15.72:0/1716810855 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b03987d30 0x563b03994050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.459+0000 7f3f3ad9d640 1 -- 172.21.15.72:0/1716810855 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b036c97d0 con 0x563b03987410 2024-03-22T01:46:01.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.459+0000 7f3f3b59e640 1 --2- 172.21.15.72:0/1716810855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b03990910 0x563b03994590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:46:01.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.459+0000 7f3f3a59c640 1 --2- 172.21.15.72:0/1716810855 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b03987d30 0x563b03994050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:46:01.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.459+0000 7f3f3ad9d640 1 --2- 172.21.15.72:0/1716810855 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563b03987410 0x563b03993b10 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c009900 tx=0x7f3f2c02ce10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f09620a25e3d327 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:01.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.459+0000 7f3f1eff5640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:46:01.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.459+0000 7f3f1eff5640 1 -- 172.21.15.72:0/1716810855 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f2c004240 con 0x563b03987410 2024-03-22T01:46:01.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.459+0000 7f3f1eff5640 1 -- 172.21.15.72:0/1716810855 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f3f2c0043e0 con 0x563b03987410 2024-03-22T01:46:01.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.459+0000 7f3f43e2b740 1 -- 172.21.15.72:0/1716810855 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f3409c930 con 0x563b03987410 2024-03-22T01:46:01.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.460+0000 7f3f1eff5640 1 -- 172.21.15.72:0/1716810855 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7f3f2c0046c0 con 0x563b03987410 2024-03-22T01:46:01.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.460+0000 7f3f43e2b740 10 client.16149 fetch_fsmap learned FSMap version 103 2024-03-22T01:46:01.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.460+0000 7f3f43e2b740 1 -- 172.21.15.72:0/1716810855 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=4+}) -- 0x563b035ff3f0 con 0x563b03987410 2024-03-22T01:46:01.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.460+0000 7f3f1eff5640 1 -- 172.21.15.72:0/1716810855 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f3409c930 con 0x563b03987410 2024-03-22T01:46:01.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.460+0000 7f3f1eff5640 1 -- 172.21.15.72:0/1716810855 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 103) ==== 8330+0+0 (secure 0 0 0) 0x7f3f2c0035f0 con 0x563b03987410 2024-03-22T01:46:01.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.460+0000 7f3f43e2b740 10 client.16149 fetch_fsmap finished waiting for FSMap version 103 2024-03-22T01:46:01.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.460+0000 7f3f43e2b740 10 client.16149 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T01:46:01.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.460+0000 7f3f43e2b740 20 client.16149 populate_metadata read hostname 'smithi072' 2024-03-22T01:46:01.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.460+0000 7f3f43e2b740 1 --2- 172.21.15.72:0/1716810855 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x563b0398ea90 0x563b0398eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:01.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.460+0000 7f3f43e2b740 4 client.16149 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T01:46:01.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.460+0000 7f3f43e2b740 1 -- 172.21.15.72:0/1716810855 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563b038f8e00 con 0x563b0398ea90 2024-03-22T01:46:01.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.460+0000 7f3f3a59c640 1 --2- 172.21.15.72:0/1716810855 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x563b0398ea90 0x563b0398eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:01.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.460+0000 7f3f3a59c640 1 --2- 172.21.15.72:0/1716810855 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x563b0398ea90 0x563b0398eef0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b06774b3eea29a2b server_cookie=1f8c5de0f29da80d in_seq=0 out_seq=0 2024-03-22T01:46:01.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.460+0000 7f3f1eff5640 10 client.16149 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:46:01.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.461+0000 7f3f1eff5640 1 -- 172.21.15.72:0/1716810855 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563b038f8e00 con 0x563b0398ea90 2024-03-22T01:46:01.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.461+0000 7f3f1eff5640 10 client.16149 handle_command_reply: tid=0 2024-03-22T01:46:01.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.461+0000 7f3f43e2b740 1 client.16149 shutdown 2024-03-22T01:46:01.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.461+0000 7f3f43e2b740 1 -- 172.21.15.72:0/1716810855 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563b03987410 msgr2=0x563b03993b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:01.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.461+0000 7f3f43e2b740 1 --2- 172.21.15.72:0/1716810855 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563b03987410 0x563b03993b10 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c009900 tx=0x7f3f2c02ce10 comp rx=0 tx=0).stop 2024-03-22T01:46:01.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.461+0000 7f3f43e2b740 1 -- 172.21.15.72:0/1716810855 shutdown_connections 2024-03-22T01:46:01.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.461+0000 7f3f43e2b740 1 --2- 172.21.15.72:0/1716810855 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x563b0398ea90 0x563b0398eef0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.461+0000 7f3f43e2b740 1 --2- 172.21.15.72:0/1716810855 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b03987d30 0x563b03994050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.461+0000 7f3f43e2b740 1 --2- 172.21.15.72:0/1716810855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b03990910 0x563b03994590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.461+0000 7f3f43e2b740 1 --2- 172.21.15.72:0/1716810855 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563b03987410 0x563b03993b10 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.461+0000 7f3f1eff5640 0 client.16149 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:46:01.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.461+0000 7f3f43e2b740 1 -- 172.21.15.72:0/1716810855 >> 172.21.15.72:0/1716810855 conn(0x563b03981b40 msgr2=0x563b03983890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:01.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.461+0000 7f3f43e2b740 1 -- 172.21.15.72:0/1716810855 shutdown_connections 2024-03-22T01:46:01.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.461+0000 7f3f43e2b740 1 -- 172.21.15.72:0/1716810855 wait complete. 2024-03-22T01:46:01.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.461+0000 7f3f43e2b740 20 client.16149 trim_cache size 0 max 16384 2024-03-22T01:46:01.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.478+0000 7f3f43e2b740 1 Processor -- start 2024-03-22T01:46:01.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.478+0000 7f3f43e2b740 1 -- start start 2024-03-22T01:46:01.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.479+0000 7f3f43e2b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b03987410 0x563b039836a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:01.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.479+0000 7f3f43e2b740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563b03987d30 0x563b039a6b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:01.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.479+0000 7f3f43e2b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b03990910 0x563b0398ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:01.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.479+0000 7f3f43e2b740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f3f3c1aaf20 con 0x563b03987d30 2024-03-22T01:46:01.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.479+0000 7f3f43e2b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3f3c1ad7a0 con 0x563b03990910 2024-03-22T01:46:01.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.479+0000 7f3f43e2b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3f3c1ab590 con 0x563b03987410 2024-03-22T01:46:01.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.479+0000 7f3f3a59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563b03987d30 0x563b039a6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:01.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.479+0000 7f3f3a59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563b03987d30 0x563b039a6b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38422/0 (socket says 172.21.15.72:38422) 2024-03-22T01:46:01.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.479+0000 7f3f3a59c640 1 -- 172.21.15.72:0/1421056423 learned_addr learned my addr 172.21.15.72:0/1421056423 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:01.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.479+0000 7f3f3b59e640 1 --2- 172.21.15.72:0/1421056423 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b03990910 0x563b0398ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:01.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.479+0000 7f3f3ad9d640 1 --2- 172.21.15.72:0/1421056423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b03987410 0x563b039836a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:01.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.479+0000 7f3f3a59c640 1 -- 172.21.15.72:0/1421056423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b03987410 msgr2=0x563b039836a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:01.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.479+0000 7f3f3a59c640 1 --2- 172.21.15.72:0/1421056423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b03987410 0x563b039836a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.479+0000 7f3f3a59c640 1 -- 172.21.15.72:0/1421056423 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b03990910 msgr2=0x563b0398ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:01.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.479+0000 7f3f3a59c640 1 --2- 172.21.15.72:0/1421056423 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b03990910 0x563b0398ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.479+0000 7f3f3a59c640 1 -- 172.21.15.72:0/1421056423 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b035bf0f0 con 0x563b03987d30 2024-03-22T01:46:01.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.479+0000 7f3f3a59c640 1 --2- 172.21.15.72:0/1421056423 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563b03987d30 0x563b039a6b00 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x563b0398d130 tx=0x7f3f30009b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad27f3a07f3a4cb3 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:01.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.479+0000 7f3f20ff9640 1 -- 172.21.15.72:0/1421056423 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f30005020 con 0x563b03987d30 2024-03-22T01:46:01.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.479+0000 7f3f20ff9640 1 -- 172.21.15.72:0/1421056423 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f300047a0 con 0x563b03987d30 2024-03-22T01:46:01.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.479+0000 7f3f20ff9640 1 -- 172.21.15.72:0/1421056423 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f300312b0 con 0x563b03987d30 2024-03-22T01:46:01.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.480+0000 7f3f43e2b740 1 -- 172.21.15.72:0/1421056423 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563b03987d30 msgr2=0x563b039a6b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:01.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.480+0000 7f3f43e2b740 1 --2- 172.21.15.72:0/1421056423 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563b03987d30 0x563b039a6b00 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x563b0398d130 tx=0x7f3f30009b70 comp rx=0 tx=0).stop 2024-03-22T01:46:01.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.480+0000 7f3f43e2b740 1 -- 172.21.15.72:0/1421056423 shutdown_connections 2024-03-22T01:46:01.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.480+0000 7f3f43e2b740 1 --2- 172.21.15.72:0/1421056423 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b03990910 0x563b0398ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.480+0000 7f3f43e2b740 1 --2- 172.21.15.72:0/1421056423 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563b03987d30 0x563b039a6b00 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.480+0000 7f3f43e2b740 1 --2- 172.21.15.72:0/1421056423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b03987410 0x563b039836a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.480+0000 7f3f43e2b740 1 -- 172.21.15.72:0/1421056423 >> 172.21.15.72:0/1421056423 conn(0x563b03981b40 msgr2=0x563b03982f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:01.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.480+0000 7f3f43e2b740 1 -- 172.21.15.72:0/1421056423 shutdown_connections 2024-03-22T01:46:01.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.480+0000 7f3f43e2b740 1 -- 172.21.15.72:0/1421056423 wait complete. 2024-03-22T01:46:01.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.480+0000 7f3f43e2b740 1 -- start start 2024-03-22T01:46:01.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.480+0000 7f3f43e2b740 1 Processor -- start 2024-03-22T01:46:01.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.480+0000 7f3f43e2b740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563b03987410 0x563b039fe750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:01.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.480+0000 7f3f43e2b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b03987d30 0x563b039fec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:01.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.480+0000 7f3f43e2b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b03990910 0x563b039a5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:01.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.481+0000 7f3f43e2b740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f3f3c1abc60 con 0x563b03987410 2024-03-22T01:46:01.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.481+0000 7f3f43e2b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3f3c1ad0d0 con 0x563b03990910 2024-03-22T01:46:01.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.481+0000 7f3f43e2b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3f3c1ac330 con 0x563b03987d30 2024-03-22T01:46:01.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.481+0000 7f3f43e2b740 10 client.0 _ll_register_callbacks cb 0x7f3f2c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:46:01.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.481+0000 7f3f3ad9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563b03987410 0x563b039fe750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:01.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.481+0000 7f3f3ad9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563b03987410 0x563b039fe750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38432/0 (socket says 172.21.15.72:38432) 2024-03-22T01:46:01.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.481+0000 7f3f3ad9d640 1 -- 172.21.15.72:0/3203885717 learned_addr learned my addr 172.21.15.72:0/3203885717 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:01.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.481+0000 7f3f3b59e640 1 --2- 172.21.15.72:0/3203885717 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b03990910 0x563b039a5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:01.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.481+0000 7f3f3a59c640 1 --2- 172.21.15.72:0/3203885717 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b03987d30 0x563b039fec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:01.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.481+0000 7f3f3ad9d640 1 -- 172.21.15.72:0/3203885717 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b03987d30 msgr2=0x563b039fec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:01.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.481+0000 7f3f3ad9d640 1 --2- 172.21.15.72:0/3203885717 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b03987d30 0x563b039fec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.481+0000 7f3f3ad9d640 1 -- 172.21.15.72:0/3203885717 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b03990910 msgr2=0x563b039a5fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:01.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.481+0000 7f3f3ad9d640 1 --2- 172.21.15.72:0/3203885717 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b03990910 0x563b039a5fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.481+0000 7f3f3ad9d640 1 -- 172.21.15.72:0/3203885717 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b039a6810 con 0x563b03987410 2024-03-22T01:46:01.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.481+0000 7f3f3b59e640 1 --2- 172.21.15.72:0/3203885717 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b03990910 0x563b039a5fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:46:01.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.481+0000 7f3f3a59c640 1 --2- 172.21.15.72:0/3203885717 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b03987d30 0x563b039fec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T01:46:01.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.481+0000 7f3f3ad9d640 1 --2- 172.21.15.72:0/3203885717 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563b03987410 0x563b039fe750 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c02bc70 tx=0x7f3f2c003920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fef82b5cb215a35 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:01.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.481+0000 7f3f1eff5640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:46:01.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.481+0000 7f3f1eff5640 1 -- 172.21.15.72:0/3203885717 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f2c005630 con 0x563b03987410 2024-03-22T01:46:01.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.481+0000 7f3f1eff5640 1 -- 172.21.15.72:0/3203885717 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f3f2c005160 con 0x563b03987410 2024-03-22T01:46:01.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.481+0000 7f3f43e2b740 1 -- 172.21.15.72:0/3203885717 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f300047a0 con 0x563b03987410 2024-03-22T01:46:01.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.481+0000 7f3f1eff5640 1 -- 172.21.15.72:0/3203885717 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7f3f2c062050 con 0x563b03987410 2024-03-22T01:46:01.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.481+0000 7f3f43e2b740 10 client.16155 fetch_fsmap learned FSMap version 103 2024-03-22T01:46:01.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.481+0000 7f3f43e2b740 1 -- 172.21.15.72:0/3203885717 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=4+}) -- 0x563b03a07060 con 0x563b03987410 2024-03-22T01:46:01.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.482+0000 7f3f1eff5640 1 -- 172.21.15.72:0/3203885717 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f300047a0 con 0x563b03987410 2024-03-22T01:46:01.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.482+0000 7f3f1eff5640 1 -- 172.21.15.72:0/3203885717 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 103) ==== 8330+0+0 (secure 0 0 0) 0x7f3f2c003c80 con 0x563b03987410 2024-03-22T01:46:01.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.482+0000 7f3f43e2b740 10 client.16155 fetch_fsmap finished waiting for FSMap version 103 2024-03-22T01:46:01.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.482+0000 7f3f43e2b740 10 client.16155 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T01:46:01.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.482+0000 7f3f43e2b740 20 client.16155 populate_metadata read hostname 'smithi072' 2024-03-22T01:46:01.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.482+0000 7f3f43e2b740 1 --2- 172.21.15.72:0/3203885717 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x563b03a07830 0x563b03a09c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:01.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.482+0000 7f3f43e2b740 4 client.16155 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T01:46:01.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.482+0000 7f3f43e2b740 1 -- 172.21.15.72:0/3203885717 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563b038f3150 con 0x563b03a07830 2024-03-22T01:46:01.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.482+0000 7f3f3a59c640 1 --2- 172.21.15.72:0/3203885717 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x563b03a07830 0x563b03a09c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:01.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.482+0000 7f3f1eff5640 10 client.16155 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:46:01.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.482+0000 7f3f3a59c640 1 --2- 172.21.15.72:0/3203885717 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x563b03a07830 0x563b03a09c50 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bbfdfc18cf6e7840 server_cookie=511821d44a4adeb5 in_seq=0 out_seq=0 2024-03-22T01:46:01.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.482+0000 7f3f1eff5640 1 -- 172.21.15.72:0/3203885717 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x563b038f3150 con 0x563b03a07830 2024-03-22T01:46:01.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.482+0000 7f3f1eff5640 10 client.16155 handle_command_reply: tid=0 2024-03-22T01:46:01.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.482+0000 7f3f43e2b740 1 client.16155 shutdown 2024-03-22T01:46:01.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.482+0000 7f3f43e2b740 1 -- 172.21.15.72:0/3203885717 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563b03987410 msgr2=0x563b039fe750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:01.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.482+0000 7f3f43e2b740 1 --2- 172.21.15.72:0/3203885717 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563b03987410 0x563b039fe750 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c02bc70 tx=0x7f3f2c003920 comp rx=0 tx=0).stop 2024-03-22T01:46:01.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.483+0000 7f3f43e2b740 1 -- 172.21.15.72:0/3203885717 shutdown_connections 2024-03-22T01:46:01.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.483+0000 7f3f43e2b740 1 --2- 172.21.15.72:0/3203885717 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x563b03a07830 0x563b03a09c50 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.483+0000 7f3f43e2b740 1 --2- 172.21.15.72:0/3203885717 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b03990910 0x563b039a5fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.483+0000 7f3f43e2b740 1 --2- 172.21.15.72:0/3203885717 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563b03987d30 0x563b039fec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.483+0000 7f3f43e2b740 1 --2- 172.21.15.72:0/3203885717 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x563b03987410 0x563b039fe750 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.483+0000 7f3f1eff5640 0 client.16155 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:46:01.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.483+0000 7f3f43e2b740 1 -- 172.21.15.72:0/3203885717 >> 172.21.15.72:0/3203885717 conn(0x563b03981b40 msgr2=0x563b03980c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:01.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.483+0000 7f3f43e2b740 1 -- 172.21.15.72:0/3203885717 shutdown_connections 2024-03-22T01:46:01.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.483+0000 7f3f43e2b740 1 -- 172.21.15.72:0/3203885717 wait complete. 2024-03-22T01:46:01.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.483+0000 7f3f43e2b740 20 client.16155 trim_cache size 0 max 16384 2024-03-22T01:46:01.491 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-22T01:46:01.491 INFO:teuthology.orchestra.run.smithi072.stdout: "status": "no active scrubs running", 2024-03-22T01:46:01.491 INFO:teuthology.orchestra.run.smithi072.stdout: "scrubs": {} 2024-03-22T01:46:01.491 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-22T01:46:01.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.483+0000 7f3f41247640 1 -- 172.21.15.72:0/699269046 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f3f240a3090 msgr2=0x7f3f240a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:01.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.483+0000 7f3f41247640 1 --2- 172.21.15.72:0/699269046 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f3f240a3090 0x7f3f240a5550 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3f30011e50 tx=0x7f3f30005f90 comp rx=0 tx=0).stop 2024-03-22T01:46:01.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.483+0000 7f3f41247640 1 -- 172.21.15.72:0/699269046 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3f3c106a30 msgr2=0x7f3f3c19bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:01.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.483+0000 7f3f41247640 1 --2- 172.21.15.72:0/699269046 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3f3c106a30 0x7f3f3c19bc60 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f3f3400eb10 tx=0x7f3f3400c2d0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.483+0000 7f3f41247640 1 -- 172.21.15.72:0/699269046 shutdown_connections 2024-03-22T01:46:01.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.483+0000 7f3f41247640 1 --2- 172.21.15.72:0/699269046 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f3f240a3090 0x7f3f240a5550 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.483+0000 7f3f41247640 1 --2- 172.21.15.72:0/699269046 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f3c1013e0 0x7f3f3c1978e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.483+0000 7f3f41247640 1 --2- 172.21.15.72:0/699269046 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3f3c106a30 0x7f3f3c19bc60 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.483+0000 7f3f41247640 1 --2- 172.21.15.72:0/699269046 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f3c100a20 0x7f3f3c1973a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.483+0000 7f3f41247640 1 -- 172.21.15.72:0/699269046 >> 172.21.15.72:0/699269046 conn(0x7f3f3c077bb0 msgr2=0x7f3f3c1032d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:01.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.483+0000 7f3f41247640 1 -- 172.21.15.72:0/699269046 shutdown_connections 2024-03-22T01:46:01.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.483+0000 7f3f41247640 1 -- 172.21.15.72:0/699269046 wait complete. 2024-03-22T01:46:01.496 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-22T01:46:01.496 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T01:46:01.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.598+0000 7f991ac95640 1 -- 172.21.15.72:0/1056241600 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9914100950 msgr2=0x7f991410e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:01.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.598+0000 7f991ac95640 1 --2- 172.21.15.72:0/1056241600 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9914100950 0x7f991410e990 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f990c009ca0 tx=0x7f990c02c140 comp rx=0 tx=0).stop 2024-03-22T01:46:01.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.598+0000 7f991ac95640 1 -- 172.21.15.72:0/1056241600 shutdown_connections 2024-03-22T01:46:01.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.599+0000 7f991ac95640 1 --2- 172.21.15.72:0/1056241600 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9914100950 0x7f991410e990 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.599+0000 7f991ac95640 1 --2- 172.21.15.72:0/1056241600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99140fffb0 0x7f9914100410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.599+0000 7f991ac95640 1 --2- 172.21.15.72:0/1056241600 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9914106b20 0x7f9914106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.599+0000 7f991ac95640 1 -- 172.21.15.72:0/1056241600 >> 172.21.15.72:0/1056241600 conn(0x7f9914078020 msgr2=0x7f99140fe2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:01.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.599+0000 7f991ac95640 1 -- 172.21.15.72:0/1056241600 shutdown_connections 2024-03-22T01:46:01.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.599+0000 7f991ac95640 1 -- 172.21.15.72:0/1056241600 wait complete. 2024-03-22T01:46:01.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.599+0000 7f991ac95640 1 Processor -- start 2024-03-22T01:46:01.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.599+0000 7f991ac95640 1 -- start start 2024-03-22T01:46:01.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.599+0000 7f991ac95640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99140fffb0 0x7f9914198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:01.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.599+0000 7f991ac95640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9914100950 0x7f9914198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:01.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.599+0000 7f991ac95640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9914106b20 0x7f991419cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:01.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.599+0000 7f991ac95640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f9914112020 con 0x7f9914106b20 2024-03-22T01:46:01.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.599+0000 7f991ac95640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9914111ea0 con 0x7f9914100950 2024-03-22T01:46:01.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.599+0000 7f991ac95640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f99141121a0 con 0x7f99140fffb0 2024-03-22T01:46:01.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.599+0000 7f9919232640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9914106b20 0x7f991419cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:01.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.599+0000 7f9919232640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9914106b20 0x7f991419cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38458/0 (socket says 172.21.15.72:38458) 2024-03-22T01:46:01.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.599+0000 7f9919232640 1 -- 172.21.15.72:0/1587081135 learned_addr learned my addr 172.21.15.72:0/1587081135 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:01.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.600+0000 7f990bfff640 1 --2- 172.21.15.72:0/1587081135 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9914100950 0x7f9914198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:01.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.600+0000 7f9918a31640 1 --2- 172.21.15.72:0/1587081135 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99140fffb0 0x7f9914198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:01.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.600+0000 7f9919232640 1 -- 172.21.15.72:0/1587081135 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99140fffb0 msgr2=0x7f9914198440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:01.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.600+0000 7f9919232640 1 --2- 172.21.15.72:0/1587081135 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99140fffb0 0x7f9914198440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.600+0000 7f9919232640 1 -- 172.21.15.72:0/1587081135 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9914100950 msgr2=0x7f9914198980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:01.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.600+0000 7f9919232640 1 --2- 172.21.15.72:0/1587081135 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9914100950 0x7f9914198980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.600+0000 7f9919232640 1 -- 172.21.15.72:0/1587081135 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f991419d430 con 0x7f9914106b20 2024-03-22T01:46:01.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.600+0000 7f9919232640 1 --2- 172.21.15.72:0/1587081135 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9914106b20 0x7f991419cd00 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f990c02c750 tx=0x7f990c0346f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:01.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.600+0000 7f9909ffb640 1 -- 172.21.15.72:0/1587081135 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f990c039050 con 0x7f9914106b20 2024-03-22T01:46:01.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.600+0000 7f991ac95640 1 -- 172.21.15.72:0/1587081135 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f991419d6c0 con 0x7f9914106b20 2024-03-22T01:46:01.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.600+0000 7f9909ffb640 1 -- 172.21.15.72:0/1587081135 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f990c045070 con 0x7f9914106b20 2024-03-22T01:46:01.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.600+0000 7f9909ffb640 1 -- 172.21.15.72:0/1587081135 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f990c004680 con 0x7f9914106b20 2024-03-22T01:46:01.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.600+0000 7f991ac95640 1 -- 172.21.15.72:0/1587081135 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99141a4fb0 con 0x7f9914106b20 2024-03-22T01:46:01.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.601+0000 7f991ac95640 1 -- 172.21.15.72:0/1587081135 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9914068e60 con 0x7f9914106b20 2024-03-22T01:46:01.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.602+0000 7f9909ffb640 1 -- 172.21.15.72:0/1587081135 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 25) ==== 137985+0+0 (secure 0 0 0) 0x7f990c003970 con 0x7f9914106b20 2024-03-22T01:46:01.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.603+0000 7f9909ffb640 1 --2- 172.21.15.72:0/1587081135 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f98fc0a3090 0x7f98fc0a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:01.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.603+0000 7f9918a31640 1 --2- 172.21.15.72:0/1587081135 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f98fc0a3090 0x7f98fc0a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:01.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.603+0000 7f9909ffb640 1 -- 172.21.15.72:0/1587081135 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f990c0f23e0 con 0x7f9914106b20 2024-03-22T01:46:01.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.603+0000 7f9918a31640 1 --2- 172.21.15.72:0/1587081135 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f98fc0a3090 0x7f98fc0a5550 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f99100095f0 tx=0x7f9910005f90 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:01.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.607+0000 7f9909ffb640 1 -- 172.21.15.72:0/1587081135 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f990c004820 con 0x7f9914106b20 2024-03-22T01:46:01.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.821+0000 7f991ac95640 1 -- 172.21.15.72:0/1587081135 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f991410a030 con 0x7f9914106b20 2024-03-22T01:46:01.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.822+0000 7f9909ffb640 1 -- 172.21.15.72:0/1587081135 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) ==== 95+0+11791 (secure 0 0 0) 0x7f990c0bec30 con 0x7f9914106b20 2024-03-22T01:46:01.821 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:46:01.821 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":103,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6835/3077216308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3077216308},{"type":"v1","addr":"172.21.15.72:6835","nonce":3077216308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2460254143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":2460254143},{"type":"v1","addr":"172.21.15.72:6837","nonce":2460254143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6839/2026432819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6838","nonce":2026432819},{"type":"v1","addr":"172.21.15.72:6839","nonce":2026432819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6841/400094545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6840","nonce":400094545},{"type":"v1","addr":"172.21.15.72:6841","nonce":400094545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6833/1578701171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":1578701171},{"type":"v1","addr":"172.21.15.89:6833","nonce":1578701171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/2703047213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2703047213},{"type":"v1","addr":"172.21.15.98:6833","nonce":2703047213}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2589468787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2589468787},{"type":"v1","addr":"172.21.15.89:6835","nonce":2589468787}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/92657636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":92657636},{"type":"v1","addr":"172.21.15.98:6835","nonce":92657636}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1662941782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1662941782},{"type":"v1","addr":"172.21.15.89:6837","nonce":1662941782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24490,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/781860371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":781860371},{"type":"v1","addr":"172.21.15.98:6837","nonce":781860371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":103,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-22T01:34:14.815677+0000","modified":"2024-03-22T01:45:27.515143+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24524},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":171,"addr":"172.21.15.98:6839/3192173650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3192173650},{"type":"v1","addr":"172.21.15.98:6839","nonce":3192173650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":171,"addr":"172.21.15.89:6839/149200280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":149200280},{"type":"v1","addr":"172.21.15.89:6839","nonce":149200280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24524]},"id":1}]} 2024-03-22T01:46:01.822 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 103 2024-03-22T01:46:01.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.824+0000 7f991ac95640 1 -- 172.21.15.72:0/1587081135 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f98fc0a3090 msgr2=0x7f98fc0a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:01.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.824+0000 7f991ac95640 1 --2- 172.21.15.72:0/1587081135 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f98fc0a3090 0x7f98fc0a5550 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f99100095f0 tx=0x7f9910005f90 comp rx=0 tx=0).stop 2024-03-22T01:46:01.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.824+0000 7f991ac95640 1 -- 172.21.15.72:0/1587081135 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9914106b20 msgr2=0x7f991419cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:01.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.824+0000 7f991ac95640 1 --2- 172.21.15.72:0/1587081135 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9914106b20 0x7f991419cd00 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f990c02c750 tx=0x7f990c0346f0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.824+0000 7f991ac95640 1 -- 172.21.15.72:0/1587081135 shutdown_connections 2024-03-22T01:46:01.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.824+0000 7f991ac95640 1 --2- 172.21.15.72:0/1587081135 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f98fc0a3090 0x7f98fc0a5550 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.824+0000 7f991ac95640 1 --2- 172.21.15.72:0/1587081135 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9914100950 0x7f9914198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.824+0000 7f991ac95640 1 --2- 172.21.15.72:0/1587081135 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9914106b20 0x7f991419cd00 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.824+0000 7f991ac95640 1 --2- 172.21.15.72:0/1587081135 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99140fffb0 0x7f9914198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.824+0000 7f991ac95640 1 -- 172.21.15.72:0/1587081135 >> 172.21.15.72:0/1587081135 conn(0x7f9914078020 msgr2=0x7f9914104b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:01.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.824+0000 7f991ac95640 1 -- 172.21.15.72:0/1587081135 shutdown_connections 2024-03-22T01:46:01.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.824+0000 7f991ac95640 1 -- 172.21.15.72:0/1587081135 wait complete. 2024-03-22T01:46:01.835 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-22T01:46:01.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.940+0000 7eff04b7a640 1 -- 172.21.15.72:0/1014882166 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff00100a30 msgr2=0x7eff00100eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:01.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.940+0000 7eff04b7a640 1 --2- 172.21.15.72:0/1014882166 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff00100a30 0x7eff00100eb0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7efef0009930 tx=0x7efef002bc30 comp rx=0 tx=0).stop 2024-03-22T01:46:01.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.940+0000 7eff04b7a640 1 -- 172.21.15.72:0/1014882166 shutdown_connections 2024-03-22T01:46:01.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.940+0000 7eff04b7a640 1 --2- 172.21.15.72:0/1014882166 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff001013f0 0x7eff0010d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.940+0000 7eff04b7a640 1 --2- 172.21.15.72:0/1014882166 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff00100a30 0x7eff00100eb0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.940+0000 7eff04b7a640 1 --2- 172.21.15.72:0/1014882166 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff00106a40 0x7eff00106e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.940+0000 7eff04b7a640 1 -- 172.21.15.72:0/1014882166 >> 172.21.15.72:0/1014882166 conn(0x7eff00077c40 msgr2=0x7eff00078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:01.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.940+0000 7eff04b7a640 1 -- 172.21.15.72:0/1014882166 shutdown_connections 2024-03-22T01:46:01.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.940+0000 7eff04b7a640 1 -- 172.21.15.72:0/1014882166 wait complete. 2024-03-22T01:46:01.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.940+0000 7eff04b7a640 1 Processor -- start 2024-03-22T01:46:01.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.940+0000 7eff04b7a640 1 -- start start 2024-03-22T01:46:01.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.941+0000 7eff04b7a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff00100a30 0x7eff00197380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:01.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.941+0000 7eff04b7a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff001013f0 0x7eff001978c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:01.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.941+0000 7eff04b7a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff00106a40 0x7eff0019bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:01.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.941+0000 7eff04b7a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7eff00110e20 con 0x7eff00106a40 2024-03-22T01:46:01.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.941+0000 7eff04b7a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7eff00110ca0 con 0x7eff00100a30 2024-03-22T01:46:01.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.941+0000 7eff04b7a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7eff00110fa0 con 0x7eff001013f0 2024-03-22T01:46:01.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.941+0000 7efefed9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff00106a40 0x7eff0019bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:01.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.941+0000 7efefed9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff00106a40 0x7eff0019bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38484/0 (socket says 172.21.15.72:38484) 2024-03-22T01:46:01.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.941+0000 7efefed9d640 1 -- 172.21.15.72:0/3330803324 learned_addr learned my addr 172.21.15.72:0/3330803324 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:01.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.941+0000 7efefe59c640 1 --2- 172.21.15.72:0/3330803324 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff00100a30 0x7eff00197380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:01.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.941+0000 7efefdd9b640 1 --2- 172.21.15.72:0/3330803324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff001013f0 0x7eff001978c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:01.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.941+0000 7efefed9d640 1 -- 172.21.15.72:0/3330803324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff001013f0 msgr2=0x7eff001978c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:01.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.941+0000 7efefed9d640 1 --2- 172.21.15.72:0/3330803324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff001013f0 0x7eff001978c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.941+0000 7efefed9d640 1 -- 172.21.15.72:0/3330803324 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff00100a30 msgr2=0x7eff00197380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:01.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.941+0000 7efefed9d640 1 --2- 172.21.15.72:0/3330803324 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff00100a30 0x7eff00197380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.941+0000 7efefed9d640 1 -- 172.21.15.72:0/3330803324 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff0019c370 con 0x7eff00106a40 2024-03-22T01:46:01.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.941+0000 7efefdd9b640 1 --2- 172.21.15.72:0/3330803324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff001013f0 0x7eff001978c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:46:01.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.941+0000 7efefed9d640 1 --2- 172.21.15.72:0/3330803324 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff00106a40 0x7eff0019bc40 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7efef800ea60 tx=0x7efef800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:01.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.941+0000 7efeef7fe640 1 -- 172.21.15.72:0/3330803324 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef800ce40 con 0x7eff00106a40 2024-03-22T01:46:01.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.941+0000 7eff04b7a640 1 -- 172.21.15.72:0/3330803324 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff0019c660 con 0x7eff00106a40 2024-03-22T01:46:01.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.941+0000 7efeef7fe640 1 -- 172.21.15.72:0/3330803324 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efef8005020 con 0x7eff00106a40 2024-03-22T01:46:01.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.941+0000 7efeef7fe640 1 -- 172.21.15.72:0/3330803324 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef8010690 con 0x7eff00106a40 2024-03-22T01:46:01.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.941+0000 7eff04b7a640 1 -- 172.21.15.72:0/3330803324 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff00068e60 con 0x7eff00106a40 2024-03-22T01:46:01.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.943+0000 7efeef7fe640 1 -- 172.21.15.72:0/3330803324 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 25) ==== 137985+0+0 (secure 0 0 0) 0x7efef8005520 con 0x7eff00106a40 2024-03-22T01:46:01.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.944+0000 7eff07839740 1 Processor -- start 2024-03-22T01:46:01.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.944+0000 7eff07839740 1 -- start start 2024-03-22T01:46:01.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.944+0000 7eff07839740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e2f2a6f800 0x55e2f2a6fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:01.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.944+0000 7eff07839740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2f2a70120 0x55e2f2a787c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:01.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.944+0000 7eff07839740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2f2a78d00 0x55e2f2a7b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:01.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.944+0000 7efeef7fe640 1 --2- 172.21.15.72:0/3330803324 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7efee80a3090 0x7efee80a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:01.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.944+0000 7eff07839740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55e2f29d38d0 con 0x55e2f2a6f800 2024-03-22T01:46:01.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.944+0000 7eff07839740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55e2f297be00 con 0x55e2f2a70120 2024-03-22T01:46:01.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.944+0000 7eff07839740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e2f29cf880 con 0x55e2f2a78d00 2024-03-22T01:46:01.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.944+0000 7efefdd9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e2f2a6f800 0x55e2f2a6fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:01.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.944+0000 7efefdd9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e2f2a6f800 0x55e2f2a6fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38500/0 (socket says 172.21.15.72:38500) 2024-03-22T01:46:01.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.944+0000 7efefdd9b640 1 -- 172.21.15.72:0/2955604997 learned_addr learned my addr 172.21.15.72:0/2955604997 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:01.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.944+0000 7efefed9d640 1 --2- 172.21.15.72:0/3330803324 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7efee80a3090 0x7efee80a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:01.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.944+0000 7efefe59c640 1 --2- 172.21.15.72:0/2955604997 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2f2a70120 0x55e2f2a787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:01.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.944+0000 7efeef7fe640 1 -- 172.21.15.72:0/3330803324 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7efef80d0130 con 0x7eff00106a40 2024-03-22T01:46:01.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.945+0000 7efefe59c640 1 -- 172.21.15.72:0/2955604997 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2f2a78d00 msgr2=0x55e2f2a7b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:01.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.945+0000 7efefe59c640 1 --2- 172.21.15.72:0/2955604997 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2f2a78d00 0x55e2f2a7b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.945+0000 7efefe59c640 1 -- 172.21.15.72:0/2955604997 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e2f2a6f800 msgr2=0x55e2f2a6fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:01.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.945+0000 7efefe59c640 1 --2- 172.21.15.72:0/2955604997 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e2f2a6f800 0x55e2f2a6fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.945+0000 7efefe59c640 1 -- 172.21.15.72:0/2955604997 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e2f26e73f0 con 0x55e2f2a70120 2024-03-22T01:46:01.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.945+0000 7efefdd9b640 1 --2- 172.21.15.72:0/2955604997 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e2f2a6f800 0x55e2f2a6fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:46:01.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.945+0000 7efefe59c640 1 --2- 172.21.15.72:0/2955604997 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2f2a70120 0x55e2f2a787c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7efef4005fd0 tx=0x7efef4012820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e3ff205f757759a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:01.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.945+0000 7efefed9d640 1 --2- 172.21.15.72:0/3330803324 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7efee80a3090 0x7efee80a5550 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7efef80cd580 tx=0x7efef80bc040 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:01.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.945+0000 7efeecff9640 1 -- 172.21.15.72:0/2955604997 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef401b420 con 0x55e2f2a70120 2024-03-22T01:46:01.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.945+0000 7efeecff9640 1 -- 172.21.15.72:0/2955604997 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efef400d8e0 con 0x55e2f2a70120 2024-03-22T01:46:01.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.945+0000 7efeecff9640 1 -- 172.21.15.72:0/2955604997 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef401a8a0 con 0x55e2f2a70120 2024-03-22T01:46:01.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.946+0000 7eff07839740 1 -- 172.21.15.72:0/2955604997 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2f2a70120 msgr2=0x55e2f2a787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:01.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.946+0000 7eff07839740 1 --2- 172.21.15.72:0/2955604997 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2f2a70120 0x55e2f2a787c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7efef4005fd0 tx=0x7efef4012820 comp rx=0 tx=0).stop 2024-03-22T01:46:01.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.946+0000 7eff07839740 1 -- 172.21.15.72:0/2955604997 shutdown_connections 2024-03-22T01:46:01.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.946+0000 7eff07839740 1 --2- 172.21.15.72:0/2955604997 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2f2a70120 0x55e2f2a787c0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.946+0000 7eff07839740 1 --2- 172.21.15.72:0/2955604997 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2f2a78d00 0x55e2f2a7b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.946+0000 7eff07839740 1 --2- 172.21.15.72:0/2955604997 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e2f2a6f800 0x55e2f2a6fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.946+0000 7eff07839740 1 -- 172.21.15.72:0/2955604997 >> 172.21.15.72:0/2955604997 conn(0x55e2f2a69f30 msgr2=0x55e2f2a6a310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:01.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.946+0000 7eff07839740 1 -- 172.21.15.72:0/2955604997 shutdown_connections 2024-03-22T01:46:01.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.946+0000 7eff07839740 1 -- 172.21.15.72:0/2955604997 wait complete. 2024-03-22T01:46:01.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.946+0000 7eff07839740 1 -- start start 2024-03-22T01:46:01.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.946+0000 7eff07839740 1 Processor -- start 2024-03-22T01:46:01.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.946+0000 7eff07839740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2f2a6f800 0x55e2f2a75d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:01.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.946+0000 7eff07839740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e2f2a70120 0x55e2f2a76280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:01.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.946+0000 7eff07839740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2f2a78d00 0x55e2f2a767c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:01.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.946+0000 7eff07839740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55e2f2a7baf0 con 0x55e2f2a70120 2024-03-22T01:46:01.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.946+0000 7eff07839740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55e2f298f300 con 0x55e2f2a78d00 2024-03-22T01:46:01.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.946+0000 7eff07839740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e2f2a7bc70 con 0x55e2f2a6f800 2024-03-22T01:46:01.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.946+0000 7eff07839740 10 client.0 _ll_register_callbacks cb 0x55e2f29cae30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:46:01.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.947+0000 7efefe59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e2f2a70120 0x55e2f2a76280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:01.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.947+0000 7efefe59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e2f2a70120 0x55e2f2a76280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38514/0 (socket says 172.21.15.72:38514) 2024-03-22T01:46:01.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.947+0000 7efefe59c640 1 -- 172.21.15.72:0/989905575 learned_addr learned my addr 172.21.15.72:0/989905575 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:01.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.947+0000 7efefdd9b640 1 --2- 172.21.15.72:0/989905575 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2f2a78d00 0x55e2f2a767c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:01.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.947+0000 7efefdd9b640 1 --2- 172.21.15.72:0/989905575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2f2a6f800 0x55e2f2a75d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:01.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.947+0000 7efefe59c640 1 -- 172.21.15.72:0/989905575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2f2a6f800 msgr2=0x55e2f2a75d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:01.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.947+0000 7efefe59c640 1 --2- 172.21.15.72:0/989905575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2f2a6f800 0x55e2f2a75d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.947+0000 7efefe59c640 1 -- 172.21.15.72:0/989905575 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2f2a78d00 msgr2=0x55e2f2a767c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:01.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.947+0000 7efefe59c640 1 --2- 172.21.15.72:0/989905575 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2f2a78d00 0x55e2f2a767c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.947+0000 7efefe59c640 1 -- 172.21.15.72:0/989905575 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e2f27b17d0 con 0x55e2f2a70120 2024-03-22T01:46:01.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.947+0000 7efefdd9b640 1 --2- 172.21.15.72:0/989905575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2f2a6f800 0x55e2f2a75d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:46:01.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.947+0000 7efefe59c640 1 --2- 172.21.15.72:0/989905575 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e2f2a70120 0x55e2f2a76280 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7efef4012d20 tx=0x7efef400d690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25842d1f7b4cd363 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:01.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.947+0000 7efee5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:46:01.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.947+0000 7efee5ffb640 1 -- 172.21.15.72:0/989905575 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef4009950 con 0x55e2f2a70120 2024-03-22T01:46:01.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.947+0000 7efee5ffb640 1 -- 172.21.15.72:0/989905575 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7efef4020040 con 0x55e2f2a70120 2024-03-22T01:46:01.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.947+0000 7eff07839740 1 -- 172.21.15.72:0/989905575 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efef400d8e0 con 0x55e2f2a70120 2024-03-22T01:46:01.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.947+0000 7efee5ffb640 1 -- 172.21.15.72:0/989905575 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7efef4033050 con 0x55e2f2a70120 2024-03-22T01:46:01.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.947+0000 7eff07839740 10 client.16173 fetch_fsmap learned FSMap version 103 2024-03-22T01:46:01.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.947+0000 7eff07839740 1 -- 172.21.15.72:0/989905575 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=4+}) -- 0x55e2f2a87de0 con 0x55e2f2a70120 2024-03-22T01:46:01.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.947+0000 7efee5ffb640 1 -- 172.21.15.72:0/989905575 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efef400d8e0 con 0x55e2f2a70120 2024-03-22T01:46:01.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.948+0000 7efee5ffb640 1 -- 172.21.15.72:0/989905575 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 103) ==== 8330+0+0 (secure 0 0 0) 0x7efef4024070 con 0x55e2f2a70120 2024-03-22T01:46:01.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.948+0000 7eff07839740 10 client.16173 fetch_fsmap finished waiting for FSMap version 103 2024-03-22T01:46:01.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.948+0000 7eff07839740 10 client.16173 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T01:46:01.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.948+0000 7eff07839740 20 client.16173 populate_metadata read hostname 'smithi072' 2024-03-22T01:46:01.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.948+0000 7eff07839740 1 --2- 172.21.15.72:0/989905575 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55e2f2a89240 0x55e2f2a8b630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:01.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.948+0000 7eff07839740 4 client.16173 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T01:46:01.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.948+0000 7eff07839740 1 -- 172.21.15.72:0/989905575 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e2f29e0e10 con 0x55e2f2a89240 2024-03-22T01:46:01.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.948+0000 7efefdd9b640 1 --2- 172.21.15.72:0/989905575 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55e2f2a89240 0x55e2f2a8b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:01.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.948+0000 7efee5ffb640 10 client.16173 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:46:01.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.948+0000 7efefdd9b640 1 --2- 172.21.15.72:0/989905575 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55e2f2a89240 0x55e2f2a8b630 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4f6033f52c987bb server_cookie=11ccc4356d986be0 in_seq=0 out_seq=0 2024-03-22T01:46:01.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.949+0000 7efee5ffb640 1 -- 172.21.15.72:0/989905575 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e2f29e0e10 con 0x55e2f2a89240 2024-03-22T01:46:01.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.949+0000 7efee5ffb640 10 client.16173 handle_command_reply: tid=0 2024-03-22T01:46:01.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.949+0000 7eff07839740 1 client.16173 shutdown 2024-03-22T01:46:01.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.949+0000 7eff07839740 1 -- 172.21.15.72:0/989905575 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e2f2a70120 msgr2=0x55e2f2a76280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:01.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.949+0000 7eff07839740 1 --2- 172.21.15.72:0/989905575 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e2f2a70120 0x55e2f2a76280 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7efef4012d20 tx=0x7efef400d690 comp rx=0 tx=0).stop 2024-03-22T01:46:01.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.949+0000 7eff07839740 1 -- 172.21.15.72:0/989905575 shutdown_connections 2024-03-22T01:46:01.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.949+0000 7eff07839740 1 --2- 172.21.15.72:0/989905575 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55e2f2a89240 0x55e2f2a8b630 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.949+0000 7eff07839740 1 --2- 172.21.15.72:0/989905575 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2f2a78d00 0x55e2f2a767c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.949+0000 7eff07839740 1 --2- 172.21.15.72:0/989905575 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e2f2a70120 0x55e2f2a76280 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.949+0000 7eff07839740 1 --2- 172.21.15.72:0/989905575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2f2a6f800 0x55e2f2a75d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.949+0000 7eff07839740 1 -- 172.21.15.72:0/989905575 >> 172.21.15.72:0/989905575 conn(0x55e2f2a69f30 msgr2=0x55e2f2a6bc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:01.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.949+0000 7efee5ffb640 0 client.16173 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:46:01.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.949+0000 7eff07839740 1 -- 172.21.15.72:0/989905575 shutdown_connections 2024-03-22T01:46:01.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.949+0000 7eff07839740 1 -- 172.21.15.72:0/989905575 wait complete. 2024-03-22T01:46:01.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.949+0000 7eff07839740 20 client.16173 trim_cache size 0 max 16384 2024-03-22T01:46:01.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.962+0000 7eff07839740 1 Processor -- start 2024-03-22T01:46:01.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.962+0000 7eff07839740 1 -- start start 2024-03-22T01:46:01.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.962+0000 7eff07839740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2f2a6f800 0x55e2f2a848c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:01.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.962+0000 7eff07839740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e2f2a70120 0x55e2f2a6ba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:01.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.962+0000 7eff07839740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2f2a78d00 0x55e2f2a90e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:01.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.962+0000 7eff07839740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7eff001aaf80 con 0x55e2f2a70120 2024-03-22T01:46:01.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.962+0000 7eff07839740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7eff001ad800 con 0x55e2f2a78d00 2024-03-22T01:46:01.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.962+0000 7eff07839740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7eff001ab5f0 con 0x55e2f2a6f800 2024-03-22T01:46:01.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.962+0000 7efefe59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e2f2a70120 0x55e2f2a6ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:01.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.962+0000 7efefe59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e2f2a70120 0x55e2f2a6ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38526/0 (socket says 172.21.15.72:38526) 2024-03-22T01:46:01.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.962+0000 7efefe59c640 1 -- 172.21.15.72:0/2544616315 learned_addr learned my addr 172.21.15.72:0/2544616315 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:01.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.962+0000 7efefdd9b640 1 --2- 172.21.15.72:0/2544616315 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2f2a78d00 0x55e2f2a90e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:01.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.962+0000 7efefe59c640 1 -- 172.21.15.72:0/2544616315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2f2a6f800 msgr2=0x55e2f2a848c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:01.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.962+0000 7efefe59c640 1 --2- 172.21.15.72:0/2544616315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2f2a6f800 0x55e2f2a848c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.962+0000 7efefe59c640 1 -- 172.21.15.72:0/2544616315 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2f2a78d00 msgr2=0x55e2f2a90e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:01.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.962+0000 7efefe59c640 1 --2- 172.21.15.72:0/2544616315 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2f2a78d00 0x55e2f2a90e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.963+0000 7efefe59c640 1 -- 172.21.15.72:0/2544616315 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e2f2a69d60 con 0x55e2f2a70120 2024-03-22T01:46:01.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.963+0000 7efefe59c640 1 --2- 172.21.15.72:0/2544616315 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e2f2a70120 0x55e2f2a6ba90 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7efef40256f0 tx=0x7efef4005b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82aada90b33d70d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:01.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.963+0000 7efee7fff640 1 -- 172.21.15.72:0/2544616315 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef4009e90 con 0x55e2f2a70120 2024-03-22T01:46:01.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.963+0000 7efee7fff640 1 -- 172.21.15.72:0/2544616315 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efef4017030 con 0x55e2f2a70120 2024-03-22T01:46:01.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.963+0000 7efee7fff640 1 -- 172.21.15.72:0/2544616315 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef4025770 con 0x55e2f2a70120 2024-03-22T01:46:01.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.963+0000 7eff07839740 1 -- 172.21.15.72:0/2544616315 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e2f2a70120 msgr2=0x55e2f2a6ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:01.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.963+0000 7eff07839740 1 --2- 172.21.15.72:0/2544616315 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e2f2a70120 0x55e2f2a6ba90 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7efef40256f0 tx=0x7efef4005b90 comp rx=0 tx=0).stop 2024-03-22T01:46:01.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.963+0000 7eff07839740 1 -- 172.21.15.72:0/2544616315 shutdown_connections 2024-03-22T01:46:01.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.963+0000 7eff07839740 1 --2- 172.21.15.72:0/2544616315 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2f2a78d00 0x55e2f2a90e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.963+0000 7eff07839740 1 --2- 172.21.15.72:0/2544616315 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e2f2a70120 0x55e2f2a6ba90 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.963+0000 7eff07839740 1 --2- 172.21.15.72:0/2544616315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2f2a6f800 0x55e2f2a848c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.963+0000 7eff07839740 1 -- 172.21.15.72:0/2544616315 >> 172.21.15.72:0/2544616315 conn(0x55e2f2a69f30 msgr2=0x55e2f2a6b380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:01.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.963+0000 7eff07839740 1 -- 172.21.15.72:0/2544616315 shutdown_connections 2024-03-22T01:46:01.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.963+0000 7eff07839740 1 -- 172.21.15.72:0/2544616315 wait complete. 2024-03-22T01:46:01.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.964+0000 7eff07839740 1 -- start start 2024-03-22T01:46:01.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.964+0000 7eff07839740 1 Processor -- start 2024-03-22T01:46:01.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.964+0000 7eff07839740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e2f2a6f800 0x55e2f2a81140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:01.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.964+0000 7eff07839740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2f2a70120 0x55e2f2a7c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:01.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.964+0000 7eff07839740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2f2a78d00 0x55e2f2a7c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:01.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.964+0000 7eff07839740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7eff001abcc0 con 0x55e2f2a6f800 2024-03-22T01:46:01.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.964+0000 7eff07839740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7eff001ad130 con 0x55e2f2a78d00 2024-03-22T01:46:01.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.964+0000 7eff07839740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7eff001ac390 con 0x55e2f2a70120 2024-03-22T01:46:01.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.964+0000 7eff07839740 10 client.0 _ll_register_callbacks cb 0x7efef80093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:46:01.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.964+0000 7efefdd9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e2f2a6f800 0x55e2f2a81140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:01.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.964+0000 7efefdd9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e2f2a6f800 0x55e2f2a81140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38542/0 (socket says 172.21.15.72:38542) 2024-03-22T01:46:01.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.964+0000 7efefdd9b640 1 -- 172.21.15.72:0/2200486185 learned_addr learned my addr 172.21.15.72:0/2200486185 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:01.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.964+0000 7efefdd9b640 1 --2- 172.21.15.72:0/2200486185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2f2a78d00 0x55e2f2a7c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:01.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.964+0000 7efefdd9b640 1 -- 172.21.15.72:0/2200486185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2f2a70120 msgr2=0x55e2f2a7c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:01.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.964+0000 7efefdd9b640 1 --2- 172.21.15.72:0/2200486185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2f2a70120 0x55e2f2a7c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.964+0000 7efefdd9b640 1 -- 172.21.15.72:0/2200486185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2f2a78d00 msgr2=0x55e2f2a7c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:01.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.964+0000 7efefdd9b640 1 --2- 172.21.15.72:0/2200486185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2f2a78d00 0x55e2f2a7c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.964+0000 7efefdd9b640 1 -- 172.21.15.72:0/2200486185 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e2f2a7b0f0 con 0x55e2f2a6f800 2024-03-22T01:46:01.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.964+0000 7efefdd9b640 1 --2- 172.21.15.72:0/2200486185 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e2f2a6f800 0x55e2f2a81140 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7efef002c170 tx=0x7efef0005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b6342b635d50a00 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:01.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.964+0000 7efee5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:46:01.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.965+0000 7efee5ffb640 1 -- 172.21.15.72:0/2200486185 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef0004480 con 0x55e2f2a6f800 2024-03-22T01:46:01.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.965+0000 7efee5ffb640 1 -- 172.21.15.72:0/2200486185 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7efef0005030 con 0x55e2f2a6f800 2024-03-22T01:46:01.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.965+0000 7eff07839740 1 -- 172.21.15.72:0/2200486185 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efef4017030 con 0x55e2f2a6f800 2024-03-22T01:46:01.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.965+0000 7efee5ffb640 1 -- 172.21.15.72:0/2200486185 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7efef0005230 con 0x55e2f2a6f800 2024-03-22T01:46:01.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.965+0000 7eff07839740 10 client.16179 fetch_fsmap learned FSMap version 103 2024-03-22T01:46:01.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.965+0000 7eff07839740 1 -- 172.21.15.72:0/2200486185 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=4+}) -- 0x55e2f2aef2e0 con 0x55e2f2a6f800 2024-03-22T01:46:01.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.965+0000 7efee5ffb640 1 -- 172.21.15.72:0/2200486185 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efef4017030 con 0x55e2f2a6f800 2024-03-22T01:46:01.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.965+0000 7efee5ffb640 1 -- 172.21.15.72:0/2200486185 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 103) ==== 8330+0+0 (secure 0 0 0) 0x7efef0003710 con 0x55e2f2a6f800 2024-03-22T01:46:01.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.965+0000 7eff07839740 10 client.16179 fetch_fsmap finished waiting for FSMap version 103 2024-03-22T01:46:01.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.965+0000 7eff07839740 10 client.16179 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T01:46:01.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.965+0000 7eff07839740 20 client.16179 populate_metadata read hostname 'smithi072' 2024-03-22T01:46:01.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.965+0000 7eff07839740 1 --2- 172.21.15.72:0/2200486185 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55e2f2aef9e0 0x55e2f2af1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:01.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.965+0000 7eff07839740 4 client.16179 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T01:46:01.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.965+0000 7eff07839740 1 -- 172.21.15.72:0/2200486185 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e2f29db160 con 0x55e2f2aef9e0 2024-03-22T01:46:01.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.965+0000 7efefe59c640 1 --2- 172.21.15.72:0/2200486185 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55e2f2aef9e0 0x55e2f2af1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:01.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.966+0000 7efee5ffb640 10 client.16179 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:46:01.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.966+0000 7efefe59c640 1 --2- 172.21.15.72:0/2200486185 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55e2f2aef9e0 0x55e2f2af1e40 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=30365c19c0d8a245 server_cookie=7a73b7b49c767f2c in_seq=0 out_seq=0 2024-03-22T01:46:01.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.966+0000 7efee5ffb640 1 -- 172.21.15.72:0/2200486185 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e2f29db160 con 0x55e2f2aef9e0 2024-03-22T01:46:01.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.966+0000 7efee5ffb640 10 client.16179 handle_command_reply: tid=0 2024-03-22T01:46:01.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.966+0000 7eff07839740 1 client.16179 shutdown 2024-03-22T01:46:01.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.966+0000 7eff07839740 1 -- 172.21.15.72:0/2200486185 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e2f2a6f800 msgr2=0x55e2f2a81140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:01.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.966+0000 7eff07839740 1 --2- 172.21.15.72:0/2200486185 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e2f2a6f800 0x55e2f2a81140 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7efef002c170 tx=0x7efef0005fd0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.966+0000 7eff07839740 1 -- 172.21.15.72:0/2200486185 shutdown_connections 2024-03-22T01:46:01.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.966+0000 7eff07839740 1 --2- 172.21.15.72:0/2200486185 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55e2f2aef9e0 0x55e2f2af1e40 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.966+0000 7eff07839740 1 --2- 172.21.15.72:0/2200486185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2f2a78d00 0x55e2f2a7c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.966+0000 7efee5ffb640 0 client.16179 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:46:01.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.966+0000 7eff07839740 1 --2- 172.21.15.72:0/2200486185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e2f2a70120 0x55e2f2a7c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.966+0000 7eff07839740 1 --2- 172.21.15.72:0/2200486185 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55e2f2a6f800 0x55e2f2a81140 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.966+0000 7eff07839740 1 -- 172.21.15.72:0/2200486185 >> 172.21.15.72:0/2200486185 conn(0x55e2f2a69f30 msgr2=0x55e2f2a68740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:01.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.966+0000 7eff07839740 1 -- 172.21.15.72:0/2200486185 shutdown_connections 2024-03-22T01:46:01.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.966+0000 7eff07839740 1 -- 172.21.15.72:0/2200486185 wait complete. 2024-03-22T01:46:01.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.966+0000 7eff07839740 20 client.16179 trim_cache size 0 max 16384 2024-03-22T01:46:01.973 INFO:teuthology.orchestra.run.smithi072.stdout:[] 2024-03-22T01:46:01.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.966+0000 7eff04b7a640 1 -- 172.21.15.72:0/3330803324 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7efee80a3090 msgr2=0x7efee80a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:01.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.966+0000 7eff04b7a640 1 --2- 172.21.15.72:0/3330803324 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7efee80a3090 0x7efee80a5550 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7efef80cd580 tx=0x7efef80bc040 comp rx=0 tx=0).stop 2024-03-22T01:46:01.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.966+0000 7eff04b7a640 1 -- 172.21.15.72:0/3330803324 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff00106a40 msgr2=0x7eff0019bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:01.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.966+0000 7eff04b7a640 1 --2- 172.21.15.72:0/3330803324 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff00106a40 0x7eff0019bc40 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7efef800ea60 tx=0x7efef800ef20 comp rx=0 tx=0).stop 2024-03-22T01:46:01.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.966+0000 7eff04b7a640 1 -- 172.21.15.72:0/3330803324 shutdown_connections 2024-03-22T01:46:01.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.966+0000 7eff04b7a640 1 --2- 172.21.15.72:0/3330803324 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7efee80a3090 0x7efee80a5550 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.966+0000 7eff04b7a640 1 --2- 172.21.15.72:0/3330803324 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff00106a40 0x7eff0019bc40 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.966+0000 7eff04b7a640 1 --2- 172.21.15.72:0/3330803324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff001013f0 0x7eff001978c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.966+0000 7eff04b7a640 1 --2- 172.21.15.72:0/3330803324 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff00100a30 0x7eff00197380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:01.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.966+0000 7eff04b7a640 1 -- 172.21.15.72:0/3330803324 >> 172.21.15.72:0/3330803324 conn(0x7eff00077c40 msgr2=0x7eff00103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:01.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.967+0000 7eff04b7a640 1 -- 172.21.15.72:0/3330803324 shutdown_connections 2024-03-22T01:46:01.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:01.967+0000 7eff04b7a640 1 -- 172.21.15.72:0/3330803324 wait complete. 2024-03-22T01:46:02.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:46:02 smithi072 ceph-mon[32414]: pgmap v284: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 5.6 KiB/s wr, 5 op/s 2024-03-22T01:46:02.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:46:02 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/1587081135' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:46:02.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:46:02 smithi089 ceph-mon[35770]: pgmap v284: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 5.6 KiB/s wr, 5 op/s 2024-03-22T01:46:02.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:46:02 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/1587081135' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:46:02.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:46:02 smithi098 ceph-mon[36468]: pgmap v284: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 5.6 KiB/s wr, 5 op/s 2024-03-22T01:46:02.473 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:46:02 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/1587081135' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:46:02.489 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.454+0000 7f0263756640 1 -- 172.21.15.72:0/1706148078 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f025c102760 msgr2=0x7f025c1067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:02.490 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.454+0000 7f0263756640 1 --2- 172.21.15.72:0/1706148078 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f025c102760 0x7f025c1067a0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f0258009900 tx=0x7f025802bc30 comp rx=0 tx=0).stop 2024-03-22T01:46:02.490 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.454+0000 7f0263756640 1 -- 172.21.15.72:0/1706148078 shutdown_connections 2024-03-22T01:46:02.490 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.454+0000 7f0263756640 1 --2- 172.21.15.72:0/1706148078 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f025c106e80 0x7f025c10ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:02.490 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.454+0000 7f0263756640 1 --2- 172.21.15.72:0/1706148078 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f025c102760 0x7f025c1067a0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:02.490 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.454+0000 7f0263756640 1 --2- 172.21.15.72:0/1706148078 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f025c101db0 0x7f025c102190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:02.490 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.454+0000 7f0263756640 1 -- 172.21.15.72:0/1706148078 >> 172.21.15.72:0/1706148078 conn(0x7f025c0fd610 msgr2=0x7f025c0ffa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:02.490 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.454+0000 7f0263756640 1 -- 172.21.15.72:0/1706148078 shutdown_connections 2024-03-22T01:46:02.490 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.454+0000 7f0263756640 1 -- 172.21.15.72:0/1706148078 wait complete. 2024-03-22T01:46:02.490 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.455+0000 7f0263756640 1 Processor -- start 2024-03-22T01:46:02.491 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.455+0000 7f0263756640 1 -- start start 2024-03-22T01:46:02.491 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.455+0000 7f0263756640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f025c101db0 0x7f025c198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:02.491 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.455+0000 7f0263756640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f025c106e80 0x7f025c198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:02.491 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.455+0000 7f0263756640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f025c19ce10 0x7f025c19d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:02.491 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.455+0000 7f0263756640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f025c111f30 con 0x7f025c101db0 2024-03-22T01:46:02.491 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.455+0000 7f0263756640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f025c111db0 con 0x7f025c106e80 2024-03-22T01:46:02.491 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.455+0000 7f0263756640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f025c1120b0 con 0x7f025c19ce10 2024-03-22T01:46:02.491 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.455+0000 7f02614f2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f025c101db0 0x7f025c198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:02.491 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.455+0000 7f02614f2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f025c101db0 0x7f025c198550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38564/0 (socket says 172.21.15.72:38564) 2024-03-22T01:46:02.491 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.455+0000 7f02614f2640 1 -- 172.21.15.72:0/1837272877 learned_addr learned my addr 172.21.15.72:0/1837272877 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:02.491 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.455+0000 7f0260cf1640 1 --2- 172.21.15.72:0/1837272877 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f025c106e80 0x7f025c198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:02.491 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.455+0000 7f0261cf3640 1 --2- 172.21.15.72:0/1837272877 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f025c19ce10 0x7f025c19d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:02.491 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.455+0000 7f02614f2640 1 -- 172.21.15.72:0/1837272877 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f025c19ce10 msgr2=0x7f025c19d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:02.491 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.455+0000 7f02614f2640 1 --2- 172.21.15.72:0/1837272877 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f025c19ce10 0x7f025c19d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:02.492 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.455+0000 7f02614f2640 1 -- 172.21.15.72:0/1837272877 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f025c106e80 msgr2=0x7f025c198a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:02.492 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.455+0000 7f02614f2640 1 --2- 172.21.15.72:0/1837272877 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f025c106e80 0x7f025c198a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:02.492 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.455+0000 7f02614f2640 1 -- 172.21.15.72:0/1837272877 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f025c19d900 con 0x7f025c101db0 2024-03-22T01:46:02.492 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.456+0000 7f02614f2640 1 --2- 172.21.15.72:0/1837272877 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f025c101db0 0x7f025c198550 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f024c00d770 tx=0x7f024c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:02.492 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.456+0000 7f02527fc640 1 -- 172.21.15.72:0/1837272877 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f024c017070 con 0x7f025c101db0 2024-03-22T01:46:02.492 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.456+0000 7f0263756640 1 -- 172.21.15.72:0/1837272877 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f025c19dbf0 con 0x7f025c101db0 2024-03-22T01:46:02.492 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.456+0000 7f02527fc640 1 -- 172.21.15.72:0/1837272877 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f024c005500 con 0x7f025c101db0 2024-03-22T01:46:02.492 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.456+0000 7f02527fc640 1 -- 172.21.15.72:0/1837272877 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f024c004e00 con 0x7f025c101db0 2024-03-22T01:46:02.492 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.456+0000 7f0263756640 1 -- 172.21.15.72:0/1837272877 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f025c1a53b0 con 0x7f025c101db0 2024-03-22T01:46:02.492 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.458+0000 7f02527fc640 1 -- 172.21.15.72:0/1837272877 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 25) ==== 137985+0+0 (secure 0 0 0) 0x7f024c004fa0 con 0x7f025c101db0 2024-03-22T01:46:02.492 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.458+0000 7f0266419740 1 Processor -- start 2024-03-22T01:46:02.492 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.458+0000 7f0266419740 1 -- start start 2024-03-22T01:46:02.492 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.459+0000 7f0266419740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5589376912d0 0x5589376916b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:02.492 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.459+0000 7f0266419740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558937691bf0 0x55893769a480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:02.492 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.459+0000 7f02527fc640 1 --2- 172.21.15.72:0/1837272877 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f02480a3080 0x7f02480a5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:02.493 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.459+0000 7f0266419740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55893769a9c0 0x55893769cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:02.493 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.459+0000 7f0266419740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x5589375f5040 con 0x55893769a9c0 2024-03-22T01:46:02.493 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.459+0000 7f0266419740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55893759d9e0 con 0x5589376912d0 2024-03-22T01:46:02.493 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.459+0000 7f0266419740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5589375f0ff0 con 0x558937691bf0 2024-03-22T01:46:02.493 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.459+0000 7f0260cf1640 1 --2- 172.21.15.72:0/1837272877 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f02480a3080 0x7f02480a5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:02.493 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.459+0000 7f0261cf3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5589376912d0 0x5589376916b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:02.493 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.459+0000 7f0261cf3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5589376912d0 0x5589376916b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.72:33408/0 (socket says 172.21.15.72:33408) 2024-03-22T01:46:02.493 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.459+0000 7f0261cf3640 1 -- 172.21.15.72:0/1914304622 learned_addr learned my addr 172.21.15.72:0/1914304622 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:02.493 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.459+0000 7f0261cf3640 1 --2- 172.21.15.72:0/1914304622 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55893769a9c0 0x55893769cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:02.493 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.459+0000 7f02527fc640 1 -- 172.21.15.72:0/1837272877 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f024c0cf610 con 0x7f025c101db0 2024-03-22T01:46:02.493 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.459+0000 7f02614f2640 1 --2- 172.21.15.72:0/1914304622 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558937691bf0 0x55893769a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:02.493 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.459+0000 7f0261cf3640 1 -- 172.21.15.72:0/1914304622 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558937691bf0 msgr2=0x55893769a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:02.493 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.459+0000 7f0261cf3640 1 --2- 172.21.15.72:0/1914304622 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558937691bf0 0x55893769a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:02.494 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.459+0000 7f0261cf3640 1 -- 172.21.15.72:0/1914304622 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55893769a9c0 msgr2=0x55893769cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:02.494 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.459+0000 7f0261cf3640 1 --2- 172.21.15.72:0/1914304622 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55893769a9c0 0x55893769cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:02.494 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.459+0000 7f0261cf3640 1 -- 172.21.15.72:0/1914304622 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558937309210 con 0x5589376912d0 2024-03-22T01:46:02.494 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.459+0000 7f02614f2640 1 --2- 172.21.15.72:0/1914304622 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558937691bf0 0x55893769a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:46:02.494 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.459+0000 7f0260cf1640 1 --2- 172.21.15.72:0/1837272877 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f02480a3080 0x7f02480a5540 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0258009230 tx=0x7f0258005fd0 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:02.494 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.459+0000 7f0261cf3640 1 --2- 172.21.15.72:0/1914304622 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5589376912d0 0x5589376916b0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f0254012a20 tx=0x7f0254009a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47db984b2469b6fc server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:02.494 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.459+0000 7f02477fe640 1 -- 172.21.15.72:0/1914304622 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0254012110 con 0x5589376912d0 2024-03-22T01:46:02.494 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.459+0000 7f02477fe640 1 -- 172.21.15.72:0/1914304622 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0254003e50 con 0x5589376912d0 2024-03-22T01:46:02.494 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.459+0000 7f02477fe640 1 -- 172.21.15.72:0/1914304622 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0254004040 con 0x5589376912d0 2024-03-22T01:46:02.494 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.460+0000 7f0266419740 1 -- 172.21.15.72:0/1914304622 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5589376912d0 msgr2=0x5589376916b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:02.494 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.460+0000 7f0266419740 1 --2- 172.21.15.72:0/1914304622 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5589376912d0 0x5589376916b0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f0254012a20 tx=0x7f0254009a20 comp rx=0 tx=0).stop 2024-03-22T01:46:02.494 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.460+0000 7f0266419740 1 -- 172.21.15.72:0/1914304622 shutdown_connections 2024-03-22T01:46:02.494 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.460+0000 7f0266419740 1 --2- 172.21.15.72:0/1914304622 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55893769a9c0 0x55893769cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:02.494 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.460+0000 7f0266419740 1 --2- 172.21.15.72:0/1914304622 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558937691bf0 0x55893769a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:02.495 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.460+0000 7f0266419740 1 --2- 172.21.15.72:0/1914304622 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5589376912d0 0x5589376916b0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:02.495 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.460+0000 7f0266419740 1 -- 172.21.15.72:0/1914304622 >> 172.21.15.72:0/1914304622 conn(0x55893768bbf0 msgr2=0x55893768bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:02.495 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.460+0000 7f0266419740 1 -- 172.21.15.72:0/1914304622 shutdown_connections 2024-03-22T01:46:02.495 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.460+0000 7f0266419740 1 -- 172.21.15.72:0/1914304622 wait complete. 2024-03-22T01:46:02.495 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.460+0000 7f0266419740 1 -- start start 2024-03-22T01:46:02.495 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.461+0000 7f0266419740 1 Processor -- start 2024-03-22T01:46:02.495 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.461+0000 7f0266419740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5589376912d0 0x55893769a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:02.495 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.461+0000 7f0266419740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558937691bf0 0x5589376976b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:02.495 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.461+0000 7f0266419740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55893769a9c0 0x558937697c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:02.495 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.461+0000 7f0266419740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55893769d5f0 con 0x55893769a9c0 2024-03-22T01:46:02.495 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.461+0000 7f0266419740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5589375b0db0 con 0x558937691bf0 2024-03-22T01:46:02.496 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.461+0000 7f0266419740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55893769d770 con 0x5589376912d0 2024-03-22T01:46:02.496 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.461+0000 7f0266419740 10 client.0 _ll_register_callbacks cb 0x558937308a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:46:02.496 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.461+0000 7f0261cf3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55893769a9c0 0x558937697c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:02.496 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.461+0000 7f0261cf3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55893769a9c0 0x558937697c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38586/0 (socket says 172.21.15.72:38586) 2024-03-22T01:46:02.496 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.461+0000 7f0261cf3640 1 -- 172.21.15.72:0/4145276135 learned_addr learned my addr 172.21.15.72:0/4145276135 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:02.496 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.461+0000 7f0260cf1640 1 --2- 172.21.15.72:0/4145276135 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558937691bf0 0x5589376976b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:02.496 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.461+0000 7f0261cf3640 1 -- 172.21.15.72:0/4145276135 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5589376912d0 msgr2=0x55893769a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:02.496 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.461+0000 7f0261cf3640 1 --2- 172.21.15.72:0/4145276135 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5589376912d0 0x55893769a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:02.496 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.461+0000 7f0261cf3640 1 -- 172.21.15.72:0/4145276135 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558937691bf0 msgr2=0x5589376976b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:02.496 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.461+0000 7f0261cf3640 1 --2- 172.21.15.72:0/4145276135 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558937691bf0 0x5589376976b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:02.496 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.461+0000 7f0261cf3640 1 -- 172.21.15.72:0/4145276135 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5589373d5620 con 0x55893769a9c0 2024-03-22T01:46:02.496 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.462+0000 7f0261cf3640 1 --2- 172.21.15.72:0/4145276135 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55893769a9c0 0x558937697c10 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f0254006010 tx=0x7f0254003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c21e4c6f1de23cb1 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:02.496 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.462+0000 7f0244ff9640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:46:02.496 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.462+0000 7f0244ff9640 1 -- 172.21.15.72:0/4145276135 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02540109d0 con 0x55893769a9c0 2024-03-22T01:46:02.497 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.462+0000 7f0244ff9640 1 -- 172.21.15.72:0/4145276135 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f0254004000 con 0x55893769a9c0 2024-03-22T01:46:02.497 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.462+0000 7f0266419740 1 -- 172.21.15.72:0/4145276135 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0254003e50 con 0x55893769a9c0 2024-03-22T01:46:02.497 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.462+0000 7f0244ff9640 1 -- 172.21.15.72:0/4145276135 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7f0254010b70 con 0x55893769a9c0 2024-03-22T01:46:02.497 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.462+0000 7f0266419740 10 client.16191 fetch_fsmap learned FSMap version 103 2024-03-22T01:46:02.497 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.462+0000 7f0266419740 1 -- 172.21.15.72:0/4145276135 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=4+}) -- 0x5589376a9490 con 0x55893769a9c0 2024-03-22T01:46:02.497 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.462+0000 7f0244ff9640 1 -- 172.21.15.72:0/4145276135 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0254003e50 con 0x55893769a9c0 2024-03-22T01:46:02.497 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.462+0000 7f0244ff9640 1 -- 172.21.15.72:0/4145276135 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 103) ==== 8330+0+0 (secure 0 0 0) 0x7f0254026020 con 0x55893769a9c0 2024-03-22T01:46:02.497 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.462+0000 7f0266419740 10 client.16191 fetch_fsmap finished waiting for FSMap version 103 2024-03-22T01:46:02.497 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.462+0000 7f0266419740 10 client.16191 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T01:46:02.497 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.462+0000 7f0266419740 20 client.16191 populate_metadata read hostname 'smithi072' 2024-03-22T01:46:02.497 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.462+0000 7f0266419740 1 --2- 172.21.15.72:0/4145276135 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x5589376aa830 0x5589376acc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:02.497 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.462+0000 7f0266419740 4 client.16191 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T01:46:02.497 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.462+0000 7f0266419740 1 -- 172.21.15.72:0/4145276135 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558937602580 con 0x5589376aa830 2024-03-22T01:46:02.497 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.462+0000 7f02614f2640 1 --2- 172.21.15.72:0/4145276135 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x5589376aa830 0x5589376acc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:02.498 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.463+0000 7f02614f2640 1 --2- 172.21.15.72:0/4145276135 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x5589376aa830 0x5589376acc20 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6d2d2db7adab2cb server_cookie=8c39051d361bacc2 in_seq=0 out_seq=0 2024-03-22T01:46:02.498 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.463+0000 7f0244ff9640 10 client.16191 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:46:02.498 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.463+0000 7f0244ff9640 1 -- 172.21.15.72:0/4145276135 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558937602580 con 0x5589376aa830 2024-03-22T01:46:02.498 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.463+0000 7f0244ff9640 10 client.16191 handle_command_reply: tid=0 2024-03-22T01:46:02.498 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.463+0000 7f0266419740 1 client.16191 shutdown 2024-03-22T01:46:02.498 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.464+0000 7f0266419740 1 -- 172.21.15.72:0/4145276135 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55893769a9c0 msgr2=0x558937697c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:02.498 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.464+0000 7f0266419740 1 --2- 172.21.15.72:0/4145276135 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55893769a9c0 0x558937697c10 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f0254006010 tx=0x7f0254003970 comp rx=0 tx=0).stop 2024-03-22T01:46:02.498 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.464+0000 7f0266419740 1 -- 172.21.15.72:0/4145276135 shutdown_connections 2024-03-22T01:46:02.498 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.464+0000 7f0266419740 1 --2- 172.21.15.72:0/4145276135 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x5589376aa830 0x5589376acc20 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:02.498 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.464+0000 7f0266419740 1 --2- 172.21.15.72:0/4145276135 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558937691bf0 0x5589376976b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:02.498 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.464+0000 7f0266419740 1 --2- 172.21.15.72:0/4145276135 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55893769a9c0 0x558937697c10 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:02.498 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.464+0000 7f0244ff9640 0 client.16191 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:46:02.498 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.464+0000 7f0266419740 1 --2- 172.21.15.72:0/4145276135 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5589376912d0 0x55893769a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:02.499 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.464+0000 7f0266419740 1 -- 172.21.15.72:0/4145276135 >> 172.21.15.72:0/4145276135 conn(0x55893768bbf0 msgr2=0x55893768d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:02.499 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.464+0000 7f0266419740 1 -- 172.21.15.72:0/4145276135 shutdown_connections 2024-03-22T01:46:02.499 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.464+0000 7f0266419740 1 -- 172.21.15.72:0/4145276135 wait complete. 2024-03-22T01:46:02.499 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.464+0000 7f0266419740 20 client.16191 trim_cache size 0 max 16384 2024-03-22T01:46:02.499 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.478+0000 7f0266419740 1 Processor -- start 2024-03-22T01:46:02.499 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.478+0000 7f0266419740 1 -- start start 2024-03-22T01:46:02.499 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.479+0000 7f0266419740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5589376912d0 0x5589376a6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:02.499 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.479+0000 7f0266419740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558937691bf0 0x55893768aad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:02.499 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.479+0000 7f0266419740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55893769a9c0 0x5589376b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:02.499 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.479+0000 7f0266419740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f025c1ad130 con 0x5589376912d0 2024-03-22T01:46:02.499 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.479+0000 7f0266419740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f025c1af9b0 con 0x558937691bf0 2024-03-22T01:46:02.499 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.479+0000 7f0266419740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f025c1ad7a0 con 0x55893769a9c0 2024-03-22T01:46:02.499 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.479+0000 7f02614f2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5589376912d0 0x5589376a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:02.499 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.479+0000 7f02614f2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5589376912d0 0x5589376a6390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38592/0 (socket says 172.21.15.72:38592) 2024-03-22T01:46:02.500 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.479+0000 7f02614f2640 1 -- 172.21.15.72:0/2807278857 learned_addr learned my addr 172.21.15.72:0/2807278857 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:02.500 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.479+0000 7f0260cf1640 1 --2- 172.21.15.72:0/2807278857 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558937691bf0 0x55893768aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:02.500 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.479+0000 7f0261cf3640 1 --2- 172.21.15.72:0/2807278857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55893769a9c0 0x5589376b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:02.500 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.479+0000 7f02614f2640 1 -- 172.21.15.72:0/2807278857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55893769a9c0 msgr2=0x5589376b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:02.500 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.479+0000 7f02614f2640 1 --2- 172.21.15.72:0/2807278857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55893769a9c0 0x5589376b2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:02.500 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.479+0000 7f02614f2640 1 -- 172.21.15.72:0/2807278857 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558937691bf0 msgr2=0x55893768aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:02.500 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.479+0000 7f02614f2640 1 --2- 172.21.15.72:0/2807278857 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558937691bf0 0x55893768aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:02.500 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.479+0000 7f02614f2640 1 -- 172.21.15.72:0/2807278857 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558937690890 con 0x5589376912d0 2024-03-22T01:46:02.500 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.479+0000 7f0260cf1640 1 --2- 172.21.15.72:0/2807278857 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558937691bf0 0x55893768aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:46:02.500 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.479+0000 7f02614f2640 1 --2- 172.21.15.72:0/2807278857 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5589376912d0 0x5589376a6390 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f024c005c20 tx=0x7f024c005bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae3faadbf2d8b785 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:02.500 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.479+0000 7f0246ffd640 1 -- 172.21.15.72:0/2807278857 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f024c014070 con 0x5589376912d0 2024-03-22T01:46:02.500 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.479+0000 7f0246ffd640 1 -- 172.21.15.72:0/2807278857 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f024c09bd50 con 0x5589376912d0 2024-03-22T01:46:02.500 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.479+0000 7f0246ffd640 1 -- 172.21.15.72:0/2807278857 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f024c0cf040 con 0x5589376912d0 2024-03-22T01:46:02.500 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.480+0000 7f0266419740 1 -- 172.21.15.72:0/2807278857 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5589376912d0 msgr2=0x5589376a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:02.501 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.480+0000 7f0266419740 1 --2- 172.21.15.72:0/2807278857 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5589376912d0 0x5589376a6390 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f024c005c20 tx=0x7f024c005bb0 comp rx=0 tx=0).stop 2024-03-22T01:46:02.501 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.480+0000 7f0266419740 1 -- 172.21.15.72:0/2807278857 shutdown_connections 2024-03-22T01:46:02.501 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.480+0000 7f0266419740 1 --2- 172.21.15.72:0/2807278857 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558937691bf0 0x55893768aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:02.501 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.480+0000 7f0266419740 1 --2- 172.21.15.72:0/2807278857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55893769a9c0 0x5589376b2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:02.501 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.480+0000 7f0266419740 1 --2- 172.21.15.72:0/2807278857 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5589376912d0 0x5589376a6390 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:02.501 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.480+0000 7f0266419740 1 -- 172.21.15.72:0/2807278857 >> 172.21.15.72:0/2807278857 conn(0x55893768bbf0 msgr2=0x55893768a3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:02.501 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.480+0000 7f0266419740 1 -- 172.21.15.72:0/2807278857 shutdown_connections 2024-03-22T01:46:02.501 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.480+0000 7f0266419740 1 -- 172.21.15.72:0/2807278857 wait complete. 2024-03-22T01:46:02.501 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.480+0000 7f0266419740 1 -- start start 2024-03-22T01:46:02.501 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.480+0000 7f0266419740 1 Processor -- start 2024-03-22T01:46:02.501 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.481+0000 7f0266419740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5589376912d0 0x5589376a2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:02.501 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.481+0000 7f0266419740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558937691bf0 0x55893769df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:02.502 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.481+0000 7f0266419740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55893769a9c0 0x55893769e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:02.502 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.481+0000 7f0266419740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f025c1ade70 con 0x5589376912d0 2024-03-22T01:46:02.502 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.481+0000 7f0266419740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f025c1af2e0 con 0x55893769a9c0 2024-03-22T01:46:02.502 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.481+0000 7f0266419740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f025c1ae540 con 0x558937691bf0 2024-03-22T01:46:02.502 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.481+0000 7f0266419740 10 client.0 _ll_register_callbacks cb 0x7f0254012510 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:46:02.502 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.481+0000 7f02614f2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5589376912d0 0x5589376a2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:02.502 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.481+0000 7f02614f2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5589376912d0 0x5589376a2a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38608/0 (socket says 172.21.15.72:38608) 2024-03-22T01:46:02.502 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.481+0000 7f02614f2640 1 -- 172.21.15.72:0/2078707575 learned_addr learned my addr 172.21.15.72:0/2078707575 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:02.502 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.481+0000 7f0261cf3640 1 --2- 172.21.15.72:0/2078707575 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55893769a9c0 0x55893769e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:02.502 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.481+0000 7f0260cf1640 1 --2- 172.21.15.72:0/2078707575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558937691bf0 0x55893769df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:02.502 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.481+0000 7f02614f2640 1 -- 172.21.15.72:0/2078707575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558937691bf0 msgr2=0x55893769df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:02.502 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.481+0000 7f02614f2640 1 --2- 172.21.15.72:0/2078707575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558937691bf0 0x55893769df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:02.503 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.481+0000 7f02614f2640 1 -- 172.21.15.72:0/2078707575 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55893769a9c0 msgr2=0x55893769e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:02.503 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.481+0000 7f02614f2640 1 --2- 172.21.15.72:0/2078707575 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55893769a9c0 0x55893769e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:02.503 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.481+0000 7f02614f2640 1 -- 172.21.15.72:0/2078707575 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55893769ea50 con 0x5589376912d0 2024-03-22T01:46:02.503 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.481+0000 7f0260cf1640 1 --2- 172.21.15.72:0/2078707575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558937691bf0 0x55893769df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:46:02.503 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.481+0000 7f02614f2640 1 --2- 172.21.15.72:0/2078707575 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5589376912d0 0x5589376a2a40 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f024c09b5e0 tx=0x7f024c09f7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40a7812fe787ef25 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:02.503 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.481+0000 7f0244ff9640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:46:02.503 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.481+0000 7f0244ff9640 1 -- 172.21.15.72:0/2078707575 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f024c014040 con 0x5589376912d0 2024-03-22T01:46:02.503 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.481+0000 7f0244ff9640 1 -- 172.21.15.72:0/2078707575 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f024c0ccc30 con 0x5589376912d0 2024-03-22T01:46:02.503 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.481+0000 7f0266419740 1 -- 172.21.15.72:0/2078707575 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f024c09bd50 con 0x5589376912d0 2024-03-22T01:46:02.503 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.481+0000 7f0244ff9640 1 -- 172.21.15.72:0/2078707575 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7f024c0cf040 con 0x5589376912d0 2024-03-22T01:46:02.503 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.481+0000 7f0266419740 10 client.16197 fetch_fsmap learned FSMap version 103 2024-03-22T01:46:02.504 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.481+0000 7f0266419740 1 -- 172.21.15.72:0/2078707575 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=4+}) -- 0x5589377108c0 con 0x5589376912d0 2024-03-22T01:46:02.504 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.482+0000 7f0244ff9640 1 -- 172.21.15.72:0/2078707575 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f024c09bd50 con 0x5589376912d0 2024-03-22T01:46:02.504 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.482+0000 7f0244ff9640 1 -- 172.21.15.72:0/2078707575 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 103) ==== 8330+0+0 (secure 0 0 0) 0x7f024c0bec00 con 0x5589376912d0 2024-03-22T01:46:02.504 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.482+0000 7f0266419740 10 client.16197 fetch_fsmap finished waiting for FSMap version 103 2024-03-22T01:46:02.504 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.482+0000 7f0266419740 10 client.16197 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T01:46:02.504 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.482+0000 7f0266419740 20 client.16197 populate_metadata read hostname 'smithi072' 2024-03-22T01:46:02.504 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.482+0000 7f0266419740 1 --2- 172.21.15.72:0/2078707575 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x5589376aa830 0x558937713050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:02.504 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.482+0000 7f0266419740 4 client.16197 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T01:46:02.504 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.482+0000 7f0266419740 1 -- 172.21.15.72:0/2078707575 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5589375fc8d0 con 0x5589376aa830 2024-03-22T01:46:02.504 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.482+0000 7f0260cf1640 1 --2- 172.21.15.72:0/2078707575 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x5589376aa830 0x558937713050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:02.504 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.482+0000 7f0244ff9640 10 client.16197 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:46:02.504 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.482+0000 7f0260cf1640 1 --2- 172.21.15.72:0/2078707575 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x5589376aa830 0x558937713050 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc07381c8bd0a264 server_cookie=6ff6a94215eb6e2b in_seq=0 out_seq=0 2024-03-22T01:46:02.506 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.492+0000 7f0244ff9640 1 -- 172.21.15.72:0/2078707575 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5589375fc8d0 con 0x5589376aa830 2024-03-22T01:46:02.506 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.492+0000 7f0244ff9640 10 client.16197 handle_command_reply: tid=0 2024-03-22T01:46:02.506 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.492+0000 7f0266419740 1 client.16197 shutdown 2024-03-22T01:46:02.506 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.492+0000 7f0266419740 1 -- 172.21.15.72:0/2078707575 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5589376912d0 msgr2=0x5589376a2a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:02.507 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.492+0000 7f0266419740 1 --2- 172.21.15.72:0/2078707575 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5589376912d0 0x5589376a2a40 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f024c09b5e0 tx=0x7f024c09f7a0 comp rx=0 tx=0).stop 2024-03-22T01:46:02.507 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.492+0000 7f0266419740 1 -- 172.21.15.72:0/2078707575 shutdown_connections 2024-03-22T01:46:02.507 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.492+0000 7f0266419740 1 --2- 172.21.15.72:0/2078707575 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x5589376aa830 0x558937713050 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:02.507 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.492+0000 7f0266419740 1 --2- 172.21.15.72:0/2078707575 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55893769a9c0 0x55893769e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:02.507 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.492+0000 7f0244ff9640 0 client.16197 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:46:02.507 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.492+0000 7f0266419740 1 --2- 172.21.15.72:0/2078707575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558937691bf0 0x55893769df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:02.507 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.492+0000 7f0266419740 1 --2- 172.21.15.72:0/2078707575 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5589376912d0 0x5589376a2a40 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:02.507 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.492+0000 7f0266419740 1 -- 172.21.15.72:0/2078707575 >> 172.21.15.72:0/2078707575 conn(0x55893768bbf0 msgr2=0x55893768ac50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:02.508 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.492+0000 7f0266419740 1 -- 172.21.15.72:0/2078707575 shutdown_connections 2024-03-22T01:46:02.508 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.492+0000 7f0266419740 1 -- 172.21.15.72:0/2078707575 wait complete. 2024-03-22T01:46:02.508 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.492+0000 7f0266419740 20 client.16197 trim_cache size 0 max 16384 2024-03-22T01:46:02.508 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout:{ 2024-03-22T01:46:02.508 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout: "message": "", 2024-03-22T01:46:02.508 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout: "return_code": 0 2024-03-22T01:46:02.509 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout:} 2024-03-22T01:46:02.510 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.493+0000 7f0263756640 1 -- 172.21.15.72:0/1837272877 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f02480a3080 msgr2=0x7f02480a5540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:02.511 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.493+0000 7f0263756640 1 --2- 172.21.15.72:0/1837272877 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f02480a3080 0x7f02480a5540 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0258009230 tx=0x7f0258005fd0 comp rx=0 tx=0).stop 2024-03-22T01:46:02.511 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.493+0000 7f0263756640 1 -- 172.21.15.72:0/1837272877 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f025c101db0 msgr2=0x7f025c198550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:02.511 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.493+0000 7f0263756640 1 --2- 172.21.15.72:0/1837272877 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f025c101db0 0x7f025c198550 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f024c00d770 tx=0x7f024c00dc30 comp rx=0 tx=0).stop 2024-03-22T01:46:02.511 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.493+0000 7f0263756640 1 -- 172.21.15.72:0/1837272877 shutdown_connections 2024-03-22T01:46:02.511 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.493+0000 7f0263756640 1 --2- 172.21.15.72:0/1837272877 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f02480a3080 0x7f02480a5540 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:02.511 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.493+0000 7f0263756640 1 --2- 172.21.15.72:0/1837272877 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f025c106e80 0x7f025c198a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:02.512 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.493+0000 7f0263756640 1 --2- 172.21.15.72:0/1837272877 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f025c19ce10 0x7f025c19d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:02.512 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.493+0000 7f0263756640 1 --2- 172.21.15.72:0/1837272877 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f025c101db0 0x7f025c198550 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:02.512 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.493+0000 7f0263756640 1 -- 172.21.15.72:0/1837272877 >> 172.21.15.72:0/1837272877 conn(0x7f025c0fd610 msgr2=0x7f025c0ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:02.512 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.493+0000 7f0263756640 1 -- 172.21.15.72:0/1837272877 shutdown_connections 2024-03-22T01:46:02.512 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:02.493+0000 7f0263756640 1 -- 172.21.15.72:0/1837272877 wait complete. 2024-03-22T01:46:02.977 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-22T01:46:02.977 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T01:46:02.977 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-22T01:46:03.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.095+0000 7f488d4ec640 1 -- 172.21.15.72:0/991375014 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48881036d0 msgr2=0x7f488810bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:03.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.095+0000 7f488d4ec640 1 --2- 172.21.15.72:0/991375014 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48881036d0 0x7f488810bbb0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f4870009900 tx=0x7f487002bcd0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.095+0000 7f488d4ec640 1 -- 172.21.15.72:0/991375014 shutdown_connections 2024-03-22T01:46:03.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.095+0000 7f488d4ec640 1 --2- 172.21.15.72:0/991375014 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48881036d0 0x7f488810bbb0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.095+0000 7f488d4ec640 1 --2- 172.21.15.72:0/991375014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f488810c180 0x7f488810e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.095+0000 7f488d4ec640 1 --2- 172.21.15.72:0/991375014 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4888102db0 0x7f4888103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.095+0000 7f488d4ec640 1 -- 172.21.15.72:0/991375014 >> 172.21.15.72:0/991375014 conn(0x7f4888077c40 msgr2=0x7f4888078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:03.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.095+0000 7f488d4ec640 1 -- 172.21.15.72:0/991375014 shutdown_connections 2024-03-22T01:46:03.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.095+0000 7f488d4ec640 1 -- 172.21.15.72:0/991375014 wait complete. 2024-03-22T01:46:03.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.095+0000 7f488d4ec640 1 Processor -- start 2024-03-22T01:46:03.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.095+0000 7f488d4ec640 1 -- start start 2024-03-22T01:46:03.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.096+0000 7f488d4ec640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4888102db0 0x7f48881984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:03.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.096+0000 7f488d4ec640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48881036d0 0x7f4888198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:03.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.096+0000 7f488d4ec640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f488810c180 0x7f488819cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:03.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.096+0000 7f488d4ec640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f4888111f60 con 0x7f4888102db0 2024-03-22T01:46:03.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.096+0000 7f488d4ec640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4888111de0 con 0x7f48881036d0 2024-03-22T01:46:03.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.096+0000 7f488d4ec640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f48881120e0 con 0x7f488810c180 2024-03-22T01:46:03.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.096+0000 7f4886ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4888102db0 0x7f48881984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:03.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.096+0000 7f4886ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4888102db0 0x7f48881984e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38636/0 (socket says 172.21.15.72:38636) 2024-03-22T01:46:03.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.096+0000 7f4886ffd640 1 -- 172.21.15.72:0/2772618429 learned_addr learned my addr 172.21.15.72:0/2772618429 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:03.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.096+0000 7f487ffff640 1 --2- 172.21.15.72:0/2772618429 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48881036d0 0x7f4888198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:03.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.096+0000 7f48877fe640 1 --2- 172.21.15.72:0/2772618429 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f488810c180 0x7f488819cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:03.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.096+0000 7f4886ffd640 1 -- 172.21.15.72:0/2772618429 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f488810c180 msgr2=0x7f488819cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:03.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.096+0000 7f4886ffd640 1 --2- 172.21.15.72:0/2772618429 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f488810c180 0x7f488819cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.096+0000 7f4886ffd640 1 -- 172.21.15.72:0/2772618429 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48881036d0 msgr2=0x7f4888198a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:03.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.096+0000 7f4886ffd640 1 --2- 172.21.15.72:0/2772618429 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48881036d0 0x7f4888198a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.096+0000 7f4886ffd640 1 -- 172.21.15.72:0/2772618429 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f488819d4d0 con 0x7f4888102db0 2024-03-22T01:46:03.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.096+0000 7f48877fe640 1 --2- 172.21.15.72:0/2772618429 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f488810c180 0x7f488819cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:46:03.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.096+0000 7f487ffff640 1 --2- 172.21.15.72:0/2772618429 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48881036d0 0x7f4888198a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:46:03.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.096+0000 7f4886ffd640 1 --2- 172.21.15.72:0/2772618429 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4888102db0 0x7f48881984e0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f487800ea60 tx=0x7f487800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:03.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.096+0000 7f4884ff9640 1 -- 172.21.15.72:0/2772618429 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f487800cde0 con 0x7f4888102db0 2024-03-22T01:46:03.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.096+0000 7f488d4ec640 1 -- 172.21.15.72:0/2772618429 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f488819d7c0 con 0x7f4888102db0 2024-03-22T01:46:03.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.097+0000 7f488d4ec640 1 -- 172.21.15.72:0/2772618429 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48881a5050 con 0x7f4888102db0 2024-03-22T01:46:03.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.097+0000 7f4884ff9640 1 -- 172.21.15.72:0/2772618429 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48780035f0 con 0x7f4888102db0 2024-03-22T01:46:03.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.097+0000 7f4884ff9640 1 -- 172.21.15.72:0/2772618429 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4878018690 con 0x7f4888102db0 2024-03-22T01:46:03.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.098+0000 7f4884ff9640 1 -- 172.21.15.72:0/2772618429 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 25) ==== 137985+0+0 (secure 0 0 0) 0x7f4878003a80 con 0x7f4888102db0 2024-03-22T01:46:03.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.099+0000 7f48901a8740 1 Processor -- start 2024-03-22T01:46:03.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.099+0000 7f48901a8740 1 -- start start 2024-03-22T01:46:03.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.099+0000 7f4884ff9640 1 --2- 172.21.15.72:0/2772618429 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f48740a3090 0x7f48740a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:03.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.099+0000 7f48877fe640 1 --2- 172.21.15.72:0/2772618429 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f48740a3090 0x7f48740a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:03.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.099+0000 7f4884ff9640 1 -- 172.21.15.72:0/2772618429 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f48780cff60 con 0x7f4888102db0 2024-03-22T01:46:03.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.099+0000 7f48901a8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7ac3eeba0 0x55a7ac3eef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:03.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.099+0000 7f48901a8740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a7ac3ef4c0 0x55a7ac3f7b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:03.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.099+0000 7f48901a8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a7ac3f80a0 0x55a7ac3fa490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:03.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.099+0000 7f48901a8740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55a7ac352570 con 0x55a7ac3ef4c0 2024-03-22T01:46:03.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.099+0000 7f48901a8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55a7ac2fae10 con 0x55a7ac3f80a0 2024-03-22T01:46:03.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.099+0000 7f48901a8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a7ac34e520 con 0x55a7ac3eeba0 2024-03-22T01:46:03.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.100+0000 7f487ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a7ac3ef4c0 0x55a7ac3f7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:03.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.100+0000 7f487ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a7ac3ef4c0 0x55a7ac3f7b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38640/0 (socket says 172.21.15.72:38640) 2024-03-22T01:46:03.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.100+0000 7f487ffff640 1 -- 172.21.15.72:0/3060502751 learned_addr learned my addr 172.21.15.72:0/3060502751 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:03.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.100+0000 7f48877fe640 1 --2- 172.21.15.72:0/3060502751 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a7ac3f80a0 0x55a7ac3fa490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:03.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.100+0000 7f4886ffd640 1 --2- 172.21.15.72:0/3060502751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7ac3eeba0 0x55a7ac3eef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:03.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.100+0000 7f48877fe640 1 --2- 172.21.15.72:0/2772618429 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f48740a3090 0x7f48740a5550 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f488000ae70 tx=0x7f488000a410 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:03.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.100+0000 7f487ffff640 1 -- 172.21.15.72:0/3060502751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7ac3eeba0 msgr2=0x55a7ac3eef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:03.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.100+0000 7f487ffff640 1 --2- 172.21.15.72:0/3060502751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7ac3eeba0 0x55a7ac3eef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.100+0000 7f487ffff640 1 -- 172.21.15.72:0/3060502751 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a7ac3f80a0 msgr2=0x55a7ac3fa490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:03.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.100+0000 7f487ffff640 1 --2- 172.21.15.72:0/3060502751 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a7ac3f80a0 0x55a7ac3fa490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.100+0000 7f487ffff640 1 -- 172.21.15.72:0/3060502751 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7ac12e7e0 con 0x55a7ac3ef4c0 2024-03-22T01:46:03.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.100+0000 7f487ffff640 1 --2- 172.21.15.72:0/3060502751 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a7ac3ef4c0 0x55a7ac3f7b60 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f4870009230 tx=0x7f487002c6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3402dcf47fe5b4d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:03.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.100+0000 7f487d7fa640 1 -- 172.21.15.72:0/3060502751 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4870043070 con 0x55a7ac3ef4c0 2024-03-22T01:46:03.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.100+0000 7f487d7fa640 1 -- 172.21.15.72:0/3060502751 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4870004af0 con 0x55a7ac3ef4c0 2024-03-22T01:46:03.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.100+0000 7f487d7fa640 1 -- 172.21.15.72:0/3060502751 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4870005120 con 0x55a7ac3ef4c0 2024-03-22T01:46:03.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.101+0000 7f48901a8740 1 -- 172.21.15.72:0/3060502751 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a7ac3ef4c0 msgr2=0x55a7ac3f7b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:03.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.101+0000 7f48901a8740 1 --2- 172.21.15.72:0/3060502751 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a7ac3ef4c0 0x55a7ac3f7b60 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f4870009230 tx=0x7f487002c6f0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.101+0000 7f48901a8740 1 -- 172.21.15.72:0/3060502751 shutdown_connections 2024-03-22T01:46:03.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.101+0000 7f48901a8740 1 --2- 172.21.15.72:0/3060502751 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a7ac3f80a0 0x55a7ac3fa490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.101+0000 7f48901a8740 1 --2- 172.21.15.72:0/3060502751 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a7ac3ef4c0 0x55a7ac3f7b60 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.101+0000 7f48901a8740 1 --2- 172.21.15.72:0/3060502751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7ac3eeba0 0x55a7ac3eef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.101+0000 7f48901a8740 1 -- 172.21.15.72:0/3060502751 >> 172.21.15.72:0/3060502751 conn(0x55a7ac3e92d0 msgr2=0x55a7ac3e96b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:03.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.101+0000 7f48901a8740 1 -- 172.21.15.72:0/3060502751 shutdown_connections 2024-03-22T01:46:03.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.101+0000 7f48901a8740 1 -- 172.21.15.72:0/3060502751 wait complete. 2024-03-22T01:46:03.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.101+0000 7f48901a8740 1 -- start start 2024-03-22T01:46:03.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.101+0000 7f48901a8740 1 Processor -- start 2024-03-22T01:46:03.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.102+0000 7f48901a8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a7ac3eeba0 0x55a7ac3f79d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:03.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.102+0000 7f48901a8740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a7ac3ef4c0 0x55a7ac3f4d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:03.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.102+0000 7f48901a8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7ac3f80a0 0x55a7ac3f52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:03.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.102+0000 7f48901a8740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55a7ac3fae30 con 0x55a7ac3ef4c0 2024-03-22T01:46:03.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.102+0000 7f48901a8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55a7ac30e110 con 0x55a7ac3eeba0 2024-03-22T01:46:03.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.102+0000 7f48901a8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a7ac3fafb0 con 0x55a7ac3f80a0 2024-03-22T01:46:03.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.102+0000 7f48901a8740 10 client.0 _ll_register_callbacks cb 0x55a7ac34abe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:46:03.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.102+0000 7f487ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a7ac3ef4c0 0x55a7ac3f4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:03.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.102+0000 7f487ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a7ac3ef4c0 0x55a7ac3f4d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38646/0 (socket says 172.21.15.72:38646) 2024-03-22T01:46:03.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.102+0000 7f4886ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a7ac3eeba0 0x55a7ac3f79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:03.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.102+0000 7f487ffff640 1 -- 172.21.15.72:0/3681879225 learned_addr learned my addr 172.21.15.72:0/3681879225 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:03.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.102+0000 7f48877fe640 1 --2- 172.21.15.72:0/3681879225 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7ac3f80a0 0x55a7ac3f52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:03.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.102+0000 7f487ffff640 1 -- 172.21.15.72:0/3681879225 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7ac3f80a0 msgr2=0x55a7ac3f52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:03.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.102+0000 7f487ffff640 1 --2- 172.21.15.72:0/3681879225 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7ac3f80a0 0x55a7ac3f52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.102+0000 7f487ffff640 1 -- 172.21.15.72:0/3681879225 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a7ac3eeba0 msgr2=0x55a7ac3f79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:03.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.102+0000 7f487ffff640 1 --2- 172.21.15.72:0/3681879225 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a7ac3eeba0 0x55a7ac3f79d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.102+0000 7f487ffff640 1 -- 172.21.15.72:0/3681879225 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a7ac18ef70 con 0x55a7ac3ef4c0 2024-03-22T01:46:03.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.102+0000 7f4886ffd640 1 --2- 172.21.15.72:0/3681879225 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a7ac3eeba0 0x55a7ac3f79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:46:03.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.102+0000 7f48877fe640 1 --2- 172.21.15.72:0/3681879225 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7ac3f80a0 0x55a7ac3f52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:46:03.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.102+0000 7f487ffff640 1 --2- 172.21.15.72:0/3681879225 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a7ac3ef4c0 0x55a7ac3f4d60 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f4870036040 tx=0x7f487002cd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=658a18270448a0ac server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:03.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.102+0000 7f486e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:46:03.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.102+0000 7f486e7fc640 1 -- 172.21.15.72:0/3681879225 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f487004f070 con 0x55a7ac3ef4c0 2024-03-22T01:46:03.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.102+0000 7f486e7fc640 1 -- 172.21.15.72:0/3681879225 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f4870048450 con 0x55a7ac3ef4c0 2024-03-22T01:46:03.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.102+0000 7f48901a8740 1 -- 172.21.15.72:0/3681879225 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4870004af0 con 0x55a7ac3ef4c0 2024-03-22T01:46:03.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.103+0000 7f486e7fc640 1 -- 172.21.15.72:0/3681879225 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7f4870058050 con 0x55a7ac3ef4c0 2024-03-22T01:46:03.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.103+0000 7f48901a8740 10 client.16209 fetch_fsmap learned FSMap version 103 2024-03-22T01:46:03.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.103+0000 7f48901a8740 1 -- 172.21.15.72:0/3681879225 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=4+}) -- 0x55a7ac026170 con 0x55a7ac3ef4c0 2024-03-22T01:46:03.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.103+0000 7f486e7fc640 1 -- 172.21.15.72:0/3681879225 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4870004af0 con 0x55a7ac3ef4c0 2024-03-22T01:46:03.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.103+0000 7f486e7fc640 1 -- 172.21.15.72:0/3681879225 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 103) ==== 8330+0+0 (secure 0 0 0) 0x7f4870052020 con 0x55a7ac3ef4c0 2024-03-22T01:46:03.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.103+0000 7f48901a8740 10 client.16209 fetch_fsmap finished waiting for FSMap version 103 2024-03-22T01:46:03.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.103+0000 7f48901a8740 10 client.16209 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T01:46:03.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.103+0000 7f48901a8740 20 client.16209 populate_metadata read hostname 'smithi072' 2024-03-22T01:46:03.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.103+0000 7f48901a8740 1 --2- 172.21.15.72:0/3681879225 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55a7ac407ea0 0x55a7ac40a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:03.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.103+0000 7f48901a8740 4 client.16209 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T01:46:03.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.103+0000 7f48901a8740 1 -- 172.21.15.72:0/3681879225 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a7ac35fab0 con 0x55a7ac407ea0 2024-03-22T01:46:03.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.103+0000 7f4886ffd640 1 --2- 172.21.15.72:0/3681879225 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55a7ac407ea0 0x55a7ac40a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:03.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.104+0000 7f486e7fc640 10 client.16209 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:46:03.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.104+0000 7f4886ffd640 1 --2- 172.21.15.72:0/3681879225 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55a7ac407ea0 0x55a7ac40a290 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13f0e7858a1c9332 server_cookie=a660a9d68edc935b in_seq=0 out_seq=0 2024-03-22T01:46:03.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.105+0000 7f486e7fc640 1 -- 172.21.15.72:0/3681879225 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a7ac35fab0 con 0x55a7ac407ea0 2024-03-22T01:46:03.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.105+0000 7f486e7fc640 10 client.16209 handle_command_reply: tid=0 2024-03-22T01:46:03.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.105+0000 7f48901a8740 1 client.16209 shutdown 2024-03-22T01:46:03.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.105+0000 7f48901a8740 1 -- 172.21.15.72:0/3681879225 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a7ac3ef4c0 msgr2=0x55a7ac3f4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:03.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.105+0000 7f48901a8740 1 --2- 172.21.15.72:0/3681879225 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a7ac3ef4c0 0x55a7ac3f4d60 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f4870036040 tx=0x7f487002cd40 comp rx=0 tx=0).stop 2024-03-22T01:46:03.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.105+0000 7f48901a8740 1 -- 172.21.15.72:0/3681879225 shutdown_connections 2024-03-22T01:46:03.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.105+0000 7f48901a8740 1 --2- 172.21.15.72:0/3681879225 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55a7ac407ea0 0x55a7ac40a290 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.105+0000 7f48901a8740 1 --2- 172.21.15.72:0/3681879225 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7ac3f80a0 0x55a7ac3f52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.105+0000 7f48901a8740 1 --2- 172.21.15.72:0/3681879225 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a7ac3ef4c0 0x55a7ac3f4d60 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.105+0000 7f486e7fc640 0 client.16209 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:46:03.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.105+0000 7f48901a8740 1 --2- 172.21.15.72:0/3681879225 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a7ac3eeba0 0x55a7ac3f79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.105+0000 7f48901a8740 1 -- 172.21.15.72:0/3681879225 >> 172.21.15.72:0/3681879225 conn(0x55a7ac3e92d0 msgr2=0x55a7ac3eb020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:03.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.105+0000 7f48901a8740 1 -- 172.21.15.72:0/3681879225 shutdown_connections 2024-03-22T01:46:03.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.105+0000 7f48901a8740 1 -- 172.21.15.72:0/3681879225 wait complete. 2024-03-22T01:46:03.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.105+0000 7f48901a8740 20 client.16209 trim_cache size 0 max 16384 2024-03-22T01:46:03.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.122+0000 7f48901a8740 1 Processor -- start 2024-03-22T01:46:03.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.122+0000 7f48901a8740 1 -- start start 2024-03-22T01:46:03.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.123+0000 7f48901a8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7ac3e92d0 0x55a7ac40f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:03.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.123+0000 7f48901a8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a7ac3eeba0 0x55a7ac400220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:03.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.123+0000 7f48901a8740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a7ac3ef4c0 0x55a7ac3e8270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:03.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.123+0000 7f48901a8740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f48881ad0d0 con 0x55a7ac3ef4c0 2024-03-22T01:46:03.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.123+0000 7f48901a8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f48881af950 con 0x55a7ac3eeba0 2024-03-22T01:46:03.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.123+0000 7f48901a8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f48881ad740 con 0x55a7ac3e92d0 2024-03-22T01:46:03.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.123+0000 7f48877fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a7ac3ef4c0 0x55a7ac3e8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:03.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.123+0000 7f48877fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a7ac3ef4c0 0x55a7ac3e8270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38662/0 (socket says 172.21.15.72:38662) 2024-03-22T01:46:03.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.123+0000 7f48877fe640 1 -- 172.21.15.72:0/1718173715 learned_addr learned my addr 172.21.15.72:0/1718173715 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:03.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.123+0000 7f487ffff640 1 --2- 172.21.15.72:0/1718173715 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a7ac3eeba0 0x55a7ac400220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:03.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.123+0000 7f4886ffd640 1 --2- 172.21.15.72:0/1718173715 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7ac3e92d0 0x55a7ac40f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:03.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.123+0000 7f48877fe640 1 -- 172.21.15.72:0/1718173715 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7ac3e92d0 msgr2=0x55a7ac40f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:03.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.123+0000 7f48877fe640 1 --2- 172.21.15.72:0/1718173715 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7ac3e92d0 0x55a7ac40f8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.123+0000 7f48877fe640 1 -- 172.21.15.72:0/1718173715 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a7ac3eeba0 msgr2=0x55a7ac400220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:03.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.123+0000 7f48877fe640 1 --2- 172.21.15.72:0/1718173715 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a7ac3eeba0 0x55a7ac400220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.123+0000 7f48877fe640 1 -- 172.21.15.72:0/1718173715 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7ac3fa490 con 0x55a7ac3ef4c0 2024-03-22T01:46:03.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.123+0000 7f48877fe640 1 --2- 172.21.15.72:0/1718173715 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a7ac3ef4c0 0x55a7ac3e8270 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f488001c5a0 tx=0x7f4880016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d97e007e58c92f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:03.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.123+0000 7f487cff9640 1 -- 172.21.15.72:0/1718173715 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4880010700 con 0x55a7ac3ef4c0 2024-03-22T01:46:03.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.123+0000 7f487cff9640 1 -- 172.21.15.72:0/1718173715 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4880016070 con 0x55a7ac3ef4c0 2024-03-22T01:46:03.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.123+0000 7f487cff9640 1 -- 172.21.15.72:0/1718173715 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f488001be00 con 0x55a7ac3ef4c0 2024-03-22T01:46:03.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.124+0000 7f48901a8740 1 -- 172.21.15.72:0/1718173715 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a7ac3ef4c0 msgr2=0x55a7ac3e8270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:03.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.124+0000 7f48901a8740 1 --2- 172.21.15.72:0/1718173715 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a7ac3ef4c0 0x55a7ac3e8270 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f488001c5a0 tx=0x7f4880016000 comp rx=0 tx=0).stop 2024-03-22T01:46:03.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.124+0000 7f48901a8740 1 -- 172.21.15.72:0/1718173715 shutdown_connections 2024-03-22T01:46:03.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.124+0000 7f48901a8740 1 --2- 172.21.15.72:0/1718173715 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a7ac3eeba0 0x55a7ac400220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.124+0000 7f48901a8740 1 --2- 172.21.15.72:0/1718173715 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a7ac3ef4c0 0x55a7ac3e8270 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.124+0000 7f48901a8740 1 --2- 172.21.15.72:0/1718173715 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7ac3e92d0 0x55a7ac40f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.124+0000 7f48901a8740 1 -- 172.21.15.72:0/1718173715 >> 172.21.15.72:0/1718173715 conn(0x55a7ac3f39a0 msgr2=0x55a7ac3e7b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:03.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.124+0000 7f48901a8740 1 -- 172.21.15.72:0/1718173715 shutdown_connections 2024-03-22T01:46:03.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.124+0000 7f48901a8740 1 -- 172.21.15.72:0/1718173715 wait complete. 2024-03-22T01:46:03.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.124+0000 7f48901a8740 1 -- start start 2024-03-22T01:46:03.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.125+0000 7f48901a8740 1 Processor -- start 2024-03-22T01:46:03.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.125+0000 7f48901a8740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a7ac3e92d0 0x55a7ac3fc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:03.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.125+0000 7f48901a8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a7ac3eeba0 0x55a7ac3fc920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:03.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.125+0000 7f48901a8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7ac3ef4c0 0x55a7ac466ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:03.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.125+0000 7f48901a8740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f48881ade10 con 0x55a7ac3e92d0 2024-03-22T01:46:03.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.125+0000 7f48901a8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f48881af280 con 0x55a7ac3eeba0 2024-03-22T01:46:03.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.125+0000 7f48901a8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f48881ae4e0 con 0x55a7ac3ef4c0 2024-03-22T01:46:03.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.125+0000 7f48901a8740 10 client.0 _ll_register_callbacks cb 0x7f4880014070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:46:03.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.125+0000 7f4886ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a7ac3e92d0 0x55a7ac3fc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:03.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.125+0000 7f4886ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a7ac3e92d0 0x55a7ac3fc3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38668/0 (socket says 172.21.15.72:38668) 2024-03-22T01:46:03.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.125+0000 7f4886ffd640 1 -- 172.21.15.72:0/3718551740 learned_addr learned my addr 172.21.15.72:0/3718551740 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:03.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.125+0000 7f487ffff640 1 --2- 172.21.15.72:0/3718551740 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a7ac3eeba0 0x55a7ac3fc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:03.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.125+0000 7f48877fe640 1 --2- 172.21.15.72:0/3718551740 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7ac3ef4c0 0x55a7ac466ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:03.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.125+0000 7f4886ffd640 1 -- 172.21.15.72:0/3718551740 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7ac3ef4c0 msgr2=0x55a7ac466ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:03.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.125+0000 7f4886ffd640 1 --2- 172.21.15.72:0/3718551740 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7ac3ef4c0 0x55a7ac466ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.125+0000 7f4886ffd640 1 -- 172.21.15.72:0/3718551740 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a7ac3eeba0 msgr2=0x55a7ac3fc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:03.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.125+0000 7f4886ffd640 1 --2- 172.21.15.72:0/3718551740 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a7ac3eeba0 0x55a7ac3fc920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.125+0000 7f4886ffd640 1 -- 172.21.15.72:0/3718551740 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a7ac4674b0 con 0x55a7ac3e92d0 2024-03-22T01:46:03.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.125+0000 7f487ffff640 1 --2- 172.21.15.72:0/3718551740 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a7ac3eeba0 0x55a7ac3fc920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:46:03.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.125+0000 7f4886ffd640 1 --2- 172.21.15.72:0/3718551740 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a7ac3e92d0 0x55a7ac3fc3e0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f487809bf30 tx=0x7f4878000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12dec0b4f100a854 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:03.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.125+0000 7f486e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:46:03.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.125+0000 7f486e7fc640 1 -- 172.21.15.72:0/3718551740 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48780cd580 con 0x55a7ac3e92d0 2024-03-22T01:46:03.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.126+0000 7f486e7fc640 1 -- 172.21.15.72:0/3718551740 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f4878098610 con 0x55a7ac3e92d0 2024-03-22T01:46:03.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.126+0000 7f48901a8740 1 -- 172.21.15.72:0/3718551740 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4880016070 con 0x55a7ac3e92d0 2024-03-22T01:46:03.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.126+0000 7f486e7fc640 1 -- 172.21.15.72:0/3718551740 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7f487809b3f0 con 0x55a7ac3e92d0 2024-03-22T01:46:03.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.126+0000 7f48901a8740 10 client.16215 fetch_fsmap learned FSMap version 103 2024-03-22T01:46:03.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.126+0000 7f48901a8740 1 -- 172.21.15.72:0/3718551740 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=4+}) -- 0x55a7ac46e8e0 con 0x55a7ac3e92d0 2024-03-22T01:46:03.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.126+0000 7f486e7fc640 1 -- 172.21.15.72:0/3718551740 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4880016070 con 0x55a7ac3e92d0 2024-03-22T01:46:03.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.126+0000 7f486e7fc640 1 -- 172.21.15.72:0/3718551740 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 103) ==== 8330+0+0 (secure 0 0 0) 0x7f4878096ca0 con 0x55a7ac3e92d0 2024-03-22T01:46:03.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.126+0000 7f48901a8740 10 client.16215 fetch_fsmap finished waiting for FSMap version 103 2024-03-22T01:46:03.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.126+0000 7f48901a8740 10 client.16215 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T01:46:03.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.126+0000 7f48901a8740 20 client.16215 populate_metadata read hostname 'smithi072' 2024-03-22T01:46:03.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.126+0000 7f48901a8740 1 --2- 172.21.15.72:0/3718551740 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55a7ac3f80a0 0x55a7ac470ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:03.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.126+0000 7f48901a8740 4 client.16215 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-22T01:46:03.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.126+0000 7f48901a8740 1 -- 172.21.15.72:0/3718551740 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55a7ac359e00 con 0x55a7ac3f80a0 2024-03-22T01:46:03.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.126+0000 7f487ffff640 1 --2- 172.21.15.72:0/3718551740 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55a7ac3f80a0 0x55a7ac470ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:03.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.126+0000 7f486e7fc640 10 client.16215 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:46:03.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.126+0000 7f487ffff640 1 --2- 172.21.15.72:0/3718551740 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55a7ac3f80a0 0x55a7ac470ff0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c7abd2baa483b6b6 server_cookie=12f78679a55d60ba in_seq=0 out_seq=0 2024-03-22T01:46:03.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.127+0000 7f486e7fc640 1 -- 172.21.15.72:0/3718551740 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55a7ac359e00 con 0x55a7ac3f80a0 2024-03-22T01:46:03.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.127+0000 7f486e7fc640 10 client.16215 handle_command_reply: tid=0 2024-03-22T01:46:03.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.127+0000 7f48901a8740 1 client.16215 shutdown 2024-03-22T01:46:03.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.127+0000 7f48901a8740 1 -- 172.21.15.72:0/3718551740 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a7ac3e92d0 msgr2=0x55a7ac3fc3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:03.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.127+0000 7f48901a8740 1 --2- 172.21.15.72:0/3718551740 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a7ac3e92d0 0x55a7ac3fc3e0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f487809bf30 tx=0x7f4878000bc0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.128+0000 7f48901a8740 1 -- 172.21.15.72:0/3718551740 shutdown_connections 2024-03-22T01:46:03.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.128+0000 7f48901a8740 1 --2- 172.21.15.72:0/3718551740 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55a7ac3f80a0 0x55a7ac470ff0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.128+0000 7f48901a8740 1 --2- 172.21.15.72:0/3718551740 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a7ac3eeba0 0x55a7ac3fc920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.128+0000 7f48901a8740 1 --2- 172.21.15.72:0/3718551740 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a7ac3ef4c0 0x55a7ac466ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.128+0000 7f486e7fc640 0 client.16215 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:46:03.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.128+0000 7f48901a8740 1 --2- 172.21.15.72:0/3718551740 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55a7ac3e92d0 0x55a7ac3fc3e0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.128+0000 7f48901a8740 1 -- 172.21.15.72:0/3718551740 >> 172.21.15.72:0/3718551740 conn(0x55a7ac3f39a0 msgr2=0x55a7ac3e83f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:03.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.128+0000 7f48901a8740 1 -- 172.21.15.72:0/3718551740 shutdown_connections 2024-03-22T01:46:03.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.128+0000 7f48901a8740 1 -- 172.21.15.72:0/3718551740 wait complete. 2024-03-22T01:46:03.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.128+0000 7f48901a8740 20 client.16215 trim_cache size 0 max 16384 2024-03-22T01:46:03.136 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-22T01:46:03.136 INFO:teuthology.orchestra.run.smithi072.stdout: "return_code": 0, 2024-03-22T01:46:03.136 INFO:teuthology.orchestra.run.smithi072.stdout: "scrub_tag": "ea880a42-3ab2-4a31-9d1f-5eaeeaaff5d6", 2024-03-22T01:46:03.136 INFO:teuthology.orchestra.run.smithi072.stdout: "mode": "asynchronous" 2024-03-22T01:46:03.137 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-22T01:46:03.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.128+0000 7f488d4ec640 1 -- 172.21.15.72:0/2772618429 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f48740a3090 msgr2=0x7f48740a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:03.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.128+0000 7f488d4ec640 1 --2- 172.21.15.72:0/2772618429 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f48740a3090 0x7f48740a5550 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f488000ae70 tx=0x7f488000a410 comp rx=0 tx=0).stop 2024-03-22T01:46:03.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.128+0000 7f488d4ec640 1 -- 172.21.15.72:0/2772618429 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4888102db0 msgr2=0x7f48881984e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:03.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.128+0000 7f488d4ec640 1 --2- 172.21.15.72:0/2772618429 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4888102db0 0x7f48881984e0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f487800ea60 tx=0x7f487800ef20 comp rx=0 tx=0).stop 2024-03-22T01:46:03.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.128+0000 7f488d4ec640 1 -- 172.21.15.72:0/2772618429 shutdown_connections 2024-03-22T01:46:03.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.128+0000 7f488d4ec640 1 --2- 172.21.15.72:0/2772618429 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f48740a3090 0x7f48740a5550 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.128+0000 7f488d4ec640 1 --2- 172.21.15.72:0/2772618429 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48881036d0 0x7f4888198a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.128+0000 7f488d4ec640 1 --2- 172.21.15.72:0/2772618429 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f488810c180 0x7f488819cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.128+0000 7f488d4ec640 1 --2- 172.21.15.72:0/2772618429 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4888102db0 0x7f48881984e0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.128+0000 7f488d4ec640 1 -- 172.21.15.72:0/2772618429 >> 172.21.15.72:0/2772618429 conn(0x7f4888077c40 msgr2=0x7f48880fee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:03.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.128+0000 7f488d4ec640 1 -- 172.21.15.72:0/2772618429 shutdown_connections 2024-03-22T01:46:03.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.128+0000 7f488d4ec640 1 -- 172.21.15.72:0/2772618429 wait complete. 2024-03-22T01:46:03.139 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T01:46:03.139 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T01:46:03.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.247+0000 7f7a9af3a640 1 -- 172.21.15.72:0/2765203295 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7a94100a30 msgr2=0x7f7a94100eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:03.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.247+0000 7f7a9af3a640 1 --2- 172.21.15.72:0/2765203295 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7a94100a30 0x7f7a94100eb0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f7a80009930 tx=0x7f7a8002bc30 comp rx=0 tx=0).stop 2024-03-22T01:46:03.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.247+0000 7f7a9af3a640 1 -- 172.21.15.72:0/2765203295 shutdown_connections 2024-03-22T01:46:03.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.247+0000 7f7a9af3a640 1 --2- 172.21.15.72:0/2765203295 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7a941013f0 0x7f7a9410d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.247+0000 7f7a9af3a640 1 --2- 172.21.15.72:0/2765203295 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7a94100a30 0x7f7a94100eb0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.247+0000 7f7a9af3a640 1 --2- 172.21.15.72:0/2765203295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a94106a40 0x7f7a94106e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.247+0000 7f7a9af3a640 1 -- 172.21.15.72:0/2765203295 >> 172.21.15.72:0/2765203295 conn(0x7f7a940fc4e0 msgr2=0x7f7a940fe900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:03.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.247+0000 7f7a9af3a640 1 -- 172.21.15.72:0/2765203295 shutdown_connections 2024-03-22T01:46:03.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.247+0000 7f7a9af3a640 1 -- 172.21.15.72:0/2765203295 wait complete. 2024-03-22T01:46:03.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.247+0000 7f7a9af3a640 1 Processor -- start 2024-03-22T01:46:03.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.248+0000 7f7a9af3a640 1 -- start start 2024-03-22T01:46:03.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.248+0000 7f7a9af3a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a94100a30 0x7f7a94197340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:03.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.248+0000 7f7a9af3a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7a941013f0 0x7f7a94197880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:03.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.248+0000 7f7a9af3a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7a94106a40 0x7f7a9419bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:03.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.248+0000 7f7a9af3a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f7a94110e20 con 0x7f7a941013f0 2024-03-22T01:46:03.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.248+0000 7f7a9af3a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f7a94110ca0 con 0x7f7a94106a40 2024-03-22T01:46:03.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.248+0000 7f7a9af3a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7a94110fa0 con 0x7f7a94100a30 2024-03-22T01:46:03.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.248+0000 7f7a93fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7a941013f0 0x7f7a94197880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:03.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.248+0000 7f7a93fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7a941013f0 0x7f7a94197880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38694/0 (socket says 172.21.15.72:38694) 2024-03-22T01:46:03.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.248+0000 7f7a93fff640 1 -- 172.21.15.72:0/3357713871 learned_addr learned my addr 172.21.15.72:0/3357713871 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:03.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.248+0000 7f7a994d7640 1 --2- 172.21.15.72:0/3357713871 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7a94106a40 0x7f7a9419bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:03.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.248+0000 7f7a98cd6640 1 --2- 172.21.15.72:0/3357713871 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a94100a30 0x7f7a94197340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:03.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.248+0000 7f7a93fff640 1 -- 172.21.15.72:0/3357713871 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a94100a30 msgr2=0x7f7a94197340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:03.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.248+0000 7f7a93fff640 1 --2- 172.21.15.72:0/3357713871 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a94100a30 0x7f7a94197340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.248+0000 7f7a93fff640 1 -- 172.21.15.72:0/3357713871 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7a94106a40 msgr2=0x7f7a9419bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:03.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.248+0000 7f7a93fff640 1 --2- 172.21.15.72:0/3357713871 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7a94106a40 0x7f7a9419bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.248+0000 7f7a93fff640 1 -- 172.21.15.72:0/3357713871 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a9419c330 con 0x7f7a941013f0 2024-03-22T01:46:03.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.248+0000 7f7a994d7640 1 --2- 172.21.15.72:0/3357713871 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7a94106a40 0x7f7a9419bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:46:03.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.248+0000 7f7a93fff640 1 --2- 172.21.15.72:0/3357713871 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7a941013f0 0x7f7a94197880 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f7a80009230 tx=0x7f7a8002cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:03.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.248+0000 7f7a91ffb640 1 -- 172.21.15.72:0/3357713871 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a80041070 con 0x7f7a941013f0 2024-03-22T01:46:03.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.248+0000 7f7a9af3a640 1 -- 172.21.15.72:0/3357713871 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a9419c5c0 con 0x7f7a941013f0 2024-03-22T01:46:03.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.248+0000 7f7a91ffb640 1 -- 172.21.15.72:0/3357713871 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a80004450 con 0x7f7a941013f0 2024-03-22T01:46:03.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.248+0000 7f7a91ffb640 1 -- 172.21.15.72:0/3357713871 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a80034440 con 0x7f7a941013f0 2024-03-22T01:46:03.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.249+0000 7f7a9af3a640 1 -- 172.21.15.72:0/3357713871 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a941a3eb0 con 0x7f7a941013f0 2024-03-22T01:46:03.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.251+0000 7f7a91ffb640 1 -- 172.21.15.72:0/3357713871 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 25) ==== 137985+0+0 (secure 0 0 0) 0x7f7a80044070 con 0x7f7a941013f0 2024-03-22T01:46:03.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.251+0000 7f7a9dbfc740 1 Processor -- start 2024-03-22T01:46:03.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.252+0000 7f7a9dbfc740 1 -- start start 2024-03-22T01:46:03.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.252+0000 7f7a9dbfc740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55efc25c8410 0x55efc25c87f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:03.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.252+0000 7f7a9dbfc740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55efc25c8d30 0x55efc25d13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:03.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.252+0000 7f7a9dbfc740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55efc25d1910 0x55efc25d3d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:03.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.252+0000 7f7a9dbfc740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55efc252c8c0 con 0x55efc25c8d30 2024-03-22T01:46:03.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.252+0000 7f7a9dbfc740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55efc24d4c20 con 0x55efc25d1910 2024-03-22T01:46:03.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.252+0000 7f7a9dbfc740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55efc2528870 con 0x55efc25c8410 2024-03-22T01:46:03.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.252+0000 7f7a98cd6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55efc25c8d30 0x55efc25d13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:03.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.252+0000 7f7a98cd6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55efc25c8d30 0x55efc25d13d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38704/0 (socket says 172.21.15.72:38704) 2024-03-22T01:46:03.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.252+0000 7f7a98cd6640 1 -- 172.21.15.72:0/2316640010 learned_addr learned my addr 172.21.15.72:0/2316640010 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:03.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.252+0000 7f7a994d7640 1 --2- 172.21.15.72:0/2316640010 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55efc25c8410 0x55efc25c87f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:03.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.252+0000 7f7a93fff640 1 --2- 172.21.15.72:0/2316640010 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55efc25d1910 0x55efc25d3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:03.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.252+0000 7f7a91ffb640 1 --2- 172.21.15.72:0/3357713871 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f7a840a3090 0x7f7a840a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:03.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.252+0000 7f7a98cd6640 1 -- 172.21.15.72:0/2316640010 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55efc25c8410 msgr2=0x55efc25c87f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:03.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.252+0000 7f7a98cd6640 1 --2- 172.21.15.72:0/2316640010 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55efc25c8410 0x55efc25c87f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.252+0000 7f7a994d7640 1 --2- 172.21.15.72:0/3357713871 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f7a840a3090 0x7f7a840a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:03.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.252+0000 7f7a98cd6640 1 -- 172.21.15.72:0/2316640010 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55efc25d1910 msgr2=0x55efc25d3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:03.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.252+0000 7f7a98cd6640 1 --2- 172.21.15.72:0/2316640010 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55efc25d1910 0x55efc25d3d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.252+0000 7f7a98cd6640 1 -- 172.21.15.72:0/2316640010 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55efc24f9150 con 0x55efc25c8d30 2024-03-22T01:46:03.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.252+0000 7f7a91ffb640 1 -- 172.21.15.72:0/3357713871 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f7a800f14c0 con 0x7f7a941013f0 2024-03-22T01:46:03.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.252+0000 7f7a98cd6640 1 --2- 172.21.15.72:0/2316640010 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55efc25c8d30 0x55efc25d13d0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f7a8800cbe0 tx=0x7f7a8800e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cd4ebf4ca4af30c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:03.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.253+0000 7f7a7effd640 1 -- 172.21.15.72:0/2316640010 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a88005220 con 0x55efc25c8d30 2024-03-22T01:46:03.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.253+0000 7f7a994d7640 1 --2- 172.21.15.72:0/3357713871 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f7a840a3090 0x7f7a840a5550 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c0049d0 tx=0x7f7a8c004ab0 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:03.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.253+0000 7f7a7effd640 1 -- 172.21.15.72:0/2316640010 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a88004c80 con 0x55efc25c8d30 2024-03-22T01:46:03.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.253+0000 7f7a7effd640 1 -- 172.21.15.72:0/2316640010 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a88010870 con 0x55efc25c8d30 2024-03-22T01:46:03.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.253+0000 7f7a9dbfc740 1 -- 172.21.15.72:0/2316640010 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55efc25c8d30 msgr2=0x55efc25d13d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:03.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.253+0000 7f7a9dbfc740 1 --2- 172.21.15.72:0/2316640010 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55efc25c8d30 0x55efc25d13d0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f7a8800cbe0 tx=0x7f7a8800e720 comp rx=0 tx=0).stop 2024-03-22T01:46:03.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.253+0000 7f7a9dbfc740 1 -- 172.21.15.72:0/2316640010 shutdown_connections 2024-03-22T01:46:03.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.253+0000 7f7a9dbfc740 1 --2- 172.21.15.72:0/2316640010 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55efc25d1910 0x55efc25d3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.253+0000 7f7a9dbfc740 1 --2- 172.21.15.72:0/2316640010 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55efc25c8d30 0x55efc25d13d0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.253+0000 7f7a9dbfc740 1 --2- 172.21.15.72:0/2316640010 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55efc25c8410 0x55efc25c87f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.253+0000 7f7a9dbfc740 1 -- 172.21.15.72:0/2316640010 >> 172.21.15.72:0/2316640010 conn(0x55efc25c2b40 msgr2=0x55efc25c2f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:03.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.253+0000 7f7a9dbfc740 1 -- 172.21.15.72:0/2316640010 shutdown_connections 2024-03-22T01:46:03.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.253+0000 7f7a9dbfc740 1 -- 172.21.15.72:0/2316640010 wait complete. 2024-03-22T01:46:03.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.254+0000 7f7a9dbfc740 1 -- start start 2024-03-22T01:46:03.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.254+0000 7f7a9dbfc740 1 Processor -- start 2024-03-22T01:46:03.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.254+0000 7f7a9dbfc740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55efc25c8410 0x55efc25ce8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:03.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.254+0000 7f7a9dbfc740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55efc25c8d30 0x55efc25cee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:03.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.254+0000 7f7a9dbfc740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55efc25d1910 0x55efc25cf360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:03.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.254+0000 7f7a9dbfc740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55efc25d4700 con 0x55efc25c8410 2024-03-22T01:46:03.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.254+0000 7f7a9dbfc740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55efc24e8130 con 0x55efc25c8d30 2024-03-22T01:46:03.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.254+0000 7f7a9dbfc740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55efc25d4880 con 0x55efc25d1910 2024-03-22T01:46:03.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.254+0000 7f7a9dbfc740 10 client.0 _ll_register_callbacks cb 0x55efc24e7b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:46:03.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.254+0000 7f7a98cd6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55efc25c8410 0x55efc25ce8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:03.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.254+0000 7f7a98cd6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55efc25c8410 0x55efc25ce8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38708/0 (socket says 172.21.15.72:38708) 2024-03-22T01:46:03.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.254+0000 7f7a98cd6640 1 -- 172.21.15.72:0/1886623879 learned_addr learned my addr 172.21.15.72:0/1886623879 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:03.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.254+0000 7f7a93fff640 1 --2- 172.21.15.72:0/1886623879 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55efc25c8d30 0x55efc25cee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:03.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.254+0000 7f7a98cd6640 1 -- 172.21.15.72:0/1886623879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55efc25d1910 msgr2=0x55efc25cf360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:03.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.254+0000 7f7a994d7640 1 --2- 172.21.15.72:0/1886623879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55efc25d1910 0x55efc25cf360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:03.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.254+0000 7f7a98cd6640 1 --2- 172.21.15.72:0/1886623879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55efc25d1910 0x55efc25cf360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.254+0000 7f7a98cd6640 1 -- 172.21.15.72:0/1886623879 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55efc25c8d30 msgr2=0x55efc25cee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:03.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.254+0000 7f7a98cd6640 1 --2- 172.21.15.72:0/1886623879 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55efc25c8d30 0x55efc25cee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.254+0000 7f7a98cd6640 1 -- 172.21.15.72:0/1886623879 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55efc230a7d0 con 0x55efc25c8410 2024-03-22T01:46:03.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.255+0000 7f7a98cd6640 1 --2- 172.21.15.72:0/1886623879 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55efc25c8410 0x55efc25ce8e0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f7a88010470 tx=0x7f7a8800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c11c5c88558fade2 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:03.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.255+0000 7f7a73fff640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:46:03.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.255+0000 7f7a73fff640 1 -- 172.21.15.72:0/1886623879 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a88005650 con 0x55efc25c8410 2024-03-22T01:46:03.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.255+0000 7f7a73fff640 1 -- 172.21.15.72:0/1886623879 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f7a88004240 con 0x55efc25c8410 2024-03-22T01:46:03.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.255+0000 7f7a9dbfc740 1 -- 172.21.15.72:0/1886623879 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7a88004c80 con 0x55efc25c8410 2024-03-22T01:46:03.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.255+0000 7f7a73fff640 1 -- 172.21.15.72:0/1886623879 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7f7a88004440 con 0x55efc25c8410 2024-03-22T01:46:03.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.255+0000 7f7a9dbfc740 10 client.16227 fetch_fsmap learned FSMap version 103 2024-03-22T01:46:03.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.255+0000 7f7a9dbfc740 1 -- 172.21.15.72:0/1886623879 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=4+}) -- 0x55efc22403f0 con 0x55efc25c8410 2024-03-22T01:46:03.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.255+0000 7f7a73fff640 1 -- 172.21.15.72:0/1886623879 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a88004c80 con 0x55efc25c8410 2024-03-22T01:46:03.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.255+0000 7f7a73fff640 1 -- 172.21.15.72:0/1886623879 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 103) ==== 8330+0+0 (secure 0 0 0) 0x7f7a88010be0 con 0x55efc25c8410 2024-03-22T01:46:03.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.255+0000 7f7a9dbfc740 10 client.16227 fetch_fsmap finished waiting for FSMap version 103 2024-03-22T01:46:03.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.255+0000 7f7a9dbfc740 10 client.16227 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T01:46:03.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.255+0000 7f7a9dbfc740 20 client.16227 populate_metadata read hostname 'smithi072' 2024-03-22T01:46:03.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.255+0000 7f7a9dbfc740 1 --2- 172.21.15.72:0/1886623879 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55efc25e1c00 0x55efc25e3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:03.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.255+0000 7f7a9dbfc740 4 client.16227 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T01:46:03.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.255+0000 7f7a9dbfc740 1 -- 172.21.15.72:0/1886623879 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55efc2539e00 con 0x55efc25e1c00 2024-03-22T01:46:03.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.256+0000 7f7a93fff640 1 --2- 172.21.15.72:0/1886623879 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55efc25e1c00 0x55efc25e3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:03.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.256+0000 7f7a93fff640 1 --2- 172.21.15.72:0/1886623879 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55efc25e1c00 0x55efc25e3ff0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf102016fb17dcd2 server_cookie=8ab7900f07c9ab2b in_seq=0 out_seq=0 2024-03-22T01:46:03.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.256+0000 7f7a73fff640 10 client.16227 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:46:03.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.256+0000 7f7a73fff640 1 -- 172.21.15.72:0/1886623879 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55efc2539e00 con 0x55efc25e1c00 2024-03-22T01:46:03.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.256+0000 7f7a73fff640 10 client.16227 handle_command_reply: tid=0 2024-03-22T01:46:03.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.256+0000 7f7a9dbfc740 1 client.16227 shutdown 2024-03-22T01:46:03.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.257+0000 7f7a9dbfc740 1 -- 172.21.15.72:0/1886623879 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55efc25c8410 msgr2=0x55efc25ce8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:03.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.257+0000 7f7a9dbfc740 1 --2- 172.21.15.72:0/1886623879 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55efc25c8410 0x55efc25ce8e0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f7a88010470 tx=0x7f7a8800c920 comp rx=0 tx=0).stop 2024-03-22T01:46:03.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.257+0000 7f7a9dbfc740 1 -- 172.21.15.72:0/1886623879 shutdown_connections 2024-03-22T01:46:03.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.257+0000 7f7a9dbfc740 1 --2- 172.21.15.72:0/1886623879 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55efc25e1c00 0x55efc25e3ff0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.257+0000 7f7a9dbfc740 1 --2- 172.21.15.72:0/1886623879 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55efc25c8d30 0x55efc25cee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.257+0000 7f7a9dbfc740 1 --2- 172.21.15.72:0/1886623879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55efc25d1910 0x55efc25cf360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.257+0000 7f7a9dbfc740 1 --2- 172.21.15.72:0/1886623879 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55efc25c8410 0x55efc25ce8e0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.257+0000 7f7a73fff640 0 client.16227 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:46:03.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.257+0000 7f7a9dbfc740 1 -- 172.21.15.72:0/1886623879 >> 172.21.15.72:0/1886623879 conn(0x55efc25c2b40 msgr2=0x55efc25c4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:03.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.257+0000 7f7a9dbfc740 1 -- 172.21.15.72:0/1886623879 shutdown_connections 2024-03-22T01:46:03.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.257+0000 7f7a9dbfc740 1 -- 172.21.15.72:0/1886623879 wait complete. 2024-03-22T01:46:03.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.257+0000 7f7a9dbfc740 20 client.16227 trim_cache size 0 max 16384 2024-03-22T01:46:03.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.274+0000 7f7a9dbfc740 1 Processor -- start 2024-03-22T01:46:03.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.274+0000 7f7a9dbfc740 1 -- start start 2024-03-22T01:46:03.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.274+0000 7f7a9dbfc740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55efc25c8410 0x55efc25dd4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:03.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.274+0000 7f7a9dbfc740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55efc25c8d30 0x55efc25e97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:03.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.274+0000 7f7a9dbfc740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55efc25d1910 0x55efc25cefb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:03.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.274+0000 7f7a9dbfc740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f7a941abf50 con 0x55efc25d1910 2024-03-22T01:46:03.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.274+0000 7f7a9dbfc740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f7a941ae7d0 con 0x55efc25c8410 2024-03-22T01:46:03.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.274+0000 7f7a9dbfc740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7a941ac5c0 con 0x55efc25c8d30 2024-03-22T01:46:03.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.274+0000 7f7a994d7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55efc25d1910 0x55efc25cefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:03.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.274+0000 7f7a994d7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55efc25d1910 0x55efc25cefb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38724/0 (socket says 172.21.15.72:38724) 2024-03-22T01:46:03.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.274+0000 7f7a994d7640 1 -- 172.21.15.72:0/1472792424 learned_addr learned my addr 172.21.15.72:0/1472792424 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:03.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.274+0000 7f7a98cd6640 1 --2- 172.21.15.72:0/1472792424 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55efc25c8410 0x55efc25dd4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:03.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.274+0000 7f7a93fff640 1 --2- 172.21.15.72:0/1472792424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55efc25c8d30 0x55efc25e97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:03.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.274+0000 7f7a994d7640 1 -- 172.21.15.72:0/1472792424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55efc25c8d30 msgr2=0x55efc25e97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:03.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.274+0000 7f7a994d7640 1 --2- 172.21.15.72:0/1472792424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55efc25c8d30 0x55efc25e97d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.274+0000 7f7a994d7640 1 -- 172.21.15.72:0/1472792424 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55efc25c8410 msgr2=0x55efc25dd4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:03.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.274+0000 7f7a994d7640 1 --2- 172.21.15.72:0/1472792424 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55efc25c8410 0x55efc25dd4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.274+0000 7f7a994d7640 1 -- 172.21.15.72:0/1472792424 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55efc25e4360 con 0x55efc25d1910 2024-03-22T01:46:03.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.274+0000 7f7a994d7640 1 --2- 172.21.15.72:0/1472792424 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55efc25d1910 0x55efc25cefb0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c013a00 tx=0x7f7a8c009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a46c82450e2f1d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:03.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.274+0000 7f7a7e7fc640 1 -- 172.21.15.72:0/1472792424 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a8c010e40 con 0x55efc25d1910 2024-03-22T01:46:03.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.275+0000 7f7a7e7fc640 1 -- 172.21.15.72:0/1472792424 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a8c026070 con 0x55efc25d1910 2024-03-22T01:46:03.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.275+0000 7f7a7e7fc640 1 -- 172.21.15.72:0/1472792424 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a8c0058e0 con 0x55efc25d1910 2024-03-22T01:46:03.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.275+0000 7f7a9dbfc740 1 -- 172.21.15.72:0/1472792424 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55efc25d1910 msgr2=0x55efc25cefb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:03.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.275+0000 7f7a9dbfc740 1 --2- 172.21.15.72:0/1472792424 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55efc25d1910 0x55efc25cefb0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c013a00 tx=0x7f7a8c009f90 comp rx=0 tx=0).stop 2024-03-22T01:46:03.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.275+0000 7f7a9dbfc740 1 -- 172.21.15.72:0/1472792424 shutdown_connections 2024-03-22T01:46:03.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.275+0000 7f7a9dbfc740 1 --2- 172.21.15.72:0/1472792424 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55efc25d1910 0x55efc25cefb0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.275+0000 7f7a9dbfc740 1 --2- 172.21.15.72:0/1472792424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55efc25c8d30 0x55efc25e97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.275+0000 7f7a9dbfc740 1 --2- 172.21.15.72:0/1472792424 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55efc25c8410 0x55efc25dd4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.275+0000 7f7a9dbfc740 1 -- 172.21.15.72:0/1472792424 >> 172.21.15.72:0/1472792424 conn(0x55efc25c2b40 msgr2=0x55efc25ce8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:03.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.275+0000 7f7a9dbfc740 1 -- 172.21.15.72:0/1472792424 shutdown_connections 2024-03-22T01:46:03.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.275+0000 7f7a9dbfc740 1 -- 172.21.15.72:0/1472792424 wait complete. 2024-03-22T01:46:03.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.275+0000 7f7a9dbfc740 1 -- start start 2024-03-22T01:46:03.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.275+0000 7f7a9dbfc740 1 Processor -- start 2024-03-22T01:46:03.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.276+0000 7f7a9dbfc740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55efc25c8410 0x55efc25d53e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:03.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.276+0000 7f7a9dbfc740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55efc25c8d30 0x55efc25d5920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:03.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.276+0000 7f7a9dbfc740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55efc25d1910 0x55efc25d5e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:03.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.276+0000 7f7a9dbfc740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f7a941acc90 con 0x55efc25c8d30 2024-03-22T01:46:03.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.276+0000 7f7a9dbfc740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f7a941ae100 con 0x55efc25d1910 2024-03-22T01:46:03.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.276+0000 7f7a9dbfc740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7a941ad360 con 0x55efc25c8410 2024-03-22T01:46:03.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.276+0000 7f7a9dbfc740 10 client.0 _ll_register_callbacks cb 0x7f7a8c009ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:46:03.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.276+0000 7f7a93fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55efc25c8d30 0x55efc25d5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:03.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.276+0000 7f7a93fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55efc25c8d30 0x55efc25d5920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38728/0 (socket says 172.21.15.72:38728) 2024-03-22T01:46:03.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.276+0000 7f7a93fff640 1 -- 172.21.15.72:0/1709628967 learned_addr learned my addr 172.21.15.72:0/1709628967 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:03.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.276+0000 7f7a994d7640 1 --2- 172.21.15.72:0/1709628967 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55efc25d1910 0x55efc25d5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:03.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.276+0000 7f7a98cd6640 1 --2- 172.21.15.72:0/1709628967 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55efc25c8410 0x55efc25d53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:03.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.276+0000 7f7a93fff640 1 -- 172.21.15.72:0/1709628967 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55efc25c8410 msgr2=0x55efc25d53e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:03.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.276+0000 7f7a93fff640 1 --2- 172.21.15.72:0/1709628967 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55efc25c8410 0x55efc25d53e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.276+0000 7f7a93fff640 1 -- 172.21.15.72:0/1709628967 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55efc25d1910 msgr2=0x55efc25d5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:03.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.276+0000 7f7a93fff640 1 --2- 172.21.15.72:0/1709628967 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55efc25d1910 0x55efc25d5e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.276+0000 7f7a93fff640 1 -- 172.21.15.72:0/1709628967 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55efc22000f0 con 0x55efc25c8d30 2024-03-22T01:46:03.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.276+0000 7f7a98cd6640 1 --2- 172.21.15.72:0/1709628967 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55efc25c8410 0x55efc25d53e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T01:46:03.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.276+0000 7f7a93fff640 1 --2- 172.21.15.72:0/1709628967 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55efc25c8d30 0x55efc25d5920 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f7a800b9fe0 tx=0x7f7a800e6040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e67d570ec3ce681 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:03.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.276+0000 7f7a73fff640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:46:03.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.276+0000 7f7a73fff640 1 -- 172.21.15.72:0/1709628967 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a8004b040 con 0x55efc25c8d30 2024-03-22T01:46:03.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.277+0000 7f7a73fff640 1 -- 172.21.15.72:0/1709628967 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f7a800b9c80 con 0x55efc25c8d30 2024-03-22T01:46:03.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.277+0000 7f7a9dbfc740 1 -- 172.21.15.72:0/1709628967 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7a8c026070 con 0x55efc25c8d30 2024-03-22T01:46:03.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.277+0000 7f7a73fff640 1 -- 172.21.15.72:0/1709628967 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7f7a800d5070 con 0x55efc25c8d30 2024-03-22T01:46:03.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.277+0000 7f7a9dbfc740 10 client.16233 fetch_fsmap learned FSMap version 103 2024-03-22T01:46:03.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.277+0000 7f7a9dbfc740 1 -- 172.21.15.72:0/1709628967 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=4+}) -- 0x55efc264c380 con 0x55efc25c8d30 2024-03-22T01:46:03.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.277+0000 7f7a73fff640 1 -- 172.21.15.72:0/1709628967 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a8c026070 con 0x55efc25c8d30 2024-03-22T01:46:03.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.277+0000 7f7a73fff640 1 -- 172.21.15.72:0/1709628967 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 103) ==== 8330+0+0 (secure 0 0 0) 0x7f7a800035f0 con 0x55efc25c8d30 2024-03-22T01:46:03.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.277+0000 7f7a9dbfc740 10 client.16233 fetch_fsmap finished waiting for FSMap version 103 2024-03-22T01:46:03.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.277+0000 7f7a9dbfc740 10 client.16233 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T01:46:03.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.277+0000 7f7a9dbfc740 20 client.16233 populate_metadata read hostname 'smithi072' 2024-03-22T01:46:03.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.277+0000 7f7a9dbfc740 1 --2- 172.21.15.72:0/1709628967 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55efc264cb80 0x55efc264ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:03.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.277+0000 7f7a9dbfc740 4 client.16233 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T01:46:03.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.277+0000 7f7a9dbfc740 1 -- 172.21.15.72:0/1709628967 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55efc2534150 con 0x55efc264cb80 2024-03-22T01:46:03.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.277+0000 7f7a98cd6640 1 --2- 172.21.15.72:0/1709628967 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55efc264cb80 0x55efc264ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:03.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.277+0000 7f7a73fff640 10 client.16233 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:46:03.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.277+0000 7f7a98cd6640 1 --2- 172.21.15.72:0/1709628967 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55efc264cb80 0x55efc264ef70 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=50f3695c754823df server_cookie=406fc77cd365a82e in_seq=0 out_seq=0 2024-03-22T01:46:03.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.278+0000 7f7a73fff640 1 -- 172.21.15.72:0/1709628967 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55efc2534150 con 0x55efc264cb80 2024-03-22T01:46:03.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.278+0000 7f7a73fff640 10 client.16233 handle_command_reply: tid=0 2024-03-22T01:46:03.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.278+0000 7f7a9dbfc740 1 client.16233 shutdown 2024-03-22T01:46:03.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.278+0000 7f7a9dbfc740 1 -- 172.21.15.72:0/1709628967 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55efc25c8d30 msgr2=0x55efc25d5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:03.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.278+0000 7f7a9dbfc740 1 --2- 172.21.15.72:0/1709628967 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55efc25c8d30 0x55efc25d5920 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f7a800b9fe0 tx=0x7f7a800e6040 comp rx=0 tx=0).stop 2024-03-22T01:46:03.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.278+0000 7f7a9dbfc740 1 -- 172.21.15.72:0/1709628967 shutdown_connections 2024-03-22T01:46:03.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.278+0000 7f7a9dbfc740 1 --2- 172.21.15.72:0/1709628967 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55efc264cb80 0x55efc264ef70 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.278+0000 7f7a9dbfc740 1 --2- 172.21.15.72:0/1709628967 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55efc25d1910 0x55efc25d5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.278+0000 7f7a9dbfc740 1 --2- 172.21.15.72:0/1709628967 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55efc25c8d30 0x55efc25d5920 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.278+0000 7f7a73fff640 0 client.16233 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:46:03.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.278+0000 7f7a9dbfc740 1 --2- 172.21.15.72:0/1709628967 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55efc25c8410 0x55efc25d53e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.278+0000 7f7a9dbfc740 1 -- 172.21.15.72:0/1709628967 >> 172.21.15.72:0/1709628967 conn(0x55efc25c2b40 msgr2=0x55efc25c1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:03.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.278+0000 7f7a9dbfc740 1 -- 172.21.15.72:0/1709628967 shutdown_connections 2024-03-22T01:46:03.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.278+0000 7f7a9dbfc740 1 -- 172.21.15.72:0/1709628967 wait complete. 2024-03-22T01:46:03.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.278+0000 7f7a9dbfc740 20 client.16233 trim_cache size 0 max 16384 2024-03-22T01:46:03.285 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-22T01:46:03.285 INFO:teuthology.orchestra.run.smithi072.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-22T01:46:03.285 INFO:teuthology.orchestra.run.smithi072.stdout: "scrubs": { 2024-03-22T01:46:03.285 INFO:teuthology.orchestra.run.smithi072.stdout: "ea880a42-3ab2-4a31-9d1f-5eaeeaaff5d6": { 2024-03-22T01:46:03.285 INFO:teuthology.orchestra.run.smithi072.stdout: "path": "/", 2024-03-22T01:46:03.285 INFO:teuthology.orchestra.run.smithi072.stdout: "tag": "ea880a42-3ab2-4a31-9d1f-5eaeeaaff5d6", 2024-03-22T01:46:03.285 INFO:teuthology.orchestra.run.smithi072.stdout: "options": "recursive,force" 2024-03-22T01:46:03.285 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:46:03.285 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:46:03.285 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-22T01:46:03.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.278+0000 7f7a9af3a640 1 -- 172.21.15.72:0/3357713871 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f7a840a3090 msgr2=0x7f7a840a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:03.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.278+0000 7f7a9af3a640 1 --2- 172.21.15.72:0/3357713871 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f7a840a3090 0x7f7a840a5550 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c0049d0 tx=0x7f7a8c004ab0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.278+0000 7f7a9af3a640 1 -- 172.21.15.72:0/3357713871 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7a941013f0 msgr2=0x7f7a94197880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:03.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.278+0000 7f7a9af3a640 1 --2- 172.21.15.72:0/3357713871 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7a941013f0 0x7f7a94197880 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f7a80009230 tx=0x7f7a8002cba0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.278+0000 7f7a9af3a640 1 -- 172.21.15.72:0/3357713871 shutdown_connections 2024-03-22T01:46:03.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.279+0000 7f7a9af3a640 1 --2- 172.21.15.72:0/3357713871 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f7a840a3090 0x7f7a840a5550 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.279+0000 7f7a9af3a640 1 --2- 172.21.15.72:0/3357713871 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7a94106a40 0x7f7a9419bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.279+0000 7f7a9af3a640 1 --2- 172.21.15.72:0/3357713871 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7a941013f0 0x7f7a94197880 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.279+0000 7f7a9af3a640 1 --2- 172.21.15.72:0/3357713871 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a94100a30 0x7f7a94197340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:03.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.279+0000 7f7a9af3a640 1 -- 172.21.15.72:0/3357713871 >> 172.21.15.72:0/3357713871 conn(0x7f7a940fc4e0 msgr2=0x7f7a940fcaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:03.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.279+0000 7f7a9af3a640 1 -- 172.21.15.72:0/3357713871 shutdown_connections 2024-03-22T01:46:03.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:03.279+0000 7f7a9af3a640 1 -- 172.21.15.72:0/3357713871 wait complete. 2024-03-22T01:46:03.288 INFO:tasks.cephfs.filesystem:scrub status for tag:ea880a42-3ab2-4a31-9d1f-5eaeeaaff5d6 - {'path': '/', 'tag': 'ea880a42-3ab2-4a31-9d1f-5eaeeaaff5d6', 'options': 'recursive,force'} 2024-03-22T01:46:04.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:46:04 smithi089 ceph-mon[35770]: pgmap v285: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 16 KiB/s wr, 5 op/s 2024-03-22T01:46:04.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:46:04 smithi089 ceph-mon[35770]: scrub queued for path: / 2024-03-22T01:46:04.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:46:04 smithi089 ceph-mon[35770]: scrub summary: idle+waiting paths [/] 2024-03-22T01:46:04.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:46:04 smithi089 ceph-mon[35770]: scrub summary: active paths [/] 2024-03-22T01:46:04.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:46:04 smithi089 ceph-mon[35770]: scrub summary: idle+waiting paths [/] 2024-03-22T01:46:04.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:46:04 smithi089 ceph-mon[35770]: scrub summary: idle 2024-03-22T01:46:04.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:46:04 smithi072 ceph-mon[32414]: pgmap v285: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 16 KiB/s wr, 5 op/s 2024-03-22T01:46:04.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:46:04 smithi072 ceph-mon[32414]: scrub queued for path: / 2024-03-22T01:46:04.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:46:04 smithi072 ceph-mon[32414]: scrub summary: idle+waiting paths [/] 2024-03-22T01:46:04.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:46:04 smithi072 ceph-mon[32414]: scrub summary: active paths [/] 2024-03-22T01:46:04.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:46:04 smithi072 ceph-mon[32414]: scrub summary: idle+waiting paths [/] 2024-03-22T01:46:04.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:46:04 smithi072 ceph-mon[32414]: scrub summary: idle 2024-03-22T01:46:04.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:46:04 smithi098 ceph-mon[36468]: pgmap v285: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 16 KiB/s wr, 5 op/s 2024-03-22T01:46:04.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:46:04 smithi098 ceph-mon[36468]: scrub queued for path: / 2024-03-22T01:46:04.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:46:04 smithi098 ceph-mon[36468]: scrub summary: idle+waiting paths [/] 2024-03-22T01:46:04.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:46:04 smithi098 ceph-mon[36468]: scrub summary: active paths [/] 2024-03-22T01:46:04.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:46:04 smithi098 ceph-mon[36468]: scrub summary: idle+waiting paths [/] 2024-03-22T01:46:04.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:46:04 smithi098 ceph-mon[36468]: scrub summary: idle 2024-03-22T01:46:06.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:46:06 smithi089 ceph-mon[35770]: pgmap v286: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 12 KiB/s wr, 3 op/s 2024-03-22T01:46:06.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:46:06 smithi072 ceph-mon[32414]: pgmap v286: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 12 KiB/s wr, 3 op/s 2024-03-22T01:46:06.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:46:06 smithi098 ceph-mon[36468]: pgmap v286: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 12 KiB/s wr, 3 op/s 2024-03-22T01:46:08.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:46:08 smithi089 ceph-mon[35770]: pgmap v287: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 25 KiB/s wr, 44 op/s 2024-03-22T01:46:08.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:46:08 smithi072 ceph-mon[32414]: pgmap v287: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 25 KiB/s wr, 44 op/s 2024-03-22T01:46:08.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:46:08 smithi098 ceph-mon[36468]: pgmap v287: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 25 KiB/s wr, 44 op/s 2024-03-22T01:46:10.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:46:10 smithi089 ceph-mon[35770]: pgmap v288: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 25 KiB/s wr, 53 op/s 2024-03-22T01:46:10.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:46:10 smithi072 ceph-mon[32414]: pgmap v288: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 25 KiB/s wr, 53 op/s 2024-03-22T01:46:10.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:46:10 smithi098 ceph-mon[36468]: pgmap v288: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 25 KiB/s wr, 53 op/s 2024-03-22T01:46:10.498 INFO:tasks.workunit.client.0.smithi072.stdout:Syncing()...50 sec 2024-03-22T01:46:10.499 INFO:tasks.workunit.client.0.smithi072.stdout:FFSB benchmark finished at: Fri Mar 22 01:46:10 2024 2024-03-22T01:46:10.499 INFO:tasks.workunit.client.0.smithi072.stdout: 2024-03-22T01:46:10.499 INFO:tasks.workunit.client.0.smithi072.stdout:Results: 2024-03-22T01:46:10.499 INFO:tasks.workunit.client.0.smithi072.stdout:Benchmark took 456.02 sec 2024-03-22T01:46:10.499 INFO:tasks.workunit.client.0.smithi072.stdout: 2024-03-22T01:46:10.499 INFO:tasks.workunit.client.0.smithi072.stdout:Total Results 2024-03-22T01:46:10.499 INFO:tasks.workunit.client.0.smithi072.stdout:=============== 2024-03-22T01:46:10.499 INFO:tasks.workunit.client.0.smithi072.stdout: Op Name Transactions Trans/sec % Trans % Op Weight Throughput 2024-03-22T01:46:10.499 INFO:tasks.workunit.client.0.smithi072.stdout: ======= ============ ========= ======= =========== ========== 2024-03-22T01:46:10.499 INFO:tasks.workunit.client.0.smithi072.stdout: write : 2636800 5782.20 100.000% 100.000% 22.6MB/sec 2024-03-22T01:46:10.499 INFO:tasks.workunit.client.0.smithi072.stdout:- 2024-03-22T01:46:10.499 INFO:tasks.workunit.client.0.smithi072.stdout:5782.20 Transactions per Second 2024-03-22T01:46:10.499 INFO:tasks.workunit.client.0.smithi072.stdout: 2024-03-22T01:46:10.499 INFO:tasks.workunit.client.0.smithi072.stdout:Throughput Results 2024-03-22T01:46:10.499 INFO:tasks.workunit.client.0.smithi072.stdout:=================== 2024-03-22T01:46:10.500 INFO:tasks.workunit.client.0.smithi072.stdout:Write Throughput: 22.6MB/sec 2024-03-22T01:46:10.511 INFO:tasks.workunit.client.0.smithi072.stdout: 2024-03-22T01:46:10.511 INFO:tasks.workunit.client.0.smithi072.stdout:System Call Latency statistics in millisecs 2024-03-22T01:46:10.511 INFO:tasks.workunit.client.0.smithi072.stdout:===== 2024-03-22T01:46:10.511 INFO:tasks.workunit.client.0.smithi072.stdout: Min Avg Max Total Calls 2024-03-22T01:46:10.511 INFO:tasks.workunit.client.0.smithi072.stdout: ======== ======== ======== ============ 2024-03-22T01:46:10.511 INFO:tasks.workunit.client.0.smithi072.stdout:[ open] 0.004000 0.081592 17.375000 2060 2024-03-22T01:46:10.511 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-03-22T01:46:10.512 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-03-22T01:46:10.512 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-03-22T01:46:10.512 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-03-22T01:46:10.512 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-03-22T01:46:10.512 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[5] 100.000000 - 200.000000 : 0 2024-03-22T01:46:10.512 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[6] 50.000000 - 100.000000 : 0 2024-03-22T01:46:10.512 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[7] 20.000000 - 50.000000 : 0 2024-03-22T01:46:10.512 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[8] 10.000000 - 20.000000 : 6 2024-03-22T01:46:10.512 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[9] 5.000000 - 10.000000 : 3 2024-03-22T01:46:10.512 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[10] 2.000000 - 5.000000 : 6 2024-03-22T01:46:10.512 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[11] 1.000000 - 2.000000 : 5 2024-03-22T01:46:10.512 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[12] 0.500000 - 1.000000 : 1 2024-03-22T01:46:10.512 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[13] 0.200000 - 0.500000 : 1 2024-03-22T01:46:10.512 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[14] 0.100000 - 0.200000 : 1 2024-03-22T01:46:10.512 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[15] 0.050000 - 0.100000 : 14 2024-03-22T01:46:10.513 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[16] 0.020000 - 0.050000 : 328 2024-03-22T01:46:10.513 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[17] 0.010000 - 0.020000 : 1108 2024-03-22T01:46:10.513 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[18] 0.000000 - 0.010000 : 587 2024-03-22T01:46:10.513 INFO:tasks.workunit.client.0.smithi072.stdout: 2024-03-22T01:46:10.513 INFO:tasks.workunit.client.0.smithi072.stdout:[ write] 0.001000 4.181582 87063.484375 2636800 2024-03-22T01:46:10.513 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[0] 5000.000000 - 10000.000000 : 77 2024-03-22T01:46:10.513 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[1] 2000.000000 - 5000.000000 : 80 2024-03-22T01:46:10.513 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[2] 1000.000000 - 2000.000000 : 90 2024-03-22T01:46:10.513 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[3] 500.000000 - 1000.000000 : 117 2024-03-22T01:46:10.513 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[4] 200.000000 - 500.000000 : 10631 2024-03-22T01:46:10.513 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[5] 100.000000 - 200.000000 : 937 2024-03-22T01:46:10.513 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[6] 50.000000 - 100.000000 : 274 2024-03-22T01:46:10.513 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[7] 20.000000 - 50.000000 : 864 2024-03-22T01:46:10.513 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[8] 10.000000 - 20.000000 : 1471 2024-03-22T01:46:10.513 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[9] 5.000000 - 10.000000 : 1528 2024-03-22T01:46:10.513 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[10] 2.000000 - 5.000000 : 282 2024-03-22T01:46:10.514 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[11] 1.000000 - 2.000000 : 234 2024-03-22T01:46:10.514 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[12] 0.500000 - 1.000000 : 191 2024-03-22T01:46:10.514 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[13] 0.200000 - 0.500000 : 180 2024-03-22T01:46:10.514 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[14] 0.100000 - 0.200000 : 159 2024-03-22T01:46:10.514 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[15] 0.050000 - 0.100000 : 562 2024-03-22T01:46:10.514 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[16] 0.020000 - 0.050000 : 5359 2024-03-22T01:46:10.514 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[17] 0.010000 - 0.020000 : 37227 2024-03-22T01:46:10.514 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[18] 0.000000 - 0.010000 : 2576286 2024-03-22T01:46:10.514 INFO:tasks.workunit.client.0.smithi072.stdout: 2024-03-22T01:46:10.514 INFO:tasks.workunit.client.0.smithi072.stdout:[ lseek] 0.000000 0.000945 85.030998 2636800 2024-03-22T01:46:10.514 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-03-22T01:46:10.514 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-03-22T01:46:10.514 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-03-22T01:46:10.514 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-03-22T01:46:10.515 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-03-22T01:46:10.515 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[5] 100.000000 - 200.000000 : 0 2024-03-22T01:46:10.515 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[6] 50.000000 - 100.000000 : 1 2024-03-22T01:46:10.515 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[7] 20.000000 - 50.000000 : 25 2024-03-22T01:46:10.515 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[8] 10.000000 - 20.000000 : 32 2024-03-22T01:46:10.515 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[9] 5.000000 - 10.000000 : 22 2024-03-22T01:46:10.515 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[10] 2.000000 - 5.000000 : 20 2024-03-22T01:46:10.515 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[11] 1.000000 - 2.000000 : 8 2024-03-22T01:46:10.515 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[12] 0.500000 - 1.000000 : 10 2024-03-22T01:46:10.515 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[13] 0.200000 - 0.500000 : 17 2024-03-22T01:46:10.515 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[14] 0.100000 - 0.200000 : 12 2024-03-22T01:46:10.515 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[15] 0.050000 - 0.100000 : 21 2024-03-22T01:46:10.515 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[16] 0.020000 - 0.050000 : 158 2024-03-22T01:46:10.515 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[17] 0.010000 - 0.020000 : 651 2024-03-22T01:46:10.515 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[18] 0.000000 - 0.010000 : 2635823 2024-03-22T01:46:10.515 INFO:tasks.workunit.client.0.smithi072.stdout: 2024-03-22T01:46:10.516 INFO:tasks.workunit.client.0.smithi072.stdout:[ close] 0.001000 0.009534 10.517000 2060 2024-03-22T01:46:10.516 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-03-22T01:46:10.516 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-03-22T01:46:10.516 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-03-22T01:46:10.516 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-03-22T01:46:10.516 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-03-22T01:46:10.516 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[5] 100.000000 - 200.000000 : 0 2024-03-22T01:46:10.516 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[6] 50.000000 - 100.000000 : 0 2024-03-22T01:46:10.516 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[7] 20.000000 - 50.000000 : 0 2024-03-22T01:46:10.516 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[8] 10.000000 - 20.000000 : 1 2024-03-22T01:46:10.516 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[9] 5.000000 - 10.000000 : 0 2024-03-22T01:46:10.516 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[10] 2.000000 - 5.000000 : 0 2024-03-22T01:46:10.516 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[11] 1.000000 - 2.000000 : 0 2024-03-22T01:46:10.516 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[12] 0.500000 - 1.000000 : 0 2024-03-22T01:46:10.516 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[13] 0.200000 - 0.500000 : 0 2024-03-22T01:46:10.517 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[14] 0.100000 - 0.200000 : 0 2024-03-22T01:46:10.517 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[15] 0.050000 - 0.100000 : 0 2024-03-22T01:46:10.517 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[16] 0.020000 - 0.050000 : 8 2024-03-22T01:46:10.517 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[17] 0.010000 - 0.020000 : 7 2024-03-22T01:46:10.517 INFO:tasks.workunit.client.0.smithi072.stdout: msec_range[18] 0.000000 - 0.010000 : 2044 2024-03-22T01:46:10.517 INFO:tasks.workunit.client.0.smithi072.stdout: 2024-03-22T01:46:10.517 INFO:tasks.workunit.client.0.smithi072.stdout: 2024-03-22T01:46:10.517 INFO:tasks.workunit.client.0.smithi072.stdout: 2024-03-22T01:46:10.517 INFO:tasks.workunit.client.0.smithi072.stdout: 2024-03-22T01:46:10.517 INFO:tasks.workunit.client.0.smithi072.stdout:0.3% User Time 2024-03-22T01:46:10.517 INFO:tasks.workunit.client.0.smithi072.stdout:1.9% System Time 2024-03-22T01:46:10.517 INFO:tasks.workunit.client.0.smithi072.stdout:2.1% CPU Utilization 2024-03-22T01:46:10.546 INFO:tasks.workunit.client.0.smithi072.stderr:+ cd .. 2024-03-22T01:46:10.546 INFO:tasks.workunit.client.0.smithi072.stderr:+ rm -r tmp ffsb 2024-03-22T01:46:12.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:46:12 smithi089 ceph-mon[35770]: pgmap v289: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 25 KiB/s wr, 53 op/s 2024-03-22T01:46:12.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:46:12 smithi072 ceph-mon[32414]: pgmap v289: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 25 KiB/s wr, 53 op/s 2024-03-22T01:46:12.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:46:12 smithi098 ceph-mon[36468]: pgmap v289: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 25 KiB/s wr, 53 op/s 2024-03-22T01:46:12.669 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-22T01:46:12.670 DEBUG:teuthology.orchestra.run.smithi072:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-22T01:46:12.746 INFO:tasks.workunit:Stopping ['suites/ffsb.sh'] on client.0... 2024-03-22T01:46:12.746 DEBUG:teuthology.orchestra.run.smithi072:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-22T01:46:14.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:46:14 smithi089 ceph-mon[35770]: pgmap v290: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 80 KiB/s wr, 54 op/s 2024-03-22T01:46:14.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:46:14 smithi072 ceph-mon[32414]: pgmap v290: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 80 KiB/s wr, 54 op/s 2024-03-22T01:46:14.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:46:14 smithi098 ceph-mon[36468]: pgmap v290: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 80 KiB/s wr, 54 op/s 2024-03-22T01:46:15.638 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.637+0000 7f6fbc25c640 1 -- 172.21.15.72:0/3288088076 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fb4116c20 msgr2=0x7f6fb4119010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:15.638 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.637+0000 7f6fbc25c640 1 --2- 172.21.15.72:0/3288088076 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fb4116c20 0x7f6fb4119010 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6fac009be0 tx=0x7f6fac02bfa0 comp rx=0 tx=0).stop 2024-03-22T01:46:15.641 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.641+0000 7f6fbc25c640 1 -- 172.21.15.72:0/3288088076 shutdown_connections 2024-03-22T01:46:15.641 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.641+0000 7f6fbc25c640 1 --2- 172.21.15.72:0/3288088076 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fb4116c20 0x7f6fb4119010 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:15.641 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.641+0000 7f6fbc25c640 1 --2- 172.21.15.72:0/3288088076 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6fb41092b0 0x7f6fb4109730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:15.641 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.641+0000 7f6fbc25c640 1 --2- 172.21.15.72:0/3288088076 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fb410c2c0 0x7f6fb4108d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:15.641 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.641+0000 7f6fbc25c640 1 -- 172.21.15.72:0/3288088076 >> 172.21.15.72:0/3288088076 conn(0x7f6fb406cc40 msgr2=0x7f6fb406d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:15.646 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.647+0000 7f6fbc25c640 1 -- 172.21.15.72:0/3288088076 shutdown_connections 2024-03-22T01:46:15.646 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.647+0000 7f6fbc25c640 1 -- 172.21.15.72:0/3288088076 wait complete. 2024-03-22T01:46:15.650 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.650+0000 7f6fbc25c640 1 Processor -- start 2024-03-22T01:46:15.652 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.653+0000 7f6fbc25c640 1 -- start start 2024-03-22T01:46:15.652 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.653+0000 7f6fbc25c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fb41092b0 0x7f6fb41a8e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:15.652 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.653+0000 7f6fbc25c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fb410c2c0 0x7f6fb41a93a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:15.652 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.653+0000 7f6fbc25c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6fb4116c20 0x7f6fb410f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:15.652 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.653+0000 7f6fbc25c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f6fb411c950 con 0x7f6fb4116c20 2024-03-22T01:46:15.652 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.653+0000 7f6fbc25c640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f6fb411c7d0 con 0x7f6fb410c2c0 2024-03-22T01:46:15.653 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.653+0000 7f6fbc25c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6fb411cad0 con 0x7f6fb41092b0 2024-03-22T01:46:15.653 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.653+0000 7f6fb9ff8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fb41092b0 0x7f6fb41a8e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:15.653 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.653+0000 7f6fb9ff8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fb41092b0 0x7f6fb41a8e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.72:39738/0 (socket says 172.21.15.72:39738) 2024-03-22T01:46:15.654 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.653+0000 7f6fb9ff8640 1 -- 172.21.15.72:0/3827652623 learned_addr learned my addr 172.21.15.72:0/3827652623 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:15.654 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.654+0000 7f6fba7f9640 1 --2- 172.21.15.72:0/3827652623 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6fb4116c20 0x7f6fb410f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:15.654 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.654+0000 7f6fb97f7640 1 --2- 172.21.15.72:0/3827652623 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fb410c2c0 0x7f6fb41a93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:15.654 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.654+0000 7f6fb9ff8640 1 -- 172.21.15.72:0/3827652623 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fb410c2c0 msgr2=0x7f6fb41a93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:15.654 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.654+0000 7f6fb9ff8640 1 --2- 172.21.15.72:0/3827652623 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fb410c2c0 0x7f6fb41a93a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:15.654 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.654+0000 7f6fb9ff8640 1 -- 172.21.15.72:0/3827652623 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6fb4116c20 msgr2=0x7f6fb410f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:15.654 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.654+0000 7f6fb9ff8640 1 --2- 172.21.15.72:0/3827652623 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6fb4116c20 0x7f6fb410f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:15.654 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.654+0000 7f6fb9ff8640 1 -- 172.21.15.72:0/3827652623 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fb410ff40 con 0x7f6fb41092b0 2024-03-22T01:46:15.655 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.654+0000 7f6fb9ff8640 1 --2- 172.21.15.72:0/3827652623 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fb41092b0 0x7f6fb41a8e60 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c00e9f0 tx=0x7f6f9c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:15.655 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.654+0000 7f6fb2ffd640 1 -- 172.21.15.72:0/3827652623 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f9c00cd80 con 0x7f6fb41092b0 2024-03-22T01:46:15.655 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.654+0000 7f6fb2ffd640 1 -- 172.21.15.72:0/3827652623 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f9c010910 con 0x7f6fb41092b0 2024-03-22T01:46:15.655 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.654+0000 7f6fb2ffd640 1 -- 172.21.15.72:0/3827652623 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f9c0188b0 con 0x7f6fb41092b0 2024-03-22T01:46:15.655 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.654+0000 7f6fbc25c640 1 -- 172.21.15.72:0/3827652623 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fb4110230 con 0x7f6fb41092b0 2024-03-22T01:46:15.655 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.655+0000 7f6fbc25c640 1 -- 172.21.15.72:0/3827652623 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fb4115a20 con 0x7f6fb41092b0 2024-03-22T01:46:15.656 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.658+0000 7f6fb2ffd640 1 -- 172.21.15.72:0/3827652623 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 137985+0+0 (secure 0 0 0) 0x7f6f9c010430 con 0x7f6fb41092b0 2024-03-22T01:46:15.658 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.659+0000 7f6fbee40740 1 Processor -- start 2024-03-22T01:46:15.659 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.660+0000 7f6fbee40740 1 -- start start 2024-03-22T01:46:15.660 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.660+0000 7f6fbee40740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dd813612d0 0x55dd813616b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:15.660 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.660+0000 7f6fbee40740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd81361bf0 0x55dd8136a480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:15.660 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.660+0000 7f6fbee40740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd8136a9c0 0x55dd8136cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:15.660 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.660+0000 7f6fbee40740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55dd812c5040 con 0x55dd813612d0 2024-03-22T01:46:15.660 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.660+0000 7f6fbee40740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55dd8126d9e0 con 0x55dd81361bf0 2024-03-22T01:46:15.660 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.660+0000 7f6fbee40740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55dd812c0ff0 con 0x55dd8136a9c0 2024-03-22T01:46:15.660 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.660+0000 7f6fb2ffd640 1 --2- 172.21.15.72:0/3827652623 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f6fa00a3180 0x7f6fa00a5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:15.661 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.660+0000 7f6fb97f7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd8136a9c0 0x55dd8136cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:15.661 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.661+0000 7f6fb97f7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd8136a9c0 0x55dd8136cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.72:39742/0 (socket says 172.21.15.72:39742) 2024-03-22T01:46:15.661 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.661+0000 7f6fb97f7640 1 -- 172.21.15.72:0/3648707699 learned_addr learned my addr 172.21.15.72:0/3648707699 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:15.661 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.661+0000 7f6fb9ff8640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd81361bf0 0x55dd8136a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:15.662 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.661+0000 7f6fba7f9640 1 --2- 172.21.15.72:0/3648707699 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dd813612d0 0x55dd813616b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:15.662 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.661+0000 7f6fba7f9640 1 --2- 172.21.15.72:0/3827652623 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f6fa00a3180 0x7f6fa00a5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:15.662 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.661+0000 7f6fb97f7640 1 -- 172.21.15.72:0/3648707699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd81361bf0 msgr2=0x55dd8136a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:15.662 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.661+0000 7f6fb97f7640 1 --2- 172.21.15.72:0/3648707699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd81361bf0 0x55dd8136a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:15.662 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.661+0000 7f6fb97f7640 1 -- 172.21.15.72:0/3648707699 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dd813612d0 msgr2=0x55dd813616b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:15.662 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.661+0000 7f6fb97f7640 1 --2- 172.21.15.72:0/3648707699 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dd813612d0 0x55dd813616b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:15.662 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.661+0000 7f6fb97f7640 1 -- 172.21.15.72:0/3648707699 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd80fd9210 con 0x55dd8136a9c0 2024-03-22T01:46:15.663 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.661+0000 7f6fb2ffd640 1 -- 172.21.15.72:0/3827652623 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f6f9c0cff40 con 0x7f6fb41092b0 2024-03-22T01:46:15.663 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.662+0000 7f6fb97f7640 1 --2- 172.21.15.72:0/3648707699 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd8136a9c0 0x55dd8136cdb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6fa400bcf0 tx=0x7f6fa400f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=19957fb46590a392 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:15.663 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.662+0000 7f6fb9ff8640 1 --2- 172.21.15.72:0/3648707699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd81361bf0 0x55dd8136a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:46:15.663 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.662+0000 7f6fba7f9640 1 --2- 172.21.15.72:0/3827652623 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f6fa00a3180 0x7f6fa00a5640 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6fac02c4e0 tx=0x7f6fac005fd0 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:15.666 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.666+0000 7f6fabfff640 1 -- 172.21.15.72:0/3648707699 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fa4015070 con 0x55dd8136a9c0 2024-03-22T01:46:15.667 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.666+0000 7f6fabfff640 1 -- 172.21.15.72:0/3648707699 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fa4005500 con 0x55dd8136a9c0 2024-03-22T01:46:15.667 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.667+0000 7f6fbee40740 1 -- 172.21.15.72:0/3648707699 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd8136a9c0 msgr2=0x55dd8136cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:15.667 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.667+0000 7f6fbee40740 1 --2- 172.21.15.72:0/3648707699 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd8136a9c0 0x55dd8136cdb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6fa400bcf0 tx=0x7f6fa400f730 comp rx=0 tx=0).stop 2024-03-22T01:46:15.668 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.669+0000 7f6fbee40740 1 -- 172.21.15.72:0/3648707699 shutdown_connections 2024-03-22T01:46:15.668 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.669+0000 7f6fbee40740 1 --2- 172.21.15.72:0/3648707699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd81361bf0 0x55dd8136a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:15.668 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.669+0000 7f6fbee40740 1 --2- 172.21.15.72:0/3648707699 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd8136a9c0 0x55dd8136cdb0 secure :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6fa400bcf0 tx=0x7f6fa400f730 comp rx=0 tx=0).stop 2024-03-22T01:46:15.668 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.669+0000 7f6fbee40740 1 --2- 172.21.15.72:0/3648707699 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dd813612d0 0x55dd813616b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:15.668 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.669+0000 7f6fbee40740 1 -- 172.21.15.72:0/3648707699 >> 172.21.15.72:0/3648707699 conn(0x55dd8135bbf0 msgr2=0x55dd8135bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:15.675 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.676+0000 7f6fbee40740 1 -- 172.21.15.72:0/3648707699 shutdown_connections 2024-03-22T01:46:15.676 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.676+0000 7f6fbee40740 1 -- 172.21.15.72:0/3648707699 wait complete. 2024-03-22T01:46:15.676 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.677+0000 7f6fbee40740 1 -- start start 2024-03-22T01:46:15.676 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.677+0000 7f6fbee40740 1 Processor -- start 2024-03-22T01:46:15.677 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.677+0000 7f6fbee40740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd813612d0 0x55dd8136a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:15.677 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.677+0000 7f6fbee40740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dd81361bf0 0x55dd813676a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:15.677 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.677+0000 7f6fbee40740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd81367c30 0x55dd81368090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:15.677 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.677+0000 7f6fbee40740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55dd8136d5f0 con 0x55dd81361bf0 2024-03-22T01:46:15.677 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.677+0000 7f6fbee40740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55dd81280db0 con 0x55dd813612d0 2024-03-22T01:46:15.677 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.677+0000 7f6fbee40740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55dd8136d770 con 0x55dd81367c30 2024-03-22T01:46:15.677 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.677+0000 7f6fbee40740 10 client.0 _ll_register_callbacks cb 0x55dd80fd8a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:46:15.678 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.678+0000 7f6fba7f9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd81367c30 0x55dd81368090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:15.678 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.678+0000 7f6fb9ff8640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd813612d0 0x55dd8136a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:15.678 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.678+0000 7f6fb97f7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dd81361bf0 0x55dd813676a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:15.678 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.678+0000 7f6fba7f9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd81367c30 0x55dd81368090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.72:39746/0 (socket says 172.21.15.72:39746) 2024-03-22T01:46:15.678 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.678+0000 7f6fba7f9640 1 -- 172.21.15.72:0/1414342988 learned_addr learned my addr 172.21.15.72:0/1414342988 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:15.678 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.678+0000 7f6fb9ff8640 1 -- 172.21.15.72:0/1414342988 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd81367c30 msgr2=0x55dd81368090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:15.678 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.678+0000 7f6fb9ff8640 1 --2- 172.21.15.72:0/1414342988 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd81367c30 0x55dd81368090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:15.678 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.678+0000 7f6fb9ff8640 1 -- 172.21.15.72:0/1414342988 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dd81361bf0 msgr2=0x55dd813676a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:15.678 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.678+0000 7f6fb9ff8640 1 --2- 172.21.15.72:0/1414342988 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dd81361bf0 0x55dd813676a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:15.678 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.678+0000 7f6fb9ff8640 1 -- 172.21.15.72:0/1414342988 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd810a5620 con 0x55dd813612d0 2024-03-22T01:46:15.678 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.678+0000 7f6fba7f9640 1 --2- 172.21.15.72:0/1414342988 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd81367c30 0x55dd81368090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:46:15.678 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.678+0000 7f6fb97f7640 1 --2- 172.21.15.72:0/1414342988 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dd81361bf0 0x55dd813676a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:46:15.679 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.679+0000 7f6fb9ff8640 1 --2- 172.21.15.72:0/1414342988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd813612d0 0x55dd8136a3b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c0cd280 tx=0x7f6f9c009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9028f7dba2af575d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:15.679 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.679+0000 7f6fa97fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-03-22T01:46:15.679 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.679+0000 7f6fa97fa640 1 -- 172.21.15.72:0/1414342988 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f9c0a4930 con 0x55dd813612d0 2024-03-22T01:46:15.680 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.679+0000 7f6fa97fa640 1 -- 172.21.15.72:0/1414342988 <== mon.2 v2:172.21.15.89:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f6f9c098590 con 0x55dd813612d0 2024-03-22T01:46:15.680 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.680+0000 7f6fbee40740 1 -- 172.21.15.72:0/1414342988 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6fa4005500 con 0x55dd813612d0 2024-03-22T01:46:15.685 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.686+0000 7f6fa97fa640 1 -- 172.21.15.72:0/1414342988 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7f6f9c09bf20 con 0x55dd813612d0 2024-03-22T01:46:15.687 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.688+0000 7f6fbee40740 10 client.26081 fetch_fsmap learned FSMap version 103 2024-03-22T01:46:15.687 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.688+0000 7f6fbee40740 1 -- 172.21.15.72:0/1414342988 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=4+}) -- 0x55dd81379af0 con 0x55dd813612d0 2024-03-22T01:46:15.692 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.690+0000 7f6fa97fa640 1 -- 172.21.15.72:0/1414342988 <== mon.2 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fa4005500 con 0x55dd813612d0 2024-03-22T01:46:15.693 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.690+0000 7f6fa97fa640 1 -- 172.21.15.72:0/1414342988 <== mon.2 v2:172.21.15.89:3300/0 5 ==== fsmap(e 103) ==== 8330+0+0 (secure 0 0 0) 0x7f6f9c010ac0 con 0x55dd813612d0 2024-03-22T01:46:15.693 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.690+0000 7f6fbee40740 10 client.26081 fetch_fsmap finished waiting for FSMap version 103 2024-03-22T01:46:15.693 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.690+0000 7f6fbee40740 10 client.26081 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T01:46:15.693 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.690+0000 7f6fbee40740 20 client.26081 populate_metadata read hostname 'smithi072' 2024-03-22T01:46:15.693 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.690+0000 7f6fbee40740 1 --2- 172.21.15.72:0/1414342988 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55dd8137b3c0 0x55dd8137d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:15.693 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.690+0000 7f6fbee40740 4 client.26081 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T01:46:15.693 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.690+0000 7f6fbee40740 1 -- 172.21.15.72:0/1414342988 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dd812d2580 con 0x55dd8137b3c0 2024-03-22T01:46:15.693 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.691+0000 7f6fb97f7640 1 --2- 172.21.15.72:0/1414342988 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55dd8137b3c0 0x55dd8137d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:15.693 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.694+0000 7f6fb97f7640 1 --2- 172.21.15.72:0/1414342988 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55dd8137b3c0 0x55dd8137d840 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db63fa0d13dea1ef server_cookie=f4e6e4a4f1c2540a in_seq=0 out_seq=0 2024-03-22T01:46:15.696 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.696+0000 7f6fa97fa640 10 client.26081 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:46:15.699 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.700+0000 7f6fa97fa640 1 -- 172.21.15.72:0/1414342988 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55dd812d2580 con 0x55dd8137b3c0 2024-03-22T01:46:15.699 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.700+0000 7f6fa97fa640 10 client.26081 handle_command_reply: tid=0 2024-03-22T01:46:15.699 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.700+0000 7f6fbee40740 1 client.26081 shutdown 2024-03-22T01:46:15.701 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.702+0000 7f6fbee40740 1 -- 172.21.15.72:0/1414342988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd813612d0 msgr2=0x55dd8136a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:15.701 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.702+0000 7f6fbee40740 1 --2- 172.21.15.72:0/1414342988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd813612d0 0x55dd8136a3b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c0cd280 tx=0x7f6f9c009310 comp rx=0 tx=0).stop 2024-03-22T01:46:15.701 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.702+0000 7f6fbee40740 1 -- 172.21.15.72:0/1414342988 shutdown_connections 2024-03-22T01:46:15.701 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.702+0000 7f6fbee40740 1 --2- 172.21.15.72:0/1414342988 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55dd8137b3c0 0x55dd8137d840 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:15.702 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.702+0000 7f6fbee40740 1 --2- 172.21.15.72:0/1414342988 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd81367c30 0x55dd81368090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:15.702 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.702+0000 7f6fbee40740 1 --2- 172.21.15.72:0/1414342988 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dd81361bf0 0x55dd813676a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:15.702 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.702+0000 7f6fbee40740 1 --2- 172.21.15.72:0/1414342988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd813612d0 0x55dd8136a3b0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:15.702 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.702+0000 7f6fbee40740 1 -- 172.21.15.72:0/1414342988 >> 172.21.15.72:0/1414342988 conn(0x55dd8135bbf0 msgr2=0x55dd8135d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:15.702 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.702+0000 7f6fa97fa640 0 client.26081 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:46:15.702 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.702+0000 7f6fbee40740 1 -- 172.21.15.72:0/1414342988 shutdown_connections 2024-03-22T01:46:15.702 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.702+0000 7f6fbee40740 1 -- 172.21.15.72:0/1414342988 wait complete. 2024-03-22T01:46:15.702 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.703+0000 7f6fbee40740 20 client.26081 trim_cache size 0 max 16384 2024-03-22T01:46:15.741 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.742+0000 7f6fbee40740 1 Processor -- start 2024-03-22T01:46:15.741 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.742+0000 7f6fbee40740 1 -- start start 2024-03-22T01:46:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.742+0000 7f6fbee40740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd813612d0 0x55dd81376440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.742+0000 7f6fbee40740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd81361bf0 0x55dd81383150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.742+0000 7f6fbee40740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dd81367c30 0x55dd81369ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.743+0000 7f6fbee40740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f6f9400e060 con 0x55dd81367c30 2024-03-22T01:46:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.743+0000 7f6fbee40740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f6f940108e0 con 0x55dd813612d0 2024-03-22T01:46:15.743 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.743+0000 7f6fbee40740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6f9400e6d0 con 0x55dd81361bf0 2024-03-22T01:46:15.743 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.743+0000 7f6fba7f9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dd81367c30 0x55dd81369ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:15.744 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.744+0000 7f6fba7f9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dd81367c30 0x55dd81369ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:35526/0 (socket says 172.21.15.72:35526) 2024-03-22T01:46:15.744 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.744+0000 7f6fba7f9640 1 -- 172.21.15.72:0/305716703 learned_addr learned my addr 172.21.15.72:0/305716703 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:15.745 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.744+0000 7f6fb9ff8640 1 --2- 172.21.15.72:0/305716703 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd813612d0 0x55dd81376440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:15.747 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.748+0000 7f6fba7f9640 1 -- 172.21.15.72:0/305716703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd81361bf0 msgr2=0x55dd81383150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:15.749 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.747+0000 7f6fb97f7640 1 --2- 172.21.15.72:0/305716703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd81361bf0 0x55dd81383150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:15.750 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.750+0000 7f6fba7f9640 1 --2- 172.21.15.72:0/305716703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd81361bf0 0x55dd81383150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:15.750 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.750+0000 7f6fba7f9640 1 -- 172.21.15.72:0/305716703 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd813612d0 msgr2=0x55dd81376440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:15.750 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.750+0000 7f6fba7f9640 1 --2- 172.21.15.72:0/305716703 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd813612d0 0x55dd81376440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:15.750 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.750+0000 7f6fba7f9640 1 -- 172.21.15.72:0/305716703 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd81361a20 con 0x55dd81367c30 2024-03-22T01:46:15.750 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.750+0000 7f6fb97f7640 1 --2- 172.21.15.72:0/305716703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd81361bf0 0x55dd81383150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T01:46:15.750 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.750+0000 7f6fb9ff8640 1 --2- 172.21.15.72:0/305716703 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd813612d0 0x55dd81376440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:46:15.753 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.750+0000 7f6fba7f9640 1 --2- 172.21.15.72:0/305716703 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dd81367c30 0x55dd81369ee0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f6fac009be0 tx=0x7f6fac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=880a14797b666eb5 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:15.753 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.753+0000 7f6fab7fe640 1 -- 172.21.15.72:0/305716703 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fac046040 con 0x55dd81367c30 2024-03-22T01:46:15.753 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.753+0000 7f6fab7fe640 1 -- 172.21.15.72:0/305716703 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fac02d590 con 0x55dd81367c30 2024-03-22T01:46:15.753 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.753+0000 7f6fab7fe640 1 -- 172.21.15.72:0/305716703 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fac033820 con 0x55dd81367c30 2024-03-22T01:46:15.755 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.755+0000 7f6fbee40740 1 -- 172.21.15.72:0/305716703 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dd81367c30 msgr2=0x55dd81369ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:15.755 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.755+0000 7f6fbee40740 1 --2- 172.21.15.72:0/305716703 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dd81367c30 0x55dd81369ee0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f6fac009be0 tx=0x7f6fac004a40 comp rx=0 tx=0).stop 2024-03-22T01:46:15.757 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.758+0000 7f6fbee40740 1 -- 172.21.15.72:0/305716703 shutdown_connections 2024-03-22T01:46:15.758 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.758+0000 7f6fbee40740 1 --2- 172.21.15.72:0/305716703 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dd81367c30 0x55dd81369ee0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:15.758 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.758+0000 7f6fbee40740 1 --2- 172.21.15.72:0/305716703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd81361bf0 0x55dd81383150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:15.758 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.758+0000 7f6fbee40740 1 --2- 172.21.15.72:0/305716703 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd813612d0 0x55dd81376440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:15.758 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.758+0000 7f6fbee40740 1 -- 172.21.15.72:0/305716703 >> 172.21.15.72:0/305716703 conn(0x55dd8135bbf0 msgr2=0x55dd8137d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:15.765 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.766+0000 7f6fbee40740 1 -- 172.21.15.72:0/305716703 shutdown_connections 2024-03-22T01:46:15.765 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.766+0000 7f6fbee40740 1 -- 172.21.15.72:0/305716703 wait complete. 2024-03-22T01:46:15.765 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.766+0000 7f6fbee40740 1 -- start start 2024-03-22T01:46:15.765 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.766+0000 7f6fbee40740 1 Processor -- start 2024-03-22T01:46:15.766 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.767+0000 7f6fbee40740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd813612d0 0x55dd81364d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:15.766 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.767+0000 7f6fbee40740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd81361bf0 0x55dd81365270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:15.766 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.767+0000 7f6fbee40740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dd81367c30 0x55dd81363910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:15.766 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.767+0000 7f6fbee40740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f6f9400eda0 con 0x55dd81367c30 2024-03-22T01:46:15.766 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.767+0000 7f6fbee40740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f6f94010210 con 0x55dd81361bf0 2024-03-22T01:46:15.766 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.767+0000 7f6fbee40740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6f9400f470 con 0x55dd813612d0 2024-03-22T01:46:15.766 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.767+0000 7f6fbee40740 10 client.0 _ll_register_callbacks cb 0x7f6f9c09b880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:46:15.767 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.768+0000 7f6fba7f9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dd81367c30 0x55dd81363910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:15.767 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.768+0000 7f6fba7f9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dd81367c30 0x55dd81363910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:35540/0 (socket says 172.21.15.72:35540) 2024-03-22T01:46:15.767 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.768+0000 7f6fba7f9640 1 -- 172.21.15.72:0/675738074 learned_addr learned my addr 172.21.15.72:0/675738074 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:15.769 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.770+0000 7f6fba7f9640 1 -- 172.21.15.72:0/675738074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd813612d0 msgr2=0x55dd81364d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:15.769 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.770+0000 7f6fba7f9640 1 --2- 172.21.15.72:0/675738074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd813612d0 0x55dd81364d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:15.769 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.770+0000 7f6fba7f9640 1 -- 172.21.15.72:0/675738074 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd81361bf0 msgr2=0x55dd81365270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:15.769 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.770+0000 7f6fba7f9640 1 --2- 172.21.15.72:0/675738074 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd81361bf0 0x55dd81365270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:15.769 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.770+0000 7f6fba7f9640 1 -- 172.21.15.72:0/675738074 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd81373250 con 0x55dd81367c30 2024-03-22T01:46:15.770 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.770+0000 7f6fba7f9640 1 --2- 172.21.15.72:0/675738074 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dd81367c30 0x55dd81363910 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f6fac02bfe0 tx=0x7f6fac009280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e209f11b6d443ccc server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:15.772 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.771+0000 7f6fa97fa640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:46:15.772 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.771+0000 7f6fa97fa640 1 -- 172.21.15.72:0/675738074 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fac046050 con 0x55dd81367c30 2024-03-22T01:46:15.772 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.771+0000 7f6fa97fa640 1 -- 172.21.15.72:0/675738074 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f6fac003f30 con 0x55dd81367c30 2024-03-22T01:46:15.772 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.773+0000 7f6fbee40740 1 -- 172.21.15.72:0/675738074 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6fac02d590 con 0x55dd81367c30 2024-03-22T01:46:15.777 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.778+0000 7f6fa97fa640 1 -- 172.21.15.72:0/675738074 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7f6fac004130 con 0x55dd81367c30 2024-03-22T01:46:15.780 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.781+0000 7f6fbee40740 10 client.16251 fetch_fsmap learned FSMap version 103 2024-03-22T01:46:15.780 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.781+0000 7f6fbee40740 1 -- 172.21.15.72:0/675738074 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=4+}) -- 0x55dd813e0d60 con 0x55dd81367c30 2024-03-22T01:46:15.781 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.782+0000 7f6fa97fa640 1 -- 172.21.15.72:0/675738074 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fac02d590 con 0x55dd81367c30 2024-03-22T01:46:15.782 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.782+0000 7f6fa97fa640 1 -- 172.21.15.72:0/675738074 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 103) ==== 8330+0+0 (secure 0 0 0) 0x7f6fac04d460 con 0x55dd81367c30 2024-03-22T01:46:15.782 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.783+0000 7f6fbee40740 10 client.16251 fetch_fsmap finished waiting for FSMap version 103 2024-03-22T01:46:15.782 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.783+0000 7f6fbee40740 10 client.16251 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T01:46:15.782 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.783+0000 7f6fbee40740 20 client.16251 populate_metadata read hostname 'smithi072' 2024-03-22T01:46:15.782 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.783+0000 7f6fbee40740 1 --2- 172.21.15.72:0/675738074 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55dd813e1400 0x55dd813e3860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:15.782 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.783+0000 7f6fbee40740 4 client.16251 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T01:46:15.782 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.783+0000 7f6fbee40740 1 -- 172.21.15.72:0/675738074 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55dd812cc8d0 con 0x55dd813e1400 2024-03-22T01:46:15.788 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.789+0000 7f6fb9ff8640 1 --2- 172.21.15.72:0/675738074 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55dd813e1400 0x55dd813e3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:15.794 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.795+0000 7f6fb9ff8640 1 --2- 172.21.15.72:0/675738074 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55dd813e1400 0x55dd813e3860 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7accd9d752b20500 server_cookie=18487ad79bd0235a in_seq=0 out_seq=0 2024-03-22T01:46:15.795 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:15.796+0000 7f6fa97fa640 10 client.16251 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:46:16.038 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:16.038+0000 7f6fa97fa640 1 -- 172.21.15.72:0/675738074 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55dd812cc8d0 con 0x55dd813e1400 2024-03-22T01:46:16.038 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:16.038+0000 7f6fa97fa640 10 client.16251 handle_command_reply: tid=0 2024-03-22T01:46:16.038 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:16.038+0000 7f6fbee40740 1 client.16251 shutdown 2024-03-22T01:46:16.038 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:16.038+0000 7f6fbee40740 1 -- 172.21.15.72:0/675738074 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dd81367c30 msgr2=0x55dd81363910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:16.038 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:16.038+0000 7f6fbee40740 1 --2- 172.21.15.72:0/675738074 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dd81367c30 0x55dd81363910 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f6fac02bfe0 tx=0x7f6fac009280 comp rx=0 tx=0).stop 2024-03-22T01:46:16.038 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:16.039+0000 7f6fbee40740 1 -- 172.21.15.72:0/675738074 shutdown_connections 2024-03-22T01:46:16.038 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:16.039+0000 7f6fbee40740 1 --2- 172.21.15.72:0/675738074 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55dd813e1400 0x55dd813e3860 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:16.038 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:16.039+0000 7f6fbee40740 1 --2- 172.21.15.72:0/675738074 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd81361bf0 0x55dd81365270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:16.039 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:16.039+0000 7f6fbee40740 1 --2- 172.21.15.72:0/675738074 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dd81367c30 0x55dd81363910 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:16.039 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:16.039+0000 7f6fbee40740 1 --2- 172.21.15.72:0/675738074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dd813612d0 0x55dd81364d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:16.039 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:16.039+0000 7f6fbee40740 1 -- 172.21.15.72:0/675738074 >> 172.21.15.72:0/675738074 conn(0x55dd8135bbf0 msgr2=0x55dd81362420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:16.039 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:16.039+0000 7f6fa97fa640 0 client.16251 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:46:16.039 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:16.039+0000 7f6fbee40740 1 -- 172.21.15.72:0/675738074 shutdown_connections 2024-03-22T01:46:16.039 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:16.039+0000 7f6fbee40740 1 -- 172.21.15.72:0/675738074 wait complete. 2024-03-22T01:46:16.039 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:16.039+0000 7f6fbee40740 20 client.16251 trim_cache size 0 max 16384 2024-03-22T01:46:16.039 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout:{ 2024-03-22T01:46:16.039 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout: "message": "", 2024-03-22T01:46:16.040 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout: "return_code": 0 2024-03-22T01:46:16.040 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout:} 2024-03-22T01:46:16.040 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:16.040+0000 7f6fbc25c640 1 -- 172.21.15.72:0/3827652623 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f6fa00a3180 msgr2=0x7f6fa00a5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:16.041 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:16.040+0000 7f6fbc25c640 1 --2- 172.21.15.72:0/3827652623 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f6fa00a3180 0x7f6fa00a5640 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6fac02c4e0 tx=0x7f6fac005fd0 comp rx=0 tx=0).stop 2024-03-22T01:46:16.041 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:16.040+0000 7f6fbc25c640 1 -- 172.21.15.72:0/3827652623 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fb41092b0 msgr2=0x7f6fb41a8e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:16.041 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:16.040+0000 7f6fbc25c640 1 --2- 172.21.15.72:0/3827652623 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fb41092b0 0x7f6fb41a8e60 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c00e9f0 tx=0x7f6f9c00eeb0 comp rx=0 tx=0).stop 2024-03-22T01:46:16.041 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:16.040+0000 7f6fbc25c640 1 -- 172.21.15.72:0/3827652623 shutdown_connections 2024-03-22T01:46:16.041 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:16.040+0000 7f6fbc25c640 1 --2- 172.21.15.72:0/3827652623 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f6fa00a3180 0x7f6fa00a5640 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:16.041 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:16.040+0000 7f6fbc25c640 1 --2- 172.21.15.72:0/3827652623 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fb410c2c0 0x7f6fb41a93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:16.041 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:16.040+0000 7f6fbc25c640 1 --2- 172.21.15.72:0/3827652623 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6fb4116c20 0x7f6fb410f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:16.041 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:16.040+0000 7f6fbc25c640 1 --2- 172.21.15.72:0/3827652623 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fb41092b0 0x7f6fb41a8e60 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:16.041 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:16.040+0000 7f6fbc25c640 1 -- 172.21.15.72:0/3827652623 >> 172.21.15.72:0/3827652623 conn(0x7f6fb406cc40 msgr2=0x7f6fb4117000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:16.041 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:16.040+0000 7f6fbc25c640 1 -- 172.21.15.72:0/3827652623 shutdown_connections 2024-03-22T01:46:16.041 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:16.040+0000 7f6fbc25c640 1 -- 172.21.15.72:0/3827652623 wait complete. 2024-03-22T01:46:16.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:46:16 smithi089 ceph-mon[35770]: pgmap v291: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 70 KiB/s wr, 53 op/s 2024-03-22T01:46:16.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:46:16 smithi072 ceph-mon[32414]: pgmap v291: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 70 KiB/s wr, 53 op/s 2024-03-22T01:46:16.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:46:16 smithi098 ceph-mon[36468]: pgmap v291: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 70 KiB/s wr, 53 op/s 2024-03-22T01:46:18.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:46:18 smithi089 ceph-mon[35770]: pgmap v292: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 226 KiB/s wr, 108 op/s 2024-03-22T01:46:18.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:46:18 smithi072 ceph-mon[32414]: pgmap v292: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 226 KiB/s wr, 108 op/s 2024-03-22T01:46:18.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:46:18 smithi098 ceph-mon[36468]: pgmap v292: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 226 KiB/s wr, 108 op/s 2024-03-22T01:46:20.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:46:20 smithi089 ceph-mon[35770]: pgmap v293: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 216 KiB/s wr, 108 op/s 2024-03-22T01:46:20.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:46:20 smithi072 ceph-mon[32414]: pgmap v293: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 216 KiB/s wr, 108 op/s 2024-03-22T01:46:20.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:46:20 smithi098 ceph-mon[36468]: pgmap v293: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 216 KiB/s wr, 108 op/s 2024-03-22T01:46:22.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:46:22 smithi089 ceph-mon[35770]: pgmap v294: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 244 KiB/s wr, 189 op/s 2024-03-22T01:46:22.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:46:22 smithi072 ceph-mon[32414]: pgmap v294: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 244 KiB/s wr, 189 op/s 2024-03-22T01:46:22.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:46:22 smithi098 ceph-mon[36468]: pgmap v294: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 244 KiB/s wr, 189 op/s 2024-03-22T01:46:24.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:46:24 smithi089 ceph-mon[35770]: pgmap v295: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 254 KiB/s wr, 229 op/s 2024-03-22T01:46:24.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:46:24 smithi072 ceph-mon[32414]: pgmap v295: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 254 KiB/s wr, 229 op/s 2024-03-22T01:46:24.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:46:24 smithi098 ceph-mon[36468]: pgmap v295: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 254 KiB/s wr, 229 op/s 2024-03-22T01:46:26.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:46:26 smithi089 ceph-mon[35770]: pgmap v296: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 199 KiB/s wr, 237 op/s 2024-03-22T01:46:26.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:46:26 smithi072 ceph-mon[32414]: pgmap v296: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 199 KiB/s wr, 237 op/s 2024-03-22T01:46:26.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:46:26 smithi098 ceph-mon[36468]: pgmap v296: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 199 KiB/s wr, 237 op/s 2024-03-22T01:46:28.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:46:28 smithi089 ceph-mon[35770]: pgmap v297: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 198 KiB/s wr, 305 op/s 2024-03-22T01:46:28.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:46:28 smithi072 ceph-mon[32414]: pgmap v297: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 198 KiB/s wr, 305 op/s 2024-03-22T01:46:28.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:46:28 smithi098 ceph-mon[36468]: pgmap v297: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 198 KiB/s wr, 305 op/s 2024-03-22T01:46:29.140 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.139+0000 7f31f8519640 1 -- 172.21.15.72:0/4131993376 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31f0101430 msgr2=0x7f31f010d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:29.140 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.139+0000 7f31f8519640 1 --2- 172.21.15.72:0/4131993376 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31f0101430 0x7f31f010d940 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f31e8009ca0 tx=0x7f31e802c0f0 comp rx=0 tx=0).stop 2024-03-22T01:46:29.141 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.139+0000 7f31f8519640 1 -- 172.21.15.72:0/4131993376 shutdown_connections 2024-03-22T01:46:29.141 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.139+0000 7f31f8519640 1 --2- 172.21.15.72:0/4131993376 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31f0101430 0x7f31f010d940 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:29.141 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.139+0000 7f31f8519640 1 --2- 172.21.15.72:0/4131993376 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31f0100a70 0x7f31f0100ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:29.141 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.139+0000 7f31f8519640 1 --2- 172.21.15.72:0/4131993376 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f31f0106a80 0x7f31f0106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:29.141 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.139+0000 7f31f8519640 1 -- 172.21.15.72:0/4131993376 >> 172.21.15.72:0/4131993376 conn(0x7f31f0077bb0 msgr2=0x7f31f0077fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:29.141 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.139+0000 7f31f8519640 1 -- 172.21.15.72:0/4131993376 shutdown_connections 2024-03-22T01:46:29.141 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.139+0000 7f31f8519640 1 -- 172.21.15.72:0/4131993376 wait complete. 2024-03-22T01:46:29.141 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.139+0000 7f31f8519640 1 Processor -- start 2024-03-22T01:46:29.141 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.139+0000 7f31f8519640 1 -- start start 2024-03-22T01:46:29.141 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.140+0000 7f31f8519640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f31f0100a70 0x7f31f0197310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:29.142 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.140+0000 7f31f8519640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31f0101430 0x7f31f0197850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:29.142 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.140+0000 7f31f8519640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31f0106a80 0x7f31f019bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:29.142 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.140+0000 7f31f8519640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f31f0110e90 con 0x7f31f0100a70 2024-03-22T01:46:29.142 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.140+0000 7f31f8519640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f31f0110d10 con 0x7f31f0101430 2024-03-22T01:46:29.142 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.140+0000 7f31f8519640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f31f0111010 con 0x7f31f0106a80 2024-03-22T01:46:29.142 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.140+0000 7f31f62b5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f31f0100a70 0x7f31f0197310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:29.142 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.140+0000 7f31f5ab4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31f0101430 0x7f31f0197850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:29.142 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.140+0000 7f31f62b5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f31f0100a70 0x7f31f0197310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59060/0 (socket says 172.21.15.72:59060) 2024-03-22T01:46:29.143 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.140+0000 7f31f62b5640 1 -- 172.21.15.72:0/290727413 learned_addr learned my addr 172.21.15.72:0/290727413 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:29.143 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.140+0000 7f31f6ab6640 1 --2- 172.21.15.72:0/290727413 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31f0106a80 0x7f31f019bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:29.143 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.140+0000 7f31f5ab4640 1 -- 172.21.15.72:0/290727413 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31f0106a80 msgr2=0x7f31f019bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:29.143 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.140+0000 7f31f5ab4640 1 --2- 172.21.15.72:0/290727413 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31f0106a80 0x7f31f019bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:29.143 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.140+0000 7f31f5ab4640 1 -- 172.21.15.72:0/290727413 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f31f0100a70 msgr2=0x7f31f0197310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:29.143 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.140+0000 7f31f5ab4640 1 --2- 172.21.15.72:0/290727413 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f31f0100a70 0x7f31f0197310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:29.143 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.140+0000 7f31f5ab4640 1 -- 172.21.15.72:0/290727413 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31f019c300 con 0x7f31f0101430 2024-03-22T01:46:29.143 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.140+0000 7f31f62b5640 1 --2- 172.21.15.72:0/290727413 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f31f0100a70 0x7f31f0197310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:46:29.144 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.140+0000 7f31f5ab4640 1 --2- 172.21.15.72:0/290727413 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31f0101430 0x7f31f0197850 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f31e000e9f0 tx=0x7f31e000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:29.144 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.140+0000 7f31e77fe640 1 -- 172.21.15.72:0/290727413 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31e000ccc0 con 0x7f31f0101430 2024-03-22T01:46:29.144 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.141+0000 7f31e77fe640 1 -- 172.21.15.72:0/290727413 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31e0005020 con 0x7f31f0101430 2024-03-22T01:46:29.144 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.141+0000 7f31f8519640 1 -- 172.21.15.72:0/290727413 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31f019c5f0 con 0x7f31f0101430 2024-03-22T01:46:29.144 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.141+0000 7f31e77fe640 1 -- 172.21.15.72:0/290727413 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31e00049e0 con 0x7f31f0101430 2024-03-22T01:46:29.144 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.141+0000 7f31f8519640 1 -- 172.21.15.72:0/290727413 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31f0069d10 con 0x7f31f0101430 2024-03-22T01:46:29.155 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.142+0000 7f31e77fe640 1 -- 172.21.15.72:0/290727413 <== mon.2 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 25) ==== 137985+0+0 (secure 0 0 0) 0x7f31e0005560 con 0x7f31f0101430 2024-03-22T01:46:29.155 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.143+0000 7f31e77fe640 1 --2- 172.21.15.72:0/290727413 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f31dc0a3080 0x7f31dc0a5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:29.155 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.143+0000 7f31fb1d7740 1 Processor -- start 2024-03-22T01:46:29.156 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.143+0000 7f31f62b5640 1 --2- 172.21.15.72:0/290727413 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f31dc0a3080 0x7f31dc0a5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:29.156 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.143+0000 7f31e77fe640 1 -- 172.21.15.72:0/290727413 <== mon.2 v2:172.21.15.89:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f31e00cfc00 con 0x7f31f0101430 2024-03-22T01:46:29.156 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.144+0000 7f31fb1d7740 1 -- start start 2024-03-22T01:46:29.156 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.144+0000 7f31fb1d7740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee70b792d0 0x55ee70b796b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:29.156 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.144+0000 7f31fb1d7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee70b79bf0 0x55ee70b82480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:29.156 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.144+0000 7f31fb1d7740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ee70b829c0 0x55ee70b84db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:29.156 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.144+0000 7f31fb1d7740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55ee70add040 con 0x55ee70b829c0 2024-03-22T01:46:29.156 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.144+0000 7f31fb1d7740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55ee70a859e0 con 0x55ee70b792d0 2024-03-22T01:46:29.156 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.144+0000 7f31fb1d7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ee70ad8ff0 con 0x55ee70b79bf0 2024-03-22T01:46:29.156 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.144+0000 7f31f62b5640 1 --2- 172.21.15.72:0/290727413 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f31dc0a3080 0x7f31dc0a5540 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f31ec0095f0 tx=0x7f31ec010040 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:29.156 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.144+0000 7f31f62b5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee70b792d0 0x55ee70b796b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:29.156 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.144+0000 7f31f62b5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee70b792d0 0x55ee70b796b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.72:53404/0 (socket says 172.21.15.72:53404) 2024-03-22T01:46:29.156 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.144+0000 7f31f62b5640 1 -- 172.21.15.72:0/2371341209 learned_addr learned my addr 172.21.15.72:0/2371341209 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:29.156 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.144+0000 7f31f6ab6640 1 --2- 172.21.15.72:0/2371341209 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ee70b829c0 0x55ee70b84db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:29.157 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.144+0000 7f31f5ab4640 1 --2- 172.21.15.72:0/2371341209 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee70b79bf0 0x55ee70b82480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:29.157 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.144+0000 7f31f62b5640 1 -- 172.21.15.72:0/2371341209 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee70b79bf0 msgr2=0x55ee70b82480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:29.157 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.144+0000 7f31f62b5640 1 --2- 172.21.15.72:0/2371341209 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee70b79bf0 0x55ee70b82480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:29.157 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.144+0000 7f31f62b5640 1 -- 172.21.15.72:0/2371341209 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ee70b829c0 msgr2=0x55ee70b84db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:29.157 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.144+0000 7f31f62b5640 1 --2- 172.21.15.72:0/2371341209 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ee70b829c0 0x55ee70b84db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:29.157 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.144+0000 7f31f62b5640 1 -- 172.21.15.72:0/2371341209 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee707f1210 con 0x55ee70b792d0 2024-03-22T01:46:29.157 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.144+0000 7f31f62b5640 1 --2- 172.21.15.72:0/2371341209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee70b792d0 0x55ee70b796b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f31ec0044d0 tx=0x7f31ec016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21a21b282cc1f75c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:29.157 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.144+0000 7f31e4bfe640 1 -- 172.21.15.72:0/2371341209 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31ec02a070 con 0x55ee70b792d0 2024-03-22T01:46:29.157 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.145+0000 7f31e4bfe640 1 -- 172.21.15.72:0/2371341209 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31ec021040 con 0x55ee70b792d0 2024-03-22T01:46:29.157 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.145+0000 7f31e4bfe640 1 -- 172.21.15.72:0/2371341209 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31ec01a960 con 0x55ee70b792d0 2024-03-22T01:46:29.157 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.145+0000 7f31fb1d7740 1 -- 172.21.15.72:0/2371341209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee70b792d0 msgr2=0x55ee70b796b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:29.157 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.145+0000 7f31fb1d7740 1 --2- 172.21.15.72:0/2371341209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee70b792d0 0x55ee70b796b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f31ec0044d0 tx=0x7f31ec016000 comp rx=0 tx=0).stop 2024-03-22T01:46:29.157 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.145+0000 7f31fb1d7740 1 -- 172.21.15.72:0/2371341209 shutdown_connections 2024-03-22T01:46:29.157 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.145+0000 7f31fb1d7740 1 --2- 172.21.15.72:0/2371341209 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ee70b829c0 0x55ee70b84db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:29.157 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.145+0000 7f31fb1d7740 1 --2- 172.21.15.72:0/2371341209 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee70b79bf0 0x55ee70b82480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:29.158 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.145+0000 7f31fb1d7740 1 --2- 172.21.15.72:0/2371341209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee70b792d0 0x55ee70b796b0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:29.158 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.145+0000 7f31fb1d7740 1 -- 172.21.15.72:0/2371341209 >> 172.21.15.72:0/2371341209 conn(0x55ee70b73bf0 msgr2=0x55ee70b73fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:29.158 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.145+0000 7f31fb1d7740 1 -- 172.21.15.72:0/2371341209 shutdown_connections 2024-03-22T01:46:29.158 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.145+0000 7f31fb1d7740 1 -- 172.21.15.72:0/2371341209 wait complete. 2024-03-22T01:46:29.158 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.146+0000 7f31fb1d7740 1 -- start start 2024-03-22T01:46:29.158 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.146+0000 7f31fb1d7740 1 Processor -- start 2024-03-22T01:46:29.158 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.146+0000 7f31fb1d7740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee70b792d0 0x55ee70b82380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:29.158 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.146+0000 7f31fb1d7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee70b79bf0 0x55ee70b7f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:29.158 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.146+0000 7f31fb1d7740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ee70b829c0 0x55ee70b7fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:29.158 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.146+0000 7f31fb1d7740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55ee70b855f0 con 0x55ee70b829c0 2024-03-22T01:46:29.158 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.146+0000 7f31fb1d7740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55ee70a98db0 con 0x55ee70b792d0 2024-03-22T01:46:29.158 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.146+0000 7f31fb1d7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ee70b85770 con 0x55ee70b79bf0 2024-03-22T01:46:29.158 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.146+0000 7f31fb1d7740 10 client.0 _ll_register_callbacks cb 0x55ee707f0a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:46:29.159 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.146+0000 7f31f6ab6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ee70b829c0 0x55ee70b7fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:29.159 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.146+0000 7f31f6ab6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ee70b829c0 0x55ee70b7fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59068/0 (socket says 172.21.15.72:59068) 2024-03-22T01:46:29.159 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.146+0000 7f31f6ab6640 1 -- 172.21.15.72:0/3757224474 learned_addr learned my addr 172.21.15.72:0/3757224474 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:29.161 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.146+0000 7f31f62b5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee70b792d0 0x55ee70b82380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:29.161 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.146+0000 7f31f5ab4640 1 --2- 172.21.15.72:0/3757224474 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee70b79bf0 0x55ee70b7f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:29.161 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.146+0000 7f31f6ab6640 1 -- 172.21.15.72:0/3757224474 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee70b79bf0 msgr2=0x55ee70b7f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:29.162 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.146+0000 7f31f6ab6640 1 --2- 172.21.15.72:0/3757224474 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee70b79bf0 0x55ee70b7f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:29.162 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.146+0000 7f31f6ab6640 1 -- 172.21.15.72:0/3757224474 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee70b792d0 msgr2=0x55ee70b82380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:29.162 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.146+0000 7f31f6ab6640 1 --2- 172.21.15.72:0/3757224474 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee70b792d0 0x55ee70b82380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:29.162 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.146+0000 7f31f6ab6640 1 -- 172.21.15.72:0/3757224474 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee708bd620 con 0x55ee70b829c0 2024-03-22T01:46:29.162 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.146+0000 7f31f5ab4640 1 --2- 172.21.15.72:0/3757224474 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee70b79bf0 0x55ee70b7f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:46:29.162 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.146+0000 7f31f62b5640 1 --2- 172.21.15.72:0/3757224474 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee70b792d0 0x55ee70b82380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:46:29.162 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.147+0000 7f31f6ab6640 1 --2- 172.21.15.72:0/3757224474 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ee70b829c0 0x55ee70b7fd20 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f31e8009430 tx=0x7f31e80324b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d220090ea1e871b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:29.162 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.147+0000 7f31d9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:46:29.162 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.147+0000 7f31d9ffb640 1 -- 172.21.15.72:0/3757224474 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31e8039040 con 0x55ee70b829c0 2024-03-22T01:46:29.162 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.147+0000 7f31d9ffb640 1 -- 172.21.15.72:0/3757224474 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f31e8032790 con 0x55ee70b829c0 2024-03-22T01:46:29.162 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.147+0000 7f31fb1d7740 1 -- 172.21.15.72:0/3757224474 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f31ec021040 con 0x55ee70b829c0 2024-03-22T01:46:29.163 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.147+0000 7f31d9ffb640 1 -- 172.21.15.72:0/3757224474 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7f31e8051050 con 0x55ee70b829c0 2024-03-22T01:46:29.163 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.147+0000 7f31fb1d7740 10 client.16263 fetch_fsmap learned FSMap version 103 2024-03-22T01:46:29.163 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.147+0000 7f31fb1d7740 1 -- 172.21.15.72:0/3757224474 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=4+}) -- 0x55ee70b91580 con 0x55ee70b829c0 2024-03-22T01:46:29.163 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.147+0000 7f31d9ffb640 1 -- 172.21.15.72:0/3757224474 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31ec021040 con 0x55ee70b829c0 2024-03-22T01:46:29.163 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.147+0000 7f31d9ffb640 1 -- 172.21.15.72:0/3757224474 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 103) ==== 8330+0+0 (secure 0 0 0) 0x7f31e8004370 con 0x55ee70b829c0 2024-03-22T01:46:29.163 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.147+0000 7f31fb1d7740 10 client.16263 fetch_fsmap finished waiting for FSMap version 103 2024-03-22T01:46:29.163 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.147+0000 7f31fb1d7740 10 client.16263 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T01:46:29.163 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.147+0000 7f31fb1d7740 20 client.16263 populate_metadata read hostname 'smithi072' 2024-03-22T01:46:29.163 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.147+0000 7f31fb1d7740 1 --2- 172.21.15.72:0/3757224474 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55ee70b92a90 0x55ee70b94e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:29.163 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.147+0000 7f31fb1d7740 4 client.16263 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T01:46:29.163 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.147+0000 7f31fb1d7740 1 -- 172.21.15.72:0/3757224474 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ee70aea580 con 0x55ee70b92a90 2024-03-22T01:46:29.163 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.148+0000 7f31f62b5640 1 --2- 172.21.15.72:0/3757224474 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55ee70b92a90 0x55ee70b94e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:29.163 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.148+0000 7f31f62b5640 1 --2- 172.21.15.72:0/3757224474 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55ee70b92a90 0x55ee70b94e80 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b8196509360ddb47 server_cookie=e50ca39aad63173e in_seq=0 out_seq=0 2024-03-22T01:46:29.163 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.148+0000 7f31d9ffb640 10 client.16263 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:46:29.164 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.148+0000 7f31d9ffb640 1 -- 172.21.15.72:0/3757224474 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ee70aea580 con 0x55ee70b92a90 2024-03-22T01:46:29.164 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.148+0000 7f31d9ffb640 10 client.16263 handle_command_reply: tid=0 2024-03-22T01:46:29.164 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.149+0000 7f31fb1d7740 1 client.16263 shutdown 2024-03-22T01:46:29.164 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.149+0000 7f31fb1d7740 1 -- 172.21.15.72:0/3757224474 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ee70b829c0 msgr2=0x55ee70b7fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:29.164 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.149+0000 7f31fb1d7740 1 --2- 172.21.15.72:0/3757224474 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ee70b829c0 0x55ee70b7fd20 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f31e8009430 tx=0x7f31e80324b0 comp rx=0 tx=0).stop 2024-03-22T01:46:29.164 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.149+0000 7f31fb1d7740 1 -- 172.21.15.72:0/3757224474 shutdown_connections 2024-03-22T01:46:29.164 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.149+0000 7f31fb1d7740 1 --2- 172.21.15.72:0/3757224474 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55ee70b92a90 0x55ee70b94e80 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:29.165 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.149+0000 7f31fb1d7740 1 --2- 172.21.15.72:0/3757224474 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ee70b829c0 0x55ee70b7fd20 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:29.165 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.149+0000 7f31fb1d7740 1 --2- 172.21.15.72:0/3757224474 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee70b79bf0 0x55ee70b7f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:29.165 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.149+0000 7f31fb1d7740 1 --2- 172.21.15.72:0/3757224474 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee70b792d0 0x55ee70b82380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:29.165 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.149+0000 7f31d9ffb640 0 client.16263 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:46:29.165 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.149+0000 7f31fb1d7740 1 -- 172.21.15.72:0/3757224474 >> 172.21.15.72:0/3757224474 conn(0x55ee70b73bf0 msgr2=0x55ee70b75940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:29.165 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.149+0000 7f31fb1d7740 1 -- 172.21.15.72:0/3757224474 shutdown_connections 2024-03-22T01:46:29.165 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.149+0000 7f31fb1d7740 1 -- 172.21.15.72:0/3757224474 wait complete. 2024-03-22T01:46:29.165 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.149+0000 7f31fb1d7740 20 client.16263 trim_cache size 0 max 16384 2024-03-22T01:46:29.171 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.164+0000 7f31fb1d7740 1 Processor -- start 2024-03-22T01:46:29.171 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.164+0000 7f31fb1d7740 1 -- start start 2024-03-22T01:46:29.172 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.164+0000 7f31fb1d7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee70b792d0 0x55ee70b8e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:29.172 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.164+0000 7f31fb1d7740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ee70b79bf0 0x55ee70b756e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:29.172 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.164+0000 7f31fb1d7740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee70b829c0 0x55ee70b9a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:29.172 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.164+0000 7f31fb1d7740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f31f01aae90 con 0x55ee70b79bf0 2024-03-22T01:46:29.172 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.164+0000 7f31fb1d7740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f31f01ad710 con 0x55ee70b829c0 2024-03-22T01:46:29.172 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.164+0000 7f31fb1d7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f31f01ab500 con 0x55ee70b792d0 2024-03-22T01:46:29.172 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.164+0000 7f31f5ab4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ee70b79bf0 0x55ee70b756e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:29.172 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.164+0000 7f31f5ab4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ee70b79bf0 0x55ee70b756e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59070/0 (socket says 172.21.15.72:59070) 2024-03-22T01:46:29.172 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.164+0000 7f31f5ab4640 1 -- 172.21.15.72:0/410151487 learned_addr learned my addr 172.21.15.72:0/410151487 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:29.172 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.164+0000 7f31f6ab6640 1 --2- 172.21.15.72:0/410151487 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee70b829c0 0x55ee70b9a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:29.172 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.164+0000 7f31f62b5640 1 --2- 172.21.15.72:0/410151487 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee70b792d0 0x55ee70b8e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:29.172 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.164+0000 7f31f5ab4640 1 -- 172.21.15.72:0/410151487 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee70b792d0 msgr2=0x55ee70b8e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:29.172 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.164+0000 7f31f5ab4640 1 --2- 172.21.15.72:0/410151487 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee70b792d0 0x55ee70b8e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:29.173 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.164+0000 7f31f5ab4640 1 -- 172.21.15.72:0/410151487 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee70b829c0 msgr2=0x55ee70b9a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:29.173 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.164+0000 7f31f5ab4640 1 --2- 172.21.15.72:0/410151487 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee70b829c0 0x55ee70b9a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:29.173 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.164+0000 7f31f5ab4640 1 -- 172.21.15.72:0/410151487 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee70b954e0 con 0x55ee70b79bf0 2024-03-22T01:46:29.173 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.164+0000 7f31f5ab4640 1 --2- 172.21.15.72:0/410151487 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ee70b79bf0 0x55ee70b756e0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f31e0098610 tx=0x7f31e00051d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7fda52fdadbc7bc server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:29.173 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.164+0000 7f31dbfff640 1 -- 172.21.15.72:0/410151487 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31e001c040 con 0x55ee70b79bf0 2024-03-22T01:46:29.173 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.164+0000 7f31dbfff640 1 -- 172.21.15.72:0/410151487 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31e0086040 con 0x55ee70b79bf0 2024-03-22T01:46:29.173 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.164+0000 7f31dbfff640 1 -- 172.21.15.72:0/410151487 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31e00a41c0 con 0x55ee70b79bf0 2024-03-22T01:46:29.173 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.165+0000 7f31fb1d7740 1 -- 172.21.15.72:0/410151487 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ee70b79bf0 msgr2=0x55ee70b756e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:29.173 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.165+0000 7f31fb1d7740 1 --2- 172.21.15.72:0/410151487 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ee70b79bf0 0x55ee70b756e0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f31e0098610 tx=0x7f31e00051d0 comp rx=0 tx=0).stop 2024-03-22T01:46:29.173 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.165+0000 7f31fb1d7740 1 -- 172.21.15.72:0/410151487 shutdown_connections 2024-03-22T01:46:29.173 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.165+0000 7f31fb1d7740 1 --2- 172.21.15.72:0/410151487 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee70b829c0 0x55ee70b9a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:29.173 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.165+0000 7f31fb1d7740 1 --2- 172.21.15.72:0/410151487 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ee70b79bf0 0x55ee70b756e0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:29.173 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.165+0000 7f31fb1d7740 1 --2- 172.21.15.72:0/410151487 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee70b792d0 0x55ee70b8e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:29.174 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.165+0000 7f31fb1d7740 1 -- 172.21.15.72:0/410151487 >> 172.21.15.72:0/410151487 conn(0x55ee70b73bf0 msgr2=0x55ee70b74fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:29.174 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.165+0000 7f31fb1d7740 1 -- 172.21.15.72:0/410151487 shutdown_connections 2024-03-22T01:46:29.174 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.165+0000 7f31fb1d7740 1 -- 172.21.15.72:0/410151487 wait complete. 2024-03-22T01:46:29.174 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.165+0000 7f31fb1d7740 1 -- start start 2024-03-22T01:46:29.174 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.165+0000 7f31fb1d7740 1 Processor -- start 2024-03-22T01:46:29.174 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.166+0000 7f31fb1d7740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ee70b792d0 0x55ee70b7c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:29.174 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.166+0000 7f31fb1d7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee70b79bf0 0x55ee70b7c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:29.174 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.166+0000 7f31fb1d7740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee70b829c0 0x55ee70b7cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:29.174 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.166+0000 7f31fb1d7740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f31f01abbd0 con 0x55ee70b792d0 2024-03-22T01:46:29.174 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.166+0000 7f31fb1d7740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f31f01ad040 con 0x55ee70b829c0 2024-03-22T01:46:29.174 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.166+0000 7f31fb1d7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f31f01ac2a0 con 0x55ee70b79bf0 2024-03-22T01:46:29.174 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.166+0000 7f31fb1d7740 10 client.0 _ll_register_callbacks cb 0x7f31ec005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:46:29.174 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.166+0000 7f31f62b5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ee70b792d0 0x55ee70b7c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:29.175 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.166+0000 7f31f6ab6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee70b829c0 0x55ee70b7cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:29.175 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.166+0000 7f31f5ab4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee70b79bf0 0x55ee70b7c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:29.175 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.166+0000 7f31f62b5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ee70b792d0 0x55ee70b7c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59082/0 (socket says 172.21.15.72:59082) 2024-03-22T01:46:29.175 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.166+0000 7f31f62b5640 1 -- 172.21.15.72:0/4138960685 learned_addr learned my addr 172.21.15.72:0/4138960685 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:29.175 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.166+0000 7f31f6ab6640 1 -- 172.21.15.72:0/4138960685 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee70b79bf0 msgr2=0x55ee70b7c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:29.175 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.166+0000 7f31f6ab6640 1 --2- 172.21.15.72:0/4138960685 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee70b79bf0 0x55ee70b7c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:29.175 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.166+0000 7f31f6ab6640 1 -- 172.21.15.72:0/4138960685 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ee70b792d0 msgr2=0x55ee70b7c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:29.175 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.166+0000 7f31f6ab6640 1 --2- 172.21.15.72:0/4138960685 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ee70b792d0 0x55ee70b7c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:29.175 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.166+0000 7f31f6ab6640 1 -- 172.21.15.72:0/4138960685 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee70bfc1e0 con 0x55ee70b829c0 2024-03-22T01:46:29.175 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.166+0000 7f31f62b5640 1 --2- 172.21.15.72:0/4138960685 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ee70b792d0 0x55ee70b7c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:46:29.175 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.166+0000 7f31f6ab6640 1 --2- 172.21.15.72:0/4138960685 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee70b829c0 0x55ee70b7cd50 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f31e80049d0 tx=0x7f31e80327c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7d926e2c3ae3064 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:29.175 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.166+0000 7f31d9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-03-22T01:46:29.175 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.166+0000 7f31d9ffb640 1 -- 172.21.15.72:0/4138960685 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31e8039040 con 0x55ee70b829c0 2024-03-22T01:46:29.176 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.167+0000 7f31fb1d7740 1 -- 172.21.15.72:0/4138960685 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f31e0086040 con 0x55ee70b829c0 2024-03-22T01:46:29.176 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.167+0000 7f31d9ffb640 1 -- 172.21.15.72:0/4138960685 <== mon.2 v2:172.21.15.89:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f31e8040460 con 0x55ee70b829c0 2024-03-22T01:46:29.176 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.167+0000 7f31d9ffb640 1 -- 172.21.15.72:0/4138960685 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7f31e8005ca0 con 0x55ee70b829c0 2024-03-22T01:46:29.176 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.167+0000 7f31fb1d7740 10 client.26102 fetch_fsmap learned FSMap version 103 2024-03-22T01:46:29.176 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.167+0000 7f31fb1d7740 1 -- 172.21.15.72:0/4138960685 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=4+}) -- 0x55ee70bfd4f0 con 0x55ee70b829c0 2024-03-22T01:46:29.176 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.167+0000 7f31d9ffb640 1 -- 172.21.15.72:0/4138960685 <== mon.2 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31e0086040 con 0x55ee70b829c0 2024-03-22T01:46:29.176 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.167+0000 7f31d9ffb640 1 -- 172.21.15.72:0/4138960685 <== mon.2 v2:172.21.15.89:3300/0 5 ==== fsmap(e 103) ==== 8330+0+0 (secure 0 0 0) 0x7f31e8034640 con 0x55ee70b829c0 2024-03-22T01:46:29.176 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.167+0000 7f31fb1d7740 10 client.26102 fetch_fsmap finished waiting for FSMap version 103 2024-03-22T01:46:29.176 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.167+0000 7f31fb1d7740 10 client.26102 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T01:46:29.176 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.167+0000 7f31fb1d7740 20 client.26102 populate_metadata read hostname 'smithi072' 2024-03-22T01:46:29.176 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.167+0000 7f31fb1d7740 1 --2- 172.21.15.72:0/4138960685 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55ee70bfdc60 0x55ee70c00080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:29.176 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.167+0000 7f31fb1d7740 4 client.26102 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T01:46:29.176 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.167+0000 7f31fb1d7740 1 -- 172.21.15.72:0/4138960685 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ee70ae48d0 con 0x55ee70bfdc60 2024-03-22T01:46:29.177 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.167+0000 7f31f62b5640 1 --2- 172.21.15.72:0/4138960685 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55ee70bfdc60 0x55ee70c00080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:29.177 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.167+0000 7f31d9ffb640 10 client.26102 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:46:29.177 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.167+0000 7f31f62b5640 1 --2- 172.21.15.72:0/4138960685 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55ee70bfdc60 0x55ee70c00080 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2d349314f8b4806 server_cookie=8a185ab3e176ad3a in_seq=0 out_seq=0 2024-03-22T01:46:29.178 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.178+0000 7f31d9ffb640 1 -- 172.21.15.72:0/4138960685 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ee70ae48d0 con 0x55ee70bfdc60 2024-03-22T01:46:29.178 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.178+0000 7f31d9ffb640 10 client.26102 handle_command_reply: tid=0 2024-03-22T01:46:29.178 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.178+0000 7f31fb1d7740 1 client.26102 shutdown 2024-03-22T01:46:29.178 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.178+0000 7f31fb1d7740 1 -- 172.21.15.72:0/4138960685 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee70b829c0 msgr2=0x55ee70b7cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:29.178 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.178+0000 7f31fb1d7740 1 --2- 172.21.15.72:0/4138960685 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee70b829c0 0x55ee70b7cd50 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f31e80049d0 tx=0x7f31e80327c0 comp rx=0 tx=0).stop 2024-03-22T01:46:29.178 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.178+0000 7f31fb1d7740 1 -- 172.21.15.72:0/4138960685 shutdown_connections 2024-03-22T01:46:29.179 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.178+0000 7f31fb1d7740 1 --2- 172.21.15.72:0/4138960685 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55ee70bfdc60 0x55ee70c00080 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:29.179 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.178+0000 7f31fb1d7740 1 --2- 172.21.15.72:0/4138960685 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee70b829c0 0x55ee70b7cd50 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:29.179 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.178+0000 7f31fb1d7740 1 --2- 172.21.15.72:0/4138960685 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee70b79bf0 0x55ee70b7c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:29.179 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.178+0000 7f31fb1d7740 1 --2- 172.21.15.72:0/4138960685 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ee70b792d0 0x55ee70b7c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:29.179 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.178+0000 7f31d9ffb640 0 client.26102 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:46:29.179 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.178+0000 7f31fb1d7740 1 -- 172.21.15.72:0/4138960685 >> 172.21.15.72:0/4138960685 conn(0x55ee70b73bf0 msgr2=0x55ee70b75710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:29.179 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.178+0000 7f31fb1d7740 1 -- 172.21.15.72:0/4138960685 shutdown_connections 2024-03-22T01:46:29.179 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.178+0000 7f31fb1d7740 1 -- 172.21.15.72:0/4138960685 wait complete. 2024-03-22T01:46:29.179 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.178+0000 7f31fb1d7740 20 client.26102 trim_cache size 0 max 16384 2024-03-22T01:46:29.180 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout:{ 2024-03-22T01:46:29.180 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout: "message": "", 2024-03-22T01:46:29.180 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout: "return_code": 0 2024-03-22T01:46:29.180 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout:} 2024-03-22T01:46:29.181 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.178+0000 7f31f8519640 1 -- 172.21.15.72:0/290727413 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f31dc0a3080 msgr2=0x7f31dc0a5540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:29.181 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.178+0000 7f31f8519640 1 --2- 172.21.15.72:0/290727413 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f31dc0a3080 0x7f31dc0a5540 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f31ec0095f0 tx=0x7f31ec010040 comp rx=0 tx=0).stop 2024-03-22T01:46:29.181 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.178+0000 7f31f8519640 1 -- 172.21.15.72:0/290727413 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31f0101430 msgr2=0x7f31f0197850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:29.181 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.178+0000 7f31f8519640 1 --2- 172.21.15.72:0/290727413 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31f0101430 0x7f31f0197850 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f31e000e9f0 tx=0x7f31e000eeb0 comp rx=0 tx=0).stop 2024-03-22T01:46:29.181 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.178+0000 7f31f8519640 1 -- 172.21.15.72:0/290727413 shutdown_connections 2024-03-22T01:46:29.181 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.178+0000 7f31f8519640 1 --2- 172.21.15.72:0/290727413 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f31dc0a3080 0x7f31dc0a5540 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:29.181 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.178+0000 7f31f8519640 1 --2- 172.21.15.72:0/290727413 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31f0101430 0x7f31f0197850 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:29.181 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.178+0000 7f31f8519640 1 --2- 172.21.15.72:0/290727413 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31f0106a80 0x7f31f019bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:29.181 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.178+0000 7f31f8519640 1 --2- 172.21.15.72:0/290727413 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f31f0100a70 0x7f31f0197310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:29.181 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.178+0000 7f31f8519640 1 -- 172.21.15.72:0/290727413 >> 172.21.15.72:0/290727413 conn(0x7f31f0077bb0 msgr2=0x7f31f0103600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:29.181 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.179+0000 7f31f8519640 1 -- 172.21.15.72:0/290727413 shutdown_connections 2024-03-22T01:46:29.182 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:29.179+0000 7f31f8519640 1 -- 172.21.15.72:0/290727413 wait complete. 2024-03-22T01:46:30.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:46:30 smithi089 ceph-mon[35770]: pgmap v298: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 42 KiB/s wr, 282 op/s 2024-03-22T01:46:30.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:46:30 smithi072 ceph-mon[32414]: pgmap v298: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 42 KiB/s wr, 282 op/s 2024-03-22T01:46:30.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:46:30 smithi098 ceph-mon[36468]: pgmap v298: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 42 KiB/s wr, 282 op/s 2024-03-22T01:46:32.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:46:32 smithi089 ceph-mon[35770]: pgmap v299: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 41 KiB/s wr, 270 op/s 2024-03-22T01:46:32.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:46:32 smithi072 ceph-mon[32414]: pgmap v299: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 41 KiB/s wr, 270 op/s 2024-03-22T01:46:32.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:46:32 smithi098 ceph-mon[36468]: pgmap v299: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 41 KiB/s wr, 270 op/s 2024-03-22T01:46:33.289 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T01:46:33.290 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T01:46:33.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:46:33 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:46:33.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.397+0000 7f3a54979640 1 -- 172.21.15.72:0/3776179508 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3a50106a40 msgr2=0x7f3a50106e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:33.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.397+0000 7f3a54979640 1 --2- 172.21.15.72:0/3776179508 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3a50106a40 0x7f3a50106e20 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f3a40009930 tx=0x7f3a4002bc30 comp rx=0 tx=0).stop 2024-03-22T01:46:33.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.397+0000 7f3a54979640 1 -- 172.21.15.72:0/3776179508 shutdown_connections 2024-03-22T01:46:33.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.397+0000 7f3a54979640 1 --2- 172.21.15.72:0/3776179508 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a501013f0 0x7f3a5010d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.397+0000 7f3a54979640 1 --2- 172.21.15.72:0/3776179508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a50100a30 0x7f3a50100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.397+0000 7f3a54979640 1 --2- 172.21.15.72:0/3776179508 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3a50106a40 0x7f3a50106e20 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.397+0000 7f3a54979640 1 -- 172.21.15.72:0/3776179508 >> 172.21.15.72:0/3776179508 conn(0x7f3a50077c40 msgr2=0x7f3a50078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:33.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.397+0000 7f3a54979640 1 -- 172.21.15.72:0/3776179508 shutdown_connections 2024-03-22T01:46:33.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.397+0000 7f3a54979640 1 -- 172.21.15.72:0/3776179508 wait complete. 2024-03-22T01:46:33.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.398+0000 7f3a54979640 1 Processor -- start 2024-03-22T01:46:33.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.398+0000 7f3a54979640 1 -- start start 2024-03-22T01:46:33.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.398+0000 7f3a54979640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a50100a30 0x7f3a50197370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:33.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.398+0000 7f3a54979640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a501013f0 0x7f3a501978b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:33.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.398+0000 7f3a54979640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3a50106a40 0x7f3a5019bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:33.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.398+0000 7f3a54979640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f3a50110e50 con 0x7f3a50106a40 2024-03-22T01:46:33.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.398+0000 7f3a54979640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3a50110cd0 con 0x7f3a50100a30 2024-03-22T01:46:33.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.398+0000 7f3a54979640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3a50110fd0 con 0x7f3a501013f0 2024-03-22T01:46:33.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.398+0000 7f3a4e59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a50100a30 0x7f3a50197370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:33.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.398+0000 7f3a4ed9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3a50106a40 0x7f3a5019bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:33.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.398+0000 7f3a4e59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a50100a30 0x7f3a50197370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.72:32866/0 (socket says 172.21.15.72:32866) 2024-03-22T01:46:33.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.398+0000 7f3a4e59c640 1 -- 172.21.15.72:0/3311003369 learned_addr learned my addr 172.21.15.72:0/3311003369 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:33.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.398+0000 7f3a47fff640 1 --2- 172.21.15.72:0/3311003369 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a501013f0 0x7f3a501978b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:33.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.398+0000 7f3a4ed9d640 1 -- 172.21.15.72:0/3311003369 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a501013f0 msgr2=0x7f3a501978b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:33.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.398+0000 7f3a4ed9d640 1 --2- 172.21.15.72:0/3311003369 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a501013f0 0x7f3a501978b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.398+0000 7f3a4ed9d640 1 -- 172.21.15.72:0/3311003369 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a50100a30 msgr2=0x7f3a50197370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:33.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.398+0000 7f3a4ed9d640 1 --2- 172.21.15.72:0/3311003369 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a50100a30 0x7f3a50197370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.398+0000 7f3a4ed9d640 1 -- 172.21.15.72:0/3311003369 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a5019c310 con 0x7f3a50106a40 2024-03-22T01:46:33.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.398+0000 7f3a4e59c640 1 --2- 172.21.15.72:0/3311003369 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a50100a30 0x7f3a50197370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:46:33.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.398+0000 7f3a4ed9d640 1 --2- 172.21.15.72:0/3311003369 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3a50106a40 0x7f3a5019bbe0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f3a4800ebc0 tx=0x7f3a4800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:33.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.398+0000 7f3a477fe640 1 -- 172.21.15.72:0/3311003369 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a48003b70 con 0x7f3a50106a40 2024-03-22T01:46:33.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.399+0000 7f3a477fe640 1 -- 172.21.15.72:0/3311003369 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a48003d10 con 0x7f3a50106a40 2024-03-22T01:46:33.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.399+0000 7f3a54979640 1 -- 172.21.15.72:0/3311003369 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a5019c600 con 0x7f3a50106a40 2024-03-22T01:46:33.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.399+0000 7f3a477fe640 1 -- 172.21.15.72:0/3311003369 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a48010720 con 0x7f3a50106a40 2024-03-22T01:46:33.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.399+0000 7f3a54979640 1 -- 172.21.15.72:0/3311003369 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a501a3e90 con 0x7f3a50106a40 2024-03-22T01:46:33.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.401+0000 7f3a477fe640 1 -- 172.21.15.72:0/3311003369 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 25) ==== 137985+0+0 (secure 0 0 0) 0x7f3a48004020 con 0x7f3a50106a40 2024-03-22T01:46:33.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.401+0000 7f3a57635740 1 Processor -- start 2024-03-22T01:46:33.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.402+0000 7f3a57635740 1 -- start start 2024-03-22T01:46:33.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.402+0000 7f3a477fe640 1 --2- 172.21.15.72:0/3311003369 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f3a3c0a3090 0x7f3a3c0a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:33.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.402+0000 7f3a57635740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5630c9a2f410 0x5630c9a2f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:33.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.402+0000 7f3a57635740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5630c9a2fd30 0x5630c9a383d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:33.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.402+0000 7f3a57635740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630c9a38910 0x5630c9a3ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:33.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.402+0000 7f3a57635740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x5630c99938c0 con 0x5630c9a2fd30 2024-03-22T01:46:33.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.402+0000 7f3a57635740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5630c993bc20 con 0x5630c9a2f410 2024-03-22T01:46:33.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.402+0000 7f3a57635740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5630c998f870 con 0x5630c9a38910 2024-03-22T01:46:33.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.402+0000 7f3a47fff640 1 --2- 172.21.15.72:0/3311003369 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f3a3c0a3090 0x7f3a3c0a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:33.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.402+0000 7f3a4e59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5630c9a2f410 0x5630c9a2f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:33.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.402+0000 7f3a477fe640 1 -- 172.21.15.72:0/3311003369 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f3a480d01a0 con 0x7f3a50106a40 2024-03-22T01:46:33.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.402+0000 7f3a47fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5630c9a2fd30 0x5630c9a383d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:33.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.402+0000 7f3a47fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5630c9a2fd30 0x5630c9a383d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36224/0 (socket says 172.21.15.72:36224) 2024-03-22T01:46:33.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.402+0000 7f3a4ed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630c9a38910 0x5630c9a3ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:33.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.402+0000 7f3a47fff640 1 -- 172.21.15.72:0/1980385680 learned_addr learned my addr 172.21.15.72:0/1980385680 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:33.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.402+0000 7f3a4e59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5630c9a2f410 0x5630c9a2f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.72:32872/0 (socket says 172.21.15.72:32872) 2024-03-22T01:46:33.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.402+0000 7f3a4ed9d640 1 -- 172.21.15.72:0/1980385680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5630c9a2f410 msgr2=0x5630c9a2f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:33.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.402+0000 7f3a4ed9d640 1 --2- 172.21.15.72:0/1980385680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5630c9a2f410 0x5630c9a2f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.402+0000 7f3a4ed9d640 1 -- 172.21.15.72:0/1980385680 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5630c9a2fd30 msgr2=0x5630c9a383d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:33.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.402+0000 7f3a4ed9d640 1 --2- 172.21.15.72:0/1980385680 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5630c9a2fd30 0x5630c9a383d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.402+0000 7f3a4ed9d640 1 -- 172.21.15.72:0/1980385680 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5630c9960150 con 0x5630c9a38910 2024-03-22T01:46:33.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.402+0000 7f3a4e59c640 1 --2- 172.21.15.72:0/1980385680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5630c9a2f410 0x5630c9a2f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:46:33.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.402+0000 7f3a47fff640 1 --2- 172.21.15.72:0/3311003369 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f3a3c0a3090 0x7f3a3c0a5550 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x5630c9a2eee0 tx=0x7f3a38009530 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:33.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.402+0000 7f3a4ed9d640 1 --2- 172.21.15.72:0/1980385680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630c9a38910 0x5630c9a3ad00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3a480cd4e0 tx=0x7f3a480984e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d23e115c88eb87f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:33.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.403+0000 7f3a44ff9640 1 -- 172.21.15.72:0/1980385680 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a4809c880 con 0x5630c9a38910 2024-03-22T01:46:33.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.403+0000 7f3a44ff9640 1 -- 172.21.15.72:0/1980385680 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a480148d0 con 0x5630c9a38910 2024-03-22T01:46:33.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.403+0000 7f3a44ff9640 1 -- 172.21.15.72:0/1980385680 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a480987f0 con 0x5630c9a38910 2024-03-22T01:46:33.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.403+0000 7f3a57635740 1 -- 172.21.15.72:0/1980385680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630c9a38910 msgr2=0x5630c9a3ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:33.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.403+0000 7f3a57635740 1 --2- 172.21.15.72:0/1980385680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630c9a38910 0x5630c9a3ad00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3a480cd4e0 tx=0x7f3a480984e0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.403+0000 7f3a57635740 1 -- 172.21.15.72:0/1980385680 shutdown_connections 2024-03-22T01:46:33.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.403+0000 7f3a57635740 1 --2- 172.21.15.72:0/1980385680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630c9a38910 0x5630c9a3ad00 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.403+0000 7f3a57635740 1 --2- 172.21.15.72:0/1980385680 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5630c9a2fd30 0x5630c9a383d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.403+0000 7f3a57635740 1 --2- 172.21.15.72:0/1980385680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5630c9a2f410 0x5630c9a2f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.403+0000 7f3a57635740 1 -- 172.21.15.72:0/1980385680 >> 172.21.15.72:0/1980385680 conn(0x5630c9a29b40 msgr2=0x5630c9a29f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:33.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.404+0000 7f3a57635740 1 -- 172.21.15.72:0/1980385680 shutdown_connections 2024-03-22T01:46:33.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.404+0000 7f3a57635740 1 -- 172.21.15.72:0/1980385680 wait complete. 2024-03-22T01:46:33.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.404+0000 7f3a57635740 1 -- start start 2024-03-22T01:46:33.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.404+0000 7f3a57635740 1 Processor -- start 2024-03-22T01:46:33.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.404+0000 7f3a57635740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5630c9a2f410 0x5630c9a35930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:33.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.404+0000 7f3a57635740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630c9a2fd30 0x5630c9a35e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:33.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.404+0000 7f3a57635740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5630c9a38910 0x5630c9a363b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:33.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.404+0000 7f3a57635740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x5630c9a3b6a0 con 0x5630c9a38910 2024-03-22T01:46:33.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.404+0000 7f3a57635740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5630c994f130 con 0x5630c9a2f410 2024-03-22T01:46:33.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.404+0000 7f3a57635740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5630c9a3b820 con 0x5630c9a2fd30 2024-03-22T01:46:33.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.404+0000 7f3a57635740 10 client.0 _ll_register_callbacks cb 0x5630c994eb00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:46:33.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.404+0000 7f3a4e59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5630c9a2f410 0x5630c9a35930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:33.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.404+0000 7f3a4ed9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5630c9a38910 0x5630c9a363b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:33.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.404+0000 7f3a4e59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5630c9a2f410 0x5630c9a35930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.72:32886/0 (socket says 172.21.15.72:32886) 2024-03-22T01:46:33.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.404+0000 7f3a4e59c640 1 -- 172.21.15.72:0/2637873277 learned_addr learned my addr 172.21.15.72:0/2637873277 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:33.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.404+0000 7f3a47fff640 1 --2- 172.21.15.72:0/2637873277 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630c9a2fd30 0x5630c9a35e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:33.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.405+0000 7f3a4e59c640 1 -- 172.21.15.72:0/2637873277 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630c9a2fd30 msgr2=0x5630c9a35e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:33.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.405+0000 7f3a4e59c640 1 --2- 172.21.15.72:0/2637873277 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630c9a2fd30 0x5630c9a35e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.405+0000 7f3a4e59c640 1 -- 172.21.15.72:0/2637873277 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5630c9a38910 msgr2=0x5630c9a363b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:33.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.405+0000 7f3a4e59c640 1 --2- 172.21.15.72:0/2637873277 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5630c9a38910 0x5630c9a363b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.405+0000 7f3a4e59c640 1 -- 172.21.15.72:0/2637873277 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5630c97717d0 con 0x5630c9a2f410 2024-03-22T01:46:33.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.405+0000 7f3a4e59c640 1 --2- 172.21.15.72:0/2637873277 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5630c9a2f410 0x5630c9a35930 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f3a40009230 tx=0x7f3a40005e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6511213bfaf6ff42 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:33.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.405+0000 7f3a35ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-03-22T01:46:33.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.405+0000 7f3a35ffb640 1 -- 172.21.15.72:0/2637873277 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a4004d060 con 0x5630c9a2f410 2024-03-22T01:46:33.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.405+0000 7f3a57635740 1 -- 172.21.15.72:0/2637873277 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a480148d0 con 0x5630c9a2f410 2024-03-22T01:46:33.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.405+0000 7f3a35ffb640 1 -- 172.21.15.72:0/2637873277 <== mon.2 v2:172.21.15.89:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f3a40004040 con 0x5630c9a2f410 2024-03-22T01:46:33.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.405+0000 7f3a35ffb640 1 -- 172.21.15.72:0/2637873277 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7f3a40054050 con 0x5630c9a2f410 2024-03-22T01:46:33.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.405+0000 7f3a57635740 10 client.26114 fetch_fsmap learned FSMap version 103 2024-03-22T01:46:33.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.405+0000 7f3a57635740 1 -- 172.21.15.72:0/2637873277 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=4+}) -- 0x5630c96a73f0 con 0x5630c9a2f410 2024-03-22T01:46:33.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.405+0000 7f3a35ffb640 1 -- 172.21.15.72:0/2637873277 <== mon.2 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a480148d0 con 0x5630c9a2f410 2024-03-22T01:46:33.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.405+0000 7f3a35ffb640 1 -- 172.21.15.72:0/2637873277 <== mon.2 v2:172.21.15.89:3300/0 5 ==== fsmap(e 103) ==== 8330+0+0 (secure 0 0 0) 0x7f3a4003e7d0 con 0x5630c9a2f410 2024-03-22T01:46:33.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.405+0000 7f3a57635740 10 client.26114 fetch_fsmap finished waiting for FSMap version 103 2024-03-22T01:46:33.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.405+0000 7f3a57635740 10 client.26114 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T01:46:33.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.405+0000 7f3a57635740 20 client.26114 populate_metadata read hostname 'smithi072' 2024-03-22T01:46:33.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.405+0000 7f3a57635740 1 --2- 172.21.15.72:0/2637873277 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x5630c9a48c10 0x5630c9a4b070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:33.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.405+0000 7f3a57635740 4 client.26114 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T01:46:33.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.405+0000 7f3a57635740 1 -- 172.21.15.72:0/2637873277 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5630c99a0e00 con 0x5630c9a48c10 2024-03-22T01:46:33.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.406+0000 7f3a47fff640 1 --2- 172.21.15.72:0/2637873277 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x5630c9a48c10 0x5630c9a4b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:33.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.406+0000 7f3a35ffb640 10 client.26114 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:46:33.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.406+0000 7f3a47fff640 1 --2- 172.21.15.72:0/2637873277 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x5630c9a48c10 0x5630c9a4b070 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f74e5563925a6912 server_cookie=bd1d1cdc90323e0f in_seq=0 out_seq=0 2024-03-22T01:46:33.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.407+0000 7f3a35ffb640 1 -- 172.21.15.72:0/2637873277 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5630c99a0e00 con 0x5630c9a48c10 2024-03-22T01:46:33.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.407+0000 7f3a35ffb640 10 client.26114 handle_command_reply: tid=0 2024-03-22T01:46:33.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.407+0000 7f3a57635740 1 client.26114 shutdown 2024-03-22T01:46:33.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.407+0000 7f3a57635740 1 -- 172.21.15.72:0/2637873277 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5630c9a2f410 msgr2=0x5630c9a35930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:33.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.407+0000 7f3a57635740 1 --2- 172.21.15.72:0/2637873277 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5630c9a2f410 0x5630c9a35930 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f3a40009230 tx=0x7f3a40005e70 comp rx=0 tx=0).stop 2024-03-22T01:46:33.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.407+0000 7f3a57635740 1 -- 172.21.15.72:0/2637873277 shutdown_connections 2024-03-22T01:46:33.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.407+0000 7f3a57635740 1 --2- 172.21.15.72:0/2637873277 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x5630c9a48c10 0x5630c9a4b070 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.407+0000 7f3a57635740 1 --2- 172.21.15.72:0/2637873277 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5630c9a38910 0x5630c9a363b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.407+0000 7f3a35ffb640 0 client.26114 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:46:33.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.407+0000 7f3a57635740 1 --2- 172.21.15.72:0/2637873277 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630c9a2fd30 0x5630c9a35e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.407+0000 7f3a57635740 1 --2- 172.21.15.72:0/2637873277 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5630c9a2f410 0x5630c9a35930 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.407+0000 7f3a57635740 1 -- 172.21.15.72:0/2637873277 >> 172.21.15.72:0/2637873277 conn(0x5630c9a29b40 msgr2=0x5630c9a2b890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:33.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.407+0000 7f3a57635740 1 -- 172.21.15.72:0/2637873277 shutdown_connections 2024-03-22T01:46:33.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.407+0000 7f3a57635740 1 -- 172.21.15.72:0/2637873277 wait complete. 2024-03-22T01:46:33.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.407+0000 7f3a57635740 20 client.26114 trim_cache size 0 max 16384 2024-03-22T01:46:33.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:46:33 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:46:33.422 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:46:33 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:46:33.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.424+0000 7f3a57635740 1 Processor -- start 2024-03-22T01:46:33.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.424+0000 7f3a57635740 1 -- start start 2024-03-22T01:46:33.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.424+0000 7f3a57635740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5630c9a2f410 0x5630c9a38060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:33.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.424+0000 7f3a57635740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630c9a2fd30 0x5630c9a44440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:33.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.424+0000 7f3a57635740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5630c9a38910 0x5630c9a34a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:33.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.424+0000 7f3a57635740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f3a501aaed0 con 0x5630c9a38910 2024-03-22T01:46:33.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.424+0000 7f3a57635740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3a501ad750 con 0x5630c9a2f410 2024-03-22T01:46:33.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.424+0000 7f3a57635740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3a501ab540 con 0x5630c9a2fd30 2024-03-22T01:46:33.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.424+0000 7f3a4ed9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5630c9a38910 0x5630c9a34a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:33.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.424+0000 7f3a4e59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5630c9a2f410 0x5630c9a38060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:33.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.424+0000 7f3a4e59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5630c9a2f410 0x5630c9a38060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.72:32894/0 (socket says 172.21.15.72:32894) 2024-03-22T01:46:33.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.424+0000 7f3a4ed9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5630c9a38910 0x5630c9a34a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36244/0 (socket says 172.21.15.72:36244) 2024-03-22T01:46:33.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.424+0000 7f3a4e59c640 1 -- 172.21.15.72:0/2240197603 learned_addr learned my addr 172.21.15.72:0/2240197603 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:33.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.424+0000 7f3a47fff640 1 --2- 172.21.15.72:0/2240197603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630c9a2fd30 0x5630c9a44440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:33.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.425+0000 7f3a4e59c640 1 -- 172.21.15.72:0/2240197603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630c9a2fd30 msgr2=0x5630c9a44440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:33.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.425+0000 7f3a4e59c640 1 --2- 172.21.15.72:0/2240197603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630c9a2fd30 0x5630c9a44440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.425+0000 7f3a4e59c640 1 -- 172.21.15.72:0/2240197603 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5630c9a38910 msgr2=0x5630c9a34a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:33.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.425+0000 7f3a4e59c640 1 --2- 172.21.15.72:0/2240197603 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5630c9a38910 0x5630c9a34a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.425+0000 7f3a4e59c640 1 -- 172.21.15.72:0/2240197603 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5630c96670f0 con 0x5630c9a2f410 2024-03-22T01:46:33.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.425+0000 7f3a4e59c640 1 --2- 172.21.15.72:0/2240197603 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5630c9a2f410 0x5630c9a38060 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f3a40009850 tx=0x7f3a40009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e1af095bce664094 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:33.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.425+0000 7f3a37fff640 1 -- 172.21.15.72:0/2240197603 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a4004d050 con 0x5630c9a2f410 2024-03-22T01:46:33.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.425+0000 7f3a37fff640 1 -- 172.21.15.72:0/2240197603 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a40009520 con 0x5630c9a2f410 2024-03-22T01:46:33.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.425+0000 7f3a37fff640 1 -- 172.21.15.72:0/2240197603 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a400486a0 con 0x5630c9a2f410 2024-03-22T01:46:33.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.425+0000 7f3a57635740 1 -- 172.21.15.72:0/2240197603 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5630c9a2f410 msgr2=0x5630c9a38060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:33.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.425+0000 7f3a57635740 1 --2- 172.21.15.72:0/2240197603 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5630c9a2f410 0x5630c9a38060 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f3a40009850 tx=0x7f3a40009260 comp rx=0 tx=0).stop 2024-03-22T01:46:33.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.425+0000 7f3a57635740 1 -- 172.21.15.72:0/2240197603 shutdown_connections 2024-03-22T01:46:33.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.425+0000 7f3a57635740 1 --2- 172.21.15.72:0/2240197603 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5630c9a38910 0x5630c9a34a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.425+0000 7f3a57635740 1 --2- 172.21.15.72:0/2240197603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630c9a2fd30 0x5630c9a44440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.425+0000 7f3a57635740 1 --2- 172.21.15.72:0/2240197603 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5630c9a2f410 0x5630c9a38060 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.425+0000 7f3a57635740 1 -- 172.21.15.72:0/2240197603 >> 172.21.15.72:0/2240197603 conn(0x5630c9a29b40 msgr2=0x5630c9a28370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:33.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.425+0000 7f3a57635740 1 -- 172.21.15.72:0/2240197603 shutdown_connections 2024-03-22T01:46:33.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.426+0000 7f3a57635740 1 -- 172.21.15.72:0/2240197603 wait complete. 2024-03-22T01:46:33.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.426+0000 7f3a57635740 1 -- start start 2024-03-22T01:46:33.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.426+0000 7f3a57635740 1 Processor -- start 2024-03-22T01:46:33.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.426+0000 7f3a57635740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5630c9a2f410 0x5630c9a40d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:33.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.426+0000 7f3a57635740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5630c9a2fd30 0x5630c9a32280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:33.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.426+0000 7f3a57635740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630c9a38910 0x5630c9a32800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:33.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.426+0000 7f3a57635740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f3a501abc10 con 0x5630c9a2f410 2024-03-22T01:46:33.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.426+0000 7f3a57635740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3a501ad080 con 0x5630c9a2fd30 2024-03-22T01:46:33.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.426+0000 7f3a57635740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3a501ac2e0 con 0x5630c9a38910 2024-03-22T01:46:33.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.426+0000 7f3a57635740 10 client.0 _ll_register_callbacks cb 0x7f3a40005d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:46:33.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.426+0000 7f3a4e59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5630c9a2f410 0x5630c9a40d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:33.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.426+0000 7f3a47fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5630c9a2fd30 0x5630c9a32280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:33.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.426+0000 7f3a4e59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5630c9a2f410 0x5630c9a40d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36256/0 (socket says 172.21.15.72:36256) 2024-03-22T01:46:33.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.426+0000 7f3a4e59c640 1 -- 172.21.15.72:0/180701069 learned_addr learned my addr 172.21.15.72:0/180701069 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:33.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.426+0000 7f3a4ed9d640 1 --2- 172.21.15.72:0/180701069 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630c9a38910 0x5630c9a32800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:33.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.426+0000 7f3a4e59c640 1 -- 172.21.15.72:0/180701069 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630c9a38910 msgr2=0x5630c9a32800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:33.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.426+0000 7f3a4e59c640 1 --2- 172.21.15.72:0/180701069 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630c9a38910 0x5630c9a32800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.426+0000 7f3a4e59c640 1 -- 172.21.15.72:0/180701069 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5630c9a2fd30 msgr2=0x5630c9a32280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:33.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.426+0000 7f3a4e59c640 1 --2- 172.21.15.72:0/180701069 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5630c9a2fd30 0x5630c9a32280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.426+0000 7f3a4e59c640 1 -- 172.21.15.72:0/180701069 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5630c9a33030 con 0x5630c9a2f410 2024-03-22T01:46:33.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.426+0000 7f3a47fff640 1 --2- 172.21.15.72:0/180701069 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5630c9a2fd30 0x5630c9a32280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:46:33.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.427+0000 7f3a4e59c640 1 --2- 172.21.15.72:0/180701069 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5630c9a2f410 0x5630c9a40d60 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f3a4003ec90 tx=0x7f3a40005e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=246c4dc7731c3ad4 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:33.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.427+0000 7f3a35ffb640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:46:33.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.427+0000 7f3a35ffb640 1 -- 172.21.15.72:0/180701069 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a4004d070 con 0x5630c9a2f410 2024-03-22T01:46:33.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.427+0000 7f3a35ffb640 1 -- 172.21.15.72:0/180701069 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f3a40004f80 con 0x5630c9a2f410 2024-03-22T01:46:33.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.427+0000 7f3a57635740 1 -- 172.21.15.72:0/180701069 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a40009520 con 0x5630c9a2f410 2024-03-22T01:46:33.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.427+0000 7f3a35ffb640 1 -- 172.21.15.72:0/180701069 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7f3a4005c040 con 0x5630c9a2f410 2024-03-22T01:46:33.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.427+0000 7f3a57635740 10 client.16287 fetch_fsmap learned FSMap version 103 2024-03-22T01:46:33.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.427+0000 7f3a57635740 1 -- 172.21.15.72:0/180701069 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=4+}) -- 0x5630c9aaf020 con 0x5630c9a2f410 2024-03-22T01:46:33.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.427+0000 7f3a35ffb640 1 -- 172.21.15.72:0/180701069 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a40009520 con 0x5630c9a2f410 2024-03-22T01:46:33.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.427+0000 7f3a35ffb640 1 -- 172.21.15.72:0/180701069 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 103) ==== 8330+0+0 (secure 0 0 0) 0x7f3a40048a80 con 0x5630c9a2f410 2024-03-22T01:46:33.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.427+0000 7f3a57635740 10 client.16287 fetch_fsmap finished waiting for FSMap version 103 2024-03-22T01:46:33.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.427+0000 7f3a57635740 10 client.16287 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T01:46:33.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.427+0000 7f3a57635740 20 client.16287 populate_metadata read hostname 'smithi072' 2024-03-22T01:46:33.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.427+0000 7f3a57635740 1 --2- 172.21.15.72:0/180701069 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x5630c9aaf6d0 0x5630c9ab1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:33.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.427+0000 7f3a57635740 4 client.16287 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T01:46:33.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.427+0000 7f3a57635740 1 -- 172.21.15.72:0/180701069 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5630c999b150 con 0x5630c9aaf6d0 2024-03-22T01:46:33.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.428+0000 7f3a47fff640 1 --2- 172.21.15.72:0/180701069 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x5630c9aaf6d0 0x5630c9ab1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:33.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.428+0000 7f3a47fff640 1 --2- 172.21.15.72:0/180701069 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x5630c9aaf6d0 0x5630c9ab1ac0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2816b3533453fb69 server_cookie=f0d0db5a7ca280d0 in_seq=0 out_seq=0 2024-03-22T01:46:33.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.428+0000 7f3a35ffb640 10 client.16287 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:46:33.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.428+0000 7f3a35ffb640 1 -- 172.21.15.72:0/180701069 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5630c999b150 con 0x5630c9aaf6d0 2024-03-22T01:46:33.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.428+0000 7f3a35ffb640 10 client.16287 handle_command_reply: tid=0 2024-03-22T01:46:33.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.428+0000 7f3a57635740 1 client.16287 shutdown 2024-03-22T01:46:33.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.428+0000 7f3a57635740 1 -- 172.21.15.72:0/180701069 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5630c9a2f410 msgr2=0x5630c9a40d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:33.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.428+0000 7f3a57635740 1 --2- 172.21.15.72:0/180701069 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5630c9a2f410 0x5630c9a40d60 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f3a4003ec90 tx=0x7f3a40005e00 comp rx=0 tx=0).stop 2024-03-22T01:46:33.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.428+0000 7f3a57635740 1 -- 172.21.15.72:0/180701069 shutdown_connections 2024-03-22T01:46:33.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.428+0000 7f3a57635740 1 --2- 172.21.15.72:0/180701069 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x5630c9aaf6d0 0x5630c9ab1ac0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.428+0000 7f3a57635740 1 --2- 172.21.15.72:0/180701069 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5630c9a2fd30 0x5630c9a32280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.428+0000 7f3a35ffb640 0 client.16287 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:46:33.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.428+0000 7f3a57635740 1 --2- 172.21.15.72:0/180701069 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5630c9a38910 0x5630c9a32800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.428+0000 7f3a57635740 1 --2- 172.21.15.72:0/180701069 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5630c9a2f410 0x5630c9a40d60 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.428+0000 7f3a57635740 1 -- 172.21.15.72:0/180701069 >> 172.21.15.72:0/180701069 conn(0x5630c9a29b40 msgr2=0x5630c9a28c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:33.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.428+0000 7f3a57635740 1 -- 172.21.15.72:0/180701069 shutdown_connections 2024-03-22T01:46:33.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.428+0000 7f3a57635740 1 -- 172.21.15.72:0/180701069 wait complete. 2024-03-22T01:46:33.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.428+0000 7f3a57635740 20 client.16287 trim_cache size 0 max 16384 2024-03-22T01:46:33.435 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-22T01:46:33.435 INFO:teuthology.orchestra.run.smithi072.stdout: "status": "no active scrubs running", 2024-03-22T01:46:33.436 INFO:teuthology.orchestra.run.smithi072.stdout: "scrubs": {} 2024-03-22T01:46:33.436 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-22T01:46:33.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.429+0000 7f3a54979640 1 -- 172.21.15.72:0/3311003369 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f3a3c0a3090 msgr2=0x7f3a3c0a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:33.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.429+0000 7f3a54979640 1 --2- 172.21.15.72:0/3311003369 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f3a3c0a3090 0x7f3a3c0a5550 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x5630c9a2eee0 tx=0x7f3a38009530 comp rx=0 tx=0).stop 2024-03-22T01:46:33.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.429+0000 7f3a54979640 1 -- 172.21.15.72:0/3311003369 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3a50106a40 msgr2=0x7f3a5019bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:33.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.429+0000 7f3a54979640 1 --2- 172.21.15.72:0/3311003369 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3a50106a40 0x7f3a5019bbe0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f3a4800ebc0 tx=0x7f3a4800c6a0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.429+0000 7f3a54979640 1 -- 172.21.15.72:0/3311003369 shutdown_connections 2024-03-22T01:46:33.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.429+0000 7f3a54979640 1 --2- 172.21.15.72:0/3311003369 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f3a3c0a3090 0x7f3a3c0a5550 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.429+0000 7f3a54979640 1 --2- 172.21.15.72:0/3311003369 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3a50106a40 0x7f3a5019bbe0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.429+0000 7f3a54979640 1 --2- 172.21.15.72:0/3311003369 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a501013f0 0x7f3a501978b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.429+0000 7f3a54979640 1 --2- 172.21.15.72:0/3311003369 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a50100a30 0x7f3a50197370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.429+0000 7f3a54979640 1 -- 172.21.15.72:0/3311003369 >> 172.21.15.72:0/3311003369 conn(0x7f3a50077c40 msgr2=0x7f3a501032b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:33.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.429+0000 7f3a54979640 1 -- 172.21.15.72:0/3311003369 shutdown_connections 2024-03-22T01:46:33.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.429+0000 7f3a54979640 1 -- 172.21.15.72:0/3311003369 wait complete. 2024-03-22T01:46:33.439 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-22T01:46:33.439 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T01:46:33.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.541+0000 7f10c993e640 1 -- 172.21.15.72:0/690765556 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10c410c140 msgr2=0x7f10c410e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:33.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.541+0000 7f10c993e640 1 --2- 172.21.15.72:0/690765556 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10c410c140 0x7f10c410e5c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f10bc009ca0 tx=0x7f10bc02c180 comp rx=0 tx=0).stop 2024-03-22T01:46:33.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.541+0000 7f10c993e640 1 -- 172.21.15.72:0/690765556 shutdown_connections 2024-03-22T01:46:33.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.541+0000 7f10c993e640 1 --2- 172.21.15.72:0/690765556 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10c410c140 0x7f10c410e5c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.541+0000 7f10c993e640 1 --2- 172.21.15.72:0/690765556 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10c4103690 0x7f10c410bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.541+0000 7f10c993e640 1 --2- 172.21.15.72:0/690765556 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10c4102d70 0x7f10c4103150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.541+0000 7f10c993e640 1 -- 172.21.15.72:0/690765556 >> 172.21.15.72:0/690765556 conn(0x7f10c4077cd0 msgr2=0x7f10c40780e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:33.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.541+0000 7f10c993e640 1 -- 172.21.15.72:0/690765556 shutdown_connections 2024-03-22T01:46:33.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.541+0000 7f10c993e640 1 -- 172.21.15.72:0/690765556 wait complete. 2024-03-22T01:46:33.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.542+0000 7f10c993e640 1 Processor -- start 2024-03-22T01:46:33.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.542+0000 7f10c993e640 1 -- start start 2024-03-22T01:46:33.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.542+0000 7f10c993e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10c4102d70 0x7f10c4198380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:33.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.542+0000 7f10c993e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10c4103690 0x7f10c41988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:33.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.542+0000 7f10c993e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10c410c140 0x7f10c419cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:33.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.542+0000 7f10c993e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f10c4111f20 con 0x7f10c4103690 2024-03-22T01:46:33.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.542+0000 7f10c993e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f10c4111da0 con 0x7f10c410c140 2024-03-22T01:46:33.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.542+0000 7f10c993e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f10c41120a0 con 0x7f10c4102d70 2024-03-22T01:46:33.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.542+0000 7f10bbfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10c4103690 0x7f10c41988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:33.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.542+0000 7f10c2ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10c4102d70 0x7f10c4198380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:33.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.542+0000 7f10bbfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10c4103690 0x7f10c41988c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36260/0 (socket says 172.21.15.72:36260) 2024-03-22T01:46:33.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.542+0000 7f10bbfff640 1 -- 172.21.15.72:0/4132052232 learned_addr learned my addr 172.21.15.72:0/4132052232 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:33.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.542+0000 7f10c37fe640 1 --2- 172.21.15.72:0/4132052232 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10c410c140 0x7f10c419cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:33.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.542+0000 7f10bbfff640 1 -- 172.21.15.72:0/4132052232 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10c4102d70 msgr2=0x7f10c4198380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:33.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.542+0000 7f10bbfff640 1 --2- 172.21.15.72:0/4132052232 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10c4102d70 0x7f10c4198380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.542+0000 7f10bbfff640 1 -- 172.21.15.72:0/4132052232 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10c410c140 msgr2=0x7f10c419cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:33.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.542+0000 7f10bbfff640 1 --2- 172.21.15.72:0/4132052232 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10c410c140 0x7f10c419cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.542+0000 7f10bbfff640 1 -- 172.21.15.72:0/4132052232 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10c419d370 con 0x7f10c4103690 2024-03-22T01:46:33.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.543+0000 7f10c37fe640 1 --2- 172.21.15.72:0/4132052232 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10c410c140 0x7f10c419cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:46:33.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.543+0000 7f10bbfff640 1 --2- 172.21.15.72:0/4132052232 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10c4103690 0x7f10c41988c0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f10ac00e9f0 tx=0x7f10ac00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:33.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.543+0000 7f10c893c640 1 -- 172.21.15.72:0/4132052232 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10ac00cd80 con 0x7f10c4103690 2024-03-22T01:46:33.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.543+0000 7f10c993e640 1 -- 172.21.15.72:0/4132052232 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10c419d660 con 0x7f10c4103690 2024-03-22T01:46:33.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.543+0000 7f10c893c640 1 -- 172.21.15.72:0/4132052232 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10ac010910 con 0x7f10c4103690 2024-03-22T01:46:33.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.543+0000 7f10c893c640 1 -- 172.21.15.72:0/4132052232 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10ac0188f0 con 0x7f10c4103690 2024-03-22T01:46:33.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.543+0000 7f10c993e640 1 -- 172.21.15.72:0/4132052232 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10c41a4ef0 con 0x7f10c4103690 2024-03-22T01:46:33.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.544+0000 7f10c993e640 1 -- 172.21.15.72:0/4132052232 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10c406ab10 con 0x7f10c4103690 2024-03-22T01:46:33.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.545+0000 7f10c893c640 1 -- 172.21.15.72:0/4132052232 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 25) ==== 137985+0+0 (secure 0 0 0) 0x7f10ac010ac0 con 0x7f10c4103690 2024-03-22T01:46:33.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.546+0000 7f10c893c640 1 --2- 172.21.15.72:0/4132052232 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f10b00a3090 0x7f10b00a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:33.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.546+0000 7f10c2ffd640 1 --2- 172.21.15.72:0/4132052232 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f10b00a3090 0x7f10b00a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:33.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.546+0000 7f10c893c640 1 -- 172.21.15.72:0/4132052232 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f10ac0cfec0 con 0x7f10c4103690 2024-03-22T01:46:33.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.546+0000 7f10c2ffd640 1 --2- 172.21.15.72:0/4132052232 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f10b00a3090 0x7f10b00a5550 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f10b4009bd0 tx=0x7f10b4005f90 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:33.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.550+0000 7f10c893c640 1 -- 172.21.15.72:0/4132052232 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f10ac09bfb0 con 0x7f10c4103690 2024-03-22T01:46:33.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.763+0000 7f10c993e640 1 -- 172.21.15.72:0/4132052232 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f10c41996d0 con 0x7f10c4103690 2024-03-22T01:46:33.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.764+0000 7f10c893c640 1 -- 172.21.15.72:0/4132052232 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) ==== 95+0+11791 (secure 0 0 0) 0x7f10ac0a0050 con 0x7f10c4103690 2024-03-22T01:46:33.763 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:46:33.764 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":103,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6835/3077216308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3077216308},{"type":"v1","addr":"172.21.15.72:6835","nonce":3077216308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2460254143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":2460254143},{"type":"v1","addr":"172.21.15.72:6837","nonce":2460254143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6839/2026432819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6838","nonce":2026432819},{"type":"v1","addr":"172.21.15.72:6839","nonce":2026432819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6841/400094545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6840","nonce":400094545},{"type":"v1","addr":"172.21.15.72:6841","nonce":400094545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6833/1578701171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":1578701171},{"type":"v1","addr":"172.21.15.89:6833","nonce":1578701171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/2703047213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2703047213},{"type":"v1","addr":"172.21.15.98:6833","nonce":2703047213}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2589468787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2589468787},{"type":"v1","addr":"172.21.15.89:6835","nonce":2589468787}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/92657636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":92657636},{"type":"v1","addr":"172.21.15.98:6835","nonce":92657636}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1662941782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1662941782},{"type":"v1","addr":"172.21.15.89:6837","nonce":1662941782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24490,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/781860371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":781860371},{"type":"v1","addr":"172.21.15.98:6837","nonce":781860371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":103,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-22T01:34:14.815677+0000","modified":"2024-03-22T01:45:27.515143+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24524},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":171,"addr":"172.21.15.98:6839/3192173650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3192173650},{"type":"v1","addr":"172.21.15.98:6839","nonce":3192173650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":171,"addr":"172.21.15.89:6839/149200280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":149200280},{"type":"v1","addr":"172.21.15.89:6839","nonce":149200280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24524]},"id":1}]} 2024-03-22T01:46:33.764 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 103 2024-03-22T01:46:33.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.766+0000 7f10c993e640 1 -- 172.21.15.72:0/4132052232 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f10b00a3090 msgr2=0x7f10b00a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:33.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.766+0000 7f10c993e640 1 --2- 172.21.15.72:0/4132052232 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f10b00a3090 0x7f10b00a5550 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f10b4009bd0 tx=0x7f10b4005f90 comp rx=0 tx=0).stop 2024-03-22T01:46:33.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.766+0000 7f10c993e640 1 -- 172.21.15.72:0/4132052232 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10c4103690 msgr2=0x7f10c41988c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:33.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.766+0000 7f10c993e640 1 --2- 172.21.15.72:0/4132052232 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10c4103690 0x7f10c41988c0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f10ac00e9f0 tx=0x7f10ac00eeb0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.766+0000 7f10c993e640 1 -- 172.21.15.72:0/4132052232 shutdown_connections 2024-03-22T01:46:33.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.766+0000 7f10c993e640 1 --2- 172.21.15.72:0/4132052232 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f10b00a3090 0x7f10b00a5550 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.766+0000 7f10c993e640 1 --2- 172.21.15.72:0/4132052232 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10c410c140 0x7f10c419cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.766+0000 7f10c993e640 1 --2- 172.21.15.72:0/4132052232 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10c4103690 0x7f10c41988c0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.766+0000 7f10c993e640 1 --2- 172.21.15.72:0/4132052232 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10c4102d70 0x7f10c4198380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.766+0000 7f10c993e640 1 -- 172.21.15.72:0/4132052232 >> 172.21.15.72:0/4132052232 conn(0x7f10c4077cd0 msgr2=0x7f10c40fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:33.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.766+0000 7f10c993e640 1 -- 172.21.15.72:0/4132052232 shutdown_connections 2024-03-22T01:46:33.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.766+0000 7f10c993e640 1 -- 172.21.15.72:0/4132052232 wait complete. 2024-03-22T01:46:33.777 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-22T01:46:33.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.883+0000 7fdc05ec9640 1 -- 172.21.15.72:0/2762226355 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc00100060 msgr2=0x7fdc001004e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:33.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.883+0000 7fdc05ec9640 1 --2- 172.21.15.72:0/2762226355 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc00100060 0x7fdc001004e0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fdbf4009930 tx=0x7fdbf402bc30 comp rx=0 tx=0).stop 2024-03-22T01:46:33.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.883+0000 7fdc05ec9640 1 -- 172.21.15.72:0/2762226355 shutdown_connections 2024-03-22T01:46:33.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.883+0000 7fdc05ec9640 1 --2- 172.21.15.72:0/2762226355 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc00100a20 0x7fdc0010e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.883+0000 7fdc05ec9640 1 --2- 172.21.15.72:0/2762226355 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc00100060 0x7fdc001004e0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.883+0000 7fdc05ec9640 1 --2- 172.21.15.72:0/2762226355 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc00106b60 0x7fdc00106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.883+0000 7fdc05ec9640 1 -- 172.21.15.72:0/2762226355 >> 172.21.15.72:0/2762226355 conn(0x7fdc00076f90 msgr2=0x7fdc000fe310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:33.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.883+0000 7fdc05ec9640 1 -- 172.21.15.72:0/2762226355 shutdown_connections 2024-03-22T01:46:33.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.883+0000 7fdc05ec9640 1 -- 172.21.15.72:0/2762226355 wait complete. 2024-03-22T01:46:33.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.884+0000 7fdc05ec9640 1 Processor -- start 2024-03-22T01:46:33.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.884+0000 7fdc05ec9640 1 -- start start 2024-03-22T01:46:33.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.884+0000 7fdc05ec9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc00100060 0x7fdc001984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:33.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.884+0000 7fdc05ec9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc00100a20 0x7fdc00198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:33.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.884+0000 7fdc05ec9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc00106b60 0x7fdc0019cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:33.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.884+0000 7fdc05ec9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fdc00112020 con 0x7fdc00106b60 2024-03-22T01:46:33.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.884+0000 7fdc05ec9640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fdc00111ea0 con 0x7fdc00100a20 2024-03-22T01:46:33.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.884+0000 7fdc05ec9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdc001121a0 con 0x7fdc00100060 2024-03-22T01:46:33.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.884+0000 7fdbfffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc00106b60 0x7fdc0019cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:33.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.884+0000 7fdbff7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc00100060 0x7fdc001984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:33.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.884+0000 7fdbfeffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc00100a20 0x7fdc00198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:33.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.884+0000 7fdbff7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc00100060 0x7fdc001984e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.72:56722/0 (socket says 172.21.15.72:56722) 2024-03-22T01:46:33.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.884+0000 7fdbfffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc00106b60 0x7fdc0019cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36276/0 (socket says 172.21.15.72:36276) 2024-03-22T01:46:33.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.884+0000 7fdbff7fe640 1 -- 172.21.15.72:0/3031721417 learned_addr learned my addr 172.21.15.72:0/3031721417 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:33.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.884+0000 7fdbff7fe640 1 -- 172.21.15.72:0/3031721417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc00100a20 msgr2=0x7fdc00198a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:33.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.884+0000 7fdbff7fe640 1 --2- 172.21.15.72:0/3031721417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc00100a20 0x7fdc00198a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.884+0000 7fdbff7fe640 1 -- 172.21.15.72:0/3031721417 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc00106b60 msgr2=0x7fdc0019cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:33.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.884+0000 7fdbff7fe640 1 --2- 172.21.15.72:0/3031721417 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc00106b60 0x7fdc0019cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.885+0000 7fdbff7fe640 1 -- 172.21.15.72:0/3031721417 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc0019d4d0 con 0x7fdc00100060 2024-03-22T01:46:33.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.885+0000 7fdbfeffd640 1 --2- 172.21.15.72:0/3031721417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc00100a20 0x7fdc00198a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:46:33.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.885+0000 7fdbfffff640 1 --2- 172.21.15.72:0/3031721417 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc00106b60 0x7fdc0019cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:46:33.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.885+0000 7fdbff7fe640 1 --2- 172.21.15.72:0/3031721417 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc00100060 0x7fdc001984e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fdbf000d770 tx=0x7fdbf000dc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:33.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.885+0000 7fdbfcff9640 1 -- 172.21.15.72:0/3031721417 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf0017070 con 0x7fdc00100060 2024-03-22T01:46:33.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.885+0000 7fdbfcff9640 1 -- 172.21.15.72:0/3031721417 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbf0005500 con 0x7fdc00100060 2024-03-22T01:46:33.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.885+0000 7fdc05ec9640 1 -- 172.21.15.72:0/3031721417 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc0019d7c0 con 0x7fdc00100060 2024-03-22T01:46:33.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.885+0000 7fdbfcff9640 1 -- 172.21.15.72:0/3031721417 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf0004e00 con 0x7fdc00100060 2024-03-22T01:46:33.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.885+0000 7fdc05ec9640 1 -- 172.21.15.72:0/3031721417 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc001a5050 con 0x7fdc00100060 2024-03-22T01:46:33.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.887+0000 7fdbfcff9640 1 -- 172.21.15.72:0/3031721417 <== mon.1 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 137985+0+0 (secure 0 0 0) 0x7fdbf0004fa0 con 0x7fdc00100060 2024-03-22T01:46:33.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.887+0000 7fdc08b8d740 1 Processor -- start 2024-03-22T01:46:33.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.887+0000 7fdc08b8d740 1 -- start start 2024-03-22T01:46:33.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.887+0000 7fdbfcff9640 1 --2- 172.21.15.72:0/3031721417 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fdbe80a3090 0x7fdbe80a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:33.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.887+0000 7fdbfffff640 1 --2- 172.21.15.72:0/3031721417 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fdbe80a3090 0x7fdbe80a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:33.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.887+0000 7fdc08b8d740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560828327800 0x560828327be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:33.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.887+0000 7fdc08b8d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560828328120 0x5608283307c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:33.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.887+0000 7fdbfcff9640 1 -- 172.21.15.72:0/3031721417 <== mon.1 v2:172.21.15.98:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fdbf00cf560 con 0x7fdc00100060 2024-03-22T01:46:33.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.887+0000 7fdc08b8d740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560828330d00 0x5608283330f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:33.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.887+0000 7fdc08b8d740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x56082828b8d0 con 0x560828327800 2024-03-22T01:46:33.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.888+0000 7fdc08b8d740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x560828233e00 con 0x560828330d00 2024-03-22T01:46:33.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.888+0000 7fdc08b8d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560828287880 con 0x560828328120 2024-03-22T01:46:33.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.888+0000 7fdbff7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560828327800 0x560828327be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:33.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.888+0000 7fdbff7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560828327800 0x560828327be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36286/0 (socket says 172.21.15.72:36286) 2024-03-22T01:46:33.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.888+0000 7fdbff7fe640 1 -- 172.21.15.72:0/960777920 learned_addr learned my addr 172.21.15.72:0/960777920 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:33.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.888+0000 7fdbfeffd640 1 --2- 172.21.15.72:0/960777920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560828328120 0x5608283307c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:33.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.888+0000 7fdbfffff640 1 --2- 172.21.15.72:0/960777920 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560828330d00 0x5608283330f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:33.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.888+0000 7fdbff7fe640 1 -- 172.21.15.72:0/960777920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560828328120 msgr2=0x5608283307c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:33.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.888+0000 7fdbff7fe640 1 --2- 172.21.15.72:0/960777920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560828328120 0x5608283307c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.888+0000 7fdbff7fe640 1 -- 172.21.15.72:0/960777920 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560828330d00 msgr2=0x5608283330f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:33.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.888+0000 7fdbff7fe640 1 --2- 172.21.15.72:0/960777920 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560828330d00 0x5608283330f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.888+0000 7fdbff7fe640 1 -- 172.21.15.72:0/960777920 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560827f9f3f0 con 0x560828327800 2024-03-22T01:46:33.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.888+0000 7fdbff7fe640 1 --2- 172.21.15.72:0/960777920 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560828327800 0x560828327be0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fdbf009b5e0 tx=0x7fdbf00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1103ed78eda7e315 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:33.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.888+0000 7fdbfffff640 1 --2- 172.21.15.72:0/3031721417 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fdbe80a3090 0x7fdbe80a5550 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fdbf8009fd0 tx=0x7fdbf800ff60 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:33.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.888+0000 7fdbe5ffb640 1 -- 172.21.15.72:0/960777920 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf0020050 con 0x560828327800 2024-03-22T01:46:33.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.888+0000 7fdbe5ffb640 1 -- 172.21.15.72:0/960777920 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbf009f6f0 con 0x560828327800 2024-03-22T01:46:33.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.888+0000 7fdbe5ffb640 1 -- 172.21.15.72:0/960777920 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf0097cc0 con 0x560828327800 2024-03-22T01:46:33.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.889+0000 7fdc08b8d740 1 -- 172.21.15.72:0/960777920 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560828327800 msgr2=0x560828327be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:33.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.889+0000 7fdc08b8d740 1 --2- 172.21.15.72:0/960777920 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560828327800 0x560828327be0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fdbf009b5e0 tx=0x7fdbf00046c0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.889+0000 7fdc08b8d740 1 -- 172.21.15.72:0/960777920 shutdown_connections 2024-03-22T01:46:33.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.889+0000 7fdc08b8d740 1 --2- 172.21.15.72:0/960777920 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560828330d00 0x5608283330f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.889+0000 7fdc08b8d740 1 --2- 172.21.15.72:0/960777920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560828328120 0x5608283307c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.889+0000 7fdc08b8d740 1 --2- 172.21.15.72:0/960777920 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560828327800 0x560828327be0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.889+0000 7fdc08b8d740 1 -- 172.21.15.72:0/960777920 >> 172.21.15.72:0/960777920 conn(0x560828321f30 msgr2=0x560828322310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:33.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.889+0000 7fdc08b8d740 1 -- 172.21.15.72:0/960777920 shutdown_connections 2024-03-22T01:46:33.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.889+0000 7fdc08b8d740 1 -- 172.21.15.72:0/960777920 wait complete. 2024-03-22T01:46:33.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.889+0000 7fdc08b8d740 1 -- start start 2024-03-22T01:46:33.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.889+0000 7fdc08b8d740 1 Processor -- start 2024-03-22T01:46:33.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.890+0000 7fdc08b8d740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560828327800 0x56082832ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:33.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.890+0000 7fdc08b8d740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560828328120 0x56082832e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:33.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.890+0000 7fdc08b8d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560828330d00 0x56082832e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:33.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.890+0000 7fdc08b8d740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x560828333a90 con 0x560828328120 2024-03-22T01:46:33.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.890+0000 7fdc08b8d740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x560828247300 con 0x560828327800 2024-03-22T01:46:33.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.890+0000 7fdc08b8d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560828333c10 con 0x560828330d00 2024-03-22T01:46:33.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.890+0000 7fdc08b8d740 10 client.0 _ll_register_callbacks cb 0x560828282e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:46:33.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.890+0000 7fdbfeffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560828328120 0x56082832e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:33.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.890+0000 7fdbff7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560828327800 0x56082832ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:33.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.890+0000 7fdbff7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560828327800 0x56082832ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.72:32966/0 (socket says 172.21.15.72:32966) 2024-03-22T01:46:33.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.890+0000 7fdbfeffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560828328120 0x56082832e300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36296/0 (socket says 172.21.15.72:36296) 2024-03-22T01:46:33.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.890+0000 7fdbff7fe640 1 -- 172.21.15.72:0/1047575419 learned_addr learned my addr 172.21.15.72:0/1047575419 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:33.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.890+0000 7fdbfffff640 1 --2- 172.21.15.72:0/1047575419 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560828330d00 0x56082832e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:33.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.890+0000 7fdbff7fe640 1 -- 172.21.15.72:0/1047575419 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560828330d00 msgr2=0x56082832e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:33.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.890+0000 7fdbff7fe640 1 --2- 172.21.15.72:0/1047575419 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560828330d00 0x56082832e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.890+0000 7fdbff7fe640 1 -- 172.21.15.72:0/1047575419 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560828328120 msgr2=0x56082832e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:33.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.890+0000 7fdbff7fe640 1 --2- 172.21.15.72:0/1047575419 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560828328120 0x56082832e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.890+0000 7fdbff7fe640 1 -- 172.21.15.72:0/1047575419 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5608280697d0 con 0x560828327800 2024-03-22T01:46:33.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.890+0000 7fdbfeffd640 1 --2- 172.21.15.72:0/1047575419 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560828328120 0x56082832e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:46:33.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.890+0000 7fdbfffff640 1 --2- 172.21.15.72:0/1047575419 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560828330d00 0x56082832e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:46:33.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.890+0000 7fdbff7fe640 1 --2- 172.21.15.72:0/1047575419 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560828327800 0x56082832ddc0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fdbf000d740 tx=0x7fdbf0005bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7b79f52919f15a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:33.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.890+0000 7fdbdeffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-03-22T01:46:33.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.890+0000 7fdbdeffd640 1 -- 172.21.15.72:0/1047575419 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf00b9060 con 0x560828327800 2024-03-22T01:46:33.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.891+0000 7fdbdeffd640 1 -- 172.21.15.72:0/1047575419 <== mon.2 v2:172.21.15.89:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fdbf0097cc0 con 0x560828327800 2024-03-22T01:46:33.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.891+0000 7fdc08b8d740 1 -- 172.21.15.72:0/1047575419 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdbf009f6f0 con 0x560828327800 2024-03-22T01:46:33.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.891+0000 7fdbdeffd640 1 -- 172.21.15.72:0/1047575419 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7fdbf0084b90 con 0x560828327800 2024-03-22T01:46:33.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.891+0000 7fdc08b8d740 10 client.26138 fetch_fsmap learned FSMap version 103 2024-03-22T01:46:33.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.891+0000 7fdc08b8d740 1 -- 172.21.15.72:0/1047575419 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=4+}) -- 0x56082833fdc0 con 0x560828327800 2024-03-22T01:46:33.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.891+0000 7fdbdeffd640 1 -- 172.21.15.72:0/1047575419 <== mon.2 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbf009f6f0 con 0x560828327800 2024-03-22T01:46:33.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.891+0000 7fdbdeffd640 1 -- 172.21.15.72:0/1047575419 <== mon.2 v2:172.21.15.89:3300/0 5 ==== fsmap(e 103) ==== 8330+0+0 (secure 0 0 0) 0x7fdbf009aaa0 con 0x560828327800 2024-03-22T01:46:33.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.891+0000 7fdc08b8d740 10 client.26138 fetch_fsmap finished waiting for FSMap version 103 2024-03-22T01:46:33.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.891+0000 7fdc08b8d740 10 client.26138 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T01:46:33.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.891+0000 7fdc08b8d740 20 client.26138 populate_metadata read hostname 'smithi072' 2024-03-22T01:46:33.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.891+0000 7fdc08b8d740 1 --2- 172.21.15.72:0/1047575419 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x560828341110 0x560828343500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:33.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.891+0000 7fdc08b8d740 4 client.26138 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T01:46:33.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.891+0000 7fdc08b8d740 1 -- 172.21.15.72:0/1047575419 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560828298e10 con 0x560828341110 2024-03-22T01:46:33.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.891+0000 7fdbfeffd640 1 --2- 172.21.15.72:0/1047575419 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x560828341110 0x560828343500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:33.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.892+0000 7fdbdeffd640 10 client.26138 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:46:33.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.892+0000 7fdbfeffd640 1 --2- 172.21.15.72:0/1047575419 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x560828341110 0x560828343500 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f72fcf0eeaca5706 server_cookie=3c04a8b23bacdbf6 in_seq=0 out_seq=0 2024-03-22T01:46:33.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.892+0000 7fdbdeffd640 1 -- 172.21.15.72:0/1047575419 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560828298e10 con 0x560828341110 2024-03-22T01:46:33.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.892+0000 7fdbdeffd640 10 client.26138 handle_command_reply: tid=0 2024-03-22T01:46:33.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.892+0000 7fdc08b8d740 1 client.26138 shutdown 2024-03-22T01:46:33.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.892+0000 7fdc08b8d740 1 -- 172.21.15.72:0/1047575419 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560828327800 msgr2=0x56082832ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:33.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.892+0000 7fdc08b8d740 1 --2- 172.21.15.72:0/1047575419 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560828327800 0x56082832ddc0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fdbf000d740 tx=0x7fdbf0005bb0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.892+0000 7fdc08b8d740 1 -- 172.21.15.72:0/1047575419 shutdown_connections 2024-03-22T01:46:33.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.892+0000 7fdc08b8d740 1 --2- 172.21.15.72:0/1047575419 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x560828341110 0x560828343500 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.892+0000 7fdc08b8d740 1 --2- 172.21.15.72:0/1047575419 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560828330d00 0x56082832e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.892+0000 7fdbdeffd640 0 client.26138 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:46:33.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.892+0000 7fdc08b8d740 1 --2- 172.21.15.72:0/1047575419 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560828328120 0x56082832e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.892+0000 7fdc08b8d740 1 --2- 172.21.15.72:0/1047575419 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560828327800 0x56082832ddc0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.892+0000 7fdc08b8d740 1 -- 172.21.15.72:0/1047575419 >> 172.21.15.72:0/1047575419 conn(0x560828321f30 msgr2=0x560828323c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:33.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.893+0000 7fdc08b8d740 1 -- 172.21.15.72:0/1047575419 shutdown_connections 2024-03-22T01:46:33.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.893+0000 7fdc08b8d740 1 -- 172.21.15.72:0/1047575419 wait complete. 2024-03-22T01:46:33.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.893+0000 7fdc08b8d740 20 client.26138 trim_cache size 0 max 16384 2024-03-22T01:46:33.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.906+0000 7fdc08b8d740 1 Processor -- start 2024-03-22T01:46:33.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.906+0000 7fdc08b8d740 1 -- start start 2024-03-22T01:46:33.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.906+0000 7fdc08b8d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560828327800 0x5608283304f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:33.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.906+0000 7fdc08b8d740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560828328120 0x56082832e660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:33.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.906+0000 7fdc08b8d740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560828330d00 0x560828346520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:33.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.906+0000 7fdc08b8d740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fdc001ad010 con 0x560828328120 2024-03-22T01:46:33.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.906+0000 7fdc08b8d740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fdc001af890 con 0x560828330d00 2024-03-22T01:46:33.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.906+0000 7fdc08b8d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdc001ad680 con 0x560828327800 2024-03-22T01:46:33.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.906+0000 7fdbfeffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560828328120 0x56082832e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:33.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.906+0000 7fdbfeffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560828328120 0x56082832e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36312/0 (socket says 172.21.15.72:36312) 2024-03-22T01:46:33.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.906+0000 7fdbfeffd640 1 -- 172.21.15.72:0/357814831 learned_addr learned my addr 172.21.15.72:0/357814831 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:33.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.906+0000 7fdbfffff640 1 --2- 172.21.15.72:0/357814831 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560828330d00 0x560828346520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:33.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.906+0000 7fdbff7fe640 1 --2- 172.21.15.72:0/357814831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560828327800 0x5608283304f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:33.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.906+0000 7fdbfeffd640 1 -- 172.21.15.72:0/357814831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560828327800 msgr2=0x5608283304f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:33.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.906+0000 7fdbfeffd640 1 --2- 172.21.15.72:0/357814831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560828327800 0x5608283304f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.906+0000 7fdbfeffd640 1 -- 172.21.15.72:0/357814831 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560828330d00 msgr2=0x560828346520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:33.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.906+0000 7fdbfeffd640 1 --2- 172.21.15.72:0/357814831 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560828330d00 0x560828346520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.906+0000 7fdbfeffd640 1 -- 172.21.15.72:0/357814831 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560828333af0 con 0x560828328120 2024-03-22T01:46:33.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.906+0000 7fdbff7fe640 1 --2- 172.21.15.72:0/357814831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560828327800 0x5608283304f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T01:46:33.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.906+0000 7fdbfeffd640 1 --2- 172.21.15.72:0/357814831 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560828328120 0x56082832e660 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fdbf4009900 tx=0x7fdbf4009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=224bef0d4547673c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:33.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.907+0000 7fdbe57fa640 1 -- 172.21.15.72:0/357814831 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf4004300 con 0x560828328120 2024-03-22T01:46:33.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.907+0000 7fdbe57fa640 1 -- 172.21.15.72:0/357814831 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbf404f070 con 0x560828328120 2024-03-22T01:46:33.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.907+0000 7fdbe57fa640 1 -- 172.21.15.72:0/357814831 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf404a070 con 0x560828328120 2024-03-22T01:46:33.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.907+0000 7fdc08b8d740 1 -- 172.21.15.72:0/357814831 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560828328120 msgr2=0x56082832e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:33.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.907+0000 7fdc08b8d740 1 --2- 172.21.15.72:0/357814831 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560828328120 0x56082832e660 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fdbf4009900 tx=0x7fdbf4009260 comp rx=0 tx=0).stop 2024-03-22T01:46:33.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.907+0000 7fdc08b8d740 1 -- 172.21.15.72:0/357814831 shutdown_connections 2024-03-22T01:46:33.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.907+0000 7fdc08b8d740 1 --2- 172.21.15.72:0/357814831 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560828330d00 0x560828346520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.907+0000 7fdc08b8d740 1 --2- 172.21.15.72:0/357814831 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560828328120 0x56082832e660 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.907+0000 7fdc08b8d740 1 --2- 172.21.15.72:0/357814831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560828327800 0x5608283304f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.907+0000 7fdc08b8d740 1 -- 172.21.15.72:0/357814831 >> 172.21.15.72:0/357814831 conn(0x560828321f30 msgr2=0x560828320740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:33.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.907+0000 7fdc08b8d740 1 -- 172.21.15.72:0/357814831 shutdown_connections 2024-03-22T01:46:33.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.907+0000 7fdc08b8d740 1 -- 172.21.15.72:0/357814831 wait complete. 2024-03-22T01:46:33.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.908+0000 7fdc08b8d740 1 -- start start 2024-03-22T01:46:33.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.908+0000 7fdc08b8d740 1 Processor -- start 2024-03-22T01:46:33.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.908+0000 7fdc08b8d740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560828327800 0x560828344260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:33.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.908+0000 7fdc08b8d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560828328120 0x5608283447a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:33.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.908+0000 7fdc08b8d740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560828330d00 0x560828344d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:33.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.908+0000 7fdc08b8d740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fdc001add50 con 0x560828327800 2024-03-22T01:46:33.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.908+0000 7fdc08b8d740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fdc001af1c0 con 0x560828330d00 2024-03-22T01:46:33.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.908+0000 7fdc08b8d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdc001ae420 con 0x560828328120 2024-03-22T01:46:33.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.908+0000 7fdc08b8d740 10 client.0 _ll_register_callbacks cb 0x7fdbf009f070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:46:33.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.908+0000 7fdbff7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560828327800 0x560828344260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:33.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.908+0000 7fdbff7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560828327800 0x560828344260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36326/0 (socket says 172.21.15.72:36326) 2024-03-22T01:46:33.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.908+0000 7fdbff7fe640 1 -- 172.21.15.72:0/231634333 learned_addr learned my addr 172.21.15.72:0/231634333 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:33.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.908+0000 7fdbfffff640 1 --2- 172.21.15.72:0/231634333 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560828330d00 0x560828344d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:33.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.908+0000 7fdbfeffd640 1 --2- 172.21.15.72:0/231634333 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560828328120 0x5608283447a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:33.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.908+0000 7fdbff7fe640 1 -- 172.21.15.72:0/231634333 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560828328120 msgr2=0x5608283447a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:33.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.908+0000 7fdbff7fe640 1 --2- 172.21.15.72:0/231634333 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560828328120 0x5608283447a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.908+0000 7fdbff7fe640 1 -- 172.21.15.72:0/231634333 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560828330d00 msgr2=0x560828344d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:33.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.908+0000 7fdbff7fe640 1 --2- 172.21.15.72:0/231634333 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560828330d00 0x560828344d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.908+0000 7fdbff7fe640 1 -- 172.21.15.72:0/231634333 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5608283a63a0 con 0x560828327800 2024-03-22T01:46:33.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.908+0000 7fdbfeffd640 1 --2- 172.21.15.72:0/231634333 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560828328120 0x5608283447a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T01:46:33.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.908+0000 7fdbff7fe640 1 --2- 172.21.15.72:0/231634333 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560828327800 0x560828344260 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fdbf0098020 tx=0x7fdbf00972b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfeacd1cc0d88725 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:33.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.908+0000 7fdbdeffd640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:46:33.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.909+0000 7fdbdeffd640 1 -- 172.21.15.72:0/231634333 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf0020040 con 0x560828327800 2024-03-22T01:46:33.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.909+0000 7fdbdeffd640 1 -- 172.21.15.72:0/231634333 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fdbf009aaa0 con 0x560828327800 2024-03-22T01:46:33.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.909+0000 7fdc08b8d740 1 -- 172.21.15.72:0/231634333 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdbf404f070 con 0x560828327800 2024-03-22T01:46:33.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.909+0000 7fdbdeffd640 1 -- 172.21.15.72:0/231634333 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7fdbf00b9040 con 0x560828327800 2024-03-22T01:46:33.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.909+0000 7fdc08b8d740 10 client.16311 fetch_fsmap learned FSMap version 103 2024-03-22T01:46:33.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.909+0000 7fdc08b8d740 1 -- 172.21.15.72:0/231634333 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=4+}) -- 0x5608283a63a0 con 0x560828327800 2024-03-22T01:46:33.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.909+0000 7fdbdeffd640 1 -- 172.21.15.72:0/231634333 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbf404f070 con 0x560828327800 2024-03-22T01:46:33.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.909+0000 7fdbdeffd640 1 -- 172.21.15.72:0/231634333 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 103) ==== 8330+0+0 (secure 0 0 0) 0x7fdbf009f120 con 0x560828327800 2024-03-22T01:46:33.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.909+0000 7fdc08b8d740 10 client.16311 fetch_fsmap finished waiting for FSMap version 103 2024-03-22T01:46:33.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.909+0000 7fdc08b8d740 10 client.16311 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T01:46:33.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.909+0000 7fdc08b8d740 20 client.16311 populate_metadata read hostname 'smithi072' 2024-03-22T01:46:33.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.909+0000 7fdc08b8d740 1 --2- 172.21.15.72:0/231634333 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x5608283a79f0 0x5608283a9e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:33.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.909+0000 7fdc08b8d740 4 client.16311 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T01:46:33.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.909+0000 7fdc08b8d740 1 -- 172.21.15.72:0/231634333 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560828293160 con 0x5608283a79f0 2024-03-22T01:46:33.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.909+0000 7fdbfeffd640 1 --2- 172.21.15.72:0/231634333 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x5608283a79f0 0x5608283a9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:33.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.910+0000 7fdbdeffd640 10 client.16311 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:46:33.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.910+0000 7fdbfeffd640 1 --2- 172.21.15.72:0/231634333 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x5608283a79f0 0x5608283a9e10 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe661050c3de0180 server_cookie=56f5b860e0624ed6 in_seq=0 out_seq=0 2024-03-22T01:46:33.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.910+0000 7fdbdeffd640 1 -- 172.21.15.72:0/231634333 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560828293160 con 0x5608283a79f0 2024-03-22T01:46:33.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.910+0000 7fdbdeffd640 10 client.16311 handle_command_reply: tid=0 2024-03-22T01:46:33.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.910+0000 7fdc08b8d740 1 client.16311 shutdown 2024-03-22T01:46:33.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.910+0000 7fdc08b8d740 1 -- 172.21.15.72:0/231634333 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560828327800 msgr2=0x560828344260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:33.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.910+0000 7fdc08b8d740 1 --2- 172.21.15.72:0/231634333 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560828327800 0x560828344260 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fdbf0098020 tx=0x7fdbf00972b0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.910+0000 7fdc08b8d740 1 -- 172.21.15.72:0/231634333 shutdown_connections 2024-03-22T01:46:33.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.910+0000 7fdc08b8d740 1 --2- 172.21.15.72:0/231634333 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x5608283a79f0 0x5608283a9e10 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.910+0000 7fdc08b8d740 1 --2- 172.21.15.72:0/231634333 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560828330d00 0x560828344d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.910+0000 7fdbdeffd640 0 client.16311 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:46:33.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.910+0000 7fdc08b8d740 1 --2- 172.21.15.72:0/231634333 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560828328120 0x5608283447a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.910+0000 7fdc08b8d740 1 --2- 172.21.15.72:0/231634333 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560828327800 0x560828344260 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.910+0000 7fdc08b8d740 1 -- 172.21.15.72:0/231634333 >> 172.21.15.72:0/231634333 conn(0x560828321f30 msgr2=0x56082833be60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:33.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.910+0000 7fdc08b8d740 1 -- 172.21.15.72:0/231634333 shutdown_connections 2024-03-22T01:46:33.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.910+0000 7fdc08b8d740 1 -- 172.21.15.72:0/231634333 wait complete. 2024-03-22T01:46:33.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.910+0000 7fdc08b8d740 20 client.16311 trim_cache size 0 max 16384 2024-03-22T01:46:33.921 INFO:teuthology.orchestra.run.smithi072.stdout:[] 2024-03-22T01:46:33.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.911+0000 7fdc05ec9640 1 -- 172.21.15.72:0/3031721417 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fdbe80a3090 msgr2=0x7fdbe80a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:33.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.911+0000 7fdc05ec9640 1 --2- 172.21.15.72:0/3031721417 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fdbe80a3090 0x7fdbe80a5550 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fdbf8009fd0 tx=0x7fdbf800ff60 comp rx=0 tx=0).stop 2024-03-22T01:46:33.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.911+0000 7fdc05ec9640 1 -- 172.21.15.72:0/3031721417 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc00100060 msgr2=0x7fdc001984e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:33.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.911+0000 7fdc05ec9640 1 --2- 172.21.15.72:0/3031721417 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc00100060 0x7fdc001984e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fdbf000d770 tx=0x7fdbf000dc30 comp rx=0 tx=0).stop 2024-03-22T01:46:33.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.911+0000 7fdc05ec9640 1 -- 172.21.15.72:0/3031721417 shutdown_connections 2024-03-22T01:46:33.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.911+0000 7fdc05ec9640 1 --2- 172.21.15.72:0/3031721417 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fdbe80a3090 0x7fdbe80a5550 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.911+0000 7fdc05ec9640 1 --2- 172.21.15.72:0/3031721417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc00100a20 0x7fdc00198a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.911+0000 7fdc05ec9640 1 --2- 172.21.15.72:0/3031721417 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc00106b60 0x7fdc0019cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.911+0000 7fdc05ec9640 1 --2- 172.21.15.72:0/3031721417 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc00100060 0x7fdc001984e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:33.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.911+0000 7fdc05ec9640 1 -- 172.21.15.72:0/3031721417 >> 172.21.15.72:0/3031721417 conn(0x7fdc00076f90 msgr2=0x7fdc00104b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:33.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.911+0000 7fdc05ec9640 1 -- 172.21.15.72:0/3031721417 shutdown_connections 2024-03-22T01:46:33.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:33.911+0000 7fdc05ec9640 1 -- 172.21.15.72:0/3031721417 wait complete. 2024-03-22T01:46:34.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:46:34 smithi089 ceph-mon[35770]: pgmap v300: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 14 KiB/s wr, 190 op/s 2024-03-22T01:46:34.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:46:34 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/4132052232' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:46:34.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:46:34 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:46:34.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:46:34 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:46:34.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:46:34 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:46:34.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:46:34 smithi072 ceph-mon[32414]: pgmap v300: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 14 KiB/s wr, 190 op/s 2024-03-22T01:46:34.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:46:34 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/4132052232' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:46:34.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:46:34 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:46:34.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:46:34 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:46:34.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:46:34 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:46:34.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:46:34 smithi098 ceph-mon[36468]: pgmap v300: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 14 KiB/s wr, 190 op/s 2024-03-22T01:46:34.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:46:34 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/4132052232' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:46:34.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:46:34 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:46:34.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:46:34 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:46:34.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:46:34 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:46:34.925 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-22T01:46:34.925 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T01:46:34.925 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-22T01:46:35.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.043+0000 7fd9d1aaf640 1 -- 172.21.15.72:0/1165366243 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9cc101b40 msgr2=0x7fd9cc101fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:35.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.043+0000 7fd9d1aaf640 1 --2- 172.21.15.72:0/1165366243 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9cc101b40 0x7fd9cc101fc0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc009900 tx=0x7fd9bc02bc30 comp rx=0 tx=0).stop 2024-03-22T01:46:35.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.043+0000 7fd9d1aaf640 1 -- 172.21.15.72:0/1165366243 shutdown_connections 2024-03-22T01:46:35.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.043+0000 7fd9d1aaf640 1 --2- 172.21.15.72:0/1165366243 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9cc102500 0x7fd9cc10e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.043+0000 7fd9d1aaf640 1 --2- 172.21.15.72:0/1165366243 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9cc101b40 0x7fd9cc101fc0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.043+0000 7fd9d1aaf640 1 --2- 172.21.15.72:0/1165366243 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9cc107c00 0x7fd9cc107fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.043+0000 7fd9d1aaf640 1 -- 172.21.15.72:0/1165366243 >> 172.21.15.72:0/1165366243 conn(0x7fd9cc0fd630 msgr2=0x7fd9cc0ffa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:35.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.043+0000 7fd9d1aaf640 1 -- 172.21.15.72:0/1165366243 shutdown_connections 2024-03-22T01:46:35.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.043+0000 7fd9d1aaf640 1 -- 172.21.15.72:0/1165366243 wait complete. 2024-03-22T01:46:35.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.043+0000 7fd9d1aaf640 1 Processor -- start 2024-03-22T01:46:35.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.044+0000 7fd9d1aaf640 1 -- start start 2024-03-22T01:46:35.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.044+0000 7fd9d1aaf640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9cc101b40 0x7fd9cc198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:35.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.044+0000 7fd9d1aaf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9cc102500 0x7fd9cc198940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:35.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.044+0000 7fd9d1aaf640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9cc107c00 0x7fd9cc19cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:35.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.044+0000 7fd9d1aaf640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fd9cc111f40 con 0x7fd9cc101b40 2024-03-22T01:46:35.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.044+0000 7fd9d1aaf640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd9cc111dc0 con 0x7fd9cc107c00 2024-03-22T01:46:35.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.044+0000 7fd9d1aaf640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd9cc1120c0 con 0x7fd9cc102500 2024-03-22T01:46:35.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.044+0000 7fd9cb7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9cc101b40 0x7fd9cc198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:35.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.044+0000 7fd9cb7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9cc101b40 0x7fd9cc198400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36348/0 (socket says 172.21.15.72:36348) 2024-03-22T01:46:35.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.044+0000 7fd9cb7fe640 1 -- 172.21.15.72:0/3783061027 learned_addr learned my addr 172.21.15.72:0/3783061027 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:35.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.044+0000 7fd9cbfff640 1 --2- 172.21.15.72:0/3783061027 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9cc107c00 0x7fd9cc19cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:35.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.044+0000 7fd9caffd640 1 --2- 172.21.15.72:0/3783061027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9cc102500 0x7fd9cc198940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:35.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.044+0000 7fd9cb7fe640 1 -- 172.21.15.72:0/3783061027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9cc102500 msgr2=0x7fd9cc198940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:35.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.044+0000 7fd9cb7fe640 1 --2- 172.21.15.72:0/3783061027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9cc102500 0x7fd9cc198940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.044+0000 7fd9cb7fe640 1 -- 172.21.15.72:0/3783061027 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9cc107c00 msgr2=0x7fd9cc19cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:35.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.044+0000 7fd9cb7fe640 1 --2- 172.21.15.72:0/3783061027 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9cc107c00 0x7fd9cc19cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.044+0000 7fd9cb7fe640 1 -- 172.21.15.72:0/3783061027 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9cc19d3a0 con 0x7fd9cc101b40 2024-03-22T01:46:35.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.044+0000 7fd9cb7fe640 1 --2- 172.21.15.72:0/3783061027 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9cc101b40 0x7fd9cc198400 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fd9c000c970 tx=0x7fd9c000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:35.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.045+0000 7fd9c8ff9640 1 -- 172.21.15.72:0/3783061027 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9c0007bb0 con 0x7fd9cc101b40 2024-03-22T01:46:35.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.045+0000 7fd9d1aaf640 1 -- 172.21.15.72:0/3783061027 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9cc19d690 con 0x7fd9cc101b40 2024-03-22T01:46:35.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.045+0000 7fd9c8ff9640 1 -- 172.21.15.72:0/3783061027 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd9c0007d50 con 0x7fd9cc101b40 2024-03-22T01:46:35.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.045+0000 7fd9c8ff9640 1 -- 172.21.15.72:0/3783061027 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9c000f450 con 0x7fd9cc101b40 2024-03-22T01:46:35.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.045+0000 7fd9d1aaf640 1 -- 172.21.15.72:0/3783061027 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9cc1a4f20 con 0x7fd9cc101b40 2024-03-22T01:46:35.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.047+0000 7fd9c8ff9640 1 -- 172.21.15.72:0/3783061027 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 25) ==== 137985+0+0 (secure 0 0 0) 0x7fd9c0005020 con 0x7fd9cc101b40 2024-03-22T01:46:35.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.047+0000 7fd9d4771740 1 Processor -- start 2024-03-22T01:46:35.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.047+0000 7fd9d4771740 1 -- start start 2024-03-22T01:46:35.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.047+0000 7fd9d4771740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5624c08b3ba0 0x5624c08b3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:35.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.047+0000 7fd9d4771740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624c08b44c0 0x5624c08bcb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:35.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.047+0000 7fd9d4771740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5624c08bd0a0 0x5624c08bf490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:35.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.047+0000 7fd9d4771740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x5624c0817570 con 0x5624c08b3ba0 2024-03-22T01:46:35.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.047+0000 7fd9d4771740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5624c07bfe10 con 0x5624c08bd0a0 2024-03-22T01:46:35.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.047+0000 7fd9d4771740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5624c0813520 con 0x5624c08b44c0 2024-03-22T01:46:35.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.047+0000 7fd9c8ff9640 1 --2- 172.21.15.72:0/3783061027 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fd9b40a3090 0x7fd9b40a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:35.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.047+0000 7fd9cbfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5624c08b3ba0 0x5624c08b3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:35.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.047+0000 7fd9cbfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5624c08b3ba0 0x5624c08b3f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36360/0 (socket says 172.21.15.72:36360) 2024-03-22T01:46:35.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.047+0000 7fd9caffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5624c08bd0a0 0x5624c08bf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:35.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.047+0000 7fd9cbfff640 1 -- 172.21.15.72:0/74145204 learned_addr learned my addr 172.21.15.72:0/74145204 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:35.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.048+0000 7fd9cb7fe640 1 --2- 172.21.15.72:0/74145204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624c08b44c0 0x5624c08bcb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:35.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.048+0000 7fd9c8ff9640 1 -- 172.21.15.72:0/3783061027 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fd9c00cf8c0 con 0x7fd9cc101b40 2024-03-22T01:46:35.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.048+0000 7fd9cbfff640 1 --2- 172.21.15.72:0/3783061027 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fd9b40a3090 0x7fd9b40a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:35.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.048+0000 7fd9caffd640 1 -- 172.21.15.72:0/74145204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624c08b44c0 msgr2=0x5624c08bcb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:35.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.048+0000 7fd9caffd640 1 --2- 172.21.15.72:0/74145204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624c08b44c0 0x5624c08bcb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.048+0000 7fd9caffd640 1 -- 172.21.15.72:0/74145204 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5624c08b3ba0 msgr2=0x5624c08b3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:35.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.048+0000 7fd9caffd640 1 --2- 172.21.15.72:0/74145204 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5624c08b3ba0 0x5624c08b3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.048+0000 7fd9caffd640 1 -- 172.21.15.72:0/74145204 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624c05f37e0 con 0x5624c08bd0a0 2024-03-22T01:46:35.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.048+0000 7fd9cbfff640 1 --2- 172.21.15.72:0/74145204 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5624c08b3ba0 0x5624c08b3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:46:35.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.048+0000 7fd9caffd640 1 --2- 172.21.15.72:0/74145204 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5624c08bd0a0 0x5624c08bf490 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc000c00 tx=0x7fd9bc02c5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c8f76be2b3e3ab4 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:35.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.048+0000 7fd9cbfff640 1 --2- 172.21.15.72:0/3783061027 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fd9b40a3090 0x7fd9b40a5550 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd9c4009fd0 tx=0x7fd9c4004ab0 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:35.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.048+0000 7fd9b1ffb640 1 -- 172.21.15.72:0/74145204 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9bc042070 con 0x5624c08bd0a0 2024-03-22T01:46:35.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.048+0000 7fd9b1ffb640 1 -- 172.21.15.72:0/74145204 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd9bc004240 con 0x5624c08bd0a0 2024-03-22T01:46:35.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.048+0000 7fd9b1ffb640 1 -- 172.21.15.72:0/74145204 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9bc004ec0 con 0x5624c08bd0a0 2024-03-22T01:46:35.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.049+0000 7fd9d4771740 1 -- 172.21.15.72:0/74145204 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5624c08bd0a0 msgr2=0x5624c08bf490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:35.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.049+0000 7fd9d4771740 1 --2- 172.21.15.72:0/74145204 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5624c08bd0a0 0x5624c08bf490 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc000c00 tx=0x7fd9bc02c5d0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.049+0000 7fd9d4771740 1 -- 172.21.15.72:0/74145204 shutdown_connections 2024-03-22T01:46:35.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.049+0000 7fd9d4771740 1 --2- 172.21.15.72:0/74145204 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5624c08bd0a0 0x5624c08bf490 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.049+0000 7fd9d4771740 1 --2- 172.21.15.72:0/74145204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624c08b44c0 0x5624c08bcb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.049+0000 7fd9d4771740 1 --2- 172.21.15.72:0/74145204 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5624c08b3ba0 0x5624c08b3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.049+0000 7fd9d4771740 1 -- 172.21.15.72:0/74145204 >> 172.21.15.72:0/74145204 conn(0x5624c08ae2d0 msgr2=0x5624c08ae6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:35.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.049+0000 7fd9d4771740 1 -- 172.21.15.72:0/74145204 shutdown_connections 2024-03-22T01:46:35.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.049+0000 7fd9d4771740 1 -- 172.21.15.72:0/74145204 wait complete. 2024-03-22T01:46:35.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.049+0000 7fd9d4771740 1 -- start start 2024-03-22T01:46:35.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.050+0000 7fd9d4771740 1 Processor -- start 2024-03-22T01:46:35.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.050+0000 7fd9d4771740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5624c08b3ba0 0x5624c08bca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:35.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.050+0000 7fd9d4771740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5624c08b44c0 0x5624c08b9d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:35.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.050+0000 7fd9d4771740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624c08bd0a0 0x5624c08ba2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:35.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.050+0000 7fd9d4771740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x5624c08bfe30 con 0x5624c08b3ba0 2024-03-22T01:46:35.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.050+0000 7fd9d4771740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5624c07d3110 con 0x5624c08b44c0 2024-03-22T01:46:35.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.050+0000 7fd9d4771740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5624c08bffb0 con 0x5624c08bd0a0 2024-03-22T01:46:35.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.050+0000 7fd9d4771740 10 client.0 _ll_register_callbacks cb 0x5624c080fbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:46:35.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.050+0000 7fd9cb7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5624c08b3ba0 0x5624c08bca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:35.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.050+0000 7fd9cb7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5624c08b3ba0 0x5624c08bca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36368/0 (socket says 172.21.15.72:36368) 2024-03-22T01:46:35.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.050+0000 7fd9cb7fe640 1 -- 172.21.15.72:0/4203030456 learned_addr learned my addr 172.21.15.72:0/4203030456 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:35.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.050+0000 7fd9caffd640 1 --2- 172.21.15.72:0/4203030456 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5624c08b44c0 0x5624c08b9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:35.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.050+0000 7fd9cbfff640 1 --2- 172.21.15.72:0/4203030456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624c08bd0a0 0x5624c08ba2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:35.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.050+0000 7fd9cb7fe640 1 -- 172.21.15.72:0/4203030456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624c08bd0a0 msgr2=0x5624c08ba2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:35.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.050+0000 7fd9cb7fe640 1 --2- 172.21.15.72:0/4203030456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624c08bd0a0 0x5624c08ba2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.050+0000 7fd9cb7fe640 1 -- 172.21.15.72:0/4203030456 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5624c08b44c0 msgr2=0x5624c08b9d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:35.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.050+0000 7fd9cb7fe640 1 --2- 172.21.15.72:0/4203030456 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5624c08b44c0 0x5624c08b9d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.050+0000 7fd9cb7fe640 1 -- 172.21.15.72:0/4203030456 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624c0653f70 con 0x5624c08b3ba0 2024-03-22T01:46:35.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.050+0000 7fd9cb7fe640 1 --2- 172.21.15.72:0/4203030456 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5624c08b3ba0 0x5624c08bca00 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fd9c00cd260 tx=0x7fd9c006b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a80df3f337e096d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:35.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.050+0000 7fd9aaffd640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:46:35.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.051+0000 7fd9aaffd640 1 -- 172.21.15.72:0/4203030456 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9c001d040 con 0x5624c08b3ba0 2024-03-22T01:46:35.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.051+0000 7fd9aaffd640 1 -- 172.21.15.72:0/4203030456 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fd9c0005310 con 0x5624c08b3ba0 2024-03-22T01:46:35.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.051+0000 7fd9d4771740 1 -- 172.21.15.72:0/4203030456 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd9bc004240 con 0x5624c08b3ba0 2024-03-22T01:46:35.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.051+0000 7fd9aaffd640 1 -- 172.21.15.72:0/4203030456 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7fd9c009b9b0 con 0x5624c08b3ba0 2024-03-22T01:46:35.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.051+0000 7fd9d4771740 10 client.16323 fetch_fsmap learned FSMap version 103 2024-03-22T01:46:35.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.051+0000 7fd9d4771740 1 -- 172.21.15.72:0/4203030456 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=4+}) -- 0x5624c04eb170 con 0x5624c08b3ba0 2024-03-22T01:46:35.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.051+0000 7fd9aaffd640 1 -- 172.21.15.72:0/4203030456 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd9bc004240 con 0x5624c08b3ba0 2024-03-22T01:46:35.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.051+0000 7fd9aaffd640 1 -- 172.21.15.72:0/4203030456 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 103) ==== 8330+0+0 (secure 0 0 0) 0x7fd9c00860b0 con 0x5624c08b3ba0 2024-03-22T01:46:35.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.051+0000 7fd9d4771740 10 client.16323 fetch_fsmap finished waiting for FSMap version 103 2024-03-22T01:46:35.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.051+0000 7fd9d4771740 10 client.16323 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T01:46:35.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.051+0000 7fd9d4771740 20 client.16323 populate_metadata read hostname 'smithi072' 2024-03-22T01:46:35.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.051+0000 7fd9d4771740 1 --2- 172.21.15.72:0/4203030456 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x5624c08ccfb0 0x5624c08cf400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:35.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.051+0000 7fd9d4771740 4 client.16323 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T01:46:35.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.051+0000 7fd9d4771740 1 -- 172.21.15.72:0/4203030456 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5624c0824ab0 con 0x5624c08ccfb0 2024-03-22T01:46:35.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.051+0000 7fd9caffd640 1 --2- 172.21.15.72:0/4203030456 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x5624c08ccfb0 0x5624c08cf400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:35.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.052+0000 7fd9aaffd640 10 client.16323 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:46:35.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.052+0000 7fd9caffd640 1 --2- 172.21.15.72:0/4203030456 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x5624c08ccfb0 0x5624c08cf400 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e09bfd82bffabbc server_cookie=fea5e2e97f14eb56 in_seq=0 out_seq=0 2024-03-22T01:46:35.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.052+0000 7fd9aaffd640 1 -- 172.21.15.72:0/4203030456 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5624c0824ab0 con 0x5624c08ccfb0 2024-03-22T01:46:35.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.052+0000 7fd9aaffd640 10 client.16323 handle_command_reply: tid=0 2024-03-22T01:46:35.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.052+0000 7fd9d4771740 1 client.16323 shutdown 2024-03-22T01:46:35.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.052+0000 7fd9d4771740 1 -- 172.21.15.72:0/4203030456 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5624c08b3ba0 msgr2=0x5624c08bca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:35.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.052+0000 7fd9d4771740 1 --2- 172.21.15.72:0/4203030456 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5624c08b3ba0 0x5624c08bca00 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fd9c00cd260 tx=0x7fd9c006b480 comp rx=0 tx=0).stop 2024-03-22T01:46:35.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.053+0000 7fd9d4771740 1 -- 172.21.15.72:0/4203030456 shutdown_connections 2024-03-22T01:46:35.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.053+0000 7fd9d4771740 1 --2- 172.21.15.72:0/4203030456 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x5624c08ccfb0 0x5624c08cf400 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.053+0000 7fd9aaffd640 0 client.16323 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:46:35.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.053+0000 7fd9d4771740 1 --2- 172.21.15.72:0/4203030456 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5624c08b44c0 0x5624c08b9d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.053+0000 7fd9d4771740 1 --2- 172.21.15.72:0/4203030456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624c08bd0a0 0x5624c08ba2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.053+0000 7fd9d4771740 1 --2- 172.21.15.72:0/4203030456 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5624c08b3ba0 0x5624c08bca00 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.053+0000 7fd9d4771740 1 -- 172.21.15.72:0/4203030456 >> 172.21.15.72:0/4203030456 conn(0x5624c08ae2d0 msgr2=0x5624c08b0020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:35.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.053+0000 7fd9d4771740 1 -- 172.21.15.72:0/4203030456 shutdown_connections 2024-03-22T01:46:35.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.053+0000 7fd9d4771740 1 -- 172.21.15.72:0/4203030456 wait complete. 2024-03-22T01:46:35.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.053+0000 7fd9d4771740 20 client.16323 trim_cache size 0 max 16384 2024-03-22T01:46:35.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.070+0000 7fd9d4771740 1 Processor -- start 2024-03-22T01:46:35.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.070+0000 7fd9d4771740 1 -- start start 2024-03-22T01:46:35.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.070+0000 7fd9d4771740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5624c08b3ba0 0x5624c08c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:35.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.070+0000 7fd9d4771740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5624c08b44c0 0x5624c08c8c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:35.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.070+0000 7fd9d4771740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624c08bd0a0 0x5624c08d4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:35.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.070+0000 7fd9d4771740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fd9cc1acf70 con 0x5624c08b44c0 2024-03-22T01:46:35.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.070+0000 7fd9d4771740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd9cc1af7f0 con 0x5624c08b3ba0 2024-03-22T01:46:35.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.070+0000 7fd9d4771740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd9cc1ad5e0 con 0x5624c08bd0a0 2024-03-22T01:46:35.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.070+0000 7fd9caffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5624c08b44c0 0x5624c08c8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:35.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.070+0000 7fd9caffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5624c08b44c0 0x5624c08c8c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36384/0 (socket says 172.21.15.72:36384) 2024-03-22T01:46:35.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.070+0000 7fd9caffd640 1 -- 172.21.15.72:0/1218348214 learned_addr learned my addr 172.21.15.72:0/1218348214 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:35.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.070+0000 7fd9cb7fe640 1 --2- 172.21.15.72:0/1218348214 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5624c08b3ba0 0x5624c08c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:35.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.070+0000 7fd9cbfff640 1 --2- 172.21.15.72:0/1218348214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624c08bd0a0 0x5624c08d4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:35.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.071+0000 7fd9caffd640 1 -- 172.21.15.72:0/1218348214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624c08bd0a0 msgr2=0x5624c08d4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:35.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.071+0000 7fd9caffd640 1 --2- 172.21.15.72:0/1218348214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624c08bd0a0 0x5624c08d4ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.071+0000 7fd9caffd640 1 -- 172.21.15.72:0/1218348214 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5624c08b3ba0 msgr2=0x5624c08c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:35.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.071+0000 7fd9caffd640 1 --2- 172.21.15.72:0/1218348214 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5624c08b3ba0 0x5624c08c5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.071+0000 7fd9caffd640 1 -- 172.21.15.72:0/1218348214 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624c08b3570 con 0x5624c08b44c0 2024-03-22T01:46:35.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.071+0000 7fd9cbfff640 1 --2- 172.21.15.72:0/1218348214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624c08bd0a0 0x5624c08d4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:46:35.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.071+0000 7fd9caffd640 1 --2- 172.21.15.72:0/1218348214 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5624c08b44c0 0x5624c08c8c00 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc0096c0 tx=0x7fd9bc004a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8806ef14de914e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:35.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.071+0000 7fd9b17fa640 1 -- 172.21.15.72:0/1218348214 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9bc056070 con 0x5624c08b44c0 2024-03-22T01:46:35.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.071+0000 7fd9b17fa640 1 -- 172.21.15.72:0/1218348214 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd9bc051b60 con 0x5624c08b44c0 2024-03-22T01:46:35.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.071+0000 7fd9b17fa640 1 -- 172.21.15.72:0/1218348214 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9bc04c040 con 0x5624c08b44c0 2024-03-22T01:46:35.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.071+0000 7fd9d4771740 1 -- 172.21.15.72:0/1218348214 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5624c08b44c0 msgr2=0x5624c08c8c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:35.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.071+0000 7fd9d4771740 1 --2- 172.21.15.72:0/1218348214 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5624c08b44c0 0x5624c08c8c00 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc0096c0 tx=0x7fd9bc004a20 comp rx=0 tx=0).stop 2024-03-22T01:46:35.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.072+0000 7fd9d4771740 1 -- 172.21.15.72:0/1218348214 shutdown_connections 2024-03-22T01:46:35.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.072+0000 7fd9d4771740 1 --2- 172.21.15.72:0/1218348214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624c08bd0a0 0x5624c08d4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.072+0000 7fd9d4771740 1 --2- 172.21.15.72:0/1218348214 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5624c08b44c0 0x5624c08c8c00 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.072+0000 7fd9d4771740 1 --2- 172.21.15.72:0/1218348214 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5624c08b3ba0 0x5624c08c5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.072+0000 7fd9d4771740 1 -- 172.21.15.72:0/1218348214 >> 172.21.15.72:0/1218348214 conn(0x5624c08ae2d0 msgr2=0x5624c08aff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:35.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.072+0000 7fd9d4771740 1 -- 172.21.15.72:0/1218348214 shutdown_connections 2024-03-22T01:46:35.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.072+0000 7fd9d4771740 1 -- 172.21.15.72:0/1218348214 wait complete. 2024-03-22T01:46:35.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.072+0000 7fd9d4771740 1 -- start start 2024-03-22T01:46:35.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.072+0000 7fd9d4771740 1 Processor -- start 2024-03-22T01:46:35.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.072+0000 7fd9d4771740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624c08b3ba0 0x5624c092f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:35.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.072+0000 7fd9d4771740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5624c08b44c0 0x5624c092fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:35.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.072+0000 7fd9d4771740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5624c08bd0a0 0x5624c092b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:35.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.072+0000 7fd9d4771740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fd9cc1adcb0 con 0x5624c08b44c0 2024-03-22T01:46:35.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.072+0000 7fd9d4771740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd9cc1af120 con 0x5624c08bd0a0 2024-03-22T01:46:35.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.072+0000 7fd9d4771740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd9cc1ae380 con 0x5624c08b3ba0 2024-03-22T01:46:35.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.072+0000 7fd9d4771740 10 client.0 _ll_register_callbacks cb 0x7fd9c0004920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:46:35.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.072+0000 7fd9caffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5624c08b44c0 0x5624c092fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:35.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.072+0000 7fd9caffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5624c08b44c0 0x5624c092fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36390/0 (socket says 172.21.15.72:36390) 2024-03-22T01:46:35.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.072+0000 7fd9caffd640 1 -- 172.21.15.72:0/1343774658 learned_addr learned my addr 172.21.15.72:0/1343774658 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:35.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.073+0000 7fd9cb7fe640 1 --2- 172.21.15.72:0/1343774658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624c08b3ba0 0x5624c092f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:35.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.073+0000 7fd9cbfff640 1 --2- 172.21.15.72:0/1343774658 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5624c08bd0a0 0x5624c092b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:35.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.073+0000 7fd9caffd640 1 -- 172.21.15.72:0/1343774658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624c08b3ba0 msgr2=0x5624c092f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:35.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.073+0000 7fd9caffd640 1 --2- 172.21.15.72:0/1343774658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624c08b3ba0 0x5624c092f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.073+0000 7fd9caffd640 1 -- 172.21.15.72:0/1343774658 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5624c08bd0a0 msgr2=0x5624c092b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:35.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.073+0000 7fd9caffd640 1 --2- 172.21.15.72:0/1343774658 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5624c08bd0a0 0x5624c092b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.073+0000 7fd9caffd640 1 -- 172.21.15.72:0/1343774658 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624c092ba90 con 0x5624c08b44c0 2024-03-22T01:46:35.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.073+0000 7fd9cbfff640 1 --2- 172.21.15.72:0/1343774658 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5624c08bd0a0 0x5624c092b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:46:35.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.073+0000 7fd9caffd640 1 --2- 172.21.15.72:0/1343774658 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5624c08b44c0 0x5624c092fbe0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc050000 tx=0x7fd9bc02c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c20349e643c765fb server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:35.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.073+0000 7fd9aaffd640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:46:35.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.073+0000 7fd9aaffd640 1 -- 172.21.15.72:0/1343774658 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9bc056070 con 0x5624c08b44c0 2024-03-22T01:46:35.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.073+0000 7fd9aaffd640 1 -- 172.21.15.72:0/1343774658 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fd9bc051380 con 0x5624c08b44c0 2024-03-22T01:46:35.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.073+0000 7fd9d4771740 1 -- 172.21.15.72:0/1343774658 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd9bc051b60 con 0x5624c08b44c0 2024-03-22T01:46:35.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.073+0000 7fd9aaffd640 1 -- 172.21.15.72:0/1343774658 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7fd9bc063050 con 0x5624c08b44c0 2024-03-22T01:46:35.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.073+0000 7fd9d4771740 10 client.16329 fetch_fsmap learned FSMap version 103 2024-03-22T01:46:35.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.073+0000 7fd9d4771740 1 -- 172.21.15.72:0/1343774658 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=4+}) -- 0x5624c0937e70 con 0x5624c08b44c0 2024-03-22T01:46:35.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.073+0000 7fd9aaffd640 1 -- 172.21.15.72:0/1343774658 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd9bc051b60 con 0x5624c08b44c0 2024-03-22T01:46:35.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.073+0000 7fd9aaffd640 1 -- 172.21.15.72:0/1343774658 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 103) ==== 8330+0+0 (secure 0 0 0) 0x7fd9bc04f070 con 0x5624c08b44c0 2024-03-22T01:46:35.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.074+0000 7fd9d4771740 10 client.16329 fetch_fsmap finished waiting for FSMap version 103 2024-03-22T01:46:35.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.074+0000 7fd9d4771740 10 client.16329 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T01:46:35.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.074+0000 7fd9d4771740 20 client.16329 populate_metadata read hostname 'smithi072' 2024-03-22T01:46:35.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.074+0000 7fd9d4771740 1 --2- 172.21.15.72:0/1343774658 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x5624c08ccfb0 0x5624c093a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:35.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.074+0000 7fd9d4771740 4 client.16329 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-22T01:46:35.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.074+0000 7fd9d4771740 1 -- 172.21.15.72:0/1343774658 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5624c081ee00 con 0x5624c08ccfb0 2024-03-22T01:46:35.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.074+0000 7fd9cb7fe640 1 --2- 172.21.15.72:0/1343774658 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x5624c08ccfb0 0x5624c093a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:35.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.074+0000 7fd9cb7fe640 1 --2- 172.21.15.72:0/1343774658 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x5624c08ccfb0 0x5624c093a1e0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=15298999df6bebe0 server_cookie=6858c3d195de8256 in_seq=0 out_seq=0 2024-03-22T01:46:35.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.074+0000 7fd9aaffd640 10 client.16329 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:46:35.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.075+0000 7fd9aaffd640 1 -- 172.21.15.72:0/1343774658 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5624c081ee00 con 0x5624c08ccfb0 2024-03-22T01:46:35.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.075+0000 7fd9aaffd640 10 client.16329 handle_command_reply: tid=0 2024-03-22T01:46:35.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.075+0000 7fd9d4771740 1 client.16329 shutdown 2024-03-22T01:46:35.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.076+0000 7fd9d4771740 1 -- 172.21.15.72:0/1343774658 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5624c08b44c0 msgr2=0x5624c092fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:35.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.076+0000 7fd9d4771740 1 --2- 172.21.15.72:0/1343774658 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5624c08b44c0 0x5624c092fbe0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc050000 tx=0x7fd9bc02c330 comp rx=0 tx=0).stop 2024-03-22T01:46:35.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.076+0000 7fd9d4771740 1 -- 172.21.15.72:0/1343774658 shutdown_connections 2024-03-22T01:46:35.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.076+0000 7fd9d4771740 1 --2- 172.21.15.72:0/1343774658 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x5624c08ccfb0 0x5624c093a1e0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.076+0000 7fd9d4771740 1 --2- 172.21.15.72:0/1343774658 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5624c08bd0a0 0x5624c092b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.076+0000 7fd9d4771740 1 --2- 172.21.15.72:0/1343774658 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5624c08b44c0 0x5624c092fbe0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.076+0000 7fd9d4771740 1 --2- 172.21.15.72:0/1343774658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624c08b3ba0 0x5624c092f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.076+0000 7fd9d4771740 1 -- 172.21.15.72:0/1343774658 >> 172.21.15.72:0/1343774658 conn(0x5624c08ae2d0 msgr2=0x5624c08af6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:35.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.076+0000 7fd9aaffd640 0 client.16329 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:46:35.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.076+0000 7fd9d4771740 1 -- 172.21.15.72:0/1343774658 shutdown_connections 2024-03-22T01:46:35.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.076+0000 7fd9d4771740 1 -- 172.21.15.72:0/1343774658 wait complete. 2024-03-22T01:46:35.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.076+0000 7fd9d4771740 20 client.16329 trim_cache size 0 max 16384 2024-03-22T01:46:35.083 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-22T01:46:35.084 INFO:teuthology.orchestra.run.smithi072.stdout: "return_code": 0, 2024-03-22T01:46:35.084 INFO:teuthology.orchestra.run.smithi072.stdout: "scrub_tag": "d726e926-2c0e-4707-b9c6-039e034b7714", 2024-03-22T01:46:35.084 INFO:teuthology.orchestra.run.smithi072.stdout: "mode": "asynchronous" 2024-03-22T01:46:35.084 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-22T01:46:35.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.076+0000 7fd9d1aaf640 1 -- 172.21.15.72:0/3783061027 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fd9b40a3090 msgr2=0x7fd9b40a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:35.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.076+0000 7fd9d1aaf640 1 --2- 172.21.15.72:0/3783061027 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fd9b40a3090 0x7fd9b40a5550 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd9c4009fd0 tx=0x7fd9c4004ab0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.076+0000 7fd9d1aaf640 1 -- 172.21.15.72:0/3783061027 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9cc101b40 msgr2=0x7fd9cc198400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:35.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.076+0000 7fd9d1aaf640 1 --2- 172.21.15.72:0/3783061027 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9cc101b40 0x7fd9cc198400 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fd9c000c970 tx=0x7fd9c000ce30 comp rx=0 tx=0).stop 2024-03-22T01:46:35.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.076+0000 7fd9d1aaf640 1 -- 172.21.15.72:0/3783061027 shutdown_connections 2024-03-22T01:46:35.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.076+0000 7fd9d1aaf640 1 --2- 172.21.15.72:0/3783061027 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fd9b40a3090 0x7fd9b40a5550 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.076+0000 7fd9d1aaf640 1 --2- 172.21.15.72:0/3783061027 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9cc107c00 0x7fd9cc19cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.076+0000 7fd9d1aaf640 1 --2- 172.21.15.72:0/3783061027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9cc102500 0x7fd9cc198940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.076+0000 7fd9d1aaf640 1 --2- 172.21.15.72:0/3783061027 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9cc101b40 0x7fd9cc198400 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.076+0000 7fd9d1aaf640 1 -- 172.21.15.72:0/3783061027 >> 172.21.15.72:0/3783061027 conn(0x7fd9cc0fd630 msgr2=0x7fd9cc0feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:35.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.077+0000 7fd9d1aaf640 1 -- 172.21.15.72:0/3783061027 shutdown_connections 2024-03-22T01:46:35.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.077+0000 7fd9d1aaf640 1 -- 172.21.15.72:0/3783061027 wait complete. 2024-03-22T01:46:35.088 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T01:46:35.088 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T01:46:35.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.193+0000 7f85e140e640 1 -- 172.21.15.72:0/2470455899 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f85dc1013f0 msgr2=0x7f85dc10d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:35.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.193+0000 7f85e140e640 1 --2- 172.21.15.72:0/2470455899 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f85dc1013f0 0x7f85dc10d900 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f85d4009cd0 tx=0x7f85d402c0f0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.193+0000 7f85e140e640 1 -- 172.21.15.72:0/2470455899 shutdown_connections 2024-03-22T01:46:35.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.193+0000 7f85e140e640 1 --2- 172.21.15.72:0/2470455899 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f85dc1013f0 0x7f85dc10d900 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.193+0000 7f85e140e640 1 --2- 172.21.15.72:0/2470455899 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85dc100a30 0x7f85dc100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.193+0000 7f85e140e640 1 --2- 172.21.15.72:0/2470455899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85dc106a40 0x7f85dc106e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.193+0000 7f85e140e640 1 -- 172.21.15.72:0/2470455899 >> 172.21.15.72:0/2470455899 conn(0x7f85dc077c40 msgr2=0x7f85dc078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:35.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.193+0000 7f85e140e640 1 -- 172.21.15.72:0/2470455899 shutdown_connections 2024-03-22T01:46:35.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.193+0000 7f85e140e640 1 -- 172.21.15.72:0/2470455899 wait complete. 2024-03-22T01:46:35.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.194+0000 7f85e140e640 1 Processor -- start 2024-03-22T01:46:35.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.194+0000 7f85e140e640 1 -- start start 2024-03-22T01:46:35.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.194+0000 7f85e140e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85dc100a30 0x7f85dc197330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:35.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.194+0000 7f85e140e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f85dc1013f0 0x7f85dc197870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:35.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.194+0000 7f85e140e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85dc106a40 0x7f85dc19bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:35.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.194+0000 7f85e140e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f85dc110e50 con 0x7f85dc1013f0 2024-03-22T01:46:35.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.194+0000 7f85e140e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f85dc110cd0 con 0x7f85dc100a30 2024-03-22T01:46:35.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.194+0000 7f85e140e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f85dc110fd0 con 0x7f85dc106a40 2024-03-22T01:46:35.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.194+0000 7f85d27fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f85dc1013f0 0x7f85dc197870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:35.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.194+0000 7f85daffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85dc100a30 0x7f85dc197330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:35.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.194+0000 7f85d27fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f85dc1013f0 0x7f85dc197870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36412/0 (socket says 172.21.15.72:36412) 2024-03-22T01:46:35.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.194+0000 7f85d27fc640 1 -- 172.21.15.72:0/426137279 learned_addr learned my addr 172.21.15.72:0/426137279 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:35.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.194+0000 7f85db7fe640 1 --2- 172.21.15.72:0/426137279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85dc106a40 0x7f85dc19bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:35.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.194+0000 7f85d27fc640 1 -- 172.21.15.72:0/426137279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85dc106a40 msgr2=0x7f85dc19bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:35.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.194+0000 7f85d27fc640 1 --2- 172.21.15.72:0/426137279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85dc106a40 0x7f85dc19bba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.194+0000 7f85d27fc640 1 -- 172.21.15.72:0/426137279 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85dc100a30 msgr2=0x7f85dc197330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:35.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.194+0000 7f85d27fc640 1 --2- 172.21.15.72:0/426137279 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85dc100a30 0x7f85dc197330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.195+0000 7f85d27fc640 1 -- 172.21.15.72:0/426137279 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85dc19c2d0 con 0x7f85dc1013f0 2024-03-22T01:46:35.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.195+0000 7f85daffd640 1 --2- 172.21.15.72:0/426137279 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85dc100a30 0x7f85dc197330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:46:35.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.195+0000 7f85d27fc640 1 --2- 172.21.15.72:0/426137279 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f85dc1013f0 0x7f85dc197870 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f85c400e9f0 tx=0x7f85c400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:35.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.195+0000 7f85d8ff9640 1 -- 172.21.15.72:0/426137279 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85c400cd80 con 0x7f85dc1013f0 2024-03-22T01:46:35.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.195+0000 7f85e140e640 1 -- 172.21.15.72:0/426137279 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85dc19c5c0 con 0x7f85dc1013f0 2024-03-22T01:46:35.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.195+0000 7f85d8ff9640 1 -- 172.21.15.72:0/426137279 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85c4005500 con 0x7f85dc1013f0 2024-03-22T01:46:35.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.195+0000 7f85d8ff9640 1 -- 172.21.15.72:0/426137279 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85c4010690 con 0x7f85dc1013f0 2024-03-22T01:46:35.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.195+0000 7f85e140e640 1 -- 172.21.15.72:0/426137279 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85dc1a3e50 con 0x7f85dc1013f0 2024-03-22T01:46:35.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.197+0000 7f85d8ff9640 1 -- 172.21.15.72:0/426137279 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 25) ==== 137985+0+0 (secure 0 0 0) 0x7f85c40046c0 con 0x7f85dc1013f0 2024-03-22T01:46:35.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.197+0000 7f85e3ffb740 1 Processor -- start 2024-03-22T01:46:35.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.197+0000 7f85d8ff9640 1 --2- 172.21.15.72:0/426137279 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f85c80a3090 0x7f85c80a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:35.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.197+0000 7f85e3ffb740 1 -- start start 2024-03-22T01:46:35.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.198+0000 7f85db7fe640 1 --2- 172.21.15.72:0/426137279 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f85c80a3090 0x7f85c80a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:35.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.198+0000 7f85d8ff9640 1 -- 172.21.15.72:0/426137279 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f85c40d0030 con 0x7f85dc1013f0 2024-03-22T01:46:35.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.198+0000 7f85e3ffb740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x562013cf1410 0x562013cf17f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:35.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.198+0000 7f85e3ffb740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562013cf1d30 0x562013cfa3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:35.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.198+0000 7f85e3ffb740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562013cfa910 0x562013cfcd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:35.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.198+0000 7f85e3ffb740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x562013c558c0 con 0x562013cf1410 2024-03-22T01:46:35.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.198+0000 7f85e3ffb740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x562013bfdc20 con 0x562013cfa910 2024-03-22T01:46:35.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.198+0000 7f85e3ffb740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562013c51870 con 0x562013cf1d30 2024-03-22T01:46:35.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.198+0000 7f85daffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x562013cf1410 0x562013cf17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:35.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.198+0000 7f85daffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x562013cf1410 0x562013cf17f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36416/0 (socket says 172.21.15.72:36416) 2024-03-22T01:46:35.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.198+0000 7f85daffd640 1 -- 172.21.15.72:0/293165345 learned_addr learned my addr 172.21.15.72:0/293165345 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:35.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.198+0000 7f85db7fe640 1 --2- 172.21.15.72:0/293165345 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562013cfa910 0x562013cfcd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:35.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.198+0000 7f85d27fc640 1 --2- 172.21.15.72:0/293165345 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562013cf1d30 0x562013cfa3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:35.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.198+0000 7f85db7fe640 1 --2- 172.21.15.72:0/426137279 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f85c80a3090 0x7f85c80a5550 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f85d4006010 tx=0x7f85d4005d80 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:35.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.198+0000 7f85daffd640 1 -- 172.21.15.72:0/293165345 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562013cf1d30 msgr2=0x562013cfa3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:35.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.198+0000 7f85daffd640 1 --2- 172.21.15.72:0/293165345 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562013cf1d30 0x562013cfa3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.198+0000 7f85daffd640 1 -- 172.21.15.72:0/293165345 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562013cfa910 msgr2=0x562013cfcd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:35.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.198+0000 7f85daffd640 1 --2- 172.21.15.72:0/293165345 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562013cfa910 0x562013cfcd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.198+0000 7f85daffd640 1 -- 172.21.15.72:0/293165345 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562013c22150 con 0x562013cf1410 2024-03-22T01:46:35.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.198+0000 7f85db7fe640 1 --2- 172.21.15.72:0/293165345 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562013cfa910 0x562013cfcd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:46:35.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.199+0000 7f85daffd640 1 --2- 172.21.15.72:0/293165345 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x562013cf1410 0x562013cf17f0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f85cc0114c0 tx=0x7f85cc011980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2980d8b46e23af8c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:35.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.199+0000 7f85d17fa640 1 -- 172.21.15.72:0/293165345 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85cc00f5c0 con 0x562013cf1410 2024-03-22T01:46:35.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.199+0000 7f85d17fa640 1 -- 172.21.15.72:0/293165345 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85cc00fc40 con 0x562013cf1410 2024-03-22T01:46:35.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.199+0000 7f85d17fa640 1 -- 172.21.15.72:0/293165345 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85cc019cc0 con 0x562013cf1410 2024-03-22T01:46:35.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.199+0000 7f85e3ffb740 1 -- 172.21.15.72:0/293165345 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x562013cf1410 msgr2=0x562013cf17f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:35.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.199+0000 7f85e3ffb740 1 --2- 172.21.15.72:0/293165345 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x562013cf1410 0x562013cf17f0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f85cc0114c0 tx=0x7f85cc011980 comp rx=0 tx=0).stop 2024-03-22T01:46:35.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.199+0000 7f85e3ffb740 1 -- 172.21.15.72:0/293165345 shutdown_connections 2024-03-22T01:46:35.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.199+0000 7f85e3ffb740 1 --2- 172.21.15.72:0/293165345 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562013cfa910 0x562013cfcd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.200+0000 7f85e3ffb740 1 --2- 172.21.15.72:0/293165345 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562013cf1d30 0x562013cfa3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.200+0000 7f85e3ffb740 1 --2- 172.21.15.72:0/293165345 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x562013cf1410 0x562013cf17f0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.200+0000 7f85e3ffb740 1 -- 172.21.15.72:0/293165345 >> 172.21.15.72:0/293165345 conn(0x562013cebb40 msgr2=0x562013cebf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:35.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.200+0000 7f85e3ffb740 1 -- 172.21.15.72:0/293165345 shutdown_connections 2024-03-22T01:46:35.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.200+0000 7f85e3ffb740 1 -- 172.21.15.72:0/293165345 wait complete. 2024-03-22T01:46:35.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.200+0000 7f85e3ffb740 1 -- start start 2024-03-22T01:46:35.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.200+0000 7f85e3ffb740 1 Processor -- start 2024-03-22T01:46:35.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.200+0000 7f85e3ffb740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562013cf1410 0x562013cf78e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:35.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.200+0000 7f85e3ffb740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x562013cf1d30 0x562013cf7e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:35.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.200+0000 7f85e3ffb740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562013cfa910 0x562013cf8360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:35.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.200+0000 7f85e3ffb740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x562013cfd700 con 0x562013cf1d30 2024-03-22T01:46:35.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.200+0000 7f85e3ffb740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x562013c11130 con 0x562013cfa910 2024-03-22T01:46:35.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.200+0000 7f85e3ffb740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562013cfd880 con 0x562013cf1410 2024-03-22T01:46:35.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.200+0000 7f85e3ffb740 10 client.0 _ll_register_callbacks cb 0x562013c10b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:46:35.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.200+0000 7f85d27fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x562013cf1d30 0x562013cf7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:35.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.200+0000 7f85d27fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x562013cf1d30 0x562013cf7e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36418/0 (socket says 172.21.15.72:36418) 2024-03-22T01:46:35.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.200+0000 7f85db7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562013cfa910 0x562013cf8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:35.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.200+0000 7f85d27fc640 1 -- 172.21.15.72:0/878513771 learned_addr learned my addr 172.21.15.72:0/878513771 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:35.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.200+0000 7f85daffd640 1 --2- 172.21.15.72:0/878513771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562013cf1410 0x562013cf78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:35.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.201+0000 7f85db7fe640 1 -- 172.21.15.72:0/878513771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562013cf1410 msgr2=0x562013cf78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:35.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.201+0000 7f85db7fe640 1 --2- 172.21.15.72:0/878513771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562013cf1410 0x562013cf78e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.201+0000 7f85db7fe640 1 -- 172.21.15.72:0/878513771 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x562013cf1d30 msgr2=0x562013cf7e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:35.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.201+0000 7f85db7fe640 1 --2- 172.21.15.72:0/878513771 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x562013cf1d30 0x562013cf7e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.201+0000 7f85db7fe640 1 -- 172.21.15.72:0/878513771 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562013a337d0 con 0x562013cfa910 2024-03-22T01:46:35.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.201+0000 7f85d27fc640 1 --2- 172.21.15.72:0/878513771 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x562013cf1d30 0x562013cf7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:46:35.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.201+0000 7f85db7fe640 1 --2- 172.21.15.72:0/878513771 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562013cfa910 0x562013cf8360 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f85d402c630 tx=0x7f85d402c0f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8aa9817bd243f9f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:35.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.201+0000 7f85c27fc640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-03-22T01:46:35.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.201+0000 7f85c27fc640 1 -- 172.21.15.72:0/878513771 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85d404d020 con 0x562013cfa910 2024-03-22T01:46:35.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.201+0000 7f85e3ffb740 1 -- 172.21.15.72:0/878513771 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f85cc00fc40 con 0x562013cfa910 2024-03-22T01:46:35.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.201+0000 7f85c27fc640 1 -- 172.21.15.72:0/878513771 <== mon.2 v2:172.21.15.89:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f85d4003f80 con 0x562013cfa910 2024-03-22T01:46:35.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.201+0000 7f85c27fc640 1 -- 172.21.15.72:0/878513771 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7f85d4058050 con 0x562013cfa910 2024-03-22T01:46:35.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.201+0000 7f85e3ffb740 10 client.26174 fetch_fsmap learned FSMap version 103 2024-03-22T01:46:35.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.201+0000 7f85e3ffb740 1 -- 172.21.15.72:0/878513771 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=4+}) -- 0x5620139693f0 con 0x562013cfa910 2024-03-22T01:46:35.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.201+0000 7f85c27fc640 1 -- 172.21.15.72:0/878513771 <== mon.2 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85cc00fc40 con 0x562013cfa910 2024-03-22T01:46:35.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.202+0000 7f85c27fc640 1 -- 172.21.15.72:0/878513771 <== mon.2 v2:172.21.15.89:3300/0 5 ==== fsmap(e 103) ==== 8330+0+0 (secure 0 0 0) 0x7f85d403f980 con 0x562013cfa910 2024-03-22T01:46:35.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.202+0000 7f85e3ffb740 10 client.26174 fetch_fsmap finished waiting for FSMap version 103 2024-03-22T01:46:35.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.202+0000 7f85e3ffb740 10 client.26174 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T01:46:35.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.202+0000 7f85e3ffb740 20 client.26174 populate_metadata read hostname 'smithi072' 2024-03-22T01:46:35.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.202+0000 7f85e3ffb740 1 --2- 172.21.15.72:0/878513771 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x562013d0ac20 0x562013d0d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:35.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.202+0000 7f85e3ffb740 4 client.26174 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T01:46:35.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.202+0000 7f85e3ffb740 1 -- 172.21.15.72:0/878513771 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562013c62e00 con 0x562013d0ac20 2024-03-22T01:46:35.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.202+0000 7f85daffd640 1 --2- 172.21.15.72:0/878513771 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x562013d0ac20 0x562013d0d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:35.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.202+0000 7f85c27fc640 10 client.26174 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:46:35.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.202+0000 7f85daffd640 1 --2- 172.21.15.72:0/878513771 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x562013d0ac20 0x562013d0d030 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=58b4fc88418c3a8c server_cookie=6234bd60e028cc5f in_seq=0 out_seq=0 2024-03-22T01:46:35.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.203+0000 7f85c27fc640 1 -- 172.21.15.72:0/878513771 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562013c62e00 con 0x562013d0ac20 2024-03-22T01:46:35.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.203+0000 7f85c27fc640 10 client.26174 handle_command_reply: tid=0 2024-03-22T01:46:35.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.203+0000 7f85e3ffb740 1 client.26174 shutdown 2024-03-22T01:46:35.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.203+0000 7f85e3ffb740 1 -- 172.21.15.72:0/878513771 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562013cfa910 msgr2=0x562013cf8360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:35.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.203+0000 7f85e3ffb740 1 --2- 172.21.15.72:0/878513771 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562013cfa910 0x562013cf8360 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f85d402c630 tx=0x7f85d402c0f0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.203+0000 7f85e3ffb740 1 -- 172.21.15.72:0/878513771 shutdown_connections 2024-03-22T01:46:35.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.203+0000 7f85e3ffb740 1 --2- 172.21.15.72:0/878513771 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x562013d0ac20 0x562013d0d030 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.203+0000 7f85e3ffb740 1 --2- 172.21.15.72:0/878513771 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562013cfa910 0x562013cf8360 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.203+0000 7f85c27fc640 0 client.26174 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:46:35.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.203+0000 7f85e3ffb740 1 --2- 172.21.15.72:0/878513771 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x562013cf1d30 0x562013cf7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.203+0000 7f85e3ffb740 1 --2- 172.21.15.72:0/878513771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562013cf1410 0x562013cf78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.203+0000 7f85e3ffb740 1 -- 172.21.15.72:0/878513771 >> 172.21.15.72:0/878513771 conn(0x562013cebb40 msgr2=0x562013ced890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:35.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.203+0000 7f85e3ffb740 1 -- 172.21.15.72:0/878513771 shutdown_connections 2024-03-22T01:46:35.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.203+0000 7f85e3ffb740 1 -- 172.21.15.72:0/878513771 wait complete. 2024-03-22T01:46:35.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.203+0000 7f85e3ffb740 20 client.26174 trim_cache size 0 max 16384 2024-03-22T01:46:35.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.220+0000 7f85e3ffb740 1 Processor -- start 2024-03-22T01:46:35.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.220+0000 7f85e3ffb740 1 -- start start 2024-03-22T01:46:35.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.221+0000 7f85e3ffb740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562013cf1410 0x562013d064a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:35.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.221+0000 7f85e3ffb740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562013cf1d30 0x562013d12900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:35.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.221+0000 7f85e3ffb740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x562013cfa910 0x562013d02ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:35.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.221+0000 7f85e3ffb740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f85dc1abef0 con 0x562013cfa910 2024-03-22T01:46:35.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.221+0000 7f85e3ffb740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f85dc1ae770 con 0x562013cf1410 2024-03-22T01:46:35.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.221+0000 7f85e3ffb740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f85dc1ac560 con 0x562013cf1d30 2024-03-22T01:46:35.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.221+0000 7f85db7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x562013cfa910 0x562013d02ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:35.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.221+0000 7f85db7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x562013cfa910 0x562013d02ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36426/0 (socket says 172.21.15.72:36426) 2024-03-22T01:46:35.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.221+0000 7f85daffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562013cf1410 0x562013d064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:35.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.221+0000 7f85db7fe640 1 -- 172.21.15.72:0/453552902 learned_addr learned my addr 172.21.15.72:0/453552902 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:35.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.221+0000 7f85d27fc640 1 --2- 172.21.15.72:0/453552902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562013cf1d30 0x562013d12900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:35.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.221+0000 7f85db7fe640 1 -- 172.21.15.72:0/453552902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562013cf1d30 msgr2=0x562013d12900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:35.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.221+0000 7f85db7fe640 1 --2- 172.21.15.72:0/453552902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562013cf1d30 0x562013d12900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.221+0000 7f85db7fe640 1 -- 172.21.15.72:0/453552902 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562013cf1410 msgr2=0x562013d064a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:35.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.221+0000 7f85db7fe640 1 --2- 172.21.15.72:0/453552902 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562013cf1410 0x562013d064a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.221+0000 7f85db7fe640 1 -- 172.21.15.72:0/453552902 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5620139290f0 con 0x562013cfa910 2024-03-22T01:46:35.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.221+0000 7f85d27fc640 1 --2- 172.21.15.72:0/453552902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562013cf1d30 0x562013d12900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T01:46:35.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.221+0000 7f85daffd640 1 --2- 172.21.15.72:0/453552902 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562013cf1410 0x562013d064a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:46:35.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.221+0000 7f85db7fe640 1 --2- 172.21.15.72:0/453552902 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x562013cfa910 0x562013d02ac0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f85d402f020 tx=0x7f85d403ff90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6878f41597df421f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:35.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.221+0000 7f85d0ff9640 1 -- 172.21.15.72:0/453552902 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85d404d040 con 0x562013cfa910 2024-03-22T01:46:35.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.221+0000 7f85d0ff9640 1 -- 172.21.15.72:0/453552902 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85d403f930 con 0x562013cfa910 2024-03-22T01:46:35.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.221+0000 7f85d0ff9640 1 -- 172.21.15.72:0/453552902 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85d4044040 con 0x562013cfa910 2024-03-22T01:46:35.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.222+0000 7f85e3ffb740 1 -- 172.21.15.72:0/453552902 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x562013cfa910 msgr2=0x562013d02ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:35.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.222+0000 7f85e3ffb740 1 --2- 172.21.15.72:0/453552902 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x562013cfa910 0x562013d02ac0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f85d402f020 tx=0x7f85d403ff90 comp rx=0 tx=0).stop 2024-03-22T01:46:35.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.222+0000 7f85e3ffb740 1 -- 172.21.15.72:0/453552902 shutdown_connections 2024-03-22T01:46:35.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.222+0000 7f85e3ffb740 1 --2- 172.21.15.72:0/453552902 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x562013cfa910 0x562013d02ac0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.222+0000 7f85e3ffb740 1 --2- 172.21.15.72:0/453552902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562013cf1d30 0x562013d12900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.222+0000 7f85e3ffb740 1 --2- 172.21.15.72:0/453552902 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562013cf1410 0x562013d064a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.222+0000 7f85e3ffb740 1 -- 172.21.15.72:0/453552902 >> 172.21.15.72:0/453552902 conn(0x562013cebb40 msgr2=0x562013cf78a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:35.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.222+0000 7f85e3ffb740 1 -- 172.21.15.72:0/453552902 shutdown_connections 2024-03-22T01:46:35.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.222+0000 7f85e3ffb740 1 -- 172.21.15.72:0/453552902 wait complete. 2024-03-22T01:46:35.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.222+0000 7f85e3ffb740 1 -- start start 2024-03-22T01:46:35.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.222+0000 7f85e3ffb740 1 Processor -- start 2024-03-22T01:46:35.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.223+0000 7f85e3ffb740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562013cf1410 0x562013cfe4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:35.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.223+0000 7f85e3ffb740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x562013cf1d30 0x562013cfea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:35.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.223+0000 7f85e3ffb740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562013cfa910 0x562013cfef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:35.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.223+0000 7f85e3ffb740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f85dc1acc30 con 0x562013cf1d30 2024-03-22T01:46:35.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.223+0000 7f85e3ffb740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f85dc1ae0a0 con 0x562013cfa910 2024-03-22T01:46:35.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.223+0000 7f85e3ffb740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f85dc1ad300 con 0x562013cf1410 2024-03-22T01:46:35.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.223+0000 7f85e3ffb740 10 client.0 _ll_register_callbacks cb 0x7f85c4005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:46:35.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.223+0000 7f85d27fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x562013cf1d30 0x562013cfea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:35.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.223+0000 7f85d27fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x562013cf1d30 0x562013cfea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36430/0 (socket says 172.21.15.72:36430) 2024-03-22T01:46:35.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.223+0000 7f85d27fc640 1 -- 172.21.15.72:0/2966504641 learned_addr learned my addr 172.21.15.72:0/2966504641 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:35.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.223+0000 7f85db7fe640 1 --2- 172.21.15.72:0/2966504641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562013cfa910 0x562013cfef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:35.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.223+0000 7f85daffd640 1 --2- 172.21.15.72:0/2966504641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562013cf1410 0x562013cfe4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:35.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.223+0000 7f85d27fc640 1 -- 172.21.15.72:0/2966504641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562013cf1410 msgr2=0x562013cfe4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:35.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.223+0000 7f85d27fc640 1 --2- 172.21.15.72:0/2966504641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562013cf1410 0x562013cfe4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.223+0000 7f85d27fc640 1 -- 172.21.15.72:0/2966504641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562013cfa910 msgr2=0x562013cfef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:35.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.223+0000 7f85d27fc640 1 --2- 172.21.15.72:0/2966504641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562013cfa910 0x562013cfef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.223+0000 7f85d27fc640 1 -- 172.21.15.72:0/2966504641 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562013d70030 con 0x562013cf1d30 2024-03-22T01:46:35.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.223+0000 7f85db7fe640 1 --2- 172.21.15.72:0/2966504641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562013cfa910 0x562013cfef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:46:35.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.223+0000 7f85d27fc640 1 --2- 172.21.15.72:0/2966504641 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x562013cf1d30 0x562013cfea10 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f85c40cd480 tx=0x7f85c409c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=743a646dfa471143 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:35.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.223+0000 7f85c27fc640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:46:35.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.223+0000 7f85c27fc640 1 -- 172.21.15.72:0/2966504641 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85c40b4070 con 0x562013cf1d30 2024-03-22T01:46:35.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.223+0000 7f85c27fc640 1 -- 172.21.15.72:0/2966504641 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f85c4096de0 con 0x562013cf1d30 2024-03-22T01:46:35.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.223+0000 7f85e3ffb740 1 -- 172.21.15.72:0/2966504641 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f85d403f930 con 0x562013cf1d30 2024-03-22T01:46:35.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.223+0000 7f85c27fc640 1 -- 172.21.15.72:0/2966504641 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7f85c40a4a80 con 0x562013cf1d30 2024-03-22T01:46:35.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.223+0000 7f85e3ffb740 10 client.16347 fetch_fsmap learned FSMap version 103 2024-03-22T01:46:35.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.223+0000 7f85e3ffb740 1 -- 172.21.15.72:0/2966504641 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=4+}) -- 0x562013d71310 con 0x562013cf1d30 2024-03-22T01:46:35.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.224+0000 7f85c27fc640 1 -- 172.21.15.72:0/2966504641 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85d403f930 con 0x562013cf1d30 2024-03-22T01:46:35.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.224+0000 7f85c27fc640 1 -- 172.21.15.72:0/2966504641 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 103) ==== 8330+0+0 (secure 0 0 0) 0x7f85c40a46f0 con 0x562013cf1d30 2024-03-22T01:46:35.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.224+0000 7f85e3ffb740 10 client.16347 fetch_fsmap finished waiting for FSMap version 103 2024-03-22T01:46:35.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.224+0000 7f85e3ffb740 10 client.16347 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T01:46:35.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.224+0000 7f85e3ffb740 20 client.16347 populate_metadata read hostname 'smithi072' 2024-03-22T01:46:35.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.224+0000 7f85e3ffb740 1 --2- 172.21.15.72:0/2966504641 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x562013d71b10 0x562013d73f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:35.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.224+0000 7f85e3ffb740 4 client.16347 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T01:46:35.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.224+0000 7f85e3ffb740 1 -- 172.21.15.72:0/2966504641 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562013c5d150 con 0x562013d71b10 2024-03-22T01:46:35.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.224+0000 7f85daffd640 1 --2- 172.21.15.72:0/2966504641 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x562013d71b10 0x562013d73f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:35.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.224+0000 7f85daffd640 1 --2- 172.21.15.72:0/2966504641 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x562013d71b10 0x562013d73f30 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5dc2ba383f96249b server_cookie=526f1adc2eddb2bf in_seq=0 out_seq=0 2024-03-22T01:46:35.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.224+0000 7f85c27fc640 10 client.16347 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:46:35.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.224+0000 7f85c27fc640 1 -- 172.21.15.72:0/2966504641 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x562013c5d150 con 0x562013d71b10 2024-03-22T01:46:35.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.224+0000 7f85c27fc640 10 client.16347 handle_command_reply: tid=0 2024-03-22T01:46:35.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.224+0000 7f85e3ffb740 1 client.16347 shutdown 2024-03-22T01:46:35.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.224+0000 7f85e3ffb740 1 -- 172.21.15.72:0/2966504641 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x562013cf1d30 msgr2=0x562013cfea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:35.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.224+0000 7f85e3ffb740 1 --2- 172.21.15.72:0/2966504641 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x562013cf1d30 0x562013cfea10 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f85c40cd480 tx=0x7f85c409c830 comp rx=0 tx=0).stop 2024-03-22T01:46:35.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.225+0000 7f85e3ffb740 1 -- 172.21.15.72:0/2966504641 shutdown_connections 2024-03-22T01:46:35.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.225+0000 7f85e3ffb740 1 --2- 172.21.15.72:0/2966504641 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x562013d71b10 0x562013d73f30 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.225+0000 7f85e3ffb740 1 --2- 172.21.15.72:0/2966504641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562013cfa910 0x562013cfef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.225+0000 7f85e3ffb740 1 --2- 172.21.15.72:0/2966504641 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x562013cf1d30 0x562013cfea10 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.225+0000 7f85c27fc640 0 client.16347 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:46:35.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.225+0000 7f85e3ffb740 1 --2- 172.21.15.72:0/2966504641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562013cf1410 0x562013cfe4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.225+0000 7f85e3ffb740 1 -- 172.21.15.72:0/2966504641 >> 172.21.15.72:0/2966504641 conn(0x562013cebb40 msgr2=0x562013cf78a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:35.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.225+0000 7f85e3ffb740 1 -- 172.21.15.72:0/2966504641 shutdown_connections 2024-03-22T01:46:35.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.225+0000 7f85e3ffb740 1 -- 172.21.15.72:0/2966504641 wait complete. 2024-03-22T01:46:35.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.225+0000 7f85e3ffb740 20 client.16347 trim_cache size 0 max 16384 2024-03-22T01:46:35.231 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-22T01:46:35.231 INFO:teuthology.orchestra.run.smithi072.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-22T01:46:35.232 INFO:teuthology.orchestra.run.smithi072.stdout: "scrubs": { 2024-03-22T01:46:35.232 INFO:teuthology.orchestra.run.smithi072.stdout: "d726e926-2c0e-4707-b9c6-039e034b7714": { 2024-03-22T01:46:35.232 INFO:teuthology.orchestra.run.smithi072.stdout: "path": "/", 2024-03-22T01:46:35.232 INFO:teuthology.orchestra.run.smithi072.stdout: "tag": "d726e926-2c0e-4707-b9c6-039e034b7714", 2024-03-22T01:46:35.232 INFO:teuthology.orchestra.run.smithi072.stdout: "options": "recursive,force" 2024-03-22T01:46:35.232 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:46:35.232 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:46:35.232 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-22T01:46:35.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.225+0000 7f85e140e640 1 -- 172.21.15.72:0/426137279 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f85c80a3090 msgr2=0x7f85c80a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:35.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.225+0000 7f85e140e640 1 --2- 172.21.15.72:0/426137279 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f85c80a3090 0x7f85c80a5550 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f85d4006010 tx=0x7f85d4005d80 comp rx=0 tx=0).stop 2024-03-22T01:46:35.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.225+0000 7f85e140e640 1 -- 172.21.15.72:0/426137279 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f85dc1013f0 msgr2=0x7f85dc197870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:35.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.225+0000 7f85e140e640 1 --2- 172.21.15.72:0/426137279 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f85dc1013f0 0x7f85dc197870 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f85c400e9f0 tx=0x7f85c400edf0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.225+0000 7f85e140e640 1 -- 172.21.15.72:0/426137279 shutdown_connections 2024-03-22T01:46:35.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.225+0000 7f85e140e640 1 --2- 172.21.15.72:0/426137279 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f85c80a3090 0x7f85c80a5550 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.225+0000 7f85e140e640 1 --2- 172.21.15.72:0/426137279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85dc106a40 0x7f85dc19bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.225+0000 7f85e140e640 1 --2- 172.21.15.72:0/426137279 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f85dc1013f0 0x7f85dc197870 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.225+0000 7f85e140e640 1 --2- 172.21.15.72:0/426137279 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85dc100a30 0x7f85dc197330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:35.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.225+0000 7f85e140e640 1 -- 172.21.15.72:0/426137279 >> 172.21.15.72:0/426137279 conn(0x7f85dc077c40 msgr2=0x7f85dc1035a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:35.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.225+0000 7f85e140e640 1 -- 172.21.15.72:0/426137279 shutdown_connections 2024-03-22T01:46:35.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:35.225+0000 7f85e140e640 1 -- 172.21.15.72:0/426137279 wait complete. 2024-03-22T01:46:35.235 INFO:tasks.cephfs.filesystem:scrub status for tag:d726e926-2c0e-4707-b9c6-039e034b7714 - {'path': '/', 'tag': 'd726e926-2c0e-4707-b9c6-039e034b7714', 'options': 'recursive,force'} 2024-03-22T01:46:36.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:46:36 smithi089 ceph-mon[35770]: pgmap v301: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3.8 KiB/s wr, 150 op/s 2024-03-22T01:46:36.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:46:36 smithi089 ceph-mon[35770]: scrub queued for path: / 2024-03-22T01:46:36.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:46:36 smithi089 ceph-mon[35770]: scrub summary: idle+waiting paths [/] 2024-03-22T01:46:36.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:46:36 smithi089 ceph-mon[35770]: scrub summary: active paths [/] 2024-03-22T01:46:36.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:46:36 smithi089 ceph-mon[35770]: scrub summary: idle+waiting paths [/] 2024-03-22T01:46:36.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:46:36 smithi072 ceph-mon[32414]: pgmap v301: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3.8 KiB/s wr, 150 op/s 2024-03-22T01:46:36.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:46:36 smithi072 ceph-mon[32414]: scrub queued for path: / 2024-03-22T01:46:36.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:46:36 smithi072 ceph-mon[32414]: scrub summary: idle+waiting paths [/] 2024-03-22T01:46:36.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:46:36 smithi072 ceph-mon[32414]: scrub summary: active paths [/] 2024-03-22T01:46:36.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:46:36 smithi072 ceph-mon[32414]: scrub summary: idle+waiting paths [/] 2024-03-22T01:46:36.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:46:36 smithi098 ceph-mon[36468]: pgmap v301: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3.8 KiB/s wr, 150 op/s 2024-03-22T01:46:36.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:46:36 smithi098 ceph-mon[36468]: scrub queued for path: / 2024-03-22T01:46:36.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:46:36 smithi098 ceph-mon[36468]: scrub summary: idle+waiting paths [/] 2024-03-22T01:46:36.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:46:36 smithi098 ceph-mon[36468]: scrub summary: active paths [/] 2024-03-22T01:46:36.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:46:36 smithi098 ceph-mon[36468]: scrub summary: idle+waiting paths [/] 2024-03-22T01:46:38.375 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:46:38 smithi072 ceph-mon[32414]: pgmap v302: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 3.8 KiB/s wr, 143 op/s 2024-03-22T01:46:38.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:46:38 smithi098 ceph-mon[36468]: pgmap v302: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 3.8 KiB/s wr, 143 op/s 2024-03-22T01:46:38.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:46:38 smithi089 ceph-mon[35770]: pgmap v302: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 3.8 KiB/s wr, 143 op/s 2024-03-22T01:46:39.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:46:39 smithi072 ceph-mon[32414]: scrub summary: idle 2024-03-22T01:46:39.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:46:39 smithi098 ceph-mon[36468]: scrub summary: idle 2024-03-22T01:46:39.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:46:39 smithi089 ceph-mon[35770]: scrub summary: idle 2024-03-22T01:46:40.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:46:40 smithi072 ceph-mon[32414]: pgmap v303: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3.1 KiB/s wr, 73 op/s 2024-03-22T01:46:40.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:46:40 smithi098 ceph-mon[36468]: pgmap v303: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3.1 KiB/s wr, 73 op/s 2024-03-22T01:46:40.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:46:40 smithi089 ceph-mon[35770]: pgmap v303: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3.1 KiB/s wr, 73 op/s 2024-03-22T01:46:42.275 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.272+0000 7f87180a4640 1 -- 172.21.15.72:0/1616753150 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87101016a0 msgr2=0x7f871010d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:42.275 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.272+0000 7f87180a4640 1 --2- 172.21.15.72:0/1616753150 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87101016a0 0x7f871010d860 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f8708009cd0 tx=0x7f870802c0f0 comp rx=0 tx=0).stop 2024-03-22T01:46:42.275 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.272+0000 7f87180a4640 1 -- 172.21.15.72:0/1616753150 shutdown_connections 2024-03-22T01:46:42.275 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.272+0000 7f87180a4640 1 --2- 172.21.15.72:0/1616753150 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87101016a0 0x7f871010d860 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:42.275 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.272+0000 7f87180a4640 1 --2- 172.21.15.72:0/1616753150 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8710100ce0 0x7f8710101160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:42.275 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.272+0000 7f87180a4640 1 --2- 172.21.15.72:0/1616753150 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8710106c20 0x7f8710107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:42.275 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.272+0000 7f87180a4640 1 -- 172.21.15.72:0/1616753150 >> 172.21.15.72:0/1616753150 conn(0x7f8710077850 msgr2=0x7f8710077c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:42.276 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.272+0000 7f87180a4640 1 -- 172.21.15.72:0/1616753150 shutdown_connections 2024-03-22T01:46:42.276 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.272+0000 7f87180a4640 1 -- 172.21.15.72:0/1616753150 wait complete. 2024-03-22T01:46:42.276 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.273+0000 7f87180a4640 1 Processor -- start 2024-03-22T01:46:42.276 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.273+0000 7f87180a4640 1 -- start start 2024-03-22T01:46:42.276 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.273+0000 7f87180a4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8710100ce0 0x7f871010e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:42.276 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.273+0000 7f87180a4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87101016a0 0x7f871010e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:42.276 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.273+0000 7f87180a4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8710106c20 0x7f871010edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:42.276 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.273+0000 7f87180a4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f8710111fe0 con 0x7f87101016a0 2024-03-22T01:46:42.276 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.273+0000 7f87180a4640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8710111e60 con 0x7f8710106c20 2024-03-22T01:46:42.276 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.273+0000 7f87180a4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8710112160 con 0x7f8710100ce0 2024-03-22T01:46:42.276 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.273+0000 7f871563f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87101016a0 0x7f871010e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:42.276 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.273+0000 7f871563f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87101016a0 0x7f871010e8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33300/0 (socket says 172.21.15.72:33300) 2024-03-22T01:46:42.277 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.273+0000 7f871563f640 1 -- 172.21.15.72:0/3181507886 learned_addr learned my addr 172.21.15.72:0/3181507886 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:42.277 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.273+0000 7f8716641640 1 --2- 172.21.15.72:0/3181507886 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8710106c20 0x7f871010edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:42.277 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.273+0000 7f8715e40640 1 --2- 172.21.15.72:0/3181507886 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8710100ce0 0x7f871010e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:42.277 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.273+0000 7f871563f640 1 -- 172.21.15.72:0/3181507886 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8710100ce0 msgr2=0x7f871010e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:42.277 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.273+0000 7f871563f640 1 --2- 172.21.15.72:0/3181507886 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8710100ce0 0x7f871010e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:42.277 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.273+0000 7f871563f640 1 -- 172.21.15.72:0/3181507886 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8710106c20 msgr2=0x7f871010edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:42.277 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.273+0000 7f871563f640 1 --2- 172.21.15.72:0/3181507886 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8710106c20 0x7f871010edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:42.277 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.273+0000 7f871563f640 1 -- 172.21.15.72:0/3181507886 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87101a6980 con 0x7f87101016a0 2024-03-22T01:46:42.277 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.274+0000 7f871563f640 1 --2- 172.21.15.72:0/3181507886 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87101016a0 0x7f871010e8b0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f870c00b4a0 tx=0x7f870c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:42.277 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.274+0000 7f8702ffd640 1 -- 172.21.15.72:0/3181507886 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f870c014070 con 0x7f87101016a0 2024-03-22T01:46:42.277 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.274+0000 7f8702ffd640 1 -- 172.21.15.72:0/3181507886 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f870c010af0 con 0x7f87101016a0 2024-03-22T01:46:42.277 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.274+0000 7f87180a4640 1 -- 172.21.15.72:0/3181507886 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87101a6c70 con 0x7f87101016a0 2024-03-22T01:46:42.278 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.274+0000 7f8702ffd640 1 -- 172.21.15.72:0/3181507886 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f870c017d10 con 0x7f87101016a0 2024-03-22T01:46:42.278 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.274+0000 7f87180a4640 1 -- 172.21.15.72:0/3181507886 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87101a71c0 con 0x7f87101016a0 2024-03-22T01:46:42.285 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.277+0000 7f8702ffd640 1 -- 172.21.15.72:0/3181507886 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 25) ==== 137985+0+0 (secure 0 0 0) 0x7f870c017070 con 0x7f87101016a0 2024-03-22T01:46:42.285 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.277+0000 7f871ad6f740 1 Processor -- start 2024-03-22T01:46:42.285 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.277+0000 7f871ad6f740 1 -- start start 2024-03-22T01:46:42.285 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.278+0000 7f871ad6f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603a74582d0 0x5603a74586b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:42.286 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.278+0000 7f871ad6f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603a7458bf0 0x5603a7461480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:42.286 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.278+0000 7f871ad6f740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603a74619c0 0x5603a7463db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:42.286 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.278+0000 7f871ad6f740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x5603a73bc040 con 0x5603a74619c0 2024-03-22T01:46:42.286 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.278+0000 7f871ad6f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5603a73649e0 con 0x5603a74582d0 2024-03-22T01:46:42.286 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.278+0000 7f8702ffd640 1 --2- 172.21.15.72:0/3181507886 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f86fc0a3100 0x7f86fc0a55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:42.286 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.278+0000 7f871ad6f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5603a73b7ff0 con 0x5603a7458bf0 2024-03-22T01:46:42.286 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.278+0000 7f8716641640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603a74582d0 0x5603a74586b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:42.286 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.278+0000 7f871563f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603a74619c0 0x5603a7463db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:42.286 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.278+0000 7f8716641640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603a74582d0 0x5603a74586b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.72:35440/0 (socket says 172.21.15.72:35440) 2024-03-22T01:46:42.286 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.278+0000 7f8716641640 1 -- 172.21.15.72:0/2946843447 learned_addr learned my addr 172.21.15.72:0/2946843447 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:42.286 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.278+0000 7f8715e40640 1 --2- 172.21.15.72:0/2946843447 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603a7458bf0 0x5603a7461480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:42.287 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.278+0000 7f8716641640 1 --2- 172.21.15.72:0/3181507886 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f86fc0a3100 0x7f86fc0a55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:42.287 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.278+0000 7f8702ffd640 1 -- 172.21.15.72:0/3181507886 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f870c0ced90 con 0x7f87101016a0 2024-03-22T01:46:42.287 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.278+0000 7f8715e40640 1 -- 172.21.15.72:0/2946843447 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603a74582d0 msgr2=0x5603a74586b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:42.287 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.278+0000 7f8715e40640 1 --2- 172.21.15.72:0/2946843447 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603a74582d0 0x5603a74586b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:42.287 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.278+0000 7f8715e40640 1 -- 172.21.15.72:0/2946843447 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603a74619c0 msgr2=0x5603a7463db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:42.287 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.278+0000 7f8715e40640 1 --2- 172.21.15.72:0/2946843447 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603a74619c0 0x5603a7463db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:42.287 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.278+0000 7f8715e40640 1 -- 172.21.15.72:0/2946843447 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5603a70d0210 con 0x5603a7458bf0 2024-03-22T01:46:42.287 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.278+0000 7f8716641640 1 --2- 172.21.15.72:0/2946843447 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603a74582d0 0x5603a74586b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:46:42.287 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.278+0000 7f871563f640 1 --2- 172.21.15.72:0/2946843447 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603a74619c0 0x5603a7463db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:46:42.287 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.278+0000 7f8716641640 1 --2- 172.21.15.72:0/3181507886 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f86fc0a3100 0x7f86fc0a55c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f8708004af0 tx=0x7f8708005c50 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:42.287 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.278+0000 7f8715e40640 1 --2- 172.21.15.72:0/2946843447 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603a7458bf0 0x5603a7461480 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8704009b70 tx=0x7f87040117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5870753b1c81ff5 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:42.287 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.279+0000 7f86fbfff640 1 -- 172.21.15.72:0/2946843447 <== mon.1 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8704009e60 con 0x5603a7458bf0 2024-03-22T01:46:42.288 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.279+0000 7f86fbfff640 1 -- 172.21.15.72:0/2946843447 <== mon.1 v2:172.21.15.98:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8704012930 con 0x5603a7458bf0 2024-03-22T01:46:42.288 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.279+0000 7f86fbfff640 1 -- 172.21.15.72:0/2946843447 <== mon.1 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f870400fa90 con 0x5603a7458bf0 2024-03-22T01:46:42.288 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.279+0000 7f871ad6f740 1 -- 172.21.15.72:0/2946843447 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603a7458bf0 msgr2=0x5603a7461480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:42.288 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.279+0000 7f871ad6f740 1 --2- 172.21.15.72:0/2946843447 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603a7458bf0 0x5603a7461480 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8704009b70 tx=0x7f87040117b0 comp rx=0 tx=0).stop 2024-03-22T01:46:42.288 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.279+0000 7f871ad6f740 1 -- 172.21.15.72:0/2946843447 shutdown_connections 2024-03-22T01:46:42.288 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.279+0000 7f871ad6f740 1 --2- 172.21.15.72:0/2946843447 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603a74619c0 0x5603a7463db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:42.288 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.279+0000 7f871ad6f740 1 --2- 172.21.15.72:0/2946843447 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603a7458bf0 0x5603a7461480 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:42.288 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.279+0000 7f871ad6f740 1 --2- 172.21.15.72:0/2946843447 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603a74582d0 0x5603a74586b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:42.288 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.279+0000 7f871ad6f740 1 -- 172.21.15.72:0/2946843447 >> 172.21.15.72:0/2946843447 conn(0x5603a7452bf0 msgr2=0x5603a7452fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:42.288 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.280+0000 7f871ad6f740 1 -- 172.21.15.72:0/2946843447 shutdown_connections 2024-03-22T01:46:42.288 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.280+0000 7f871ad6f740 1 -- 172.21.15.72:0/2946843447 wait complete. 2024-03-22T01:46:42.288 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.280+0000 7f871ad6f740 1 -- start start 2024-03-22T01:46:42.289 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.280+0000 7f871ad6f740 1 Processor -- start 2024-03-22T01:46:42.289 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.280+0000 7f871ad6f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603a74582d0 0x5603a74612b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:42.289 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.280+0000 7f871ad6f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603a7458bf0 0x5603a745e640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:42.289 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.280+0000 7f871ad6f740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603a74619c0 0x5603a745eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:42.289 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.280+0000 7f871ad6f740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x5603a74645f0 con 0x5603a74619c0 2024-03-22T01:46:42.289 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.280+0000 7f871ad6f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5603a7377db0 con 0x5603a7458bf0 2024-03-22T01:46:42.289 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.280+0000 7f871ad6f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5603a7464770 con 0x5603a74582d0 2024-03-22T01:46:42.290 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.281+0000 7f871ad6f740 10 client.0 _ll_register_callbacks cb 0x5603a70cfa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:46:42.290 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.281+0000 7f8716641640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603a74619c0 0x5603a745eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:42.290 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.281+0000 7f8716641640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603a74619c0 0x5603a745eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33318/0 (socket says 172.21.15.72:33318) 2024-03-22T01:46:42.290 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.281+0000 7f8716641640 1 -- 172.21.15.72:0/4068788939 learned_addr learned my addr 172.21.15.72:0/4068788939 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:42.290 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.281+0000 7f871563f640 1 --2- 172.21.15.72:0/4068788939 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603a7458bf0 0x5603a745e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:42.290 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.281+0000 7f8715e40640 1 --2- 172.21.15.72:0/4068788939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603a74582d0 0x5603a74612b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:42.290 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.281+0000 7f8716641640 1 -- 172.21.15.72:0/4068788939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603a74582d0 msgr2=0x5603a74612b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:42.290 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.281+0000 7f8716641640 1 --2- 172.21.15.72:0/4068788939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603a74582d0 0x5603a74612b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:42.290 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.281+0000 7f8716641640 1 -- 172.21.15.72:0/4068788939 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603a7458bf0 msgr2=0x5603a745e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:42.291 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.281+0000 7f8716641640 1 --2- 172.21.15.72:0/4068788939 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603a7458bf0 0x5603a745e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:42.291 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.281+0000 7f8716641640 1 -- 172.21.15.72:0/4068788939 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603a719c620 con 0x5603a74619c0 2024-03-22T01:46:42.291 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.281+0000 7f871563f640 1 --2- 172.21.15.72:0/4068788939 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603a7458bf0 0x5603a745e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:46:42.291 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.281+0000 7f8716641640 1 --2- 172.21.15.72:0/4068788939 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603a74619c0 0x5603a745eb80 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f8708037040 tx=0x7f870802c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37ce9b89db51506f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:42.291 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.281+0000 7f86f97fa640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:46:42.291 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.281+0000 7f86f97fa640 1 -- 172.21.15.72:0/4068788939 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8708053060 con 0x5603a74619c0 2024-03-22T01:46:42.291 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.281+0000 7f86f97fa640 1 -- 172.21.15.72:0/4068788939 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f8708004040 con 0x5603a74619c0 2024-03-22T01:46:42.291 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.281+0000 7f871ad6f740 1 -- 172.21.15.72:0/4068788939 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8704012930 con 0x5603a74619c0 2024-03-22T01:46:42.291 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.281+0000 7f86f97fa640 1 -- 172.21.15.72:0/4068788939 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7f8708039040 con 0x5603a74619c0 2024-03-22T01:46:42.291 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.281+0000 7f871ad6f740 10 client.16359 fetch_fsmap learned FSMap version 103 2024-03-22T01:46:42.291 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.281+0000 7f871ad6f740 1 -- 172.21.15.72:0/4068788939 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=4+}) -- 0x5603a74704c0 con 0x5603a74619c0 2024-03-22T01:46:42.292 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.282+0000 7f86f97fa640 1 -- 172.21.15.72:0/4068788939 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8704012930 con 0x5603a74619c0 2024-03-22T01:46:42.292 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.282+0000 7f86f97fa640 1 -- 172.21.15.72:0/4068788939 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 103) ==== 8330+0+0 (secure 0 0 0) 0x7f870804e970 con 0x5603a74619c0 2024-03-22T01:46:42.292 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.282+0000 7f871ad6f740 10 client.16359 fetch_fsmap finished waiting for FSMap version 103 2024-03-22T01:46:42.292 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.282+0000 7f871ad6f740 10 client.16359 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T01:46:42.292 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.282+0000 7f871ad6f740 20 client.16359 populate_metadata read hostname 'smithi072' 2024-03-22T01:46:42.292 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.282+0000 7f871ad6f740 1 --2- 172.21.15.72:0/4068788939 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x5603a7471980 0x5603a7473d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:42.292 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.282+0000 7f871ad6f740 4 client.16359 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T01:46:42.292 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.282+0000 7f871ad6f740 1 -- 172.21.15.72:0/4068788939 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5603a73c9580 con 0x5603a7471980 2024-03-22T01:46:42.292 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.282+0000 7f8715e40640 1 --2- 172.21.15.72:0/4068788939 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x5603a7471980 0x5603a7473d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:42.292 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.283+0000 7f86f97fa640 10 client.16359 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:46:42.292 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.283+0000 7f8715e40640 1 --2- 172.21.15.72:0/4068788939 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x5603a7471980 0x5603a7473d70 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ae916fe2e265554 server_cookie=f9368c76f3a9512a in_seq=0 out_seq=0 2024-03-22T01:46:42.293 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.283+0000 7f86f97fa640 1 -- 172.21.15.72:0/4068788939 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5603a73c9580 con 0x5603a7471980 2024-03-22T01:46:42.293 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.283+0000 7f86f97fa640 10 client.16359 handle_command_reply: tid=0 2024-03-22T01:46:42.293 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.283+0000 7f871ad6f740 1 client.16359 shutdown 2024-03-22T01:46:42.293 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.283+0000 7f871ad6f740 1 -- 172.21.15.72:0/4068788939 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603a74619c0 msgr2=0x5603a745eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:42.293 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.283+0000 7f871ad6f740 1 --2- 172.21.15.72:0/4068788939 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603a74619c0 0x5603a745eb80 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f8708037040 tx=0x7f870802c0f0 comp rx=0 tx=0).stop 2024-03-22T01:46:42.293 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.283+0000 7f871ad6f740 1 -- 172.21.15.72:0/4068788939 shutdown_connections 2024-03-22T01:46:42.293 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.283+0000 7f871ad6f740 1 --2- 172.21.15.72:0/4068788939 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x5603a7471980 0x5603a7473d70 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:42.293 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.283+0000 7f871ad6f740 1 --2- 172.21.15.72:0/4068788939 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603a7458bf0 0x5603a745e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:42.293 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.283+0000 7f871ad6f740 1 --2- 172.21.15.72:0/4068788939 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603a74619c0 0x5603a745eb80 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:42.294 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.283+0000 7f86f97fa640 0 client.16359 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:46:42.294 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.283+0000 7f871ad6f740 1 --2- 172.21.15.72:0/4068788939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603a74582d0 0x5603a74612b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:42.294 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.283+0000 7f871ad6f740 1 -- 172.21.15.72:0/4068788939 >> 172.21.15.72:0/4068788939 conn(0x5603a7452bf0 msgr2=0x5603a7454940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:42.294 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.283+0000 7f871ad6f740 1 -- 172.21.15.72:0/4068788939 shutdown_connections 2024-03-22T01:46:42.294 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.284+0000 7f871ad6f740 1 -- 172.21.15.72:0/4068788939 wait complete. 2024-03-22T01:46:42.294 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.284+0000 7f871ad6f740 20 client.16359 trim_cache size 0 max 16384 2024-03-22T01:46:42.303 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.298+0000 7f871ad6f740 1 Processor -- start 2024-03-22T01:46:42.303 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.298+0000 7f871ad6f740 1 -- start start 2024-03-22T01:46:42.303 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.298+0000 7f871ad6f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603a74582d0 0x5603a746d390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:42.304 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.298+0000 7f871ad6f740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603a7458bf0 0x5603a7451ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:42.304 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.298+0000 7f871ad6f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603a74619c0 0x5603a7479680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:42.304 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.298+0000 7f871ad6f740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f87101af3a0 con 0x5603a7458bf0 2024-03-22T01:46:42.304 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.298+0000 7f871ad6f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f87101b1c20 con 0x5603a74582d0 2024-03-22T01:46:42.304 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.298+0000 7f871ad6f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f87101afa10 con 0x5603a74619c0 2024-03-22T01:46:42.304 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.299+0000 7f871563f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603a7458bf0 0x5603a7451ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:42.304 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.299+0000 7f871563f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603a7458bf0 0x5603a7451ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33330/0 (socket says 172.21.15.72:33330) 2024-03-22T01:46:42.304 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.299+0000 7f871563f640 1 -- 172.21.15.72:0/257580598 learned_addr learned my addr 172.21.15.72:0/257580598 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:42.304 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.299+0000 7f8715e40640 1 --2- 172.21.15.72:0/257580598 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603a74582d0 0x5603a746d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:42.304 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.299+0000 7f8716641640 1 --2- 172.21.15.72:0/257580598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603a74619c0 0x5603a7479680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:42.304 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.299+0000 7f871563f640 1 -- 172.21.15.72:0/257580598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603a74619c0 msgr2=0x5603a7479680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:42.304 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.299+0000 7f871563f640 1 --2- 172.21.15.72:0/257580598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603a74619c0 0x5603a7479680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:42.304 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.299+0000 7f871563f640 1 -- 172.21.15.72:0/257580598 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603a74582d0 msgr2=0x5603a746d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:42.304 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.299+0000 7f871563f640 1 --2- 172.21.15.72:0/257580598 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603a74582d0 0x5603a746d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:42.305 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.299+0000 7f871563f640 1 -- 172.21.15.72:0/257580598 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5603a74740e0 con 0x5603a7458bf0 2024-03-22T01:46:42.305 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.299+0000 7f8716641640 1 --2- 172.21.15.72:0/257580598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603a74619c0 0x5603a7479680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T01:46:42.305 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.299+0000 7f871563f640 1 --2- 172.21.15.72:0/257580598 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603a7458bf0 0x5603a7451ad0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f870c0cc730 tx=0x7f870c06f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5f6b53d3810ced server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:42.305 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.299+0000 7f86fb7fe640 1 -- 172.21.15.72:0/257580598 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f870c01e040 con 0x5603a7458bf0 2024-03-22T01:46:42.305 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.299+0000 7f86fb7fe640 1 -- 172.21.15.72:0/257580598 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f870c0a38f0 con 0x5603a7458bf0 2024-03-22T01:46:42.305 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.299+0000 7f86fb7fe640 1 -- 172.21.15.72:0/257580598 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f870c010dc0 con 0x5603a7458bf0 2024-03-22T01:46:42.305 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.300+0000 7f871ad6f740 1 -- 172.21.15.72:0/257580598 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603a7458bf0 msgr2=0x5603a7451ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:42.305 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.300+0000 7f871ad6f740 1 --2- 172.21.15.72:0/257580598 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603a7458bf0 0x5603a7451ad0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f870c0cc730 tx=0x7f870c06f040 comp rx=0 tx=0).stop 2024-03-22T01:46:42.305 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.300+0000 7f871ad6f740 1 -- 172.21.15.72:0/257580598 shutdown_connections 2024-03-22T01:46:42.305 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.300+0000 7f871ad6f740 1 --2- 172.21.15.72:0/257580598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603a74619c0 0x5603a7479680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:42.305 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.300+0000 7f871ad6f740 1 --2- 172.21.15.72:0/257580598 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603a7458bf0 0x5603a7451ad0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:42.305 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.300+0000 7f871ad6f740 1 --2- 172.21.15.72:0/257580598 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603a74582d0 0x5603a746d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:42.305 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.300+0000 7f871ad6f740 1 -- 172.21.15.72:0/257580598 >> 172.21.15.72:0/257580598 conn(0x5603a7452bf0 msgr2=0x5603a74513c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:42.305 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.300+0000 7f871ad6f740 1 -- 172.21.15.72:0/257580598 shutdown_connections 2024-03-22T01:46:42.306 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.300+0000 7f871ad6f740 1 -- 172.21.15.72:0/257580598 wait complete. 2024-03-22T01:46:42.306 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.300+0000 7f871ad6f740 1 -- start start 2024-03-22T01:46:42.306 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.300+0000 7f871ad6f740 1 Processor -- start 2024-03-22T01:46:42.306 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.301+0000 7f871ad6f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603a74582d0 0x5603a7469d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:42.306 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.301+0000 7f871ad6f740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603a7458bf0 0x5603a74650b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:42.306 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.301+0000 7f871ad6f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603a74619c0 0x5603a74656c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:42.306 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.301+0000 7f871ad6f740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f87101b00e0 con 0x5603a7458bf0 2024-03-22T01:46:42.306 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.301+0000 7f871ad6f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f87101b1550 con 0x5603a74582d0 2024-03-22T01:46:42.306 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.301+0000 7f871ad6f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f87101b07b0 con 0x5603a74619c0 2024-03-22T01:46:42.306 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.301+0000 7f871ad6f740 10 client.0 _ll_register_callbacks cb 0x7f8708009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:46:42.306 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.301+0000 7f871563f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603a7458bf0 0x5603a74650b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:42.306 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.301+0000 7f871563f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603a7458bf0 0x5603a74650b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33340/0 (socket says 172.21.15.72:33340) 2024-03-22T01:46:42.306 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.301+0000 7f871563f640 1 -- 172.21.15.72:0/449570128 learned_addr learned my addr 172.21.15.72:0/449570128 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:42.306 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.301+0000 7f8715e40640 1 --2- 172.21.15.72:0/449570128 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603a74582d0 0x5603a7469d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:42.307 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.301+0000 7f8716641640 1 --2- 172.21.15.72:0/449570128 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603a74619c0 0x5603a74656c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:42.307 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.301+0000 7f871563f640 1 -- 172.21.15.72:0/449570128 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603a74619c0 msgr2=0x5603a74656c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:42.307 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.301+0000 7f871563f640 1 --2- 172.21.15.72:0/449570128 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603a74619c0 0x5603a74656c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:42.307 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.301+0000 7f871563f640 1 -- 172.21.15.72:0/449570128 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603a74582d0 msgr2=0x5603a7469d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:42.307 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.301+0000 7f871563f640 1 --2- 172.21.15.72:0/449570128 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603a74582d0 0x5603a7469d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:42.307 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.301+0000 7f871563f640 1 -- 172.21.15.72:0/449570128 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603a7465ec0 con 0x5603a7458bf0 2024-03-22T01:46:42.307 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.301+0000 7f8715e40640 1 --2- 172.21.15.72:0/449570128 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603a74582d0 0x5603a7469d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:46:42.307 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.301+0000 7f871563f640 1 --2- 172.21.15.72:0/449570128 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603a7458bf0 0x5603a74650b0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f870c00be60 tx=0x7f870c0047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=922ecf49b59ce93c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:42.307 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.301+0000 7f86f97fa640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:46:42.307 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.301+0000 7f86f97fa640 1 -- 172.21.15.72:0/449570128 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f870c01e050 con 0x5603a7458bf0 2024-03-22T01:46:42.307 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.301+0000 7f86f97fa640 1 -- 172.21.15.72:0/449570128 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f870c097550 con 0x5603a7458bf0 2024-03-22T01:46:42.307 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.301+0000 7f871ad6f740 1 -- 172.21.15.72:0/449570128 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f870c0a38f0 con 0x5603a7458bf0 2024-03-22T01:46:42.307 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.302+0000 7f86f97fa640 1 -- 172.21.15.72:0/449570128 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7f870c0af0d0 con 0x5603a7458bf0 2024-03-22T01:46:42.308 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.302+0000 7f871ad6f740 10 client.16365 fetch_fsmap learned FSMap version 103 2024-03-22T01:46:42.308 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.302+0000 7f871ad6f740 1 -- 172.21.15.72:0/449570128 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=4+}) -- 0x5603a74d7d30 con 0x5603a7458bf0 2024-03-22T01:46:42.308 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.302+0000 7f86f97fa640 1 -- 172.21.15.72:0/449570128 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f870c0a38f0 con 0x5603a7458bf0 2024-03-22T01:46:42.308 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.302+0000 7f86f97fa640 1 -- 172.21.15.72:0/449570128 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 103) ==== 8330+0+0 (secure 0 0 0) 0x7f870c0a3460 con 0x5603a7458bf0 2024-03-22T01:46:42.308 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.302+0000 7f871ad6f740 10 client.16365 fetch_fsmap finished waiting for FSMap version 103 2024-03-22T01:46:42.308 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.302+0000 7f871ad6f740 10 client.16365 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T01:46:42.308 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.302+0000 7f871ad6f740 20 client.16365 populate_metadata read hostname 'smithi072' 2024-03-22T01:46:42.308 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.302+0000 7f871ad6f740 1 --2- 172.21.15.72:0/449570128 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x5603a74d8420 0x5603a74da840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:42.308 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.302+0000 7f871ad6f740 4 client.16365 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T01:46:42.308 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.302+0000 7f871ad6f740 1 -- 172.21.15.72:0/449570128 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5603a745ef10 con 0x5603a74d8420 2024-03-22T01:46:42.308 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.302+0000 7f8715e40640 1 --2- 172.21.15.72:0/449570128 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x5603a74d8420 0x5603a74da840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:42.308 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.303+0000 7f8715e40640 1 --2- 172.21.15.72:0/449570128 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x5603a74d8420 0x5603a74da840 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5356111321e549 server_cookie=7eeec1685dfe2c27 in_seq=0 out_seq=0 2024-03-22T01:46:42.308 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.303+0000 7f86f97fa640 10 client.16365 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:46:42.310 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.306+0000 7f86f97fa640 1 -- 172.21.15.72:0/449570128 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5603a745ef10 con 0x5603a74d8420 2024-03-22T01:46:42.310 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.306+0000 7f86f97fa640 10 client.16365 handle_command_reply: tid=0 2024-03-22T01:46:42.310 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.306+0000 7f871ad6f740 1 client.16365 shutdown 2024-03-22T01:46:42.310 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.307+0000 7f871ad6f740 1 -- 172.21.15.72:0/449570128 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603a7458bf0 msgr2=0x5603a74650b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:42.310 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.307+0000 7f871ad6f740 1 --2- 172.21.15.72:0/449570128 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603a7458bf0 0x5603a74650b0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f870c00be60 tx=0x7f870c0047d0 comp rx=0 tx=0).stop 2024-03-22T01:46:42.310 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.307+0000 7f871ad6f740 1 -- 172.21.15.72:0/449570128 shutdown_connections 2024-03-22T01:46:42.310 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.307+0000 7f871ad6f740 1 --2- 172.21.15.72:0/449570128 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x5603a74d8420 0x5603a74da840 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:42.310 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.307+0000 7f871ad6f740 1 --2- 172.21.15.72:0/449570128 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5603a74619c0 0x5603a74656c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:42.310 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.307+0000 7f871ad6f740 1 --2- 172.21.15.72:0/449570128 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5603a7458bf0 0x5603a74650b0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:42.310 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.307+0000 7f871ad6f740 1 --2- 172.21.15.72:0/449570128 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603a74582d0 0x5603a7469d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:42.310 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.307+0000 7f86f97fa640 0 client.16365 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:46:42.310 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.307+0000 7f871ad6f740 1 -- 172.21.15.72:0/449570128 >> 172.21.15.72:0/449570128 conn(0x5603a7452bf0 msgr2=0x5603a7451c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:42.310 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.307+0000 7f871ad6f740 1 -- 172.21.15.72:0/449570128 shutdown_connections 2024-03-22T01:46:42.311 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.307+0000 7f871ad6f740 1 -- 172.21.15.72:0/449570128 wait complete. 2024-03-22T01:46:42.311 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.307+0000 7f871ad6f740 20 client.16365 trim_cache size 0 max 16384 2024-03-22T01:46:42.311 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout:{ 2024-03-22T01:46:42.311 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout: "message": "", 2024-03-22T01:46:42.311 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout: "return_code": 0 2024-03-22T01:46:42.311 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout:} 2024-03-22T01:46:42.312 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.307+0000 7f87180a4640 1 -- 172.21.15.72:0/3181507886 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f86fc0a3100 msgr2=0x7f86fc0a55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:42.312 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.307+0000 7f87180a4640 1 --2- 172.21.15.72:0/3181507886 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f86fc0a3100 0x7f86fc0a55c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f8708004af0 tx=0x7f8708005c50 comp rx=0 tx=0).stop 2024-03-22T01:46:42.312 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.307+0000 7f87180a4640 1 -- 172.21.15.72:0/3181507886 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87101016a0 msgr2=0x7f871010e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:42.312 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.307+0000 7f87180a4640 1 --2- 172.21.15.72:0/3181507886 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87101016a0 0x7f871010e8b0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f870c00b4a0 tx=0x7f870c00b960 comp rx=0 tx=0).stop 2024-03-22T01:46:42.312 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.307+0000 7f87180a4640 1 -- 172.21.15.72:0/3181507886 shutdown_connections 2024-03-22T01:46:42.312 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.307+0000 7f87180a4640 1 --2- 172.21.15.72:0/3181507886 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f86fc0a3100 0x7f86fc0a55c0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:42.312 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.307+0000 7f87180a4640 1 --2- 172.21.15.72:0/3181507886 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8710106c20 0x7f871010edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:42.312 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.307+0000 7f87180a4640 1 --2- 172.21.15.72:0/3181507886 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87101016a0 0x7f871010e8b0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:42.313 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.307+0000 7f87180a4640 1 --2- 172.21.15.72:0/3181507886 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8710100ce0 0x7f871010e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:42.313 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.307+0000 7f87180a4640 1 -- 172.21.15.72:0/3181507886 >> 172.21.15.72:0/3181507886 conn(0x7f8710077850 msgr2=0x7f87100ff840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:42.313 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.308+0000 7f87180a4640 1 -- 172.21.15.72:0/3181507886 shutdown_connections 2024-03-22T01:46:42.313 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:42.308+0000 7f87180a4640 1 -- 172.21.15.72:0/3181507886 wait complete. 2024-03-22T01:46:42.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:46:42 smithi098 ceph-mon[36468]: pgmap v304: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 3.1 KiB/s wr, 42 op/s 2024-03-22T01:46:42.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:46:42 smithi089 ceph-mon[35770]: pgmap v304: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 3.1 KiB/s wr, 42 op/s 2024-03-22T01:46:42.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:46:42 smithi072 ceph-mon[32414]: pgmap v304: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 3.1 KiB/s wr, 42 op/s 2024-03-22T01:46:44.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:46:44 smithi098 ceph-mon[36468]: pgmap v305: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1023 B/s wr, 14 op/s 2024-03-22T01:46:44.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:46:44 smithi089 ceph-mon[35770]: pgmap v305: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1023 B/s wr, 14 op/s 2024-03-22T01:46:44.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:46:44 smithi072 ceph-mon[32414]: pgmap v305: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1023 B/s wr, 14 op/s 2024-03-22T01:46:46.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:46:46 smithi098 ceph-mon[36468]: pgmap v306: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-22T01:46:46.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:46:46 smithi089 ceph-mon[35770]: pgmap v306: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-22T01:46:46.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:46:46 smithi072 ceph-mon[32414]: pgmap v306: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-22T01:46:48.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:46:48 smithi098 ceph-mon[36468]: pgmap v307: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-22T01:46:48.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:46:48 smithi089 ceph-mon[35770]: pgmap v307: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-22T01:46:48.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:46:48 smithi072 ceph-mon[32414]: pgmap v307: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-22T01:46:50.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:46:50 smithi098 ceph-mon[36468]: pgmap v308: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-22T01:46:50.424 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:46:50 smithi072 ceph-mon[32414]: pgmap v308: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-22T01:46:50.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:46:50 smithi089 ceph-mon[35770]: pgmap v308: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-22T01:46:52.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:46:52 smithi098 ceph-mon[36468]: pgmap v309: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-22T01:46:52.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:46:52 smithi089 ceph-mon[35770]: pgmap v309: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-22T01:46:52.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:46:52 smithi072 ceph-mon[32414]: pgmap v309: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-22T01:46:54.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:46:54 smithi098 ceph-mon[36468]: pgmap v310: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-22T01:46:54.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:46:54 smithi089 ceph-mon[35770]: pgmap v310: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-22T01:46:54.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:46:54 smithi072 ceph-mon[32414]: pgmap v310: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-22T01:46:55.405 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.403+0000 7fbcfcd81640 1 -- 172.21.15.72:0/3240200294 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbcf8100a40 msgr2=0x7fbcf8100ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:55.405 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.403+0000 7fbcfcd81640 1 --2- 172.21.15.72:0/3240200294 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbcf8100a40 0x7fbcf8100ec0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fbce8009860 tx=0x7fbce802bb70 comp rx=0 tx=0).stop 2024-03-22T01:46:55.406 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.403+0000 7fbcfcd81640 1 -- 172.21.15.72:0/3240200294 shutdown_connections 2024-03-22T01:46:55.406 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.403+0000 7fbcfcd81640 1 --2- 172.21.15.72:0/3240200294 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbcf8100a40 0x7fbcf8100ec0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:55.406 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.403+0000 7fbcfcd81640 1 --2- 172.21.15.72:0/3240200294 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbcf8101400 0x7fbcf810d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:55.406 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.403+0000 7fbcfcd81640 1 --2- 172.21.15.72:0/3240200294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbcf8106a50 0x7fbcf8106e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:55.406 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.403+0000 7fbcfcd81640 1 -- 172.21.15.72:0/3240200294 >> 172.21.15.72:0/3240200294 conn(0x7fbcf80fc530 msgr2=0x7fbcf80fe950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:55.406 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.403+0000 7fbcfcd81640 1 -- 172.21.15.72:0/3240200294 shutdown_connections 2024-03-22T01:46:55.406 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.403+0000 7fbcfcd81640 1 -- 172.21.15.72:0/3240200294 wait complete. 2024-03-22T01:46:55.406 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.404+0000 7fbcfcd81640 1 Processor -- start 2024-03-22T01:46:55.406 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.404+0000 7fbcfcd81640 1 -- start start 2024-03-22T01:46:55.406 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.404+0000 7fbcfcd81640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbcf8100a40 0x7fbcf8197340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:55.406 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.404+0000 7fbcfcd81640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbcf8101400 0x7fbcf8197880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:55.406 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.404+0000 7fbcfcd81640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbcf8106a50 0x7fbcf819bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:55.407 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.404+0000 7fbcfcd81640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fbcf8110e60 con 0x7fbcf8106a50 2024-03-22T01:46:55.407 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.404+0000 7fbcfcd81640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fbcf8110ce0 con 0x7fbcf8100a40 2024-03-22T01:46:55.407 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.404+0000 7fbcfcd81640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbcf8110fe0 con 0x7fbcf8101400 2024-03-22T01:46:55.407 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.404+0000 7fbcf659c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbcf8100a40 0x7fbcf8197340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:55.407 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.404+0000 7fbcf6d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbcf8106a50 0x7fbcf819bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:55.407 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.404+0000 7fbcf659c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbcf8100a40 0x7fbcf8197340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.72:49776/0 (socket says 172.21.15.72:49776) 2024-03-22T01:46:55.407 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.404+0000 7fbcf659c640 1 -- 172.21.15.72:0/74359279 learned_addr learned my addr 172.21.15.72:0/74359279 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:55.407 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.404+0000 7fbcf5d9b640 1 --2- 172.21.15.72:0/74359279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbcf8101400 0x7fbcf8197880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:55.407 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.405+0000 7fbcf659c640 1 -- 172.21.15.72:0/74359279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbcf8101400 msgr2=0x7fbcf8197880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:55.407 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.405+0000 7fbcf659c640 1 --2- 172.21.15.72:0/74359279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbcf8101400 0x7fbcf8197880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:55.407 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.405+0000 7fbcf659c640 1 -- 172.21.15.72:0/74359279 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbcf8106a50 msgr2=0x7fbcf819bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:55.407 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.405+0000 7fbcf659c640 1 --2- 172.21.15.72:0/74359279 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbcf8106a50 0x7fbcf819bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:55.407 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.405+0000 7fbcf659c640 1 -- 172.21.15.72:0/74359279 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcf819c2e0 con 0x7fbcf8100a40 2024-03-22T01:46:55.407 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.405+0000 7fbcf5d9b640 1 --2- 172.21.15.72:0/74359279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbcf8101400 0x7fbcf8197880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:46:55.407 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.405+0000 7fbcf6d9d640 1 --2- 172.21.15.72:0/74359279 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbcf8106a50 0x7fbcf819bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:46:55.408 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.405+0000 7fbcf659c640 1 --2- 172.21.15.72:0/74359279 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbcf8100a40 0x7fbcf8197340 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fbcec00d770 tx=0x7fbcec00dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:55.408 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.405+0000 7fbce77fe640 1 -- 172.21.15.72:0/74359279 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcec017070 con 0x7fbcf8100a40 2024-03-22T01:46:55.408 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.405+0000 7fbce77fe640 1 -- 172.21.15.72:0/74359279 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbcec005500 con 0x7fbcf8100a40 2024-03-22T01:46:55.408 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.405+0000 7fbcfcd81640 1 -- 172.21.15.72:0/74359279 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcf819c5d0 con 0x7fbcf8100a40 2024-03-22T01:46:55.408 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.405+0000 7fbce77fe640 1 -- 172.21.15.72:0/74359279 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcec004e00 con 0x7fbcf8100a40 2024-03-22T01:46:55.408 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.405+0000 7fbcfcd81640 1 -- 172.21.15.72:0/74359279 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcf81a3e60 con 0x7fbcf8100a40 2024-03-22T01:46:55.414 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.408+0000 7fbce77fe640 1 -- 172.21.15.72:0/74359279 <== mon.2 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 25) ==== 137985+0+0 (secure 0 0 0) 0x7fbcec004fa0 con 0x7fbcf8100a40 2024-03-22T01:46:55.414 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.408+0000 7fbcff96b740 1 Processor -- start 2024-03-22T01:46:55.415 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.408+0000 7fbcff96b740 1 -- start start 2024-03-22T01:46:55.415 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.408+0000 7fbce77fe640 1 --2- 172.21.15.72:0/74359279 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fbce00a3100 0x7fbce00a55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:55.415 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.409+0000 7fbcff96b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55911b194d80 0x55911b195160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:55.415 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.409+0000 7fbcff96b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55911b1956a0 0x55911b1a0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:55.415 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.409+0000 7fbcff96b740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55911b1a0bd0 0x55911b1a2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:55.415 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.409+0000 7fbcf6d9d640 1 --2- 172.21.15.72:0/74359279 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fbce00a3100 0x7fbce00a55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:55.415 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.409+0000 7fbcff96b740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55911b0f9040 con 0x55911b1a0bd0 2024-03-22T01:46:55.415 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.409+0000 7fbcff96b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55911b0a19e0 con 0x55911b194d80 2024-03-22T01:46:55.415 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.409+0000 7fbcff96b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55911b0f4ff0 con 0x55911b1956a0 2024-03-22T01:46:55.415 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.409+0000 7fbce77fe640 1 -- 172.21.15.72:0/74359279 <== mon.2 v2:172.21.15.89:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fbcec0cf490 con 0x7fbcf8100a40 2024-03-22T01:46:55.415 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.409+0000 7fbcf6d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55911b1a0bd0 0x55911b1a2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:55.415 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.409+0000 7fbcf659c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55911b194d80 0x55911b195160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:55.415 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.409+0000 7fbcf6d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55911b1a0bd0 0x55911b1a2fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:40512/0 (socket says 172.21.15.72:40512) 2024-03-22T01:46:55.415 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.409+0000 7fbcf6d9d640 1 -- 172.21.15.72:0/188047526 learned_addr learned my addr 172.21.15.72:0/188047526 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:55.416 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.409+0000 7fbcf5d9b640 1 --2- 172.21.15.72:0/188047526 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55911b1956a0 0x55911b1a0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:55.416 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.409+0000 7fbcf659c640 1 -- 172.21.15.72:0/188047526 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55911b1956a0 msgr2=0x55911b1a0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:55.416 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.409+0000 7fbcf659c640 1 --2- 172.21.15.72:0/188047526 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55911b1956a0 0x55911b1a0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:55.416 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.409+0000 7fbcf659c640 1 -- 172.21.15.72:0/188047526 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55911b1a0bd0 msgr2=0x55911b1a2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:55.416 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.409+0000 7fbcf659c640 1 --2- 172.21.15.72:0/188047526 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55911b1a0bd0 0x55911b1a2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:55.416 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.409+0000 7fbcf659c640 1 -- 172.21.15.72:0/188047526 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55911ae0d210 con 0x55911b194d80 2024-03-22T01:46:55.416 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.409+0000 7fbcf6d9d640 1 --2- 172.21.15.72:0/74359279 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fbce00a3100 0x7fbce00a55c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fbcf0009fd0 tx=0x7fbcf000e7a0 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:55.416 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.409+0000 7fbcf659c640 1 --2- 172.21.15.72:0/188047526 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55911b194d80 0x55911b195160 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fbcec0cc8d0 tx=0x7fbcec09af90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8dc64278ef557ef2 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:55.416 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.410+0000 7fbce4ff9640 1 -- 172.21.15.72:0/188047526 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcec020050 con 0x55911b194d80 2024-03-22T01:46:55.416 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.410+0000 7fbce4ff9640 1 -- 172.21.15.72:0/188047526 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbcec09f6c0 con 0x55911b194d80 2024-03-22T01:46:55.416 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.410+0000 7fbce4ff9640 1 -- 172.21.15.72:0/188047526 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcec0ccb50 con 0x55911b194d80 2024-03-22T01:46:55.416 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.410+0000 7fbcff96b740 1 -- 172.21.15.72:0/188047526 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55911b194d80 msgr2=0x55911b195160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:55.416 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.410+0000 7fbcff96b740 1 --2- 172.21.15.72:0/188047526 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55911b194d80 0x55911b195160 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fbcec0cc8d0 tx=0x7fbcec09af90 comp rx=0 tx=0).stop 2024-03-22T01:46:55.416 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.410+0000 7fbcff96b740 1 -- 172.21.15.72:0/188047526 shutdown_connections 2024-03-22T01:46:55.417 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.410+0000 7fbcff96b740 1 --2- 172.21.15.72:0/188047526 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55911b1a0bd0 0x55911b1a2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:55.417 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.410+0000 7fbcff96b740 1 --2- 172.21.15.72:0/188047526 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55911b1956a0 0x55911b1a0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:55.417 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.410+0000 7fbcff96b740 1 --2- 172.21.15.72:0/188047526 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55911b194d80 0x55911b195160 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:55.417 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.410+0000 7fbcff96b740 1 -- 172.21.15.72:0/188047526 >> 172.21.15.72:0/188047526 conn(0x55911b18fbf0 msgr2=0x55911b18ffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:55.417 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.410+0000 7fbcff96b740 1 -- 172.21.15.72:0/188047526 shutdown_connections 2024-03-22T01:46:55.417 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.410+0000 7fbcff96b740 1 -- 172.21.15.72:0/188047526 wait complete. 2024-03-22T01:46:55.417 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.411+0000 7fbcff96b740 1 -- start start 2024-03-22T01:46:55.417 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.411+0000 7fbcff96b740 1 Processor -- start 2024-03-22T01:46:55.417 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.411+0000 7fbcff96b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55911b194d80 0x55911b1a0530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:55.417 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.411+0000 7fbcff96b740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55911b1956a0 0x55911b19d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:55.417 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.411+0000 7fbcff96b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55911b1a0bd0 0x55911b19de20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:55.417 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.411+0000 7fbcff96b740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55911b1a3740 con 0x55911b1956a0 2024-03-22T01:46:55.417 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.411+0000 7fbcff96b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55911b0b4db0 con 0x55911b194d80 2024-03-22T01:46:55.417 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.411+0000 7fbcff96b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55911b1a38c0 con 0x55911b1a0bd0 2024-03-22T01:46:55.418 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.411+0000 7fbcff96b740 10 client.0 _ll_register_callbacks cb 0x55911ae0ca00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:46:55.418 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.411+0000 7fbcf5d9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55911b1956a0 0x55911b19d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:55.418 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.411+0000 7fbcf5d9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55911b1956a0 0x55911b19d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:40524/0 (socket says 172.21.15.72:40524) 2024-03-22T01:46:55.418 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.411+0000 7fbcf5d9b640 1 -- 172.21.15.72:0/4160914829 learned_addr learned my addr 172.21.15.72:0/4160914829 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:55.418 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.411+0000 7fbcf659c640 1 --2- 172.21.15.72:0/4160914829 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55911b194d80 0x55911b1a0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:55.418 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.411+0000 7fbcf6d9d640 1 --2- 172.21.15.72:0/4160914829 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55911b1a0bd0 0x55911b19de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:55.418 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.411+0000 7fbcf5d9b640 1 -- 172.21.15.72:0/4160914829 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55911b1a0bd0 msgr2=0x55911b19de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:55.418 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.411+0000 7fbcf5d9b640 1 --2- 172.21.15.72:0/4160914829 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55911b1a0bd0 0x55911b19de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:55.419 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.411+0000 7fbcf5d9b640 1 -- 172.21.15.72:0/4160914829 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55911b194d80 msgr2=0x55911b1a0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:55.419 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.411+0000 7fbcf5d9b640 1 --2- 172.21.15.72:0/4160914829 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55911b194d80 0x55911b1a0530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:55.419 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.411+0000 7fbcf5d9b640 1 -- 172.21.15.72:0/4160914829 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55911aed9620 con 0x55911b1956a0 2024-03-22T01:46:55.419 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.411+0000 7fbcf659c640 1 --2- 172.21.15.72:0/4160914829 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55911b194d80 0x55911b1a0530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:46:55.420 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.412+0000 7fbcf5d9b640 1 --2- 172.21.15.72:0/4160914829 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55911b1956a0 0x55911b19d8c0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fbce802c0b0 tx=0x7fbce8004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b80767b8fc5ccd6 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:55.420 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.412+0000 7fbcddffb640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:46:55.420 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.412+0000 7fbcddffb640 1 -- 172.21.15.72:0/4160914829 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbce8039040 con 0x55911b1956a0 2024-03-22T01:46:55.420 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.412+0000 7fbcff96b740 1 -- 172.21.15.72:0/4160914829 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbcec09f6c0 con 0x55911b1956a0 2024-03-22T01:46:55.420 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.412+0000 7fbcddffb640 1 -- 172.21.15.72:0/4160914829 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fbce8034440 con 0x55911b1956a0 2024-03-22T01:46:55.420 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.412+0000 7fbcddffb640 1 -- 172.21.15.72:0/4160914829 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7fbce804d050 con 0x55911b1956a0 2024-03-22T01:46:55.420 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.412+0000 7fbcff96b740 10 client.16377 fetch_fsmap learned FSMap version 103 2024-03-22T01:46:55.420 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.412+0000 7fbcff96b740 1 -- 172.21.15.72:0/4160914829 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=4+}) -- 0x55911b1af740 con 0x55911b1956a0 2024-03-22T01:46:55.420 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.412+0000 7fbcddffb640 1 -- 172.21.15.72:0/4160914829 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbcec09f6c0 con 0x55911b1956a0 2024-03-22T01:46:55.420 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.412+0000 7fbcddffb640 1 -- 172.21.15.72:0/4160914829 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 103) ==== 8330+0+0 (secure 0 0 0) 0x7fbce802c5a0 con 0x55911b1956a0 2024-03-22T01:46:55.420 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.412+0000 7fbcff96b740 10 client.16377 fetch_fsmap finished waiting for FSMap version 103 2024-03-22T01:46:55.420 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.412+0000 7fbcff96b740 10 client.16377 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T01:46:55.420 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.412+0000 7fbcff96b740 20 client.16377 populate_metadata read hostname 'smithi072' 2024-03-22T01:46:55.420 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.412+0000 7fbcff96b740 1 --2- 172.21.15.72:0/4160914829 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55911b1b0bb0 0x55911b1b2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:55.420 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.412+0000 7fbcff96b740 4 client.16377 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T01:46:55.421 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.412+0000 7fbcff96b740 1 -- 172.21.15.72:0/4160914829 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55911b106580 con 0x55911b1b0bb0 2024-03-22T01:46:55.421 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.413+0000 7fbcf659c640 1 --2- 172.21.15.72:0/4160914829 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55911b1b0bb0 0x55911b1b2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:55.421 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.413+0000 7fbcf659c640 1 --2- 172.21.15.72:0/4160914829 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55911b1b0bb0 0x55911b1b2fc0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6377ec6a9ea6848f server_cookie=5bc0d846346bbb0 in_seq=0 out_seq=0 2024-03-22T01:46:55.421 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.413+0000 7fbcddffb640 10 client.16377 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:46:55.421 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.414+0000 7fbcddffb640 1 -- 172.21.15.72:0/4160914829 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55911b106580 con 0x55911b1b0bb0 2024-03-22T01:46:55.421 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.414+0000 7fbcddffb640 10 client.16377 handle_command_reply: tid=0 2024-03-22T01:46:55.421 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.414+0000 7fbcff96b740 1 client.16377 shutdown 2024-03-22T01:46:55.421 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.414+0000 7fbcff96b740 1 -- 172.21.15.72:0/4160914829 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55911b1956a0 msgr2=0x55911b19d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:55.421 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.414+0000 7fbcff96b740 1 --2- 172.21.15.72:0/4160914829 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55911b1956a0 0x55911b19d8c0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fbce802c0b0 tx=0x7fbce8004270 comp rx=0 tx=0).stop 2024-03-22T01:46:55.421 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.414+0000 7fbcff96b740 1 -- 172.21.15.72:0/4160914829 shutdown_connections 2024-03-22T01:46:55.421 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.414+0000 7fbcff96b740 1 --2- 172.21.15.72:0/4160914829 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55911b1b0bb0 0x55911b1b2fc0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:55.421 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.414+0000 7fbcff96b740 1 --2- 172.21.15.72:0/4160914829 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55911b1a0bd0 0x55911b19de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:55.421 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.414+0000 7fbcff96b740 1 --2- 172.21.15.72:0/4160914829 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55911b1956a0 0x55911b19d8c0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:55.422 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.414+0000 7fbcff96b740 1 --2- 172.21.15.72:0/4160914829 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55911b194d80 0x55911b1a0530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:55.422 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.414+0000 7fbcddffb640 0 client.16377 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:46:55.422 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.414+0000 7fbcff96b740 1 -- 172.21.15.72:0/4160914829 >> 172.21.15.72:0/4160914829 conn(0x55911b18fbf0 msgr2=0x55911b191940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:55.422 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.414+0000 7fbcff96b740 1 -- 172.21.15.72:0/4160914829 shutdown_connections 2024-03-22T01:46:55.422 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.414+0000 7fbcff96b740 1 -- 172.21.15.72:0/4160914829 wait complete. 2024-03-22T01:46:55.422 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.414+0000 7fbcff96b740 20 client.16377 trim_cache size 0 max 16384 2024-03-22T01:46:55.432 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.429+0000 7fbcff96b740 1 Processor -- start 2024-03-22T01:46:55.433 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.429+0000 7fbcff96b740 1 -- start start 2024-03-22T01:46:55.433 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.429+0000 7fbcff96b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55911b194d80 0x55911b1b2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:55.433 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.429+0000 7fbcff96b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55911b1956a0 0x55911b1ac520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:55.433 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.429+0000 7fbcff96b740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55911b1a0bd0 0x55911b1b8920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:55.433 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.429+0000 7fbcff96b740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fbcf81aade0 con 0x55911b1a0bd0 2024-03-22T01:46:55.433 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.429+0000 7fbcff96b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fbcf81ad660 con 0x55911b1956a0 2024-03-22T01:46:55.433 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.429+0000 7fbcff96b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbcf81ab450 con 0x55911b194d80 2024-03-22T01:46:55.433 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.429+0000 7fbcf6d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55911b1a0bd0 0x55911b1b8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:55.433 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.429+0000 7fbcf6d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55911b1a0bd0 0x55911b1b8920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:40532/0 (socket says 172.21.15.72:40532) 2024-03-22T01:46:55.433 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.429+0000 7fbcf6d9d640 1 -- 172.21.15.72:0/4231005142 learned_addr learned my addr 172.21.15.72:0/4231005142 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:55.433 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.429+0000 7fbcf5d9b640 1 --2- 172.21.15.72:0/4231005142 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55911b1956a0 0x55911b1ac520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:55.433 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.429+0000 7fbcf659c640 1 --2- 172.21.15.72:0/4231005142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55911b194d80 0x55911b1b2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:55.433 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.429+0000 7fbcf6d9d640 1 -- 172.21.15.72:0/4231005142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55911b194d80 msgr2=0x55911b1b2fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:55.434 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.429+0000 7fbcf6d9d640 1 --2- 172.21.15.72:0/4231005142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55911b194d80 0x55911b1b2fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:55.434 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.429+0000 7fbcf6d9d640 1 -- 172.21.15.72:0/4231005142 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55911b1956a0 msgr2=0x55911b1ac520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:55.434 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.429+0000 7fbcf6d9d640 1 --2- 172.21.15.72:0/4231005142 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55911b1956a0 0x55911b1ac520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:55.434 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.429+0000 7fbcf6d9d640 1 -- 172.21.15.72:0/4231005142 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55911b19be10 con 0x55911b1a0bd0 2024-03-22T01:46:55.434 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.430+0000 7fbcf6d9d640 1 --2- 172.21.15.72:0/4231005142 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55911b1a0bd0 0x55911b1b8920 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fbcf000d010 tx=0x7fbcf00095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97686186ae16ace0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:55.434 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.430+0000 7fbcdffff640 1 -- 172.21.15.72:0/4231005142 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcf0030070 con 0x55911b1a0bd0 2024-03-22T01:46:55.434 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.430+0000 7fbcdffff640 1 -- 172.21.15.72:0/4231005142 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbcf0018040 con 0x55911b1a0bd0 2024-03-22T01:46:55.434 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.430+0000 7fbcdffff640 1 -- 172.21.15.72:0/4231005142 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcf002bcf0 con 0x55911b1a0bd0 2024-03-22T01:46:55.434 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.430+0000 7fbcff96b740 1 -- 172.21.15.72:0/4231005142 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55911b1a0bd0 msgr2=0x55911b1b8920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:55.434 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.430+0000 7fbcff96b740 1 --2- 172.21.15.72:0/4231005142 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55911b1a0bd0 0x55911b1b8920 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fbcf000d010 tx=0x7fbcf00095c0 comp rx=0 tx=0).stop 2024-03-22T01:46:55.434 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.430+0000 7fbcff96b740 1 -- 172.21.15.72:0/4231005142 shutdown_connections 2024-03-22T01:46:55.434 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.430+0000 7fbcff96b740 1 --2- 172.21.15.72:0/4231005142 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55911b1956a0 0x55911b1ac520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:55.434 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.430+0000 7fbcff96b740 1 --2- 172.21.15.72:0/4231005142 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55911b1a0bd0 0x55911b1b8920 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:55.434 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.430+0000 7fbcff96b740 1 --2- 172.21.15.72:0/4231005142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55911b194d80 0x55911b1b2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:55.435 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.430+0000 7fbcff96b740 1 -- 172.21.15.72:0/4231005142 >> 172.21.15.72:0/4231005142 conn(0x55911b18fbf0 msgr2=0x55911b18aae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:55.435 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.430+0000 7fbcff96b740 1 -- 172.21.15.72:0/4231005142 shutdown_connections 2024-03-22T01:46:55.435 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.430+0000 7fbcff96b740 1 -- 172.21.15.72:0/4231005142 wait complete. 2024-03-22T01:46:55.435 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.431+0000 7fbcff96b740 1 -- start start 2024-03-22T01:46:55.435 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.431+0000 7fbcff96b740 1 Processor -- start 2024-03-22T01:46:55.435 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.431+0000 7fbcff96b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55911b194d80 0x55911b1a44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:55.435 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.431+0000 7fbcff96b740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55911b1956a0 0x55911b1a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:55.435 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.431+0000 7fbcff96b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55911b1a0bd0 0x55911b1a4f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:55.435 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.431+0000 7fbcff96b740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fbcf81abb20 con 0x55911b1956a0 2024-03-22T01:46:55.435 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.431+0000 7fbcff96b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fbcf81acf90 con 0x55911b194d80 2024-03-22T01:46:55.435 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.431+0000 7fbcff96b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbcf81ac1f0 con 0x55911b1a0bd0 2024-03-22T01:46:55.435 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.431+0000 7fbcff96b740 10 client.0 _ll_register_callbacks cb 0x7fbcf0009460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:46:55.435 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.431+0000 7fbcf5d9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55911b1956a0 0x55911b1a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:55.435 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.431+0000 7fbcf5d9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55911b1956a0 0x55911b1a49f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:40548/0 (socket says 172.21.15.72:40548) 2024-03-22T01:46:55.436 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.431+0000 7fbcf5d9b640 1 -- 172.21.15.72:0/3140670023 learned_addr learned my addr 172.21.15.72:0/3140670023 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:55.436 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.431+0000 7fbcf659c640 1 --2- 172.21.15.72:0/3140670023 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55911b194d80 0x55911b1a44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:55.436 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.431+0000 7fbcf6d9d640 1 --2- 172.21.15.72:0/3140670023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55911b1a0bd0 0x55911b1a4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:55.436 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.431+0000 7fbcf5d9b640 1 -- 172.21.15.72:0/3140670023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55911b1a0bd0 msgr2=0x55911b1a4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:55.436 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.431+0000 7fbcf5d9b640 1 --2- 172.21.15.72:0/3140670023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55911b1a0bd0 0x55911b1a4f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:55.436 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.431+0000 7fbcf5d9b640 1 -- 172.21.15.72:0/3140670023 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55911b194d80 msgr2=0x55911b1a44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:46:55.436 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.431+0000 7fbcf5d9b640 1 --2- 172.21.15.72:0/3140670023 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55911b194d80 0x55911b1a44b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:55.436 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.431+0000 7fbcf5d9b640 1 -- 172.21.15.72:0/3140670023 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55911b215fa0 con 0x55911b1956a0 2024-03-22T01:46:55.436 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.431+0000 7fbcf659c640 1 --2- 172.21.15.72:0/3140670023 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55911b194d80 0x55911b1a44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:46:55.436 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.432+0000 7fbcf5d9b640 1 --2- 172.21.15.72:0/3140670023 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55911b1956a0 0x55911b1a49f0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x55911b19d120 tx=0x7fbce8032470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68a48d1f73c99d30 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:55.436 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.432+0000 7fbcddffb640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:46:55.436 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.432+0000 7fbcddffb640 1 -- 172.21.15.72:0/3140670023 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbce8039040 con 0x55911b1956a0 2024-03-22T01:46:55.436 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.432+0000 7fbcddffb640 1 -- 172.21.15.72:0/3140670023 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fbce8034440 con 0x55911b1956a0 2024-03-22T01:46:55.436 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.432+0000 7fbcff96b740 1 -- 172.21.15.72:0/3140670023 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbcf0018040 con 0x55911b1956a0 2024-03-22T01:46:55.437 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.432+0000 7fbcddffb640 1 -- 172.21.15.72:0/3140670023 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7fbce8061050 con 0x55911b1956a0 2024-03-22T01:46:55.437 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.432+0000 7fbcff96b740 10 client.16383 fetch_fsmap learned FSMap version 103 2024-03-22T01:46:55.437 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.432+0000 7fbcff96b740 1 -- 172.21.15.72:0/3140670023 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=4+}) -- 0x55911b2172e0 con 0x55911b1956a0 2024-03-22T01:46:55.437 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.432+0000 7fbcddffb640 1 -- 172.21.15.72:0/3140670023 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbcf0018040 con 0x55911b1956a0 2024-03-22T01:46:55.437 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.432+0000 7fbcddffb640 1 -- 172.21.15.72:0/3140670023 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 103) ==== 8330+0+0 (secure 0 0 0) 0x7fbce80346d0 con 0x55911b1956a0 2024-03-22T01:46:55.437 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.432+0000 7fbcff96b740 10 client.16383 fetch_fsmap finished waiting for FSMap version 103 2024-03-22T01:46:55.437 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.432+0000 7fbcff96b740 10 client.16383 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T01:46:55.437 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.432+0000 7fbcff96b740 20 client.16383 populate_metadata read hostname 'smithi072' 2024-03-22T01:46:55.437 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.432+0000 7fbcff96b740 1 --2- 172.21.15.72:0/3140670023 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55911b217ac0 0x55911b219ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:55.438 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.432+0000 7fbcff96b740 4 client.16383 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T01:46:55.438 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.432+0000 7fbcff96b740 1 -- 172.21.15.72:0/3140670023 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55911b1008d0 con 0x55911b217ac0 2024-03-22T01:46:55.438 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.432+0000 7fbcf659c640 1 --2- 172.21.15.72:0/3140670023 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55911b217ac0 0x55911b219ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:55.438 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.433+0000 7fbcf659c640 1 --2- 172.21.15.72:0/3140670023 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55911b217ac0 0x55911b219ee0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e15b167bc0d96b7e server_cookie=5389400ae1738077 in_seq=0 out_seq=0 2024-03-22T01:46:55.438 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.433+0000 7fbcddffb640 10 client.16383 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:46:55.439 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.436+0000 7fbcddffb640 1 -- 172.21.15.72:0/3140670023 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55911b1008d0 con 0x55911b217ac0 2024-03-22T01:46:55.439 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.436+0000 7fbcddffb640 10 client.16383 handle_command_reply: tid=0 2024-03-22T01:46:55.439 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.436+0000 7fbcff96b740 1 client.16383 shutdown 2024-03-22T01:46:55.439 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.437+0000 7fbcff96b740 1 -- 172.21.15.72:0/3140670023 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55911b1956a0 msgr2=0x55911b1a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:55.439 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.437+0000 7fbcff96b740 1 --2- 172.21.15.72:0/3140670023 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55911b1956a0 0x55911b1a49f0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x55911b19d120 tx=0x7fbce8032470 comp rx=0 tx=0).stop 2024-03-22T01:46:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.437+0000 7fbcff96b740 1 -- 172.21.15.72:0/3140670023 shutdown_connections 2024-03-22T01:46:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.437+0000 7fbcff96b740 1 --2- 172.21.15.72:0/3140670023 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55911b217ac0 0x55911b219ee0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.437+0000 7fbcff96b740 1 --2- 172.21.15.72:0/3140670023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55911b1a0bd0 0x55911b1a4f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.437+0000 7fbcff96b740 1 --2- 172.21.15.72:0/3140670023 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55911b1956a0 0x55911b1a49f0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.437+0000 7fbcddffb640 0 client.16383 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:46:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.437+0000 7fbcff96b740 1 --2- 172.21.15.72:0/3140670023 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55911b194d80 0x55911b1a44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.437+0000 7fbcff96b740 1 -- 172.21.15.72:0/3140670023 >> 172.21.15.72:0/3140670023 conn(0x55911b18fbf0 msgr2=0x55911b18e3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.437+0000 7fbcff96b740 1 -- 172.21.15.72:0/3140670023 shutdown_connections 2024-03-22T01:46:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.437+0000 7fbcff96b740 1 -- 172.21.15.72:0/3140670023 wait complete. 2024-03-22T01:46:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.437+0000 7fbcff96b740 20 client.16383 trim_cache size 0 max 16384 2024-03-22T01:46:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout:{ 2024-03-22T01:46:55.441 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout: "message": "", 2024-03-22T01:46:55.441 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout: "return_code": 0 2024-03-22T01:46:55.441 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stdout:} 2024-03-22T01:46:55.441 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.437+0000 7fbcfcd81640 1 -- 172.21.15.72:0/74359279 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fbce00a3100 msgr2=0x7fbce00a55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:55.442 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.437+0000 7fbcfcd81640 1 --2- 172.21.15.72:0/74359279 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fbce00a3100 0x7fbce00a55c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fbcf0009fd0 tx=0x7fbcf000e7a0 comp rx=0 tx=0).stop 2024-03-22T01:46:55.442 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.437+0000 7fbcfcd81640 1 -- 172.21.15.72:0/74359279 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbcf8100a40 msgr2=0x7fbcf8197340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:55.442 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.437+0000 7fbcfcd81640 1 --2- 172.21.15.72:0/74359279 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbcf8100a40 0x7fbcf8197340 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fbcec00d770 tx=0x7fbcec00dc30 comp rx=0 tx=0).stop 2024-03-22T01:46:55.442 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.437+0000 7fbcfcd81640 1 -- 172.21.15.72:0/74359279 shutdown_connections 2024-03-22T01:46:55.442 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.437+0000 7fbcfcd81640 1 --2- 172.21.15.72:0/74359279 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fbce00a3100 0x7fbce00a55c0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:55.442 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.437+0000 7fbcfcd81640 1 --2- 172.21.15.72:0/74359279 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbcf8106a50 0x7fbcf819bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:55.442 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.437+0000 7fbcfcd81640 1 --2- 172.21.15.72:0/74359279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbcf8101400 0x7fbcf8197880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:55.442 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.437+0000 7fbcfcd81640 1 --2- 172.21.15.72:0/74359279 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbcf8100a40 0x7fbcf8197340 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:55.442 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.437+0000 7fbcfcd81640 1 -- 172.21.15.72:0/74359279 >> 172.21.15.72:0/74359279 conn(0x7fbcf80fc530 msgr2=0x7fbcf80fcd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:55.442 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.437+0000 7fbcfcd81640 1 -- 172.21.15.72:0/74359279 shutdown_connections 2024-03-22T01:46:55.442 INFO:teuthology.task.background_exec.ubuntu@smithi072.front.sepia.ceph.com.smithi072.stderr:2024-03-22T01:46:55.437+0000 7fbcfcd81640 1 -- 172.21.15.72:0/74359279 wait complete. 2024-03-22T01:46:56.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:46:56 smithi098 ceph-mon[36468]: pgmap v311: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-22T01:46:56.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:46:56 smithi089 ceph-mon[35770]: pgmap v311: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-22T01:46:56.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:46:56 smithi072 ceph-mon[32414]: pgmap v311: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-22T01:46:58.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:46:58 smithi098 ceph-mon[36468]: pgmap v312: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 341 B/s wr, 3 op/s 2024-03-22T01:46:58.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:46:58 smithi089 ceph-mon[35770]: pgmap v312: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 341 B/s wr, 3 op/s 2024-03-22T01:46:58.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:46:58 smithi072 ceph-mon[32414]: pgmap v312: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 341 B/s wr, 3 op/s 2024-03-22T01:46:59.044 DEBUG:teuthology.parallel:result is None 2024-03-22T01:46:59.044 DEBUG:teuthology.orchestra.run.smithi072:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-03-22T01:46:59.119 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-03-22T01:46:59.119 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-03-22T01:46:59.131 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi072.front.sepia.ceph.com 2024-03-22T01:46:59.168 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-03-22T01:46:59.178 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-03-22T01:46:59.178 INFO:teuthology.task.exec:Executing custom commands... 2024-03-22T01:46:59.178 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi072.front.sepia.ceph.com 2024-03-22T01:46:59.179 DEBUG:teuthology.orchestra.run.smithi072:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-03-22T01:46:59.547 DEBUG:teuthology.orchestra.run.smithi072:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-03-22T01:46:59.717 DEBUG:teuthology.orchestra.run.smithi072:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-03-22T01:46:59.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:59.830+0000 7fa2fd90c640 1 -- 172.21.15.72:0/233475290 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa2f8069450 msgr2=0x7fa2f80698d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:59.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:59.830+0000 7fa2fd90c640 1 --2- 172.21.15.72:0/233475290 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa2f8069450 0x7fa2f80698d0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fa2e0009900 tx=0x7fa2e002bc30 comp rx=0 tx=0).stop 2024-03-22T01:46:59.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:59.830+0000 7fa2fd90c640 1 -- 172.21.15.72:0/233475290 shutdown_connections 2024-03-22T01:46:59.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:59.830+0000 7fa2fd90c640 1 --2- 172.21.15.72:0/233475290 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2f8069e10 0x7fa2f810e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:59.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:59.830+0000 7fa2fd90c640 1 --2- 172.21.15.72:0/233475290 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa2f8069450 0x7fa2f80698d0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:59.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:59.830+0000 7fa2fd90c640 1 --2- 172.21.15.72:0/233475290 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2f8107c10 0x7fa2f8107ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:59.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:59.830+0000 7fa2fd90c640 1 -- 172.21.15.72:0/233475290 >> 172.21.15.72:0/233475290 conn(0x7fa2f8077b80 msgr2=0x7fa2f8077f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:46:59.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:59.831+0000 7fa2fd90c640 1 -- 172.21.15.72:0/233475290 shutdown_connections 2024-03-22T01:46:59.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:59.831+0000 7fa2fd90c640 1 -- 172.21.15.72:0/233475290 wait complete. 2024-03-22T01:46:59.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:59.831+0000 7fa2fd90c640 1 Processor -- start 2024-03-22T01:46:59.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:59.831+0000 7fa2fd90c640 1 -- start start 2024-03-22T01:46:59.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:59.832+0000 7fa2fd90c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa2f8069450 0x7fa2f81983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:59.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:59.832+0000 7fa2fd90c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2f8069e10 0x7fa2f8198930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:59.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:59.832+0000 7fa2fd90c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2f8107c10 0x7fa2f819cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:59.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:59.832+0000 7fa2fd90c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fa2f8111f50 con 0x7fa2f8069450 2024-03-22T01:46:59.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:59.832+0000 7fa2fd90c640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa2f8111dd0 con 0x7fa2f8107c10 2024-03-22T01:46:59.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:59.832+0000 7fa2fd90c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa2f81120d0 con 0x7fa2f8069e10 2024-03-22T01:46:59.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:59.832+0000 7fa2f6ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa2f8069450 0x7fa2f81983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:59.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:59.832+0000 7fa2f6ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa2f8069450 0x7fa2f81983f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:40564/0 (socket says 172.21.15.72:40564) 2024-03-22T01:46:59.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:59.832+0000 7fa2f6ffd640 1 -- 172.21.15.72:0/411021689 learned_addr learned my addr 172.21.15.72:0/411021689 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:46:59.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:59.832+0000 7fa2f77fe640 1 --2- 172.21.15.72:0/411021689 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2f8107c10 0x7fa2f819cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:59.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:59.832+0000 7fa2f67fc640 1 --2- 172.21.15.72:0/411021689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2f8069e10 0x7fa2f8198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:59.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:59.832+0000 7fa2f6ffd640 1 -- 172.21.15.72:0/411021689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2f8069e10 msgr2=0x7fa2f8198930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:59.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:59.832+0000 7fa2f6ffd640 1 --2- 172.21.15.72:0/411021689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2f8069e10 0x7fa2f8198930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:59.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:59.832+0000 7fa2f6ffd640 1 -- 172.21.15.72:0/411021689 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2f8107c10 msgr2=0x7fa2f819cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:46:59.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:59.832+0000 7fa2f6ffd640 1 --2- 172.21.15.72:0/411021689 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2f8107c10 0x7fa2f819cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:46:59.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:59.832+0000 7fa2f6ffd640 1 -- 172.21.15.72:0/411021689 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2f819d390 con 0x7fa2f8069450 2024-03-22T01:46:59.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:59.832+0000 7fa2f67fc640 1 --2- 172.21.15.72:0/411021689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2f8069e10 0x7fa2f8198930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T01:46:59.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:59.832+0000 7fa2f6ffd640 1 --2- 172.21.15.72:0/411021689 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa2f8069450 0x7fa2f81983f0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fa2e800ebc0 tx=0x7fa2e800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:59.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:59.832+0000 7fa2fc90a640 1 -- 172.21.15.72:0/411021689 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2e8010630 con 0x7fa2f8069450 2024-03-22T01:46:59.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:59.832+0000 7fa2fd90c640 1 -- 172.21.15.72:0/411021689 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2f819d680 con 0x7fa2f8069450 2024-03-22T01:46:59.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:59.832+0000 7fa2fc90a640 1 -- 172.21.15.72:0/411021689 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2e8010cb0 con 0x7fa2f8069450 2024-03-22T01:46:59.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:59.832+0000 7fa2fc90a640 1 -- 172.21.15.72:0/411021689 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2e8018bd0 con 0x7fa2f8069450 2024-03-22T01:46:59.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:59.832+0000 7fa2fd90c640 1 -- 172.21.15.72:0/411021689 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2f81a4f10 con 0x7fa2f8069450 2024-03-22T01:46:59.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:59.833+0000 7fa2fd90c640 1 -- 172.21.15.72:0/411021689 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2f8050140 con 0x7fa2f8069450 2024-03-22T01:46:59.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:59.835+0000 7fa2fc90a640 1 -- 172.21.15.72:0/411021689 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 25) ==== 137985+0+0 (secure 0 0 0) 0x7fa2e80107d0 con 0x7fa2f8069450 2024-03-22T01:46:59.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:59.835+0000 7fa2fc90a640 1 --2- 172.21.15.72:0/411021689 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fa2e40a3080 0x7fa2e40a5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:46:59.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:59.835+0000 7fa2f67fc640 1 --2- 172.21.15.72:0/411021689 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fa2e40a3080 0x7fa2e40a5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:46:59.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:59.835+0000 7fa2fc90a640 1 -- 172.21.15.72:0/411021689 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fa2e80d0440 con 0x7fa2f8069450 2024-03-22T01:46:59.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:59.836+0000 7fa2f67fc640 1 --2- 172.21.15.72:0/411021689 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fa2e40a3080 0x7fa2e40a5540 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa2e002efd0 tx=0x7fa2e0000bc0 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:46:59.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:46:59.839+0000 7fa2fc90a640 1 -- 172.21.15.72:0/411021689 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa2e809cb80 con 0x7fa2f8069450 2024-03-22T01:47:00.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.035+0000 7fa2fd90c640 1 -- 172.21.15.72:0/411021689 --> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7fa2f8061fd0 con 0x7fa2e40a3080 2024-03-22T01:47:00.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.042+0000 7fa2fc90a640 1 -- 172.21.15.72:0/411021689 <== mgr.15381 v2:172.21.15.72:6800/1379139045 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+332 (secure 0 0 0) 0x7fa2f8061fd0 con 0x7fa2e40a3080 2024-03-22T01:47:00.041 INFO:teuthology.orchestra.run.smithi072.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-03-22T00:00:00", "created": "2024-03-22T01:36:43", "first": "2024-03-22T01:37:00", "last": "2024-03-22T01:46:00", "last_pruned": null, "created_count": 4, "pruned_count": 0, "active": true} 2024-03-22T01:47:00.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.044+0000 7fa2fd90c640 1 -- 172.21.15.72:0/411021689 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fa2e40a3080 msgr2=0x7fa2e40a5540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:00.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.044+0000 7fa2fd90c640 1 --2- 172.21.15.72:0/411021689 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fa2e40a3080 0x7fa2e40a5540 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa2e002efd0 tx=0x7fa2e0000bc0 comp rx=0 tx=0).stop 2024-03-22T01:47:00.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.044+0000 7fa2fd90c640 1 -- 172.21.15.72:0/411021689 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa2f8069450 msgr2=0x7fa2f81983f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:00.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.044+0000 7fa2fd90c640 1 --2- 172.21.15.72:0/411021689 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa2f8069450 0x7fa2f81983f0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fa2e800ebc0 tx=0x7fa2e800c6a0 comp rx=0 tx=0).stop 2024-03-22T01:47:00.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.044+0000 7fa2fd90c640 1 -- 172.21.15.72:0/411021689 shutdown_connections 2024-03-22T01:47:00.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.044+0000 7fa2fd90c640 1 --2- 172.21.15.72:0/411021689 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fa2e40a3080 0x7fa2e40a5540 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:00.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.044+0000 7fa2fd90c640 1 --2- 172.21.15.72:0/411021689 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2f8107c10 0x7fa2f819cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:00.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.044+0000 7fa2fd90c640 1 --2- 172.21.15.72:0/411021689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2f8069e10 0x7fa2f8198930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:00.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.044+0000 7fa2fd90c640 1 --2- 172.21.15.72:0/411021689 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa2f8069450 0x7fa2f81983f0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:00.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.044+0000 7fa2fd90c640 1 -- 172.21.15.72:0/411021689 >> 172.21.15.72:0/411021689 conn(0x7fa2f8077b80 msgr2=0x7fa2f80ff5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:47:00.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.044+0000 7fa2fd90c640 1 -- 172.21.15.72:0/411021689 shutdown_connections 2024-03-22T01:47:00.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.044+0000 7fa2fd90c640 1 -- 172.21.15.72:0/411021689 wait complete. 2024-03-22T01:47:00.054 DEBUG:teuthology.orchestra.run.smithi072:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-03-22T01:47:00.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.164+0000 7f6218fd6640 1 -- 172.21.15.72:0/4229403987 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6214100a40 msgr2=0x7f6214100ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:00.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.164+0000 7f6218fd6640 1 --2- 172.21.15.72:0/4229403987 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6214100a40 0x7f6214100ec0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f61fc009860 tx=0x7f61fc02bb70 comp rx=0 tx=0).stop 2024-03-22T01:47:00.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.164+0000 7f6218fd6640 1 -- 172.21.15.72:0/4229403987 shutdown_connections 2024-03-22T01:47:00.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.164+0000 7f6218fd6640 1 --2- 172.21.15.72:0/4229403987 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6214100a40 0x7f6214100ec0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:00.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.164+0000 7f6218fd6640 1 --2- 172.21.15.72:0/4229403987 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6214101400 0x7f621410d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:00.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.164+0000 7f6218fd6640 1 --2- 172.21.15.72:0/4229403987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6214106a50 0x7f6214106e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:00.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.164+0000 7f6218fd6640 1 -- 172.21.15.72:0/4229403987 >> 172.21.15.72:0/4229403987 conn(0x7f6214077be0 msgr2=0x7f6214077ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:47:00.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.164+0000 7f6218fd6640 1 -- 172.21.15.72:0/4229403987 shutdown_connections 2024-03-22T01:47:00.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.164+0000 7f6218fd6640 1 -- 172.21.15.72:0/4229403987 wait complete. 2024-03-22T01:47:00.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.165+0000 7f6218fd6640 1 Processor -- start 2024-03-22T01:47:00.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.165+0000 7f6218fd6640 1 -- start start 2024-03-22T01:47:00.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.165+0000 7f6218fd6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6214100a40 0x7f6214198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:00.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.165+0000 7f6218fd6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6214101400 0x7f6214198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:00.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.165+0000 7f6218fd6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6214106a50 0x7f621419ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:00.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.166+0000 7f6218fd6640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f6214110e60 con 0x7f6214101400 2024-03-22T01:47:00.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.166+0000 7f6218fd6640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f6214110ce0 con 0x7f6214100a40 2024-03-22T01:47:00.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.166+0000 7f6218fd6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6214110fe0 con 0x7f6214106a50 2024-03-22T01:47:00.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.166+0000 7f620bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6214101400 0x7f6214198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:00.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.166+0000 7f621259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6214100a40 0x7f6214198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:00.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.166+0000 7f6212d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6214106a50 0x7f621419ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:00.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.166+0000 7f621259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6214100a40 0x7f6214198430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.72:49852/0 (socket says 172.21.15.72:49852) 2024-03-22T01:47:00.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.166+0000 7f621259c640 1 -- 172.21.15.72:0/2197470973 learned_addr learned my addr 172.21.15.72:0/2197470973 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:47:00.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.166+0000 7f6212d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6214106a50 0x7f621419ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.72:58756/0 (socket says 172.21.15.72:58756) 2024-03-22T01:47:00.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.166+0000 7f621259c640 1 -- 172.21.15.72:0/2197470973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6214106a50 msgr2=0x7f621419ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:00.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.166+0000 7f621259c640 1 --2- 172.21.15.72:0/2197470973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6214106a50 0x7f621419ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:00.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.166+0000 7f621259c640 1 -- 172.21.15.72:0/2197470973 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6214101400 msgr2=0x7f6214198970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:00.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.166+0000 7f621259c640 1 --2- 172.21.15.72:0/2197470973 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6214101400 0x7f6214198970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:00.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.166+0000 7f621259c640 1 -- 172.21.15.72:0/2197470973 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f621419d420 con 0x7f6214100a40 2024-03-22T01:47:00.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.166+0000 7f621259c640 1 --2- 172.21.15.72:0/2197470973 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6214100a40 0x7f6214198430 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f620400e9f0 tx=0x7f620400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:47:00.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.166+0000 7f620b7fe640 1 -- 172.21.15.72:0/2197470973 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f620400cd80 con 0x7f6214100a40 2024-03-22T01:47:00.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.167+0000 7f620b7fe640 1 -- 172.21.15.72:0/2197470973 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6204005500 con 0x7f6214100a40 2024-03-22T01:47:00.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.167+0000 7f6218fd6640 1 -- 172.21.15.72:0/2197470973 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f621419d710 con 0x7f6214100a40 2024-03-22T01:47:00.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.167+0000 7f620b7fe640 1 -- 172.21.15.72:0/2197470973 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6204010690 con 0x7f6214100a40 2024-03-22T01:47:00.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.167+0000 7f6218fd6640 1 -- 172.21.15.72:0/2197470973 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62141a4fa0 con 0x7f6214100a40 2024-03-22T01:47:00.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.168+0000 7f6218fd6640 1 -- 172.21.15.72:0/2197470973 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6214050140 con 0x7f6214100a40 2024-03-22T01:47:00.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.169+0000 7f620b7fe640 1 -- 172.21.15.72:0/2197470973 <== mon.2 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 25) ==== 137985+0+0 (secure 0 0 0) 0x7f6204004f60 con 0x7f6214100a40 2024-03-22T01:47:00.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.169+0000 7f620b7fe640 1 --2- 172.21.15.72:0/2197470973 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f62000a3080 0x7f62000a5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:00.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.169+0000 7f620bfff640 1 --2- 172.21.15.72:0/2197470973 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f62000a3080 0x7f62000a5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:00.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.169+0000 7f620b7fe640 1 -- 172.21.15.72:0/2197470973 <== mon.2 v2:172.21.15.89:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f62040cff40 con 0x7f6214100a40 2024-03-22T01:47:00.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.170+0000 7f620bfff640 1 --2- 172.21.15.72:0/2197470973 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f62000a3080 0x7f62000a5540 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f61fc02c180 tx=0x7f61fc0056d0 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:47:00.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.174+0000 7f620b7fe640 1 -- 172.21.15.72:0/2197470973 <== mon.2 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f620409c680 con 0x7f6214100a40 2024-03-22T01:47:00.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:00 smithi072 ceph-mon[32414]: pgmap v313: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 85 B/s wr, 1 op/s 2024-03-22T01:47:00.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.375+0000 7f6218fd6640 1 -- 172.21.15.72:0/2197470973 --> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f62141033d0 con 0x7f62000a3080 2024-03-22T01:47:00.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.382+0000 7f620b7fe640 1 -- 172.21.15.72:0/2197470973 <== mgr.15381 v2:172.21.15.72:6800/1379139045 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f62141033d0 con 0x7f62000a3080 2024-03-22T01:47:00.381 INFO:teuthology.orchestra.run.smithi072.stdout:/ 1m 6m3h 2024-03-22T01:47:00.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.384+0000 7f6218fd6640 1 -- 172.21.15.72:0/2197470973 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f62000a3080 msgr2=0x7f62000a5540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:00.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.384+0000 7f6218fd6640 1 --2- 172.21.15.72:0/2197470973 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f62000a3080 0x7f62000a5540 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f61fc02c180 tx=0x7f61fc0056d0 comp rx=0 tx=0).stop 2024-03-22T01:47:00.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.384+0000 7f6218fd6640 1 -- 172.21.15.72:0/2197470973 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6214100a40 msgr2=0x7f6214198430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:00.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.384+0000 7f6218fd6640 1 --2- 172.21.15.72:0/2197470973 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6214100a40 0x7f6214198430 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f620400e9f0 tx=0x7f620400edf0 comp rx=0 tx=0).stop 2024-03-22T01:47:00.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.384+0000 7f6218fd6640 1 -- 172.21.15.72:0/2197470973 shutdown_connections 2024-03-22T01:47:00.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.384+0000 7f6218fd6640 1 --2- 172.21.15.72:0/2197470973 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f62000a3080 0x7f62000a5540 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:00.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.384+0000 7f6218fd6640 1 --2- 172.21.15.72:0/2197470973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6214106a50 0x7f621419ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:00.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.384+0000 7f6218fd6640 1 --2- 172.21.15.72:0/2197470973 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6214101400 0x7f6214198970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:00.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.384+0000 7f6218fd6640 1 --2- 172.21.15.72:0/2197470973 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6214100a40 0x7f6214198430 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:00.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.384+0000 7f6218fd6640 1 -- 172.21.15.72:0/2197470973 >> 172.21.15.72:0/2197470973 conn(0x7f6214077be0 msgr2=0x7f62140ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:47:00.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.384+0000 7f6218fd6640 1 -- 172.21.15.72:0/2197470973 shutdown_connections 2024-03-22T01:47:00.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:00.384+0000 7f6218fd6640 1 -- 172.21.15.72:0/2197470973 wait complete. 2024-03-22T01:47:00.394 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-03-22T01:47:00.405 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-03-22T01:47:00.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:00 smithi098 ceph-mon[36468]: pgmap v313: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 85 B/s wr, 1 op/s 2024-03-22T01:47:00.429 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:00 smithi089 ceph-mon[35770]: pgmap v313: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 85 B/s wr, 1 op/s 2024-03-22T01:47:01.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:01 smithi098 ceph-mon[36468]: from='client.16389 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:47:01.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:01 smithi089 ceph-mon[35770]: from='client.16389 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:47:01.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:01 smithi072 ceph-mon[32414]: from='client.16389 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:47:02.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:02 smithi098 ceph-mon[36468]: from='client.26228 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:47:02.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:02 smithi098 ceph-mon[36468]: pgmap v314: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-22T01:47:02.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:02 smithi089 ceph-mon[35770]: from='client.26228 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:47:02.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:02 smithi089 ceph-mon[35770]: pgmap v314: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-22T01:47:02.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:02 smithi072 ceph-mon[32414]: from='client.26228 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T01:47:02.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:02 smithi072 ceph-mon[32414]: pgmap v314: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-22T01:47:04.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:04 smithi098 ceph-mon[36468]: pgmap v315: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 19 op/s 2024-03-22T01:47:04.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:04 smithi089 ceph-mon[35770]: pgmap v315: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 19 op/s 2024-03-22T01:47:04.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:04 smithi072 ceph-mon[32414]: pgmap v315: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 19 op/s 2024-03-22T01:47:05.236 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T01:47:05.236 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T01:47:05.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.349+0000 7fa439884640 1 -- 172.21.15.72:0/1732635192 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa434106a40 msgr2=0x7fa434106e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:05.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.349+0000 7fa439884640 1 --2- 172.21.15.72:0/1732635192 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa434106a40 0x7fa434106e20 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fa424009930 tx=0x7fa42402bc30 comp rx=0 tx=0).stop 2024-03-22T01:47:05.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.349+0000 7fa439884640 1 -- 172.21.15.72:0/1732635192 shutdown_connections 2024-03-22T01:47:05.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.349+0000 7fa439884640 1 --2- 172.21.15.72:0/1732635192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa434100a30 0x7fa434100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.349+0000 7fa439884640 1 --2- 172.21.15.72:0/1732635192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4341013f0 0x7fa43410d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.349+0000 7fa439884640 1 --2- 172.21.15.72:0/1732635192 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa434106a40 0x7fa434106e20 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.349+0000 7fa439884640 1 -- 172.21.15.72:0/1732635192 >> 172.21.15.72:0/1732635192 conn(0x7fa434077c40 msgr2=0x7fa434078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:47:05.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.349+0000 7fa439884640 1 -- 172.21.15.72:0/1732635192 shutdown_connections 2024-03-22T01:47:05.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.349+0000 7fa439884640 1 -- 172.21.15.72:0/1732635192 wait complete. 2024-03-22T01:47:05.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.349+0000 7fa439884640 1 Processor -- start 2024-03-22T01:47:05.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.350+0000 7fa439884640 1 -- start start 2024-03-22T01:47:05.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.350+0000 7fa439884640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa434100a30 0x7fa434197390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:05.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.350+0000 7fa439884640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4341013f0 0x7fa4341978d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:05.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.350+0000 7fa439884640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa434106a40 0x7fa43419bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:05.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.350+0000 7fa439884640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fa434110e50 con 0x7fa434106a40 2024-03-22T01:47:05.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.350+0000 7fa439884640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa434110cd0 con 0x7fa4341013f0 2024-03-22T01:47:05.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.350+0000 7fa439884640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa434110fd0 con 0x7fa434100a30 2024-03-22T01:47:05.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.350+0000 7fa4337fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa434106a40 0x7fa43419bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:05.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.350+0000 7fa4337fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa434106a40 0x7fa43419bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59252/0 (socket says 172.21.15.72:59252) 2024-03-22T01:47:05.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.350+0000 7fa4337fe640 1 -- 172.21.15.72:0/4169468437 learned_addr learned my addr 172.21.15.72:0/4169468437 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:47:05.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.350+0000 7fa4327fc640 1 --2- 172.21.15.72:0/4169468437 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4341013f0 0x7fa4341978d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:05.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.350+0000 7fa432ffd640 1 --2- 172.21.15.72:0/4169468437 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa434100a30 0x7fa434197390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:05.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.350+0000 7fa4327fc640 1 -- 172.21.15.72:0/4169468437 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa434100a30 msgr2=0x7fa434197390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:05.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.350+0000 7fa4327fc640 1 --2- 172.21.15.72:0/4169468437 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa434100a30 0x7fa434197390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.350+0000 7fa4327fc640 1 -- 172.21.15.72:0/4169468437 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa434106a40 msgr2=0x7fa43419bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:05.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.350+0000 7fa4327fc640 1 --2- 172.21.15.72:0/4169468437 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa434106a40 0x7fa43419bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.350+0000 7fa4327fc640 1 -- 172.21.15.72:0/4169468437 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa43419c330 con 0x7fa4341013f0 2024-03-22T01:47:05.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.350+0000 7fa4337fe640 1 --2- 172.21.15.72:0/4169468437 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa434106a40 0x7fa43419bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:47:05.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.350+0000 7fa4327fc640 1 --2- 172.21.15.72:0/4169468437 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4341013f0 0x7fa4341978d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa42800e9f0 tx=0x7fa42800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:47:05.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.351+0000 7fa438882640 1 -- 172.21.15.72:0/4169468437 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa42800ccc0 con 0x7fa4341013f0 2024-03-22T01:47:05.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.351+0000 7fa438882640 1 -- 172.21.15.72:0/4169468437 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa428010910 con 0x7fa4341013f0 2024-03-22T01:47:05.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.351+0000 7fa439884640 1 -- 172.21.15.72:0/4169468437 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa43419c620 con 0x7fa4341013f0 2024-03-22T01:47:05.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.351+0000 7fa438882640 1 -- 172.21.15.72:0/4169468437 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa428018960 con 0x7fa4341013f0 2024-03-22T01:47:05.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.351+0000 7fa439884640 1 -- 172.21.15.72:0/4169468437 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa434068e60 con 0x7fa4341013f0 2024-03-22T01:47:05.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.352+0000 7fa438882640 1 -- 172.21.15.72:0/4169468437 <== mon.2 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 25) ==== 137985+0+0 (secure 0 0 0) 0x7fa428010ac0 con 0x7fa4341013f0 2024-03-22T01:47:05.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.353+0000 7fa438882640 1 --2- 172.21.15.72:0/4169468437 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fa41c0a3090 0x7fa41c0a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:05.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.353+0000 7fa43c46d740 1 Processor -- start 2024-03-22T01:47:05.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.353+0000 7fa43c46d740 1 -- start start 2024-03-22T01:47:05.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.353+0000 7fa432ffd640 1 --2- 172.21.15.72:0/4169468437 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fa41c0a3090 0x7fa41c0a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:05.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.353+0000 7fa438882640 1 -- 172.21.15.72:0/4169468437 <== mon.2 v2:172.21.15.89:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fa4280cffa0 con 0x7fa4341013f0 2024-03-22T01:47:05.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.353+0000 7fa43c46d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d298539410 0x55d2985397f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:05.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.353+0000 7fa43c46d740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d298539d30 0x55d2985423d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:05.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.353+0000 7fa43c46d740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d298542910 0x55d298544d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:05.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.353+0000 7fa43c46d740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55d29849d8c0 con 0x55d298539d30 2024-03-22T01:47:05.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.353+0000 7fa43c46d740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55d298445c20 con 0x55d298542910 2024-03-22T01:47:05.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.353+0000 7fa43c46d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d298499870 con 0x55d298539410 2024-03-22T01:47:05.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.353+0000 7fa4327fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d298539d30 0x55d2985423d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:05.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.354+0000 7fa4327fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d298539d30 0x55d2985423d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59262/0 (socket says 172.21.15.72:59262) 2024-03-22T01:47:05.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.354+0000 7fa4327fc640 1 -- 172.21.15.72:0/1500233641 learned_addr learned my addr 172.21.15.72:0/1500233641 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:47:05.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.354+0000 7fa4337fe640 1 --2- 172.21.15.72:0/1500233641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d298542910 0x55d298544d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:05.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.354+0000 7fa432ffd640 1 --2- 172.21.15.72:0/1500233641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d298539410 0x55d2985397f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:05.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.354+0000 7fa432ffd640 1 --2- 172.21.15.72:0/4169468437 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fa41c0a3090 0x7fa41c0a5550 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa424005ab0 tx=0x7fa424005fd0 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:47:05.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.354+0000 7fa4337fe640 1 -- 172.21.15.72:0/1500233641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d298539410 msgr2=0x55d2985397f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:47:05.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.354+0000 7fa4337fe640 1 --2- 172.21.15.72:0/1500233641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d298539410 0x55d2985397f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.354+0000 7fa4337fe640 1 -- 172.21.15.72:0/1500233641 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d298539d30 msgr2=0x55d2985423d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:47:05.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.354+0000 7fa4337fe640 1 --2- 172.21.15.72:0/1500233641 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d298539d30 0x55d2985423d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.354+0000 7fa4337fe640 1 -- 172.21.15.72:0/1500233641 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d29846a150 con 0x55d298542910 2024-03-22T01:47:05.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.354+0000 7fa432ffd640 1 --2- 172.21.15.72:0/1500233641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d298539410 0x55d2985397f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:47:05.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.354+0000 7fa4327fc640 1 --2- 172.21.15.72:0/1500233641 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d298539d30 0x55d2985423d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:47:05.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.354+0000 7fa4337fe640 1 --2- 172.21.15.72:0/1500233641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d298542910 0x55d298544d00 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa42c0049d0 tx=0x7fa42c009500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=737b1ae8605c7f68 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:47:05.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.354+0000 7fa419ffb640 1 -- 172.21.15.72:0/1500233641 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa42c01c070 con 0x55d298542910 2024-03-22T01:47:05.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.354+0000 7fa419ffb640 1 -- 172.21.15.72:0/1500233641 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa42c004d20 con 0x55d298542910 2024-03-22T01:47:05.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.354+0000 7fa419ffb640 1 -- 172.21.15.72:0/1500233641 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa42c005750 con 0x55d298542910 2024-03-22T01:47:05.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.355+0000 7fa43c46d740 1 -- 172.21.15.72:0/1500233641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d298542910 msgr2=0x55d298544d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:05.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.355+0000 7fa43c46d740 1 --2- 172.21.15.72:0/1500233641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d298542910 0x55d298544d00 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa42c0049d0 tx=0x7fa42c009500 comp rx=0 tx=0).stop 2024-03-22T01:47:05.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.355+0000 7fa43c46d740 1 -- 172.21.15.72:0/1500233641 shutdown_connections 2024-03-22T01:47:05.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.355+0000 7fa43c46d740 1 --2- 172.21.15.72:0/1500233641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d298542910 0x55d298544d00 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.355+0000 7fa43c46d740 1 --2- 172.21.15.72:0/1500233641 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d298539d30 0x55d2985423d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.355+0000 7fa43c46d740 1 --2- 172.21.15.72:0/1500233641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d298539410 0x55d2985397f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.355+0000 7fa43c46d740 1 -- 172.21.15.72:0/1500233641 >> 172.21.15.72:0/1500233641 conn(0x55d298533b40 msgr2=0x55d298533f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:47:05.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.355+0000 7fa43c46d740 1 -- 172.21.15.72:0/1500233641 shutdown_connections 2024-03-22T01:47:05.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.355+0000 7fa43c46d740 1 -- 172.21.15.72:0/1500233641 wait complete. 2024-03-22T01:47:05.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.355+0000 7fa43c46d740 1 -- start start 2024-03-22T01:47:05.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.355+0000 7fa43c46d740 1 Processor -- start 2024-03-22T01:47:05.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.356+0000 7fa43c46d740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d298539410 0x55d29853f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:05.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.356+0000 7fa43c46d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d298539d30 0x55d29853fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:05.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.356+0000 7fa43c46d740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d298542910 0x55d298540350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:05.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.356+0000 7fa43c46d740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55d298545700 con 0x55d298542910 2024-03-22T01:47:05.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.356+0000 7fa43c46d740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55d298459130 con 0x55d298539410 2024-03-22T01:47:05.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.356+0000 7fa43c46d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d298545880 con 0x55d298539d30 2024-03-22T01:47:05.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.356+0000 7fa43c46d740 10 client.0 _ll_register_callbacks cb 0x55d298458b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:47:05.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.356+0000 7fa432ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d298539410 0x55d29853f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:05.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.356+0000 7fa4337fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d298542910 0x55d298540350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:05.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.356+0000 7fa432ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d298539410 0x55d29853f8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.72:51294/0 (socket says 172.21.15.72:51294) 2024-03-22T01:47:05.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.356+0000 7fa432ffd640 1 -- 172.21.15.72:0/4143846133 learned_addr learned my addr 172.21.15.72:0/4143846133 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:47:05.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.356+0000 7fa4327fc640 1 --2- 172.21.15.72:0/4143846133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d298539d30 0x55d29853fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:05.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.356+0000 7fa4337fe640 1 -- 172.21.15.72:0/4143846133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d298539d30 msgr2=0x55d29853fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:47:05.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.356+0000 7fa4337fe640 1 --2- 172.21.15.72:0/4143846133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d298539d30 0x55d29853fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.356+0000 7fa4337fe640 1 -- 172.21.15.72:0/4143846133 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d298539410 msgr2=0x55d29853f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:47:05.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.356+0000 7fa4337fe640 1 --2- 172.21.15.72:0/4143846133 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d298539410 0x55d29853f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.356+0000 7fa4337fe640 1 -- 172.21.15.72:0/4143846133 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d29827b7d0 con 0x55d298542910 2024-03-22T01:47:05.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.356+0000 7fa432ffd640 1 --2- 172.21.15.72:0/4143846133 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d298539410 0x55d29853f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:47:05.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.356+0000 7fa4337fe640 1 --2- 172.21.15.72:0/4143846133 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d298542910 0x55d298540350 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fa42c010010 tx=0x7fa42c00e500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cb35180ce8b53da server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:47:05.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.356+0000 7fa412ffd640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:47:05.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.356+0000 7fa412ffd640 1 -- 172.21.15.72:0/4143846133 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa42c01c070 con 0x55d298542910 2024-03-22T01:47:05.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.356+0000 7fa412ffd640 1 -- 172.21.15.72:0/4143846133 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fa42c005870 con 0x55d298542910 2024-03-22T01:47:05.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.356+0000 7fa43c46d740 1 -- 172.21.15.72:0/4143846133 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa42c004d20 con 0x55d298542910 2024-03-22T01:47:05.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.357+0000 7fa412ffd640 1 -- 172.21.15.72:0/4143846133 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7fa42c00b820 con 0x55d298542910 2024-03-22T01:47:05.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.357+0000 7fa43c46d740 10 client.16407 fetch_fsmap learned FSMap version 103 2024-03-22T01:47:05.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.357+0000 7fa43c46d740 1 -- 172.21.15.72:0/4143846133 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=4+}) -- 0x55d2981b13f0 con 0x55d298542910 2024-03-22T01:47:05.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.357+0000 7fa412ffd640 1 -- 172.21.15.72:0/4143846133 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa42c004d20 con 0x55d298542910 2024-03-22T01:47:05.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.357+0000 7fa412ffd640 1 -- 172.21.15.72:0/4143846133 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 103) ==== 8330+0+0 (secure 0 0 0) 0x7fa42c02aa80 con 0x55d298542910 2024-03-22T01:47:05.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.357+0000 7fa43c46d740 10 client.16407 fetch_fsmap finished waiting for FSMap version 103 2024-03-22T01:47:05.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.357+0000 7fa43c46d740 10 client.16407 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T01:47:05.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.357+0000 7fa43c46d740 20 client.16407 populate_metadata read hostname 'smithi072' 2024-03-22T01:47:05.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.357+0000 7fa43c46d740 1 --2- 172.21.15.72:0/4143846133 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55d298552b70 0x55d298554f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:05.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.357+0000 7fa43c46d740 4 client.16407 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T01:47:05.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.357+0000 7fa43c46d740 1 -- 172.21.15.72:0/4143846133 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d2984aae00 con 0x55d298552b70 2024-03-22T01:47:05.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.357+0000 7fa432ffd640 1 --2- 172.21.15.72:0/4143846133 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55d298552b70 0x55d298554f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:05.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.358+0000 7fa412ffd640 10 client.16407 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:47:05.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.358+0000 7fa432ffd640 1 --2- 172.21.15.72:0/4143846133 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55d298552b70 0x55d298554f60 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a756fd9f62f0a772 server_cookie=aec9d6114ee5960a in_seq=0 out_seq=0 2024-03-22T01:47:05.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.358+0000 7fa412ffd640 1 -- 172.21.15.72:0/4143846133 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d2984aae00 con 0x55d298552b70 2024-03-22T01:47:05.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.358+0000 7fa412ffd640 10 client.16407 handle_command_reply: tid=0 2024-03-22T01:47:05.371 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.358+0000 7fa43c46d740 1 client.16407 shutdown 2024-03-22T01:47:05.371 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.358+0000 7fa43c46d740 1 -- 172.21.15.72:0/4143846133 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d298542910 msgr2=0x55d298540350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:05.371 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.358+0000 7fa43c46d740 1 --2- 172.21.15.72:0/4143846133 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d298542910 0x55d298540350 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fa42c010010 tx=0x7fa42c00e500 comp rx=0 tx=0).stop 2024-03-22T01:47:05.371 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.358+0000 7fa43c46d740 1 -- 172.21.15.72:0/4143846133 shutdown_connections 2024-03-22T01:47:05.371 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.358+0000 7fa43c46d740 1 --2- 172.21.15.72:0/4143846133 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55d298552b70 0x55d298554f60 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.371 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.358+0000 7fa43c46d740 1 --2- 172.21.15.72:0/4143846133 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d298542910 0x55d298540350 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.371 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.358+0000 7fa43c46d740 1 --2- 172.21.15.72:0/4143846133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d298539d30 0x55d29853fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.371 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.358+0000 7fa412ffd640 0 client.16407 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:47:05.371 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.358+0000 7fa43c46d740 1 --2- 172.21.15.72:0/4143846133 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d298539410 0x55d29853f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.371 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.358+0000 7fa43c46d740 1 -- 172.21.15.72:0/4143846133 >> 172.21.15.72:0/4143846133 conn(0x55d298533b40 msgr2=0x55d298535890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:47:05.371 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.358+0000 7fa43c46d740 1 -- 172.21.15.72:0/4143846133 shutdown_connections 2024-03-22T01:47:05.371 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.358+0000 7fa43c46d740 1 -- 172.21.15.72:0/4143846133 wait complete. 2024-03-22T01:47:05.371 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.359+0000 7fa43c46d740 20 client.16407 trim_cache size 0 max 16384 2024-03-22T01:47:05.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.376+0000 7fa43c46d740 1 Processor -- start 2024-03-22T01:47:05.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.376+0000 7fa43c46d740 1 -- start start 2024-03-22T01:47:05.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.376+0000 7fa43c46d740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d298539410 0x55d298542000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:05.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.376+0000 7fa43c46d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d298539d30 0x55d298532e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:05.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.376+0000 7fa43c46d740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d298542910 0x55d29854aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:05.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.376+0000 7fa43c46d740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fa4341aaf30 con 0x55d298542910 2024-03-22T01:47:05.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.376+0000 7fa43c46d740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa4341ad7b0 con 0x55d298539410 2024-03-22T01:47:05.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.376+0000 7fa43c46d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa4341ab5a0 con 0x55d298539d30 2024-03-22T01:47:05.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.376+0000 7fa4337fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d298542910 0x55d29854aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:05.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.376+0000 7fa432ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d298539410 0x55d298542000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:05.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.376+0000 7fa432ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d298539410 0x55d298542000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.72:51300/0 (socket says 172.21.15.72:51300) 2024-03-22T01:47:05.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.376+0000 7fa432ffd640 1 -- 172.21.15.72:0/3070227784 learned_addr learned my addr 172.21.15.72:0/3070227784 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:47:05.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.376+0000 7fa4327fc640 1 --2- 172.21.15.72:0/3070227784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d298539d30 0x55d298532e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:05.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.376+0000 7fa432ffd640 1 -- 172.21.15.72:0/3070227784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d298539d30 msgr2=0x55d298532e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:47:05.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.376+0000 7fa432ffd640 1 --2- 172.21.15.72:0/3070227784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d298539d30 0x55d298532e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.376+0000 7fa432ffd640 1 -- 172.21.15.72:0/3070227784 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d298542910 msgr2=0x55d29854aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:47:05.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.376+0000 7fa432ffd640 1 --2- 172.21.15.72:0/3070227784 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d298542910 0x55d29854aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.376+0000 7fa432ffd640 1 -- 172.21.15.72:0/3070227784 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d2981710f0 con 0x55d298539410 2024-03-22T01:47:05.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.376+0000 7fa432ffd640 1 --2- 172.21.15.72:0/3070227784 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d298539410 0x55d298542000 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa424009230 tx=0x7fa42403f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1915c8d4e61167f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:47:05.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.376+0000 7fa4197fa640 1 -- 172.21.15.72:0/3070227784 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa42405a070 con 0x55d298539410 2024-03-22T01:47:05.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.376+0000 7fa4197fa640 1 -- 172.21.15.72:0/3070227784 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4240542d0 con 0x55d298539410 2024-03-22T01:47:05.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.376+0000 7fa4197fa640 1 -- 172.21.15.72:0/3070227784 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa424004af0 con 0x55d298539410 2024-03-22T01:47:05.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.377+0000 7fa43c46d740 1 -- 172.21.15.72:0/3070227784 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d298539410 msgr2=0x55d298542000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:05.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.377+0000 7fa43c46d740 1 --2- 172.21.15.72:0/3070227784 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d298539410 0x55d298542000 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa424009230 tx=0x7fa42403f000 comp rx=0 tx=0).stop 2024-03-22T01:47:05.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.377+0000 7fa43c46d740 1 -- 172.21.15.72:0/3070227784 shutdown_connections 2024-03-22T01:47:05.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.377+0000 7fa43c46d740 1 --2- 172.21.15.72:0/3070227784 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d298542910 0x55d29854aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.377+0000 7fa43c46d740 1 --2- 172.21.15.72:0/3070227784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d298539d30 0x55d298532e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.377+0000 7fa43c46d740 1 --2- 172.21.15.72:0/3070227784 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d298539410 0x55d298542000 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.377+0000 7fa43c46d740 1 -- 172.21.15.72:0/3070227784 >> 172.21.15.72:0/3070227784 conn(0x55d298533b40 msgr2=0x55d29853e900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:47:05.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.377+0000 7fa43c46d740 1 -- 172.21.15.72:0/3070227784 shutdown_connections 2024-03-22T01:47:05.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.377+0000 7fa43c46d740 1 -- 172.21.15.72:0/3070227784 wait complete. 2024-03-22T01:47:05.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.377+0000 7fa43c46d740 1 -- start start 2024-03-22T01:47:05.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.378+0000 7fa43c46d740 1 Processor -- start 2024-03-22T01:47:05.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.378+0000 7fa43c46d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d298539410 0x55d29853bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:05.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.378+0000 7fa43c46d740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d298539d30 0x55d29853c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:05.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.378+0000 7fa43c46d740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d298542910 0x55d29853c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:05.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.378+0000 7fa43c46d740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fa4341abc70 con 0x55d298542910 2024-03-22T01:47:05.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.378+0000 7fa43c46d740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa4341ad0e0 con 0x55d298539d30 2024-03-22T01:47:05.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.378+0000 7fa43c46d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa4341ac340 con 0x55d298539410 2024-03-22T01:47:05.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.378+0000 7fa43c46d740 10 client.0 _ll_register_callbacks cb 0x7fa428010eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:47:05.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.378+0000 7fa4337fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d298542910 0x55d29853c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:05.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.378+0000 7fa4337fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d298542910 0x55d29853c5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59280/0 (socket says 172.21.15.72:59280) 2024-03-22T01:47:05.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.378+0000 7fa4337fe640 1 -- 172.21.15.72:0/1448793231 learned_addr learned my addr 172.21.15.72:0/1448793231 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:47:05.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.378+0000 7fa432ffd640 1 --2- 172.21.15.72:0/1448793231 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d298539410 0x55d29853bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:05.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.378+0000 7fa4327fc640 1 --2- 172.21.15.72:0/1448793231 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d298539d30 0x55d29853c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:05.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.378+0000 7fa4337fe640 1 -- 172.21.15.72:0/1448793231 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d298539410 msgr2=0x55d29853bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:47:05.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.378+0000 7fa4337fe640 1 --2- 172.21.15.72:0/1448793231 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d298539410 0x55d29853bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.378+0000 7fa4337fe640 1 -- 172.21.15.72:0/1448793231 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d298539d30 msgr2=0x55d29853c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:47:05.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.378+0000 7fa4337fe640 1 --2- 172.21.15.72:0/1448793231 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d298539d30 0x55d29853c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.378+0000 7fa4337fe640 1 -- 172.21.15.72:0/1448793231 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d29853ccb0 con 0x55d298542910 2024-03-22T01:47:05.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.378+0000 7fa4337fe640 1 --2- 172.21.15.72:0/1448793231 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d298542910 0x55d29853c5c0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fa42c009560 tx=0x7fa42c0051e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=287009fd05810d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:47:05.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.378+0000 7fa412ffd640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:47:05.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.379+0000 7fa412ffd640 1 -- 172.21.15.72:0/1448793231 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa42c03d060 con 0x55d298542910 2024-03-22T01:47:05.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.379+0000 7fa412ffd640 1 -- 172.21.15.72:0/1448793231 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fa42c00b9c0 con 0x55d298542910 2024-03-22T01:47:05.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.379+0000 7fa43c46d740 1 -- 172.21.15.72:0/1448793231 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa4240542d0 con 0x55d298542910 2024-03-22T01:47:05.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.379+0000 7fa412ffd640 1 -- 172.21.15.72:0/1448793231 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7fa42c01c040 con 0x55d298542910 2024-03-22T01:47:05.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.379+0000 7fa43c46d740 10 client.16413 fetch_fsmap learned FSMap version 103 2024-03-22T01:47:05.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.379+0000 7fa43c46d740 1 -- 172.21.15.72:0/1448793231 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=4+}) -- 0x55d2985bd2b0 con 0x55d298542910 2024-03-22T01:47:05.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.379+0000 7fa412ffd640 1 -- 172.21.15.72:0/1448793231 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4240542d0 con 0x55d298542910 2024-03-22T01:47:05.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.379+0000 7fa412ffd640 1 -- 172.21.15.72:0/1448793231 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 103) ==== 8330+0+0 (secure 0 0 0) 0x7fa42c02aa30 con 0x55d298542910 2024-03-22T01:47:05.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.379+0000 7fa43c46d740 10 client.16413 fetch_fsmap finished waiting for FSMap version 103 2024-03-22T01:47:05.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.379+0000 7fa43c46d740 10 client.16413 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T01:47:05.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.379+0000 7fa43c46d740 20 client.16413 populate_metadata read hostname 'smithi072' 2024-03-22T01:47:05.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.379+0000 7fa43c46d740 1 --2- 172.21.15.72:0/1448793231 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55d2985bd840 0x55d2985bfc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:05.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.379+0000 7fa43c46d740 4 client.16413 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T01:47:05.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.379+0000 7fa43c46d740 1 -- 172.21.15.72:0/1448793231 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d2984a5150 con 0x55d2985bd840 2024-03-22T01:47:05.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.379+0000 7fa432ffd640 1 --2- 172.21.15.72:0/1448793231 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55d2985bd840 0x55d2985bfc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:05.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.380+0000 7fa412ffd640 10 client.16413 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:47:05.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.380+0000 7fa432ffd640 1 --2- 172.21.15.72:0/1448793231 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55d2985bd840 0x55d2985bfc30 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a89d19ef21bb44e server_cookie=e25d9b5b559233a2 in_seq=0 out_seq=0 2024-03-22T01:47:05.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.380+0000 7fa412ffd640 1 -- 172.21.15.72:0/1448793231 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55d2984a5150 con 0x55d2985bd840 2024-03-22T01:47:05.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.380+0000 7fa412ffd640 10 client.16413 handle_command_reply: tid=0 2024-03-22T01:47:05.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.380+0000 7fa43c46d740 1 client.16413 shutdown 2024-03-22T01:47:05.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.380+0000 7fa43c46d740 1 -- 172.21.15.72:0/1448793231 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d298542910 msgr2=0x55d29853c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:05.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.380+0000 7fa43c46d740 1 --2- 172.21.15.72:0/1448793231 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d298542910 0x55d29853c5c0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fa42c009560 tx=0x7fa42c0051e0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.380+0000 7fa43c46d740 1 -- 172.21.15.72:0/1448793231 shutdown_connections 2024-03-22T01:47:05.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.380+0000 7fa43c46d740 1 --2- 172.21.15.72:0/1448793231 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x55d2985bd840 0x55d2985bfc30 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.380+0000 7fa43c46d740 1 --2- 172.21.15.72:0/1448793231 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d298539d30 0x55d29853c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.380+0000 7fa43c46d740 1 --2- 172.21.15.72:0/1448793231 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55d298542910 0x55d29853c5c0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.380+0000 7fa412ffd640 0 client.16413 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:47:05.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.380+0000 7fa43c46d740 1 --2- 172.21.15.72:0/1448793231 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d298539410 0x55d29853bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.380+0000 7fa43c46d740 1 -- 172.21.15.72:0/1448793231 >> 172.21.15.72:0/1448793231 conn(0x55d298533b40 msgr2=0x55d29853e900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:47:05.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.380+0000 7fa43c46d740 1 -- 172.21.15.72:0/1448793231 shutdown_connections 2024-03-22T01:47:05.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.380+0000 7fa43c46d740 1 -- 172.21.15.72:0/1448793231 wait complete. 2024-03-22T01:47:05.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.380+0000 7fa43c46d740 20 client.16413 trim_cache size 0 max 16384 2024-03-22T01:47:05.391 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-22T01:47:05.391 INFO:teuthology.orchestra.run.smithi072.stdout: "status": "no active scrubs running", 2024-03-22T01:47:05.391 INFO:teuthology.orchestra.run.smithi072.stdout: "scrubs": {} 2024-03-22T01:47:05.391 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-22T01:47:05.393 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.381+0000 7fa439884640 1 -- 172.21.15.72:0/4169468437 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fa41c0a3090 msgr2=0x7fa41c0a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:05.393 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.381+0000 7fa439884640 1 --2- 172.21.15.72:0/4169468437 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fa41c0a3090 0x7fa41c0a5550 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa424005ab0 tx=0x7fa424005fd0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.393 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.381+0000 7fa439884640 1 -- 172.21.15.72:0/4169468437 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4341013f0 msgr2=0x7fa4341978d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:05.393 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.381+0000 7fa439884640 1 --2- 172.21.15.72:0/4169468437 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4341013f0 0x7fa4341978d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa42800e9f0 tx=0x7fa42800eeb0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.393 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.381+0000 7fa439884640 1 -- 172.21.15.72:0/4169468437 shutdown_connections 2024-03-22T01:47:05.393 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.381+0000 7fa439884640 1 --2- 172.21.15.72:0/4169468437 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fa41c0a3090 0x7fa41c0a5550 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.393 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.381+0000 7fa439884640 1 --2- 172.21.15.72:0/4169468437 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4341013f0 0x7fa4341978d0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.381+0000 7fa439884640 1 --2- 172.21.15.72:0/4169468437 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa434106a40 0x7fa43419bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.381+0000 7fa439884640 1 --2- 172.21.15.72:0/4169468437 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa434100a30 0x7fa434197390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.381+0000 7fa439884640 1 -- 172.21.15.72:0/4169468437 >> 172.21.15.72:0/4169468437 conn(0x7fa434077c40 msgr2=0x7fa434103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:47:05.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.381+0000 7fa439884640 1 -- 172.21.15.72:0/4169468437 shutdown_connections 2024-03-22T01:47:05.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.381+0000 7fa439884640 1 -- 172.21.15.72:0/4169468437 wait complete. 2024-03-22T01:47:05.394 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-22T01:47:05.394 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T01:47:05.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.497+0000 7f0b32e6f640 1 -- 172.21.15.72:0/1187168616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b2c1020b0 msgr2=0x7f0b2c1056c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:05.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.497+0000 7f0b32e6f640 1 --2- 172.21.15.72:0/1187168616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b2c1020b0 0x7f0b2c1056c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0b18009900 tx=0x7f0b1802bc30 comp rx=0 tx=0).stop 2024-03-22T01:47:05.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.497+0000 7f0b32e6f640 1 -- 172.21.15.72:0/1187168616 shutdown_connections 2024-03-22T01:47:05.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.497+0000 7f0b32e6f640 1 --2- 172.21.15.72:0/1187168616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b2c1020b0 0x7f0b2c1056c0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.497+0000 7f0b32e6f640 1 --2- 172.21.15.72:0/1187168616 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b2c105da0 0x7f0b2c10d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.497+0000 7f0b32e6f640 1 --2- 172.21.15.72:0/1187168616 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0b2c101700 0x7f0b2c101ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.497+0000 7f0b32e6f640 1 -- 172.21.15.72:0/1187168616 >> 172.21.15.72:0/1187168616 conn(0x7f0b2c076ee0 msgr2=0x7f0b2c0772f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:47:05.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.497+0000 7f0b32e6f640 1 -- 172.21.15.72:0/1187168616 shutdown_connections 2024-03-22T01:47:05.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.497+0000 7f0b32e6f640 1 -- 172.21.15.72:0/1187168616 wait complete. 2024-03-22T01:47:05.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.497+0000 7f0b32e6f640 1 Processor -- start 2024-03-22T01:47:05.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.497+0000 7f0b32e6f640 1 -- start start 2024-03-22T01:47:05.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.498+0000 7f0b32e6f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0b2c101700 0x7f0b2c1992b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:05.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.498+0000 7f0b32e6f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b2c105da0 0x7f0b2c1997f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:05.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.498+0000 7f0b32e6f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b2c199d80 0x7f0b2c1a4330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:05.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.498+0000 7f0b32e6f640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f0b2c110e10 con 0x7f0b2c101700 2024-03-22T01:47:05.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.498+0000 7f0b32e6f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f0b2c110c90 con 0x7f0b2c105da0 2024-03-22T01:47:05.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.498+0000 7f0b32e6f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0b2c110f90 con 0x7f0b2c199d80 2024-03-22T01:47:05.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.498+0000 7f0b30c0b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0b2c101700 0x7f0b2c1992b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:05.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.498+0000 7f0b30c0b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0b2c101700 0x7f0b2c1992b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59294/0 (socket says 172.21.15.72:59294) 2024-03-22T01:47:05.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.498+0000 7f0b30c0b640 1 -- 172.21.15.72:0/30144594 learned_addr learned my addr 172.21.15.72:0/30144594 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:47:05.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.498+0000 7f0b2bfff640 1 --2- 172.21.15.72:0/30144594 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b2c105da0 0x7f0b2c1997f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:05.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.498+0000 7f0b3140c640 1 --2- 172.21.15.72:0/30144594 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b2c199d80 0x7f0b2c1a4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:05.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.498+0000 7f0b30c0b640 1 -- 172.21.15.72:0/30144594 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b2c199d80 msgr2=0x7f0b2c1a4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:05.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.498+0000 7f0b30c0b640 1 --2- 172.21.15.72:0/30144594 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b2c199d80 0x7f0b2c1a4330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.498+0000 7f0b30c0b640 1 -- 172.21.15.72:0/30144594 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b2c105da0 msgr2=0x7f0b2c1997f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:05.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.498+0000 7f0b30c0b640 1 --2- 172.21.15.72:0/30144594 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b2c105da0 0x7f0b2c1997f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.498+0000 7f0b30c0b640 1 -- 172.21.15.72:0/30144594 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b2c1a4b00 con 0x7f0b2c101700 2024-03-22T01:47:05.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.498+0000 7f0b3140c640 1 --2- 172.21.15.72:0/30144594 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b2c199d80 0x7f0b2c1a4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T01:47:05.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.498+0000 7f0b2bfff640 1 --2- 172.21.15.72:0/30144594 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b2c105da0 0x7f0b2c1997f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:47:05.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.498+0000 7f0b30c0b640 1 --2- 172.21.15.72:0/30144594 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0b2c101700 0x7f0b2c1992b0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f0b2000e9f0 tx=0x7f0b2000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:47:05.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.498+0000 7f0b29ffb640 1 -- 172.21.15.72:0/30144594 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b2000cd80 con 0x7f0b2c101700 2024-03-22T01:47:05.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.498+0000 7f0b32e6f640 1 -- 172.21.15.72:0/30144594 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b2c1a4df0 con 0x7f0b2c101700 2024-03-22T01:47:05.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.498+0000 7f0b29ffb640 1 -- 172.21.15.72:0/30144594 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b20010910 con 0x7f0b2c101700 2024-03-22T01:47:05.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.498+0000 7f0b29ffb640 1 -- 172.21.15.72:0/30144594 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b200188f0 con 0x7f0b2c101700 2024-03-22T01:47:05.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.499+0000 7f0b32e6f640 1 -- 172.21.15.72:0/30144594 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b2c1a5260 con 0x7f0b2c101700 2024-03-22T01:47:05.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.500+0000 7f0b32e6f640 1 -- 172.21.15.72:0/30144594 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b2c068e60 con 0x7f0b2c101700 2024-03-22T01:47:05.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.501+0000 7f0b29ffb640 1 -- 172.21.15.72:0/30144594 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 25) ==== 137985+0+0 (secure 0 0 0) 0x7f0b20010ac0 con 0x7f0b2c101700 2024-03-22T01:47:05.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.501+0000 7f0b29ffb640 1 --2- 172.21.15.72:0/30144594 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f0b1c0a3090 0x7f0b1c0a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:05.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.501+0000 7f0b2bfff640 1 --2- 172.21.15.72:0/30144594 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f0b1c0a3090 0x7f0b1c0a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:05.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.501+0000 7f0b29ffb640 1 -- 172.21.15.72:0/30144594 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f0b200cffd0 con 0x7f0b2c101700 2024-03-22T01:47:05.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.502+0000 7f0b2bfff640 1 --2- 172.21.15.72:0/30144594 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f0b1c0a3090 0x7f0b1c0a5550 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0b18000c00 tx=0x7f0b18005fd0 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:47:05.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.506+0000 7f0b29ffb640 1 -- 172.21.15.72:0/30144594 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0b2009c710 con 0x7f0b2c101700 2024-03-22T01:47:05.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.722+0000 7f0b32e6f640 1 -- 172.21.15.72:0/30144594 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0b2c1020b0 con 0x7f0b2c101700 2024-03-22T01:47:05.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.722+0000 7f0b29ffb640 1 -- 172.21.15.72:0/30144594 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) ==== 95+0+11791 (secure 0 0 0) 0x7f0b2009c010 con 0x7f0b2c101700 2024-03-22T01:47:05.722 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:47:05.722 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":103,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6835/3077216308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3077216308},{"type":"v1","addr":"172.21.15.72:6835","nonce":3077216308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2460254143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":2460254143},{"type":"v1","addr":"172.21.15.72:6837","nonce":2460254143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6839/2026432819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6838","nonce":2026432819},{"type":"v1","addr":"172.21.15.72:6839","nonce":2026432819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6841/400094545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6840","nonce":400094545},{"type":"v1","addr":"172.21.15.72:6841","nonce":400094545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6833/1578701171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":1578701171},{"type":"v1","addr":"172.21.15.89:6833","nonce":1578701171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/2703047213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2703047213},{"type":"v1","addr":"172.21.15.98:6833","nonce":2703047213}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2589468787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2589468787},{"type":"v1","addr":"172.21.15.89:6835","nonce":2589468787}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/92657636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":92657636},{"type":"v1","addr":"172.21.15.98:6835","nonce":92657636}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1662941782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1662941782},{"type":"v1","addr":"172.21.15.89:6837","nonce":1662941782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24490,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/781860371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":781860371},{"type":"v1","addr":"172.21.15.98:6837","nonce":781860371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":103,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-22T01:34:14.815677+0000","modified":"2024-03-22T01:45:27.515143+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24524},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":171,"addr":"172.21.15.98:6839/3192173650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3192173650},{"type":"v1","addr":"172.21.15.98:6839","nonce":3192173650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":171,"addr":"172.21.15.89:6839/149200280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":149200280},{"type":"v1","addr":"172.21.15.89:6839","nonce":149200280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24524]},"id":1}]} 2024-03-22T01:47:05.722 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 103 2024-03-22T01:47:05.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.724+0000 7f0b32e6f640 1 -- 172.21.15.72:0/30144594 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f0b1c0a3090 msgr2=0x7f0b1c0a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:05.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.724+0000 7f0b32e6f640 1 --2- 172.21.15.72:0/30144594 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f0b1c0a3090 0x7f0b1c0a5550 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0b18000c00 tx=0x7f0b18005fd0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.724+0000 7f0b32e6f640 1 -- 172.21.15.72:0/30144594 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0b2c101700 msgr2=0x7f0b2c1992b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:05.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.724+0000 7f0b32e6f640 1 --2- 172.21.15.72:0/30144594 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0b2c101700 0x7f0b2c1992b0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f0b2000e9f0 tx=0x7f0b2000eeb0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.724+0000 7f0b32e6f640 1 -- 172.21.15.72:0/30144594 shutdown_connections 2024-03-22T01:47:05.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.724+0000 7f0b32e6f640 1 --2- 172.21.15.72:0/30144594 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f0b1c0a3090 0x7f0b1c0a5550 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.724+0000 7f0b32e6f640 1 --2- 172.21.15.72:0/30144594 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b2c105da0 0x7f0b2c1997f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.724+0000 7f0b32e6f640 1 --2- 172.21.15.72:0/30144594 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b2c199d80 0x7f0b2c1a4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.724+0000 7f0b32e6f640 1 --2- 172.21.15.72:0/30144594 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0b2c101700 0x7f0b2c1992b0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.724+0000 7f0b32e6f640 1 -- 172.21.15.72:0/30144594 >> 172.21.15.72:0/30144594 conn(0x7f0b2c076ee0 msgr2=0x7f0b2c0fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:47:05.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.725+0000 7f0b32e6f640 1 -- 172.21.15.72:0/30144594 shutdown_connections 2024-03-22T01:47:05.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.725+0000 7f0b32e6f640 1 -- 172.21.15.72:0/30144594 wait complete. 2024-03-22T01:47:05.735 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-22T01:47:05.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.839+0000 7f196cf25640 1 -- 172.21.15.72:0/442133272 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1968101400 msgr2=0x7f19681017e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:05.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.839+0000 7f196cf25640 1 --2- 172.21.15.72:0/442133272 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1968101400 0x7f19681017e0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f1958009900 tx=0x7f195802bc30 comp rx=0 tx=0).stop 2024-03-22T01:47:05.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.839+0000 7f196cf25640 1 -- 172.21.15.72:0/442133272 shutdown_connections 2024-03-22T01:47:05.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.839+0000 7f196cf25640 1 --2- 172.21.15.72:0/442133272 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1968101db0 0x7f19681056c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.839+0000 7f196cf25640 1 --2- 172.21.15.72:0/442133272 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1968105df0 0x7f196810d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.839+0000 7f196cf25640 1 --2- 172.21.15.72:0/442133272 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1968101400 0x7f19681017e0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.839+0000 7f196cf25640 1 -- 172.21.15.72:0/442133272 >> 172.21.15.72:0/442133272 conn(0x7f1968076f90 msgr2=0x7f19680fe310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:47:05.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.839+0000 7f196cf25640 1 -- 172.21.15.72:0/442133272 shutdown_connections 2024-03-22T01:47:05.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.839+0000 7f196cf25640 1 -- 172.21.15.72:0/442133272 wait complete. 2024-03-22T01:47:05.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.840+0000 7f196cf25640 1 Processor -- start 2024-03-22T01:47:05.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.840+0000 7f196cf25640 1 -- start start 2024-03-22T01:47:05.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.840+0000 7f196cf25640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1968101400 0x7f19681973b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:05.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.840+0000 7f196cf25640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1968105df0 0x7f19681978f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:05.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.840+0000 7f196cf25640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f196819bc70 0x7f196819c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:05.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.840+0000 7f196cf25640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f1968110e50 con 0x7f196819bc70 2024-03-22T01:47:05.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.840+0000 7f196cf25640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1968110cd0 con 0x7f1968105df0 2024-03-22T01:47:05.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.840+0000 7f196cf25640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1968110fd0 con 0x7f1968101400 2024-03-22T01:47:05.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.840+0000 7f1966d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f196819bc70 0x7f196819c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:05.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.840+0000 7f1966d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f196819bc70 0x7f196819c120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59310/0 (socket says 172.21.15.72:59310) 2024-03-22T01:47:05.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.840+0000 7f1966d9d640 1 -- 172.21.15.72:0/4003402914 learned_addr learned my addr 172.21.15.72:0/4003402914 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:47:05.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.840+0000 7f195ffff640 1 --2- 172.21.15.72:0/4003402914 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1968105df0 0x7f19681978f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:05.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.840+0000 7f196659c640 1 --2- 172.21.15.72:0/4003402914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1968101400 0x7f19681973b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:05.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.840+0000 7f1966d9d640 1 -- 172.21.15.72:0/4003402914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1968101400 msgr2=0x7f19681973b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:05.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.840+0000 7f1966d9d640 1 --2- 172.21.15.72:0/4003402914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1968101400 0x7f19681973b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.840+0000 7f1966d9d640 1 -- 172.21.15.72:0/4003402914 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1968105df0 msgr2=0x7f19681978f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:05.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.840+0000 7f1966d9d640 1 --2- 172.21.15.72:0/4003402914 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1968105df0 0x7f19681978f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.840+0000 7f1966d9d640 1 -- 172.21.15.72:0/4003402914 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f196819c760 con 0x7f196819bc70 2024-03-22T01:47:05.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.840+0000 7f196659c640 1 --2- 172.21.15.72:0/4003402914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1968101400 0x7f19681973b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T01:47:05.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.840+0000 7f1966d9d640 1 --2- 172.21.15.72:0/4003402914 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f196819bc70 0x7f196819c120 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f196000eb10 tx=0x7f196000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:47:05.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.841+0000 7f195f7fe640 1 -- 172.21.15.72:0/4003402914 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1960004b20 con 0x7f196819bc70 2024-03-22T01:47:05.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.841+0000 7f196cf25640 1 -- 172.21.15.72:0/4003402914 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f196819ca50 con 0x7f196819bc70 2024-03-22T01:47:05.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.841+0000 7f195f7fe640 1 -- 172.21.15.72:0/4003402914 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1960004cc0 con 0x7f196819bc70 2024-03-22T01:47:05.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.841+0000 7f195f7fe640 1 -- 172.21.15.72:0/4003402914 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1960010760 con 0x7f196819bc70 2024-03-22T01:47:05.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.841+0000 7f196cf25640 1 -- 172.21.15.72:0/4003402914 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19681a4240 con 0x7f196819bc70 2024-03-22T01:47:05.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.843+0000 7f195f7fe640 1 -- 172.21.15.72:0/4003402914 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 25) ==== 137985+0+0 (secure 0 0 0) 0x7f1960005020 con 0x7f196819bc70 2024-03-22T01:47:05.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.843+0000 7f196fbef740 1 Processor -- start 2024-03-22T01:47:05.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.843+0000 7f196fbef740 1 -- start start 2024-03-22T01:47:05.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.844+0000 7f196fbef740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564bebeaa800 0x564bebeaabe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:05.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.844+0000 7f196fbef740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bebeab120 0x564bebeb37c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:05.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.844+0000 7f196fbef740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564bebeb3d00 0x564bebeb60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:05.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.844+0000 7f196fbef740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x564bebe0e8d0 con 0x564bebeaa800 2024-03-22T01:47:05.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.844+0000 7f196fbef740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x564bebdb6e00 con 0x564bebeb3d00 2024-03-22T01:47:05.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.844+0000 7f196fbef740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x564bebe0a880 con 0x564bebeab120 2024-03-22T01:47:05.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.844+0000 7f195f7fe640 1 --2- 172.21.15.72:0/4003402914 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f19540a3090 0x7f19540a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:05.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.844+0000 7f195ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564bebeaa800 0x564bebeaabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:05.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.844+0000 7f195ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564bebeaa800 0x564bebeaabe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59322/0 (socket says 172.21.15.72:59322) 2024-03-22T01:47:05.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.844+0000 7f196659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bebeab120 0x564bebeb37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:05.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.844+0000 7f195ffff640 1 -- 172.21.15.72:0/3112052458 learned_addr learned my addr 172.21.15.72:0/3112052458 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:47:05.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.844+0000 7f195ffff640 1 --2- 172.21.15.72:0/3112052458 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564bebeb3d00 0x564bebeb60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:05.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.844+0000 7f1966d9d640 1 --2- 172.21.15.72:0/4003402914 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f19540a3090 0x7f19540a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:05.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.844+0000 7f195f7fe640 1 -- 172.21.15.72:0/4003402914 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f19600d02a0 con 0x7f196819bc70 2024-03-22T01:47:05.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.844+0000 7f195ffff640 1 -- 172.21.15.72:0/3112052458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bebeab120 msgr2=0x564bebeb37c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:47:05.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.844+0000 7f195ffff640 1 --2- 172.21.15.72:0/3112052458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bebeab120 0x564bebeb37c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.844+0000 7f195ffff640 1 -- 172.21.15.72:0/3112052458 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564bebeb3d00 msgr2=0x564bebeb60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:47:05.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.844+0000 7f195ffff640 1 --2- 172.21.15.72:0/3112052458 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564bebeb3d00 0x564bebeb60f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.844+0000 7f195ffff640 1 -- 172.21.15.72:0/3112052458 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564bebb223f0 con 0x564bebeaa800 2024-03-22T01:47:05.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.844+0000 7f195ffff640 1 --2- 172.21.15.72:0/3112052458 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564bebeaa800 0x564bebeaabe0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f1950013790 tx=0x7f1950005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33cf2e9b7ebdc3e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:47:05.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.844+0000 7f1966d9d640 1 --2- 172.21.15.72:0/4003402914 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f19540a3090 0x7f19540a5550 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f196009c270 tx=0x7f196000efd0 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:47:05.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.845+0000 7f195cff9640 1 -- 172.21.15.72:0/3112052458 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1950013d10 con 0x564bebeaa800 2024-03-22T01:47:05.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.845+0000 7f195cff9640 1 -- 172.21.15.72:0/3112052458 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1950005020 con 0x564bebeaa800 2024-03-22T01:47:05.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.845+0000 7f195cff9640 1 -- 172.21.15.72:0/3112052458 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1950004a00 con 0x564bebeaa800 2024-03-22T01:47:05.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.845+0000 7f196fbef740 1 -- 172.21.15.72:0/3112052458 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564bebeaa800 msgr2=0x564bebeaabe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:05.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.845+0000 7f196fbef740 1 --2- 172.21.15.72:0/3112052458 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564bebeaa800 0x564bebeaabe0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f1950013790 tx=0x7f1950005850 comp rx=0 tx=0).stop 2024-03-22T01:47:05.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.845+0000 7f196fbef740 1 -- 172.21.15.72:0/3112052458 shutdown_connections 2024-03-22T01:47:05.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.845+0000 7f196fbef740 1 --2- 172.21.15.72:0/3112052458 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564bebeb3d00 0x564bebeb60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.845+0000 7f196fbef740 1 --2- 172.21.15.72:0/3112052458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bebeab120 0x564bebeb37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.845+0000 7f196fbef740 1 --2- 172.21.15.72:0/3112052458 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564bebeaa800 0x564bebeaabe0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.845+0000 7f196fbef740 1 -- 172.21.15.72:0/3112052458 >> 172.21.15.72:0/3112052458 conn(0x564bebea4f30 msgr2=0x564bebea5310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:47:05.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.845+0000 7f196fbef740 1 -- 172.21.15.72:0/3112052458 shutdown_connections 2024-03-22T01:47:05.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.845+0000 7f196fbef740 1 -- 172.21.15.72:0/3112052458 wait complete. 2024-03-22T01:47:05.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.846+0000 7f196fbef740 1 -- start start 2024-03-22T01:47:05.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.846+0000 7f196fbef740 1 Processor -- start 2024-03-22T01:47:05.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.846+0000 7f196fbef740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564bebeaa800 0x564bebeb0d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:05.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.846+0000 7f196fbef740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bebeab120 0x564bebeb1280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:05.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.846+0000 7f196fbef740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564bebeb3d00 0x564bebeb17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:05.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.846+0000 7f196fbef740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x564bebeb6a90 con 0x564bebeaa800 2024-03-22T01:47:05.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.846+0000 7f196fbef740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x564bebdca300 con 0x564bebeb3d00 2024-03-22T01:47:05.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.846+0000 7f196fbef740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x564bebeb6c10 con 0x564bebeab120 2024-03-22T01:47:05.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.846+0000 7f196fbef740 10 client.0 _ll_register_callbacks cb 0x564bebe05e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:47:05.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.846+0000 7f195ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564bebeaa800 0x564bebeb0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:05.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.846+0000 7f195ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564bebeaa800 0x564bebeb0d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59332/0 (socket says 172.21.15.72:59332) 2024-03-22T01:47:05.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.846+0000 7f195ffff640 1 -- 172.21.15.72:0/4183554019 learned_addr learned my addr 172.21.15.72:0/4183554019 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:47:05.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.846+0000 7f196659c640 1 --2- 172.21.15.72:0/4183554019 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bebeab120 0x564bebeb1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:05.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.846+0000 7f195ffff640 1 --2- 172.21.15.72:0/4183554019 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564bebeb3d00 0x564bebeb17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:05.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.846+0000 7f195ffff640 1 -- 172.21.15.72:0/4183554019 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bebeab120 msgr2=0x564bebeb1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:47:05.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.846+0000 7f195ffff640 1 --2- 172.21.15.72:0/4183554019 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bebeab120 0x564bebeb1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.846+0000 7f195ffff640 1 -- 172.21.15.72:0/4183554019 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564bebeb3d00 msgr2=0x564bebeb17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:47:05.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.846+0000 7f195ffff640 1 --2- 172.21.15.72:0/4183554019 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564bebeb3d00 0x564bebeb17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.846+0000 7f195ffff640 1 -- 172.21.15.72:0/4183554019 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564bebbec7d0 con 0x564bebeaa800 2024-03-22T01:47:05.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.846+0000 7f196659c640 1 --2- 172.21.15.72:0/4183554019 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bebeab120 0x564bebeb1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:47:05.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.847+0000 7f195ffff640 1 --2- 172.21.15.72:0/4183554019 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564bebeaa800 0x564bebeb0d40 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f19500056d0 tx=0x7f1950005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e3a7796a81a3de7 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:47:05.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.847+0000 7f194dffb640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:47:05.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.847+0000 7f194dffb640 1 -- 172.21.15.72:0/4183554019 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19500045f0 con 0x564bebeaa800 2024-03-22T01:47:05.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.847+0000 7f194dffb640 1 -- 172.21.15.72:0/4183554019 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f1950004790 con 0x564bebeaa800 2024-03-22T01:47:05.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.847+0000 7f196fbef740 1 -- 172.21.15.72:0/4183554019 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1950005020 con 0x564bebeaa800 2024-03-22T01:47:05.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.847+0000 7f194dffb640 1 -- 172.21.15.72:0/4183554019 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7f1950036050 con 0x564bebeaa800 2024-03-22T01:47:05.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.847+0000 7f196fbef740 10 client.16431 fetch_fsmap learned FSMap version 103 2024-03-22T01:47:05.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.847+0000 7f196fbef740 1 -- 172.21.15.72:0/4183554019 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=4+}) -- 0x564bebec2d50 con 0x564bebeaa800 2024-03-22T01:47:05.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.847+0000 7f194dffb640 1 -- 172.21.15.72:0/4183554019 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1950005020 con 0x564bebeaa800 2024-03-22T01:47:05.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.847+0000 7f194dffb640 1 -- 172.21.15.72:0/4183554019 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 103) ==== 8330+0+0 (secure 0 0 0) 0x7f19500209e0 con 0x564bebeaa800 2024-03-22T01:47:05.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.847+0000 7f196fbef740 10 client.16431 fetch_fsmap finished waiting for FSMap version 103 2024-03-22T01:47:05.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.847+0000 7f196fbef740 10 client.16431 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T01:47:05.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.847+0000 7f196fbef740 20 client.16431 populate_metadata read hostname 'smithi072' 2024-03-22T01:47:05.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.847+0000 7f196fbef740 1 --2- 172.21.15.72:0/4183554019 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x564bebec4100 0x564bebec64f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:05.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.847+0000 7f196fbef740 4 client.16431 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T01:47:05.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.847+0000 7f196fbef740 1 -- 172.21.15.72:0/4183554019 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564bebe1be10 con 0x564bebec4100 2024-03-22T01:47:05.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.848+0000 7f196659c640 1 --2- 172.21.15.72:0/4183554019 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x564bebec4100 0x564bebec64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:05.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.848+0000 7f194dffb640 10 client.16431 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:47:05.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.848+0000 7f196659c640 1 --2- 172.21.15.72:0/4183554019 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x564bebec4100 0x564bebec64f0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a27ad18350b94cb8 server_cookie=72c412d48251ece4 in_seq=0 out_seq=0 2024-03-22T01:47:05.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.848+0000 7f194dffb640 1 -- 172.21.15.72:0/4183554019 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564bebe1be10 con 0x564bebec4100 2024-03-22T01:47:05.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.848+0000 7f194dffb640 10 client.16431 handle_command_reply: tid=0 2024-03-22T01:47:05.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.848+0000 7f196fbef740 1 client.16431 shutdown 2024-03-22T01:47:05.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.849+0000 7f196fbef740 1 -- 172.21.15.72:0/4183554019 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564bebeaa800 msgr2=0x564bebeb0d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:05.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.849+0000 7f196fbef740 1 --2- 172.21.15.72:0/4183554019 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564bebeaa800 0x564bebeb0d40 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f19500056d0 tx=0x7f1950005f90 comp rx=0 tx=0).stop 2024-03-22T01:47:05.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.849+0000 7f196fbef740 1 -- 172.21.15.72:0/4183554019 shutdown_connections 2024-03-22T01:47:05.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.849+0000 7f196fbef740 1 --2- 172.21.15.72:0/4183554019 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x564bebec4100 0x564bebec64f0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.849+0000 7f196fbef740 1 --2- 172.21.15.72:0/4183554019 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564bebeb3d00 0x564bebeb17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.849+0000 7f196fbef740 1 --2- 172.21.15.72:0/4183554019 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bebeab120 0x564bebeb1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.849+0000 7f196fbef740 1 --2- 172.21.15.72:0/4183554019 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564bebeaa800 0x564bebeb0d40 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.849+0000 7f194dffb640 0 client.16431 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:47:05.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.849+0000 7f196fbef740 1 -- 172.21.15.72:0/4183554019 >> 172.21.15.72:0/4183554019 conn(0x564bebea4f30 msgr2=0x564bebea6c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:47:05.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.849+0000 7f196fbef740 1 -- 172.21.15.72:0/4183554019 shutdown_connections 2024-03-22T01:47:05.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.849+0000 7f196fbef740 1 -- 172.21.15.72:0/4183554019 wait complete. 2024-03-22T01:47:05.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.849+0000 7f196fbef740 20 client.16431 trim_cache size 0 max 16384 2024-03-22T01:47:05.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.862+0000 7f196fbef740 1 Processor -- start 2024-03-22T01:47:05.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.862+0000 7f196fbef740 1 -- start start 2024-03-22T01:47:05.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.862+0000 7f196fbef740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564bebeaa800 0x564bebeaf4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:05.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.862+0000 7f196fbef740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bebeab120 0x564bebebf830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:05.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.862+0000 7f196fbef740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564bebeb3d00 0x564bebeb15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:05.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.862+0000 7f196fbef740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f19681aaec0 con 0x564bebeaa800 2024-03-22T01:47:05.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.862+0000 7f196fbef740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f19681ad740 con 0x564bebeb3d00 2024-03-22T01:47:05.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.862+0000 7f196fbef740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f19681ab530 con 0x564bebeab120 2024-03-22T01:47:05.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.862+0000 7f195ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564bebeaa800 0x564bebeaf4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:05.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.862+0000 7f195ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564bebeaa800 0x564bebeaf4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59342/0 (socket says 172.21.15.72:59342) 2024-03-22T01:47:05.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.862+0000 7f195ffff640 1 -- 172.21.15.72:0/1595434124 learned_addr learned my addr 172.21.15.72:0/1595434124 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:47:05.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.862+0000 7f195ffff640 1 --2- 172.21.15.72:0/1595434124 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564bebeb3d00 0x564bebeb15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:05.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.862+0000 7f196659c640 1 --2- 172.21.15.72:0/1595434124 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bebeab120 0x564bebebf830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:05.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.862+0000 7f195ffff640 1 -- 172.21.15.72:0/1595434124 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bebeab120 msgr2=0x564bebebf830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:47:05.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.862+0000 7f195ffff640 1 --2- 172.21.15.72:0/1595434124 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bebeab120 0x564bebebf830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.862+0000 7f195ffff640 1 -- 172.21.15.72:0/1595434124 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564bebeb3d00 msgr2=0x564bebeb15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:47:05.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.862+0000 7f195ffff640 1 --2- 172.21.15.72:0/1595434124 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564bebeb3d00 0x564bebeb15e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.862+0000 7f195ffff640 1 -- 172.21.15.72:0/1595434124 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564bebec6860 con 0x564bebeaa800 2024-03-22T01:47:05.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.862+0000 7f195ffff640 1 --2- 172.21.15.72:0/1595434124 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564bebeaa800 0x564bebeaf4e0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f19500096d0 tx=0x7f1950009640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85bd8f79a938fa59 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:47:05.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.862+0000 7f194ffff640 1 -- 172.21.15.72:0/1595434124 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19500119f0 con 0x564bebeaa800 2024-03-22T01:47:05.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.863+0000 7f194ffff640 1 -- 172.21.15.72:0/1595434124 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1950017040 con 0x564bebeaa800 2024-03-22T01:47:05.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.863+0000 7f194ffff640 1 -- 172.21.15.72:0/1595434124 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f195001fc00 con 0x564bebeaa800 2024-03-22T01:47:05.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.863+0000 7f196fbef740 1 -- 172.21.15.72:0/1595434124 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564bebeaa800 msgr2=0x564bebeaf4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:05.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.863+0000 7f196fbef740 1 --2- 172.21.15.72:0/1595434124 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564bebeaa800 0x564bebeaf4e0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f19500096d0 tx=0x7f1950009640 comp rx=0 tx=0).stop 2024-03-22T01:47:05.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.863+0000 7f196fbef740 1 -- 172.21.15.72:0/1595434124 shutdown_connections 2024-03-22T01:47:05.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.863+0000 7f196fbef740 1 --2- 172.21.15.72:0/1595434124 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564bebeb3d00 0x564bebeb15e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.863+0000 7f196fbef740 1 --2- 172.21.15.72:0/1595434124 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bebeab120 0x564bebebf830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.863+0000 7f196fbef740 1 --2- 172.21.15.72:0/1595434124 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564bebeaa800 0x564bebeaf4e0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.863+0000 7f196fbef740 1 -- 172.21.15.72:0/1595434124 >> 172.21.15.72:0/1595434124 conn(0x564bebea4f30 msgr2=0x564bebea3fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:47:05.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.863+0000 7f196fbef740 1 -- 172.21.15.72:0/1595434124 shutdown_connections 2024-03-22T01:47:05.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.863+0000 7f196fbef740 1 -- 172.21.15.72:0/1595434124 wait complete. 2024-03-22T01:47:05.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.863+0000 7f196fbef740 1 -- start start 2024-03-22T01:47:05.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.863+0000 7f196fbef740 1 Processor -- start 2024-03-22T01:47:05.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.864+0000 7f196fbef740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564bebeaa800 0x564bebebc1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:05.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.864+0000 7f196fbef740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bebeab120 0x564bebeb74c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:05.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.864+0000 7f196fbef740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564bebeb3d00 0x564bebeb7a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:05.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.864+0000 7f196fbef740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f19681abc00 con 0x564bebeaa800 2024-03-22T01:47:05.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.864+0000 7f196fbef740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f19681ad070 con 0x564bebeb3d00 2024-03-22T01:47:05.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.864+0000 7f196fbef740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f19681ac2d0 con 0x564bebeab120 2024-03-22T01:47:05.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.864+0000 7f196fbef740 10 client.0 _ll_register_callbacks cb 0x7f1950009e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T01:47:05.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.864+0000 7f195ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564bebeaa800 0x564bebebc1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:05.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.864+0000 7f195ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564bebeaa800 0x564bebebc1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59352/0 (socket says 172.21.15.72:59352) 2024-03-22T01:47:05.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.864+0000 7f195ffff640 1 -- 172.21.15.72:0/1610435308 learned_addr learned my addr 172.21.15.72:0/1610435308 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:47:05.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.864+0000 7f196659c640 1 --2- 172.21.15.72:0/1610435308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bebeab120 0x564bebeb74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:05.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.864+0000 7f195ffff640 1 --2- 172.21.15.72:0/1610435308 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564bebeb3d00 0x564bebeb7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:05.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.864+0000 7f195ffff640 1 -- 172.21.15.72:0/1610435308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bebeab120 msgr2=0x564bebeb74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:47:05.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.864+0000 7f195ffff640 1 --2- 172.21.15.72:0/1610435308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bebeab120 0x564bebeb74c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.864+0000 7f195ffff640 1 -- 172.21.15.72:0/1610435308 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564bebeb3d00 msgr2=0x564bebeb7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T01:47:05.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.864+0000 7f195ffff640 1 --2- 172.21.15.72:0/1610435308 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564bebeb3d00 0x564bebeb7a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.864+0000 7f195ffff640 1 -- 172.21.15.72:0/1610435308 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564bebeaaf50 con 0x564bebeaa800 2024-03-22T01:47:05.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.864+0000 7f195ffff640 1 --2- 172.21.15.72:0/1610435308 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564bebeaa800 0x564bebebc1a0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f19500058c0 tx=0x7f1950011f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e7f2c81c48b7fa4 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:47:05.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.864+0000 7f194dffb640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-22T01:47:05.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.864+0000 7f194dffb640 1 -- 172.21.15.72:0/1610435308 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f195003f040 con 0x564bebeaa800 2024-03-22T01:47:05.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.864+0000 7f194dffb640 1 -- 172.21.15.72:0/1610435308 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f195001f720 con 0x564bebeaa800 2024-03-22T01:47:05.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.864+0000 7f196fbef740 1 -- 172.21.15.72:0/1610435308 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1950017040 con 0x564bebeaa800 2024-03-22T01:47:05.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.865+0000 7f194dffb640 1 -- 172.21.15.72:0/1610435308 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7f1950005020 con 0x564bebeaa800 2024-03-22T01:47:05.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.865+0000 7f196fbef740 10 client.16437 fetch_fsmap learned FSMap version 103 2024-03-22T01:47:05.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.865+0000 7f196fbef740 1 -- 172.21.15.72:0/1610435308 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=4+}) -- 0x564bebf2a190 con 0x564bebeaa800 2024-03-22T01:47:05.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.865+0000 7f194dffb640 1 -- 172.21.15.72:0/1610435308 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1950017040 con 0x564bebeaa800 2024-03-22T01:47:05.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.865+0000 7f194dffb640 1 -- 172.21.15.72:0/1610435308 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 103) ==== 8330+0+0 (secure 0 0 0) 0x7f1950045070 con 0x564bebeaa800 2024-03-22T01:47:05.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.865+0000 7f196fbef740 10 client.16437 fetch_fsmap finished waiting for FSMap version 103 2024-03-22T01:47:05.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.865+0000 7f196fbef740 10 client.16437 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T01:47:05.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.865+0000 7f196fbef740 20 client.16437 populate_metadata read hostname 'smithi072' 2024-03-22T01:47:05.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.865+0000 7f196fbef740 1 --2- 172.21.15.72:0/1610435308 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x564bebf2a770 0x564bebf2cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:05.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.865+0000 7f196fbef740 4 client.16437 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T01:47:05.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.865+0000 7f196fbef740 1 -- 172.21.15.72:0/1610435308 --> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564bebe16160 con 0x564bebf2a770 2024-03-22T01:47:05.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.865+0000 7f196659c640 1 --2- 172.21.15.72:0/1610435308 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x564bebf2a770 0x564bebf2cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:05.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.865+0000 7f194dffb640 10 client.16437 ms_handle_connect on v2:172.21.15.89:6838/149200280 2024-03-22T01:47:05.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.865+0000 7f196659c640 1 --2- 172.21.15.72:0/1610435308 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x564bebf2a770 0x564bebf2cb90 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d49e422071a19fe server_cookie=2554ffc7c975afc2 in_seq=0 out_seq=0 2024-03-22T01:47:05.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.866+0000 7f194dffb640 1 -- 172.21.15.72:0/1610435308 <== mds.0 v2:172.21.15.89:6838/149200280 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564bebe16160 con 0x564bebf2a770 2024-03-22T01:47:05.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.866+0000 7f194dffb640 10 client.16437 handle_command_reply: tid=0 2024-03-22T01:47:05.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.866+0000 7f196fbef740 1 client.16437 shutdown 2024-03-22T01:47:05.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.866+0000 7f196fbef740 1 -- 172.21.15.72:0/1610435308 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564bebeaa800 msgr2=0x564bebebc1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:05.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.866+0000 7f196fbef740 1 --2- 172.21.15.72:0/1610435308 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564bebeaa800 0x564bebebc1a0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f19500058c0 tx=0x7f1950011f90 comp rx=0 tx=0).stop 2024-03-22T01:47:05.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.866+0000 7f196fbef740 1 -- 172.21.15.72:0/1610435308 shutdown_connections 2024-03-22T01:47:05.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.866+0000 7f196fbef740 1 --2- 172.21.15.72:0/1610435308 >> [v2:172.21.15.89:6838/149200280,v1:172.21.15.89:6839/149200280] conn(0x564bebf2a770 0x564bebf2cb90 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.866+0000 7f196fbef740 1 --2- 172.21.15.72:0/1610435308 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564bebeb3d00 0x564bebeb7a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.866+0000 7f196fbef740 1 --2- 172.21.15.72:0/1610435308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bebeab120 0x564bebeb74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.866+0000 7f194dffb640 0 client.16437 ms_handle_reset on v2:172.21.15.89:6838/149200280 2024-03-22T01:47:05.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.866+0000 7f196fbef740 1 --2- 172.21.15.72:0/1610435308 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564bebeaa800 0x564bebebc1a0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.866+0000 7f196fbef740 1 -- 172.21.15.72:0/1610435308 >> 172.21.15.72:0/1610435308 conn(0x564bebea4f30 msgr2=0x564bebea6190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:47:05.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.866+0000 7f196fbef740 1 -- 172.21.15.72:0/1610435308 shutdown_connections 2024-03-22T01:47:05.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.866+0000 7f196fbef740 1 -- 172.21.15.72:0/1610435308 wait complete. 2024-03-22T01:47:05.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.866+0000 7f196fbef740 20 client.16437 trim_cache size 0 max 16384 2024-03-22T01:47:05.878 INFO:teuthology.orchestra.run.smithi072.stdout:[] 2024-03-22T01:47:05.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.866+0000 7f196cf25640 1 -- 172.21.15.72:0/4003402914 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f19540a3090 msgr2=0x7f19540a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:05.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.866+0000 7f196cf25640 1 --2- 172.21.15.72:0/4003402914 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f19540a3090 0x7f19540a5550 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f196009c270 tx=0x7f196000efd0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.866+0000 7f196cf25640 1 -- 172.21.15.72:0/4003402914 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f196819bc70 msgr2=0x7f196819c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:05.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.866+0000 7f196cf25640 1 --2- 172.21.15.72:0/4003402914 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f196819bc70 0x7f196819c120 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f196000eb10 tx=0x7f196000c2d0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.866+0000 7f196cf25640 1 -- 172.21.15.72:0/4003402914 shutdown_connections 2024-03-22T01:47:05.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.866+0000 7f196cf25640 1 --2- 172.21.15.72:0/4003402914 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f19540a3090 0x7f19540a5550 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.866+0000 7f196cf25640 1 --2- 172.21.15.72:0/4003402914 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1968105df0 0x7f19681978f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.866+0000 7f196cf25640 1 --2- 172.21.15.72:0/4003402914 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f196819bc70 0x7f196819c120 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.866+0000 7f196cf25640 1 --2- 172.21.15.72:0/4003402914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1968101400 0x7f19681973b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:05.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.866+0000 7f196cf25640 1 -- 172.21.15.72:0/4003402914 >> 172.21.15.72:0/4003402914 conn(0x7f1968076f90 msgr2=0x7f1968103ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:47:05.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.866+0000 7f196cf25640 1 -- 172.21.15.72:0/4003402914 shutdown_connections 2024-03-22T01:47:05.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:05.866+0000 7f196cf25640 1 -- 172.21.15.72:0/4003402914 wait complete. 2024-03-22T01:47:06.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:06 smithi098 ceph-mon[36468]: pgmap v316: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 19 op/s 2024-03-22T01:47:06.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:06 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/30144594' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:47:06.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:06 smithi089 ceph-mon[35770]: pgmap v316: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 19 op/s 2024-03-22T01:47:06.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:06 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/30144594' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:47:06.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:06 smithi072 ceph-mon[32414]: pgmap v316: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 19 op/s 2024-03-22T01:47:06.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:06 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/30144594' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:47:06.881 INFO:tasks.fwd_scrub.fs.[cephfs]:end scrubbing fs: cephfs 2024-03-22T01:47:06.882 INFO:tasks.fwd_scrub:done joining 2024-03-22T01:47:06.882 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-03-22T01:47:06.896 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T01:47:07.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.011+0000 7f3b25ec9640 1 -- 172.21.15.72:0/4028531107 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3b20100a20 msgr2=0x7f3b20100ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:07.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.011+0000 7f3b25ec9640 1 --2- 172.21.15.72:0/4028531107 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3b20100a20 0x7f3b20100ea0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f3b14009930 tx=0x7f3b1402bc30 comp rx=0 tx=0).stop 2024-03-22T01:47:07.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.011+0000 7f3b25ec9640 1 -- 172.21.15.72:0/4028531107 shutdown_connections 2024-03-22T01:47:07.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.011+0000 7f3b25ec9640 1 --2- 172.21.15.72:0/4028531107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b201013e0 0x7f3b2010d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:07.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.011+0000 7f3b25ec9640 1 --2- 172.21.15.72:0/4028531107 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3b20100a20 0x7f3b20100ea0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:07.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.011+0000 7f3b25ec9640 1 --2- 172.21.15.72:0/4028531107 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b20106a30 0x7f3b20106e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:07.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.011+0000 7f3b25ec9640 1 -- 172.21.15.72:0/4028531107 >> 172.21.15.72:0/4028531107 conn(0x7f3b200fc510 msgr2=0x7f3b200fe930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:47:07.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.011+0000 7f3b25ec9640 1 -- 172.21.15.72:0/4028531107 shutdown_connections 2024-03-22T01:47:07.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.011+0000 7f3b25ec9640 1 -- 172.21.15.72:0/4028531107 wait complete. 2024-03-22T01:47:07.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.012+0000 7f3b25ec9640 1 Processor -- start 2024-03-22T01:47:07.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.012+0000 7f3b25ec9640 1 -- start start 2024-03-22T01:47:07.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.012+0000 7f3b25ec9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3b20100a20 0x7f3b200fcb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:07.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.012+0000 7f3b25ec9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b201013e0 0x7f3b200fd050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:07.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.012+0000 7f3b25ec9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b20106a30 0x7f3b200fd590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:07.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.012+0000 7f3b25ec9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f3b20110e40 con 0x7f3b20100a20 2024-03-22T01:47:07.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.012+0000 7f3b25ec9640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3b20110cc0 con 0x7f3b20106a30 2024-03-22T01:47:07.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.012+0000 7f3b25ec9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3b20110fc0 con 0x7f3b201013e0 2024-03-22T01:47:07.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.012+0000 7f3b1f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3b20100a20 0x7f3b200fcb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:07.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.012+0000 7f3b1f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3b20100a20 0x7f3b200fcb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59376/0 (socket says 172.21.15.72:59376) 2024-03-22T01:47:07.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.012+0000 7f3b1f7fe640 1 -- 172.21.15.72:0/3888304572 learned_addr learned my addr 172.21.15.72:0/3888304572 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:47:07.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.012+0000 7f3b1ffff640 1 --2- 172.21.15.72:0/3888304572 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b20106a30 0x7f3b200fd590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:07.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.013+0000 7f3b1effd640 1 --2- 172.21.15.72:0/3888304572 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b201013e0 0x7f3b200fd050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:07.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.013+0000 7f3b1f7fe640 1 -- 172.21.15.72:0/3888304572 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b201013e0 msgr2=0x7f3b200fd050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:07.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.013+0000 7f3b1f7fe640 1 --2- 172.21.15.72:0/3888304572 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b201013e0 0x7f3b200fd050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:07.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.013+0000 7f3b1f7fe640 1 -- 172.21.15.72:0/3888304572 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b20106a30 msgr2=0x7f3b200fd590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:07.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.013+0000 7f3b1f7fe640 1 --2- 172.21.15.72:0/3888304572 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b20106a30 0x7f3b200fd590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:07.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.013+0000 7f3b1f7fe640 1 -- 172.21.15.72:0/3888304572 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b201a6980 con 0x7f3b20100a20 2024-03-22T01:47:07.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.013+0000 7f3b1f7fe640 1 --2- 172.21.15.72:0/3888304572 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3b20100a20 0x7f3b200fcb10 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f3b1000c970 tx=0x7f3b1000cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:47:07.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.013+0000 7f3b1cff9640 1 -- 172.21.15.72:0/3888304572 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b10007bb0 con 0x7f3b20100a20 2024-03-22T01:47:07.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.013+0000 7f3b1cff9640 1 -- 172.21.15.72:0/3888304572 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b10007d50 con 0x7f3b20100a20 2024-03-22T01:47:07.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.013+0000 7f3b1cff9640 1 -- 172.21.15.72:0/3888304572 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b1000f450 con 0x7f3b20100a20 2024-03-22T01:47:07.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.013+0000 7f3b25ec9640 1 -- 172.21.15.72:0/3888304572 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b201a6c70 con 0x7f3b20100a20 2024-03-22T01:47:07.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.013+0000 7f3b25ec9640 1 -- 172.21.15.72:0/3888304572 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b201a7140 con 0x7f3b20100a20 2024-03-22T01:47:07.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.014+0000 7f3b25ec9640 1 -- 172.21.15.72:0/3888304572 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b20102160 con 0x7f3b20100a20 2024-03-22T01:47:07.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.015+0000 7f3b1cff9640 1 -- 172.21.15.72:0/3888304572 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 25) ==== 137985+0+0 (secure 0 0 0) 0x7f3b10004f20 con 0x7f3b20100a20 2024-03-22T01:47:07.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.016+0000 7f3b1cff9640 1 --2- 172.21.15.72:0/3888304572 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f3b080a3110 0x7f3b080a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:07.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.016+0000 7f3b1effd640 1 --2- 172.21.15.72:0/3888304572 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f3b080a3110 0x7f3b080a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:07.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.016+0000 7f3b1cff9640 1 -- 172.21.15.72:0/3888304572 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f3b100cfb80 con 0x7f3b20100a20 2024-03-22T01:47:07.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.016+0000 7f3b1effd640 1 --2- 172.21.15.72:0/3888304572 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f3b080a3110 0x7f3b080a55d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f3b14006010 tx=0x7f3b14000bc0 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:47:07.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.019+0000 7f3b1cff9640 1 -- 172.21.15.72:0/3888304572 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3b10014030 con 0x7f3b20100a20 2024-03-22T01:47:07.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.199+0000 7f3b25ec9640 1 -- 172.21.15.72:0/3888304572 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3b20068e60 con 0x7f3b20100a20 2024-03-22T01:47:07.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.200+0000 7f3b1cff9640 1 -- 172.21.15.72:0/3888304572 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) ==== 74+0+25134 (secure 0 0 0) 0x7f3b1009c340 con 0x7f3b20100a20 2024-03-22T01:47:07.200 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:47:07.200 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":96,"fsid":"022a99bc-e7eb-11ee-95cd-87774f69a715","created":"2024-03-22T01:26:23.893672+0000","modified":"2024-03-22T01:36:38.183837+0000","last_up_change":"2024-03-22T01:33:25.223684+0000","last_in_change":"2024-03-22T01:33:05.010331+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T01:29:37.438503+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-22T01:34:12.610422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-22T01:34:13.761686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-22T01:34:17.507039+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"16c15bb0-db92-4191-9063-38b4890e81e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6803","nonce":661906791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6805","nonce":661906791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6809","nonce":661906791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6807","nonce":661906791}]},"public_addr":"172.21.15.72:6803/661906791","cluster_addr":"172.21.15.72:6805/661906791","heartbeat_back_addr":"172.21.15.72:6809/661906791","heartbeat_front_addr":"172.21.15.72:6807/661906791","state":["exists","up"]},{"osd":1,"uuid":"981290ba-97dc-496b-aa22-d51704184b10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6801","nonce":2870590550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6803","nonce":2870590550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6807","nonce":2870590550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6805","nonce":2870590550}]},"public_addr":"172.21.15.89:6801/2870590550","cluster_addr":"172.21.15.89:6803/2870590550","heartbeat_back_addr":"172.21.15.89:6807/2870590550","heartbeat_front_addr":"172.21.15.89:6805/2870590550","state":["exists","up"]},{"osd":2,"uuid":"f07ed658-f545-46a1-8edd-8b1d5d49885c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6801","nonce":3865778118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6803","nonce":3865778118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6807","nonce":3865778118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6805","nonce":3865778118}]},"public_addr":"172.21.15.98:6801/3865778118","cluster_addr":"172.21.15.98:6803/3865778118","heartbeat_back_addr":"172.21.15.98:6807/3865778118","heartbeat_front_addr":"172.21.15.98:6805/3865778118","state":["exists","up"]},{"osd":3,"uuid":"ccc4bc40-2c48-46d2-8548-0c05b44f9f1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6811","nonce":1865951395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6813","nonce":1865951395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6817","nonce":1865951395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6815","nonce":1865951395}]},"public_addr":"172.21.15.72:6811/1865951395","cluster_addr":"172.21.15.72:6813/1865951395","heartbeat_back_addr":"172.21.15.72:6817/1865951395","heartbeat_front_addr":"172.21.15.72:6815/1865951395","state":["exists","up"]},{"osd":4,"uuid":"1812439f-810a-4ed2-9395-c00c9a085a13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6809","nonce":2683760795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6811","nonce":2683760795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6815","nonce":2683760795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6813","nonce":2683760795}]},"public_addr":"172.21.15.89:6809/2683760795","cluster_addr":"172.21.15.89:6811/2683760795","heartbeat_back_addr":"172.21.15.89:6815/2683760795","heartbeat_front_addr":"172.21.15.89:6813/2683760795","state":["exists","up"]},{"osd":5,"uuid":"e78954d8-f4e3-4d58-8536-d2d59c843386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6809","nonce":2635838572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6811","nonce":2635838572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6815","nonce":2635838572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6813","nonce":2635838572}]},"public_addr":"172.21.15.98:6809/2635838572","cluster_addr":"172.21.15.98:6811/2635838572","heartbeat_back_addr":"172.21.15.98:6815/2635838572","heartbeat_front_addr":"172.21.15.98:6813/2635838572","state":["exists","up"]},{"osd":6,"uuid":"dc6c8be2-b64c-4959-8015-7b0ea6d25599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6819","nonce":3158982661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6821","nonce":3158982661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6825","nonce":3158982661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6823","nonce":3158982661}]},"public_addr":"172.21.15.72:6819/3158982661","cluster_addr":"172.21.15.72:6821/3158982661","heartbeat_back_addr":"172.21.15.72:6825/3158982661","heartbeat_front_addr":"172.21.15.72:6823/3158982661","state":["exists","up"]},{"osd":7,"uuid":"7a73469f-cf3a-4306-ae93-ea46e21ebbbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6817","nonce":3075913234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6819","nonce":3075913234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6823","nonce":3075913234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6821","nonce":3075913234}]},"public_addr":"172.21.15.89:6817/3075913234","cluster_addr":"172.21.15.89:6819/3075913234","heartbeat_back_addr":"172.21.15.89:6823/3075913234","heartbeat_front_addr":"172.21.15.89:6821/3075913234","state":["exists","up"]},{"osd":8,"uuid":"c4b13bee-ea4d-4344-95fb-2622ecbb8196","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6817","nonce":1813255577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6819","nonce":1813255577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6823","nonce":1813255577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6821","nonce":1813255577}]},"public_addr":"172.21.15.98:6817/1813255577","cluster_addr":"172.21.15.98:6819/1813255577","heartbeat_back_addr":"172.21.15.98:6823/1813255577","heartbeat_front_addr":"172.21.15.98:6821/1813255577","state":["exists","up"]},{"osd":9,"uuid":"eefb0a44-4a36-4e55-9d04-0501bf718db6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6827","nonce":3377749206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6829","nonce":3377749206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6832","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6833","nonce":3377749206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6831","nonce":3377749206}]},"public_addr":"172.21.15.72:6827/3377749206","cluster_addr":"172.21.15.72:6829/3377749206","heartbeat_back_addr":"172.21.15.72:6833/3377749206","heartbeat_front_addr":"172.21.15.72:6831/3377749206","state":["exists","up"]},{"osd":10,"uuid":"8741d25f-40dd-40d7-b040-5cc5e411eb98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6825","nonce":885058786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6827","nonce":885058786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6831","nonce":885058786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6829","nonce":885058786}]},"public_addr":"172.21.15.89:6825/885058786","cluster_addr":"172.21.15.89:6827/885058786","heartbeat_back_addr":"172.21.15.89:6831/885058786","heartbeat_front_addr":"172.21.15.89:6829/885058786","state":["exists","up"]},{"osd":11,"uuid":"8d02e957-eda4-4184-b5c6-5eb3b8b38060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6825","nonce":1517095838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6827","nonce":1517095838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6831","nonce":1517095838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6829","nonce":1517095838}]},"public_addr":"172.21.15.98:6825/1517095838","cluster_addr":"172.21.15.98:6827/1517095838","heartbeat_back_addr":"172.21.15.98:6831/1517095838","heartbeat_front_addr":"172.21.15.98:6829/1517095838","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:28:47.307652+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:11.037135+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:34.619188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:59.350738+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:30:23.787337+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:30:48.994154+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:31:13.690350+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:31:39.628116+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:04.723912+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:30.689741+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:57.229508+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:33:23.154537+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":7}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:0/2032770149":"2024-03-23T01:36:38.183806+0000","172.21.15.72:6801/401403200":"2024-03-23T01:36:38.183806+0000","172.21.15.72:0/1161486989":"2024-03-23T01:36:38.183806+0000","172.21.15.72:6800/401403200":"2024-03-23T01:36:38.183806+0000","172.21.15.72:0/121087196":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/2451857761":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/1228477995":"2024-03-23T01:27:15.339809+0000","172.21.15.72:6801/1513232913":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/4223447038":"2024-03-23T01:36:38.183806+0000","172.21.15.72:0/3558958264":"2024-03-23T01:26:50.074871+0000","172.21.15.72:0/1510187198":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6800/3518012916":"2024-03-23T01:26:50.074871+0000","172.21.15.72:0/608566205":"2024-03-23T01:36:38.183806+0000","172.21.15.72:6801/3518012916":"2024-03-23T01:26:50.074871+0000","172.21.15.72:0/1372801041":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6800/1513232913":"2024-03-23T01:27:15.339809+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T01:47:07.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.202+0000 7f3b25ec9640 1 -- 172.21.15.72:0/3888304572 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f3b080a3110 msgr2=0x7f3b080a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:07.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.202+0000 7f3b25ec9640 1 --2- 172.21.15.72:0/3888304572 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f3b080a3110 0x7f3b080a55d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f3b14006010 tx=0x7f3b14000bc0 comp rx=0 tx=0).stop 2024-03-22T01:47:07.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.202+0000 7f3b25ec9640 1 -- 172.21.15.72:0/3888304572 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3b20100a20 msgr2=0x7f3b200fcb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:07.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.202+0000 7f3b25ec9640 1 --2- 172.21.15.72:0/3888304572 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3b20100a20 0x7f3b200fcb10 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f3b1000c970 tx=0x7f3b1000cd70 comp rx=0 tx=0).stop 2024-03-22T01:47:07.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.202+0000 7f3b25ec9640 1 -- 172.21.15.72:0/3888304572 shutdown_connections 2024-03-22T01:47:07.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.202+0000 7f3b25ec9640 1 --2- 172.21.15.72:0/3888304572 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f3b080a3110 0x7f3b080a55d0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:07.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.202+0000 7f3b25ec9640 1 --2- 172.21.15.72:0/3888304572 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b20106a30 0x7f3b200fd590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:07.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.202+0000 7f3b25ec9640 1 --2- 172.21.15.72:0/3888304572 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b201013e0 0x7f3b200fd050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:07.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.202+0000 7f3b25ec9640 1 --2- 172.21.15.72:0/3888304572 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3b20100a20 0x7f3b200fcb10 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:07.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.202+0000 7f3b25ec9640 1 -- 172.21.15.72:0/3888304572 >> 172.21.15.72:0/3888304572 conn(0x7f3b200fc510 msgr2=0x7f3b2006a5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:47:07.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.202+0000 7f3b25ec9640 1 -- 172.21.15.72:0/3888304572 shutdown_connections 2024-03-22T01:47:07.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.202+0000 7f3b25ec9640 1 -- 172.21.15.72:0/3888304572 wait complete. 2024-03-22T01:47:07.213 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-22T01:29:37.438503+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-22T01:34:12.610422+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-22T01:34:13.761686+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-22T01:34:17.507039+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-22T01:47:07.213 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-22T01:47:07.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.319+0000 7effe6e55640 1 -- 172.21.15.72:0/1274103948 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7effe0100a50 msgr2=0x7effe0100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:07.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.319+0000 7effe6e55640 1 --2- 172.21.15.72:0/1274103948 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7effe0100a50 0x7effe0100ed0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7effcc009930 tx=0x7effcc02bc30 comp rx=0 tx=0).stop 2024-03-22T01:47:07.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.319+0000 7effe6e55640 1 -- 172.21.15.72:0/1274103948 shutdown_connections 2024-03-22T01:47:07.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.319+0000 7effe6e55640 1 --2- 172.21.15.72:0/1274103948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7effe0101410 0x7effe010da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:07.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.319+0000 7effe6e55640 1 --2- 172.21.15.72:0/1274103948 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7effe0100a50 0x7effe0100ed0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:07.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.319+0000 7effe6e55640 1 --2- 172.21.15.72:0/1274103948 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effe0106b70 0x7effe0106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:07.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.319+0000 7effe6e55640 1 -- 172.21.15.72:0/1274103948 >> 172.21.15.72:0/1274103948 conn(0x7effe0077c00 msgr2=0x7effe0078010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:47:07.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.319+0000 7effe6e55640 1 -- 172.21.15.72:0/1274103948 shutdown_connections 2024-03-22T01:47:07.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.319+0000 7effe6e55640 1 -- 172.21.15.72:0/1274103948 wait complete. 2024-03-22T01:47:07.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.320+0000 7effe6e55640 1 Processor -- start 2024-03-22T01:47:07.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.320+0000 7effe6e55640 1 -- start start 2024-03-22T01:47:07.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.320+0000 7effe6e55640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7effe0100a50 0x7effe01985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:07.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.320+0000 7effe6e55640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effe0101410 0x7effe0198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:07.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.320+0000 7effe6e55640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7effe0106b70 0x7effe019ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:07.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.320+0000 7effe6e55640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7effe0110f80 con 0x7effe0106b70 2024-03-22T01:47:07.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.320+0000 7effe6e55640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7effe0110e00 con 0x7effe0101410 2024-03-22T01:47:07.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.320+0000 7effe6e55640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7effe0111100 con 0x7effe0100a50 2024-03-22T01:47:07.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.320+0000 7effe53f2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7effe0106b70 0x7effe019ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:07.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.320+0000 7effe53f2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7effe0106b70 0x7effe019ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59392/0 (socket says 172.21.15.72:59392) 2024-03-22T01:47:07.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.320+0000 7effe53f2640 1 -- 172.21.15.72:0/3494585581 learned_addr learned my addr 172.21.15.72:0/3494585581 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:47:07.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.320+0000 7effdffff640 1 --2- 172.21.15.72:0/3494585581 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effe0101410 0x7effe0198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:07.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.320+0000 7effe4bf1640 1 --2- 172.21.15.72:0/3494585581 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7effe0100a50 0x7effe01985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:07.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.320+0000 7effe53f2640 1 -- 172.21.15.72:0/3494585581 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7effe0100a50 msgr2=0x7effe01985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:07.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.320+0000 7effe53f2640 1 --2- 172.21.15.72:0/3494585581 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7effe0100a50 0x7effe01985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:07.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.320+0000 7effe53f2640 1 -- 172.21.15.72:0/3494585581 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effe0101410 msgr2=0x7effe0198b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:07.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.320+0000 7effe53f2640 1 --2- 172.21.15.72:0/3494585581 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effe0101410 0x7effe0198b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:07.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.320+0000 7effe53f2640 1 -- 172.21.15.72:0/3494585581 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effe019d5c0 con 0x7effe0106b70 2024-03-22T01:47:07.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.320+0000 7effe4bf1640 1 --2- 172.21.15.72:0/3494585581 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7effe0100a50 0x7effe01985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:47:07.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.321+0000 7effe53f2640 1 --2- 172.21.15.72:0/3494585581 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7effe0106b70 0x7effe019ce90 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7effd800e960 tx=0x7effd800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:47:07.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.321+0000 7effddffb640 1 -- 172.21.15.72:0/3494585581 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effd8004d20 con 0x7effe0106b70 2024-03-22T01:47:07.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.321+0000 7effddffb640 1 -- 172.21.15.72:0/3494585581 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7effd8005020 con 0x7effe0106b70 2024-03-22T01:47:07.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.321+0000 7effe6e55640 1 -- 172.21.15.72:0/3494585581 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effe019d8b0 con 0x7effe0106b70 2024-03-22T01:47:07.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.321+0000 7effddffb640 1 -- 172.21.15.72:0/3494585581 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effd80116d0 con 0x7effe0106b70 2024-03-22T01:47:07.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.321+0000 7effe6e55640 1 -- 172.21.15.72:0/3494585581 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effe01a5140 con 0x7effe0106b70 2024-03-22T01:47:07.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.322+0000 7effe6e55640 1 -- 172.21.15.72:0/3494585581 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effe0068e60 con 0x7effe0106b70 2024-03-22T01:47:07.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.323+0000 7effddffb640 1 -- 172.21.15.72:0/3494585581 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 25) ==== 137985+0+0 (secure 0 0 0) 0x7effd8005560 con 0x7effe0106b70 2024-03-22T01:47:07.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.323+0000 7effddffb640 1 --2- 172.21.15.72:0/3494585581 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7effd00a3090 0x7effd00a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:07.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.323+0000 7effe4bf1640 1 --2- 172.21.15.72:0/3494585581 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7effd00a3090 0x7effd00a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:07.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.323+0000 7effddffb640 1 -- 172.21.15.72:0/3494585581 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7effd80d0070 con 0x7effe0106b70 2024-03-22T01:47:07.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.324+0000 7effe4bf1640 1 --2- 172.21.15.72:0/3494585581 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7effd00a3090 0x7effd00a5550 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7effd4009ba0 tx=0x7effd4005f90 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:47:07.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.328+0000 7effddffb640 1 -- 172.21.15.72:0/3494585581 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7effd8016030 con 0x7effe0106b70 2024-03-22T01:47:07.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.503+0000 7effe6e55640 1 -- 172.21.15.72:0/3494585581 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7effe0102190 con 0x7effe0106b70 2024-03-22T01:47:07.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.503+0000 7effddffb640 1 -- 172.21.15.72:0/3494585581 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v96) ==== 93+0+10 (secure 0 0 0) 0x7effd80bae20 con 0x7effe0106b70 2024-03-22T01:47:07.503 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-22T01:47:07.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.505+0000 7effe6e55640 1 -- 172.21.15.72:0/3494585581 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7effd00a3090 msgr2=0x7effd00a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:07.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.505+0000 7effe6e55640 1 --2- 172.21.15.72:0/3494585581 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7effd00a3090 0x7effd00a5550 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7effd4009ba0 tx=0x7effd4005f90 comp rx=0 tx=0).stop 2024-03-22T01:47:07.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.505+0000 7effe6e55640 1 -- 172.21.15.72:0/3494585581 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7effe0106b70 msgr2=0x7effe019ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:07.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.505+0000 7effe6e55640 1 --2- 172.21.15.72:0/3494585581 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7effe0106b70 0x7effe019ce90 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7effd800e960 tx=0x7effd800ee20 comp rx=0 tx=0).stop 2024-03-22T01:47:07.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.505+0000 7effe6e55640 1 -- 172.21.15.72:0/3494585581 shutdown_connections 2024-03-22T01:47:07.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.505+0000 7effe6e55640 1 --2- 172.21.15.72:0/3494585581 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7effd00a3090 0x7effd00a5550 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:07.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.505+0000 7effe6e55640 1 --2- 172.21.15.72:0/3494585581 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effe0101410 0x7effe0198b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:07.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.505+0000 7effe6e55640 1 --2- 172.21.15.72:0/3494585581 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7effe0106b70 0x7effe019ce90 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:07.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.505+0000 7effe6e55640 1 --2- 172.21.15.72:0/3494585581 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7effe0100a50 0x7effe01985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:07.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.505+0000 7effe6e55640 1 -- 172.21.15.72:0/3494585581 >> 172.21.15.72:0/3494585581 conn(0x7effe0077c00 msgr2=0x7effe006a590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:47:07.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.505+0000 7effe6e55640 1 -- 172.21.15.72:0/3494585581 shutdown_connections 2024-03-22T01:47:07.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.505+0000 7effe6e55640 1 -- 172.21.15.72:0/3494585581 wait complete. 2024-03-22T01:47:07.515 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-22T01:47:07.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.621+0000 7fadb7dd7640 1 -- 172.21.15.72:0/1523508292 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fadb0107c10 msgr2=0x7fadb0107ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:07.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.621+0000 7fadb7dd7640 1 --2- 172.21.15.72:0/1523508292 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fadb0107c10 0x7fadb0107ff0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fadac009900 tx=0x7fadac02bc30 comp rx=0 tx=0).stop 2024-03-22T01:47:07.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.621+0000 7fadb7dd7640 1 -- 172.21.15.72:0/1523508292 shutdown_connections 2024-03-22T01:47:07.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.621+0000 7fadb7dd7640 1 --2- 172.21.15.72:0/1523508292 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fadb0102510 0x7fadb010e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:07.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.621+0000 7fadb7dd7640 1 --2- 172.21.15.72:0/1523508292 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fadb0101b50 0x7fadb0101fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:07.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.621+0000 7fadb7dd7640 1 --2- 172.21.15.72:0/1523508292 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fadb0107c10 0x7fadb0107ff0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:07.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.621+0000 7fadb7dd7640 1 -- 172.21.15.72:0/1523508292 >> 172.21.15.72:0/1523508292 conn(0x7fadb00fd620 msgr2=0x7fadb00ffa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:47:07.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.621+0000 7fadb7dd7640 1 -- 172.21.15.72:0/1523508292 shutdown_connections 2024-03-22T01:47:07.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.621+0000 7fadb7dd7640 1 -- 172.21.15.72:0/1523508292 wait complete. 2024-03-22T01:47:07.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.622+0000 7fadb7dd7640 1 Processor -- start 2024-03-22T01:47:07.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.622+0000 7fadb7dd7640 1 -- start start 2024-03-22T01:47:07.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.622+0000 7fadb7dd7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fadb0101b50 0x7fadb0198370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:07.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.622+0000 7fadb7dd7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fadb0102510 0x7fadb01988b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:07.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.622+0000 7fadb7dd7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fadb0107c10 0x7fadb019cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:07.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.622+0000 7fadb7dd7640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fadb0111f50 con 0x7fadb0102510 2024-03-22T01:47:07.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.622+0000 7fadb7dd7640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fadb0111dd0 con 0x7fadb0107c10 2024-03-22T01:47:07.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.622+0000 7fadb7dd7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fadb01120d0 con 0x7fadb0101b50 2024-03-22T01:47:07.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.622+0000 7fadb5372640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fadb0102510 0x7fadb01988b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:07.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.622+0000 7fadb5372640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fadb0102510 0x7fadb01988b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59404/0 (socket says 172.21.15.72:59404) 2024-03-22T01:47:07.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.622+0000 7fadb5372640 1 -- 172.21.15.72:0/665360941 learned_addr learned my addr 172.21.15.72:0/665360941 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:47:07.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.622+0000 7fadb6374640 1 --2- 172.21.15.72:0/665360941 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fadb0107c10 0x7fadb019cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:07.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.623+0000 7fadb5b73640 1 --2- 172.21.15.72:0/665360941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fadb0101b50 0x7fadb0198370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:07.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.623+0000 7fadb5372640 1 -- 172.21.15.72:0/665360941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fadb0101b50 msgr2=0x7fadb0198370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:07.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.623+0000 7fadb5372640 1 --2- 172.21.15.72:0/665360941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fadb0101b50 0x7fadb0198370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:07.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.623+0000 7fadb5372640 1 -- 172.21.15.72:0/665360941 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fadb0107c10 msgr2=0x7fadb019cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:07.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.623+0000 7fadb5372640 1 --2- 172.21.15.72:0/665360941 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fadb0107c10 0x7fadb019cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:07.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.623+0000 7fadb5372640 1 -- 172.21.15.72:0/665360941 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadb019d360 con 0x7fadb0102510 2024-03-22T01:47:07.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.623+0000 7fadb6374640 1 --2- 172.21.15.72:0/665360941 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fadb0107c10 0x7fadb019cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:47:07.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.623+0000 7fadb5372640 1 --2- 172.21.15.72:0/665360941 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fadb0102510 0x7fadb01988b0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fada000c970 tx=0x7fada000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:47:07.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.623+0000 7fada6ffd640 1 -- 172.21.15.72:0/665360941 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fada0007bb0 con 0x7fadb0102510 2024-03-22T01:47:07.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.623+0000 7fada6ffd640 1 -- 172.21.15.72:0/665360941 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fada0007d50 con 0x7fadb0102510 2024-03-22T01:47:07.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.623+0000 7fadb7dd7640 1 -- 172.21.15.72:0/665360941 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadb019d650 con 0x7fadb0102510 2024-03-22T01:47:07.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.623+0000 7fada6ffd640 1 -- 172.21.15.72:0/665360941 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fada000f450 con 0x7fadb0102510 2024-03-22T01:47:07.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.623+0000 7fadb7dd7640 1 -- 172.21.15.72:0/665360941 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadb01a4ee0 con 0x7fadb0102510 2024-03-22T01:47:07.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.624+0000 7fadb7dd7640 1 -- 172.21.15.72:0/665360941 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fadb0068a10 con 0x7fadb0102510 2024-03-22T01:47:07.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.625+0000 7fada6ffd640 1 -- 172.21.15.72:0/665360941 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 25) ==== 137985+0+0 (secure 0 0 0) 0x7fada0005020 con 0x7fadb0102510 2024-03-22T01:47:07.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.625+0000 7fada6ffd640 1 --2- 172.21.15.72:0/665360941 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fad9c0a3090 0x7fad9c0a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:07.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.626+0000 7fadb5b73640 1 --2- 172.21.15.72:0/665360941 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fad9c0a3090 0x7fad9c0a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:07.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.626+0000 7fada6ffd640 1 -- 172.21.15.72:0/665360941 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fada00cf970 con 0x7fadb0102510 2024-03-22T01:47:07.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.626+0000 7fadb5b73640 1 --2- 172.21.15.72:0/665360941 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fad9c0a3090 0x7fad9c0a5550 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fadac02efd0 tx=0x7fadac000bc0 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:47:07.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.630+0000 7fada6ffd640 1 -- 172.21.15.72:0/665360941 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fada0014030 con 0x7fadb0102510 2024-03-22T01:47:07.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.807+0000 7fadb7dd7640 1 -- 172.21.15.72:0/665360941 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fadb002d3b0 con 0x7fadb0102510 2024-03-22T01:47:07.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.808+0000 7fada6ffd640 1 -- 172.21.15.72:0/665360941 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v96) ==== 104+0+11 (secure 0 0 0) 0x7fada009c0b0 con 0x7fadb0102510 2024-03-22T01:47:07.807 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-03-22T01:47:07.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.810+0000 7fadb7dd7640 1 -- 172.21.15.72:0/665360941 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fad9c0a3090 msgr2=0x7fad9c0a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:07.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.810+0000 7fadb7dd7640 1 --2- 172.21.15.72:0/665360941 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fad9c0a3090 0x7fad9c0a5550 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fadac02efd0 tx=0x7fadac000bc0 comp rx=0 tx=0).stop 2024-03-22T01:47:07.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.810+0000 7fadb7dd7640 1 -- 172.21.15.72:0/665360941 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fadb0102510 msgr2=0x7fadb01988b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:07.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.810+0000 7fadb7dd7640 1 --2- 172.21.15.72:0/665360941 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fadb0102510 0x7fadb01988b0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fada000c970 tx=0x7fada000ce30 comp rx=0 tx=0).stop 2024-03-22T01:47:07.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.810+0000 7fadb7dd7640 1 -- 172.21.15.72:0/665360941 shutdown_connections 2024-03-22T01:47:07.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.810+0000 7fadb7dd7640 1 --2- 172.21.15.72:0/665360941 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fad9c0a3090 0x7fad9c0a5550 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:07.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.810+0000 7fadb7dd7640 1 --2- 172.21.15.72:0/665360941 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fadb0107c10 0x7fadb019cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:07.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.810+0000 7fadb7dd7640 1 --2- 172.21.15.72:0/665360941 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fadb0102510 0x7fadb01988b0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:07.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.810+0000 7fadb7dd7640 1 --2- 172.21.15.72:0/665360941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fadb0101b50 0x7fadb0198370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:07.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.810+0000 7fadb7dd7640 1 -- 172.21.15.72:0/665360941 >> 172.21.15.72:0/665360941 conn(0x7fadb00fd620 msgr2=0x7fadb00fece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:47:07.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.810+0000 7fadb7dd7640 1 -- 172.21.15.72:0/665360941 shutdown_connections 2024-03-22T01:47:07.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.810+0000 7fadb7dd7640 1 -- 172.21.15.72:0/665360941 wait complete. 2024-03-22T01:47:07.821 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-22T01:47:07.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.927+0000 7fe29f439640 1 -- 172.21.15.72:0/142882099 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe298106a30 msgr2=0x7fe298106e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:07.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.927+0000 7fe29f439640 1 --2- 172.21.15.72:0/142882099 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe298106a30 0x7fe298106e10 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fe288009930 tx=0x7fe28802bc30 comp rx=0 tx=0).stop 2024-03-22T01:47:07.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.927+0000 7fe29f439640 1 -- 172.21.15.72:0/142882099 shutdown_connections 2024-03-22T01:47:07.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.927+0000 7fe29f439640 1 --2- 172.21.15.72:0/142882099 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe2981013e0 0x7fe29810d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:07.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.927+0000 7fe29f439640 1 --2- 172.21.15.72:0/142882099 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe298100a20 0x7fe298100ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:07.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.927+0000 7fe29f439640 1 --2- 172.21.15.72:0/142882099 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe298106a30 0x7fe298106e10 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:07.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.927+0000 7fe29f439640 1 -- 172.21.15.72:0/142882099 >> 172.21.15.72:0/142882099 conn(0x7fe2980fc4f0 msgr2=0x7fe2980fe910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:47:07.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.927+0000 7fe29f439640 1 -- 172.21.15.72:0/142882099 shutdown_connections 2024-03-22T01:47:07.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.927+0000 7fe29f439640 1 -- 172.21.15.72:0/142882099 wait complete. 2024-03-22T01:47:07.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.927+0000 7fe29f439640 1 Processor -- start 2024-03-22T01:47:07.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.927+0000 7fe29f439640 1 -- start start 2024-03-22T01:47:07.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.928+0000 7fe29f439640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe298100a20 0x7fe298198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:07.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.928+0000 7fe29f439640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2981013e0 0x7fe298198940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:07.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.928+0000 7fe29f439640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe298106a30 0x7fe29819ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:07.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.928+0000 7fe29f439640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fe298111f00 con 0x7fe298106a30 2024-03-22T01:47:07.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.928+0000 7fe29f439640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe298111d80 con 0x7fe298100a20 2024-03-22T01:47:07.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.928+0000 7fe29f439640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe298112080 con 0x7fe2981013e0 2024-03-22T01:47:07.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.928+0000 7fe29d9d6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe298106a30 0x7fe29819ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:07.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.928+0000 7fe29d9d6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe298106a30 0x7fe29819ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59412/0 (socket says 172.21.15.72:59412) 2024-03-22T01:47:07.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.928+0000 7fe29d9d6640 1 -- 172.21.15.72:0/1492914133 learned_addr learned my addr 172.21.15.72:0/1492914133 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:47:07.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.928+0000 7fe29d1d5640 1 --2- 172.21.15.72:0/1492914133 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe298100a20 0x7fe298198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:07.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.928+0000 7fe29c9d4640 1 --2- 172.21.15.72:0/1492914133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2981013e0 0x7fe298198940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:07.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.928+0000 7fe29d9d6640 1 -- 172.21.15.72:0/1492914133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2981013e0 msgr2=0x7fe298198940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:07.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.928+0000 7fe29d9d6640 1 --2- 172.21.15.72:0/1492914133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2981013e0 0x7fe298198940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:07.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.928+0000 7fe29d9d6640 1 -- 172.21.15.72:0/1492914133 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe298100a20 msgr2=0x7fe298198400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:07.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.928+0000 7fe29d9d6640 1 --2- 172.21.15.72:0/1492914133 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe298100a20 0x7fe298198400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:07.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.928+0000 7fe29d9d6640 1 -- 172.21.15.72:0/1492914133 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe29819d3f0 con 0x7fe298106a30 2024-03-22T01:47:07.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.928+0000 7fe29d1d5640 1 --2- 172.21.15.72:0/1492914133 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe298100a20 0x7fe298198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:47:07.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.928+0000 7fe29d9d6640 1 --2- 172.21.15.72:0/1492914133 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe298106a30 0x7fe29819ccc0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fe29000eb10 tx=0x7fe29000ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:47:07.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.928+0000 7fe28e7fc640 1 -- 172.21.15.72:0/1492914133 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe29000ce20 con 0x7fe298106a30 2024-03-22T01:47:07.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.928+0000 7fe28e7fc640 1 -- 172.21.15.72:0/1492914133 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe290005020 con 0x7fe298106a30 2024-03-22T01:47:07.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.928+0000 7fe29f439640 1 -- 172.21.15.72:0/1492914133 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe29819d6e0 con 0x7fe298106a30 2024-03-22T01:47:07.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.928+0000 7fe28e7fc640 1 -- 172.21.15.72:0/1492914133 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe290010690 con 0x7fe298106a30 2024-03-22T01:47:07.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.928+0000 7fe29f439640 1 -- 172.21.15.72:0/1492914133 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2981a4f70 con 0x7fe298106a30 2024-03-22T01:47:07.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.929+0000 7fe29f439640 1 -- 172.21.15.72:0/1492914133 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe298102160 con 0x7fe298106a30 2024-03-22T01:47:07.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.930+0000 7fe28e7fc640 1 -- 172.21.15.72:0/1492914133 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 25) ==== 137985+0+0 (secure 0 0 0) 0x7fe290005540 con 0x7fe298106a30 2024-03-22T01:47:07.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.931+0000 7fe28e7fc640 1 --2- 172.21.15.72:0/1492914133 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fe2840a3090 0x7fe2840a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:07.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.931+0000 7fe29d1d5640 1 --2- 172.21.15.72:0/1492914133 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fe2840a3090 0x7fe2840a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:07.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.931+0000 7fe28e7fc640 1 -- 172.21.15.72:0/1492914133 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fe2900d00a0 con 0x7fe298106a30 2024-03-22T01:47:07.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.932+0000 7fe29d1d5640 1 --2- 172.21.15.72:0/1492914133 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fe2840a3090 0x7fe2840a5550 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe288005ab0 tx=0x7fe288005fd0 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:47:07.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:07.936+0000 7fe28e7fc640 1 -- 172.21.15.72:0/1492914133 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe29009c7e0 con 0x7fe298106a30 2024-03-22T01:47:08.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.113+0000 7fe29f439640 1 -- 172.21.15.72:0/1492914133 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe298199750 con 0x7fe298106a30 2024-03-22T01:47:08.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.114+0000 7fe28e7fc640 1 -- 172.21.15.72:0/1492914133 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v96) ==== 100+0+11 (secure 0 0 0) 0x7fe29009c0e0 con 0x7fe298106a30 2024-03-22T01:47:08.113 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-03-22T01:47:08.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.116+0000 7fe29f439640 1 -- 172.21.15.72:0/1492914133 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fe2840a3090 msgr2=0x7fe2840a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:08.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.116+0000 7fe29f439640 1 --2- 172.21.15.72:0/1492914133 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fe2840a3090 0x7fe2840a5550 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe288005ab0 tx=0x7fe288005fd0 comp rx=0 tx=0).stop 2024-03-22T01:47:08.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.116+0000 7fe29f439640 1 -- 172.21.15.72:0/1492914133 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe298106a30 msgr2=0x7fe29819ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:08.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.116+0000 7fe29f439640 1 --2- 172.21.15.72:0/1492914133 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe298106a30 0x7fe29819ccc0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fe29000eb10 tx=0x7fe29000ef10 comp rx=0 tx=0).stop 2024-03-22T01:47:08.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.116+0000 7fe29f439640 1 -- 172.21.15.72:0/1492914133 shutdown_connections 2024-03-22T01:47:08.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.116+0000 7fe29f439640 1 --2- 172.21.15.72:0/1492914133 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fe2840a3090 0x7fe2840a5550 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:08.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.116+0000 7fe29f439640 1 --2- 172.21.15.72:0/1492914133 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe298106a30 0x7fe29819ccc0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:08.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.116+0000 7fe29f439640 1 --2- 172.21.15.72:0/1492914133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2981013e0 0x7fe298198940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:08.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.116+0000 7fe29f439640 1 --2- 172.21.15.72:0/1492914133 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe298100a20 0x7fe298198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:08.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.116+0000 7fe29f439640 1 -- 172.21.15.72:0/1492914133 >> 172.21.15.72:0/1492914133 conn(0x7fe2980fc4f0 msgr2=0x7fe2980fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:47:08.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.116+0000 7fe29f439640 1 -- 172.21.15.72:0/1492914133 shutdown_connections 2024-03-22T01:47:08.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.116+0000 7fe29f439640 1 -- 172.21.15.72:0/1492914133 wait complete. 2024-03-22T01:47:08.125 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-22T01:47:08.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.231+0000 7f0b494d4640 1 -- 172.21.15.72:0/3051187532 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0b44106a40 msgr2=0x7f0b44106e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:08.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.231+0000 7f0b494d4640 1 --2- 172.21.15.72:0/3051187532 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0b44106a40 0x7f0b44106e20 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f0b3c009cd0 tx=0x7f0b3c02c0f0 comp rx=0 tx=0).stop 2024-03-22T01:47:08.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.231+0000 7f0b494d4640 1 -- 172.21.15.72:0/3051187532 shutdown_connections 2024-03-22T01:47:08.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.231+0000 7f0b494d4640 1 --2- 172.21.15.72:0/3051187532 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b441013f0 0x7f0b4410d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:08.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.231+0000 7f0b494d4640 1 --2- 172.21.15.72:0/3051187532 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b44100a30 0x7f0b44100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:08.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.231+0000 7f0b494d4640 1 --2- 172.21.15.72:0/3051187532 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0b44106a40 0x7f0b44106e20 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:08.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.231+0000 7f0b494d4640 1 -- 172.21.15.72:0/3051187532 >> 172.21.15.72:0/3051187532 conn(0x7f0b44077c40 msgr2=0x7f0b44078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:47:08.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.232+0000 7f0b494d4640 1 -- 172.21.15.72:0/3051187532 shutdown_connections 2024-03-22T01:47:08.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.232+0000 7f0b494d4640 1 -- 172.21.15.72:0/3051187532 wait complete. 2024-03-22T01:47:08.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.232+0000 7f0b494d4640 1 Processor -- start 2024-03-22T01:47:08.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.232+0000 7f0b494d4640 1 -- start start 2024-03-22T01:47:08.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.232+0000 7f0b494d4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b44100a30 0x7f0b441992c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:08.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.232+0000 7f0b494d4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b441013f0 0x7f0b44199800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:08.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.232+0000 7f0b494d4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0b44106a40 0x7f0b441a4340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:08.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.232+0000 7f0b494d4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f0b44110e20 con 0x7f0b44106a40 2024-03-22T01:47:08.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.232+0000 7f0b494d4640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f0b44110ca0 con 0x7f0b441013f0 2024-03-22T01:47:08.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.232+0000 7f0b494d4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0b44110fa0 con 0x7f0b44100a30 2024-03-22T01:47:08.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.233+0000 7f0b437fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0b44106a40 0x7f0b441a4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:08.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.233+0000 7f0b437fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0b44106a40 0x7f0b441a4340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59436/0 (socket says 172.21.15.72:59436) 2024-03-22T01:47:08.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.233+0000 7f0b437fe640 1 -- 172.21.15.72:0/1991817475 learned_addr learned my addr 172.21.15.72:0/1991817475 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:47:08.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.233+0000 7f0b427fc640 1 --2- 172.21.15.72:0/1991817475 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b441013f0 0x7f0b44199800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:08.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.233+0000 7f0b42ffd640 1 --2- 172.21.15.72:0/1991817475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b44100a30 0x7f0b441992c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:08.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.233+0000 7f0b437fe640 1 -- 172.21.15.72:0/1991817475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b44100a30 msgr2=0x7f0b441992c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:08.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.233+0000 7f0b437fe640 1 --2- 172.21.15.72:0/1991817475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b44100a30 0x7f0b441992c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:08.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.233+0000 7f0b437fe640 1 -- 172.21.15.72:0/1991817475 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b441013f0 msgr2=0x7f0b44199800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:08.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.233+0000 7f0b437fe640 1 --2- 172.21.15.72:0/1991817475 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b441013f0 0x7f0b44199800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:08.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.233+0000 7f0b437fe640 1 -- 172.21.15.72:0/1991817475 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b441a4880 con 0x7f0b44106a40 2024-03-22T01:47:08.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.233+0000 7f0b42ffd640 1 --2- 172.21.15.72:0/1991817475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b44100a30 0x7f0b441992c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T01:47:08.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.233+0000 7f0b437fe640 1 --2- 172.21.15.72:0/1991817475 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0b44106a40 0x7f0b441a4340 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f0b3400e970 tx=0x7f0b3400ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:47:08.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.233+0000 7f0b3bfff640 1 -- 172.21.15.72:0/1991817475 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b3400ccc0 con 0x7f0b44106a40 2024-03-22T01:47:08.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.233+0000 7f0b3bfff640 1 -- 172.21.15.72:0/1991817475 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b3400ce60 con 0x7f0b44106a40 2024-03-22T01:47:08.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.233+0000 7f0b494d4640 1 -- 172.21.15.72:0/1991817475 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b441a4a50 con 0x7f0b44106a40 2024-03-22T01:47:08.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.233+0000 7f0b3bfff640 1 -- 172.21.15.72:0/1991817475 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b34010430 con 0x7f0b44106a40 2024-03-22T01:47:08.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.233+0000 7f0b494d4640 1 -- 172.21.15.72:0/1991817475 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b441a4ef0 con 0x7f0b44106a40 2024-03-22T01:47:08.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.234+0000 7f0b494d4640 1 -- 172.21.15.72:0/1991817475 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b44068a10 con 0x7f0b44106a40 2024-03-22T01:47:08.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.235+0000 7f0b3bfff640 1 -- 172.21.15.72:0/1991817475 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 25) ==== 137985+0+0 (secure 0 0 0) 0x7f0b34005000 con 0x7f0b44106a40 2024-03-22T01:47:08.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.236+0000 7f0b3bfff640 1 --2- 172.21.15.72:0/1991817475 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f0b300a3090 0x7f0b300a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:08.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.236+0000 7f0b42ffd640 1 --2- 172.21.15.72:0/1991817475 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f0b300a3090 0x7f0b300a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:08.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.236+0000 7f0b3bfff640 1 -- 172.21.15.72:0/1991817475 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f0b340cfe80 con 0x7f0b44106a40 2024-03-22T01:47:08.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.236+0000 7f0b42ffd640 1 --2- 172.21.15.72:0/1991817475 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f0b300a3090 0x7f0b300a5550 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f0b3c037040 tx=0x7f0b3c005d80 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:47:08.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.240+0000 7f0b3bfff640 1 -- 172.21.15.72:0/1991817475 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0b3409c5c0 con 0x7f0b44106a40 2024-03-22T01:47:08.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.418+0000 7f0b494d4640 1 -- 172.21.15.72:0/1991817475 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f0b44108fa0 con 0x7f0b44106a40 2024-03-22T01:47:08.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.418+0000 7f0b3bfff640 1 -- 172.21.15.72:0/1991817475 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v96) ==== 103+0+11 (secure 0 0 0) 0x7f0b3409bec0 con 0x7f0b44106a40 2024-03-22T01:47:08.417 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-03-22T01:47:08.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.420+0000 7f0b494d4640 1 -- 172.21.15.72:0/1991817475 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f0b300a3090 msgr2=0x7f0b300a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:08.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.420+0000 7f0b494d4640 1 --2- 172.21.15.72:0/1991817475 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f0b300a3090 0x7f0b300a5550 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f0b3c037040 tx=0x7f0b3c005d80 comp rx=0 tx=0).stop 2024-03-22T01:47:08.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.420+0000 7f0b494d4640 1 -- 172.21.15.72:0/1991817475 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0b44106a40 msgr2=0x7f0b441a4340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:08.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.420+0000 7f0b494d4640 1 --2- 172.21.15.72:0/1991817475 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0b44106a40 0x7f0b441a4340 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f0b3400e970 tx=0x7f0b3400ee30 comp rx=0 tx=0).stop 2024-03-22T01:47:08.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.420+0000 7f0b494d4640 1 -- 172.21.15.72:0/1991817475 shutdown_connections 2024-03-22T01:47:08.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.420+0000 7f0b494d4640 1 --2- 172.21.15.72:0/1991817475 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f0b300a3090 0x7f0b300a5550 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:08.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.420+0000 7f0b494d4640 1 --2- 172.21.15.72:0/1991817475 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b441013f0 0x7f0b44199800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:08.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.420+0000 7f0b494d4640 1 --2- 172.21.15.72:0/1991817475 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0b44106a40 0x7f0b441a4340 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:08.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.420+0000 7f0b494d4640 1 --2- 172.21.15.72:0/1991817475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b44100a30 0x7f0b441992c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:08.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.420+0000 7f0b494d4640 1 -- 172.21.15.72:0/1991817475 >> 172.21.15.72:0/1991817475 conn(0x7f0b44077c40 msgr2=0x7f0b44103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:47:08.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.420+0000 7f0b494d4640 1 -- 172.21.15.72:0/1991817475 shutdown_connections 2024-03-22T01:47:08.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.420+0000 7f0b494d4640 1 -- 172.21.15.72:0/1991817475 wait complete. 2024-03-22T01:47:08.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:08 smithi098 ceph-mon[36468]: pgmap v317: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 20 op/s 2024-03-22T01:47:08.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:08 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/3888304572' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:47:08.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:08 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/3494585581' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T01:47:08.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:08 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/665360941' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T01:47:08.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:08 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/1492914133' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T01:47:08.429 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:08 smithi072 ceph-mon[32414]: pgmap v317: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 20 op/s 2024-03-22T01:47:08.429 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:08 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/3888304572' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:47:08.429 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:08 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/3494585581' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T01:47:08.429 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:08 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/665360941' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T01:47:08.429 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:08 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/1492914133' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T01:47:08.430 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-03-22T01:47:08.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.532+0000 7fd4ae7a0640 1 -- 172.21.15.72:0/804798303 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd4a810c020 msgr2=0x7fd4a810e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:08.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.533+0000 7fd4ae7a0640 1 --2- 172.21.15.72:0/804798303 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd4a810c020 0x7fd4a810e4a0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fd4a0009900 tx=0x7fd4a002bcd0 comp rx=0 tx=0).stop 2024-03-22T01:47:08.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.533+0000 7fd4ae7a0640 1 -- 172.21.15.72:0/804798303 shutdown_connections 2024-03-22T01:47:08.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.533+0000 7fd4ae7a0640 1 --2- 172.21.15.72:0/804798303 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd4a8103570 0x7fd4a810ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:08.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.533+0000 7fd4ae7a0640 1 --2- 172.21.15.72:0/804798303 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd4a810c020 0x7fd4a810e4a0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:08.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.533+0000 7fd4ae7a0640 1 --2- 172.21.15.72:0/804798303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4a8102c50 0x7fd4a8103030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:08.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.533+0000 7fd4ae7a0640 1 -- 172.21.15.72:0/804798303 >> 172.21.15.72:0/804798303 conn(0x7fd4a80fc3f0 msgr2=0x7fd4a80fe810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:47:08.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.533+0000 7fd4ae7a0640 1 -- 172.21.15.72:0/804798303 shutdown_connections 2024-03-22T01:47:08.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.533+0000 7fd4ae7a0640 1 -- 172.21.15.72:0/804798303 wait complete. 2024-03-22T01:47:08.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.533+0000 7fd4ae7a0640 1 Processor -- start 2024-03-22T01:47:08.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.533+0000 7fd4ae7a0640 1 -- start start 2024-03-22T01:47:08.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.533+0000 7fd4ae7a0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd4a8102c50 0x7fd4a8198340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:08.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.534+0000 7fd4ae7a0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4a8103570 0x7fd4a8198880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:08.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.534+0000 7fd4ae7a0640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd4a810c020 0x7fd4a819cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:08.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.534+0000 7fd4ae7a0640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fd4a8111e00 con 0x7fd4a810c020 2024-03-22T01:47:08.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.534+0000 7fd4ae7a0640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd4a8111c80 con 0x7fd4a8102c50 2024-03-22T01:47:08.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.534+0000 7fd4ae7a0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd4a8111f80 con 0x7fd4a8103570 2024-03-22T01:47:08.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.534+0000 7fd4adf9f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd4a810c020 0x7fd4a819cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:08.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.534+0000 7fd4adf9f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd4a810c020 0x7fd4a819cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59454/0 (socket says 172.21.15.72:59454) 2024-03-22T01:47:08.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.534+0000 7fd4adf9f640 1 -- 172.21.15.72:0/1227938334 learned_addr learned my addr 172.21.15.72:0/1227938334 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:47:08.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.534+0000 7fd4ad79e640 1 --2- 172.21.15.72:0/1227938334 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd4a8102c50 0x7fd4a8198340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:08.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.534+0000 7fd4acf9d640 1 --2- 172.21.15.72:0/1227938334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4a8103570 0x7fd4a8198880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:08.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.534+0000 7fd4adf9f640 1 -- 172.21.15.72:0/1227938334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4a8103570 msgr2=0x7fd4a8198880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:08.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.534+0000 7fd4adf9f640 1 --2- 172.21.15.72:0/1227938334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4a8103570 0x7fd4a8198880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:08.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.534+0000 7fd4adf9f640 1 -- 172.21.15.72:0/1227938334 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd4a8102c50 msgr2=0x7fd4a8198340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:08.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.534+0000 7fd4adf9f640 1 --2- 172.21.15.72:0/1227938334 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd4a8102c50 0x7fd4a8198340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:08.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.534+0000 7fd4adf9f640 1 -- 172.21.15.72:0/1227938334 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4a819d2e0 con 0x7fd4a810c020 2024-03-22T01:47:08.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.534+0000 7fd4acf9d640 1 --2- 172.21.15.72:0/1227938334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4a8103570 0x7fd4a8198880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T01:47:08.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.534+0000 7fd4adf9f640 1 --2- 172.21.15.72:0/1227938334 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd4a810c020 0x7fd4a819cbb0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fd4a0005940 tx=0x7fd4a002cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:47:08.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.534+0000 7fd49e7fc640 1 -- 172.21.15.72:0/1227938334 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4a0041070 con 0x7fd4a810c020 2024-03-22T01:47:08.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.534+0000 7fd49e7fc640 1 -- 172.21.15.72:0/1227938334 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd4a0004450 con 0x7fd4a810c020 2024-03-22T01:47:08.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.534+0000 7fd49e7fc640 1 -- 172.21.15.72:0/1227938334 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4a0003db0 con 0x7fd4a810c020 2024-03-22T01:47:08.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.534+0000 7fd4ae7a0640 1 -- 172.21.15.72:0/1227938334 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4a819d570 con 0x7fd4a810c020 2024-03-22T01:47:08.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.534+0000 7fd4ae7a0640 1 -- 172.21.15.72:0/1227938334 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4a81a4e60 con 0x7fd4a810c020 2024-03-22T01:47:08.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.535+0000 7fd4ae7a0640 1 -- 172.21.15.72:0/1227938334 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4a804f960 con 0x7fd4a810c020 2024-03-22T01:47:08.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.536+0000 7fd49e7fc640 1 -- 172.21.15.72:0/1227938334 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 25) ==== 137985+0+0 (secure 0 0 0) 0x7fd4a0044070 con 0x7fd4a810c020 2024-03-22T01:47:08.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.537+0000 7fd49e7fc640 1 --2- 172.21.15.72:0/1227938334 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fd4940a3100 0x7fd4940a55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:08.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.537+0000 7fd4ad79e640 1 --2- 172.21.15.72:0/1227938334 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fd4940a3100 0x7fd4940a55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:08.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.537+0000 7fd49e7fc640 1 -- 172.21.15.72:0/1227938334 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fd4a00f16b0 con 0x7fd4a810c020 2024-03-22T01:47:08.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.537+0000 7fd4ad79e640 1 --2- 172.21.15.72:0/1227938334 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fd4940a3100 0x7fd4940a55c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd4a4005fd0 tx=0x7fd4a4009680 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:47:08.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.541+0000 7fd49e7fc640 1 -- 172.21.15.72:0/1227938334 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd4a00bdf00 con 0x7fd4a810c020 2024-03-22T01:47:08.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:08 smithi089 ceph-mon[35770]: pgmap v317: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 20 op/s 2024-03-22T01:47:08.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:08 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/3888304572' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:47:08.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:08 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/3494585581' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T01:47:08.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:08 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/665360941' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T01:47:08.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:08 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/1492914133' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T01:47:08.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.764+0000 7fd4ae7a0640 1 -- 172.21.15.72:0/1227938334 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7fd4a8107110 con 0x7fd4a810c020 2024-03-22T01:47:08.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.772+0000 7fd49e7fc640 1 -- 172.21.15.72:0/1227938334 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v25) ==== 81+0+742375 (secure 0 0 0) 0x7fd4a00bd800 con 0x7fd4a810c020 2024-03-22T01:47:08.783 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:47:08.783 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-22T01:47:08.783 INFO:teuthology.orchestra.run.smithi072.stdout: "epoch": 25, 2024-03-22T01:47:08.783 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.783 INFO:teuthology.orchestra.run.smithi072.stdout: "active_gid": 15381, 2024-03-22T01:47:08.784 INFO:teuthology.orchestra.run.smithi072.stdout: "active_name": "x", 2024-03-22T01:47:08.784 INFO:teuthology.orchestra.run.smithi072.stdout: "active_addrs": { 2024-03-22T01:47:08.784 INFO:teuthology.orchestra.run.smithi072.stdout: "addrvec": [ 2024-03-22T01:47:08.784 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:08.784 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "v2", 2024-03-22T01:47:08.784 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": "172.21.15.72:6800", 2024-03-22T01:47:08.784 INFO:teuthology.orchestra.run.smithi072.stdout: "nonce": 1379139045 2024-03-22T01:47:08.784 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.784 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:08.784 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "v1", 2024-03-22T01:47:08.784 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": "172.21.15.72:6801", 2024-03-22T01:47:08.784 INFO:teuthology.orchestra.run.smithi072.stdout: "nonce": 1379139045 2024-03-22T01:47:08.784 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:08.784 INFO:teuthology.orchestra.run.smithi072.stdout: ] 2024-03-22T01:47:08.785 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.785 INFO:teuthology.orchestra.run.smithi072.stdout: "active_addr": "172.21.15.72:6801/1379139045", 2024-03-22T01:47:08.785 INFO:teuthology.orchestra.run.smithi072.stdout: "active_change": "2024-03-22T01:36:38.184045+0000", 2024-03-22T01:47:08.785 INFO:teuthology.orchestra.run.smithi072.stdout: "active_mgr_features": 4540701547738038271, 2024-03-22T01:47:08.785 INFO:teuthology.orchestra.run.smithi072.stdout: "available": true, 2024-03-22T01:47:08.785 INFO:teuthology.orchestra.run.smithi072.stdout: "standbys": [ 2024-03-22T01:47:08.785 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:08.785 INFO:teuthology.orchestra.run.smithi072.stdout: "gid": 15369, 2024-03-22T01:47:08.785 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "y", 2024-03-22T01:47:08.785 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr_features": 4540701547738038271, 2024-03-22T01:47:08.785 INFO:teuthology.orchestra.run.smithi072.stdout: "available_modules": [ 2024-03-22T01:47:08.785 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:08.785 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "alerts", 2024-03-22T01:47:08.785 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:08.786 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:08.786 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:08.786 INFO:teuthology.orchestra.run.smithi072.stdout: "interval": { 2024-03-22T01:47:08.786 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "interval", 2024-03-22T01:47:08.786 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "secs", 2024-03-22T01:47:08.786 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.786 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.786 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "60", 2024-03-22T01:47:08.786 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.786 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.786 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.786 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "How frequently to reexamine health status", 2024-03-22T01:47:08.786 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.787 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.787 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.787 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.787 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:08.787 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:08.787 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.787 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.787 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.787 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:08.787 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.787 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.787 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:08.787 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:08.787 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:08.787 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:08.788 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:08.788 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:08.788 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:08.788 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:08.788 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.788 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.788 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.788 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.788 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.788 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:08.788 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:08.788 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.788 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.789 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.789 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:08.789 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.789 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.789 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.789 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.789 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.789 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.789 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.789 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.789 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:08.789 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:08.789 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.789 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.789 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.790 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:08.790 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.790 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.790 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:08.790 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:08.790 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:08.790 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:08.790 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:08.790 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:08.790 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:08.790 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:08.790 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.790 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.790 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.791 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.791 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.791 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:08.791 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:08.791 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.791 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.791 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.791 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:08.791 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.791 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.791 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.791 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.791 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.791 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.792 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.792 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.792 INFO:teuthology.orchestra.run.smithi072.stdout: "smtp_destination": { 2024-03-22T01:47:08.792 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "smtp_destination", 2024-03-22T01:47:08.792 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.792 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.792 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.792 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:08.792 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.792 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.792 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.792 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Email address to send alerts to", 2024-03-22T01:47:08.792 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.792 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.793 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.793 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.793 INFO:teuthology.orchestra.run.smithi072.stdout: "smtp_from_name": { 2024-03-22T01:47:08.793 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "smtp_from_name", 2024-03-22T01:47:08.793 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.793 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.793 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.793 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "Ceph", 2024-03-22T01:47:08.793 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.793 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.793 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.793 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Email From: name", 2024-03-22T01:47:08.793 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.793 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.794 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.794 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.794 INFO:teuthology.orchestra.run.smithi072.stdout: "smtp_host": { 2024-03-22T01:47:08.794 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "smtp_host", 2024-03-22T01:47:08.794 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.794 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.794 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.794 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:08.794 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.794 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.794 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.794 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "SMTP server", 2024-03-22T01:47:08.794 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.795 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.795 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.795 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.795 INFO:teuthology.orchestra.run.smithi072.stdout: "smtp_password": { 2024-03-22T01:47:08.795 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "smtp_password", 2024-03-22T01:47:08.795 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.795 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.795 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.795 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:08.795 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.795 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.795 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.795 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Password to authenticate with", 2024-03-22T01:47:08.795 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.795 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.796 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.796 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.796 INFO:teuthology.orchestra.run.smithi072.stdout: "smtp_port": { 2024-03-22T01:47:08.796 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "smtp_port", 2024-03-22T01:47:08.796 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:08.796 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.796 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.796 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "465", 2024-03-22T01:47:08.796 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.796 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.796 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.796 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "SMTP port", 2024-03-22T01:47:08.796 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.796 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.796 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.797 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.797 INFO:teuthology.orchestra.run.smithi072.stdout: "smtp_sender": { 2024-03-22T01:47:08.797 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "smtp_sender", 2024-03-22T01:47:08.797 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.797 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.797 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.797 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:08.797 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.797 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.797 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.797 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "SMTP envelope sender", 2024-03-22T01:47:08.797 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.797 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.798 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.798 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.798 INFO:teuthology.orchestra.run.smithi072.stdout: "smtp_ssl": { 2024-03-22T01:47:08.798 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "smtp_ssl", 2024-03-22T01:47:08.798 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.798 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.798 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.798 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:08.798 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.798 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.798 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.798 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-22T01:47:08.799 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.799 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.799 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.799 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.799 INFO:teuthology.orchestra.run.smithi072.stdout: "smtp_user": { 2024-03-22T01:47:08.799 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "smtp_user", 2024-03-22T01:47:08.799 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.799 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.799 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.799 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:08.799 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.799 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.799 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.800 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "User to authenticate as", 2024-03-22T01:47:08.800 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.800 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.800 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.800 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:08.800 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:08.800 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.800 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:08.800 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "balancer", 2024-03-22T01:47:08.800 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:08.800 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:08.800 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:08.800 INFO:teuthology.orchestra.run.smithi072.stdout: "active": { 2024-03-22T01:47:08.800 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "active", 2024-03-22T01:47:08.801 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.801 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.801 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.801 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:08.801 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.801 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.801 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.801 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "automatically balance PGs across cluster", 2024-03-22T01:47:08.801 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.801 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.801 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.801 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.801 INFO:teuthology.orchestra.run.smithi072.stdout: "begin_time": { 2024-03-22T01:47:08.801 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "begin_time", 2024-03-22T01:47:08.801 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.802 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.802 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.802 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "0000", 2024-03-22T01:47:08.802 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.802 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.802 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.802 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "beginning time of day to automatically balance", 2024-03-22T01:47:08.802 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-22T01:47:08.802 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.802 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.802 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.802 INFO:teuthology.orchestra.run.smithi072.stdout: "begin_weekday": { 2024-03-22T01:47:08.802 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "begin_weekday", 2024-03-22T01:47:08.802 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "uint", 2024-03-22T01:47:08.803 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.803 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.803 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "0", 2024-03-22T01:47:08.803 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "0", 2024-03-22T01:47:08.803 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "6", 2024-03-22T01:47:08.803 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.803 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-22T01:47:08.803 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-22T01:47:08.803 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.803 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.803 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.803 INFO:teuthology.orchestra.run.smithi072.stdout: "crush_compat_max_iterations": { 2024-03-22T01:47:08.803 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "crush_compat_max_iterations", 2024-03-22T01:47:08.803 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "uint", 2024-03-22T01:47:08.803 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.804 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.804 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "25", 2024-03-22T01:47:08.804 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "1", 2024-03-22T01:47:08.804 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "250", 2024-03-22T01:47:08.804 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.804 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-22T01:47:08.804 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.804 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.804 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.804 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.804 INFO:teuthology.orchestra.run.smithi072.stdout: "crush_compat_metrics": { 2024-03-22T01:47:08.804 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "crush_compat_metrics", 2024-03-22T01:47:08.804 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.804 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.804 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.805 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "pgs,objects,bytes", 2024-03-22T01:47:08.805 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.805 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.805 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.805 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-22T01:47:08.805 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-22T01:47:08.805 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.805 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.805 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.805 INFO:teuthology.orchestra.run.smithi072.stdout: "crush_compat_step": { 2024-03-22T01:47:08.805 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "crush_compat_step", 2024-03-22T01:47:08.805 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "float", 2024-03-22T01:47:08.805 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.805 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.806 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "0.5", 2024-03-22T01:47:08.806 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "0.001", 2024-03-22T01:47:08.806 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "0.999", 2024-03-22T01:47:08.806 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.806 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "aggressiveness of optimization", 2024-03-22T01:47:08.806 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-22T01:47:08.806 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.806 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.806 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.806 INFO:teuthology.orchestra.run.smithi072.stdout: "end_time": { 2024-03-22T01:47:08.806 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "end_time", 2024-03-22T01:47:08.806 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.806 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.806 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.807 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "2359", 2024-03-22T01:47:08.807 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.807 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.807 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.807 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "ending time of day to automatically balance", 2024-03-22T01:47:08.807 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-22T01:47:08.807 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.807 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.807 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.807 INFO:teuthology.orchestra.run.smithi072.stdout: "end_weekday": { 2024-03-22T01:47:08.807 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "end_weekday", 2024-03-22T01:47:08.807 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "uint", 2024-03-22T01:47:08.807 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.807 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.807 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "0", 2024-03-22T01:47:08.808 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "0", 2024-03-22T01:47:08.808 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "6", 2024-03-22T01:47:08.808 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.808 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-22T01:47:08.808 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-22T01:47:08.808 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.808 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.808 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.808 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:08.808 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:08.808 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.808 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.808 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.808 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:08.808 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.809 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.809 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:08.809 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:08.809 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:08.809 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:08.809 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:08.809 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:08.809 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:08.809 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:08.809 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.809 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.809 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.809 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.809 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.810 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:08.810 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:08.810 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.810 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.810 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.810 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:08.810 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.810 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.810 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.810 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.810 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.811 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.811 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.811 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.811 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:08.811 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:08.811 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.811 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.811 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.811 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:08.811 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.811 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.811 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:08.811 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:08.811 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:08.812 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:08.812 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:08.812 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:08.812 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:08.812 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:08.812 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.812 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.812 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.812 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.812 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.812 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:08.812 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:08.812 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.812 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.813 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.813 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:08.813 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.813 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.813 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.813 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.813 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.813 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.813 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.813 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.813 INFO:teuthology.orchestra.run.smithi072.stdout: "min_score": { 2024-03-22T01:47:08.813 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "min_score", 2024-03-22T01:47:08.813 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "float", 2024-03-22T01:47:08.813 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.814 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.814 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "0", 2024-03-22T01:47:08.814 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.814 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.814 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.814 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-22T01:47:08.814 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.814 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.814 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.814 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.814 INFO:teuthology.orchestra.run.smithi072.stdout: "mode": { 2024-03-22T01:47:08.814 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "mode", 2024-03-22T01:47:08.814 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.814 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.815 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.815 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "upmap", 2024-03-22T01:47:08.815 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.815 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.815 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:08.815 INFO:teuthology.orchestra.run.smithi072.stdout: "crush-compat", 2024-03-22T01:47:08.815 INFO:teuthology.orchestra.run.smithi072.stdout: "none", 2024-03-22T01:47:08.815 INFO:teuthology.orchestra.run.smithi072.stdout: "read", 2024-03-22T01:47:08.815 INFO:teuthology.orchestra.run.smithi072.stdout: "upmap", 2024-03-22T01:47:08.815 INFO:teuthology.orchestra.run.smithi072.stdout: "upmap-read" 2024-03-22T01:47:08.815 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:08.815 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Balancer mode", 2024-03-22T01:47:08.815 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.815 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.815 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.816 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.816 INFO:teuthology.orchestra.run.smithi072.stdout: "pool_ids": { 2024-03-22T01:47:08.816 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "pool_ids", 2024-03-22T01:47:08.816 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.816 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.816 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.816 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:08.816 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.816 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.816 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.816 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-22T01:47:08.817 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.817 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.817 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.817 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.817 INFO:teuthology.orchestra.run.smithi072.stdout: "sleep_interval": { 2024-03-22T01:47:08.817 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "sleep_interval", 2024-03-22T01:47:08.817 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "secs", 2024-03-22T01:47:08.817 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.817 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.817 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "60", 2024-03-22T01:47:08.817 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.817 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.817 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.817 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-22T01:47:08.817 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.818 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.818 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.818 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.818 INFO:teuthology.orchestra.run.smithi072.stdout: "upmap_max_deviation": { 2024-03-22T01:47:08.818 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "upmap_max_deviation", 2024-03-22T01:47:08.818 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:08.818 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.818 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.818 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "5", 2024-03-22T01:47:08.818 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "1", 2024-03-22T01:47:08.818 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.818 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.818 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-22T01:47:08.818 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-22T01:47:08.818 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.819 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.819 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.819 INFO:teuthology.orchestra.run.smithi072.stdout: "upmap_max_optimizations": { 2024-03-22T01:47:08.819 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "upmap_max_optimizations", 2024-03-22T01:47:08.819 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "uint", 2024-03-22T01:47:08.819 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.819 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.819 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "10", 2024-03-22T01:47:08.819 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.819 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.819 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.819 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-22T01:47:08.819 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.819 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.819 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.820 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:08.820 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:08.820 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.820 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:08.820 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "cephadm", 2024-03-22T01:47:08.820 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:08.820 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:08.820 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:08.820 INFO:teuthology.orchestra.run.smithi072.stdout: "agent_down_multiplier": { 2024-03-22T01:47:08.820 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "agent_down_multiplier", 2024-03-22T01:47:08.820 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "float", 2024-03-22T01:47:08.820 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.820 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.820 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "3.0", 2024-03-22T01:47:08.820 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.821 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.821 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.821 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-22T01:47:08.821 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.821 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.821 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.821 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.821 INFO:teuthology.orchestra.run.smithi072.stdout: "agent_refresh_rate": { 2024-03-22T01:47:08.821 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "agent_refresh_rate", 2024-03-22T01:47:08.821 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "secs", 2024-03-22T01:47:08.821 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.821 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.821 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "20", 2024-03-22T01:47:08.821 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.821 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.822 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.822 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-22T01:47:08.822 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.822 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.822 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.822 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.822 INFO:teuthology.orchestra.run.smithi072.stdout: "agent_starting_port": { 2024-03-22T01:47:08.822 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "agent_starting_port", 2024-03-22T01:47:08.822 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:08.822 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.822 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.822 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "4721", 2024-03-22T01:47:08.822 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.822 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.822 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.823 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-22T01:47:08.823 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.823 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.823 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.823 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.823 INFO:teuthology.orchestra.run.smithi072.stdout: "allow_ptrace": { 2024-03-22T01:47:08.823 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "allow_ptrace", 2024-03-22T01:47:08.823 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.823 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.823 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.823 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:08.824 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.824 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.824 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.824 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-22T01:47:08.824 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-22T01:47:08.824 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.824 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.824 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.824 INFO:teuthology.orchestra.run.smithi072.stdout: "autotune_interval": { 2024-03-22T01:47:08.824 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "autotune_interval", 2024-03-22T01:47:08.824 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "secs", 2024-03-22T01:47:08.824 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.824 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.824 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "600", 2024-03-22T01:47:08.824 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.825 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.825 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.825 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-22T01:47:08.825 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.825 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.825 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.825 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.825 INFO:teuthology.orchestra.run.smithi072.stdout: "autotune_memory_target_ratio": { 2024-03-22T01:47:08.825 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "autotune_memory_target_ratio", 2024-03-22T01:47:08.825 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "float", 2024-03-22T01:47:08.825 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.825 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.825 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "0.7", 2024-03-22T01:47:08.825 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.825 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.826 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.826 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-22T01:47:08.826 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.826 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.826 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.826 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.826 INFO:teuthology.orchestra.run.smithi072.stdout: "cephadm_log_destination": { 2024-03-22T01:47:08.826 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "cephadm_log_destination", 2024-03-22T01:47:08.826 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.826 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.826 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.826 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:08.826 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.826 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.827 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:08.827 INFO:teuthology.orchestra.run.smithi072.stdout: "file", 2024-03-22T01:47:08.827 INFO:teuthology.orchestra.run.smithi072.stdout: "file,syslog", 2024-03-22T01:47:08.827 INFO:teuthology.orchestra.run.smithi072.stdout: "syslog" 2024-03-22T01:47:08.827 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:08.827 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-22T01:47:08.827 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.827 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.827 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.827 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.827 INFO:teuthology.orchestra.run.smithi072.stdout: "cgroups_split": { 2024-03-22T01:47:08.827 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "cgroups_split", 2024-03-22T01:47:08.827 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.827 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.827 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.828 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:08.828 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.828 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.828 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.828 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-22T01:47:08.828 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.828 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.828 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.828 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.828 INFO:teuthology.orchestra.run.smithi072.stdout: "config_checks_enabled": { 2024-03-22T01:47:08.828 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "config_checks_enabled", 2024-03-22T01:47:08.828 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.828 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.828 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.828 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:08.829 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.829 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.829 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.829 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-22T01:47:08.829 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.829 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.829 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.829 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.829 INFO:teuthology.orchestra.run.smithi072.stdout: "config_dashboard": { 2024-03-22T01:47:08.829 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "config_dashboard", 2024-03-22T01:47:08.829 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.829 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.829 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.829 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:08.830 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.830 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.830 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.830 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-22T01:47:08.830 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.830 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.830 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.830 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.830 INFO:teuthology.orchestra.run.smithi072.stdout: "container_image_alertmanager": { 2024-03-22T01:47:08.830 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "container_image_alertmanager", 2024-03-22T01:47:08.830 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.830 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.830 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.830 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-22T01:47:08.830 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.831 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.831 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.831 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Prometheus container image", 2024-03-22T01:47:08.831 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.831 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.831 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.831 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.831 INFO:teuthology.orchestra.run.smithi072.stdout: "container_image_base": { 2024-03-22T01:47:08.831 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "container_image_base", 2024-03-22T01:47:08.831 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.831 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.831 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.831 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-22T01:47:08.831 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.831 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.832 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.832 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Container image name, without the tag", 2024-03-22T01:47:08.832 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.832 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.832 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.832 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.832 INFO:teuthology.orchestra.run.smithi072.stdout: "container_image_elasticsearch": { 2024-03-22T01:47:08.832 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "container_image_elasticsearch", 2024-03-22T01:47:08.832 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.832 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.832 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.832 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-22T01:47:08.832 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.832 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.832 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.833 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "elasticsearch container image", 2024-03-22T01:47:08.833 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.833 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.833 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.833 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.833 INFO:teuthology.orchestra.run.smithi072.stdout: "container_image_grafana": { 2024-03-22T01:47:08.833 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "container_image_grafana", 2024-03-22T01:47:08.833 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.833 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.833 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.833 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-03-22T01:47:08.833 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.833 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.833 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.833 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Prometheus container image", 2024-03-22T01:47:08.834 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.834 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.834 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.834 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.834 INFO:teuthology.orchestra.run.smithi072.stdout: "container_image_haproxy": { 2024-03-22T01:47:08.834 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "container_image_haproxy", 2024-03-22T01:47:08.834 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.834 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.834 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.834 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-22T01:47:08.834 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.834 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.835 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.835 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "HAproxy container image", 2024-03-22T01:47:08.835 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.835 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.835 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.835 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.835 INFO:teuthology.orchestra.run.smithi072.stdout: "container_image_jaeger_agent": { 2024-03-22T01:47:08.835 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "container_image_jaeger_agent", 2024-03-22T01:47:08.835 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.835 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.835 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.835 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-22T01:47:08.835 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.835 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.835 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.836 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Jaeger agent container image", 2024-03-22T01:47:08.836 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.836 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.836 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.836 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.836 INFO:teuthology.orchestra.run.smithi072.stdout: "container_image_jaeger_collector": { 2024-03-22T01:47:08.836 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "container_image_jaeger_collector", 2024-03-22T01:47:08.836 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.836 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.836 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.836 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-22T01:47:08.837 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.837 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.837 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.837 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Jaeger collector container image", 2024-03-22T01:47:08.837 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.837 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.837 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.837 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.837 INFO:teuthology.orchestra.run.smithi072.stdout: "container_image_jaeger_query": { 2024-03-22T01:47:08.837 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "container_image_jaeger_query", 2024-03-22T01:47:08.837 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.837 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.837 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.838 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-22T01:47:08.838 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.838 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.838 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.838 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Jaeger query container image", 2024-03-22T01:47:08.838 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.838 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.838 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.838 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.838 INFO:teuthology.orchestra.run.smithi072.stdout: "container_image_keepalived": { 2024-03-22T01:47:08.838 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "container_image_keepalived", 2024-03-22T01:47:08.838 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.839 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.839 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.839 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-22T01:47:08.839 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.839 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.839 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.839 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Keepalived container image", 2024-03-22T01:47:08.839 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.839 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.839 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.839 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.839 INFO:teuthology.orchestra.run.smithi072.stdout: "container_image_loki": { 2024-03-22T01:47:08.839 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "container_image_loki", 2024-03-22T01:47:08.839 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.840 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.840 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.840 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-22T01:47:08.840 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.840 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.840 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.840 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Loki container image", 2024-03-22T01:47:08.840 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.840 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.840 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.840 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.840 INFO:teuthology.orchestra.run.smithi072.stdout: "container_image_node_exporter": { 2024-03-22T01:47:08.840 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "container_image_node_exporter", 2024-03-22T01:47:08.841 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.841 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.841 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.841 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-22T01:47:08.841 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.841 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.841 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.841 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Prometheus container image", 2024-03-22T01:47:08.841 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.841 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.842 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.842 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.842 INFO:teuthology.orchestra.run.smithi072.stdout: "container_image_nvmeof": { 2024-03-22T01:47:08.842 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "container_image_nvmeof", 2024-03-22T01:47:08.842 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.842 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.842 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.842 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-22T01:47:08.842 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.842 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.842 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.842 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Nvme-of container image", 2024-03-22T01:47:08.843 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.843 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.843 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.843 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.843 INFO:teuthology.orchestra.run.smithi072.stdout: "container_image_prometheus": { 2024-03-22T01:47:08.843 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "container_image_prometheus", 2024-03-22T01:47:08.843 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.843 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.843 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.843 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-22T01:47:08.843 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.843 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.843 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.843 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Prometheus container image", 2024-03-22T01:47:08.843 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.844 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.844 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.844 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.844 INFO:teuthology.orchestra.run.smithi072.stdout: "container_image_promtail": { 2024-03-22T01:47:08.844 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "container_image_promtail", 2024-03-22T01:47:08.844 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.844 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.844 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.844 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-22T01:47:08.844 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.844 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.844 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.844 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Promtail container image", 2024-03-22T01:47:08.844 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.845 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.845 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.845 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.845 INFO:teuthology.orchestra.run.smithi072.stdout: "container_image_snmp_gateway": { 2024-03-22T01:47:08.845 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "container_image_snmp_gateway", 2024-03-22T01:47:08.845 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.845 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.845 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.845 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-22T01:47:08.845 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.845 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.845 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.845 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "SNMP Gateway container image", 2024-03-22T01:47:08.845 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.845 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.846 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.846 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.846 INFO:teuthology.orchestra.run.smithi072.stdout: "container_init": { 2024-03-22T01:47:08.846 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "container_init", 2024-03-22T01:47:08.846 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.846 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.846 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.846 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:08.846 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.846 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.846 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.846 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Run podman/docker with `--init`", 2024-03-22T01:47:08.846 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.846 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.846 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.847 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.847 INFO:teuthology.orchestra.run.smithi072.stdout: "daemon_cache_timeout": { 2024-03-22T01:47:08.847 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "daemon_cache_timeout", 2024-03-22T01:47:08.847 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "secs", 2024-03-22T01:47:08.847 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.847 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.847 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "600", 2024-03-22T01:47:08.847 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.847 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.847 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.847 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-22T01:47:08.847 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.848 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.848 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.848 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.848 INFO:teuthology.orchestra.run.smithi072.stdout: "default_cephadm_command_timeout": { 2024-03-22T01:47:08.848 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "default_cephadm_command_timeout", 2024-03-22T01:47:08.848 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:08.848 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.848 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.848 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "900", 2024-03-22T01:47:08.848 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.848 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.848 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.848 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-22T01:47:08.848 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.849 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.849 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.849 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.849 INFO:teuthology.orchestra.run.smithi072.stdout: "default_registry": { 2024-03-22T01:47:08.849 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "default_registry", 2024-03-22T01:47:08.849 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.849 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.849 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.849 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "docker.io", 2024-03-22T01:47:08.849 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.849 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.849 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.849 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-22T01:47:08.849 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.849 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.850 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.850 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.850 INFO:teuthology.orchestra.run.smithi072.stdout: "device_cache_timeout": { 2024-03-22T01:47:08.850 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "device_cache_timeout", 2024-03-22T01:47:08.850 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "secs", 2024-03-22T01:47:08.850 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.850 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.850 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "1800", 2024-03-22T01:47:08.850 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.850 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.850 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.850 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "seconds to cache device inventory", 2024-03-22T01:47:08.850 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.850 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.851 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.851 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.851 INFO:teuthology.orchestra.run.smithi072.stdout: "device_enhanced_scan": { 2024-03-22T01:47:08.851 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "device_enhanced_scan", 2024-03-22T01:47:08.851 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.851 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.851 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.851 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:08.851 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.851 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.851 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.851 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-22T01:47:08.851 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.851 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.852 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.852 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.852 INFO:teuthology.orchestra.run.smithi072.stdout: "facts_cache_timeout": { 2024-03-22T01:47:08.852 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "facts_cache_timeout", 2024-03-22T01:47:08.852 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "secs", 2024-03-22T01:47:08.852 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.852 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.852 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "60", 2024-03-22T01:47:08.852 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.852 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.852 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.852 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "seconds to cache host facts data", 2024-03-22T01:47:08.852 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.852 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.853 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.853 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.853 INFO:teuthology.orchestra.run.smithi072.stdout: "grafana_dashboards_path": { 2024-03-22T01:47:08.853 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "grafana_dashboards_path", 2024-03-22T01:47:08.853 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.853 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.853 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.853 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-03-22T01:47:08.853 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.853 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.853 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.853 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-03-22T01:47:08.853 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.854 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.854 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.854 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.854 INFO:teuthology.orchestra.run.smithi072.stdout: "host_check_interval": { 2024-03-22T01:47:08.854 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "host_check_interval", 2024-03-22T01:47:08.854 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "secs", 2024-03-22T01:47:08.854 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.854 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.854 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "600", 2024-03-22T01:47:08.854 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.854 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.854 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.854 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "how frequently to perform a host check", 2024-03-22T01:47:08.854 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.855 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.855 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.855 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.855 INFO:teuthology.orchestra.run.smithi072.stdout: "hw_monitoring": { 2024-03-22T01:47:08.855 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "hw_monitoring", 2024-03-22T01:47:08.855 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.855 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.855 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.855 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:08.855 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.855 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.855 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.855 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-22T01:47:08.856 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.856 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.856 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.856 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.856 INFO:teuthology.orchestra.run.smithi072.stdout: "inventory_list_all": { 2024-03-22T01:47:08.856 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "inventory_list_all", 2024-03-22T01:47:08.856 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.856 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.856 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.856 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:08.856 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.856 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.856 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.856 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-22T01:47:08.856 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.857 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.857 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.857 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.857 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:08.857 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:08.857 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.857 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.857 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.857 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:08.857 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.857 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.857 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:08.857 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:08.857 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:08.858 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:08.858 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:08.858 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:08.858 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:08.858 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:08.858 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.858 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.858 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.858 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.858 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.858 INFO:teuthology.orchestra.run.smithi072.stdout: "log_refresh_metadata": { 2024-03-22T01:47:08.858 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_refresh_metadata", 2024-03-22T01:47:08.858 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.858 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.859 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.859 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:08.859 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.859 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.859 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.859 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-22T01:47:08.859 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.859 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.859 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.859 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.859 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:08.859 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:08.859 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.860 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.860 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.860 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:08.860 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.860 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.860 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.860 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-22T01:47:08.860 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.860 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.860 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.860 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.860 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:08.860 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:08.861 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.861 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.861 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.861 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:08.861 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.861 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.861 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:08.861 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:08.861 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:08.861 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:08.861 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:08.861 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:08.861 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:08.861 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:08.862 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.862 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.862 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.862 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.862 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.862 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:08.862 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:08.862 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.862 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.862 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.862 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:08.863 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.863 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.863 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.863 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.863 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.863 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.863 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.863 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.863 INFO:teuthology.orchestra.run.smithi072.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-22T01:47:08.863 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-22T01:47:08.863 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.863 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.863 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.864 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:08.864 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.864 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.864 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.864 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-22T01:47:08.864 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.864 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.864 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.864 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.864 INFO:teuthology.orchestra.run.smithi072.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-22T01:47:08.864 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-22T01:47:08.864 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.864 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.865 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.865 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "*", 2024-03-22T01:47:08.865 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.865 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.865 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.865 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-22T01:47:08.865 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.865 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.865 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.865 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.865 INFO:teuthology.orchestra.run.smithi072.stdout: "max_count_per_host": { 2024-03-22T01:47:08.865 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "max_count_per_host", 2024-03-22T01:47:08.865 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:08.865 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.866 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.866 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "10", 2024-03-22T01:47:08.866 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.866 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.866 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.866 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "max number of daemons per service per host", 2024-03-22T01:47:08.866 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.866 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.866 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.866 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.866 INFO:teuthology.orchestra.run.smithi072.stdout: "max_osd_draining_count": { 2024-03-22T01:47:08.866 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "max_osd_draining_count", 2024-03-22T01:47:08.866 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:08.867 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.867 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.867 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "10", 2024-03-22T01:47:08.867 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.867 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.867 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.867 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-22T01:47:08.867 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.867 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.867 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.867 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.867 INFO:teuthology.orchestra.run.smithi072.stdout: "migration_current": { 2024-03-22T01:47:08.867 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "migration_current", 2024-03-22T01:47:08.868 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:08.868 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.868 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.868 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:08.868 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.868 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.868 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.868 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "internal - do not modify", 2024-03-22T01:47:08.868 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.868 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.868 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.868 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.868 INFO:teuthology.orchestra.run.smithi072.stdout: "mode": { 2024-03-22T01:47:08.868 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "mode", 2024-03-22T01:47:08.868 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.869 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.869 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.869 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "root", 2024-03-22T01:47:08.869 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.869 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.869 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:08.869 INFO:teuthology.orchestra.run.smithi072.stdout: "cephadm-package", 2024-03-22T01:47:08.869 INFO:teuthology.orchestra.run.smithi072.stdout: "root" 2024-03-22T01:47:08.869 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:08.869 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "mode for remote execution of cephadm", 2024-03-22T01:47:08.869 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.869 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.869 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.869 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.869 INFO:teuthology.orchestra.run.smithi072.stdout: "oob_default_addr": { 2024-03-22T01:47:08.870 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "oob_default_addr", 2024-03-22T01:47:08.870 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.870 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.870 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.870 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "169.254.1.1", 2024-03-22T01:47:08.870 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.870 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.870 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.870 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-22T01:47:08.870 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.870 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.870 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.870 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.870 INFO:teuthology.orchestra.run.smithi072.stdout: "prometheus_alerts_path": { 2024-03-22T01:47:08.870 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "prometheus_alerts_path", 2024-03-22T01:47:08.871 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.871 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.871 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.871 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-22T01:47:08.871 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.871 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.871 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.871 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-22T01:47:08.871 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.871 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.871 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.871 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.871 INFO:teuthology.orchestra.run.smithi072.stdout: "registry_insecure": { 2024-03-22T01:47:08.872 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "registry_insecure", 2024-03-22T01:47:08.872 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.872 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.872 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.872 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:08.872 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.872 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.872 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.872 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-22T01:47:08.872 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.872 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.872 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.872 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.872 INFO:teuthology.orchestra.run.smithi072.stdout: "registry_password": { 2024-03-22T01:47:08.873 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "registry_password", 2024-03-22T01:47:08.873 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.873 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.873 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.873 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:08.873 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.873 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.873 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.873 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-22T01:47:08.873 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.873 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.873 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.873 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.874 INFO:teuthology.orchestra.run.smithi072.stdout: "registry_url": { 2024-03-22T01:47:08.874 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "registry_url", 2024-03-22T01:47:08.874 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.874 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.874 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.874 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:08.874 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.874 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.874 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.874 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-22T01:47:08.874 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.874 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.874 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.874 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.875 INFO:teuthology.orchestra.run.smithi072.stdout: "registry_username": { 2024-03-22T01:47:08.875 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "registry_username", 2024-03-22T01:47:08.875 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.875 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.875 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.875 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:08.875 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.875 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.875 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.875 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-22T01:47:08.875 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.875 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.875 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.875 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.876 INFO:teuthology.orchestra.run.smithi072.stdout: "secure_monitoring_stack": { 2024-03-22T01:47:08.876 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "secure_monitoring_stack", 2024-03-22T01:47:08.876 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.876 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.876 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.876 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:08.876 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.876 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.876 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.876 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-22T01:47:08.876 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.876 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.876 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.876 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.877 INFO:teuthology.orchestra.run.smithi072.stdout: "service_discovery_port": { 2024-03-22T01:47:08.877 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "service_discovery_port", 2024-03-22T01:47:08.877 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:08.877 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.877 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.877 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "8765", 2024-03-22T01:47:08.877 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.877 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.877 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.877 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "cephadm service discovery port", 2024-03-22T01:47:08.877 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.877 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.877 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.877 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.877 INFO:teuthology.orchestra.run.smithi072.stdout: "ssh_config_file": { 2024-03-22T01:47:08.878 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "ssh_config_file", 2024-03-22T01:47:08.878 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.878 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.878 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.878 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:08.878 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.878 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.878 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.878 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-22T01:47:08.878 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.878 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.878 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.878 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.878 INFO:teuthology.orchestra.run.smithi072.stdout: "use_agent": { 2024-03-22T01:47:08.879 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "use_agent", 2024-03-22T01:47:08.879 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.879 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.879 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.879 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:08.879 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.879 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.879 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.879 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-22T01:47:08.879 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.879 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.879 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.879 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.879 INFO:teuthology.orchestra.run.smithi072.stdout: "use_repo_digest": { 2024-03-22T01:47:08.879 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "use_repo_digest", 2024-03-22T01:47:08.880 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.880 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.880 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.880 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:08.880 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.880 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.880 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.880 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-22T01:47:08.880 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.880 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.880 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.880 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.880 INFO:teuthology.orchestra.run.smithi072.stdout: "warn_on_failed_host_check": { 2024-03-22T01:47:08.880 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "warn_on_failed_host_check", 2024-03-22T01:47:08.881 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.881 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.881 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.881 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:08.881 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.881 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.881 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.881 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "raise a health warning if the host check fails", 2024-03-22T01:47:08.881 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.881 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.881 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.881 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.881 INFO:teuthology.orchestra.run.smithi072.stdout: "warn_on_stray_daemons": { 2024-03-22T01:47:08.881 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "warn_on_stray_daemons", 2024-03-22T01:47:08.882 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.882 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.882 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.882 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:08.882 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.882 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.882 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.882 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-22T01:47:08.882 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.882 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.882 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.882 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.882 INFO:teuthology.orchestra.run.smithi072.stdout: "warn_on_stray_hosts": { 2024-03-22T01:47:08.882 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "warn_on_stray_hosts", 2024-03-22T01:47:08.882 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.883 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.883 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.883 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:08.883 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.883 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.883 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.883 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-22T01:47:08.883 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.883 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.883 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.883 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:08.883 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:08.883 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.883 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:08.883 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "crash", 2024-03-22T01:47:08.884 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:08.884 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:08.884 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:08.884 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:08.884 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:08.884 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.884 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.884 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.884 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:08.884 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.884 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.884 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:08.884 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:08.884 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:08.884 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:08.885 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:08.885 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:08.885 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:08.885 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:08.885 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.885 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.885 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.885 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.885 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.885 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:08.885 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:08.885 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.885 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.886 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.886 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:08.886 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.886 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.886 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.886 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.886 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.886 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.886 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.886 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.886 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:08.886 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:08.886 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.886 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.887 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.887 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:08.887 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.887 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.887 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:08.887 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:08.887 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:08.887 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:08.887 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:08.887 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:08.887 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:08.887 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:08.887 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.887 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.887 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.888 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.888 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.888 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:08.888 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:08.888 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.888 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.888 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.888 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:08.888 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.888 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.888 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.888 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.888 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.888 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.889 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.889 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.889 INFO:teuthology.orchestra.run.smithi072.stdout: "retain_interval": { 2024-03-22T01:47:08.889 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "retain_interval", 2024-03-22T01:47:08.889 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "secs", 2024-03-22T01:47:08.889 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.889 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.889 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "31536000", 2024-03-22T01:47:08.889 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.889 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.889 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.889 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-22T01:47:08.889 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.889 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.889 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.890 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.890 INFO:teuthology.orchestra.run.smithi072.stdout: "warn_recent_interval": { 2024-03-22T01:47:08.890 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "warn_recent_interval", 2024-03-22T01:47:08.890 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "secs", 2024-03-22T01:47:08.890 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.890 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.890 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "1209600", 2024-03-22T01:47:08.890 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.890 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.890 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.890 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-22T01:47:08.890 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.890 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.890 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.891 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:08.891 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:08.891 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.891 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:08.891 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "dashboard", 2024-03-22T01:47:08.891 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:08.891 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:08.891 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:08.891 INFO:teuthology.orchestra.run.smithi072.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-22T01:47:08.891 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-22T01:47:08.891 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:08.891 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.891 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.891 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "10", 2024-03-22T01:47:08.891 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.892 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.892 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.892 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.892 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.892 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.892 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.892 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.892 INFO:teuthology.orchestra.run.smithi072.stdout: "ALERTMANAGER_API_HOST": { 2024-03-22T01:47:08.892 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-22T01:47:08.892 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.892 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.892 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.892 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:08.892 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.892 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.893 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.893 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.893 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.893 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.893 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.893 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.893 INFO:teuthology.orchestra.run.smithi072.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-22T01:47:08.893 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-22T01:47:08.893 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.893 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.893 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.893 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:08.893 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.893 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.893 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.894 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.894 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.894 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.894 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.894 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.894 INFO:teuthology.orchestra.run.smithi072.stdout: "AUDIT_API_ENABLED": { 2024-03-22T01:47:08.894 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "AUDIT_API_ENABLED", 2024-03-22T01:47:08.894 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.894 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.894 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.894 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:08.894 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.894 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.894 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.895 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.895 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.895 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.895 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.895 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.895 INFO:teuthology.orchestra.run.smithi072.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-22T01:47:08.895 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-22T01:47:08.895 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.895 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.895 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.895 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:08.895 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.895 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.895 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.895 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.896 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.896 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.896 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.896 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.896 INFO:teuthology.orchestra.run.smithi072.stdout: "ENABLE_BROWSABLE_API": { 2024-03-22T01:47:08.896 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-22T01:47:08.896 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.896 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.896 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.896 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:08.896 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.896 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.896 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.896 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.896 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.897 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.897 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.897 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.897 INFO:teuthology.orchestra.run.smithi072.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-22T01:47:08.897 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-22T01:47:08.897 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.897 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.897 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.897 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:08.897 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.897 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.897 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.897 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.897 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.898 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.898 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.898 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.898 INFO:teuthology.orchestra.run.smithi072.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-22T01:47:08.898 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-22T01:47:08.898 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.898 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.898 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.898 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:08.898 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.898 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.898 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.898 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.899 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.899 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.899 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.899 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.899 INFO:teuthology.orchestra.run.smithi072.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-22T01:47:08.899 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-22T01:47:08.899 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.899 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.899 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.899 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:08.899 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.899 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.899 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.899 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.899 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.900 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.900 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.900 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.900 INFO:teuthology.orchestra.run.smithi072.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-22T01:47:08.900 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-22T01:47:08.900 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.900 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.900 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.900 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:08.900 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.900 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.900 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.900 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.900 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.900 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.901 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.901 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.901 INFO:teuthology.orchestra.run.smithi072.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-22T01:47:08.901 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-22T01:47:08.901 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.901 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.901 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.901 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:08.901 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.901 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.901 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.901 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.901 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.901 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.902 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.902 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.902 INFO:teuthology.orchestra.run.smithi072.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-22T01:47:08.902 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-22T01:47:08.902 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.902 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.902 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.902 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:08.902 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.902 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.902 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.902 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.902 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.902 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.903 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.903 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.903 INFO:teuthology.orchestra.run.smithi072.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-22T01:47:08.903 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-22T01:47:08.903 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.903 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.903 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.903 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:08.903 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.903 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.903 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.903 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.903 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.903 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.903 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.904 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.904 INFO:teuthology.orchestra.run.smithi072.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-22T01:47:08.904 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-22T01:47:08.904 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.904 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.904 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.904 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:08.904 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.904 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.904 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.904 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.904 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.904 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.904 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.905 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.905 INFO:teuthology.orchestra.run.smithi072.stdout: "GRAFANA_API_PASSWORD": { 2024-03-22T01:47:08.905 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-22T01:47:08.905 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.905 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.905 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.905 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "admin", 2024-03-22T01:47:08.905 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.905 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.905 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.905 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.905 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.905 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.905 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.905 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.906 INFO:teuthology.orchestra.run.smithi072.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-22T01:47:08.906 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-22T01:47:08.906 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.906 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.906 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.906 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:08.906 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.906 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.906 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.906 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.906 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.906 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.906 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.907 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.907 INFO:teuthology.orchestra.run.smithi072.stdout: "GRAFANA_API_URL": { 2024-03-22T01:47:08.907 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "GRAFANA_API_URL", 2024-03-22T01:47:08.907 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.907 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.907 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.907 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:08.907 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.907 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.907 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.907 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.907 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.907 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.908 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.908 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.908 INFO:teuthology.orchestra.run.smithi072.stdout: "GRAFANA_API_USERNAME": { 2024-03-22T01:47:08.908 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-22T01:47:08.908 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.908 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.908 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.908 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "admin", 2024-03-22T01:47:08.908 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.908 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.908 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.908 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.908 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.908 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.909 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.909 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.909 INFO:teuthology.orchestra.run.smithi072.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-22T01:47:08.909 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-22T01:47:08.909 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.909 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.909 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.909 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:08.909 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.909 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.909 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.909 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.909 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.909 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.910 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.910 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.910 INFO:teuthology.orchestra.run.smithi072.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-22T01:47:08.910 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-22T01:47:08.910 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.910 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.910 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.910 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:08.910 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.910 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.910 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.910 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.911 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.911 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.911 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.911 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.911 INFO:teuthology.orchestra.run.smithi072.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-22T01:47:08.911 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-22T01:47:08.911 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.911 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.911 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.911 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:08.911 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.911 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.911 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.912 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.912 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.912 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.912 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.912 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.912 INFO:teuthology.orchestra.run.smithi072.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-22T01:47:08.912 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-22T01:47:08.912 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.912 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.912 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.912 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:08.912 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.912 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.912 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.913 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.913 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.913 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.913 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.913 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.913 INFO:teuthology.orchestra.run.smithi072.stdout: "MULTICLUSTER_CONFIG": { 2024-03-22T01:47:08.913 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-22T01:47:08.913 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.913 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.913 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.913 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "{}", 2024-03-22T01:47:08.913 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.913 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.913 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.913 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.914 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.914 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.914 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.914 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.914 INFO:teuthology.orchestra.run.smithi072.stdout: "PROMETHEUS_API_HOST": { 2024-03-22T01:47:08.914 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-22T01:47:08.914 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.914 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.914 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.914 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:08.914 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.914 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.914 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.915 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.915 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.915 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.915 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.915 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.915 INFO:teuthology.orchestra.run.smithi072.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-22T01:47:08.915 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-22T01:47:08.915 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.915 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.915 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.915 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:08.915 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.915 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.915 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.916 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.916 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.916 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.916 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.916 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.916 INFO:teuthology.orchestra.run.smithi072.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-22T01:47:08.916 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-22T01:47:08.916 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.916 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.916 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.916 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:08.916 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.916 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.916 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.916 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.917 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.917 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.917 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.917 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.917 INFO:teuthology.orchestra.run.smithi072.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-22T01:47:08.917 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-22T01:47:08.917 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.917 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.917 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.917 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:08.917 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.917 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.917 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.917 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.918 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.918 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.918 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.918 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.918 INFO:teuthology.orchestra.run.smithi072.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-22T01:47:08.918 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-22T01:47:08.918 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.918 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.918 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.918 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:08.918 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.918 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.918 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.919 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.919 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.919 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.919 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.919 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.919 INFO:teuthology.orchestra.run.smithi072.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-22T01:47:08.919 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-22T01:47:08.919 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.919 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.919 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.919 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:08.919 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.919 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.919 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.919 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.920 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.920 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.920 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.920 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.920 INFO:teuthology.orchestra.run.smithi072.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-22T01:47:08.920 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-22T01:47:08.920 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.920 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.920 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.920 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:08.920 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.920 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.920 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.920 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.921 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.921 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.921 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.921 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.921 INFO:teuthology.orchestra.run.smithi072.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-22T01:47:08.921 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-22T01:47:08.921 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.921 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.921 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.921 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:08.921 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.921 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.921 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.921 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.922 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.922 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.922 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.922 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.922 INFO:teuthology.orchestra.run.smithi072.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-22T01:47:08.922 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-22T01:47:08.922 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.922 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.922 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.922 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:08.922 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.922 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.922 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.922 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.923 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.923 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.923 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.923 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.923 INFO:teuthology.orchestra.run.smithi072.stdout: "PWD_POLICY_ENABLED": { 2024-03-22T01:47:08.923 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-22T01:47:08.923 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.923 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.923 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.923 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:08.923 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.923 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.923 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.923 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.924 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.924 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.924 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.924 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.924 INFO:teuthology.orchestra.run.smithi072.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-22T01:47:08.924 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-22T01:47:08.924 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.924 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.924 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.924 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-22T01:47:08.924 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.924 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.925 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.925 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.925 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.925 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.925 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.925 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.925 INFO:teuthology.orchestra.run.smithi072.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-22T01:47:08.925 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-22T01:47:08.925 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:08.925 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.925 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.925 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "10", 2024-03-22T01:47:08.925 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.925 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.926 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.926 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.926 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.926 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.926 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.926 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.926 INFO:teuthology.orchestra.run.smithi072.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-22T01:47:08.926 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-22T01:47:08.926 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:08.926 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.926 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.926 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "8", 2024-03-22T01:47:08.926 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.927 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.927 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.927 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.927 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.927 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.927 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.927 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.927 INFO:teuthology.orchestra.run.smithi072.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-22T01:47:08.927 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-22T01:47:08.927 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:08.927 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.927 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.927 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "45", 2024-03-22T01:47:08.927 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.928 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.928 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.928 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.928 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.928 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.928 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.928 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.928 INFO:teuthology.orchestra.run.smithi072.stdout: "RGW_API_ACCESS_KEY": { 2024-03-22T01:47:08.928 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-22T01:47:08.928 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.928 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.928 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.928 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:08.928 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.928 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.929 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.929 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.929 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.929 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.929 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.929 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.929 INFO:teuthology.orchestra.run.smithi072.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-22T01:47:08.929 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-22T01:47:08.929 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.929 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.929 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.929 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "admin", 2024-03-22T01:47:08.929 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.929 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.930 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.930 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.930 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.930 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.930 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.930 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.930 INFO:teuthology.orchestra.run.smithi072.stdout: "RGW_API_SECRET_KEY": { 2024-03-22T01:47:08.930 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-22T01:47:08.930 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.930 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.930 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.930 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:08.931 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.931 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.931 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.931 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.931 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.931 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.931 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.931 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.931 INFO:teuthology.orchestra.run.smithi072.stdout: "RGW_API_SSL_VERIFY": { 2024-03-22T01:47:08.931 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-22T01:47:08.931 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.931 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.931 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.931 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:08.931 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.932 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.932 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.932 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.932 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.932 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.932 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.932 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.932 INFO:teuthology.orchestra.run.smithi072.stdout: "UNSAFE_TLS_v1_2": { 2024-03-22T01:47:08.932 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-22T01:47:08.932 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.932 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.932 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.932 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:08.932 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.932 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.933 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.933 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.933 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.933 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.933 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.933 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.933 INFO:teuthology.orchestra.run.smithi072.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-22T01:47:08.933 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-22T01:47:08.933 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:08.933 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.933 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.933 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "0", 2024-03-22T01:47:08.933 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.933 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.933 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.934 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.934 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.934 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.934 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.934 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.934 INFO:teuthology.orchestra.run.smithi072.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-22T01:47:08.934 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-22T01:47:08.934 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:08.934 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.934 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.934 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "10", 2024-03-22T01:47:08.934 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.934 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.934 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.934 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.935 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.935 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.935 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.935 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.935 INFO:teuthology.orchestra.run.smithi072.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-22T01:47:08.935 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-22T01:47:08.935 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:08.935 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.935 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.935 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "5", 2024-03-22T01:47:08.935 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.935 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.935 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.935 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.936 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.936 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.936 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.936 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.936 INFO:teuthology.orchestra.run.smithi072.stdout: "cross_origin_url": { 2024-03-22T01:47:08.936 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "cross_origin_url", 2024-03-22T01:47:08.936 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.936 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.936 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.936 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:08.936 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.936 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.936 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.936 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.937 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.937 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.937 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.937 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.937 INFO:teuthology.orchestra.run.smithi072.stdout: "crt_file": { 2024-03-22T01:47:08.937 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "crt_file", 2024-03-22T01:47:08.937 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.937 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.937 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.937 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:08.937 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.937 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.937 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.938 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.938 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.938 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.938 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.938 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.938 INFO:teuthology.orchestra.run.smithi072.stdout: "debug": { 2024-03-22T01:47:08.938 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "debug", 2024-03-22T01:47:08.938 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.938 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.938 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.938 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:08.938 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.938 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.938 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.939 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Enable/disable debug options", 2024-03-22T01:47:08.939 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.939 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.939 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.939 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.939 INFO:teuthology.orchestra.run.smithi072.stdout: "jwt_token_ttl": { 2024-03-22T01:47:08.939 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "jwt_token_ttl", 2024-03-22T01:47:08.939 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:08.939 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.939 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.939 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "28800", 2024-03-22T01:47:08.939 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.939 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.939 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.940 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.940 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.940 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.940 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.940 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.940 INFO:teuthology.orchestra.run.smithi072.stdout: "key_file": { 2024-03-22T01:47:08.940 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "key_file", 2024-03-22T01:47:08.940 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.940 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.940 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.940 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:08.940 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.940 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.940 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.940 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.941 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.941 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.941 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.941 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.941 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:08.941 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:08.941 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.941 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.941 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.941 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:08.941 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.941 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.941 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:08.941 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:08.941 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:08.942 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:08.942 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:08.942 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:08.942 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:08.942 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:08.942 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.942 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.942 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.942 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.942 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.942 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:08.942 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:08.943 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.943 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.943 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.943 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:08.943 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.943 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.943 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.943 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.943 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.943 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.943 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.943 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.943 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:08.944 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:08.944 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.944 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.944 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.944 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:08.944 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.944 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.944 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:08.944 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:08.944 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:08.944 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:08.944 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:08.944 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:08.944 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:08.945 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:08.945 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.945 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.945 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.945 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.945 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.945 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:08.945 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:08.945 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.945 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.945 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.945 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:08.946 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.946 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.946 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.946 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.946 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.946 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.946 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.946 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.946 INFO:teuthology.orchestra.run.smithi072.stdout: "motd": { 2024-03-22T01:47:08.946 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "motd", 2024-03-22T01:47:08.946 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.946 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.947 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.947 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:08.947 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.947 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.947 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.947 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "The message of the day", 2024-03-22T01:47:08.947 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.947 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.947 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.947 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.947 INFO:teuthology.orchestra.run.smithi072.stdout: "redirect_resolve_ip_addr": { 2024-03-22T01:47:08.947 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "redirect_resolve_ip_addr", 2024-03-22T01:47:08.947 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.947 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.948 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.948 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:08.948 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.948 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.948 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.948 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.948 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.948 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.948 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.948 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.948 INFO:teuthology.orchestra.run.smithi072.stdout: "server_addr": { 2024-03-22T01:47:08.948 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "server_addr", 2024-03-22T01:47:08.948 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.948 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.948 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.949 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "::", 2024-03-22T01:47:08.949 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.949 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.949 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.949 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.949 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.949 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.949 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.949 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.949 INFO:teuthology.orchestra.run.smithi072.stdout: "server_port": { 2024-03-22T01:47:08.949 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "server_port", 2024-03-22T01:47:08.949 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:08.949 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.950 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.950 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "8080", 2024-03-22T01:47:08.950 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.950 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.950 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.950 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.950 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.950 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.950 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.950 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.950 INFO:teuthology.orchestra.run.smithi072.stdout: "ssl": { 2024-03-22T01:47:08.950 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "ssl", 2024-03-22T01:47:08.950 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.950 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.951 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.951 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:08.951 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.951 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.951 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.951 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.951 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.951 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.951 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.951 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.951 INFO:teuthology.orchestra.run.smithi072.stdout: "ssl_server_port": { 2024-03-22T01:47:08.952 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "ssl_server_port", 2024-03-22T01:47:08.952 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:08.952 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.952 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.952 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "8443", 2024-03-22T01:47:08.952 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.952 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.952 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.952 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.952 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.952 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.952 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.953 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.953 INFO:teuthology.orchestra.run.smithi072.stdout: "standby_behaviour": { 2024-03-22T01:47:08.953 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "standby_behaviour", 2024-03-22T01:47:08.953 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.953 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.953 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.953 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "redirect", 2024-03-22T01:47:08.953 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.953 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.953 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:08.953 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:08.953 INFO:teuthology.orchestra.run.smithi072.stdout: "redirect" 2024-03-22T01:47:08.954 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:08.954 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.954 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.954 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.954 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.954 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.954 INFO:teuthology.orchestra.run.smithi072.stdout: "standby_error_status_code": { 2024-03-22T01:47:08.954 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "standby_error_status_code", 2024-03-22T01:47:08.954 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:08.954 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.954 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.954 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "500", 2024-03-22T01:47:08.954 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "400", 2024-03-22T01:47:08.955 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "599", 2024-03-22T01:47:08.955 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.955 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.955 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.955 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.955 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.955 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.955 INFO:teuthology.orchestra.run.smithi072.stdout: "url_prefix": { 2024-03-22T01:47:08.955 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "url_prefix", 2024-03-22T01:47:08.955 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.955 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.955 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.955 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:08.956 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.956 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.956 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.956 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.956 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.956 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.956 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.956 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:08.956 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:08.956 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.956 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:08.956 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "devicehealth", 2024-03-22T01:47:08.956 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:08.957 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:08.957 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:08.957 INFO:teuthology.orchestra.run.smithi072.stdout: "enable_monitoring": { 2024-03-22T01:47:08.957 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "enable_monitoring", 2024-03-22T01:47:08.957 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.957 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.957 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.957 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:08.957 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.957 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.957 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.957 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "monitor device health metrics", 2024-03-22T01:47:08.957 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.958 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.958 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.958 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.958 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:08.958 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:08.958 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.958 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.958 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.958 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:08.958 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.958 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.958 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:08.958 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:08.959 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:08.959 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:08.959 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:08.959 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:08.959 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:08.959 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:08.959 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.959 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.959 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.959 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.959 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.959 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:08.959 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:08.959 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.960 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.960 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.960 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:08.960 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.960 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.960 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.960 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.960 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.960 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.960 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.960 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.960 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:08.960 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:08.960 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.961 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.961 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.961 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:08.961 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.961 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.961 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:08.961 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:08.961 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:08.961 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:08.962 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:08.962 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:08.962 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:08.962 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:08.962 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.962 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.962 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.962 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.962 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.962 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:08.962 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:08.962 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.962 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.963 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.963 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:08.963 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.963 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.963 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.963 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.963 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.963 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.963 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.963 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.963 INFO:teuthology.orchestra.run.smithi072.stdout: "mark_out_threshold": { 2024-03-22T01:47:08.963 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "mark_out_threshold", 2024-03-22T01:47:08.963 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "secs", 2024-03-22T01:47:08.963 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.963 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.964 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "2419200", 2024-03-22T01:47:08.964 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.964 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.964 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.964 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-22T01:47:08.964 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.964 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.964 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.964 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.964 INFO:teuthology.orchestra.run.smithi072.stdout: "pool_name": { 2024-03-22T01:47:08.964 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "pool_name", 2024-03-22T01:47:08.964 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.964 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.964 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.965 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "device_health_metrics", 2024-03-22T01:47:08.965 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.965 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.965 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.965 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-22T01:47:08.965 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.965 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.965 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.965 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.965 INFO:teuthology.orchestra.run.smithi072.stdout: "retention_period": { 2024-03-22T01:47:08.965 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "retention_period", 2024-03-22T01:47:08.965 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "secs", 2024-03-22T01:47:08.965 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.965 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.965 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "15552000", 2024-03-22T01:47:08.966 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.966 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.966 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.966 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "how long to retain device health metrics", 2024-03-22T01:47:08.966 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.966 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.966 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.966 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.966 INFO:teuthology.orchestra.run.smithi072.stdout: "scrape_frequency": { 2024-03-22T01:47:08.966 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "scrape_frequency", 2024-03-22T01:47:08.966 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "secs", 2024-03-22T01:47:08.966 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.966 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.967 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "86400", 2024-03-22T01:47:08.967 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.967 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.967 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.967 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-22T01:47:08.967 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.967 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.967 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.967 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.967 INFO:teuthology.orchestra.run.smithi072.stdout: "self_heal": { 2024-03-22T01:47:08.967 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "self_heal", 2024-03-22T01:47:08.967 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.967 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.968 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.968 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:08.968 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.968 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.968 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.968 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-22T01:47:08.968 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.968 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.968 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.968 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.968 INFO:teuthology.orchestra.run.smithi072.stdout: "sleep_interval": { 2024-03-22T01:47:08.968 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "sleep_interval", 2024-03-22T01:47:08.968 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "secs", 2024-03-22T01:47:08.968 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.968 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.969 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "600", 2024-03-22T01:47:08.969 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.969 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.969 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.969 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "how frequently to wake up and check device health", 2024-03-22T01:47:08.969 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.969 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.969 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.969 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.969 INFO:teuthology.orchestra.run.smithi072.stdout: "warn_threshold": { 2024-03-22T01:47:08.969 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "warn_threshold", 2024-03-22T01:47:08.969 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "secs", 2024-03-22T01:47:08.969 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.970 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.970 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "7257600", 2024-03-22T01:47:08.970 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.970 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.970 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.970 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-22T01:47:08.970 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.970 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.970 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.970 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:08.970 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:08.970 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.970 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:08.970 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "diskprediction_local", 2024-03-22T01:47:08.971 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:08.971 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:08.971 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:08.971 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:08.971 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:08.971 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.971 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.971 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.971 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:08.971 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.971 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.971 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:08.971 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:08.971 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:08.971 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:08.972 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:08.972 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:08.972 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:08.972 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:08.972 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.972 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.972 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.972 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.972 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.972 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:08.972 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:08.972 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.972 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.973 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.973 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:08.973 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.973 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.973 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.973 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.973 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.973 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.973 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.973 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.973 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:08.973 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:08.973 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.973 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.973 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.974 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:08.974 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.974 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.974 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:08.974 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:08.974 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:08.974 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:08.974 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:08.974 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:08.974 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:08.974 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:08.974 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.974 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.975 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.975 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.975 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.975 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:08.975 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:08.975 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.975 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.975 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.975 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:08.975 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.975 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.975 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.975 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.976 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.976 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.976 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.976 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.976 INFO:teuthology.orchestra.run.smithi072.stdout: "predict_interval": { 2024-03-22T01:47:08.976 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "predict_interval", 2024-03-22T01:47:08.976 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.976 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.976 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.976 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "86400", 2024-03-22T01:47:08.976 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.976 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.976 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.976 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.976 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.977 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.977 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.977 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.977 INFO:teuthology.orchestra.run.smithi072.stdout: "predictor_model": { 2024-03-22T01:47:08.977 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "predictor_model", 2024-03-22T01:47:08.977 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.977 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.977 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.977 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "prophetstor", 2024-03-22T01:47:08.977 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.977 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.977 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.977 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.977 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.977 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.978 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.978 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.978 INFO:teuthology.orchestra.run.smithi072.stdout: "sleep_interval": { 2024-03-22T01:47:08.978 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "sleep_interval", 2024-03-22T01:47:08.978 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.978 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.978 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.978 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "600", 2024-03-22T01:47:08.978 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.978 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.978 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.978 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.978 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.978 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.979 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.979 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:08.979 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:08.979 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.979 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:08.979 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "influx", 2024-03-22T01:47:08.979 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": false, 2024-03-22T01:47:08.979 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "influxdb python module not found", 2024-03-22T01:47:08.979 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:08.979 INFO:teuthology.orchestra.run.smithi072.stdout: "batch_size": { 2024-03-22T01:47:08.979 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "batch_size", 2024-03-22T01:47:08.979 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:08.979 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.980 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.980 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "5000", 2024-03-22T01:47:08.980 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.980 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.980 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.980 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-22T01:47:08.980 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.980 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.980 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.980 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.980 INFO:teuthology.orchestra.run.smithi072.stdout: "database": { 2024-03-22T01:47:08.980 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "database", 2024-03-22T01:47:08.980 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.980 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.980 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.981 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "ceph", 2024-03-22T01:47:08.981 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.981 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.981 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.981 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-22T01:47:08.981 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.981 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.981 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.981 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.981 INFO:teuthology.orchestra.run.smithi072.stdout: "hostname": { 2024-03-22T01:47:08.981 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "hostname", 2024-03-22T01:47:08.981 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.981 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.981 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.982 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:08.982 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.982 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.982 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.982 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "InfluxDB server hostname", 2024-03-22T01:47:08.982 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.982 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.982 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.982 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.982 INFO:teuthology.orchestra.run.smithi072.stdout: "interval": { 2024-03-22T01:47:08.982 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "interval", 2024-03-22T01:47:08.982 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "secs", 2024-03-22T01:47:08.982 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.982 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.983 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "30", 2024-03-22T01:47:08.983 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "5", 2024-03-22T01:47:08.983 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.983 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.983 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-22T01:47:08.983 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.983 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.983 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.983 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.983 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:08.983 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:08.983 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.983 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.983 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.983 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:08.984 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.984 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.984 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:08.984 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:08.984 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:08.984 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:08.984 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:08.984 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:08.984 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:08.984 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:08.984 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.984 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.984 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.984 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.984 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.985 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:08.985 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:08.985 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.985 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.985 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.985 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:08.985 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.985 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.985 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.985 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.985 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.985 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.985 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.985 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.985 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:08.986 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:08.986 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.986 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.986 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.986 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:08.986 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.986 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.986 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:08.986 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:08.986 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:08.986 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:08.986 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:08.986 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:08.986 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:08.986 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:08.987 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.987 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.987 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.987 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.987 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.987 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:08.987 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:08.987 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.987 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.987 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.987 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:08.987 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.987 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.987 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.987 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.988 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.988 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.988 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.988 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.988 INFO:teuthology.orchestra.run.smithi072.stdout: "password": { 2024-03-22T01:47:08.988 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "password", 2024-03-22T01:47:08.988 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.988 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.988 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.988 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:08.988 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.988 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.988 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.988 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "password of InfluxDB server user", 2024-03-22T01:47:08.988 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.989 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.989 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.989 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.989 INFO:teuthology.orchestra.run.smithi072.stdout: "port": { 2024-03-22T01:47:08.989 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "port", 2024-03-22T01:47:08.989 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:08.989 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.989 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.989 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "8086", 2024-03-22T01:47:08.989 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.989 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.989 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.989 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "InfluxDB server port", 2024-03-22T01:47:08.989 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.989 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.990 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.990 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.990 INFO:teuthology.orchestra.run.smithi072.stdout: "ssl": { 2024-03-22T01:47:08.990 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "ssl", 2024-03-22T01:47:08.990 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.990 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.990 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.990 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "false", 2024-03-22T01:47:08.990 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.990 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.990 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.990 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-22T01:47:08.990 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.991 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.991 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.991 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.991 INFO:teuthology.orchestra.run.smithi072.stdout: "threads": { 2024-03-22T01:47:08.991 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "threads", 2024-03-22T01:47:08.991 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:08.991 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.991 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.991 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "5", 2024-03-22T01:47:08.991 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "1", 2024-03-22T01:47:08.991 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "32", 2024-03-22T01:47:08.991 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.992 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-22T01:47:08.992 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.992 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.992 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.992 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.992 INFO:teuthology.orchestra.run.smithi072.stdout: "username": { 2024-03-22T01:47:08.992 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "username", 2024-03-22T01:47:08.992 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.992 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.992 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.992 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:08.992 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.992 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.993 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.993 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "username of InfluxDB server user", 2024-03-22T01:47:08.993 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.993 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.993 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.993 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.993 INFO:teuthology.orchestra.run.smithi072.stdout: "verify_ssl": { 2024-03-22T01:47:08.993 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "verify_ssl", 2024-03-22T01:47:08.993 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.993 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.993 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:08.993 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "true", 2024-03-22T01:47:08.993 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.993 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.994 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.994 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-22T01:47:08.994 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.994 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.994 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.994 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:08.994 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:08.994 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.994 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:08.994 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "insights", 2024-03-22T01:47:08.994 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:08.994 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:08.994 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:08.994 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:08.995 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:08.995 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.995 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.995 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.995 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:08.995 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.995 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.995 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:08.995 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:08.995 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:08.995 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:08.995 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:08.995 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:08.996 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:08.996 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:08.996 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.996 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.996 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.996 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.996 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.996 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:08.996 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:08.996 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.996 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.996 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.996 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:08.997 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.997 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.997 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.997 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.997 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.997 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.997 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.997 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.997 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:08.997 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:08.997 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:08.997 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.997 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.997 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:08.998 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.998 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.998 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:08.998 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:08.998 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:08.998 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:08.998 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:08.998 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:08.998 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:08.998 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:08.998 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.998 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.998 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.998 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.999 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:08.999 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:08.999 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:08.999 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:08.999 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:08.999 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:08.999 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:08.999 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:08.999 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:08.999 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:08.999 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:08.999 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:08.999 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:08.999 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:08.999 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.000 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.000 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.000 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.000 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "iostat", 2024-03-22T01:47:09.000 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.000 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.000 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.000 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.000 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.000 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.000 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.000 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.000 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.000 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.001 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.001 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.001 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.001 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.001 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.001 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.001 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.001 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.001 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.001 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.001 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.001 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.001 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.001 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.001 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.001 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.002 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.002 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.002 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.002 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.002 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.002 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.002 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.002 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.002 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.002 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.002 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.002 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.002 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.002 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.003 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.003 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.003 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.003 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.003 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.003 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.003 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.003 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.003 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.003 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.003 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.003 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.003 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.003 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.003 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.004 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.004 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.004 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.004 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.004 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.004 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.004 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.004 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.004 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.004 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.004 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.004 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.004 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.004 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.004 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.005 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.005 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.005 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.005 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.005 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.005 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.005 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "k8sevents", 2024-03-22T01:47:09.005 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.005 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.005 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.005 INFO:teuthology.orchestra.run.smithi072.stdout: "ceph_event_retention_days": { 2024-03-22T01:47:09.005 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "ceph_event_retention_days", 2024-03-22T01:47:09.005 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.005 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.005 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.006 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "7", 2024-03-22T01:47:09.006 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.006 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.006 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.006 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-22T01:47:09.006 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.006 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.006 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.006 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.006 INFO:teuthology.orchestra.run.smithi072.stdout: "config_check_secs": { 2024-03-22T01:47:09.006 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "config_check_secs", 2024-03-22T01:47:09.006 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.006 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.007 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.007 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "10", 2024-03-22T01:47:09.007 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "10", 2024-03-22T01:47:09.007 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.007 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.007 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-22T01:47:09.007 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.007 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.007 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.007 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.007 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.007 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.007 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.007 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.008 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.008 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.008 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.008 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.008 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.008 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.008 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.008 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.008 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.008 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.008 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.008 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.008 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.008 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.008 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.009 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.009 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.009 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.009 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.009 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.009 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.009 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.009 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.009 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.009 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.009 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.009 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.009 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.009 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.010 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.010 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.010 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.010 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.010 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.010 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.010 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.010 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.010 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.010 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.010 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.010 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.010 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.010 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.011 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.011 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.011 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.011 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.011 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.011 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.011 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.011 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.011 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.011 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.011 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.011 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.012 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.012 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.012 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.012 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.012 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.012 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.012 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.012 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.012 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.012 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.012 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.012 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.012 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.012 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.013 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "localpool", 2024-03-22T01:47:09.013 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.013 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.013 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.013 INFO:teuthology.orchestra.run.smithi072.stdout: "failure_domain": { 2024-03-22T01:47:09.013 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "failure_domain", 2024-03-22T01:47:09.013 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.013 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.013 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.013 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "host", 2024-03-22T01:47:09.013 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.013 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.013 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.013 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "failure domain for any created local pool", 2024-03-22T01:47:09.014 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-22T01:47:09.014 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.014 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.014 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.014 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.014 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.014 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.014 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.014 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.014 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.014 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.014 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.014 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.014 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.014 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.015 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.015 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.015 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.015 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.015 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.015 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.015 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.015 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.015 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.015 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.015 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.015 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.015 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.015 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.015 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.016 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.016 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.016 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.016 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.016 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.016 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.016 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.016 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.016 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.016 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.016 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.016 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.016 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.016 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.017 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.017 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.017 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.017 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.017 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.017 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.017 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.017 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.017 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.017 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.017 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.017 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.017 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.017 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.017 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.018 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.018 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.018 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.018 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.018 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.018 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.018 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.018 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.018 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.018 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.018 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.018 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.018 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.019 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.019 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.019 INFO:teuthology.orchestra.run.smithi072.stdout: "min_size": { 2024-03-22T01:47:09.019 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "min_size", 2024-03-22T01:47:09.019 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.019 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.019 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.019 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.019 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.019 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.019 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.019 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "default min_size for any created local pool", 2024-03-22T01:47:09.019 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-22T01:47:09.019 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.020 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.020 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.020 INFO:teuthology.orchestra.run.smithi072.stdout: "num_rep": { 2024-03-22T01:47:09.020 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "num_rep", 2024-03-22T01:47:09.020 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.020 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.020 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.020 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "3", 2024-03-22T01:47:09.020 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.020 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.020 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.020 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "default replica count for any created local pool", 2024-03-22T01:47:09.020 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.020 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.021 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.021 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.021 INFO:teuthology.orchestra.run.smithi072.stdout: "pg_num": { 2024-03-22T01:47:09.021 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "pg_num", 2024-03-22T01:47:09.021 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.021 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.021 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.021 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "128", 2024-03-22T01:47:09.021 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.021 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.021 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.021 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "default pg_num for any created local pool", 2024-03-22T01:47:09.021 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.021 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.022 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.022 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.022 INFO:teuthology.orchestra.run.smithi072.stdout: "prefix": { 2024-03-22T01:47:09.022 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "prefix", 2024-03-22T01:47:09.022 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.022 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.022 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.022 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.022 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.022 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.022 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.022 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "name prefix for any created local pool", 2024-03-22T01:47:09.022 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.023 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.023 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.023 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.023 INFO:teuthology.orchestra.run.smithi072.stdout: "subtree": { 2024-03-22T01:47:09.023 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "subtree", 2024-03-22T01:47:09.023 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.023 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.023 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.023 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "rack", 2024-03-22T01:47:09.023 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.023 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.023 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.023 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-22T01:47:09.023 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-22T01:47:09.023 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.024 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.024 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.024 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.024 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.024 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.024 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "mds_autoscaler", 2024-03-22T01:47:09.024 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.024 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.024 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.024 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.024 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.024 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.024 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.024 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.024 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.025 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.025 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.025 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.025 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.025 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.025 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.025 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.025 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.025 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.025 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.025 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.025 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.025 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.025 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.026 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.026 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.026 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.026 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.026 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.026 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.026 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.026 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.026 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.026 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.026 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.026 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.026 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.026 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.027 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.027 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.027 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.027 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.027 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.027 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.027 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.027 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.027 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.027 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.027 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.027 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.027 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.027 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.027 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.028 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.028 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.028 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.028 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.028 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.028 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.028 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.028 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.028 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.028 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.028 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.028 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.028 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.028 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.029 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.029 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.029 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.029 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.029 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.029 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.029 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.029 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.029 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.029 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.029 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "mirroring", 2024-03-22T01:47:09.029 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.029 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.029 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.030 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.030 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.030 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.030 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.030 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.030 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.030 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.030 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.030 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.030 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.030 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.030 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.030 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.030 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.031 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.031 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.031 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.031 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.031 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.031 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.031 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.031 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.031 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.031 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.031 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.031 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.031 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.031 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.032 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.032 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.032 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.032 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.032 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.032 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.032 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.032 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.032 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.032 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.032 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.032 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.032 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.032 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.032 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.033 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.033 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.033 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.033 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.033 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.033 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.033 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.033 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.033 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.033 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.033 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.033 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.034 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.034 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.034 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.034 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.034 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.034 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.034 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.034 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.034 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.034 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.034 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.034 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.034 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.035 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.035 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.035 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.035 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.035 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.035 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "nfs", 2024-03-22T01:47:09.035 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.035 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.035 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.035 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.035 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.035 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.035 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.035 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.035 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.036 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.036 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.036 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.036 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.036 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.036 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.036 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.036 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.036 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.036 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.036 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.036 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.036 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.036 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.036 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.036 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.037 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.037 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.037 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.037 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.037 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.037 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.037 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.037 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.037 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.037 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.037 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.037 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.037 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.037 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.038 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.038 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.038 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.038 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.038 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.038 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.038 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.038 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.038 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.038 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.038 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.038 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.038 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.038 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.039 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.039 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.039 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.039 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.039 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.039 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.039 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.039 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.039 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.039 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.039 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.039 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.039 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.039 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.040 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.040 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.040 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.040 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.040 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.040 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.040 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.040 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.040 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.040 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "orchestrator", 2024-03-22T01:47:09.040 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.040 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.040 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.040 INFO:teuthology.orchestra.run.smithi072.stdout: "fail_fs": { 2024-03-22T01:47:09.040 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "fail_fs", 2024-03-22T01:47:09.041 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.041 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.041 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.041 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.041 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.041 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.041 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.041 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-22T01:47:09.041 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.041 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.041 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.041 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.041 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.041 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.041 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.042 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.042 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.042 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.042 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.042 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.042 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.042 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.042 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.042 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.042 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.042 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.042 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.042 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.042 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.042 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.043 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.043 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.043 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.043 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.043 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.043 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.043 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.043 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.043 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.043 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.043 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.043 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.043 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.043 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.044 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.044 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.044 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.044 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.044 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.044 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.044 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.044 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.044 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.044 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.044 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.044 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.044 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.044 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.044 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.044 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.045 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.045 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.045 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.045 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.045 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.045 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.045 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.045 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.045 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.045 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.045 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.045 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.045 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.045 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.045 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.046 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.046 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.046 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.046 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.046 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.046 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.046 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.046 INFO:teuthology.orchestra.run.smithi072.stdout: "orchestrator": { 2024-03-22T01:47:09.046 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "orchestrator", 2024-03-22T01:47:09.046 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.046 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.046 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.046 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.047 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.047 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.047 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.047 INFO:teuthology.orchestra.run.smithi072.stdout: "cephadm", 2024-03-22T01:47:09.047 INFO:teuthology.orchestra.run.smithi072.stdout: "rook", 2024-03-22T01:47:09.047 INFO:teuthology.orchestra.run.smithi072.stdout: "test_orchestrator" 2024-03-22T01:47:09.047 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.047 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Orchestrator backend", 2024-03-22T01:47:09.047 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.047 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.047 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.047 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.047 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.047 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.047 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.048 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "osd_perf_query", 2024-03-22T01:47:09.048 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.048 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.048 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.048 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.048 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.048 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.048 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.048 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.048 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.048 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.049 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.049 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.049 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.049 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.049 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.049 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.049 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.049 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.049 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.049 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.049 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.049 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.049 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.050 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.050 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.050 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.050 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.050 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.050 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.050 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.050 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.050 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.050 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.050 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.050 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.050 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.050 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.050 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.051 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.051 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.051 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.051 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.051 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.051 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.051 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.051 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.051 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.051 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.051 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.051 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.051 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.051 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.052 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.052 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.052 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.052 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.052 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.052 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.052 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.052 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.052 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.052 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.052 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.052 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.052 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.052 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.053 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.053 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.053 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.053 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.053 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.053 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.053 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.053 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.053 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.053 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.053 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "osd_support", 2024-03-22T01:47:09.053 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.053 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.053 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.053 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.054 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.054 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.054 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.054 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.054 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.054 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.054 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.054 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.054 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.054 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.054 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.054 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.054 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.054 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.055 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.055 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.055 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.055 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.055 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.055 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.055 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.055 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.055 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.055 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.055 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.055 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.055 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.055 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.055 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.056 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.056 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.056 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.056 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.056 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.056 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.056 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.056 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.056 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.056 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.056 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.056 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.057 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.057 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.057 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.057 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.057 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.057 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.057 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.057 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.057 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.057 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.057 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.057 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.057 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.057 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.057 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.057 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.058 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.058 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.058 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.058 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.058 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.058 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.058 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.058 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.058 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.058 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.058 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.058 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.058 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.058 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.059 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.059 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "pg_autoscaler", 2024-03-22T01:47:09.059 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.059 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.059 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.059 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.059 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.059 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.059 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.059 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.059 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.059 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.059 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.059 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.059 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.060 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.060 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.060 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.060 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.060 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.060 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.060 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.060 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.060 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.060 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.060 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.060 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.060 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.060 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.060 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.061 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.061 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.061 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.061 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.061 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.061 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.061 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.061 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.061 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.061 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.061 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.061 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.061 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.062 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.062 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.062 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.062 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.062 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.062 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.062 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.062 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.062 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.062 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.062 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.062 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.062 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.062 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.063 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.063 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.063 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.063 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.063 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.063 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.063 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.063 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.063 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.063 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.063 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.063 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.063 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.063 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.064 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.064 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.064 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.064 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.064 INFO:teuthology.orchestra.run.smithi072.stdout: "sleep_interval": { 2024-03-22T01:47:09.064 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "sleep_interval", 2024-03-22T01:47:09.064 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "secs", 2024-03-22T01:47:09.064 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.064 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.064 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "60", 2024-03-22T01:47:09.064 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.064 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.064 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.064 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.064 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.065 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.065 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.065 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.065 INFO:teuthology.orchestra.run.smithi072.stdout: "threshold": { 2024-03-22T01:47:09.065 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "threshold", 2024-03-22T01:47:09.065 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "float", 2024-03-22T01:47:09.065 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.065 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.065 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "3.0", 2024-03-22T01:47:09.065 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "1.0", 2024-03-22T01:47:09.065 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.065 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.065 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "scaling threshold", 2024-03-22T01:47:09.065 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-22T01:47:09.065 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.066 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.066 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.066 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.066 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.066 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.066 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "progress", 2024-03-22T01:47:09.066 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.066 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.066 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.066 INFO:teuthology.orchestra.run.smithi072.stdout: "allow_pg_recovery_event": { 2024-03-22T01:47:09.066 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "allow_pg_recovery_event", 2024-03-22T01:47:09.066 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.066 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.066 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.067 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.067 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.067 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.067 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.067 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-22T01:47:09.067 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.067 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.067 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.067 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.067 INFO:teuthology.orchestra.run.smithi072.stdout: "enabled": { 2024-03-22T01:47:09.067 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "enabled", 2024-03-22T01:47:09.067 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.067 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.067 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.068 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.068 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.068 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.068 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.068 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.068 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.068 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.068 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.068 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.068 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.068 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.068 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.068 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.068 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.068 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.069 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.069 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.069 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.069 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.069 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.069 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.069 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.069 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.069 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.069 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.069 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.069 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.069 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.069 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.069 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.070 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.070 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.070 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.070 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.070 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.070 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.070 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.070 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.070 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.070 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.070 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.070 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.070 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.070 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.071 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.071 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.071 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.071 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.071 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.071 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.071 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.071 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.071 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.071 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.071 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.071 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.071 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.071 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.071 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.072 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.072 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.072 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.072 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.072 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.072 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.072 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.072 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.072 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.072 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.072 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.072 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.072 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.072 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.072 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.073 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.073 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.073 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.073 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.073 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.073 INFO:teuthology.orchestra.run.smithi072.stdout: "max_completed_events": { 2024-03-22T01:47:09.073 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "max_completed_events", 2024-03-22T01:47:09.073 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.073 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.073 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.073 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "50", 2024-03-22T01:47:09.073 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.073 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.074 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.074 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "number of past completed events to remember", 2024-03-22T01:47:09.074 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.074 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.074 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.074 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.074 INFO:teuthology.orchestra.run.smithi072.stdout: "sleep_interval": { 2024-03-22T01:47:09.074 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "sleep_interval", 2024-03-22T01:47:09.074 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "secs", 2024-03-22T01:47:09.074 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.074 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.074 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "5", 2024-03-22T01:47:09.074 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.075 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.075 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.075 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "how long the module is going to sleep", 2024-03-22T01:47:09.075 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.075 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.075 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.075 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.075 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.075 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.075 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.075 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "prometheus", 2024-03-22T01:47:09.075 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.075 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.075 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.076 INFO:teuthology.orchestra.run.smithi072.stdout: "cache": { 2024-03-22T01:47:09.076 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "cache", 2024-03-22T01:47:09.076 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.076 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.076 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.076 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.076 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.076 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.076 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.076 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.076 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.076 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.076 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.076 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.077 INFO:teuthology.orchestra.run.smithi072.stdout: "exclude_perf_counters": { 2024-03-22T01:47:09.077 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "exclude_perf_counters", 2024-03-22T01:47:09.077 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.077 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.077 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.077 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.077 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.077 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.077 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.077 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-22T01:47:09.077 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-22T01:47:09.077 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.077 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.077 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.078 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.078 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.078 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.078 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.078 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.078 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.078 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.078 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.078 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.078 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.078 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.078 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.078 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.079 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.079 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.079 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.079 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.079 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.079 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.079 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.079 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.079 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.079 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.079 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.079 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.079 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.080 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.080 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.080 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.080 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.080 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.080 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.080 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.080 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.080 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.080 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.080 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.080 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.080 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.081 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.081 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.081 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.081 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.081 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.081 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.081 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.081 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.081 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.081 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.081 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.081 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.081 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.082 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.082 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.082 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.082 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.082 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.082 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.082 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.082 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.082 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.082 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.082 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.082 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.082 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.082 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.083 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.083 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.083 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.083 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.083 INFO:teuthology.orchestra.run.smithi072.stdout: "rbd_stats_pools": { 2024-03-22T01:47:09.083 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "rbd_stats_pools", 2024-03-22T01:47:09.083 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.083 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.083 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.083 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.083 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.083 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.083 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.084 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.084 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.084 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.084 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.084 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.084 INFO:teuthology.orchestra.run.smithi072.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-22T01:47:09.084 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-22T01:47:09.084 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.084 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.084 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.084 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "300", 2024-03-22T01:47:09.084 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.084 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.084 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.085 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.085 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.085 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.085 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.085 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.085 INFO:teuthology.orchestra.run.smithi072.stdout: "scrape_interval": { 2024-03-22T01:47:09.085 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "scrape_interval", 2024-03-22T01:47:09.085 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "float", 2024-03-22T01:47:09.085 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.085 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.085 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "15.0", 2024-03-22T01:47:09.085 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.085 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.085 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.085 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.086 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.086 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.086 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.086 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.086 INFO:teuthology.orchestra.run.smithi072.stdout: "server_addr": { 2024-03-22T01:47:09.086 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "server_addr", 2024-03-22T01:47:09.086 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.086 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.086 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.086 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "::", 2024-03-22T01:47:09.086 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.086 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.086 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.086 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-22T01:47:09.087 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.087 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.087 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.087 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.087 INFO:teuthology.orchestra.run.smithi072.stdout: "server_port": { 2024-03-22T01:47:09.087 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "server_port", 2024-03-22T01:47:09.087 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.087 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.087 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.087 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "9283", 2024-03-22T01:47:09.087 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.087 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.087 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.088 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-22T01:47:09.088 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.088 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.088 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.088 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.088 INFO:teuthology.orchestra.run.smithi072.stdout: "stale_cache_strategy": { 2024-03-22T01:47:09.088 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "stale_cache_strategy", 2024-03-22T01:47:09.088 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.088 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.088 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.088 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "log", 2024-03-22T01:47:09.088 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.088 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.088 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.088 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.089 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.089 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.089 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.089 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.089 INFO:teuthology.orchestra.run.smithi072.stdout: "standby_behaviour": { 2024-03-22T01:47:09.089 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "standby_behaviour", 2024-03-22T01:47:09.089 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.089 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.089 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.089 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "default", 2024-03-22T01:47:09.089 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.089 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.089 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.089 INFO:teuthology.orchestra.run.smithi072.stdout: "default", 2024-03-22T01:47:09.089 INFO:teuthology.orchestra.run.smithi072.stdout: "error" 2024-03-22T01:47:09.090 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.090 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.090 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.090 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.090 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.090 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.090 INFO:teuthology.orchestra.run.smithi072.stdout: "standby_error_status_code": { 2024-03-22T01:47:09.090 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "standby_error_status_code", 2024-03-22T01:47:09.090 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.090 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.090 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.090 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "500", 2024-03-22T01:47:09.090 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "400", 2024-03-22T01:47:09.090 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "599", 2024-03-22T01:47:09.091 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.091 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.091 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.091 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.091 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.091 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.091 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.091 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.091 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.091 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "rbd_support", 2024-03-22T01:47:09.091 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.091 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.091 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.091 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.091 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.092 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.092 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.092 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.092 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.092 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.092 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.092 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.092 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.092 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.092 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.092 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.092 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.092 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.092 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.092 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.093 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.093 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.093 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.093 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.093 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.093 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.093 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.093 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.093 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.093 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.093 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.093 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.093 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.093 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.094 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.094 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.094 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.094 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.094 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.094 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.094 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.094 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.094 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.094 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.094 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.094 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.094 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.094 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.095 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.095 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.095 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.095 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.095 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.095 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.095 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.095 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.095 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.095 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.095 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.095 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.095 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.095 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.095 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.095 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.096 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.096 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.096 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.096 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.096 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.096 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.096 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.096 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.096 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.096 INFO:teuthology.orchestra.run.smithi072.stdout: "max_concurrent_snap_create": { 2024-03-22T01:47:09.096 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "max_concurrent_snap_create", 2024-03-22T01:47:09.096 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.096 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.096 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.097 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "10", 2024-03-22T01:47:09.097 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.097 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.097 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.097 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.097 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.097 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.097 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.097 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.097 INFO:teuthology.orchestra.run.smithi072.stdout: "mirror_snapshot_schedule": { 2024-03-22T01:47:09.097 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "mirror_snapshot_schedule", 2024-03-22T01:47:09.097 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.097 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.097 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.097 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.098 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.098 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.098 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.098 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.098 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.098 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.098 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.098 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.098 INFO:teuthology.orchestra.run.smithi072.stdout: "trash_purge_schedule": { 2024-03-22T01:47:09.098 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "trash_purge_schedule", 2024-03-22T01:47:09.098 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.098 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.098 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.098 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.098 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.099 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.099 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.099 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.099 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.099 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.099 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.099 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.099 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.099 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.099 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.099 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "restful", 2024-03-22T01:47:09.099 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.099 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.099 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.099 INFO:teuthology.orchestra.run.smithi072.stdout: "enable_auth": { 2024-03-22T01:47:09.100 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "enable_auth", 2024-03-22T01:47:09.100 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.100 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.100 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.100 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.100 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.100 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.100 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.100 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.100 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.100 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.100 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.101 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.101 INFO:teuthology.orchestra.run.smithi072.stdout: "key_file": { 2024-03-22T01:47:09.101 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "key_file", 2024-03-22T01:47:09.101 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.101 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.101 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.101 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.101 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.101 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.101 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.101 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.101 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.101 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.102 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.102 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.102 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.102 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.102 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.102 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.102 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.102 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.102 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.102 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.102 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.102 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.102 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.102 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.102 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.103 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.103 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.103 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.103 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.103 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.103 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.103 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.103 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.103 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.103 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.103 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.103 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.103 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.104 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.104 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.104 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.104 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.104 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.104 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.104 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.104 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.104 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.104 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.104 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.104 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.104 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.104 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.105 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.105 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.105 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.105 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.105 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.105 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.105 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.105 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.105 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.105 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.105 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.105 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.105 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.105 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.105 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.106 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.106 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.106 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.106 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.106 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.106 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.106 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.106 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.106 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.106 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.106 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.106 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.106 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.106 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.107 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.107 INFO:teuthology.orchestra.run.smithi072.stdout: "server_addr": { 2024-03-22T01:47:09.107 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "server_addr", 2024-03-22T01:47:09.107 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.107 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.107 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.107 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.107 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.107 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.107 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.107 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.107 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.107 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.107 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.107 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.108 INFO:teuthology.orchestra.run.smithi072.stdout: "server_port": { 2024-03-22T01:47:09.108 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "server_port", 2024-03-22T01:47:09.108 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.108 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.108 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.108 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.108 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.108 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.108 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.108 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.108 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.108 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.108 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.108 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.108 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.109 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.109 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.109 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "rgw", 2024-03-22T01:47:09.109 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.109 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.109 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.109 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.109 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.109 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.109 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.109 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.109 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.109 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.109 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.109 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.110 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.110 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.110 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.110 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.110 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.110 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.110 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.110 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.110 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.110 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.110 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.110 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.110 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.110 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.111 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.111 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.111 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.111 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.111 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.111 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.111 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.111 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.111 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.111 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.111 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.111 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.111 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.111 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.111 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.111 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.112 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.112 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.112 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.112 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.112 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.112 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.112 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.112 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.112 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.112 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.112 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.112 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.112 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.112 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.112 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.113 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.113 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.113 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.113 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.113 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.113 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.113 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.113 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.113 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.113 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.113 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.113 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.113 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.113 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.113 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.114 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.114 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.114 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.114 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.114 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "rook", 2024-03-22T01:47:09.114 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.114 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.114 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.114 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.114 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.114 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.114 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.114 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.115 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.115 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.115 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.115 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.115 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.115 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.115 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.115 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.115 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.115 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.115 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.115 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.115 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.115 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.115 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.116 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.116 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.116 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.116 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.116 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.116 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.116 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.116 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.116 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.116 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.116 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.116 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.116 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.116 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.116 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.117 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.117 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.117 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.117 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.117 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.117 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.117 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.117 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.117 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.117 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.117 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.117 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.117 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.117 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.117 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.118 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.118 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.118 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.118 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.118 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.118 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.118 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.118 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.118 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.118 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.118 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.118 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.118 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.119 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.119 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.119 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.119 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.119 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.119 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.119 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.119 INFO:teuthology.orchestra.run.smithi072.stdout: "storage_class": { 2024-03-22T01:47:09.119 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "storage_class", 2024-03-22T01:47:09.119 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.119 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.119 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.119 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "local", 2024-03-22T01:47:09.119 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.120 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.120 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.120 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-22T01:47:09.120 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.120 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.120 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.120 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.120 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.120 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.120 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.120 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "selftest", 2024-03-22T01:47:09.120 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.120 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.120 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.120 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.121 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.121 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.121 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.121 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.121 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.121 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.121 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.121 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.121 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.121 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.121 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.121 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.121 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.121 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.122 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.122 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.122 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.122 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.122 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.122 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.122 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.122 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.122 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.122 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.122 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.122 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.123 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.123 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.123 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.123 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.123 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.123 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.123 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.123 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.123 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.123 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.123 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.123 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.123 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.123 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.123 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.124 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.124 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.124 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.124 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.124 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.124 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.124 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.124 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.124 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.124 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.124 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.124 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.124 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.125 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.125 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.125 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.125 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.125 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.125 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.125 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.125 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.125 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.125 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.125 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.125 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.125 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.125 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.125 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.125 INFO:teuthology.orchestra.run.smithi072.stdout: "roption1": { 2024-03-22T01:47:09.126 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "roption1", 2024-03-22T01:47:09.126 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.126 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.126 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.126 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.126 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.126 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.126 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.126 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.126 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.126 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.126 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.126 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.126 INFO:teuthology.orchestra.run.smithi072.stdout: "roption2": { 2024-03-22T01:47:09.127 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "roption2", 2024-03-22T01:47:09.127 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.127 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.127 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.127 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "xyz", 2024-03-22T01:47:09.127 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.127 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.127 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.127 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.127 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.127 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.127 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.127 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.127 INFO:teuthology.orchestra.run.smithi072.stdout: "rwoption1": { 2024-03-22T01:47:09.127 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "rwoption1", 2024-03-22T01:47:09.128 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.128 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.128 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.128 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.128 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.128 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.128 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.128 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.128 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.128 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.128 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.128 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.128 INFO:teuthology.orchestra.run.smithi072.stdout: "rwoption2": { 2024-03-22T01:47:09.129 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "rwoption2", 2024-03-22T01:47:09.129 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.129 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.129 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.129 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.129 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.129 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.129 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.129 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.129 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.129 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.129 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.129 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.129 INFO:teuthology.orchestra.run.smithi072.stdout: "rwoption3": { 2024-03-22T01:47:09.130 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "rwoption3", 2024-03-22T01:47:09.130 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "float", 2024-03-22T01:47:09.130 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.130 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.130 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.130 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.130 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.130 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.130 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.130 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.130 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.130 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.130 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.130 INFO:teuthology.orchestra.run.smithi072.stdout: "rwoption4": { 2024-03-22T01:47:09.131 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "rwoption4", 2024-03-22T01:47:09.131 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.131 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.131 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.131 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.131 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.131 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.131 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.131 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.131 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.131 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.131 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.131 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.131 INFO:teuthology.orchestra.run.smithi072.stdout: "rwoption5": { 2024-03-22T01:47:09.132 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "rwoption5", 2024-03-22T01:47:09.132 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.132 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.133 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.133 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.133 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.133 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.134 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.134 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.134 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.134 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.134 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.134 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.134 INFO:teuthology.orchestra.run.smithi072.stdout: "rwoption6": { 2024-03-22T01:47:09.134 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "rwoption6", 2024-03-22T01:47:09.134 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.134 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.134 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.134 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.135 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.135 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.135 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.135 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.135 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.135 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.135 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.135 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.135 INFO:teuthology.orchestra.run.smithi072.stdout: "rwoption7": { 2024-03-22T01:47:09.135 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "rwoption7", 2024-03-22T01:47:09.135 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.135 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.135 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.135 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.135 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "1", 2024-03-22T01:47:09.135 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "42", 2024-03-22T01:47:09.136 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.136 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.136 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.136 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.136 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.136 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.136 INFO:teuthology.orchestra.run.smithi072.stdout: "testkey": { 2024-03-22T01:47:09.136 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "testkey", 2024-03-22T01:47:09.136 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.136 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.136 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.136 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.136 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.136 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.136 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.137 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.137 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.137 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.137 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.137 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.137 INFO:teuthology.orchestra.run.smithi072.stdout: "testlkey": { 2024-03-22T01:47:09.137 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "testlkey", 2024-03-22T01:47:09.137 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.137 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.137 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.137 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.137 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.137 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.137 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.137 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.138 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.138 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.138 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.138 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.138 INFO:teuthology.orchestra.run.smithi072.stdout: "testnewline": { 2024-03-22T01:47:09.138 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "testnewline", 2024-03-22T01:47:09.138 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.138 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.138 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.138 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.138 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.138 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.138 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.139 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.139 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.139 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.139 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.139 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.139 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.139 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.139 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.139 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "snap_schedule", 2024-03-22T01:47:09.139 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.139 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.139 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.139 INFO:teuthology.orchestra.run.smithi072.stdout: "allow_m_granularity": { 2024-03-22T01:47:09.139 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "allow_m_granularity", 2024-03-22T01:47:09.139 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.140 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.140 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.140 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.140 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.140 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.140 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.140 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "allow minute scheduled snapshots", 2024-03-22T01:47:09.140 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.140 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.140 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.140 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.140 INFO:teuthology.orchestra.run.smithi072.stdout: "dump_on_update": { 2024-03-22T01:47:09.140 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "dump_on_update", 2024-03-22T01:47:09.140 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.141 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.141 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.141 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.141 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.141 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.141 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.141 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "dump database to debug log on update", 2024-03-22T01:47:09.141 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.141 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.141 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.141 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.141 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.141 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.141 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.141 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.142 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.142 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.142 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.142 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.142 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.142 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.142 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.142 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.142 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.142 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.142 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.142 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.142 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.142 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.142 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.143 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.143 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.143 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.143 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.143 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.143 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.143 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.143 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.143 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.143 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.143 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.143 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.143 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.144 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.144 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.144 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.144 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.144 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.144 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.144 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.144 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.144 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.144 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.144 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.144 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.144 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.144 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.144 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.144 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.145 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.145 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.145 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.145 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.145 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.145 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.145 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.145 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.145 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.145 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.145 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.145 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.145 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.145 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.145 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.145 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.146 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.146 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.146 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.146 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.146 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.146 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.146 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.146 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.146 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.146 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "stats", 2024-03-22T01:47:09.146 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.146 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.146 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.146 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.147 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.147 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.147 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.147 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.147 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.147 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.147 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.147 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.147 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.147 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.147 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.147 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.147 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.147 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.147 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.148 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.148 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.148 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.148 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.148 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.148 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.148 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.148 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.148 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.148 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.148 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.148 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.148 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.149 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.149 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.149 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.149 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.149 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.149 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.149 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.149 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.149 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.149 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.149 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.149 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.149 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.149 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.149 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.150 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.150 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.150 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.150 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.150 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.150 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.150 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.150 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.150 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.150 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.150 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.150 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.150 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.150 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.151 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.151 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.151 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.151 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.151 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.151 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.151 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.151 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.151 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.151 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.151 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.151 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.151 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.151 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.151 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.152 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "status", 2024-03-22T01:47:09.152 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.152 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.152 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.152 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.152 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.152 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.152 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.152 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.152 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.152 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.152 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.152 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.152 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.152 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.153 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.153 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.153 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.153 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.153 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.153 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.153 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.153 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.153 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.153 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.153 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.153 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.153 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.153 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.153 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.153 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.154 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.154 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.154 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.154 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.154 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.154 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.154 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.154 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.154 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.154 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.154 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.154 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.154 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.154 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.155 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.155 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.155 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.155 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.155 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.155 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.155 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.155 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.155 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.155 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.155 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.155 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.155 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.155 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.155 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.156 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.156 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.156 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.156 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.156 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.156 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.156 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.156 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.156 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.156 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.156 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.156 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.157 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.157 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.157 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.157 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.157 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.157 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "telegraf", 2024-03-22T01:47:09.157 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.157 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.157 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.157 INFO:teuthology.orchestra.run.smithi072.stdout: "address": { 2024-03-22T01:47:09.157 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "address", 2024-03-22T01:47:09.157 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.157 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.157 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.158 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-22T01:47:09.158 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.158 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.158 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.158 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.158 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.158 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.158 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.158 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.158 INFO:teuthology.orchestra.run.smithi072.stdout: "interval": { 2024-03-22T01:47:09.158 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "interval", 2024-03-22T01:47:09.158 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "secs", 2024-03-22T01:47:09.158 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.159 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.159 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "15", 2024-03-22T01:47:09.159 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.159 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.159 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.159 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.159 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.159 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.159 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.159 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.159 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.159 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.159 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.159 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.159 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.159 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.160 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.160 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.160 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.160 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.160 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.160 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.160 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.160 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.160 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.160 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.160 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.160 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.160 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.160 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.160 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.160 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.161 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.161 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.161 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.161 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.161 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.161 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.161 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.161 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.161 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.161 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.161 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.161 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.161 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.161 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.162 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.162 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.162 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.162 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.162 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.162 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.162 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.162 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.162 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.162 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.162 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.162 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.162 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.162 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.163 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.163 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.163 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.163 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.163 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.163 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.163 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.163 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.163 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.163 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.163 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.163 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.163 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.163 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.163 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.164 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.164 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.164 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.164 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.164 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.164 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.164 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.164 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.164 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "telemetry", 2024-03-22T01:47:09.164 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.164 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.164 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.164 INFO:teuthology.orchestra.run.smithi072.stdout: "channel_basic": { 2024-03-22T01:47:09.164 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "channel_basic", 2024-03-22T01:47:09.164 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.164 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.165 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.165 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.165 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.165 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.165 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.165 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-22T01:47:09.165 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.165 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.165 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.165 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.165 INFO:teuthology.orchestra.run.smithi072.stdout: "channel_crash": { 2024-03-22T01:47:09.165 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "channel_crash", 2024-03-22T01:47:09.165 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.165 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.166 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.166 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.166 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.166 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.166 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.166 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-22T01:47:09.166 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.166 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.166 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.166 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.166 INFO:teuthology.orchestra.run.smithi072.stdout: "channel_device": { 2024-03-22T01:47:09.166 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "channel_device", 2024-03-22T01:47:09.166 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.166 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.166 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.167 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.167 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.167 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.167 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.167 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-22T01:47:09.167 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.167 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.167 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.167 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.167 INFO:teuthology.orchestra.run.smithi072.stdout: "channel_ident": { 2024-03-22T01:47:09.167 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "channel_ident", 2024-03-22T01:47:09.167 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.167 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.167 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.168 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.168 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.168 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.168 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.168 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-22T01:47:09.168 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.168 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.168 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.168 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.168 INFO:teuthology.orchestra.run.smithi072.stdout: "channel_perf": { 2024-03-22T01:47:09.168 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "channel_perf", 2024-03-22T01:47:09.168 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.168 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.168 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.169 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.169 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.169 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.169 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.169 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-22T01:47:09.169 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.169 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.169 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.169 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.169 INFO:teuthology.orchestra.run.smithi072.stdout: "contact": { 2024-03-22T01:47:09.169 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "contact", 2024-03-22T01:47:09.169 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.169 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.169 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.170 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.170 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.170 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.170 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.170 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.170 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.170 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.170 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.170 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.170 INFO:teuthology.orchestra.run.smithi072.stdout: "description": { 2024-03-22T01:47:09.170 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "description", 2024-03-22T01:47:09.170 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.170 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.170 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.170 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.171 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.171 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.171 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.171 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.171 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.171 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.171 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.171 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.171 INFO:teuthology.orchestra.run.smithi072.stdout: "device_url": { 2024-03-22T01:47:09.171 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "device_url", 2024-03-22T01:47:09.171 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.171 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.171 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.171 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-22T01:47:09.171 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.172 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.172 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.172 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.172 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.172 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.172 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.172 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.172 INFO:teuthology.orchestra.run.smithi072.stdout: "enabled": { 2024-03-22T01:47:09.172 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "enabled", 2024-03-22T01:47:09.172 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.172 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.172 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.172 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.172 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.172 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.173 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.173 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.173 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.173 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.173 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.173 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.173 INFO:teuthology.orchestra.run.smithi072.stdout: "interval": { 2024-03-22T01:47:09.173 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "interval", 2024-03-22T01:47:09.173 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.173 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.173 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.173 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "24", 2024-03-22T01:47:09.173 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "8", 2024-03-22T01:47:09.173 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.174 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.174 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.174 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.174 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.174 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.174 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.174 INFO:teuthology.orchestra.run.smithi072.stdout: "last_opt_revision": { 2024-03-22T01:47:09.174 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "last_opt_revision", 2024-03-22T01:47:09.174 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.174 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.174 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.174 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "1", 2024-03-22T01:47:09.174 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.174 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.175 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.175 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.175 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.175 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.175 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.175 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.175 INFO:teuthology.orchestra.run.smithi072.stdout: "leaderboard": { 2024-03-22T01:47:09.175 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "leaderboard", 2024-03-22T01:47:09.175 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.175 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.175 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.175 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.175 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.175 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.175 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.176 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.176 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.176 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.176 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.176 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.176 INFO:teuthology.orchestra.run.smithi072.stdout: "leaderboard_description": { 2024-03-22T01:47:09.176 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "leaderboard_description", 2024-03-22T01:47:09.176 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.176 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.176 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.176 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.176 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.176 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.176 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.177 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.177 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.177 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.177 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.177 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.177 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.177 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.177 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.177 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.177 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.177 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.177 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.177 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.177 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.177 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.178 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.178 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.178 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.178 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.178 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.178 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.178 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.178 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.178 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.178 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.178 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.178 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.178 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.178 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.179 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.179 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.179 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.179 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.179 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.179 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.179 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.179 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.179 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.179 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.179 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.179 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.179 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.179 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.179 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.180 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.180 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.180 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.180 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.180 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.180 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.180 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.180 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.180 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.180 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.180 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.180 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.180 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.180 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.180 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.181 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.181 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.181 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.181 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.181 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.181 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.181 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.181 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.181 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.181 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.181 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.181 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.181 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.181 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.181 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.181 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.182 INFO:teuthology.orchestra.run.smithi072.stdout: "organization": { 2024-03-22T01:47:09.182 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "organization", 2024-03-22T01:47:09.182 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.182 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.182 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.182 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.182 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.182 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.182 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.182 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.182 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.182 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.182 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.182 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.182 INFO:teuthology.orchestra.run.smithi072.stdout: "proxy": { 2024-03-22T01:47:09.183 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "proxy", 2024-03-22T01:47:09.183 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.183 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.183 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.183 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.183 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.183 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.183 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.183 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.183 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.183 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.183 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.183 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.184 INFO:teuthology.orchestra.run.smithi072.stdout: "url": { 2024-03-22T01:47:09.184 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "url", 2024-03-22T01:47:09.184 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.184 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.184 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.184 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-22T01:47:09.184 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.184 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.184 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.184 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.184 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.184 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.184 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.184 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.184 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.185 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.185 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.185 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "test_orchestrator", 2024-03-22T01:47:09.185 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.185 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.185 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.185 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.185 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.185 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.185 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.185 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.185 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.185 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.185 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.185 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.186 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.186 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.186 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.186 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.186 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.186 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.186 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.186 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.186 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.186 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.186 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.186 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.186 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.186 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.187 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.187 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.187 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.187 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.187 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.187 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.187 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.187 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.187 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.187 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.187 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.187 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.187 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.187 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.187 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.188 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.188 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.188 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.188 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.188 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.188 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.188 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.188 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.188 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.188 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.188 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.188 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.188 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.188 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.189 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.189 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.189 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.189 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.189 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.189 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.189 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.189 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.189 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.189 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.189 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.189 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.189 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.189 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.189 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.190 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.190 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.190 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.190 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.190 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.190 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.190 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "volumes", 2024-03-22T01:47:09.190 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.190 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.190 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.190 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.190 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.190 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.190 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.191 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.191 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.191 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.191 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.191 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.191 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.191 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.191 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.191 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.191 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.191 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.191 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.191 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.191 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.192 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.192 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.192 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.192 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.192 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.192 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.192 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.192 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.192 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.192 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.192 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.192 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.192 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.192 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.192 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.193 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.193 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.193 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.193 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.193 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.193 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.193 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.193 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.193 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.193 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.193 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.193 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.193 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.193 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.193 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.194 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.194 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.194 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.194 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.194 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.194 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.194 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.194 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.194 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.194 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.194 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.194 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.194 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.194 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.194 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.195 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.195 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.195 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.195 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.195 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.195 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.195 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.195 INFO:teuthology.orchestra.run.smithi072.stdout: "max_concurrent_clones": { 2024-03-22T01:47:09.195 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "max_concurrent_clones", 2024-03-22T01:47:09.195 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.195 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.195 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.195 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "4", 2024-03-22T01:47:09.195 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.195 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.196 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.196 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-22T01:47:09.196 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.196 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.196 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.196 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.196 INFO:teuthology.orchestra.run.smithi072.stdout: "periodic_async_work": { 2024-03-22T01:47:09.196 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "periodic_async_work", 2024-03-22T01:47:09.196 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.196 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.196 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.196 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.196 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.196 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.197 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.197 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Periodically check for async work", 2024-03-22T01:47:09.197 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.197 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.197 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.197 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.197 INFO:teuthology.orchestra.run.smithi072.stdout: "snapshot_clone_delay": { 2024-03-22T01:47:09.197 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "snapshot_clone_delay", 2024-03-22T01:47:09.197 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.197 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.197 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.197 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "0", 2024-03-22T01:47:09.197 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.197 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.197 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.198 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-22T01:47:09.198 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.198 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.198 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.198 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.198 INFO:teuthology.orchestra.run.smithi072.stdout: "snapshot_clone_no_wait": { 2024-03-22T01:47:09.198 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "snapshot_clone_no_wait", 2024-03-22T01:47:09.198 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.198 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.198 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.198 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.198 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.198 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.198 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.199 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-22T01:47:09.199 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.199 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.199 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.199 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.199 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.199 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.199 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.199 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "zabbix", 2024-03-22T01:47:09.199 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.199 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.199 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.199 INFO:teuthology.orchestra.run.smithi072.stdout: "discovery_interval": { 2024-03-22T01:47:09.199 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "discovery_interval", 2024-03-22T01:47:09.199 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "uint", 2024-03-22T01:47:09.199 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.200 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.200 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "100", 2024-03-22T01:47:09.200 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.200 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.200 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.200 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.200 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.200 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.200 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.200 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.200 INFO:teuthology.orchestra.run.smithi072.stdout: "identifier": { 2024-03-22T01:47:09.200 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "identifier", 2024-03-22T01:47:09.200 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.200 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.200 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.201 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.201 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.201 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.201 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.201 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.201 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.201 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.201 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.201 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.201 INFO:teuthology.orchestra.run.smithi072.stdout: "interval": { 2024-03-22T01:47:09.201 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "interval", 2024-03-22T01:47:09.201 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "secs", 2024-03-22T01:47:09.201 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.202 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.202 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "60", 2024-03-22T01:47:09.202 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.202 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.202 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.202 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.202 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.202 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.202 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.202 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.202 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.202 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.202 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.202 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.202 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.203 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.203 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.203 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.203 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.203 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.203 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.203 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.203 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.203 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.203 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.203 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.203 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.203 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.204 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.204 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.204 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.204 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.204 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.204 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.204 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.204 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.204 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.204 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.204 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.204 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.204 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.204 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.204 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.205 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.205 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.205 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.205 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.205 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.205 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.205 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.205 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.205 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.205 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.205 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.205 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.205 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.206 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.206 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.206 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.206 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.206 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.206 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.206 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.206 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.206 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.206 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.206 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.206 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.206 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.206 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.206 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.207 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.207 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.207 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.207 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.207 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.207 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.207 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.207 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.207 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.207 INFO:teuthology.orchestra.run.smithi072.stdout: "zabbix_host": { 2024-03-22T01:47:09.207 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "zabbix_host", 2024-03-22T01:47:09.207 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.207 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.207 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.208 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.208 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.208 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.208 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.208 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.208 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.208 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.208 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.208 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.208 INFO:teuthology.orchestra.run.smithi072.stdout: "zabbix_port": { 2024-03-22T01:47:09.208 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "zabbix_port", 2024-03-22T01:47:09.208 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.208 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.208 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.208 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "10051", 2024-03-22T01:47:09.209 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.209 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.209 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.209 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.209 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.209 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.209 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.209 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.209 INFO:teuthology.orchestra.run.smithi072.stdout: "zabbix_sender": { 2024-03-22T01:47:09.209 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "zabbix_sender", 2024-03-22T01:47:09.209 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.209 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.209 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.209 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-22T01:47:09.210 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.210 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.210 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.210 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.210 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.210 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.210 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.210 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.210 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.210 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.210 INFO:teuthology.orchestra.run.smithi072.stdout: ] 2024-03-22T01:47:09.210 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.210 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.210 INFO:teuthology.orchestra.run.smithi072.stdout: "gid": 24985, 2024-03-22T01:47:09.211 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "z", 2024-03-22T01:47:09.211 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr_features": 4540701547738038271, 2024-03-22T01:47:09.211 INFO:teuthology.orchestra.run.smithi072.stdout: "available_modules": [ 2024-03-22T01:47:09.211 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.211 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "alerts", 2024-03-22T01:47:09.211 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.211 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.211 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.211 INFO:teuthology.orchestra.run.smithi072.stdout: "interval": { 2024-03-22T01:47:09.211 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "interval", 2024-03-22T01:47:09.211 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "secs", 2024-03-22T01:47:09.211 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.211 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.211 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "60", 2024-03-22T01:47:09.211 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.211 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.212 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.212 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "How frequently to reexamine health status", 2024-03-22T01:47:09.212 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.212 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.212 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.212 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.212 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.212 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.212 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.212 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.212 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.212 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.212 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.212 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.212 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.213 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.213 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.213 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.213 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.213 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.213 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.213 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.213 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.213 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.213 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.213 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.213 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.213 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.213 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.214 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.214 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.214 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.214 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.214 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.214 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.214 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.214 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.214 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.214 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.214 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.214 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.214 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.215 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.215 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.215 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.215 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.215 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.215 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.215 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.215 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.215 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.215 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.215 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.215 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.215 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.215 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.215 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.215 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.215 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.216 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.216 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.216 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.216 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.216 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.216 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.216 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.216 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.216 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.216 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.216 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.216 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.216 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.216 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.217 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.217 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.217 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.217 INFO:teuthology.orchestra.run.smithi072.stdout: "smtp_destination": { 2024-03-22T01:47:09.217 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "smtp_destination", 2024-03-22T01:47:09.217 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.217 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.217 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.217 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.217 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.217 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.217 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.217 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Email address to send alerts to", 2024-03-22T01:47:09.217 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.218 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.218 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.218 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.218 INFO:teuthology.orchestra.run.smithi072.stdout: "smtp_from_name": { 2024-03-22T01:47:09.218 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "smtp_from_name", 2024-03-22T01:47:09.218 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.218 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.218 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.218 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "Ceph", 2024-03-22T01:47:09.218 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.218 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.218 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.218 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Email From: name", 2024-03-22T01:47:09.218 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.218 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.219 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.219 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.219 INFO:teuthology.orchestra.run.smithi072.stdout: "smtp_host": { 2024-03-22T01:47:09.219 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "smtp_host", 2024-03-22T01:47:09.219 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.219 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.219 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.219 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.219 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.219 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.219 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.219 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "SMTP server", 2024-03-22T01:47:09.219 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.219 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.219 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.220 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.220 INFO:teuthology.orchestra.run.smithi072.stdout: "smtp_password": { 2024-03-22T01:47:09.220 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "smtp_password", 2024-03-22T01:47:09.220 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.220 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.220 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.220 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.220 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.220 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.220 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.220 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Password to authenticate with", 2024-03-22T01:47:09.220 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.220 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.220 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.220 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.220 INFO:teuthology.orchestra.run.smithi072.stdout: "smtp_port": { 2024-03-22T01:47:09.221 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "smtp_port", 2024-03-22T01:47:09.221 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.221 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.221 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.221 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "465", 2024-03-22T01:47:09.221 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.221 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.221 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.221 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "SMTP port", 2024-03-22T01:47:09.221 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.221 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.221 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.221 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.221 INFO:teuthology.orchestra.run.smithi072.stdout: "smtp_sender": { 2024-03-22T01:47:09.221 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "smtp_sender", 2024-03-22T01:47:09.221 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.221 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.222 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.222 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.222 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.222 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.222 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.222 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "SMTP envelope sender", 2024-03-22T01:47:09.222 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.222 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.222 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.222 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.222 INFO:teuthology.orchestra.run.smithi072.stdout: "smtp_ssl": { 2024-03-22T01:47:09.222 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "smtp_ssl", 2024-03-22T01:47:09.222 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.222 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.223 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.223 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.223 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.223 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.223 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.223 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-22T01:47:09.223 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.223 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.223 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.223 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.223 INFO:teuthology.orchestra.run.smithi072.stdout: "smtp_user": { 2024-03-22T01:47:09.223 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "smtp_user", 2024-03-22T01:47:09.223 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.223 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.224 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.224 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.224 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.224 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.224 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.224 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "User to authenticate as", 2024-03-22T01:47:09.224 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.224 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.224 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.224 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.224 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.224 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.224 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.224 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "balancer", 2024-03-22T01:47:09.224 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.225 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.225 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.225 INFO:teuthology.orchestra.run.smithi072.stdout: "active": { 2024-03-22T01:47:09.225 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "active", 2024-03-22T01:47:09.225 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.225 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.225 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.225 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.225 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.225 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.225 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.225 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "automatically balance PGs across cluster", 2024-03-22T01:47:09.225 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.225 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.225 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.226 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.226 INFO:teuthology.orchestra.run.smithi072.stdout: "begin_time": { 2024-03-22T01:47:09.226 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "begin_time", 2024-03-22T01:47:09.226 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.226 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.226 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.226 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "0000", 2024-03-22T01:47:09.226 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.226 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.226 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.226 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "beginning time of day to automatically balance", 2024-03-22T01:47:09.226 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-22T01:47:09.227 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.227 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.227 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.227 INFO:teuthology.orchestra.run.smithi072.stdout: "begin_weekday": { 2024-03-22T01:47:09.227 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "begin_weekday", 2024-03-22T01:47:09.227 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "uint", 2024-03-22T01:47:09.227 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.227 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.228 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "0", 2024-03-22T01:47:09.228 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "0", 2024-03-22T01:47:09.228 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "6", 2024-03-22T01:47:09.228 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.228 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-22T01:47:09.228 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-22T01:47:09.228 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.228 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.228 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.228 INFO:teuthology.orchestra.run.smithi072.stdout: "crush_compat_max_iterations": { 2024-03-22T01:47:09.228 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "crush_compat_max_iterations", 2024-03-22T01:47:09.228 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "uint", 2024-03-22T01:47:09.228 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.229 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.229 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "25", 2024-03-22T01:47:09.229 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "1", 2024-03-22T01:47:09.229 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "250", 2024-03-22T01:47:09.229 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.229 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-22T01:47:09.229 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.229 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.229 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.229 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.229 INFO:teuthology.orchestra.run.smithi072.stdout: "crush_compat_metrics": { 2024-03-22T01:47:09.229 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "crush_compat_metrics", 2024-03-22T01:47:09.229 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.229 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.229 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.230 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "pgs,objects,bytes", 2024-03-22T01:47:09.230 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.230 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.230 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.230 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-22T01:47:09.230 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-22T01:47:09.230 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.230 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.230 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.230 INFO:teuthology.orchestra.run.smithi072.stdout: "crush_compat_step": { 2024-03-22T01:47:09.230 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "crush_compat_step", 2024-03-22T01:47:09.230 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "float", 2024-03-22T01:47:09.230 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.230 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.230 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "0.5", 2024-03-22T01:47:09.231 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "0.001", 2024-03-22T01:47:09.231 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "0.999", 2024-03-22T01:47:09.231 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.231 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "aggressiveness of optimization", 2024-03-22T01:47:09.231 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-22T01:47:09.231 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.231 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.231 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.231 INFO:teuthology.orchestra.run.smithi072.stdout: "end_time": { 2024-03-22T01:47:09.231 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "end_time", 2024-03-22T01:47:09.231 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.231 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.231 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.231 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "2359", 2024-03-22T01:47:09.231 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.231 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.232 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.232 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "ending time of day to automatically balance", 2024-03-22T01:47:09.232 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-22T01:47:09.232 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.232 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.232 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.232 INFO:teuthology.orchestra.run.smithi072.stdout: "end_weekday": { 2024-03-22T01:47:09.232 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "end_weekday", 2024-03-22T01:47:09.232 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "uint", 2024-03-22T01:47:09.232 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.232 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.232 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "0", 2024-03-22T01:47:09.232 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "0", 2024-03-22T01:47:09.232 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "6", 2024-03-22T01:47:09.232 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.233 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-22T01:47:09.233 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-22T01:47:09.233 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.233 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.233 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.233 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.233 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.233 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.233 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.233 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.233 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.233 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.233 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.233 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.233 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.233 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.234 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.234 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.234 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.234 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.234 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.234 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.234 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.234 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.234 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.234 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.234 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.234 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.234 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.234 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.235 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.235 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.235 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.235 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.235 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.235 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.235 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.235 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.235 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.235 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.235 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.235 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.236 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.236 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.236 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.236 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.236 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.236 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.236 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.236 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.236 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.236 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.236 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.236 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.236 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.236 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.236 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.237 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.237 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.237 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.237 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.237 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.237 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.237 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.237 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.237 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.237 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.237 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.237 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.237 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.237 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.237 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.238 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.238 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.238 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.238 INFO:teuthology.orchestra.run.smithi072.stdout: "min_score": { 2024-03-22T01:47:09.238 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "min_score", 2024-03-22T01:47:09.238 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "float", 2024-03-22T01:47:09.238 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.238 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.238 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "0", 2024-03-22T01:47:09.238 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.238 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.238 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.238 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-22T01:47:09.238 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.238 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.239 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.239 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.239 INFO:teuthology.orchestra.run.smithi072.stdout: "mode": { 2024-03-22T01:47:09.239 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "mode", 2024-03-22T01:47:09.239 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.239 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.239 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.239 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "upmap", 2024-03-22T01:47:09.239 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.239 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.239 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.239 INFO:teuthology.orchestra.run.smithi072.stdout: "crush-compat", 2024-03-22T01:47:09.239 INFO:teuthology.orchestra.run.smithi072.stdout: "none", 2024-03-22T01:47:09.239 INFO:teuthology.orchestra.run.smithi072.stdout: "read", 2024-03-22T01:47:09.239 INFO:teuthology.orchestra.run.smithi072.stdout: "upmap", 2024-03-22T01:47:09.240 INFO:teuthology.orchestra.run.smithi072.stdout: "upmap-read" 2024-03-22T01:47:09.240 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.240 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Balancer mode", 2024-03-22T01:47:09.240 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.240 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.240 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.240 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.240 INFO:teuthology.orchestra.run.smithi072.stdout: "pool_ids": { 2024-03-22T01:47:09.240 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "pool_ids", 2024-03-22T01:47:09.240 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.240 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.240 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.240 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.240 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.241 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.241 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.241 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-22T01:47:09.241 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.241 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.241 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.241 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.241 INFO:teuthology.orchestra.run.smithi072.stdout: "sleep_interval": { 2024-03-22T01:47:09.241 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "sleep_interval", 2024-03-22T01:47:09.241 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "secs", 2024-03-22T01:47:09.241 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.241 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.241 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "60", 2024-03-22T01:47:09.241 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.241 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.242 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.242 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-22T01:47:09.242 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.242 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.242 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.242 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.242 INFO:teuthology.orchestra.run.smithi072.stdout: "upmap_max_deviation": { 2024-03-22T01:47:09.242 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "upmap_max_deviation", 2024-03-22T01:47:09.242 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.242 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.242 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.242 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "5", 2024-03-22T01:47:09.242 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "1", 2024-03-22T01:47:09.242 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.242 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.243 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-22T01:47:09.243 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-22T01:47:09.243 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.243 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.243 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.243 INFO:teuthology.orchestra.run.smithi072.stdout: "upmap_max_optimizations": { 2024-03-22T01:47:09.243 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "upmap_max_optimizations", 2024-03-22T01:47:09.243 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "uint", 2024-03-22T01:47:09.243 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.243 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.243 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "10", 2024-03-22T01:47:09.243 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.243 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.243 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.244 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-22T01:47:09.244 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.244 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.244 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.244 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.244 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.244 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.244 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.244 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "cephadm", 2024-03-22T01:47:09.244 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.244 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.244 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.244 INFO:teuthology.orchestra.run.smithi072.stdout: "agent_down_multiplier": { 2024-03-22T01:47:09.244 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "agent_down_multiplier", 2024-03-22T01:47:09.244 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "float", 2024-03-22T01:47:09.245 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.245 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.245 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "3.0", 2024-03-22T01:47:09.245 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.245 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.245 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.245 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-22T01:47:09.245 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.245 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.245 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.245 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.245 INFO:teuthology.orchestra.run.smithi072.stdout: "agent_refresh_rate": { 2024-03-22T01:47:09.245 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "agent_refresh_rate", 2024-03-22T01:47:09.245 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "secs", 2024-03-22T01:47:09.245 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.245 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.246 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "20", 2024-03-22T01:47:09.246 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.246 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.246 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.246 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-22T01:47:09.246 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.246 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.246 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.246 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.246 INFO:teuthology.orchestra.run.smithi072.stdout: "agent_starting_port": { 2024-03-22T01:47:09.246 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "agent_starting_port", 2024-03-22T01:47:09.246 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.246 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.246 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.246 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "4721", 2024-03-22T01:47:09.247 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.247 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.247 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.247 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-22T01:47:09.247 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.247 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.247 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.247 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.247 INFO:teuthology.orchestra.run.smithi072.stdout: "allow_ptrace": { 2024-03-22T01:47:09.247 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "allow_ptrace", 2024-03-22T01:47:09.247 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.247 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.247 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.248 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.248 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.248 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.248 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.248 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-22T01:47:09.248 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-22T01:47:09.248 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.248 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.248 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.248 INFO:teuthology.orchestra.run.smithi072.stdout: "autotune_interval": { 2024-03-22T01:47:09.248 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "autotune_interval", 2024-03-22T01:47:09.248 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "secs", 2024-03-22T01:47:09.248 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.248 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.248 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "600", 2024-03-22T01:47:09.249 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.249 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.249 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.249 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-22T01:47:09.249 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.249 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.249 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.249 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.249 INFO:teuthology.orchestra.run.smithi072.stdout: "autotune_memory_target_ratio": { 2024-03-22T01:47:09.249 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "autotune_memory_target_ratio", 2024-03-22T01:47:09.249 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "float", 2024-03-22T01:47:09.249 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.249 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.249 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "0.7", 2024-03-22T01:47:09.249 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.250 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.250 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.250 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-22T01:47:09.250 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.250 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.250 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.250 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.250 INFO:teuthology.orchestra.run.smithi072.stdout: "cephadm_log_destination": { 2024-03-22T01:47:09.250 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "cephadm_log_destination", 2024-03-22T01:47:09.250 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.250 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.250 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.250 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.250 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.251 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.251 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.251 INFO:teuthology.orchestra.run.smithi072.stdout: "file", 2024-03-22T01:47:09.251 INFO:teuthology.orchestra.run.smithi072.stdout: "file,syslog", 2024-03-22T01:47:09.251 INFO:teuthology.orchestra.run.smithi072.stdout: "syslog" 2024-03-22T01:47:09.251 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.251 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-22T01:47:09.251 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.251 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.251 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.251 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.251 INFO:teuthology.orchestra.run.smithi072.stdout: "cgroups_split": { 2024-03-22T01:47:09.251 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "cgroups_split", 2024-03-22T01:47:09.251 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.252 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.252 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.252 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.252 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.252 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.252 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.252 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-22T01:47:09.252 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.252 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.252 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.252 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.252 INFO:teuthology.orchestra.run.smithi072.stdout: "config_checks_enabled": { 2024-03-22T01:47:09.252 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "config_checks_enabled", 2024-03-22T01:47:09.252 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.252 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.252 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.253 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.253 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.253 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.253 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.253 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-22T01:47:09.253 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.253 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.253 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.253 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.253 INFO:teuthology.orchestra.run.smithi072.stdout: "config_dashboard": { 2024-03-22T01:47:09.253 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "config_dashboard", 2024-03-22T01:47:09.253 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.253 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.253 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.253 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.253 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.254 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.254 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.254 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-22T01:47:09.254 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.254 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.254 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.254 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.254 INFO:teuthology.orchestra.run.smithi072.stdout: "container_image_alertmanager": { 2024-03-22T01:47:09.254 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "container_image_alertmanager", 2024-03-22T01:47:09.254 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.254 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.254 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.254 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-22T01:47:09.254 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.255 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.255 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.255 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Prometheus container image", 2024-03-22T01:47:09.255 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.255 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.255 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.255 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.255 INFO:teuthology.orchestra.run.smithi072.stdout: "container_image_base": { 2024-03-22T01:47:09.255 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "container_image_base", 2024-03-22T01:47:09.255 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.255 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.255 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.255 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-22T01:47:09.255 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.255 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.256 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.256 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Container image name, without the tag", 2024-03-22T01:47:09.256 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.256 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.256 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.256 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.256 INFO:teuthology.orchestra.run.smithi072.stdout: "container_image_elasticsearch": { 2024-03-22T01:47:09.256 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "container_image_elasticsearch", 2024-03-22T01:47:09.256 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.256 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.256 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.256 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-22T01:47:09.256 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.256 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.256 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.257 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "elasticsearch container image", 2024-03-22T01:47:09.257 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.257 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.257 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.257 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.257 INFO:teuthology.orchestra.run.smithi072.stdout: "container_image_grafana": { 2024-03-22T01:47:09.257 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "container_image_grafana", 2024-03-22T01:47:09.257 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.257 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.257 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.257 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-03-22T01:47:09.257 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.257 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.257 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.257 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Prometheus container image", 2024-03-22T01:47:09.258 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.258 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.258 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.258 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.258 INFO:teuthology.orchestra.run.smithi072.stdout: "container_image_haproxy": { 2024-03-22T01:47:09.258 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "container_image_haproxy", 2024-03-22T01:47:09.258 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.258 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.258 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.258 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-22T01:47:09.258 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.258 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.258 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.259 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "HAproxy container image", 2024-03-22T01:47:09.259 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.259 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.259 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.259 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.259 INFO:teuthology.orchestra.run.smithi072.stdout: "container_image_jaeger_agent": { 2024-03-22T01:47:09.259 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "container_image_jaeger_agent", 2024-03-22T01:47:09.259 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.259 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.259 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.259 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-22T01:47:09.259 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.259 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.260 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.260 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Jaeger agent container image", 2024-03-22T01:47:09.260 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.260 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.260 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.260 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.260 INFO:teuthology.orchestra.run.smithi072.stdout: "container_image_jaeger_collector": { 2024-03-22T01:47:09.260 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "container_image_jaeger_collector", 2024-03-22T01:47:09.260 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.260 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.260 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.260 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-22T01:47:09.260 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.260 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.260 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.261 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Jaeger collector container image", 2024-03-22T01:47:09.261 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.261 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.261 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.261 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.261 INFO:teuthology.orchestra.run.smithi072.stdout: "container_image_jaeger_query": { 2024-03-22T01:47:09.261 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "container_image_jaeger_query", 2024-03-22T01:47:09.261 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.261 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.261 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.261 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-22T01:47:09.261 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.261 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.261 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.262 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Jaeger query container image", 2024-03-22T01:47:09.262 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.262 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.262 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.262 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.262 INFO:teuthology.orchestra.run.smithi072.stdout: "container_image_keepalived": { 2024-03-22T01:47:09.262 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "container_image_keepalived", 2024-03-22T01:47:09.262 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.262 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.262 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.262 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-22T01:47:09.262 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.262 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.262 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.262 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Keepalived container image", 2024-03-22T01:47:09.263 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.263 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.263 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.263 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.263 INFO:teuthology.orchestra.run.smithi072.stdout: "container_image_loki": { 2024-03-22T01:47:09.263 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "container_image_loki", 2024-03-22T01:47:09.263 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.263 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.263 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.263 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-22T01:47:09.263 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.263 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.263 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.263 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Loki container image", 2024-03-22T01:47:09.264 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.264 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.264 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.264 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.264 INFO:teuthology.orchestra.run.smithi072.stdout: "container_image_node_exporter": { 2024-03-22T01:47:09.264 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "container_image_node_exporter", 2024-03-22T01:47:09.264 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.264 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.264 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.264 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-22T01:47:09.264 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.264 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.264 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.264 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Prometheus container image", 2024-03-22T01:47:09.264 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.265 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.265 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.265 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.265 INFO:teuthology.orchestra.run.smithi072.stdout: "container_image_nvmeof": { 2024-03-22T01:47:09.265 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "container_image_nvmeof", 2024-03-22T01:47:09.265 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.265 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.265 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.265 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-22T01:47:09.265 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.265 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.265 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.265 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Nvme-of container image", 2024-03-22T01:47:09.265 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.265 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.266 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.266 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.266 INFO:teuthology.orchestra.run.smithi072.stdout: "container_image_prometheus": { 2024-03-22T01:47:09.266 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "container_image_prometheus", 2024-03-22T01:47:09.266 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.266 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.266 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.266 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-22T01:47:09.266 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.266 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.266 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.266 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Prometheus container image", 2024-03-22T01:47:09.267 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.267 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.267 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.267 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.267 INFO:teuthology.orchestra.run.smithi072.stdout: "container_image_promtail": { 2024-03-22T01:47:09.267 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "container_image_promtail", 2024-03-22T01:47:09.267 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.267 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.267 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.267 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-22T01:47:09.267 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.267 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.267 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.267 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Promtail container image", 2024-03-22T01:47:09.267 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.267 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.268 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.268 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.268 INFO:teuthology.orchestra.run.smithi072.stdout: "container_image_snmp_gateway": { 2024-03-22T01:47:09.268 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "container_image_snmp_gateway", 2024-03-22T01:47:09.268 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.268 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.268 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.268 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-22T01:47:09.268 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.268 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.268 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.268 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "SNMP Gateway container image", 2024-03-22T01:47:09.268 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.268 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.268 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.269 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.269 INFO:teuthology.orchestra.run.smithi072.stdout: "container_init": { 2024-03-22T01:47:09.269 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "container_init", 2024-03-22T01:47:09.269 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.269 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.269 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.269 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.269 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.269 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.269 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.269 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Run podman/docker with `--init`", 2024-03-22T01:47:09.269 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.269 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.270 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.270 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.270 INFO:teuthology.orchestra.run.smithi072.stdout: "daemon_cache_timeout": { 2024-03-22T01:47:09.270 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "daemon_cache_timeout", 2024-03-22T01:47:09.270 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "secs", 2024-03-22T01:47:09.270 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.270 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.270 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "600", 2024-03-22T01:47:09.270 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.270 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.270 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.270 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-22T01:47:09.270 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.271 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.271 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.271 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.271 INFO:teuthology.orchestra.run.smithi072.stdout: "default_cephadm_command_timeout": { 2024-03-22T01:47:09.271 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "default_cephadm_command_timeout", 2024-03-22T01:47:09.271 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.271 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.271 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.271 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "900", 2024-03-22T01:47:09.271 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.271 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.271 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.271 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-22T01:47:09.271 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.272 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.272 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.272 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.272 INFO:teuthology.orchestra.run.smithi072.stdout: "default_registry": { 2024-03-22T01:47:09.272 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "default_registry", 2024-03-22T01:47:09.272 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.272 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.272 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.272 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "docker.io", 2024-03-22T01:47:09.272 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.272 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.272 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.272 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-22T01:47:09.272 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.272 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.273 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.273 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.273 INFO:teuthology.orchestra.run.smithi072.stdout: "device_cache_timeout": { 2024-03-22T01:47:09.273 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "device_cache_timeout", 2024-03-22T01:47:09.273 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "secs", 2024-03-22T01:47:09.273 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.273 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.273 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "1800", 2024-03-22T01:47:09.273 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.273 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.273 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.273 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "seconds to cache device inventory", 2024-03-22T01:47:09.273 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.274 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.274 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.274 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.274 INFO:teuthology.orchestra.run.smithi072.stdout: "device_enhanced_scan": { 2024-03-22T01:47:09.274 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "device_enhanced_scan", 2024-03-22T01:47:09.274 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.274 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.274 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.274 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.274 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.274 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.274 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.274 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-22T01:47:09.275 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.275 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.275 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.275 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.275 INFO:teuthology.orchestra.run.smithi072.stdout: "facts_cache_timeout": { 2024-03-22T01:47:09.275 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "facts_cache_timeout", 2024-03-22T01:47:09.275 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "secs", 2024-03-22T01:47:09.275 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.275 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.275 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "60", 2024-03-22T01:47:09.275 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.275 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.275 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.275 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "seconds to cache host facts data", 2024-03-22T01:47:09.276 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.276 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.276 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.276 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.276 INFO:teuthology.orchestra.run.smithi072.stdout: "grafana_dashboards_path": { 2024-03-22T01:47:09.276 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "grafana_dashboards_path", 2024-03-22T01:47:09.276 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.276 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.276 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.276 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-03-22T01:47:09.276 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.276 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.276 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.276 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-03-22T01:47:09.277 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.277 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.277 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.277 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.277 INFO:teuthology.orchestra.run.smithi072.stdout: "host_check_interval": { 2024-03-22T01:47:09.277 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "host_check_interval", 2024-03-22T01:47:09.277 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "secs", 2024-03-22T01:47:09.277 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.277 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.277 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "600", 2024-03-22T01:47:09.277 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.277 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.277 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.277 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "how frequently to perform a host check", 2024-03-22T01:47:09.277 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.278 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.278 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.278 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.278 INFO:teuthology.orchestra.run.smithi072.stdout: "hw_monitoring": { 2024-03-22T01:47:09.278 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "hw_monitoring", 2024-03-22T01:47:09.278 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.278 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.278 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.278 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.278 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.278 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.278 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.278 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-22T01:47:09.278 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.279 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.279 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.279 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.279 INFO:teuthology.orchestra.run.smithi072.stdout: "inventory_list_all": { 2024-03-22T01:47:09.279 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "inventory_list_all", 2024-03-22T01:47:09.279 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.279 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.279 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.279 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.279 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.279 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.279 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.280 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-22T01:47:09.280 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.280 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.280 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.280 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.280 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.280 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.280 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.280 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.280 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.280 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.280 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.280 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.280 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.280 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.281 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.281 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.281 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.281 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.281 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.281 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.281 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.281 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.281 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.281 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.281 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.281 INFO:teuthology.orchestra.run.smithi072.stdout: "log_refresh_metadata": { 2024-03-22T01:47:09.281 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_refresh_metadata", 2024-03-22T01:47:09.282 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.282 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.282 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.282 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.282 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.282 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.282 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.282 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-22T01:47:09.282 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.282 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.282 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.282 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.282 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.283 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.283 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.283 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.283 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.283 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.283 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.283 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.283 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.283 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-22T01:47:09.283 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.283 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.283 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.283 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.283 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.284 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.284 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.284 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.284 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.284 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.284 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.284 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.284 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.284 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.284 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.284 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.284 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.284 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.284 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.284 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.285 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.285 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.285 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.285 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.285 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.285 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.285 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.285 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.285 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.285 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.285 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.285 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.285 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.285 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.286 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.286 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.286 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.286 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.286 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.286 INFO:teuthology.orchestra.run.smithi072.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-22T01:47:09.286 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-22T01:47:09.286 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.286 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.286 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.286 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.286 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.286 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.286 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.287 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-22T01:47:09.287 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.287 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.287 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.287 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.287 INFO:teuthology.orchestra.run.smithi072.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-22T01:47:09.287 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-22T01:47:09.287 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.287 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.287 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.287 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "*", 2024-03-22T01:47:09.287 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.287 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.287 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.288 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-22T01:47:09.288 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.288 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.288 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.288 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.288 INFO:teuthology.orchestra.run.smithi072.stdout: "max_count_per_host": { 2024-03-22T01:47:09.288 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "max_count_per_host", 2024-03-22T01:47:09.288 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.288 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.288 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.288 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "10", 2024-03-22T01:47:09.288 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.288 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.289 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.289 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "max number of daemons per service per host", 2024-03-22T01:47:09.289 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.289 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.289 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.289 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.289 INFO:teuthology.orchestra.run.smithi072.stdout: "max_osd_draining_count": { 2024-03-22T01:47:09.289 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "max_osd_draining_count", 2024-03-22T01:47:09.289 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.289 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.289 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.289 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "10", 2024-03-22T01:47:09.289 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.289 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.290 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.290 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-22T01:47:09.290 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.290 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.290 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.290 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.290 INFO:teuthology.orchestra.run.smithi072.stdout: "migration_current": { 2024-03-22T01:47:09.290 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "migration_current", 2024-03-22T01:47:09.290 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.290 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.290 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.290 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.290 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.291 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.291 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.291 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "internal - do not modify", 2024-03-22T01:47:09.291 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.291 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.291 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.291 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.291 INFO:teuthology.orchestra.run.smithi072.stdout: "mode": { 2024-03-22T01:47:09.291 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "mode", 2024-03-22T01:47:09.291 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.291 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.291 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.291 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "root", 2024-03-22T01:47:09.291 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.292 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.292 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.292 INFO:teuthology.orchestra.run.smithi072.stdout: "cephadm-package", 2024-03-22T01:47:09.292 INFO:teuthology.orchestra.run.smithi072.stdout: "root" 2024-03-22T01:47:09.292 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.292 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "mode for remote execution of cephadm", 2024-03-22T01:47:09.292 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.292 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.292 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.292 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.292 INFO:teuthology.orchestra.run.smithi072.stdout: "oob_default_addr": { 2024-03-22T01:47:09.292 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "oob_default_addr", 2024-03-22T01:47:09.292 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.292 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.293 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.293 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "169.254.1.1", 2024-03-22T01:47:09.293 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.293 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.293 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.293 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-22T01:47:09.293 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.293 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.293 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.293 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.293 INFO:teuthology.orchestra.run.smithi072.stdout: "prometheus_alerts_path": { 2024-03-22T01:47:09.293 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "prometheus_alerts_path", 2024-03-22T01:47:09.293 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.293 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.294 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.294 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-22T01:47:09.294 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.294 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.294 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.294 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-22T01:47:09.294 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.294 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.294 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.294 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.294 INFO:teuthology.orchestra.run.smithi072.stdout: "registry_insecure": { 2024-03-22T01:47:09.294 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "registry_insecure", 2024-03-22T01:47:09.294 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.294 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.295 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.295 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.295 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.295 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.295 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.295 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-22T01:47:09.295 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.295 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.295 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.295 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.295 INFO:teuthology.orchestra.run.smithi072.stdout: "registry_password": { 2024-03-22T01:47:09.295 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "registry_password", 2024-03-22T01:47:09.295 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.295 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.296 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.296 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.296 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.296 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.296 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.296 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-22T01:47:09.296 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.296 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.296 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.296 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.296 INFO:teuthology.orchestra.run.smithi072.stdout: "registry_url": { 2024-03-22T01:47:09.296 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "registry_url", 2024-03-22T01:47:09.296 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.296 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.297 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.297 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.297 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.297 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.297 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.297 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-22T01:47:09.297 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.297 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.297 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.297 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.297 INFO:teuthology.orchestra.run.smithi072.stdout: "registry_username": { 2024-03-22T01:47:09.297 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "registry_username", 2024-03-22T01:47:09.297 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.298 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.298 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.298 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.298 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.298 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.298 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.298 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-22T01:47:09.298 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.298 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.298 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.298 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.298 INFO:teuthology.orchestra.run.smithi072.stdout: "secure_monitoring_stack": { 2024-03-22T01:47:09.298 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "secure_monitoring_stack", 2024-03-22T01:47:09.298 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.299 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.299 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.299 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.299 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.299 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.299 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.299 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-22T01:47:09.299 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.299 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.299 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.299 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.299 INFO:teuthology.orchestra.run.smithi072.stdout: "service_discovery_port": { 2024-03-22T01:47:09.299 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "service_discovery_port", 2024-03-22T01:47:09.299 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.300 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.300 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.300 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "8765", 2024-03-22T01:47:09.300 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.300 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.300 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.300 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "cephadm service discovery port", 2024-03-22T01:47:09.300 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.300 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.300 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.300 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.301 INFO:teuthology.orchestra.run.smithi072.stdout: "ssh_config_file": { 2024-03-22T01:47:09.301 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "ssh_config_file", 2024-03-22T01:47:09.301 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.301 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.301 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.301 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.301 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.301 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.301 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.301 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-22T01:47:09.301 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.301 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.301 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.301 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.301 INFO:teuthology.orchestra.run.smithi072.stdout: "use_agent": { 2024-03-22T01:47:09.302 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "use_agent", 2024-03-22T01:47:09.302 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.302 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.302 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.302 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.302 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.302 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.302 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.302 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-22T01:47:09.302 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.302 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.302 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.302 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.303 INFO:teuthology.orchestra.run.smithi072.stdout: "use_repo_digest": { 2024-03-22T01:47:09.303 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "use_repo_digest", 2024-03-22T01:47:09.303 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.303 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.303 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.303 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.303 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.303 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.303 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.303 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-22T01:47:09.303 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.303 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.303 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.303 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.303 INFO:teuthology.orchestra.run.smithi072.stdout: "warn_on_failed_host_check": { 2024-03-22T01:47:09.304 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "warn_on_failed_host_check", 2024-03-22T01:47:09.304 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.304 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.304 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.304 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.304 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.304 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.304 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.304 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "raise a health warning if the host check fails", 2024-03-22T01:47:09.304 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.304 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.304 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.304 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.304 INFO:teuthology.orchestra.run.smithi072.stdout: "warn_on_stray_daemons": { 2024-03-22T01:47:09.305 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "warn_on_stray_daemons", 2024-03-22T01:47:09.305 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.305 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.305 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.305 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.305 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.305 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.305 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.305 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-22T01:47:09.305 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.305 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.305 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.305 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.305 INFO:teuthology.orchestra.run.smithi072.stdout: "warn_on_stray_hosts": { 2024-03-22T01:47:09.305 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "warn_on_stray_hosts", 2024-03-22T01:47:09.306 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.306 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.306 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.306 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.306 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.306 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.306 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.306 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-22T01:47:09.306 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.306 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.306 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.306 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.306 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.307 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.307 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.307 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "crash", 2024-03-22T01:47:09.307 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.307 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.307 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.307 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.307 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.307 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.307 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.307 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.307 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.307 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.307 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.308 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.308 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.308 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.308 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.308 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.308 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.308 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.308 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.308 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.308 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.308 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.308 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.308 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.309 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.309 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.309 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.309 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.309 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.309 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.309 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.309 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.309 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.309 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.309 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.309 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.309 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.309 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.310 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.310 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.310 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.310 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.310 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.310 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.310 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.310 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.310 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.310 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.310 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.310 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.310 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.310 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.311 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.311 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.311 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.311 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.311 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.311 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.311 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.311 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.311 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.311 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.311 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.311 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.311 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.311 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.311 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.312 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.312 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.312 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.312 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.312 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.312 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.312 INFO:teuthology.orchestra.run.smithi072.stdout: "retain_interval": { 2024-03-22T01:47:09.312 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "retain_interval", 2024-03-22T01:47:09.312 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "secs", 2024-03-22T01:47:09.312 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.312 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.312 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "31536000", 2024-03-22T01:47:09.312 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.312 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.312 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.313 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-22T01:47:09.313 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.313 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.313 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.313 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.313 INFO:teuthology.orchestra.run.smithi072.stdout: "warn_recent_interval": { 2024-03-22T01:47:09.313 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "warn_recent_interval", 2024-03-22T01:47:09.313 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "secs", 2024-03-22T01:47:09.313 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.313 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.313 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "1209600", 2024-03-22T01:47:09.313 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.313 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.313 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.314 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-22T01:47:09.314 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.314 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.314 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.314 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.314 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.314 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.314 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.314 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "dashboard", 2024-03-22T01:47:09.314 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.314 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.314 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.314 INFO:teuthology.orchestra.run.smithi072.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-22T01:47:09.314 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-22T01:47:09.315 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.315 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.315 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.315 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "10", 2024-03-22T01:47:09.315 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.315 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.315 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.315 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.315 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.315 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.315 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.315 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.315 INFO:teuthology.orchestra.run.smithi072.stdout: "ALERTMANAGER_API_HOST": { 2024-03-22T01:47:09.315 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-22T01:47:09.315 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.316 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.316 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.316 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.316 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.316 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.316 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.316 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.316 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.316 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.316 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.316 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.316 INFO:teuthology.orchestra.run.smithi072.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-22T01:47:09.316 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-22T01:47:09.317 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.317 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.317 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.317 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.317 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.317 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.317 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.317 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.317 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.317 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.317 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.318 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.318 INFO:teuthology.orchestra.run.smithi072.stdout: "AUDIT_API_ENABLED": { 2024-03-22T01:47:09.318 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "AUDIT_API_ENABLED", 2024-03-22T01:47:09.318 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.318 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.318 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.318 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.318 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.318 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.318 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.318 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.319 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.319 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.319 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.319 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.319 INFO:teuthology.orchestra.run.smithi072.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-22T01:47:09.319 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-22T01:47:09.319 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.319 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.319 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.319 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.319 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.319 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.319 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.319 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.320 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.320 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.320 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.320 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.320 INFO:teuthology.orchestra.run.smithi072.stdout: "ENABLE_BROWSABLE_API": { 2024-03-22T01:47:09.320 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-22T01:47:09.320 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.320 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.320 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.320 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.320 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.320 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.320 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.320 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.321 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.321 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.321 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.321 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.321 INFO:teuthology.orchestra.run.smithi072.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-22T01:47:09.321 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-22T01:47:09.321 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.321 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.321 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.321 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.322 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.322 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.322 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.322 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.322 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.322 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.322 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.322 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.322 INFO:teuthology.orchestra.run.smithi072.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-22T01:47:09.322 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-22T01:47:09.322 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.323 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.323 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.323 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.323 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.323 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.323 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.323 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.323 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.323 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.323 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.324 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.324 INFO:teuthology.orchestra.run.smithi072.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-22T01:47:09.324 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-22T01:47:09.324 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.324 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.324 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.324 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.324 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.324 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.325 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.325 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.325 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.325 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.325 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.325 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.325 INFO:teuthology.orchestra.run.smithi072.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-22T01:47:09.325 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-22T01:47:09.326 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.326 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.326 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.326 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.326 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.326 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.326 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.326 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.326 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.326 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.326 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.326 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.326 INFO:teuthology.orchestra.run.smithi072.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-22T01:47:09.327 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-22T01:47:09.327 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.327 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.327 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.327 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.327 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.327 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.327 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.327 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.327 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.327 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.327 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.327 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.328 INFO:teuthology.orchestra.run.smithi072.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-22T01:47:09.328 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-22T01:47:09.328 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.328 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.328 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.328 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.328 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.328 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.328 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.328 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.328 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.328 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.328 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.328 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.329 INFO:teuthology.orchestra.run.smithi072.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-22T01:47:09.329 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-22T01:47:09.329 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.329 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.329 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.329 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.329 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.329 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.329 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.329 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.330 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.330 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.330 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.330 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.330 INFO:teuthology.orchestra.run.smithi072.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-22T01:47:09.330 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-22T01:47:09.330 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.330 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.330 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.330 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.330 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.330 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.330 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.330 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.331 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.331 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.331 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.331 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.331 INFO:teuthology.orchestra.run.smithi072.stdout: "GRAFANA_API_PASSWORD": { 2024-03-22T01:47:09.331 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-22T01:47:09.331 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.331 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.331 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.331 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "admin", 2024-03-22T01:47:09.331 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.331 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.331 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.331 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.332 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.332 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.332 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.332 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.332 INFO:teuthology.orchestra.run.smithi072.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-22T01:47:09.332 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-22T01:47:09.332 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.332 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.332 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.332 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.332 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.332 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.332 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.332 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.332 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.333 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.333 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.333 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.333 INFO:teuthology.orchestra.run.smithi072.stdout: "GRAFANA_API_URL": { 2024-03-22T01:47:09.333 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "GRAFANA_API_URL", 2024-03-22T01:47:09.333 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.333 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.333 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.333 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.333 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.333 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.333 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.333 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.333 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.333 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.334 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.334 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.334 INFO:teuthology.orchestra.run.smithi072.stdout: "GRAFANA_API_USERNAME": { 2024-03-22T01:47:09.334 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-22T01:47:09.334 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.334 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.334 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.334 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "admin", 2024-03-22T01:47:09.334 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.334 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.334 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.334 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.334 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.334 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.335 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.335 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.335 INFO:teuthology.orchestra.run.smithi072.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-22T01:47:09.335 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-22T01:47:09.337 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.337 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.337 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.337 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.337 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.338 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.338 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.338 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.338 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.338 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.338 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.338 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.338 INFO:teuthology.orchestra.run.smithi072.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-22T01:47:09.338 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-22T01:47:09.338 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.338 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.338 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.338 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.339 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.339 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.339 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.339 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.339 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.339 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.339 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.339 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.339 INFO:teuthology.orchestra.run.smithi072.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-22T01:47:09.339 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-22T01:47:09.339 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.339 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.340 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.340 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.340 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.340 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.340 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.340 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.340 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.340 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.340 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.340 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.340 INFO:teuthology.orchestra.run.smithi072.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-22T01:47:09.340 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-22T01:47:09.340 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.340 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.340 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.341 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.341 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.341 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.341 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.341 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.341 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.341 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.341 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.341 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.341 INFO:teuthology.orchestra.run.smithi072.stdout: "MULTICLUSTER_CONFIG": { 2024-03-22T01:47:09.341 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-22T01:47:09.341 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.341 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.341 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.342 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "{}", 2024-03-22T01:47:09.342 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.342 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.342 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.342 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.342 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.342 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.342 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.342 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.342 INFO:teuthology.orchestra.run.smithi072.stdout: "PROMETHEUS_API_HOST": { 2024-03-22T01:47:09.342 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-22T01:47:09.342 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.342 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.342 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.343 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.343 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.343 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.343 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.343 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.343 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.343 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.343 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.343 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.343 INFO:teuthology.orchestra.run.smithi072.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-22T01:47:09.343 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-22T01:47:09.343 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.343 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.344 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.344 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.344 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.344 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.344 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.344 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.344 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.344 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.344 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.344 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.344 INFO:teuthology.orchestra.run.smithi072.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-22T01:47:09.344 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-22T01:47:09.344 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.344 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.344 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.345 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.345 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.345 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.345 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.345 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.345 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.345 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.345 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.345 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.345 INFO:teuthology.orchestra.run.smithi072.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-22T01:47:09.345 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-22T01:47:09.345 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.345 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.345 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.346 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.346 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.346 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.346 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.346 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.346 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.346 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.346 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.346 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.346 INFO:teuthology.orchestra.run.smithi072.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-22T01:47:09.346 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-22T01:47:09.346 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.346 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.346 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.347 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.347 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.347 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.347 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.347 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.347 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.347 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.347 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.347 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.347 INFO:teuthology.orchestra.run.smithi072.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-22T01:47:09.347 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-22T01:47:09.347 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.347 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.347 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.347 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.348 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.348 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.348 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.348 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.348 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.348 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.348 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.348 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.348 INFO:teuthology.orchestra.run.smithi072.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-22T01:47:09.348 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-22T01:47:09.348 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.348 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.348 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.349 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.349 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.349 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.349 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.349 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.349 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.349 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.349 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.349 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.349 INFO:teuthology.orchestra.run.smithi072.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-22T01:47:09.349 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-22T01:47:09.349 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.349 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.349 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.350 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.350 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.350 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.350 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.350 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.350 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.350 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.350 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.350 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.350 INFO:teuthology.orchestra.run.smithi072.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-22T01:47:09.350 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-22T01:47:09.350 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.351 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.351 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.351 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.351 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.351 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.351 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.351 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.351 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.351 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.351 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.351 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.351 INFO:teuthology.orchestra.run.smithi072.stdout: "PWD_POLICY_ENABLED": { 2024-03-22T01:47:09.352 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-22T01:47:09.352 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.352 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.352 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.352 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.352 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.352 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.352 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.352 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.352 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.352 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.352 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.352 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.352 INFO:teuthology.orchestra.run.smithi072.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-22T01:47:09.353 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-22T01:47:09.353 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.353 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.353 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.353 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-22T01:47:09.353 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.353 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.353 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.353 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.353 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.353 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.354 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.354 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.354 INFO:teuthology.orchestra.run.smithi072.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-22T01:47:09.354 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-22T01:47:09.354 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.354 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.354 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.354 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "10", 2024-03-22T01:47:09.354 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.354 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.354 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.355 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.355 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.355 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.355 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.355 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.355 INFO:teuthology.orchestra.run.smithi072.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-22T01:47:09.355 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-22T01:47:09.355 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.355 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.355 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.355 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "8", 2024-03-22T01:47:09.355 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.356 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.356 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.356 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.356 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.356 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.356 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.356 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.356 INFO:teuthology.orchestra.run.smithi072.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-22T01:47:09.356 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-22T01:47:09.356 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.356 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.356 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.356 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "45", 2024-03-22T01:47:09.357 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.357 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.357 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.357 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.357 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.357 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.357 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.357 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.357 INFO:teuthology.orchestra.run.smithi072.stdout: "RGW_API_ACCESS_KEY": { 2024-03-22T01:47:09.357 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-22T01:47:09.357 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.357 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.357 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.358 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.358 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.358 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.358 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.358 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.358 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.358 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.358 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.358 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.358 INFO:teuthology.orchestra.run.smithi072.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-22T01:47:09.358 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-22T01:47:09.358 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.358 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.359 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.359 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "admin", 2024-03-22T01:47:09.359 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.359 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.359 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.359 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.359 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.359 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.359 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.359 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.359 INFO:teuthology.orchestra.run.smithi072.stdout: "RGW_API_SECRET_KEY": { 2024-03-22T01:47:09.359 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-22T01:47:09.359 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.360 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.360 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.360 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.360 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.360 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.360 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.360 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.360 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.360 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.360 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.360 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.360 INFO:teuthology.orchestra.run.smithi072.stdout: "RGW_API_SSL_VERIFY": { 2024-03-22T01:47:09.360 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-22T01:47:09.360 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.361 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.361 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.361 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.361 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.361 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.361 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.361 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.361 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.361 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.361 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.361 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.361 INFO:teuthology.orchestra.run.smithi072.stdout: "UNSAFE_TLS_v1_2": { 2024-03-22T01:47:09.361 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-22T01:47:09.361 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.362 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.362 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.362 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.362 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.362 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.362 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.362 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.362 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.362 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.362 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.362 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.362 INFO:teuthology.orchestra.run.smithi072.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-22T01:47:09.362 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-22T01:47:09.362 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.363 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.363 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.363 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "0", 2024-03-22T01:47:09.363 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.363 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.363 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.363 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.363 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.363 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.363 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.363 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.363 INFO:teuthology.orchestra.run.smithi072.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-22T01:47:09.363 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-22T01:47:09.363 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.363 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.364 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.364 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "10", 2024-03-22T01:47:09.364 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.364 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.364 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.364 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.364 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.364 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.364 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.364 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.364 INFO:teuthology.orchestra.run.smithi072.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-22T01:47:09.364 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-22T01:47:09.364 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.364 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.365 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.365 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "5", 2024-03-22T01:47:09.365 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.365 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.365 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.365 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.365 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.365 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.365 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.365 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.365 INFO:teuthology.orchestra.run.smithi072.stdout: "cross_origin_url": { 2024-03-22T01:47:09.365 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "cross_origin_url", 2024-03-22T01:47:09.365 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.365 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.365 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.366 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.366 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.366 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.366 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.366 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.366 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.366 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.366 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.366 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.366 INFO:teuthology.orchestra.run.smithi072.stdout: "crt_file": { 2024-03-22T01:47:09.366 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "crt_file", 2024-03-22T01:47:09.366 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.366 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.367 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.367 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.367 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.367 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.367 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.367 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.367 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.367 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.367 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.367 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.367 INFO:teuthology.orchestra.run.smithi072.stdout: "debug": { 2024-03-22T01:47:09.367 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "debug", 2024-03-22T01:47:09.367 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.367 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.368 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.368 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.368 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.368 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.368 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.368 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Enable/disable debug options", 2024-03-22T01:47:09.368 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.368 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.368 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.368 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.368 INFO:teuthology.orchestra.run.smithi072.stdout: "jwt_token_ttl": { 2024-03-22T01:47:09.368 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "jwt_token_ttl", 2024-03-22T01:47:09.368 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.368 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.368 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.369 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "28800", 2024-03-22T01:47:09.369 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.369 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.369 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.369 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.369 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.369 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.369 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.369 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.369 INFO:teuthology.orchestra.run.smithi072.stdout: "key_file": { 2024-03-22T01:47:09.369 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "key_file", 2024-03-22T01:47:09.369 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.369 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.369 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.369 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.370 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.370 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.370 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.370 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.370 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.370 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.370 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.370 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.370 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.370 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.370 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.370 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.371 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.371 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.371 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.371 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.371 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.371 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.371 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.371 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.371 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.371 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.371 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.371 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.371 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.372 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.372 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.372 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.372 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.372 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.372 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.372 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.372 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.372 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.372 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.372 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.372 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.372 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.372 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.373 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.373 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.373 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.373 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.373 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.373 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.373 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.373 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.373 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.373 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.373 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.373 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.373 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.373 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.373 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.374 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.374 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.374 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.374 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.374 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.374 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.374 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.374 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.374 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.374 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.374 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.374 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.374 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.374 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.375 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.375 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.375 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.375 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.375 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.375 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.375 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.375 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.375 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.375 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.375 INFO:teuthology.orchestra.run.smithi072.stdout: "motd": { 2024-03-22T01:47:09.375 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "motd", 2024-03-22T01:47:09.375 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.375 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.375 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.376 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.376 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.376 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.376 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.376 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "The message of the day", 2024-03-22T01:47:09.376 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.376 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.376 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.376 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.376 INFO:teuthology.orchestra.run.smithi072.stdout: "redirect_resolve_ip_addr": { 2024-03-22T01:47:09.376 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "redirect_resolve_ip_addr", 2024-03-22T01:47:09.376 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.376 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.376 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.377 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.377 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.377 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.377 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.377 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.377 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.377 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.377 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.377 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.377 INFO:teuthology.orchestra.run.smithi072.stdout: "server_addr": { 2024-03-22T01:47:09.377 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "server_addr", 2024-03-22T01:47:09.377 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.377 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.377 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.377 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "::", 2024-03-22T01:47:09.378 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.378 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.378 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.378 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.378 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.378 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.378 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.378 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.378 INFO:teuthology.orchestra.run.smithi072.stdout: "server_port": { 2024-03-22T01:47:09.378 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "server_port", 2024-03-22T01:47:09.378 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.378 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.378 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.379 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "8080", 2024-03-22T01:47:09.379 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.379 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.379 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.379 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.379 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.379 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.379 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.379 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.379 INFO:teuthology.orchestra.run.smithi072.stdout: "ssl": { 2024-03-22T01:47:09.379 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "ssl", 2024-03-22T01:47:09.379 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.379 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.379 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.379 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.380 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.380 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.380 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.380 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.380 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.380 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.380 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.380 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.380 INFO:teuthology.orchestra.run.smithi072.stdout: "ssl_server_port": { 2024-03-22T01:47:09.380 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "ssl_server_port", 2024-03-22T01:47:09.380 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.380 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.380 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.380 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "8443", 2024-03-22T01:47:09.381 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.381 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.381 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.381 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.381 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.381 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.381 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.381 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.381 INFO:teuthology.orchestra.run.smithi072.stdout: "standby_behaviour": { 2024-03-22T01:47:09.381 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "standby_behaviour", 2024-03-22T01:47:09.381 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.381 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.382 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.382 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "redirect", 2024-03-22T01:47:09.382 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.382 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.382 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.382 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.382 INFO:teuthology.orchestra.run.smithi072.stdout: "redirect" 2024-03-22T01:47:09.382 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.382 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.382 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.382 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.382 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.382 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.383 INFO:teuthology.orchestra.run.smithi072.stdout: "standby_error_status_code": { 2024-03-22T01:47:09.383 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "standby_error_status_code", 2024-03-22T01:47:09.383 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.383 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.383 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.383 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "500", 2024-03-22T01:47:09.383 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "400", 2024-03-22T01:47:09.383 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "599", 2024-03-22T01:47:09.383 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.383 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.383 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.383 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.383 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.383 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.383 INFO:teuthology.orchestra.run.smithi072.stdout: "url_prefix": { 2024-03-22T01:47:09.383 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "url_prefix", 2024-03-22T01:47:09.384 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.384 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.384 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.384 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.384 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.384 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.384 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.384 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.384 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.384 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.384 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.384 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.384 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.385 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.385 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.385 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "devicehealth", 2024-03-22T01:47:09.385 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.385 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.385 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.385 INFO:teuthology.orchestra.run.smithi072.stdout: "enable_monitoring": { 2024-03-22T01:47:09.385 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "enable_monitoring", 2024-03-22T01:47:09.385 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.385 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.385 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.385 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.385 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.385 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.386 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.386 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "monitor device health metrics", 2024-03-22T01:47:09.386 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.386 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.386 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.386 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.386 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.386 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.386 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.386 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.386 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.386 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.386 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.387 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.387 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.387 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.387 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.387 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.387 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.387 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.387 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.387 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.387 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.387 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.387 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.387 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.387 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.387 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.388 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.388 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.388 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.388 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.388 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.388 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.388 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.388 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.388 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.388 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.388 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.388 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.388 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.388 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.388 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.389 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.389 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.389 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.389 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.389 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.389 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.389 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.389 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.389 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.389 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.389 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.389 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.389 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.389 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.390 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.390 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.390 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.390 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.390 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.390 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.390 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.390 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.390 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.390 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.390 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.390 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.390 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.390 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.391 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.391 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.391 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.391 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.391 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.391 INFO:teuthology.orchestra.run.smithi072.stdout: "mark_out_threshold": { 2024-03-22T01:47:09.391 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "mark_out_threshold", 2024-03-22T01:47:09.391 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "secs", 2024-03-22T01:47:09.391 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.391 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.391 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "2419200", 2024-03-22T01:47:09.391 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.391 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.391 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.391 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-22T01:47:09.392 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.392 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.392 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.392 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.392 INFO:teuthology.orchestra.run.smithi072.stdout: "pool_name": { 2024-03-22T01:47:09.392 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "pool_name", 2024-03-22T01:47:09.392 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.392 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.392 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.392 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "device_health_metrics", 2024-03-22T01:47:09.392 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.392 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.392 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.393 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-22T01:47:09.393 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.393 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.393 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.393 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.393 INFO:teuthology.orchestra.run.smithi072.stdout: "retention_period": { 2024-03-22T01:47:09.393 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "retention_period", 2024-03-22T01:47:09.393 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "secs", 2024-03-22T01:47:09.393 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.393 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.393 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "15552000", 2024-03-22T01:47:09.393 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.393 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.393 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.394 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "how long to retain device health metrics", 2024-03-22T01:47:09.394 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.394 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.394 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.394 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.394 INFO:teuthology.orchestra.run.smithi072.stdout: "scrape_frequency": { 2024-03-22T01:47:09.394 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "scrape_frequency", 2024-03-22T01:47:09.394 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "secs", 2024-03-22T01:47:09.394 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.394 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.394 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "86400", 2024-03-22T01:47:09.394 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.395 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.395 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.395 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-22T01:47:09.395 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.395 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.395 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.395 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.395 INFO:teuthology.orchestra.run.smithi072.stdout: "self_heal": { 2024-03-22T01:47:09.395 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "self_heal", 2024-03-22T01:47:09.395 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.395 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.395 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.395 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.395 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.395 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.396 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.396 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-22T01:47:09.396 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.396 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.396 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.396 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.396 INFO:teuthology.orchestra.run.smithi072.stdout: "sleep_interval": { 2024-03-22T01:47:09.396 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "sleep_interval", 2024-03-22T01:47:09.396 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "secs", 2024-03-22T01:47:09.396 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.396 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.396 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "600", 2024-03-22T01:47:09.396 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.396 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.396 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.397 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "how frequently to wake up and check device health", 2024-03-22T01:47:09.397 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.397 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.397 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.397 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.397 INFO:teuthology.orchestra.run.smithi072.stdout: "warn_threshold": { 2024-03-22T01:47:09.397 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "warn_threshold", 2024-03-22T01:47:09.397 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "secs", 2024-03-22T01:47:09.397 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.397 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.397 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "7257600", 2024-03-22T01:47:09.397 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.397 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.397 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.397 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-22T01:47:09.398 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.398 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.398 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.398 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.398 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.398 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.398 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.398 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "diskprediction_local", 2024-03-22T01:47:09.398 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.398 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.398 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.398 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.398 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.398 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.399 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.399 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.399 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.399 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.399 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.399 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.399 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.399 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.399 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.399 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.399 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.399 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.399 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.399 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.400 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.400 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.400 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.400 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.400 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.400 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.400 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.400 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.400 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.400 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.400 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.400 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.400 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.400 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.400 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.401 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.401 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.401 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.401 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.401 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.401 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.401 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.401 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.401 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.401 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.401 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.401 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.401 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.401 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.401 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.402 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.402 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.402 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.402 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.402 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.402 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.402 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.402 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.402 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.402 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.402 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.402 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.403 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.403 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.403 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.403 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.403 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.403 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.403 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.403 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.403 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.403 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.403 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.403 INFO:teuthology.orchestra.run.smithi072.stdout: "predict_interval": { 2024-03-22T01:47:09.403 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "predict_interval", 2024-03-22T01:47:09.403 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.403 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.404 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.404 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "86400", 2024-03-22T01:47:09.404 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.404 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.404 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.404 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.404 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.404 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.404 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.404 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.404 INFO:teuthology.orchestra.run.smithi072.stdout: "predictor_model": { 2024-03-22T01:47:09.404 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "predictor_model", 2024-03-22T01:47:09.404 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.404 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.404 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.405 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "prophetstor", 2024-03-22T01:47:09.405 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.405 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.405 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.405 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.405 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.405 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.405 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.405 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.405 INFO:teuthology.orchestra.run.smithi072.stdout: "sleep_interval": { 2024-03-22T01:47:09.405 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "sleep_interval", 2024-03-22T01:47:09.405 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.405 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.405 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.405 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "600", 2024-03-22T01:47:09.405 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.406 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.406 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.406 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.406 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.406 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.406 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.406 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.406 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.406 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.406 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.406 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "influx", 2024-03-22T01:47:09.406 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": false, 2024-03-22T01:47:09.406 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "influxdb python module not found", 2024-03-22T01:47:09.406 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.407 INFO:teuthology.orchestra.run.smithi072.stdout: "batch_size": { 2024-03-22T01:47:09.407 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "batch_size", 2024-03-22T01:47:09.407 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.407 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.407 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.407 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "5000", 2024-03-22T01:47:09.407 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.407 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.407 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.407 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-22T01:47:09.407 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.407 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.407 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.407 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.408 INFO:teuthology.orchestra.run.smithi072.stdout: "database": { 2024-03-22T01:47:09.408 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "database", 2024-03-22T01:47:09.408 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.408 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.408 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.408 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "ceph", 2024-03-22T01:47:09.408 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.408 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.408 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.408 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-22T01:47:09.408 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.408 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.408 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.409 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.409 INFO:teuthology.orchestra.run.smithi072.stdout: "hostname": { 2024-03-22T01:47:09.409 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "hostname", 2024-03-22T01:47:09.409 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.409 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.409 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.409 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.409 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.409 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.409 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.409 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "InfluxDB server hostname", 2024-03-22T01:47:09.409 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.409 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.409 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.409 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.410 INFO:teuthology.orchestra.run.smithi072.stdout: "interval": { 2024-03-22T01:47:09.410 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "interval", 2024-03-22T01:47:09.410 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "secs", 2024-03-22T01:47:09.410 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.410 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.410 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "30", 2024-03-22T01:47:09.410 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "5", 2024-03-22T01:47:09.410 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.410 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.410 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-22T01:47:09.410 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.410 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.410 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.410 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.411 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.411 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.411 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.411 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.411 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.411 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.411 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.411 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.411 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.411 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.411 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.411 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.411 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.411 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.412 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.412 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.412 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.412 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.412 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.412 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.412 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.412 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.412 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.412 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.412 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.412 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.412 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.412 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.412 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.413 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.413 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.413 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.413 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.413 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.413 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.413 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.413 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.413 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.413 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.413 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.413 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.413 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.413 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.413 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.414 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.414 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.414 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.414 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.414 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.414 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.414 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.414 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.414 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.414 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.414 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.414 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.414 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.414 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.414 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.415 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.415 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.415 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.415 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.415 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.415 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.415 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.415 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.415 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.415 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.415 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.415 INFO:teuthology.orchestra.run.smithi072.stdout: "password": { 2024-03-22T01:47:09.415 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "password", 2024-03-22T01:47:09.415 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.415 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.416 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.416 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.416 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.416 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.416 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.416 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "password of InfluxDB server user", 2024-03-22T01:47:09.416 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.416 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.416 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.416 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.416 INFO:teuthology.orchestra.run.smithi072.stdout: "port": { 2024-03-22T01:47:09.416 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "port", 2024-03-22T01:47:09.416 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.416 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.417 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.417 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "8086", 2024-03-22T01:47:09.417 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.417 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.417 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.417 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "InfluxDB server port", 2024-03-22T01:47:09.417 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.417 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.417 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.417 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.417 INFO:teuthology.orchestra.run.smithi072.stdout: "ssl": { 2024-03-22T01:47:09.417 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "ssl", 2024-03-22T01:47:09.417 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.417 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.418 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.418 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "false", 2024-03-22T01:47:09.418 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.418 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.418 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.418 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-22T01:47:09.418 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.418 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.418 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.418 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.418 INFO:teuthology.orchestra.run.smithi072.stdout: "threads": { 2024-03-22T01:47:09.418 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "threads", 2024-03-22T01:47:09.418 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.418 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.419 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.419 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "5", 2024-03-22T01:47:09.419 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "1", 2024-03-22T01:47:09.419 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "32", 2024-03-22T01:47:09.419 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.419 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-22T01:47:09.419 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.419 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.419 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.419 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.419 INFO:teuthology.orchestra.run.smithi072.stdout: "username": { 2024-03-22T01:47:09.419 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "username", 2024-03-22T01:47:09.419 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.419 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.420 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.420 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.420 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.420 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.420 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.420 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "username of InfluxDB server user", 2024-03-22T01:47:09.420 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.420 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.420 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.420 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.420 INFO:teuthology.orchestra.run.smithi072.stdout: "verify_ssl": { 2024-03-22T01:47:09.420 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "verify_ssl", 2024-03-22T01:47:09.420 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.420 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.421 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.421 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "true", 2024-03-22T01:47:09.421 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.421 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.421 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.421 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-22T01:47:09.421 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.421 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.421 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.421 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.421 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.421 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.421 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.421 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "insights", 2024-03-22T01:47:09.422 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.422 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.422 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.422 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.422 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.422 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.422 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.422 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.422 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.422 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.422 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.422 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.422 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.422 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.422 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.423 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.423 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.423 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.423 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.423 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.423 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.423 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.423 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.423 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.423 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.423 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.423 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.423 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.424 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.424 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.424 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.424 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.424 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.424 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.424 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.424 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.424 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.424 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.424 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.424 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.424 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.424 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.424 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.425 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.425 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.425 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.425 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.425 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.425 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.425 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.425 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.425 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.425 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.425 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.426 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.426 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.426 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.426 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.426 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.426 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.426 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.426 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.426 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.426 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.426 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.426 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.426 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.426 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.426 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.427 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.427 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.427 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.427 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.427 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.427 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.427 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.427 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "iostat", 2024-03-22T01:47:09.427 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.427 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.427 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.427 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.427 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.427 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.427 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.428 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.428 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.428 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.428 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.428 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.428 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.428 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.428 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.428 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.428 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.428 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.428 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.428 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.428 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.428 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.429 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.429 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.429 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.429 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.429 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.429 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.429 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.429 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.429 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.429 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.429 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.429 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.429 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.429 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.429 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.430 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.430 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.430 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.430 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.430 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.430 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.430 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.430 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.430 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.430 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.430 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.430 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.430 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.430 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.431 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.431 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.431 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.431 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.431 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.431 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.431 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.431 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.431 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.431 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.431 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.431 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.431 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.431 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.432 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.432 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.432 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.432 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.432 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.432 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.432 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.432 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.432 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.432 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.432 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.432 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "k8sevents", 2024-03-22T01:47:09.432 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.432 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.432 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.432 INFO:teuthology.orchestra.run.smithi072.stdout: "ceph_event_retention_days": { 2024-03-22T01:47:09.433 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "ceph_event_retention_days", 2024-03-22T01:47:09.433 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.433 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.433 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.433 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "7", 2024-03-22T01:47:09.433 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.433 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.433 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.433 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-22T01:47:09.433 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.433 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.433 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.433 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.434 INFO:teuthology.orchestra.run.smithi072.stdout: "config_check_secs": { 2024-03-22T01:47:09.434 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "config_check_secs", 2024-03-22T01:47:09.434 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.434 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.434 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.434 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "10", 2024-03-22T01:47:09.434 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "10", 2024-03-22T01:47:09.434 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.434 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.434 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-22T01:47:09.434 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.434 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.434 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.435 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.435 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.435 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.435 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.435 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.435 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.435 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.435 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.435 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.435 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.435 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.435 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.435 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.435 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.435 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.436 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.436 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.436 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.436 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.436 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.436 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.436 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.436 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.436 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.436 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.436 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.436 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.436 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.436 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.436 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.437 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.437 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.437 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.437 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.437 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.437 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.437 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.437 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.437 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.437 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.437 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.437 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.437 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.437 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.438 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.438 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.438 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.438 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.438 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.438 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.438 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.438 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.438 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.438 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.438 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.438 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.438 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.438 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.438 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.439 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.439 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.439 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.439 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.439 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.439 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.439 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.439 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.439 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.439 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.439 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.439 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.439 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.439 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.439 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.440 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "localpool", 2024-03-22T01:47:09.440 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.440 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.440 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.440 INFO:teuthology.orchestra.run.smithi072.stdout: "failure_domain": { 2024-03-22T01:47:09.440 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "failure_domain", 2024-03-22T01:47:09.440 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.440 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.440 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.440 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "host", 2024-03-22T01:47:09.440 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.440 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.440 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.440 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "failure domain for any created local pool", 2024-03-22T01:47:09.441 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-22T01:47:09.441 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.441 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.441 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.441 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.441 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.441 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.441 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.441 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.441 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.441 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.441 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.441 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.441 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.441 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.441 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.442 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.442 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.442 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.442 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.442 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.442 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.442 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.442 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.442 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.442 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.442 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.442 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.442 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.442 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.442 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.443 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.443 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.443 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.443 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.443 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.443 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.443 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.443 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.443 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.443 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.443 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.443 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.443 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.443 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.443 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.444 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.444 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.444 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.444 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.444 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.444 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.444 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.444 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.444 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.444 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.444 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.444 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.444 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.444 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.444 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.444 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.445 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.445 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.445 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.445 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.445 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.445 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.445 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.445 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.445 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.445 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.445 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.445 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.445 INFO:teuthology.orchestra.run.smithi072.stdout: "min_size": { 2024-03-22T01:47:09.445 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "min_size", 2024-03-22T01:47:09.445 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.446 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.446 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.446 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.446 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.446 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.446 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.446 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "default min_size for any created local pool", 2024-03-22T01:47:09.446 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-22T01:47:09.446 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.446 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.447 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.447 INFO:teuthology.orchestra.run.smithi072.stdout: "num_rep": { 2024-03-22T01:47:09.447 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "num_rep", 2024-03-22T01:47:09.447 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.447 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.447 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.447 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "3", 2024-03-22T01:47:09.447 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.447 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.447 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.447 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "default replica count for any created local pool", 2024-03-22T01:47:09.447 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.447 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.447 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.447 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.448 INFO:teuthology.orchestra.run.smithi072.stdout: "pg_num": { 2024-03-22T01:47:09.448 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "pg_num", 2024-03-22T01:47:09.448 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.448 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.448 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.448 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "128", 2024-03-22T01:47:09.448 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.448 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.448 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.448 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "default pg_num for any created local pool", 2024-03-22T01:47:09.448 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.448 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.448 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.448 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.448 INFO:teuthology.orchestra.run.smithi072.stdout: "prefix": { 2024-03-22T01:47:09.449 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "prefix", 2024-03-22T01:47:09.449 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.449 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.449 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.449 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.449 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.449 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.449 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.449 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "name prefix for any created local pool", 2024-03-22T01:47:09.449 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.449 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.449 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.449 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.449 INFO:teuthology.orchestra.run.smithi072.stdout: "subtree": { 2024-03-22T01:47:09.449 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "subtree", 2024-03-22T01:47:09.450 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.450 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.450 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.450 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "rack", 2024-03-22T01:47:09.450 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.450 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.450 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.450 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-22T01:47:09.450 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-22T01:47:09.450 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.450 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.450 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.450 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.450 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.451 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.451 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "mds_autoscaler", 2024-03-22T01:47:09.451 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.451 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.451 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.451 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.451 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.451 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.451 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.451 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.451 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.451 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.451 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.451 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.452 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.452 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.452 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.452 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.452 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.452 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.452 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.452 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.452 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.452 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.452 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.452 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.452 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.452 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.452 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.453 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.453 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.453 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.453 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.453 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.453 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.453 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.453 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.453 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.453 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.453 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.453 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.453 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.453 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.453 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.453 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.454 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.454 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.454 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.454 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.454 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.454 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.454 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.454 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.454 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.454 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.454 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.454 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.454 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.454 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.454 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.455 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.455 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.455 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.455 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.455 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.455 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.455 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.455 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.455 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.455 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.455 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.455 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.455 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.455 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.455 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.456 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.456 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.456 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.456 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "mirroring", 2024-03-22T01:47:09.456 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.456 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.456 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.456 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.456 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.456 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.456 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.456 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.456 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.456 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.456 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.456 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.457 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.457 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.457 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.457 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.457 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.457 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.457 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.457 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.457 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.457 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.457 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.457 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.457 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.457 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.457 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.458 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.458 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.458 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.458 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.458 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.458 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.458 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.458 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.458 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.458 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.458 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.458 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.458 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.458 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.459 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.459 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.459 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.459 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.459 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.459 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.459 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.459 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.459 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.459 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.459 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.459 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.459 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.459 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.460 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.460 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.460 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.460 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.460 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.460 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.460 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.460 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.460 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.460 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.460 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.460 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.460 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.460 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.460 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.461 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.461 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.461 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.461 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.461 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.461 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.461 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "nfs", 2024-03-22T01:47:09.461 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.461 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.461 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.461 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.461 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.461 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.461 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.461 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.462 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.462 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.462 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.462 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.462 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.462 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.462 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.462 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.462 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.462 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.462 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.462 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.462 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.462 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.462 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.463 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.463 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.463 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.463 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.463 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.463 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.463 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.463 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.463 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.463 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.463 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.463 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.463 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.464 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.464 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.464 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.464 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.464 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.464 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.464 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.464 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.464 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.464 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.464 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.464 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.464 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.464 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.464 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.465 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.465 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.465 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.465 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.465 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.465 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.465 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.465 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.465 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.465 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.465 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.465 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.465 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.465 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.465 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.466 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.466 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.466 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.466 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.466 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.466 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.466 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.466 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.466 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.466 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.466 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "orchestrator", 2024-03-22T01:47:09.466 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.466 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.466 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.466 INFO:teuthology.orchestra.run.smithi072.stdout: "fail_fs": { 2024-03-22T01:47:09.467 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "fail_fs", 2024-03-22T01:47:09.467 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.467 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.467 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.467 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.467 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.467 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.467 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.467 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-22T01:47:09.467 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.467 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.467 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.467 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.467 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.468 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.468 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.468 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.468 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.468 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.468 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.468 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.468 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.468 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.468 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.468 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.468 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.468 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.468 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.468 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.468 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.469 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.469 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.469 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.469 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.469 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.469 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.469 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.469 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.469 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.469 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.469 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.469 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.469 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.469 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.469 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.470 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.470 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.470 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.470 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.470 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.470 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.470 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.470 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.470 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.470 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.470 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.470 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.470 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.470 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.470 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.471 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.471 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.471 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.471 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.471 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.471 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.471 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.471 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.471 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.471 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.472 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.472 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.472 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.472 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.472 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.472 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.472 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.473 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.473 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.473 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.473 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.473 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.473 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.473 INFO:teuthology.orchestra.run.smithi072.stdout: "orchestrator": { 2024-03-22T01:47:09.473 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "orchestrator", 2024-03-22T01:47:09.473 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.473 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.473 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.473 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.473 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.473 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.474 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.474 INFO:teuthology.orchestra.run.smithi072.stdout: "cephadm", 2024-03-22T01:47:09.474 INFO:teuthology.orchestra.run.smithi072.stdout: "rook", 2024-03-22T01:47:09.474 INFO:teuthology.orchestra.run.smithi072.stdout: "test_orchestrator" 2024-03-22T01:47:09.474 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.474 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Orchestrator backend", 2024-03-22T01:47:09.474 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.474 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.474 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.474 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.474 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.474 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.474 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.474 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "osd_perf_query", 2024-03-22T01:47:09.474 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.475 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.475 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.475 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.475 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.475 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.475 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.475 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.475 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.475 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.475 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.475 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.475 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.475 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.475 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.475 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.476 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.476 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.476 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.476 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.476 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.476 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.476 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.476 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.476 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.476 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.476 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.476 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.476 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.477 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.477 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.477 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.477 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.477 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.477 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.477 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.477 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.477 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.477 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.477 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.477 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.477 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.478 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.478 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.478 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.478 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.478 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.478 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.478 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.478 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.478 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.478 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.478 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.478 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.478 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.479 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.479 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.479 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.479 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.479 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.479 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.479 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.479 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.479 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.479 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.479 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.479 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.479 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.479 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.480 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.480 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.480 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.480 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.480 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.480 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.480 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.480 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "osd_support", 2024-03-22T01:47:09.480 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.480 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.480 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.480 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.480 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.480 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.481 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.481 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.481 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.481 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.481 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.481 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.481 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.481 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.481 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.481 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.481 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.481 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.481 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.482 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.482 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.482 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.482 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.482 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.482 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.482 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.482 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.482 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.482 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.482 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.482 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.482 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.482 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.483 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.483 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.483 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.483 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.483 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.483 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.483 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.483 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.483 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.483 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.483 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.483 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.483 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.483 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.483 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.484 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.484 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.484 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.484 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.484 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.484 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.484 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.484 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.484 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.484 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.484 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.484 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.484 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.484 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.485 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.485 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.485 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.485 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.485 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.485 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.485 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.485 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.485 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.485 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.485 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.485 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.485 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.485 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.486 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "pg_autoscaler", 2024-03-22T01:47:09.486 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.486 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.486 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.486 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.486 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.486 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.486 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.486 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.486 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.486 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.486 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.486 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.487 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.487 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.487 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.487 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.487 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.487 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.487 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.487 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.487 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.487 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.487 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.487 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.487 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.488 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.488 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.488 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.488 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.488 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.488 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.488 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.488 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.488 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.488 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.488 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.488 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.488 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.488 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.489 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.489 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.489 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.489 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.489 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.489 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.489 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.489 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.489 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.489 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.489 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.489 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.490 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.490 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.490 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.490 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.490 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.490 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.490 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.490 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.490 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.490 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.490 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.490 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.490 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.490 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.491 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.491 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.491 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.491 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.491 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.491 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.491 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.491 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.491 INFO:teuthology.orchestra.run.smithi072.stdout: "sleep_interval": { 2024-03-22T01:47:09.491 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "sleep_interval", 2024-03-22T01:47:09.491 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "secs", 2024-03-22T01:47:09.491 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.491 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.491 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "60", 2024-03-22T01:47:09.492 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.492 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.492 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.492 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.492 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.492 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.492 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.492 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.492 INFO:teuthology.orchestra.run.smithi072.stdout: "threshold": { 2024-03-22T01:47:09.492 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "threshold", 2024-03-22T01:47:09.492 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "float", 2024-03-22T01:47:09.492 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.492 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.492 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "3.0", 2024-03-22T01:47:09.493 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "1.0", 2024-03-22T01:47:09.493 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.493 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.493 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "scaling threshold", 2024-03-22T01:47:09.493 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-22T01:47:09.493 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.493 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.493 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.493 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.493 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.493 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.493 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "progress", 2024-03-22T01:47:09.493 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.494 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.494 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.494 INFO:teuthology.orchestra.run.smithi072.stdout: "allow_pg_recovery_event": { 2024-03-22T01:47:09.494 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "allow_pg_recovery_event", 2024-03-22T01:47:09.494 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.494 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.494 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.494 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.494 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.494 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.494 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.494 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-22T01:47:09.494 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.494 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.495 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.495 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.495 INFO:teuthology.orchestra.run.smithi072.stdout: "enabled": { 2024-03-22T01:47:09.495 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "enabled", 2024-03-22T01:47:09.495 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.495 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.495 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.495 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.495 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.495 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.495 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.495 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.495 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.495 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.495 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.496 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.496 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.496 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.496 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.496 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.496 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.496 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.496 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.496 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.496 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.496 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.496 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.496 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.496 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.496 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.496 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.497 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.497 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.497 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.497 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.497 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.497 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.497 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.497 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.497 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.497 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.497 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.497 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.497 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.497 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.498 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.498 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.498 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.498 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.498 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.498 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.498 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.498 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.498 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.498 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.498 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.498 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.498 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.498 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.498 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.499 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.499 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.499 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.499 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.499 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.499 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.499 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.499 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.499 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.499 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.499 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.499 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.499 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.499 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.500 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.500 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.500 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.500 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.500 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.500 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.500 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.500 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.500 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.500 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.500 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.500 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.500 INFO:teuthology.orchestra.run.smithi072.stdout: "max_completed_events": { 2024-03-22T01:47:09.500 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "max_completed_events", 2024-03-22T01:47:09.500 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.500 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.501 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.501 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "50", 2024-03-22T01:47:09.501 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.501 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.501 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.501 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "number of past completed events to remember", 2024-03-22T01:47:09.501 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.501 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.501 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.501 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.501 INFO:teuthology.orchestra.run.smithi072.stdout: "sleep_interval": { 2024-03-22T01:47:09.502 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "sleep_interval", 2024-03-22T01:47:09.502 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "secs", 2024-03-22T01:47:09.502 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.502 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.502 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "5", 2024-03-22T01:47:09.502 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.502 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.502 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.502 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "how long the module is going to sleep", 2024-03-22T01:47:09.502 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.502 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.502 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.502 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.503 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.503 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.503 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.503 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "prometheus", 2024-03-22T01:47:09.503 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.503 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.503 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.503 INFO:teuthology.orchestra.run.smithi072.stdout: "cache": { 2024-03-22T01:47:09.503 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "cache", 2024-03-22T01:47:09.503 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.503 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.503 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.503 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.503 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.503 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.503 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.504 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.504 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.504 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.504 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.504 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.504 INFO:teuthology.orchestra.run.smithi072.stdout: "exclude_perf_counters": { 2024-03-22T01:47:09.504 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "exclude_perf_counters", 2024-03-22T01:47:09.504 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.504 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.504 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.504 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.504 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.504 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.505 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.505 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-22T01:47:09.505 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-22T01:47:09.505 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.505 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.505 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.505 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.505 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.505 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.505 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.505 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.505 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.505 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.505 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.506 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.506 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.506 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.506 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.506 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.506 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.506 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.506 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.506 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.506 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.506 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.506 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.506 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.506 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.506 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.507 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.507 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.507 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.507 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.507 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.507 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.507 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.507 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.507 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.507 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.507 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.507 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.507 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.507 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.507 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.508 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.508 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.508 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.508 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.508 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.508 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.508 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.508 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.508 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.508 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.508 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.509 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.509 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.509 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.509 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.509 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.509 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.509 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.509 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.509 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.509 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.509 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.509 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.509 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.509 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.510 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.510 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.510 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.510 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.510 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.510 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.510 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.510 INFO:teuthology.orchestra.run.smithi072.stdout: "rbd_stats_pools": { 2024-03-22T01:47:09.510 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "rbd_stats_pools", 2024-03-22T01:47:09.510 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.510 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.510 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.511 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.511 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.511 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.511 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.511 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.511 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.511 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.511 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.511 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.511 INFO:teuthology.orchestra.run.smithi072.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-22T01:47:09.511 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-22T01:47:09.511 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.511 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.512 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.512 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "300", 2024-03-22T01:47:09.512 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.512 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.512 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.512 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.512 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.512 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.512 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.512 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.512 INFO:teuthology.orchestra.run.smithi072.stdout: "scrape_interval": { 2024-03-22T01:47:09.512 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "scrape_interval", 2024-03-22T01:47:09.512 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "float", 2024-03-22T01:47:09.512 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.512 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.513 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "15.0", 2024-03-22T01:47:09.513 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.513 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.513 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.513 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.513 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.513 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.513 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.513 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.513 INFO:teuthology.orchestra.run.smithi072.stdout: "server_addr": { 2024-03-22T01:47:09.513 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "server_addr", 2024-03-22T01:47:09.513 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.513 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.513 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.514 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "::", 2024-03-22T01:47:09.514 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.514 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.514 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.514 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-22T01:47:09.514 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.514 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.514 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.514 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.514 INFO:teuthology.orchestra.run.smithi072.stdout: "server_port": { 2024-03-22T01:47:09.514 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "server_port", 2024-03-22T01:47:09.514 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.514 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.515 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.515 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "9283", 2024-03-22T01:47:09.515 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.515 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.515 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.515 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-22T01:47:09.515 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.515 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.515 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.515 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.515 INFO:teuthology.orchestra.run.smithi072.stdout: "stale_cache_strategy": { 2024-03-22T01:47:09.516 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "stale_cache_strategy", 2024-03-22T01:47:09.516 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.516 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.516 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.516 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "log", 2024-03-22T01:47:09.516 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.516 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.516 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.516 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.516 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.516 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.516 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.516 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.516 INFO:teuthology.orchestra.run.smithi072.stdout: "standby_behaviour": { 2024-03-22T01:47:09.516 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "standby_behaviour", 2024-03-22T01:47:09.517 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.517 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.517 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.517 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "default", 2024-03-22T01:47:09.517 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.517 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.517 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.517 INFO:teuthology.orchestra.run.smithi072.stdout: "default", 2024-03-22T01:47:09.517 INFO:teuthology.orchestra.run.smithi072.stdout: "error" 2024-03-22T01:47:09.517 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.517 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.517 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.517 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.517 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.517 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.518 INFO:teuthology.orchestra.run.smithi072.stdout: "standby_error_status_code": { 2024-03-22T01:47:09.518 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "standby_error_status_code", 2024-03-22T01:47:09.518 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.518 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.518 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.518 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "500", 2024-03-22T01:47:09.518 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "400", 2024-03-22T01:47:09.518 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "599", 2024-03-22T01:47:09.518 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.518 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.518 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.518 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.518 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.519 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.519 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.519 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.519 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.519 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "rbd_support", 2024-03-22T01:47:09.519 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.519 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.519 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.519 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.519 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.519 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.519 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.519 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.519 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.520 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.520 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.520 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.520 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.520 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.520 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.520 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.520 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.520 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.520 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.520 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.520 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.520 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.520 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.520 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.521 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.521 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.521 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.521 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.521 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.521 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.521 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.521 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.521 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.521 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.521 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.521 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.521 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.521 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.522 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.522 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.522 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.522 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.522 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.522 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.522 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.522 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.522 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.522 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.522 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.522 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.523 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.523 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.523 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.523 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.523 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.523 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.523 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.523 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.523 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.523 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.523 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.523 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.523 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.523 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.523 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.523 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.524 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.524 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.524 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.524 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.524 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.524 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.524 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.524 INFO:teuthology.orchestra.run.smithi072.stdout: "max_concurrent_snap_create": { 2024-03-22T01:47:09.524 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "max_concurrent_snap_create", 2024-03-22T01:47:09.524 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.524 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.524 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.524 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "10", 2024-03-22T01:47:09.524 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.525 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.525 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.525 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.525 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.525 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.525 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.525 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.525 INFO:teuthology.orchestra.run.smithi072.stdout: "mirror_snapshot_schedule": { 2024-03-22T01:47:09.525 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "mirror_snapshot_schedule", 2024-03-22T01:47:09.525 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.525 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.525 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.525 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.526 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.526 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.526 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.526 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.526 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.526 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.526 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.526 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.526 INFO:teuthology.orchestra.run.smithi072.stdout: "trash_purge_schedule": { 2024-03-22T01:47:09.526 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "trash_purge_schedule", 2024-03-22T01:47:09.526 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.526 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.526 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.527 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.527 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.527 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.527 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.527 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.527 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.527 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.527 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.527 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.527 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.527 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.527 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.527 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "restful", 2024-03-22T01:47:09.528 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.528 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.528 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.528 INFO:teuthology.orchestra.run.smithi072.stdout: "enable_auth": { 2024-03-22T01:47:09.528 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "enable_auth", 2024-03-22T01:47:09.528 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.528 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.528 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.528 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.528 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.528 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.528 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.528 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.528 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.529 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.529 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.529 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.529 INFO:teuthology.orchestra.run.smithi072.stdout: "key_file": { 2024-03-22T01:47:09.529 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "key_file", 2024-03-22T01:47:09.529 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.529 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.529 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.529 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.529 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.529 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.529 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.529 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.529 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.530 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.530 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.530 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.530 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.530 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.530 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.530 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.530 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.530 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.530 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.530 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.531 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.531 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.531 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.531 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.531 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.531 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.531 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.531 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.531 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.531 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.531 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.531 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.532 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.532 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.532 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.532 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.532 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.532 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.532 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.532 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.532 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.532 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.532 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.532 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.532 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.533 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.533 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.533 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.533 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.533 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.533 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.533 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.533 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.533 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.533 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.533 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.533 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.533 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.533 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.533 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.534 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.534 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.534 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.534 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.534 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.534 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.534 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.534 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.534 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.534 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.534 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.534 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.534 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.535 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.535 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.535 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.535 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.535 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.535 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.535 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.535 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.535 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.535 INFO:teuthology.orchestra.run.smithi072.stdout: "server_addr": { 2024-03-22T01:47:09.535 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "server_addr", 2024-03-22T01:47:09.535 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.535 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.535 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.535 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.535 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.536 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.536 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.536 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.536 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.536 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.536 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.536 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.536 INFO:teuthology.orchestra.run.smithi072.stdout: "server_port": { 2024-03-22T01:47:09.536 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "server_port", 2024-03-22T01:47:09.536 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.536 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.536 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.536 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.536 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.537 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.537 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.537 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.537 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.537 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.537 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.537 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.537 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.537 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.537 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.537 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "rgw", 2024-03-22T01:47:09.537 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.537 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.537 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.537 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.538 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.538 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.538 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.538 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.538 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.538 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.538 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.538 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.538 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.538 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.538 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.538 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.538 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.538 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.539 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.539 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.539 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.539 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.539 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.539 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.539 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.539 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.539 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.539 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.539 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.539 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.539 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.539 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.539 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.540 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.540 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.541 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.541 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.541 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.541 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.541 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.541 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.541 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.541 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.542 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.542 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.542 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.542 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.542 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.542 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.542 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.542 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.542 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.542 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.542 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.542 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.542 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.542 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.542 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.543 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.543 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.543 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.543 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.543 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.543 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.543 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.543 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.543 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.543 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.543 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.543 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.543 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.543 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.543 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.544 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.544 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.544 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.544 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "rook", 2024-03-22T01:47:09.544 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.544 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.544 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.544 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.544 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.544 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.544 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.544 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.544 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.544 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.545 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.545 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.545 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.545 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.545 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.545 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.545 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.545 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.545 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.545 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.545 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.545 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.545 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.545 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.545 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.546 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.546 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.546 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.546 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.546 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.546 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.546 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.546 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.546 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.546 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.546 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.546 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.546 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.546 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.547 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.547 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.547 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.547 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.547 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.547 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.547 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.547 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.547 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.547 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.547 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.547 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.547 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.547 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.547 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.548 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.548 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.548 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.548 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.548 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.548 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.548 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.548 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.548 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.548 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.548 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.548 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.548 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.548 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.548 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.549 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.549 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.549 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.549 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.549 INFO:teuthology.orchestra.run.smithi072.stdout: "storage_class": { 2024-03-22T01:47:09.549 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "storage_class", 2024-03-22T01:47:09.549 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.549 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.549 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.549 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "local", 2024-03-22T01:47:09.549 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.549 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.549 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.549 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-22T01:47:09.550 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.550 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.550 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.550 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.550 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.550 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.550 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.550 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "selftest", 2024-03-22T01:47:09.550 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.550 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.550 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.550 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.550 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.550 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.550 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.551 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.551 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.551 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.551 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.551 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.551 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.551 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.551 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.551 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.551 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.551 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.551 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.551 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.551 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.551 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.552 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.552 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.552 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.552 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.552 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.552 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.552 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.552 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.552 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.552 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.552 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.552 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.552 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.552 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.552 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.553 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.553 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.553 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.553 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.553 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.553 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.553 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.553 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.553 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.553 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.553 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.553 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.553 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.553 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.554 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.554 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.554 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.554 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.554 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.554 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.554 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.554 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.554 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.554 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.554 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.554 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.554 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.555 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.555 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.555 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.555 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.555 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.555 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.555 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.555 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.555 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.555 INFO:teuthology.orchestra.run.smithi072.stdout: "roption1": { 2024-03-22T01:47:09.555 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "roption1", 2024-03-22T01:47:09.555 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.556 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.556 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.556 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.556 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.556 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.556 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.556 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.556 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.556 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.556 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.556 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.556 INFO:teuthology.orchestra.run.smithi072.stdout: "roption2": { 2024-03-22T01:47:09.556 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "roption2", 2024-03-22T01:47:09.556 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.556 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.557 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.557 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "xyz", 2024-03-22T01:47:09.557 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.557 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.557 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.557 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.557 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.557 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.557 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.557 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.557 INFO:teuthology.orchestra.run.smithi072.stdout: "rwoption1": { 2024-03-22T01:47:09.557 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "rwoption1", 2024-03-22T01:47:09.557 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.557 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.557 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.558 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.558 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.558 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.558 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.558 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.558 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.558 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.558 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.558 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.558 INFO:teuthology.orchestra.run.smithi072.stdout: "rwoption2": { 2024-03-22T01:47:09.558 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "rwoption2", 2024-03-22T01:47:09.558 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.558 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.559 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.559 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.559 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.559 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.559 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.559 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.559 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.559 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.559 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.559 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.559 INFO:teuthology.orchestra.run.smithi072.stdout: "rwoption3": { 2024-03-22T01:47:09.559 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "rwoption3", 2024-03-22T01:47:09.559 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "float", 2024-03-22T01:47:09.559 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.559 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.560 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.560 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.560 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.560 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.560 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.560 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.560 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.560 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.560 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.560 INFO:teuthology.orchestra.run.smithi072.stdout: "rwoption4": { 2024-03-22T01:47:09.560 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "rwoption4", 2024-03-22T01:47:09.560 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.560 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.560 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.561 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.561 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.561 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.561 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.561 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.561 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.561 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.561 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.561 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.561 INFO:teuthology.orchestra.run.smithi072.stdout: "rwoption5": { 2024-03-22T01:47:09.561 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "rwoption5", 2024-03-22T01:47:09.561 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.561 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.562 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.562 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.562 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.562 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.562 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.562 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.562 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.562 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.562 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.562 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.562 INFO:teuthology.orchestra.run.smithi072.stdout: "rwoption6": { 2024-03-22T01:47:09.562 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "rwoption6", 2024-03-22T01:47:09.562 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.563 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.563 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.563 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.563 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.563 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.563 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.563 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.563 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.563 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.563 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.563 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.563 INFO:teuthology.orchestra.run.smithi072.stdout: "rwoption7": { 2024-03-22T01:47:09.563 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "rwoption7", 2024-03-22T01:47:09.564 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.564 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.564 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.564 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.564 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "1", 2024-03-22T01:47:09.564 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "42", 2024-03-22T01:47:09.564 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.564 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.564 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.564 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.564 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.564 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.564 INFO:teuthology.orchestra.run.smithi072.stdout: "testkey": { 2024-03-22T01:47:09.564 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "testkey", 2024-03-22T01:47:09.565 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.565 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.565 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.565 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.565 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.565 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.565 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.565 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.565 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.565 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.565 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.565 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.565 INFO:teuthology.orchestra.run.smithi072.stdout: "testlkey": { 2024-03-22T01:47:09.565 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "testlkey", 2024-03-22T01:47:09.565 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.566 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.566 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.566 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.566 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.566 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.566 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.566 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.566 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.566 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.566 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.566 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.566 INFO:teuthology.orchestra.run.smithi072.stdout: "testnewline": { 2024-03-22T01:47:09.566 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "testnewline", 2024-03-22T01:47:09.566 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.567 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.567 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.567 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.567 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.567 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.567 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.567 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.567 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.567 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.567 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.567 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.567 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.567 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.567 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.568 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "snap_schedule", 2024-03-22T01:47:09.568 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.568 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.568 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.568 INFO:teuthology.orchestra.run.smithi072.stdout: "allow_m_granularity": { 2024-03-22T01:47:09.568 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "allow_m_granularity", 2024-03-22T01:47:09.568 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.568 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.568 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.568 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.568 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.568 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.568 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.568 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "allow minute scheduled snapshots", 2024-03-22T01:47:09.569 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.569 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.569 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.569 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.569 INFO:teuthology.orchestra.run.smithi072.stdout: "dump_on_update": { 2024-03-22T01:47:09.569 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "dump_on_update", 2024-03-22T01:47:09.569 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.569 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.569 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.569 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.569 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.569 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.569 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.569 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "dump database to debug log on update", 2024-03-22T01:47:09.569 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.569 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.570 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.570 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.570 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.570 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.570 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.570 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.570 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.570 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.570 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.570 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.570 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.570 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.570 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.570 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.571 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.571 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.571 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.571 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.571 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.571 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.571 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.571 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.571 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.571 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.571 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.571 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.571 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.571 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.571 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.571 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.572 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.572 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.572 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.572 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.572 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.572 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.572 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.572 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.572 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.572 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.572 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.572 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.572 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.572 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.573 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.573 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.573 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.573 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.573 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.573 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.573 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.573 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.573 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.573 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.573 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.573 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.573 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.573 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.573 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.573 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.573 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.574 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.574 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.574 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.574 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.574 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.574 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.574 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.574 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.574 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.574 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.574 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.574 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.574 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.574 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.574 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "stats", 2024-03-22T01:47:09.575 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.575 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.575 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.575 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.575 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.575 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.575 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.575 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.575 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.575 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.575 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.575 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.575 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.575 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.575 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.576 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.576 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.576 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.576 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.576 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.576 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.576 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.576 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.576 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.576 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.576 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.576 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.576 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.576 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.576 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.576 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.576 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.577 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.577 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.577 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.577 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.577 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.577 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.577 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.577 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.577 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.577 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.577 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.577 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.577 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.578 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.578 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.578 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.578 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.578 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.578 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.578 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.578 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.578 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.578 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.578 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.578 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.578 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.578 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.578 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.579 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.579 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.579 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.579 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.579 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.579 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.579 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.579 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.579 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.579 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.579 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.579 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.579 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.579 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.579 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.580 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.580 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "status", 2024-03-22T01:47:09.580 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.580 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.580 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.580 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.580 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.580 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.580 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.580 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.580 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.580 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.580 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.581 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.581 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.581 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.581 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.581 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.581 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.581 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.581 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.581 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.581 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.581 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.581 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.581 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.581 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.582 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.582 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.582 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.582 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.582 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.582 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.582 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.582 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.582 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.582 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.582 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.582 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.582 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.582 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.582 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.583 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.583 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.583 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.583 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.583 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.583 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.583 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.583 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.583 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.583 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.583 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.583 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.583 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.583 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.583 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.583 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.584 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.584 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.584 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.584 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.584 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.584 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.584 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.584 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.584 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.584 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.584 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.584 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.584 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.584 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.584 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.585 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.585 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.585 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.585 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.585 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.585 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "telegraf", 2024-03-22T01:47:09.585 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.585 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.585 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.585 INFO:teuthology.orchestra.run.smithi072.stdout: "address": { 2024-03-22T01:47:09.585 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "address", 2024-03-22T01:47:09.585 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.585 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.585 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.586 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-22T01:47:09.586 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.586 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.586 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.586 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.586 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.586 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.586 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.586 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.586 INFO:teuthology.orchestra.run.smithi072.stdout: "interval": { 2024-03-22T01:47:09.586 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "interval", 2024-03-22T01:47:09.586 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "secs", 2024-03-22T01:47:09.586 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.586 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.586 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "15", 2024-03-22T01:47:09.587 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.587 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.587 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.587 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.587 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.587 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.587 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.587 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.587 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.587 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.587 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.587 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.587 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.587 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.588 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.588 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.588 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.588 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.588 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.588 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.588 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.588 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.588 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.588 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.588 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.588 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.588 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.588 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.588 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.588 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.589 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.589 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.589 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.589 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.589 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.589 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.589 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.589 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.589 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.589 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.589 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.589 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.589 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.589 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.589 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.590 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.590 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.590 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.590 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.590 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.590 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.590 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.590 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.590 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.590 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.590 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.590 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.590 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.590 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.590 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.591 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.591 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.591 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.591 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.591 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.591 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.591 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.591 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.591 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.591 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.591 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.591 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.591 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.591 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.591 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.591 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.592 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.592 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.592 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.592 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.592 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.592 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "telemetry", 2024-03-22T01:47:09.592 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.592 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.592 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.592 INFO:teuthology.orchestra.run.smithi072.stdout: "channel_basic": { 2024-03-22T01:47:09.592 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "channel_basic", 2024-03-22T01:47:09.592 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.592 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.592 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.592 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.592 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.593 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.593 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.593 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-22T01:47:09.593 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.593 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.593 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.593 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.593 INFO:teuthology.orchestra.run.smithi072.stdout: "channel_crash": { 2024-03-22T01:47:09.593 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "channel_crash", 2024-03-22T01:47:09.593 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.593 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.593 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.593 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.593 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.594 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.594 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.594 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-22T01:47:09.594 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.594 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.594 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.594 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.594 INFO:teuthology.orchestra.run.smithi072.stdout: "channel_device": { 2024-03-22T01:47:09.594 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "channel_device", 2024-03-22T01:47:09.594 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.594 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.594 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.594 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.594 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.595 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.595 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.595 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-22T01:47:09.595 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.595 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.595 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.595 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.595 INFO:teuthology.orchestra.run.smithi072.stdout: "channel_ident": { 2024-03-22T01:47:09.595 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "channel_ident", 2024-03-22T01:47:09.595 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.595 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.595 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.595 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.595 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.596 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.596 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.596 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-22T01:47:09.596 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.596 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.596 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.596 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.596 INFO:teuthology.orchestra.run.smithi072.stdout: "channel_perf": { 2024-03-22T01:47:09.596 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "channel_perf", 2024-03-22T01:47:09.596 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.596 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.596 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.596 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.596 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.597 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.597 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.597 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-22T01:47:09.597 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.597 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.597 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.597 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.597 INFO:teuthology.orchestra.run.smithi072.stdout: "contact": { 2024-03-22T01:47:09.597 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "contact", 2024-03-22T01:47:09.597 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.597 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.597 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.597 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.597 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.598 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.598 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.598 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.598 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.598 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.598 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.598 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.598 INFO:teuthology.orchestra.run.smithi072.stdout: "description": { 2024-03-22T01:47:09.598 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "description", 2024-03-22T01:47:09.598 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.598 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.598 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.598 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.598 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.598 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.599 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.599 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.599 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.599 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.599 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.599 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.599 INFO:teuthology.orchestra.run.smithi072.stdout: "device_url": { 2024-03-22T01:47:09.599 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "device_url", 2024-03-22T01:47:09.599 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.599 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.599 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.599 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-22T01:47:09.599 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.599 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.599 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.600 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.600 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.600 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.600 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.600 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.600 INFO:teuthology.orchestra.run.smithi072.stdout: "enabled": { 2024-03-22T01:47:09.600 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "enabled", 2024-03-22T01:47:09.600 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.600 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.600 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.600 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.600 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.600 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.600 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.601 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.601 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.601 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.601 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.601 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.601 INFO:teuthology.orchestra.run.smithi072.stdout: "interval": { 2024-03-22T01:47:09.601 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "interval", 2024-03-22T01:47:09.601 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.601 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.601 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.601 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "24", 2024-03-22T01:47:09.601 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "8", 2024-03-22T01:47:09.601 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.601 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.601 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.602 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.602 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.602 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.602 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.602 INFO:teuthology.orchestra.run.smithi072.stdout: "last_opt_revision": { 2024-03-22T01:47:09.602 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "last_opt_revision", 2024-03-22T01:47:09.602 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.602 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.602 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.602 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "1", 2024-03-22T01:47:09.602 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.602 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.602 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.603 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.603 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.603 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.603 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.603 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.603 INFO:teuthology.orchestra.run.smithi072.stdout: "leaderboard": { 2024-03-22T01:47:09.603 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "leaderboard", 2024-03-22T01:47:09.603 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.603 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.603 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.603 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.603 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.603 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.603 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.603 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.604 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.604 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.604 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.604 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.604 INFO:teuthology.orchestra.run.smithi072.stdout: "leaderboard_description": { 2024-03-22T01:47:09.604 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "leaderboard_description", 2024-03-22T01:47:09.604 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.604 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.604 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.604 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.604 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.604 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.604 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.604 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.604 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.605 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.605 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.605 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.605 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.605 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.605 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.605 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.605 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.605 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.605 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.605 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.605 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.605 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.605 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.605 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.606 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.606 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.606 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.606 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.606 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.606 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.606 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.606 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.606 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.606 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.606 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.606 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.606 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.606 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.607 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.607 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.607 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.607 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.607 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.607 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.607 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.607 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.607 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.607 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.607 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.607 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.607 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.607 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.608 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.608 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.608 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.608 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.608 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.608 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.608 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.608 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.608 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.608 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.608 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.608 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.608 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.608 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.608 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.608 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.609 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.609 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.609 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.609 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.609 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.609 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.609 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.609 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.609 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.609 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.609 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.609 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.609 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.609 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.609 INFO:teuthology.orchestra.run.smithi072.stdout: "organization": { 2024-03-22T01:47:09.609 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "organization", 2024-03-22T01:47:09.610 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.610 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.610 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.610 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.610 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.610 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.610 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.610 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.610 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.610 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.610 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.610 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.610 INFO:teuthology.orchestra.run.smithi072.stdout: "proxy": { 2024-03-22T01:47:09.611 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "proxy", 2024-03-22T01:47:09.611 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.611 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.611 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.611 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.611 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.611 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.611 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.611 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.611 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.611 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.611 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.611 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.611 INFO:teuthology.orchestra.run.smithi072.stdout: "url": { 2024-03-22T01:47:09.611 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "url", 2024-03-22T01:47:09.612 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.612 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.612 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.612 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-22T01:47:09.612 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.612 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.612 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.612 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.612 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.612 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.612 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.612 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.612 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.612 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.612 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.612 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "test_orchestrator", 2024-03-22T01:47:09.613 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.613 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.613 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.613 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.613 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.613 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.613 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.613 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.613 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.613 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.613 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.613 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.613 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.613 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.613 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.613 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.614 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.614 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.614 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.614 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.614 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.614 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.614 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.614 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.614 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.614 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.614 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.614 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.614 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.614 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.614 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.615 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.615 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.615 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.615 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.615 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.615 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.615 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.615 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.615 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.615 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.615 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.615 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.615 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.615 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.615 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.615 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.615 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.616 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.616 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.616 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.616 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.616 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.616 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.616 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.616 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.616 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.616 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.616 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.616 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.616 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.616 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.617 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.617 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.617 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.617 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.617 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.617 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.617 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.617 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.617 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.617 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.617 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.617 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.617 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.617 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.617 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "volumes", 2024-03-22T01:47:09.618 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.618 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.618 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.618 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.618 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.618 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.618 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.618 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.618 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.618 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.618 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.618 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.618 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.618 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.619 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.619 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.619 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.619 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.619 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.619 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.619 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.619 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.619 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.619 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.619 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.619 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.619 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.619 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.619 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.620 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.620 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.620 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.620 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.620 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.620 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.620 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.620 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.620 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.620 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.620 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.620 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.620 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.620 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.620 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.621 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.621 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.621 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.621 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.621 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.621 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.621 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.621 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.621 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.621 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.621 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.621 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.621 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.621 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.621 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.622 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.622 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.622 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.622 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.622 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.622 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.622 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.622 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.622 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.622 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.622 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.622 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.622 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.622 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.622 INFO:teuthology.orchestra.run.smithi072.stdout: "max_concurrent_clones": { 2024-03-22T01:47:09.623 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "max_concurrent_clones", 2024-03-22T01:47:09.623 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.623 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.623 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.623 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "4", 2024-03-22T01:47:09.623 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.623 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.623 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.623 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-22T01:47:09.623 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.623 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.623 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.623 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.623 INFO:teuthology.orchestra.run.smithi072.stdout: "periodic_async_work": { 2024-03-22T01:47:09.624 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "periodic_async_work", 2024-03-22T01:47:09.624 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.624 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.624 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.624 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.624 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.624 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.624 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.624 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Periodically check for async work", 2024-03-22T01:47:09.624 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.624 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.624 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.624 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.624 INFO:teuthology.orchestra.run.smithi072.stdout: "snapshot_clone_delay": { 2024-03-22T01:47:09.624 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "snapshot_clone_delay", 2024-03-22T01:47:09.625 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.625 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.625 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.625 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "0", 2024-03-22T01:47:09.625 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.625 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.625 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.625 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-22T01:47:09.625 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.625 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.625 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.625 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.625 INFO:teuthology.orchestra.run.smithi072.stdout: "snapshot_clone_no_wait": { 2024-03-22T01:47:09.625 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "snapshot_clone_no_wait", 2024-03-22T01:47:09.625 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.626 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.626 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.626 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.626 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.626 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.626 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.626 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-22T01:47:09.626 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.626 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.626 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.626 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.626 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.626 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.627 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.627 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "zabbix", 2024-03-22T01:47:09.627 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.627 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.627 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.627 INFO:teuthology.orchestra.run.smithi072.stdout: "discovery_interval": { 2024-03-22T01:47:09.627 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "discovery_interval", 2024-03-22T01:47:09.627 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "uint", 2024-03-22T01:47:09.627 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.627 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.627 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "100", 2024-03-22T01:47:09.627 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.627 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.627 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.627 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.628 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.628 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.628 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.628 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.628 INFO:teuthology.orchestra.run.smithi072.stdout: "identifier": { 2024-03-22T01:47:09.628 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "identifier", 2024-03-22T01:47:09.628 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.628 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.628 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.628 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.628 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.628 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.628 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.628 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.628 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.628 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.629 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.629 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.629 INFO:teuthology.orchestra.run.smithi072.stdout: "interval": { 2024-03-22T01:47:09.629 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "interval", 2024-03-22T01:47:09.629 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "secs", 2024-03-22T01:47:09.629 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.629 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.629 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "60", 2024-03-22T01:47:09.629 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.629 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.629 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.629 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.629 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.630 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.630 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.630 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.630 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.630 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.630 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.630 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.630 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.630 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.630 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.630 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.630 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.630 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.630 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.630 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.631 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.631 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.631 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.631 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.631 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.631 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.631 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.631 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.631 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.631 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.631 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.631 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.631 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.631 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.632 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.632 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.632 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.632 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.632 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.632 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.632 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.632 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.632 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.632 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.632 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.632 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.632 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.632 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.632 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.633 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.633 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.633 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.633 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.633 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.633 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.633 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.633 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.633 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.633 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.633 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.633 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.633 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.633 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.634 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.634 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.634 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.634 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.634 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.634 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.634 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.634 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.634 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.634 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.634 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.634 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.634 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.634 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.635 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.635 INFO:teuthology.orchestra.run.smithi072.stdout: "zabbix_host": { 2024-03-22T01:47:09.635 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "zabbix_host", 2024-03-22T01:47:09.635 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.635 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.635 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.635 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.635 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.635 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.635 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.635 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.635 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.635 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.635 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.635 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.636 INFO:teuthology.orchestra.run.smithi072.stdout: "zabbix_port": { 2024-03-22T01:47:09.636 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "zabbix_port", 2024-03-22T01:47:09.636 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.636 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.636 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.636 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "10051", 2024-03-22T01:47:09.636 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.636 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.636 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.636 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.636 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.636 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.636 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.636 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.636 INFO:teuthology.orchestra.run.smithi072.stdout: "zabbix_sender": { 2024-03-22T01:47:09.637 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "zabbix_sender", 2024-03-22T01:47:09.637 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.637 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.637 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.637 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-22T01:47:09.637 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.637 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.637 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.637 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.637 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.637 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.637 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.637 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.637 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.637 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.638 INFO:teuthology.orchestra.run.smithi072.stdout: ] 2024-03-22T01:47:09.638 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.638 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.638 INFO:teuthology.orchestra.run.smithi072.stdout: "modules": [ 2024-03-22T01:47:09.638 INFO:teuthology.orchestra.run.smithi072.stdout: "cephadm", 2024-03-22T01:47:09.638 INFO:teuthology.orchestra.run.smithi072.stdout: "dashboard", 2024-03-22T01:47:09.638 INFO:teuthology.orchestra.run.smithi072.stdout: "iostat", 2024-03-22T01:47:09.638 INFO:teuthology.orchestra.run.smithi072.stdout: "nfs", 2024-03-22T01:47:09.638 INFO:teuthology.orchestra.run.smithi072.stdout: "restful", 2024-03-22T01:47:09.638 INFO:teuthology.orchestra.run.smithi072.stdout: "snap_schedule" 2024-03-22T01:47:09.638 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.638 INFO:teuthology.orchestra.run.smithi072.stdout: "available_modules": [ 2024-03-22T01:47:09.638 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.638 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "alerts", 2024-03-22T01:47:09.639 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.639 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.639 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.639 INFO:teuthology.orchestra.run.smithi072.stdout: "interval": { 2024-03-22T01:47:09.639 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "interval", 2024-03-22T01:47:09.639 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "secs", 2024-03-22T01:47:09.639 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.639 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.639 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "60", 2024-03-22T01:47:09.639 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.639 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.639 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.639 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "How frequently to reexamine health status", 2024-03-22T01:47:09.639 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.639 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.640 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.640 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.640 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.640 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.640 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.640 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.640 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.640 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.640 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.640 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.640 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.640 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.640 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.640 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.640 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.641 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.641 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.641 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.641 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.641 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.641 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.641 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.641 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.641 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.641 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.641 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.641 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.641 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.641 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.642 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.642 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.642 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.642 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.642 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.642 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.642 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.642 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.642 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.642 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.642 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.642 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.642 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.642 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.643 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.643 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.643 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.643 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.643 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.643 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.643 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.643 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.643 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.643 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.643 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.643 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.643 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.643 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.643 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.644 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.644 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.644 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.644 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.644 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.644 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.644 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.644 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.644 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.644 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.644 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.644 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.644 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.644 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.644 INFO:teuthology.orchestra.run.smithi072.stdout: "smtp_destination": { 2024-03-22T01:47:09.645 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "smtp_destination", 2024-03-22T01:47:09.645 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.645 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.645 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.645 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.645 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.645 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.645 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.645 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Email address to send alerts to", 2024-03-22T01:47:09.645 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.645 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.645 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.645 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.645 INFO:teuthology.orchestra.run.smithi072.stdout: "smtp_from_name": { 2024-03-22T01:47:09.646 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "smtp_from_name", 2024-03-22T01:47:09.646 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.646 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.646 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.646 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "Ceph", 2024-03-22T01:47:09.646 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.646 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.646 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.646 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Email From: name", 2024-03-22T01:47:09.646 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.646 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.646 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.646 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.646 INFO:teuthology.orchestra.run.smithi072.stdout: "smtp_host": { 2024-03-22T01:47:09.646 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "smtp_host", 2024-03-22T01:47:09.647 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.647 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.647 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.647 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.647 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.647 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.647 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.647 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "SMTP server", 2024-03-22T01:47:09.647 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.647 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.647 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.647 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.647 INFO:teuthology.orchestra.run.smithi072.stdout: "smtp_password": { 2024-03-22T01:47:09.647 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "smtp_password", 2024-03-22T01:47:09.647 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.648 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.648 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.648 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.648 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.648 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.648 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.648 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Password to authenticate with", 2024-03-22T01:47:09.648 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.648 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.648 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.648 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.648 INFO:teuthology.orchestra.run.smithi072.stdout: "smtp_port": { 2024-03-22T01:47:09.648 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "smtp_port", 2024-03-22T01:47:09.648 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.648 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.649 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.649 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "465", 2024-03-22T01:47:09.649 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.649 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.649 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.649 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "SMTP port", 2024-03-22T01:47:09.649 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.649 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.649 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.649 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.649 INFO:teuthology.orchestra.run.smithi072.stdout: "smtp_sender": { 2024-03-22T01:47:09.649 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "smtp_sender", 2024-03-22T01:47:09.649 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.649 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.650 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.650 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.650 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.650 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.650 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.650 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "SMTP envelope sender", 2024-03-22T01:47:09.650 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.650 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.650 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.650 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.650 INFO:teuthology.orchestra.run.smithi072.stdout: "smtp_ssl": { 2024-03-22T01:47:09.650 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "smtp_ssl", 2024-03-22T01:47:09.650 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.651 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.651 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.651 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.651 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.651 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.651 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.651 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-22T01:47:09.651 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.651 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.651 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.651 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.651 INFO:teuthology.orchestra.run.smithi072.stdout: "smtp_user": { 2024-03-22T01:47:09.651 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "smtp_user", 2024-03-22T01:47:09.651 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.651 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.651 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.652 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.652 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.652 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.652 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.652 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "User to authenticate as", 2024-03-22T01:47:09.652 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.652 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.652 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.652 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.652 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.652 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.652 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.652 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "balancer", 2024-03-22T01:47:09.652 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.652 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.653 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.653 INFO:teuthology.orchestra.run.smithi072.stdout: "active": { 2024-03-22T01:47:09.653 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "active", 2024-03-22T01:47:09.653 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.653 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.653 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.653 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.653 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.653 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.653 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.653 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "automatically balance PGs across cluster", 2024-03-22T01:47:09.653 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.653 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.653 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.654 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.654 INFO:teuthology.orchestra.run.smithi072.stdout: "begin_time": { 2024-03-22T01:47:09.654 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "begin_time", 2024-03-22T01:47:09.654 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.654 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.654 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.654 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "0000", 2024-03-22T01:47:09.654 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.654 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.654 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.654 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "beginning time of day to automatically balance", 2024-03-22T01:47:09.654 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-22T01:47:09.654 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.654 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.655 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.655 INFO:teuthology.orchestra.run.smithi072.stdout: "begin_weekday": { 2024-03-22T01:47:09.655 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "begin_weekday", 2024-03-22T01:47:09.655 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "uint", 2024-03-22T01:47:09.655 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.655 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.655 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "0", 2024-03-22T01:47:09.655 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "0", 2024-03-22T01:47:09.655 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "6", 2024-03-22T01:47:09.655 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.655 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-22T01:47:09.655 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-22T01:47:09.655 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.655 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.655 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.656 INFO:teuthology.orchestra.run.smithi072.stdout: "crush_compat_max_iterations": { 2024-03-22T01:47:09.656 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "crush_compat_max_iterations", 2024-03-22T01:47:09.656 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "uint", 2024-03-22T01:47:09.656 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.656 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.656 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "25", 2024-03-22T01:47:09.656 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "1", 2024-03-22T01:47:09.656 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "250", 2024-03-22T01:47:09.656 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.656 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-22T01:47:09.656 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.656 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.656 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.656 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.657 INFO:teuthology.orchestra.run.smithi072.stdout: "crush_compat_metrics": { 2024-03-22T01:47:09.657 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "crush_compat_metrics", 2024-03-22T01:47:09.657 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.657 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.657 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.657 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "pgs,objects,bytes", 2024-03-22T01:47:09.657 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.657 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.657 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.657 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-22T01:47:09.657 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-22T01:47:09.657 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.657 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.657 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.657 INFO:teuthology.orchestra.run.smithi072.stdout: "crush_compat_step": { 2024-03-22T01:47:09.658 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "crush_compat_step", 2024-03-22T01:47:09.658 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "float", 2024-03-22T01:47:09.658 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.658 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.658 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "0.5", 2024-03-22T01:47:09.658 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "0.001", 2024-03-22T01:47:09.658 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "0.999", 2024-03-22T01:47:09.658 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.658 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "aggressiveness of optimization", 2024-03-22T01:47:09.658 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-22T01:47:09.658 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.658 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.658 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.659 INFO:teuthology.orchestra.run.smithi072.stdout: "end_time": { 2024-03-22T01:47:09.659 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "end_time", 2024-03-22T01:47:09.659 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.659 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.659 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.659 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "2359", 2024-03-22T01:47:09.659 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.659 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.659 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.659 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "ending time of day to automatically balance", 2024-03-22T01:47:09.659 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-22T01:47:09.659 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.659 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.659 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.660 INFO:teuthology.orchestra.run.smithi072.stdout: "end_weekday": { 2024-03-22T01:47:09.660 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "end_weekday", 2024-03-22T01:47:09.660 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "uint", 2024-03-22T01:47:09.660 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.660 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.660 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "0", 2024-03-22T01:47:09.660 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "0", 2024-03-22T01:47:09.660 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "6", 2024-03-22T01:47:09.660 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.660 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-22T01:47:09.660 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-22T01:47:09.660 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.660 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.660 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.660 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.660 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.661 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.661 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.661 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.661 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.661 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.661 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.661 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.661 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.661 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.661 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.661 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.661 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.661 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.661 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.661 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.661 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.662 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.662 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.662 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.662 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.662 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.662 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.662 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.662 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.662 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.662 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.662 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.662 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.662 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.662 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.662 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.663 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.663 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.663 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.663 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.663 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.663 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.663 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.663 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.663 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.663 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.663 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.663 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.663 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.663 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.663 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.663 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.664 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.664 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.664 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.664 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.664 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.664 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.664 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.664 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.664 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.664 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.664 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.664 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.664 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.664 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.665 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.665 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.665 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.665 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.665 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.665 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.665 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.665 INFO:teuthology.orchestra.run.smithi072.stdout: "min_score": { 2024-03-22T01:47:09.665 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "min_score", 2024-03-22T01:47:09.665 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "float", 2024-03-22T01:47:09.665 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.665 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.665 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "0", 2024-03-22T01:47:09.665 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.666 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.666 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.666 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-22T01:47:09.666 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.666 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.666 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.666 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.666 INFO:teuthology.orchestra.run.smithi072.stdout: "mode": { 2024-03-22T01:47:09.666 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "mode", 2024-03-22T01:47:09.666 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.666 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.666 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.667 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "upmap", 2024-03-22T01:47:09.667 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.667 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.667 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.667 INFO:teuthology.orchestra.run.smithi072.stdout: "crush-compat", 2024-03-22T01:47:09.667 INFO:teuthology.orchestra.run.smithi072.stdout: "none", 2024-03-22T01:47:09.667 INFO:teuthology.orchestra.run.smithi072.stdout: "read", 2024-03-22T01:47:09.667 INFO:teuthology.orchestra.run.smithi072.stdout: "upmap", 2024-03-22T01:47:09.667 INFO:teuthology.orchestra.run.smithi072.stdout: "upmap-read" 2024-03-22T01:47:09.667 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.667 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Balancer mode", 2024-03-22T01:47:09.667 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.667 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.667 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.667 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.668 INFO:teuthology.orchestra.run.smithi072.stdout: "pool_ids": { 2024-03-22T01:47:09.668 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "pool_ids", 2024-03-22T01:47:09.668 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.668 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.668 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.668 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.668 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.668 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.668 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.668 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-22T01:47:09.668 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.668 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.668 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.669 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.669 INFO:teuthology.orchestra.run.smithi072.stdout: "sleep_interval": { 2024-03-22T01:47:09.669 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "sleep_interval", 2024-03-22T01:47:09.669 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "secs", 2024-03-22T01:47:09.669 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.669 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.669 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "60", 2024-03-22T01:47:09.669 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.669 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.669 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.669 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-22T01:47:09.669 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.669 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.669 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.669 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.670 INFO:teuthology.orchestra.run.smithi072.stdout: "upmap_max_deviation": { 2024-03-22T01:47:09.670 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "upmap_max_deviation", 2024-03-22T01:47:09.670 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.670 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.670 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.670 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "5", 2024-03-22T01:47:09.670 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "1", 2024-03-22T01:47:09.670 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.670 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.670 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-22T01:47:09.670 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-22T01:47:09.670 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.670 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.670 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.671 INFO:teuthology.orchestra.run.smithi072.stdout: "upmap_max_optimizations": { 2024-03-22T01:47:09.671 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "upmap_max_optimizations", 2024-03-22T01:47:09.671 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "uint", 2024-03-22T01:47:09.671 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.671 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.671 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "10", 2024-03-22T01:47:09.671 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.671 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.671 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.671 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-22T01:47:09.671 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.671 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.671 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.671 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.672 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.672 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.672 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.672 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "cephadm", 2024-03-22T01:47:09.672 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.672 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.672 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.672 INFO:teuthology.orchestra.run.smithi072.stdout: "agent_down_multiplier": { 2024-03-22T01:47:09.672 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "agent_down_multiplier", 2024-03-22T01:47:09.672 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "float", 2024-03-22T01:47:09.672 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.672 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.672 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "3.0", 2024-03-22T01:47:09.672 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.673 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.673 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.673 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-22T01:47:09.673 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.673 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.673 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.673 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.673 INFO:teuthology.orchestra.run.smithi072.stdout: "agent_refresh_rate": { 2024-03-22T01:47:09.673 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "agent_refresh_rate", 2024-03-22T01:47:09.673 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "secs", 2024-03-22T01:47:09.673 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.673 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.673 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "20", 2024-03-22T01:47:09.674 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.674 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.674 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.674 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-22T01:47:09.674 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.674 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.674 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.674 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.674 INFO:teuthology.orchestra.run.smithi072.stdout: "agent_starting_port": { 2024-03-22T01:47:09.674 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "agent_starting_port", 2024-03-22T01:47:09.674 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.674 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.674 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.674 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "4721", 2024-03-22T01:47:09.675 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.675 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.675 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.675 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-22T01:47:09.675 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.675 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.675 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.675 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.675 INFO:teuthology.orchestra.run.smithi072.stdout: "allow_ptrace": { 2024-03-22T01:47:09.675 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "allow_ptrace", 2024-03-22T01:47:09.675 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.675 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.675 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.675 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.676 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.676 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.676 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.676 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-22T01:47:09.676 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-22T01:47:09.676 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.676 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.676 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.676 INFO:teuthology.orchestra.run.smithi072.stdout: "autotune_interval": { 2024-03-22T01:47:09.676 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "autotune_interval", 2024-03-22T01:47:09.676 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "secs", 2024-03-22T01:47:09.676 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.676 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.677 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "600", 2024-03-22T01:47:09.677 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.677 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.677 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.677 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-22T01:47:09.677 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.677 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.677 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.677 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.677 INFO:teuthology.orchestra.run.smithi072.stdout: "autotune_memory_target_ratio": { 2024-03-22T01:47:09.677 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "autotune_memory_target_ratio", 2024-03-22T01:47:09.677 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "float", 2024-03-22T01:47:09.677 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.677 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.677 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "0.7", 2024-03-22T01:47:09.678 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.678 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.678 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.678 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-22T01:47:09.678 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.678 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.678 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.678 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.678 INFO:teuthology.orchestra.run.smithi072.stdout: "cephadm_log_destination": { 2024-03-22T01:47:09.678 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "cephadm_log_destination", 2024-03-22T01:47:09.678 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.678 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.678 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.679 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.679 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.679 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.679 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.679 INFO:teuthology.orchestra.run.smithi072.stdout: "file", 2024-03-22T01:47:09.679 INFO:teuthology.orchestra.run.smithi072.stdout: "file,syslog", 2024-03-22T01:47:09.679 INFO:teuthology.orchestra.run.smithi072.stdout: "syslog" 2024-03-22T01:47:09.679 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.679 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-22T01:47:09.679 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.679 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.679 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.680 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.680 INFO:teuthology.orchestra.run.smithi072.stdout: "cgroups_split": { 2024-03-22T01:47:09.680 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "cgroups_split", 2024-03-22T01:47:09.680 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.680 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.680 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.680 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.680 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.680 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.680 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.680 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-22T01:47:09.680 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.681 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.681 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.681 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.681 INFO:teuthology.orchestra.run.smithi072.stdout: "config_checks_enabled": { 2024-03-22T01:47:09.681 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "config_checks_enabled", 2024-03-22T01:47:09.681 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.681 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.681 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.681 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.681 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.681 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.681 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.681 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-22T01:47:09.681 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.682 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.682 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.682 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.682 INFO:teuthology.orchestra.run.smithi072.stdout: "config_dashboard": { 2024-03-22T01:47:09.682 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "config_dashboard", 2024-03-22T01:47:09.682 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.682 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.682 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.682 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.682 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.682 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.682 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.682 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-22T01:47:09.683 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.683 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.683 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.683 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.683 INFO:teuthology.orchestra.run.smithi072.stdout: "container_image_alertmanager": { 2024-03-22T01:47:09.683 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "container_image_alertmanager", 2024-03-22T01:47:09.683 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.683 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.683 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.683 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-22T01:47:09.683 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.683 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.683 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.684 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Prometheus container image", 2024-03-22T01:47:09.684 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.684 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.684 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.684 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.684 INFO:teuthology.orchestra.run.smithi072.stdout: "container_image_base": { 2024-03-22T01:47:09.684 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "container_image_base", 2024-03-22T01:47:09.684 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.684 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.684 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.684 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-22T01:47:09.684 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.684 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.684 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.684 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Container image name, without the tag", 2024-03-22T01:47:09.685 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.685 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.685 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.685 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.685 INFO:teuthology.orchestra.run.smithi072.stdout: "container_image_elasticsearch": { 2024-03-22T01:47:09.685 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "container_image_elasticsearch", 2024-03-22T01:47:09.685 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.685 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.685 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.685 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-22T01:47:09.685 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.685 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.685 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.685 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "elasticsearch container image", 2024-03-22T01:47:09.686 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.686 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.686 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.686 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.686 INFO:teuthology.orchestra.run.smithi072.stdout: "container_image_grafana": { 2024-03-22T01:47:09.686 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "container_image_grafana", 2024-03-22T01:47:09.686 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.686 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.686 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.686 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-03-22T01:47:09.686 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.686 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.686 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.686 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Prometheus container image", 2024-03-22T01:47:09.687 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.687 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.687 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.687 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.687 INFO:teuthology.orchestra.run.smithi072.stdout: "container_image_haproxy": { 2024-03-22T01:47:09.687 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "container_image_haproxy", 2024-03-22T01:47:09.687 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.687 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.687 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.687 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-22T01:47:09.687 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.687 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.687 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.687 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "HAproxy container image", 2024-03-22T01:47:09.687 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.688 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.688 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.688 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.688 INFO:teuthology.orchestra.run.smithi072.stdout: "container_image_jaeger_agent": { 2024-03-22T01:47:09.688 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "container_image_jaeger_agent", 2024-03-22T01:47:09.688 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.688 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.688 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.688 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-22T01:47:09.688 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.688 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.688 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.688 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Jaeger agent container image", 2024-03-22T01:47:09.688 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.688 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.688 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.689 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.689 INFO:teuthology.orchestra.run.smithi072.stdout: "container_image_jaeger_collector": { 2024-03-22T01:47:09.689 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "container_image_jaeger_collector", 2024-03-22T01:47:09.689 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.689 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.689 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.689 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-22T01:47:09.689 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.689 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.689 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.689 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Jaeger collector container image", 2024-03-22T01:47:09.689 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.689 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.689 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.689 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.690 INFO:teuthology.orchestra.run.smithi072.stdout: "container_image_jaeger_query": { 2024-03-22T01:47:09.690 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "container_image_jaeger_query", 2024-03-22T01:47:09.690 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.690 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.690 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.690 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-22T01:47:09.690 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.690 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.690 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.690 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Jaeger query container image", 2024-03-22T01:47:09.690 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.690 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.690 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.690 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.691 INFO:teuthology.orchestra.run.smithi072.stdout: "container_image_keepalived": { 2024-03-22T01:47:09.691 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "container_image_keepalived", 2024-03-22T01:47:09.691 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.691 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.691 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.691 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-22T01:47:09.691 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.691 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.691 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.691 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Keepalived container image", 2024-03-22T01:47:09.691 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.691 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.691 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.691 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.692 INFO:teuthology.orchestra.run.smithi072.stdout: "container_image_loki": { 2024-03-22T01:47:09.692 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "container_image_loki", 2024-03-22T01:47:09.692 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.692 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.692 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.692 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-22T01:47:09.692 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.692 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.692 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.692 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Loki container image", 2024-03-22T01:47:09.692 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.692 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.692 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.692 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.692 INFO:teuthology.orchestra.run.smithi072.stdout: "container_image_node_exporter": { 2024-03-22T01:47:09.693 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "container_image_node_exporter", 2024-03-22T01:47:09.693 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.693 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.693 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.693 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-22T01:47:09.693 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.693 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.693 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.693 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Prometheus container image", 2024-03-22T01:47:09.693 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.693 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.693 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.693 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.693 INFO:teuthology.orchestra.run.smithi072.stdout: "container_image_nvmeof": { 2024-03-22T01:47:09.693 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "container_image_nvmeof", 2024-03-22T01:47:09.694 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.694 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.694 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.694 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-22T01:47:09.694 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.694 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.694 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.694 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Nvme-of container image", 2024-03-22T01:47:09.694 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.694 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.694 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.694 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.695 INFO:teuthology.orchestra.run.smithi072.stdout: "container_image_prometheus": { 2024-03-22T01:47:09.695 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "container_image_prometheus", 2024-03-22T01:47:09.695 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.695 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.695 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.695 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-22T01:47:09.695 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.695 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.695 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.695 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Prometheus container image", 2024-03-22T01:47:09.695 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.695 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.695 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.695 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.695 INFO:teuthology.orchestra.run.smithi072.stdout: "container_image_promtail": { 2024-03-22T01:47:09.696 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "container_image_promtail", 2024-03-22T01:47:09.696 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.696 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.696 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.696 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-22T01:47:09.696 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.696 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.696 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.696 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Promtail container image", 2024-03-22T01:47:09.696 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.696 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.696 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.696 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.696 INFO:teuthology.orchestra.run.smithi072.stdout: "container_image_snmp_gateway": { 2024-03-22T01:47:09.696 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "container_image_snmp_gateway", 2024-03-22T01:47:09.697 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.697 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.697 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.697 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-22T01:47:09.697 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.697 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.697 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.697 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "SNMP Gateway container image", 2024-03-22T01:47:09.697 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.697 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.697 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.697 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.697 INFO:teuthology.orchestra.run.smithi072.stdout: "container_init": { 2024-03-22T01:47:09.697 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "container_init", 2024-03-22T01:47:09.698 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.698 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.698 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.698 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.698 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.698 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.698 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.698 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Run podman/docker with `--init`", 2024-03-22T01:47:09.698 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.698 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.698 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.698 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.698 INFO:teuthology.orchestra.run.smithi072.stdout: "daemon_cache_timeout": { 2024-03-22T01:47:09.698 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "daemon_cache_timeout", 2024-03-22T01:47:09.699 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "secs", 2024-03-22T01:47:09.699 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.699 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.699 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "600", 2024-03-22T01:47:09.699 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.699 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.699 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.699 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-22T01:47:09.699 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.699 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.699 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.699 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.699 INFO:teuthology.orchestra.run.smithi072.stdout: "default_cephadm_command_timeout": { 2024-03-22T01:47:09.699 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "default_cephadm_command_timeout", 2024-03-22T01:47:09.700 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.700 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.700 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.700 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "900", 2024-03-22T01:47:09.700 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.700 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.700 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.700 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-22T01:47:09.700 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.700 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.700 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.700 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.700 INFO:teuthology.orchestra.run.smithi072.stdout: "default_registry": { 2024-03-22T01:47:09.700 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "default_registry", 2024-03-22T01:47:09.701 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.701 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.701 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.701 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "docker.io", 2024-03-22T01:47:09.701 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.701 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.701 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.701 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-22T01:47:09.701 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.701 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.701 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.701 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.701 INFO:teuthology.orchestra.run.smithi072.stdout: "device_cache_timeout": { 2024-03-22T01:47:09.701 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "device_cache_timeout", 2024-03-22T01:47:09.701 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "secs", 2024-03-22T01:47:09.702 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.702 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.702 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "1800", 2024-03-22T01:47:09.702 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.702 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.702 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.702 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "seconds to cache device inventory", 2024-03-22T01:47:09.702 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.702 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.702 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.702 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.702 INFO:teuthology.orchestra.run.smithi072.stdout: "device_enhanced_scan": { 2024-03-22T01:47:09.702 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "device_enhanced_scan", 2024-03-22T01:47:09.703 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.703 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.703 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.703 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.703 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.703 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.703 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.703 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-22T01:47:09.703 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.703 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.703 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.703 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.703 INFO:teuthology.orchestra.run.smithi072.stdout: "facts_cache_timeout": { 2024-03-22T01:47:09.703 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "facts_cache_timeout", 2024-03-22T01:47:09.703 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "secs", 2024-03-22T01:47:09.704 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.704 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.704 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "60", 2024-03-22T01:47:09.704 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.704 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.704 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.704 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "seconds to cache host facts data", 2024-03-22T01:47:09.704 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.704 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.704 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.704 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.704 INFO:teuthology.orchestra.run.smithi072.stdout: "grafana_dashboards_path": { 2024-03-22T01:47:09.704 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "grafana_dashboards_path", 2024-03-22T01:47:09.704 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.705 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.705 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.705 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-03-22T01:47:09.705 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.705 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.705 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.705 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-03-22T01:47:09.705 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.705 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.705 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.705 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.705 INFO:teuthology.orchestra.run.smithi072.stdout: "host_check_interval": { 2024-03-22T01:47:09.705 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "host_check_interval", 2024-03-22T01:47:09.705 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "secs", 2024-03-22T01:47:09.706 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.706 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.706 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "600", 2024-03-22T01:47:09.706 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.706 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.706 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.706 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "how frequently to perform a host check", 2024-03-22T01:47:09.706 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.706 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.706 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.706 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.706 INFO:teuthology.orchestra.run.smithi072.stdout: "hw_monitoring": { 2024-03-22T01:47:09.706 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "hw_monitoring", 2024-03-22T01:47:09.706 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.706 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.707 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.707 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.707 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.707 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.707 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.707 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-22T01:47:09.707 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.707 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.707 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.707 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.707 INFO:teuthology.orchestra.run.smithi072.stdout: "inventory_list_all": { 2024-03-22T01:47:09.707 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "inventory_list_all", 2024-03-22T01:47:09.707 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.707 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.707 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.707 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.708 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.708 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.708 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.708 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-22T01:47:09.708 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.708 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.708 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.708 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.708 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.708 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.708 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.708 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.708 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.708 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.708 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.708 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.709 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.709 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.709 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.709 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.709 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.709 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.709 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.709 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.709 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.709 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.709 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.709 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.709 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.709 INFO:teuthology.orchestra.run.smithi072.stdout: "log_refresh_metadata": { 2024-03-22T01:47:09.709 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_refresh_metadata", 2024-03-22T01:47:09.710 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.710 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.710 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.710 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.710 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.710 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.710 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.710 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-22T01:47:09.710 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.710 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.710 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.710 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.710 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.710 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.711 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.711 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.711 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.711 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.711 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.711 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.711 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.711 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-22T01:47:09.711 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.711 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.711 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.711 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.711 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.712 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.712 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.712 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.712 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.712 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.712 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.712 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.712 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.712 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.712 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.712 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.712 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.712 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.713 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.713 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.713 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.713 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.713 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.713 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.713 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.713 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.713 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.713 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.713 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.713 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.713 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.713 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.713 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.714 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.714 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.714 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.714 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.714 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.714 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.714 INFO:teuthology.orchestra.run.smithi072.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-22T01:47:09.714 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-22T01:47:09.714 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.714 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.714 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.714 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.714 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.714 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.715 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.715 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-22T01:47:09.715 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.715 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.715 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.715 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.715 INFO:teuthology.orchestra.run.smithi072.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-22T01:47:09.715 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-22T01:47:09.715 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.715 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.715 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.715 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "*", 2024-03-22T01:47:09.715 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.715 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.715 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.716 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-22T01:47:09.716 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.716 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.716 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.716 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.716 INFO:teuthology.orchestra.run.smithi072.stdout: "max_count_per_host": { 2024-03-22T01:47:09.716 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "max_count_per_host", 2024-03-22T01:47:09.716 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.716 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.716 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.716 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "10", 2024-03-22T01:47:09.716 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.716 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.716 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.716 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "max number of daemons per service per host", 2024-03-22T01:47:09.717 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.717 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.717 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.717 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.717 INFO:teuthology.orchestra.run.smithi072.stdout: "max_osd_draining_count": { 2024-03-22T01:47:09.717 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "max_osd_draining_count", 2024-03-22T01:47:09.717 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.717 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.717 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.717 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "10", 2024-03-22T01:47:09.717 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.717 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.717 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.717 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-22T01:47:09.718 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.718 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.718 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.718 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.718 INFO:teuthology.orchestra.run.smithi072.stdout: "migration_current": { 2024-03-22T01:47:09.718 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "migration_current", 2024-03-22T01:47:09.718 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.718 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.718 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.718 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.718 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.718 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.718 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.718 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "internal - do not modify", 2024-03-22T01:47:09.719 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.719 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.719 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.719 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.719 INFO:teuthology.orchestra.run.smithi072.stdout: "mode": { 2024-03-22T01:47:09.719 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "mode", 2024-03-22T01:47:09.719 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.719 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.719 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.719 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "root", 2024-03-22T01:47:09.719 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.719 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.719 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.719 INFO:teuthology.orchestra.run.smithi072.stdout: "cephadm-package", 2024-03-22T01:47:09.720 INFO:teuthology.orchestra.run.smithi072.stdout: "root" 2024-03-22T01:47:09.720 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.720 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "mode for remote execution of cephadm", 2024-03-22T01:47:09.720 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.720 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.720 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.720 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.720 INFO:teuthology.orchestra.run.smithi072.stdout: "oob_default_addr": { 2024-03-22T01:47:09.720 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "oob_default_addr", 2024-03-22T01:47:09.720 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.720 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.720 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.720 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "169.254.1.1", 2024-03-22T01:47:09.720 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.720 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.721 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.721 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-22T01:47:09.721 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.721 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.721 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.721 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.721 INFO:teuthology.orchestra.run.smithi072.stdout: "prometheus_alerts_path": { 2024-03-22T01:47:09.721 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "prometheus_alerts_path", 2024-03-22T01:47:09.721 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.721 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.721 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.721 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-22T01:47:09.721 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.721 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.721 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.721 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-22T01:47:09.722 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.722 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.722 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.722 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.722 INFO:teuthology.orchestra.run.smithi072.stdout: "registry_insecure": { 2024-03-22T01:47:09.722 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "registry_insecure", 2024-03-22T01:47:09.722 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.722 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.722 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.722 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.722 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.722 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.722 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.722 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-22T01:47:09.723 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.723 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.723 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.723 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.723 INFO:teuthology.orchestra.run.smithi072.stdout: "registry_password": { 2024-03-22T01:47:09.723 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "registry_password", 2024-03-22T01:47:09.723 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.723 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.723 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.723 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.723 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.723 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.723 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.723 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-22T01:47:09.723 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.724 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.724 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.724 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.724 INFO:teuthology.orchestra.run.smithi072.stdout: "registry_url": { 2024-03-22T01:47:09.724 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "registry_url", 2024-03-22T01:47:09.724 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.724 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.724 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.724 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.724 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.724 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.724 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.724 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-22T01:47:09.724 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.724 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.725 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.725 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.725 INFO:teuthology.orchestra.run.smithi072.stdout: "registry_username": { 2024-03-22T01:47:09.725 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "registry_username", 2024-03-22T01:47:09.725 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.725 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.725 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.725 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.725 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.725 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.725 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.725 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-22T01:47:09.725 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.725 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.726 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.726 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.726 INFO:teuthology.orchestra.run.smithi072.stdout: "secure_monitoring_stack": { 2024-03-22T01:47:09.726 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "secure_monitoring_stack", 2024-03-22T01:47:09.726 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.726 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.726 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.726 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.726 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.726 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.726 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.726 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-22T01:47:09.726 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.726 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.727 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.727 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.727 INFO:teuthology.orchestra.run.smithi072.stdout: "service_discovery_port": { 2024-03-22T01:47:09.727 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "service_discovery_port", 2024-03-22T01:47:09.727 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.727 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.727 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.727 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "8765", 2024-03-22T01:47:09.727 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.727 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.727 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.727 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "cephadm service discovery port", 2024-03-22T01:47:09.727 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.727 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.727 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.728 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.728 INFO:teuthology.orchestra.run.smithi072.stdout: "ssh_config_file": { 2024-03-22T01:47:09.728 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "ssh_config_file", 2024-03-22T01:47:09.728 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.728 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.728 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.728 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.728 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.728 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.728 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.728 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-22T01:47:09.728 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.728 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.728 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.729 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.729 INFO:teuthology.orchestra.run.smithi072.stdout: "use_agent": { 2024-03-22T01:47:09.729 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "use_agent", 2024-03-22T01:47:09.729 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.729 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.729 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.729 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.729 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.729 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.729 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.729 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-22T01:47:09.729 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.729 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.729 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.729 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.730 INFO:teuthology.orchestra.run.smithi072.stdout: "use_repo_digest": { 2024-03-22T01:47:09.730 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "use_repo_digest", 2024-03-22T01:47:09.730 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.730 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.730 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.730 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.730 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.730 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.730 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.730 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-22T01:47:09.730 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.730 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.730 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.730 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.730 INFO:teuthology.orchestra.run.smithi072.stdout: "warn_on_failed_host_check": { 2024-03-22T01:47:09.731 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "warn_on_failed_host_check", 2024-03-22T01:47:09.731 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.731 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.731 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.731 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.731 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.731 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.731 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.731 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "raise a health warning if the host check fails", 2024-03-22T01:47:09.731 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.731 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.731 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.731 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.731 INFO:teuthology.orchestra.run.smithi072.stdout: "warn_on_stray_daemons": { 2024-03-22T01:47:09.732 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "warn_on_stray_daemons", 2024-03-22T01:47:09.732 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.732 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.732 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.732 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.732 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.732 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.732 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.732 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-22T01:47:09.732 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.732 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.732 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.732 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.732 INFO:teuthology.orchestra.run.smithi072.stdout: "warn_on_stray_hosts": { 2024-03-22T01:47:09.732 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "warn_on_stray_hosts", 2024-03-22T01:47:09.733 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.733 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.733 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.733 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.733 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.733 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.733 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.733 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-22T01:47:09.733 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.733 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.733 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.733 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.733 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.733 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.734 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.734 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "crash", 2024-03-22T01:47:09.734 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.734 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.734 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.734 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.734 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.734 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.734 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.734 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.734 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.734 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.734 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.734 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.735 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.735 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.735 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.735 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.735 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.735 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.735 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.735 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.735 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.735 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.735 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.735 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.735 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.735 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.735 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.736 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.736 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.736 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.736 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.736 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.736 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.736 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.736 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.736 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.736 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.736 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.736 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.737 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.737 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.737 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.737 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.737 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.737 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.737 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.737 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.737 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.737 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.737 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.737 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.737 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.737 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.737 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.738 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.738 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.738 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.738 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.738 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.738 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.738 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.738 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.738 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.738 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.738 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.738 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.738 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.739 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.739 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.739 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.739 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.739 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.739 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.739 INFO:teuthology.orchestra.run.smithi072.stdout: "retain_interval": { 2024-03-22T01:47:09.739 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "retain_interval", 2024-03-22T01:47:09.739 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "secs", 2024-03-22T01:47:09.739 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.739 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.739 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "31536000", 2024-03-22T01:47:09.739 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.740 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.740 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.740 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-22T01:47:09.740 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.740 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.740 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.740 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.740 INFO:teuthology.orchestra.run.smithi072.stdout: "warn_recent_interval": { 2024-03-22T01:47:09.740 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "warn_recent_interval", 2024-03-22T01:47:09.740 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "secs", 2024-03-22T01:47:09.740 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.741 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.741 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "1209600", 2024-03-22T01:47:09.741 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.741 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.741 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.741 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-22T01:47:09.741 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.741 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.741 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.741 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.741 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.741 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.741 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.741 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "dashboard", 2024-03-22T01:47:09.742 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.742 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.742 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.742 INFO:teuthology.orchestra.run.smithi072.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-22T01:47:09.742 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-22T01:47:09.742 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.742 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.742 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.742 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "10", 2024-03-22T01:47:09.742 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.742 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.742 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.742 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.742 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.743 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.743 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.743 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.743 INFO:teuthology.orchestra.run.smithi072.stdout: "ALERTMANAGER_API_HOST": { 2024-03-22T01:47:09.743 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-22T01:47:09.743 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.743 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.743 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.743 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.743 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.743 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.743 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.743 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.743 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.744 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.744 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.744 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.744 INFO:teuthology.orchestra.run.smithi072.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-22T01:47:09.745 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-22T01:47:09.745 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.745 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.745 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.745 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.746 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.746 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.746 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.746 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.746 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.746 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.746 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.746 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.746 INFO:teuthology.orchestra.run.smithi072.stdout: "AUDIT_API_ENABLED": { 2024-03-22T01:47:09.746 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "AUDIT_API_ENABLED", 2024-03-22T01:47:09.746 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.746 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.746 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.747 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.747 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.747 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.747 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.747 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.747 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.747 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.747 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.747 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.747 INFO:teuthology.orchestra.run.smithi072.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-22T01:47:09.747 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-22T01:47:09.747 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.747 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.747 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.747 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.748 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.748 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.748 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.748 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.748 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.748 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.748 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.748 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.748 INFO:teuthology.orchestra.run.smithi072.stdout: "ENABLE_BROWSABLE_API": { 2024-03-22T01:47:09.748 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-22T01:47:09.748 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.748 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.748 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.748 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.749 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.749 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.749 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.749 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.749 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.749 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.749 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.749 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.749 INFO:teuthology.orchestra.run.smithi072.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-22T01:47:09.749 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-22T01:47:09.749 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.749 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.749 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.749 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.749 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.750 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.750 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.750 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.750 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.750 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.750 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.750 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.750 INFO:teuthology.orchestra.run.smithi072.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-22T01:47:09.750 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-22T01:47:09.750 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.750 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.750 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.750 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.750 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.750 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.751 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.751 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.751 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.751 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.751 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.751 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.751 INFO:teuthology.orchestra.run.smithi072.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-22T01:47:09.751 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-22T01:47:09.751 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.751 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.751 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.751 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.751 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.751 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.752 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.752 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.752 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.752 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.752 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.752 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.752 INFO:teuthology.orchestra.run.smithi072.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-22T01:47:09.752 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-22T01:47:09.752 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.752 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.752 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.752 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.752 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.752 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.752 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.753 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.753 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.753 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.753 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.753 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.753 INFO:teuthology.orchestra.run.smithi072.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-22T01:47:09.753 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-22T01:47:09.753 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.753 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.753 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.754 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.754 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.754 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.754 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.754 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.754 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.754 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.754 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.754 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.754 INFO:teuthology.orchestra.run.smithi072.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-22T01:47:09.754 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-22T01:47:09.755 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.755 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.755 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.755 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.755 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.755 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.755 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.755 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.755 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.755 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.755 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.755 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.755 INFO:teuthology.orchestra.run.smithi072.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-22T01:47:09.755 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-22T01:47:09.755 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.756 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.756 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.756 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.756 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.756 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.756 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.756 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.756 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.756 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.756 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.756 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.756 INFO:teuthology.orchestra.run.smithi072.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-22T01:47:09.756 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-22T01:47:09.756 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.757 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.757 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.757 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.757 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.757 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.757 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.757 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.757 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.757 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.757 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.757 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.757 INFO:teuthology.orchestra.run.smithi072.stdout: "GRAFANA_API_PASSWORD": { 2024-03-22T01:47:09.757 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-22T01:47:09.757 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.757 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.758 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.758 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "admin", 2024-03-22T01:47:09.758 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.758 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.758 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.758 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.758 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.758 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.758 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.758 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.758 INFO:teuthology.orchestra.run.smithi072.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-22T01:47:09.758 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-22T01:47:09.758 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.758 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.759 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.759 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.759 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.759 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.759 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.759 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.759 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.759 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.759 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.759 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.759 INFO:teuthology.orchestra.run.smithi072.stdout: "GRAFANA_API_URL": { 2024-03-22T01:47:09.759 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "GRAFANA_API_URL", 2024-03-22T01:47:09.759 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.760 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.760 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.760 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.760 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.760 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.760 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.760 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.760 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.760 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.760 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.760 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.760 INFO:teuthology.orchestra.run.smithi072.stdout: "GRAFANA_API_USERNAME": { 2024-03-22T01:47:09.760 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-22T01:47:09.760 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.760 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.761 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.761 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "admin", 2024-03-22T01:47:09.761 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.761 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.761 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.761 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.761 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.761 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.761 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.761 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.761 INFO:teuthology.orchestra.run.smithi072.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-22T01:47:09.761 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-22T01:47:09.761 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.761 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.762 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.762 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.762 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.762 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.762 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.762 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.762 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.762 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.762 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.762 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.762 INFO:teuthology.orchestra.run.smithi072.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-22T01:47:09.762 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-22T01:47:09.762 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.762 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.763 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.763 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.763 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.763 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.763 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.763 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.763 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.763 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.763 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.763 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.763 INFO:teuthology.orchestra.run.smithi072.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-22T01:47:09.763 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-22T01:47:09.763 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.763 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.763 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.763 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.764 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.764 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.764 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.764 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.764 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.764 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.764 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.764 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.764 INFO:teuthology.orchestra.run.smithi072.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-22T01:47:09.764 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-22T01:47:09.764 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.764 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.764 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.765 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.765 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.765 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.765 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.765 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.765 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.765 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.765 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.765 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.765 INFO:teuthology.orchestra.run.smithi072.stdout: "MULTICLUSTER_CONFIG": { 2024-03-22T01:47:09.765 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-22T01:47:09.765 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.765 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.765 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.765 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "{}", 2024-03-22T01:47:09.766 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.766 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.766 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.766 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.766 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.766 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.766 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.766 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.766 INFO:teuthology.orchestra.run.smithi072.stdout: "PROMETHEUS_API_HOST": { 2024-03-22T01:47:09.766 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-22T01:47:09.766 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.766 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.766 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.766 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.767 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.767 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.767 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.767 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.767 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.767 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.767 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.767 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.767 INFO:teuthology.orchestra.run.smithi072.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-22T01:47:09.767 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-22T01:47:09.767 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.767 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.767 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.767 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.768 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.768 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.768 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.768 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.768 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.768 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.768 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.768 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.768 INFO:teuthology.orchestra.run.smithi072.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-22T01:47:09.768 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-22T01:47:09.768 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.768 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.768 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.768 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.769 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.769 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.769 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.769 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.769 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.769 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.769 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.769 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.769 INFO:teuthology.orchestra.run.smithi072.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-22T01:47:09.769 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-22T01:47:09.769 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.769 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.769 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.770 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.770 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.770 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.770 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.770 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.770 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.770 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.770 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.770 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.770 INFO:teuthology.orchestra.run.smithi072.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-22T01:47:09.770 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-22T01:47:09.770 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.770 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.770 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.770 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.771 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.771 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.771 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.771 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.771 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.771 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.771 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.771 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.771 INFO:teuthology.orchestra.run.smithi072.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-22T01:47:09.771 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-22T01:47:09.771 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.771 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.771 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.771 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.772 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.772 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.772 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.772 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.772 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.772 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.772 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.772 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.772 INFO:teuthology.orchestra.run.smithi072.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-22T01:47:09.772 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-22T01:47:09.772 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.772 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.772 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.772 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.772 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.773 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.773 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.773 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.773 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.773 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.773 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.773 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.773 INFO:teuthology.orchestra.run.smithi072.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-22T01:47:09.773 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-22T01:47:09.773 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.773 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.773 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.773 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.773 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.773 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.774 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.774 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.774 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.774 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.774 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.774 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.774 INFO:teuthology.orchestra.run.smithi072.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-22T01:47:09.774 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-22T01:47:09.774 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.774 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.774 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.774 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.774 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.774 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.775 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.775 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.775 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.775 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.775 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.775 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.775 INFO:teuthology.orchestra.run.smithi072.stdout: "PWD_POLICY_ENABLED": { 2024-03-22T01:47:09.775 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-22T01:47:09.775 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.775 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.775 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.775 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.775 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.775 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.776 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.776 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.776 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.776 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.776 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.776 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.776 INFO:teuthology.orchestra.run.smithi072.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-22T01:47:09.776 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-22T01:47:09.776 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.776 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.776 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.776 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-22T01:47:09.776 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.777 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.777 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.777 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.777 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.777 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.777 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.777 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.777 INFO:teuthology.orchestra.run.smithi072.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-22T01:47:09.777 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-22T01:47:09.777 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.777 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.777 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.777 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "10", 2024-03-22T01:47:09.777 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.777 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.778 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.778 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.778 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.778 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.778 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.778 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.778 INFO:teuthology.orchestra.run.smithi072.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-22T01:47:09.778 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-22T01:47:09.778 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.778 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.778 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.778 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "8", 2024-03-22T01:47:09.778 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.779 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.779 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.779 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.779 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.779 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.779 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.779 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.779 INFO:teuthology.orchestra.run.smithi072.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-22T01:47:09.779 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-22T01:47:09.779 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.779 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.779 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.779 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "45", 2024-03-22T01:47:09.779 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.779 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.780 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.780 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.780 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.780 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.780 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.780 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.780 INFO:teuthology.orchestra.run.smithi072.stdout: "RGW_API_ACCESS_KEY": { 2024-03-22T01:47:09.780 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-22T01:47:09.780 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.780 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.780 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.780 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.780 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.780 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.780 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.781 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.781 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.781 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.781 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.781 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.781 INFO:teuthology.orchestra.run.smithi072.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-22T01:47:09.781 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-22T01:47:09.781 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.781 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.781 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.781 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "admin", 2024-03-22T01:47:09.781 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.781 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.781 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.781 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.781 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.782 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.782 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.782 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.782 INFO:teuthology.orchestra.run.smithi072.stdout: "RGW_API_SECRET_KEY": { 2024-03-22T01:47:09.782 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-22T01:47:09.782 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.782 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.782 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.782 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.782 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.782 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.782 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.782 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.782 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.783 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.783 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.783 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.783 INFO:teuthology.orchestra.run.smithi072.stdout: "RGW_API_SSL_VERIFY": { 2024-03-22T01:47:09.783 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-22T01:47:09.783 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.783 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.783 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.783 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.783 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.783 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.783 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.783 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.783 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.783 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.783 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.784 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.784 INFO:teuthology.orchestra.run.smithi072.stdout: "UNSAFE_TLS_v1_2": { 2024-03-22T01:47:09.784 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-22T01:47:09.784 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.784 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.784 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.784 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.784 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.784 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.784 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.784 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.784 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.784 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.784 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.784 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.784 INFO:teuthology.orchestra.run.smithi072.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-22T01:47:09.784 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-22T01:47:09.785 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.785 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.785 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.785 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "0", 2024-03-22T01:47:09.785 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.785 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.785 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.785 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.785 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.785 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.785 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.785 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.785 INFO:teuthology.orchestra.run.smithi072.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-22T01:47:09.785 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-22T01:47:09.785 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.785 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.786 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.786 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "10", 2024-03-22T01:47:09.786 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.786 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.786 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.786 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.786 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.786 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.786 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.786 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.786 INFO:teuthology.orchestra.run.smithi072.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-22T01:47:09.786 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-22T01:47:09.786 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.786 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.787 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.787 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "5", 2024-03-22T01:47:09.787 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.787 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.787 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.787 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.787 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.787 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.787 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.787 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.787 INFO:teuthology.orchestra.run.smithi072.stdout: "cross_origin_url": { 2024-03-22T01:47:09.787 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "cross_origin_url", 2024-03-22T01:47:09.787 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.787 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.788 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.788 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.788 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.788 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.788 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.788 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.788 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.788 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.788 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.788 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.788 INFO:teuthology.orchestra.run.smithi072.stdout: "crt_file": { 2024-03-22T01:47:09.788 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "crt_file", 2024-03-22T01:47:09.788 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.788 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.789 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.789 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.789 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.789 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.789 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.789 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.789 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.789 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.789 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.789 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.789 INFO:teuthology.orchestra.run.smithi072.stdout: "debug": { 2024-03-22T01:47:09.789 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "debug", 2024-03-22T01:47:09.789 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.789 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.790 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.790 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.790 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.790 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.790 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.790 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Enable/disable debug options", 2024-03-22T01:47:09.790 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.790 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.790 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.790 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.790 INFO:teuthology.orchestra.run.smithi072.stdout: "jwt_token_ttl": { 2024-03-22T01:47:09.790 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "jwt_token_ttl", 2024-03-22T01:47:09.790 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.791 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.791 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.791 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "28800", 2024-03-22T01:47:09.791 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.791 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.791 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.791 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.791 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.791 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.791 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.791 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.791 INFO:teuthology.orchestra.run.smithi072.stdout: "key_file": { 2024-03-22T01:47:09.791 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "key_file", 2024-03-22T01:47:09.791 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.792 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.792 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.792 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.792 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.792 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.792 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.792 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.792 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.792 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.792 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.792 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.792 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.792 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.793 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.793 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.793 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.793 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.793 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.793 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.793 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.793 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.793 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.793 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.793 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.793 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.793 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.794 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.794 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.794 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.794 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.794 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.794 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.794 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.794 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.794 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.794 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.794 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.795 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.795 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.795 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.795 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.795 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.795 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.795 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.795 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.795 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.795 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.795 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.795 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.795 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.795 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.795 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.796 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.796 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.796 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.796 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.796 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.796 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.796 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.796 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.796 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.796 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.796 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.796 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.796 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.796 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.797 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.797 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.797 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.797 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.797 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.797 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.797 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.797 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.797 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.797 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.797 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.797 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.798 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.798 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.798 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.798 INFO:teuthology.orchestra.run.smithi072.stdout: "motd": { 2024-03-22T01:47:09.798 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "motd", 2024-03-22T01:47:09.798 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.798 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.798 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.798 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.798 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.798 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.798 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.799 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "The message of the day", 2024-03-22T01:47:09.799 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.799 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.799 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.799 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.799 INFO:teuthology.orchestra.run.smithi072.stdout: "redirect_resolve_ip_addr": { 2024-03-22T01:47:09.799 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "redirect_resolve_ip_addr", 2024-03-22T01:47:09.799 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.799 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.799 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.799 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.799 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.799 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.799 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.800 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.800 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.800 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.800 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.800 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.800 INFO:teuthology.orchestra.run.smithi072.stdout: "server_addr": { 2024-03-22T01:47:09.800 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "server_addr", 2024-03-22T01:47:09.800 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.800 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.800 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.800 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "::", 2024-03-22T01:47:09.800 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.801 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.801 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.801 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.801 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.801 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.801 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.801 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.801 INFO:teuthology.orchestra.run.smithi072.stdout: "server_port": { 2024-03-22T01:47:09.801 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "server_port", 2024-03-22T01:47:09.801 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.801 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.801 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.801 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "8080", 2024-03-22T01:47:09.801 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.802 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.802 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.802 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.802 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.802 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.802 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.802 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.802 INFO:teuthology.orchestra.run.smithi072.stdout: "ssl": { 2024-03-22T01:47:09.802 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "ssl", 2024-03-22T01:47:09.802 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.802 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.802 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.802 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.803 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.803 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.803 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.803 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.803 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.803 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.803 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.803 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.803 INFO:teuthology.orchestra.run.smithi072.stdout: "ssl_server_port": { 2024-03-22T01:47:09.803 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "ssl_server_port", 2024-03-22T01:47:09.803 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.803 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.803 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.804 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "8443", 2024-03-22T01:47:09.804 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.804 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.804 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.804 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.804 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.804 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.804 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.804 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.804 INFO:teuthology.orchestra.run.smithi072.stdout: "standby_behaviour": { 2024-03-22T01:47:09.804 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "standby_behaviour", 2024-03-22T01:47:09.804 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.804 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.804 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.804 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "redirect", 2024-03-22T01:47:09.804 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.805 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.805 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.805 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.805 INFO:teuthology.orchestra.run.smithi072.stdout: "redirect" 2024-03-22T01:47:09.805 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.805 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.805 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.805 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.805 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.805 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.805 INFO:teuthology.orchestra.run.smithi072.stdout: "standby_error_status_code": { 2024-03-22T01:47:09.805 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "standby_error_status_code", 2024-03-22T01:47:09.805 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.805 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.806 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.806 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "500", 2024-03-22T01:47:09.806 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "400", 2024-03-22T01:47:09.806 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "599", 2024-03-22T01:47:09.806 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.806 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.806 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.806 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.806 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.806 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.806 INFO:teuthology.orchestra.run.smithi072.stdout: "url_prefix": { 2024-03-22T01:47:09.806 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "url_prefix", 2024-03-22T01:47:09.806 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.807 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.807 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.807 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.807 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.807 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.807 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.807 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.807 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.807 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.807 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.807 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.807 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.807 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.807 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.808 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "devicehealth", 2024-03-22T01:47:09.808 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.808 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.808 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.808 INFO:teuthology.orchestra.run.smithi072.stdout: "enable_monitoring": { 2024-03-22T01:47:09.808 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "enable_monitoring", 2024-03-22T01:47:09.808 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.808 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.808 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.808 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.808 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.808 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.808 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.808 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "monitor device health metrics", 2024-03-22T01:47:09.809 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.809 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.809 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.809 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.809 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.809 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.809 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.809 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.809 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.809 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.809 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.809 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.809 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.809 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.810 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.810 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.810 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.810 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.810 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.810 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.810 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.810 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.810 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.810 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.810 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.810 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.810 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.811 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.811 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.811 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.811 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.811 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.811 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.811 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.811 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.811 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.811 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.811 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.811 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.811 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.811 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.812 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.812 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.812 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.812 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.812 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.812 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.812 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.812 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.812 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.812 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.812 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.812 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.812 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.812 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.812 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.813 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.813 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.813 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.813 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.813 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.813 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.813 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.813 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.813 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.813 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.813 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.813 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.813 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.813 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.814 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.814 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.814 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.814 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.814 INFO:teuthology.orchestra.run.smithi072.stdout: "mark_out_threshold": { 2024-03-22T01:47:09.814 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "mark_out_threshold", 2024-03-22T01:47:09.814 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "secs", 2024-03-22T01:47:09.814 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.814 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.814 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "2419200", 2024-03-22T01:47:09.814 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.814 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.814 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.814 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-22T01:47:09.815 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.815 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.815 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.815 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.815 INFO:teuthology.orchestra.run.smithi072.stdout: "pool_name": { 2024-03-22T01:47:09.815 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "pool_name", 2024-03-22T01:47:09.815 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.815 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.815 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.815 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "device_health_metrics", 2024-03-22T01:47:09.815 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.815 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.815 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.815 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-22T01:47:09.815 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.816 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.816 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.816 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.816 INFO:teuthology.orchestra.run.smithi072.stdout: "retention_period": { 2024-03-22T01:47:09.816 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "retention_period", 2024-03-22T01:47:09.816 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "secs", 2024-03-22T01:47:09.816 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.816 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.816 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "15552000", 2024-03-22T01:47:09.816 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.816 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.816 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.816 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "how long to retain device health metrics", 2024-03-22T01:47:09.816 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.816 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.817 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.817 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.817 INFO:teuthology.orchestra.run.smithi072.stdout: "scrape_frequency": { 2024-03-22T01:47:09.817 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "scrape_frequency", 2024-03-22T01:47:09.817 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "secs", 2024-03-22T01:47:09.817 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.817 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.817 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "86400", 2024-03-22T01:47:09.817 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.817 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.817 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.817 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-22T01:47:09.817 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.817 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.817 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.818 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.818 INFO:teuthology.orchestra.run.smithi072.stdout: "self_heal": { 2024-03-22T01:47:09.818 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "self_heal", 2024-03-22T01:47:09.818 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.818 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.818 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.818 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.818 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.818 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.818 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.818 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-22T01:47:09.818 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.818 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.818 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.818 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.819 INFO:teuthology.orchestra.run.smithi072.stdout: "sleep_interval": { 2024-03-22T01:47:09.819 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "sleep_interval", 2024-03-22T01:47:09.819 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "secs", 2024-03-22T01:47:09.819 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.819 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.819 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "600", 2024-03-22T01:47:09.819 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.819 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.819 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.819 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "how frequently to wake up and check device health", 2024-03-22T01:47:09.819 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.819 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.819 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.819 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.819 INFO:teuthology.orchestra.run.smithi072.stdout: "warn_threshold": { 2024-03-22T01:47:09.820 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "warn_threshold", 2024-03-22T01:47:09.820 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "secs", 2024-03-22T01:47:09.820 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.820 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.820 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "7257600", 2024-03-22T01:47:09.820 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.820 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.820 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.820 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-22T01:47:09.820 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.820 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.820 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.820 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.820 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.821 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.821 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.821 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "diskprediction_local", 2024-03-22T01:47:09.821 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.821 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.821 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.821 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.821 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.821 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.821 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.821 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.821 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.821 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.821 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.822 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.822 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.822 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.822 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.822 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.822 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.822 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.822 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.822 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.822 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.822 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.822 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.822 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.822 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.822 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.823 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.823 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.823 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.823 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.823 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.823 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.823 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.823 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.823 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.823 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.824 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.824 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.824 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.824 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.824 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.824 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.824 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.824 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.824 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.824 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.824 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.824 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.824 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.824 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.825 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.825 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.825 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.825 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.825 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.825 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.825 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.825 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.825 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.825 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.825 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.825 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.825 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.825 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.825 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.825 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.826 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.826 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.826 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.826 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.826 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.826 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.826 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.826 INFO:teuthology.orchestra.run.smithi072.stdout: "predict_interval": { 2024-03-22T01:47:09.826 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "predict_interval", 2024-03-22T01:47:09.826 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.826 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.826 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.826 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "86400", 2024-03-22T01:47:09.826 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.827 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.827 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.827 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.827 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.827 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.827 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.827 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.827 INFO:teuthology.orchestra.run.smithi072.stdout: "predictor_model": { 2024-03-22T01:47:09.827 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "predictor_model", 2024-03-22T01:47:09.827 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.827 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.827 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.827 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "prophetstor", 2024-03-22T01:47:09.827 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.828 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.828 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.828 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.828 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.828 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.828 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.828 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.828 INFO:teuthology.orchestra.run.smithi072.stdout: "sleep_interval": { 2024-03-22T01:47:09.828 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "sleep_interval", 2024-03-22T01:47:09.828 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.828 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.828 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.828 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "600", 2024-03-22T01:47:09.828 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.828 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.828 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.829 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.829 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.829 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.829 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.829 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.829 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.829 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.829 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.829 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "influx", 2024-03-22T01:47:09.829 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": false, 2024-03-22T01:47:09.829 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "influxdb python module not found", 2024-03-22T01:47:09.829 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.829 INFO:teuthology.orchestra.run.smithi072.stdout: "batch_size": { 2024-03-22T01:47:09.829 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "batch_size", 2024-03-22T01:47:09.829 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.830 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.830 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.830 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "5000", 2024-03-22T01:47:09.830 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.830 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.830 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.830 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-22T01:47:09.830 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.830 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.830 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.830 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.830 INFO:teuthology.orchestra.run.smithi072.stdout: "database": { 2024-03-22T01:47:09.830 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "database", 2024-03-22T01:47:09.830 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.831 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.831 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.831 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "ceph", 2024-03-22T01:47:09.831 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.831 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.831 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.831 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-22T01:47:09.831 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.831 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.831 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.831 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.831 INFO:teuthology.orchestra.run.smithi072.stdout: "hostname": { 2024-03-22T01:47:09.831 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "hostname", 2024-03-22T01:47:09.831 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.831 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.832 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.832 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.832 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.832 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.832 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.832 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "InfluxDB server hostname", 2024-03-22T01:47:09.832 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.832 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.832 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.832 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.832 INFO:teuthology.orchestra.run.smithi072.stdout: "interval": { 2024-03-22T01:47:09.832 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "interval", 2024-03-22T01:47:09.832 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "secs", 2024-03-22T01:47:09.832 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.832 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.833 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "30", 2024-03-22T01:47:09.833 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "5", 2024-03-22T01:47:09.833 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.833 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.833 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-22T01:47:09.833 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.833 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.833 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.833 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.833 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.833 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.833 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.833 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.833 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.834 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.834 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.834 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.834 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.834 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.834 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.834 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.834 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.834 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.834 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.834 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.834 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.834 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.834 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.834 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.835 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.835 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.835 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.835 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.835 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.835 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.835 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.835 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.835 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.835 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.835 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.835 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.835 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.835 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.835 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.836 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.836 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.836 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.836 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.836 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.836 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.836 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.836 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.836 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.836 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.836 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.836 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.836 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.836 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.836 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.836 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.837 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.837 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.837 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.837 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.837 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.837 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.837 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.837 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.837 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.837 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.837 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.837 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.837 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.837 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.838 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.838 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.838 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.838 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.838 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.838 INFO:teuthology.orchestra.run.smithi072.stdout: "password": { 2024-03-22T01:47:09.838 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "password", 2024-03-22T01:47:09.838 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.838 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.838 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.838 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.838 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.838 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.838 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.838 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "password of InfluxDB server user", 2024-03-22T01:47:09.839 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.839 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.839 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.839 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.839 INFO:teuthology.orchestra.run.smithi072.stdout: "port": { 2024-03-22T01:47:09.839 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "port", 2024-03-22T01:47:09.839 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.839 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.839 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.839 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "8086", 2024-03-22T01:47:09.839 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.839 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.839 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.840 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "InfluxDB server port", 2024-03-22T01:47:09.840 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.840 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.840 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.840 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.840 INFO:teuthology.orchestra.run.smithi072.stdout: "ssl": { 2024-03-22T01:47:09.840 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "ssl", 2024-03-22T01:47:09.840 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.840 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.840 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.840 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "false", 2024-03-22T01:47:09.840 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.840 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.840 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.840 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-22T01:47:09.841 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.841 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.841 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.841 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.841 INFO:teuthology.orchestra.run.smithi072.stdout: "threads": { 2024-03-22T01:47:09.841 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "threads", 2024-03-22T01:47:09.841 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.841 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.841 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.841 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "5", 2024-03-22T01:47:09.841 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "1", 2024-03-22T01:47:09.841 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "32", 2024-03-22T01:47:09.842 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.842 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-22T01:47:09.842 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.842 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.842 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.842 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.842 INFO:teuthology.orchestra.run.smithi072.stdout: "username": { 2024-03-22T01:47:09.842 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "username", 2024-03-22T01:47:09.842 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.842 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.842 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.842 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.842 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.842 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.843 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.843 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "username of InfluxDB server user", 2024-03-22T01:47:09.843 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.843 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.843 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.843 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.843 INFO:teuthology.orchestra.run.smithi072.stdout: "verify_ssl": { 2024-03-22T01:47:09.843 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "verify_ssl", 2024-03-22T01:47:09.843 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.843 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.843 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.843 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "true", 2024-03-22T01:47:09.843 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.843 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.843 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.844 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-22T01:47:09.844 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.844 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.844 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.844 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.844 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.844 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.844 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.844 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "insights", 2024-03-22T01:47:09.844 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.844 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.844 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.844 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.844 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.844 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.845 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.845 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.845 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.845 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.845 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.845 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.845 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.845 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.845 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.845 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.845 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.845 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.845 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.845 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.845 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.846 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.846 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.846 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.846 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.846 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.846 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.846 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.846 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.846 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.846 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.846 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.846 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.846 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.846 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.847 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.847 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.847 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.847 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.847 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.847 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.847 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.847 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.847 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.847 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.847 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.847 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.847 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.847 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.847 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.848 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.848 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.848 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.848 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.848 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.848 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.848 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.848 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.848 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.848 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.848 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.848 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.848 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.848 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.849 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.849 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.849 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.849 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.849 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.849 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.849 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.849 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.849 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.849 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.849 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.849 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.849 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "iostat", 2024-03-22T01:47:09.849 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.850 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.850 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.850 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.850 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.850 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.850 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.850 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.850 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.850 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.850 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.850 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.850 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.850 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.851 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.851 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.851 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.851 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.851 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.851 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.851 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.851 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.851 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.851 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.851 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.851 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.851 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.851 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.852 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.852 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.852 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.852 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.852 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.852 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.852 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.852 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.852 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.852 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.852 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.852 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.852 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.852 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.852 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.853 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.853 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.853 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.853 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.853 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.853 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.853 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.853 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.853 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.853 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.853 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.853 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.853 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.854 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.854 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.854 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.854 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.854 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.854 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.854 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.854 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.854 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.854 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.854 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.854 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.855 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.855 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.855 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.855 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.855 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.855 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.855 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.855 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.855 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "k8sevents", 2024-03-22T01:47:09.855 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.855 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.855 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.855 INFO:teuthology.orchestra.run.smithi072.stdout: "ceph_event_retention_days": { 2024-03-22T01:47:09.855 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "ceph_event_retention_days", 2024-03-22T01:47:09.855 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.856 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.856 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.856 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "7", 2024-03-22T01:47:09.856 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.856 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.856 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.856 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-22T01:47:09.856 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.856 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.856 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.856 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.856 INFO:teuthology.orchestra.run.smithi072.stdout: "config_check_secs": { 2024-03-22T01:47:09.856 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "config_check_secs", 2024-03-22T01:47:09.857 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.857 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.857 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.857 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "10", 2024-03-22T01:47:09.857 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "10", 2024-03-22T01:47:09.857 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.857 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.857 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-22T01:47:09.857 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.857 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.857 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.857 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.857 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.857 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.857 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.858 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.858 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.858 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.858 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.858 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.858 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.858 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.858 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.858 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.858 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.858 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.858 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.858 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.858 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.859 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.859 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.859 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.859 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.859 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.859 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.859 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.859 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.859 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.859 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.859 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.859 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.859 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.859 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.860 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.860 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.860 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.860 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.860 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.860 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.860 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.860 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.860 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.860 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.860 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.860 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.860 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.860 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.860 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.861 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.861 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.861 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.861 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.861 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.861 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.861 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.861 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.861 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.861 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.861 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.861 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.861 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.861 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.862 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.862 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.862 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.862 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.862 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.862 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.862 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.862 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.862 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.862 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.862 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.862 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.862 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.863 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "localpool", 2024-03-22T01:47:09.863 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.863 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.863 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.863 INFO:teuthology.orchestra.run.smithi072.stdout: "failure_domain": { 2024-03-22T01:47:09.863 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "failure_domain", 2024-03-22T01:47:09.863 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.863 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.863 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.863 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "host", 2024-03-22T01:47:09.863 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.863 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.863 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.864 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "failure domain for any created local pool", 2024-03-22T01:47:09.864 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-22T01:47:09.864 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.864 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.864 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.864 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.864 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.864 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.864 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.864 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.864 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.864 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.864 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.864 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.864 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.865 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.865 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.865 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.865 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.865 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.865 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.865 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.865 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.865 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.865 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.865 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.865 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.865 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.865 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.865 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.866 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.866 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.866 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.866 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.866 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.866 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.866 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.866 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.866 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.866 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.866 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.866 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.866 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.866 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.867 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.867 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.867 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.867 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.867 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.867 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.867 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.867 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.867 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.867 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.867 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.867 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.867 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.867 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.867 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.868 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.868 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.868 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.868 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.868 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.868 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.868 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.868 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.868 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.868 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.868 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.868 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.868 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.868 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.868 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.868 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.869 INFO:teuthology.orchestra.run.smithi072.stdout: "min_size": { 2024-03-22T01:47:09.869 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "min_size", 2024-03-22T01:47:09.869 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.869 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.869 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.869 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.869 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.869 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.869 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.869 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "default min_size for any created local pool", 2024-03-22T01:47:09.869 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-22T01:47:09.869 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.869 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.869 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.869 INFO:teuthology.orchestra.run.smithi072.stdout: "num_rep": { 2024-03-22T01:47:09.870 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "num_rep", 2024-03-22T01:47:09.870 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.870 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.870 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.870 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "3", 2024-03-22T01:47:09.870 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.870 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.870 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.870 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "default replica count for any created local pool", 2024-03-22T01:47:09.870 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.870 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.870 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.870 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.870 INFO:teuthology.orchestra.run.smithi072.stdout: "pg_num": { 2024-03-22T01:47:09.871 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "pg_num", 2024-03-22T01:47:09.871 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.871 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.871 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.871 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "128", 2024-03-22T01:47:09.871 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.871 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.871 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.871 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "default pg_num for any created local pool", 2024-03-22T01:47:09.871 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.871 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.871 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.871 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.871 INFO:teuthology.orchestra.run.smithi072.stdout: "prefix": { 2024-03-22T01:47:09.871 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "prefix", 2024-03-22T01:47:09.871 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.872 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.872 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.872 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.872 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.872 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.872 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.872 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "name prefix for any created local pool", 2024-03-22T01:47:09.872 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.872 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.872 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.872 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.872 INFO:teuthology.orchestra.run.smithi072.stdout: "subtree": { 2024-03-22T01:47:09.872 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "subtree", 2024-03-22T01:47:09.873 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.873 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.873 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.873 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "rack", 2024-03-22T01:47:09.873 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.873 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.873 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.873 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-22T01:47:09.873 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-22T01:47:09.873 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.873 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.873 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.873 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.873 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.873 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.874 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "mds_autoscaler", 2024-03-22T01:47:09.874 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.874 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.874 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.874 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.874 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.874 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.874 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.874 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.874 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.874 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.874 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.874 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.874 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.874 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.875 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.875 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.875 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.875 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.875 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.875 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.875 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.875 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.875 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.875 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.875 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.875 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.875 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.875 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.875 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.876 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.876 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.876 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.876 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.876 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.876 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.876 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.876 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.876 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.876 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.876 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.876 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.876 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.876 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.876 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.877 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.877 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.877 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.877 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.877 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.877 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.877 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.877 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.877 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.877 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.877 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.877 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.877 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.877 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.877 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.878 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.878 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.878 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.878 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.878 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.878 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.878 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.878 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.878 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.878 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.878 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.878 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.878 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.878 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.879 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.879 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.879 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.879 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "mirroring", 2024-03-22T01:47:09.879 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.879 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.879 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.879 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.879 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.879 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.879 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.879 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.879 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.879 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.879 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.880 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.880 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.880 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.880 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.880 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.880 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.880 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.880 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.880 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.880 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.880 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.880 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.880 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.880 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.881 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.881 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.881 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.881 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.881 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.881 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.881 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.881 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.881 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.881 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.881 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.881 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.881 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.881 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.882 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.882 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.882 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.882 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.882 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.882 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.882 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.882 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.882 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.882 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.882 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.882 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.882 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.883 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.883 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.883 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.883 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.883 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.883 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.883 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.883 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.883 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.883 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.883 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.883 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.883 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.883 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.884 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.884 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.884 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.884 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.884 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.884 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.884 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.884 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.884 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.884 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.884 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "nfs", 2024-03-22T01:47:09.884 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.884 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.884 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.885 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.885 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.885 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.885 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.885 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.885 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.885 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.885 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.885 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.885 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.885 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.885 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.885 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.885 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.885 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.886 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.886 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.886 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.886 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.886 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.886 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.886 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.886 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.886 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.886 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.886 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.886 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.886 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.886 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.886 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.887 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.887 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.887 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.887 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.887 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.887 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.887 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.887 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.887 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.887 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.887 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.887 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.887 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.887 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.888 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.888 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.888 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.888 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.888 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.888 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.888 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.888 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.888 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.888 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.888 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.888 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.888 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.888 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.888 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.889 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.889 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.889 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.889 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.889 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.889 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.889 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.889 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.889 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.889 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.889 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.889 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.889 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.890 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.890 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "orchestrator", 2024-03-22T01:47:09.890 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.890 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.890 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.890 INFO:teuthology.orchestra.run.smithi072.stdout: "fail_fs": { 2024-03-22T01:47:09.890 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "fail_fs", 2024-03-22T01:47:09.890 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.890 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.890 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.890 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.890 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.890 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.890 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.891 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-22T01:47:09.891 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.891 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.891 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.891 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.891 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.891 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.891 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.891 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.891 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.891 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.891 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.891 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.891 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.892 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.892 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.892 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.892 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.892 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.892 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.892 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.892 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.892 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.892 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.892 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.892 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.892 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.892 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.892 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.893 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.893 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.893 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.893 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.893 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.893 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.893 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.893 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.893 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.893 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.893 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.893 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.893 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.893 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.894 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.894 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.894 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.894 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.894 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.894 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.894 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.894 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.894 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.894 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.894 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.894 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.894 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.894 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.895 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.895 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.895 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.895 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.895 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.895 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.895 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.895 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.895 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.895 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.895 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.895 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.895 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.895 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.895 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.896 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.896 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.896 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.896 INFO:teuthology.orchestra.run.smithi072.stdout: "orchestrator": { 2024-03-22T01:47:09.896 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "orchestrator", 2024-03-22T01:47:09.896 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.896 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.896 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.896 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.896 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.896 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.896 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.896 INFO:teuthology.orchestra.run.smithi072.stdout: "cephadm", 2024-03-22T01:47:09.896 INFO:teuthology.orchestra.run.smithi072.stdout: "rook", 2024-03-22T01:47:09.896 INFO:teuthology.orchestra.run.smithi072.stdout: "test_orchestrator" 2024-03-22T01:47:09.897 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.897 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Orchestrator backend", 2024-03-22T01:47:09.897 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.897 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.897 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.897 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.897 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.897 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.897 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.897 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "osd_perf_query", 2024-03-22T01:47:09.897 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.897 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.897 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.897 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.898 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.898 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.898 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.898 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.898 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.898 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.898 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.898 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.898 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.898 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.898 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.898 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.898 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.898 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.898 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.899 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.899 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.899 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.899 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.899 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.899 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.899 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.899 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.899 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.899 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.899 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.899 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.899 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.899 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.899 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.900 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.900 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.900 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.900 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.900 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.900 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.900 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.900 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.900 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.900 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.900 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.900 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.900 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.900 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.900 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.901 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.901 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.901 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.901 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.901 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.901 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.901 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.901 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.901 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.901 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.901 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.901 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.901 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.901 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.901 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.902 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.902 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.902 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.902 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.902 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.902 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.902 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.902 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.902 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.902 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.902 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.902 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.902 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "osd_support", 2024-03-22T01:47:09.902 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.902 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.903 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.903 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.903 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.903 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.903 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.903 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.903 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.903 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.903 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.903 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.903 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.903 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.903 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.903 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.903 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.903 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.904 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.904 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.904 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.904 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.904 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.904 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.904 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.904 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.904 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.904 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.904 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.904 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.904 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.904 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.904 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.904 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.904 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.905 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.905 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.905 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.905 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.905 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.905 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.905 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.905 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.905 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.905 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.905 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.905 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.905 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.905 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.905 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.905 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.906 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.906 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.906 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.906 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.906 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.906 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.906 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.906 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.906 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.906 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.906 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.906 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.906 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.906 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.907 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.907 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.907 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.907 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.907 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.907 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.907 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.907 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.907 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.907 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.907 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.907 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "pg_autoscaler", 2024-03-22T01:47:09.907 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.907 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.908 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.908 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.908 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.908 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.908 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.908 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.908 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.908 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.908 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.908 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.908 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.908 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.908 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.908 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.909 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.909 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.909 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.909 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.909 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.909 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.909 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.909 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.909 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.909 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.909 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.909 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.910 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.910 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.910 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.910 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.910 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.910 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.910 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.910 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.910 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.910 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.910 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.910 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.911 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.911 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.911 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.911 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.911 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.911 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.911 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.911 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.911 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.911 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.911 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.911 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.911 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.911 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.912 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.912 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.912 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.912 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.912 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.912 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.912 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.912 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.912 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.912 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.912 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.912 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.912 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.913 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.913 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.913 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.913 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.913 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.913 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.913 INFO:teuthology.orchestra.run.smithi072.stdout: "sleep_interval": { 2024-03-22T01:47:09.913 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "sleep_interval", 2024-03-22T01:47:09.913 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "secs", 2024-03-22T01:47:09.913 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.913 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.913 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "60", 2024-03-22T01:47:09.913 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.914 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.914 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.914 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.914 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.914 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.914 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.914 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.914 INFO:teuthology.orchestra.run.smithi072.stdout: "threshold": { 2024-03-22T01:47:09.914 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "threshold", 2024-03-22T01:47:09.914 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "float", 2024-03-22T01:47:09.914 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.914 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.914 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "3.0", 2024-03-22T01:47:09.914 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "1.0", 2024-03-22T01:47:09.914 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.915 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.915 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "scaling threshold", 2024-03-22T01:47:09.915 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-22T01:47:09.915 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.915 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.915 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.915 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.915 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.915 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.915 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "progress", 2024-03-22T01:47:09.915 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.915 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.915 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.915 INFO:teuthology.orchestra.run.smithi072.stdout: "allow_pg_recovery_event": { 2024-03-22T01:47:09.915 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "allow_pg_recovery_event", 2024-03-22T01:47:09.916 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.916 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.916 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.916 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.916 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.916 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.916 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.916 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-22T01:47:09.916 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.916 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.916 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.916 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.916 INFO:teuthology.orchestra.run.smithi072.stdout: "enabled": { 2024-03-22T01:47:09.916 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "enabled", 2024-03-22T01:47:09.916 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.917 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.917 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.917 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.917 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.917 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.917 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.917 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.917 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.917 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.917 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.917 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.917 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.917 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.917 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.918 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.918 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.918 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.918 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.918 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.918 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.918 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.918 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.918 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.918 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.918 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.918 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.918 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.918 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.918 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.919 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.919 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.919 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.919 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.919 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.919 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.919 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.919 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.919 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.919 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.919 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.919 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.919 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.919 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.919 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.920 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.920 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.920 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.920 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.920 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.920 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.920 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.920 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.920 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.920 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.920 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.920 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.920 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.921 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.921 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.921 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.921 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.921 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.921 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.921 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.921 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.921 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.921 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.921 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.921 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.921 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.921 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.922 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.922 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.922 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.922 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.922 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.922 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.922 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.922 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.922 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.922 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.922 INFO:teuthology.orchestra.run.smithi072.stdout: "max_completed_events": { 2024-03-22T01:47:09.922 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "max_completed_events", 2024-03-22T01:47:09.922 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.922 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.922 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.923 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "50", 2024-03-22T01:47:09.923 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.923 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.923 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.923 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "number of past completed events to remember", 2024-03-22T01:47:09.923 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.923 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.923 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.923 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.923 INFO:teuthology.orchestra.run.smithi072.stdout: "sleep_interval": { 2024-03-22T01:47:09.923 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "sleep_interval", 2024-03-22T01:47:09.923 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "secs", 2024-03-22T01:47:09.923 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.923 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.924 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "5", 2024-03-22T01:47:09.924 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.924 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.924 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.924 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "how long the module is going to sleep", 2024-03-22T01:47:09.924 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.924 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.924 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.924 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.924 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.924 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.924 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.924 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "prometheus", 2024-03-22T01:47:09.924 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.924 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.925 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.925 INFO:teuthology.orchestra.run.smithi072.stdout: "cache": { 2024-03-22T01:47:09.925 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "cache", 2024-03-22T01:47:09.925 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.925 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.925 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.925 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.925 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.925 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.925 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.925 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.925 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.925 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.925 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.925 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.926 INFO:teuthology.orchestra.run.smithi072.stdout: "exclude_perf_counters": { 2024-03-22T01:47:09.926 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "exclude_perf_counters", 2024-03-22T01:47:09.926 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.926 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.926 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.926 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.926 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.926 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.926 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.926 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-22T01:47:09.926 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-22T01:47:09.926 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.926 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.927 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.927 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.927 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.927 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.927 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.927 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.927 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.927 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.927 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.927 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.927 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.927 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.927 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.927 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.928 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.928 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.928 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.928 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.928 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.928 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.928 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.928 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.928 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.928 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.928 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.928 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.928 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.928 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.929 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.929 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.929 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.929 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.929 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.929 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.929 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.929 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.929 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.929 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.929 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.929 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.929 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.930 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.930 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.930 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.930 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.930 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.930 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.930 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.930 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.930 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.930 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.930 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.930 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.930 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.930 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.931 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.931 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.931 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.931 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.931 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.931 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.931 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.931 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.931 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.931 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.931 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.931 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.931 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.931 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.931 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.932 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.932 INFO:teuthology.orchestra.run.smithi072.stdout: "rbd_stats_pools": { 2024-03-22T01:47:09.932 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "rbd_stats_pools", 2024-03-22T01:47:09.932 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.932 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.932 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.932 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.932 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.932 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.932 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.932 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.932 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.932 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.932 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.933 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.933 INFO:teuthology.orchestra.run.smithi072.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-22T01:47:09.933 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-22T01:47:09.933 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.933 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.933 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.933 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "300", 2024-03-22T01:47:09.933 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.933 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.933 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.933 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.933 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.934 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.934 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.934 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.934 INFO:teuthology.orchestra.run.smithi072.stdout: "scrape_interval": { 2024-03-22T01:47:09.934 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "scrape_interval", 2024-03-22T01:47:09.934 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "float", 2024-03-22T01:47:09.934 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.934 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.934 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "15.0", 2024-03-22T01:47:09.934 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.934 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.934 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.934 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.935 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.935 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.935 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.935 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.935 INFO:teuthology.orchestra.run.smithi072.stdout: "server_addr": { 2024-03-22T01:47:09.935 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "server_addr", 2024-03-22T01:47:09.935 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.935 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.935 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.935 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "::", 2024-03-22T01:47:09.935 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.935 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.935 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.935 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-22T01:47:09.936 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.936 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.936 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.936 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.936 INFO:teuthology.orchestra.run.smithi072.stdout: "server_port": { 2024-03-22T01:47:09.936 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "server_port", 2024-03-22T01:47:09.936 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.936 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.936 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.936 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "9283", 2024-03-22T01:47:09.936 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.936 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.936 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.937 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-22T01:47:09.937 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.937 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.937 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.937 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.937 INFO:teuthology.orchestra.run.smithi072.stdout: "stale_cache_strategy": { 2024-03-22T01:47:09.937 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "stale_cache_strategy", 2024-03-22T01:47:09.937 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.937 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.937 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.937 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "log", 2024-03-22T01:47:09.937 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.937 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.938 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.938 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.938 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.938 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.938 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.938 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.938 INFO:teuthology.orchestra.run.smithi072.stdout: "standby_behaviour": { 2024-03-22T01:47:09.938 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "standby_behaviour", 2024-03-22T01:47:09.938 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.938 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.938 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.938 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "default", 2024-03-22T01:47:09.938 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.938 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.939 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.939 INFO:teuthology.orchestra.run.smithi072.stdout: "default", 2024-03-22T01:47:09.939 INFO:teuthology.orchestra.run.smithi072.stdout: "error" 2024-03-22T01:47:09.939 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.939 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.939 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.939 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.939 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.939 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.939 INFO:teuthology.orchestra.run.smithi072.stdout: "standby_error_status_code": { 2024-03-22T01:47:09.939 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "standby_error_status_code", 2024-03-22T01:47:09.939 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.939 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.939 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.940 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "500", 2024-03-22T01:47:09.940 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "400", 2024-03-22T01:47:09.940 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "599", 2024-03-22T01:47:09.940 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.940 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.940 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.940 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.940 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.940 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.940 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.940 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.940 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.940 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "rbd_support", 2024-03-22T01:47:09.940 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.941 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.941 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.941 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.941 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.941 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.941 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.941 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.941 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.941 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.941 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.941 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.941 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.941 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.941 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.942 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.942 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.942 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.942 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.942 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.942 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.942 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.942 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.942 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.942 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.942 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.942 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.942 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.943 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.943 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.943 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.943 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.943 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.943 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.943 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.943 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.943 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.943 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.943 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.943 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.944 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.944 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.944 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.944 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.944 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.944 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.944 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.944 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.944 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.944 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.944 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.944 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.944 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.944 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.944 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.945 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.945 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.945 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.945 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.945 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.945 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.945 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.945 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.945 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.945 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.945 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.945 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.945 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.945 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.946 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.946 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.946 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.946 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.946 INFO:teuthology.orchestra.run.smithi072.stdout: "max_concurrent_snap_create": { 2024-03-22T01:47:09.946 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "max_concurrent_snap_create", 2024-03-22T01:47:09.946 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.946 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.946 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.946 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "10", 2024-03-22T01:47:09.946 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.946 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.946 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.947 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.947 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.947 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.947 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.947 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.947 INFO:teuthology.orchestra.run.smithi072.stdout: "mirror_snapshot_schedule": { 2024-03-22T01:47:09.947 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "mirror_snapshot_schedule", 2024-03-22T01:47:09.947 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.947 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.947 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.947 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.949 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.949 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.949 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.949 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.950 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.950 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.950 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.950 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.950 INFO:teuthology.orchestra.run.smithi072.stdout: "trash_purge_schedule": { 2024-03-22T01:47:09.950 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "trash_purge_schedule", 2024-03-22T01:47:09.950 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.950 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.950 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.950 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.950 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.950 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.950 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.951 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.951 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.951 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.951 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.951 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.951 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.951 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.951 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.951 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "restful", 2024-03-22T01:47:09.951 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.951 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.951 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.951 INFO:teuthology.orchestra.run.smithi072.stdout: "enable_auth": { 2024-03-22T01:47:09.951 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "enable_auth", 2024-03-22T01:47:09.952 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.952 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.952 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.952 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.952 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.952 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.952 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.952 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.952 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.952 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.952 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.952 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.952 INFO:teuthology.orchestra.run.smithi072.stdout: "key_file": { 2024-03-22T01:47:09.953 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "key_file", 2024-03-22T01:47:09.953 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.953 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.953 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.953 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.953 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.953 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.953 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.953 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.953 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.953 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.953 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.953 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.953 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.954 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.954 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.954 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.954 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.954 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.954 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.954 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.954 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.954 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.954 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.954 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.954 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.955 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.955 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.955 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.955 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.955 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.955 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.955 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.955 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.955 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.955 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.955 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.955 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.955 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.956 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.956 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.956 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.956 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.956 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.956 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.956 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.956 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.956 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.956 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.956 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.956 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.956 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.956 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.957 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.957 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.957 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.957 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.957 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.957 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.957 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.957 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.957 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.957 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.957 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.957 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.957 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.958 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.958 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.958 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.958 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.958 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.958 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.958 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.958 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.958 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.958 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.958 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.958 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.958 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.959 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.959 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.959 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.959 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.959 INFO:teuthology.orchestra.run.smithi072.stdout: "server_addr": { 2024-03-22T01:47:09.959 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "server_addr", 2024-03-22T01:47:09.959 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.959 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.959 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.959 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.959 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.959 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.959 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.959 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.960 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.960 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.960 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.960 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.960 INFO:teuthology.orchestra.run.smithi072.stdout: "server_port": { 2024-03-22T01:47:09.960 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "server_port", 2024-03-22T01:47:09.960 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.960 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.960 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.960 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.960 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.960 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.960 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.961 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.961 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.961 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.961 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.961 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.961 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.961 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.961 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.961 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "rgw", 2024-03-22T01:47:09.961 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.961 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.961 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.961 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.962 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.962 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.962 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.962 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.962 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.962 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.962 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.962 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.962 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.962 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.962 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.962 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.962 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.962 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.963 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.963 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.963 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.963 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.963 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.963 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.963 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.963 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.963 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.963 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.963 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.963 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.963 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.963 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.963 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.963 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.964 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.964 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.964 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.964 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.964 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.964 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.964 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.964 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.964 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.964 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.964 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.964 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.964 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.964 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.964 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.965 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.965 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.965 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.965 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.965 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.965 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.965 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.965 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.965 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.965 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.965 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.965 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.965 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.965 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.966 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.966 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.966 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.966 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.966 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.966 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.966 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.966 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.966 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.966 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.966 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.966 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.966 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.966 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "rook", 2024-03-22T01:47:09.967 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.967 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.967 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.967 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.967 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.967 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.967 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.967 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.967 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.967 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.967 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.967 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.967 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.967 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.967 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.968 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.968 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.968 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.968 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.968 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.968 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.968 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.968 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.968 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.968 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.968 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.968 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.968 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.968 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.968 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.968 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.969 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.969 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.969 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.969 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.969 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.969 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.969 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.969 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.969 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.969 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.969 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.969 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.969 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.969 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.970 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.970 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.970 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.970 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.970 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.970 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.970 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.970 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.970 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.970 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.970 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.970 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.970 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.970 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.971 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.971 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.971 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.971 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.971 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.971 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.971 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.971 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.971 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.971 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.971 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.971 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.971 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.971 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.971 INFO:teuthology.orchestra.run.smithi072.stdout: "storage_class": { 2024-03-22T01:47:09.972 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "storage_class", 2024-03-22T01:47:09.972 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.972 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.972 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.972 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "local", 2024-03-22T01:47:09.972 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.972 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.972 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.972 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-22T01:47:09.972 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.972 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.972 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.972 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.972 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.972 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.972 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.973 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "selftest", 2024-03-22T01:47:09.973 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.973 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.973 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.973 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.973 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.973 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.973 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.973 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.973 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.973 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.973 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.973 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.974 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.974 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.974 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.974 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.974 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.974 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.974 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.974 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.974 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.974 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.974 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.974 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.974 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.974 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.974 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.975 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.975 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.975 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.975 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.975 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.975 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.975 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.975 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.975 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.975 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.975 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.975 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.975 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.975 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.975 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.976 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.976 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.976 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.976 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.976 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.976 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.976 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.976 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.976 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.976 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.976 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.976 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.976 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.976 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.976 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.977 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.977 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.977 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.977 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.977 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.977 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.977 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.977 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.977 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.977 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.977 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.977 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.977 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.977 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.978 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.978 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.978 INFO:teuthology.orchestra.run.smithi072.stdout: "roption1": { 2024-03-22T01:47:09.978 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "roption1", 2024-03-22T01:47:09.978 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.978 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.978 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.978 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.978 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.978 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.978 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.978 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.978 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.978 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.979 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.979 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.979 INFO:teuthology.orchestra.run.smithi072.stdout: "roption2": { 2024-03-22T01:47:09.979 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "roption2", 2024-03-22T01:47:09.979 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.979 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.979 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.979 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "xyz", 2024-03-22T01:47:09.979 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.979 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.979 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.979 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.979 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.979 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.979 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.980 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.980 INFO:teuthology.orchestra.run.smithi072.stdout: "rwoption1": { 2024-03-22T01:47:09.980 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "rwoption1", 2024-03-22T01:47:09.980 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.980 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.980 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.980 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.980 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.980 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.980 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.980 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.980 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.980 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.980 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.980 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.981 INFO:teuthology.orchestra.run.smithi072.stdout: "rwoption2": { 2024-03-22T01:47:09.981 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "rwoption2", 2024-03-22T01:47:09.981 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.981 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.981 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.981 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.981 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.981 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.981 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.981 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.981 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.981 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.981 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.981 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.982 INFO:teuthology.orchestra.run.smithi072.stdout: "rwoption3": { 2024-03-22T01:47:09.982 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "rwoption3", 2024-03-22T01:47:09.982 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "float", 2024-03-22T01:47:09.982 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.982 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.982 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.982 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.982 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.982 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.982 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.982 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.982 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.982 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.982 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.983 INFO:teuthology.orchestra.run.smithi072.stdout: "rwoption4": { 2024-03-22T01:47:09.983 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "rwoption4", 2024-03-22T01:47:09.983 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.983 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.983 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.983 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.983 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.983 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.983 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.983 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.983 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.983 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.983 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.983 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.983 INFO:teuthology.orchestra.run.smithi072.stdout: "rwoption5": { 2024-03-22T01:47:09.984 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "rwoption5", 2024-03-22T01:47:09.984 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.984 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.984 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.984 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.984 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.984 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.984 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.984 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.984 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.984 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.984 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.984 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.984 INFO:teuthology.orchestra.run.smithi072.stdout: "rwoption6": { 2024-03-22T01:47:09.984 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "rwoption6", 2024-03-22T01:47:09.984 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.985 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.985 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.985 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:09.985 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.985 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.985 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.985 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.985 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.985 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.985 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.985 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.985 INFO:teuthology.orchestra.run.smithi072.stdout: "rwoption7": { 2024-03-22T01:47:09.985 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "rwoption7", 2024-03-22T01:47:09.985 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:09.985 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.986 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.986 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.986 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "1", 2024-03-22T01:47:09.986 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "42", 2024-03-22T01:47:09.986 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.986 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.986 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.986 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.986 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.986 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.986 INFO:teuthology.orchestra.run.smithi072.stdout: "testkey": { 2024-03-22T01:47:09.986 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "testkey", 2024-03-22T01:47:09.986 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.986 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.987 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.987 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.987 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.987 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.987 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.987 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.987 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.987 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.987 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.987 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.987 INFO:teuthology.orchestra.run.smithi072.stdout: "testlkey": { 2024-03-22T01:47:09.987 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "testlkey", 2024-03-22T01:47:09.987 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.987 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.987 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.988 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.988 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.988 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.988 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.988 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.988 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.988 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.988 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.988 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.988 INFO:teuthology.orchestra.run.smithi072.stdout: "testnewline": { 2024-03-22T01:47:09.988 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "testnewline", 2024-03-22T01:47:09.988 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.988 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.988 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:09.988 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.988 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.989 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.989 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.989 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.989 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.989 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.989 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.989 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.989 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.989 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.989 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.989 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "snap_schedule", 2024-03-22T01:47:09.989 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.989 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.989 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.989 INFO:teuthology.orchestra.run.smithi072.stdout: "allow_m_granularity": { 2024-03-22T01:47:09.989 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "allow_m_granularity", 2024-03-22T01:47:09.990 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.990 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.990 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.990 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.990 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.990 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.990 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.990 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "allow minute scheduled snapshots", 2024-03-22T01:47:09.990 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.990 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.990 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.990 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.990 INFO:teuthology.orchestra.run.smithi072.stdout: "dump_on_update": { 2024-03-22T01:47:09.990 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "dump_on_update", 2024-03-22T01:47:09.991 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.991 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.991 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.991 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.991 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.991 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.991 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.991 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "dump database to debug log on update", 2024-03-22T01:47:09.991 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.991 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.991 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.991 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.991 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.991 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.991 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.991 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.992 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.992 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.992 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.992 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.992 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.992 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.992 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.992 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.992 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.992 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.992 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.992 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.992 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.992 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.992 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.993 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.993 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.993 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.993 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.993 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.993 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.993 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.993 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.993 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.993 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.993 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.993 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.993 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.993 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.993 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.993 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.994 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.994 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.994 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.994 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.994 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.994 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.994 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.994 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.994 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.994 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.994 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.994 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.994 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.994 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.995 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.995 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.995 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.995 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.995 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.995 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.995 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.995 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:09.995 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:09.995 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.995 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.995 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.995 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.995 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.995 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.995 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.996 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.996 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.996 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.996 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.996 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.996 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:09.996 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.996 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:09.996 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "stats", 2024-03-22T01:47:09.996 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:09.996 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:09.996 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:09.996 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:09.996 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:09.996 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.996 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.997 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.997 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:09.997 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.997 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.997 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:09.997 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:09.997 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:09.997 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:09.997 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:09.997 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:09.997 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:09.998 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:09.998 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.998 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.998 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.998 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.998 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.998 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:09.998 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:09.998 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:09.998 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.998 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.998 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:09.998 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.998 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.999 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:09.999 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:09.999 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:09.999 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:09.999 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:09.999 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:09.999 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:09.999 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:09.999 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:09.999 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:09.999 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:09.999 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:09.999 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:09.999 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:09.999 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:10.000 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:10.000 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:10.000 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:10.000 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:10.000 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:10.000 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:10.000 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:10.000 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:10.000 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:10.000 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:10.000 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:10.000 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:10.000 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:10.000 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:10.000 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:10.001 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:10.001 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:10.001 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:10.001 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:10.001 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:10.001 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:10.001 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:10.001 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:10.001 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:10.001 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:10.001 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:10.001 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:10.002 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:10.002 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:10.002 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "status", 2024-03-22T01:47:10.002 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:10.002 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:10.002 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:10.002 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:10.002 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:10.002 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:10.002 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:10.002 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:10.002 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:10.002 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:10.002 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:10.003 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:10.003 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:10.003 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:10.003 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:10.003 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:10.003 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:10.003 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:10.003 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:10.003 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:10.003 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:10.003 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:10.003 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:10.003 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:10.004 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:10.004 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:10.004 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:10.004 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:10.004 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:10.004 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:10.004 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:10.004 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:10.004 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:10.004 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:10.004 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:10.004 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:10.004 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:10.004 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:10.004 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:10.005 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:10.005 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:10.005 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:10.005 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:10.005 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:10.005 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:10.005 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:10.005 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:10.005 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:10.005 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:10.005 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:10.005 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:10.005 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:10.005 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:10.006 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:10.006 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:10.006 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:10.006 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:10.006 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:10.006 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:10.006 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:10.006 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:10.006 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:10.006 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:10.006 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:10.006 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:10.006 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:10.007 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:10.007 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:10.007 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:10.007 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:10.007 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:10.007 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:10.007 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:10.007 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:10.007 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:10.007 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:10.007 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "telegraf", 2024-03-22T01:47:10.007 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:10.007 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:10.008 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:10.008 INFO:teuthology.orchestra.run.smithi072.stdout: "address": { 2024-03-22T01:47:10.008 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "address", 2024-03-22T01:47:10.008 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:10.008 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:10.008 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:10.008 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-22T01:47:10.008 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:10.008 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:10.008 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:10.008 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:10.008 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:10.008 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:10.008 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:10.009 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:10.009 INFO:teuthology.orchestra.run.smithi072.stdout: "interval": { 2024-03-22T01:47:10.009 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "interval", 2024-03-22T01:47:10.009 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "secs", 2024-03-22T01:47:10.009 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:10.009 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:10.009 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "15", 2024-03-22T01:47:10.009 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:10.009 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:10.009 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:10.009 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:10.009 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:10.009 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:10.009 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:10.010 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:10.010 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:10.010 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:10.010 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:10.010 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:10.010 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:10.010 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:10.010 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:10.010 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:10.010 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:10.010 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:10.010 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:10.010 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:10.010 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:10.011 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:10.011 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:10.011 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:10.011 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:10.011 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:10.011 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:10.011 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:10.011 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:10.011 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:10.011 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:10.011 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:10.011 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:10.011 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:10.012 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:10.012 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:10.012 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:10.012 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:10.012 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:10.012 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:10.012 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:10.012 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:10.012 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:10.012 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:10.012 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:10.012 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:10.012 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:10.012 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:10.013 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:10.013 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:10.013 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:10.013 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:10.013 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:10.013 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:10.013 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:10.013 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:10.013 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:10.013 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:10.013 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:10.013 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:10.013 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:10.014 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:10.014 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:10.014 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:10.014 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:10.014 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:10.014 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:10.014 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:10.014 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:10.014 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:10.014 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:10.014 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:10.014 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:10.014 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:10.015 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:10.015 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:10.015 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:10.015 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:10.015 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:10.015 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:10.015 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:10.015 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "telemetry", 2024-03-22T01:47:10.015 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:10.015 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:10.015 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:10.015 INFO:teuthology.orchestra.run.smithi072.stdout: "channel_basic": { 2024-03-22T01:47:10.015 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "channel_basic", 2024-03-22T01:47:10.015 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:10.015 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:10.016 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:10.016 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:10.016 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:10.016 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:10.016 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:10.016 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-22T01:47:10.016 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:10.016 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:10.016 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:10.016 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:10.016 INFO:teuthology.orchestra.run.smithi072.stdout: "channel_crash": { 2024-03-22T01:47:10.016 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "channel_crash", 2024-03-22T01:47:10.016 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:10.016 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:10.016 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:10.017 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:10.017 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:10.017 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:10.017 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:10.017 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-22T01:47:10.017 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:10.017 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:10.017 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:10.017 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:10.017 INFO:teuthology.orchestra.run.smithi072.stdout: "channel_device": { 2024-03-22T01:47:10.017 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "channel_device", 2024-03-22T01:47:10.017 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:10.017 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:10.017 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:10.017 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:10.018 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:10.018 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:10.018 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:10.018 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-22T01:47:10.018 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:10.018 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:10.018 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:10.018 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:10.018 INFO:teuthology.orchestra.run.smithi072.stdout: "channel_ident": { 2024-03-22T01:47:10.018 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "channel_ident", 2024-03-22T01:47:10.018 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:10.018 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:10.018 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:10.019 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:10.019 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:10.019 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:10.019 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:10.019 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-22T01:47:10.019 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:10.019 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:10.019 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:10.019 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:10.019 INFO:teuthology.orchestra.run.smithi072.stdout: "channel_perf": { 2024-03-22T01:47:10.019 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "channel_perf", 2024-03-22T01:47:10.019 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:10.019 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:10.019 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:10.020 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:10.020 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:10.020 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:10.020 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:10.020 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-22T01:47:10.020 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:10.020 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:10.020 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:10.020 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:10.020 INFO:teuthology.orchestra.run.smithi072.stdout: "contact": { 2024-03-22T01:47:10.020 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "contact", 2024-03-22T01:47:10.020 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:10.020 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:10.020 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:10.020 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:10.021 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:10.021 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:10.021 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:10.021 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:10.021 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:10.021 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:10.021 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:10.021 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:10.021 INFO:teuthology.orchestra.run.smithi072.stdout: "description": { 2024-03-22T01:47:10.021 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "description", 2024-03-22T01:47:10.021 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:10.021 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:10.021 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:10.021 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:10.021 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:10.021 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:10.022 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:10.022 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:10.022 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:10.022 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:10.022 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:10.022 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:10.022 INFO:teuthology.orchestra.run.smithi072.stdout: "device_url": { 2024-03-22T01:47:10.022 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "device_url", 2024-03-22T01:47:10.022 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:10.022 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:10.022 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:10.022 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-22T01:47:10.022 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:10.022 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:10.023 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:10.023 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:10.023 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:10.023 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:10.023 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:10.023 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:10.023 INFO:teuthology.orchestra.run.smithi072.stdout: "enabled": { 2024-03-22T01:47:10.023 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "enabled", 2024-03-22T01:47:10.023 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:10.023 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:10.023 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:10.023 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:10.023 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:10.023 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:10.023 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:10.024 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:10.024 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:10.024 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:10.024 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:10.024 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:10.024 INFO:teuthology.orchestra.run.smithi072.stdout: "interval": { 2024-03-22T01:47:10.024 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "interval", 2024-03-22T01:47:10.024 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:10.024 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:10.024 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:10.024 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "24", 2024-03-22T01:47:10.024 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "8", 2024-03-22T01:47:10.024 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:10.024 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:10.024 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:10.025 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:10.025 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:10.025 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:10.025 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:10.025 INFO:teuthology.orchestra.run.smithi072.stdout: "last_opt_revision": { 2024-03-22T01:47:10.025 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "last_opt_revision", 2024-03-22T01:47:10.025 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:10.025 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:10.025 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:10.025 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "1", 2024-03-22T01:47:10.025 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:10.025 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:10.025 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:10.025 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:10.026 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:10.026 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:10.026 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:10.026 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:10.026 INFO:teuthology.orchestra.run.smithi072.stdout: "leaderboard": { 2024-03-22T01:47:10.026 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "leaderboard", 2024-03-22T01:47:10.026 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:10.026 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:10.026 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:10.026 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:10.026 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:10.026 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:10.026 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:10.026 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:10.027 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:10.027 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:10.027 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:10.027 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:10.027 INFO:teuthology.orchestra.run.smithi072.stdout: "leaderboard_description": { 2024-03-22T01:47:10.027 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "leaderboard_description", 2024-03-22T01:47:10.027 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:10.027 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:10.027 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:10.027 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:10.027 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:10.027 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:10.027 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:10.028 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:10.028 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:10.028 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:10.028 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:10.028 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:10.028 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:10.028 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:10.028 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:10.028 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:10.028 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:10.028 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:10.028 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:10.028 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:10.028 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:10.028 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:10.029 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:10.029 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:10.029 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:10.029 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:10.029 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:10.029 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:10.029 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:10.029 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:10.029 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:10.029 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:10.029 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:10.029 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:10.029 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:10.029 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:10.029 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:10.030 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:10.030 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:10.030 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:10.030 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:10.030 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:10.030 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:10.030 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:10.030 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:10.030 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:10.030 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:10.030 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:10.030 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:10.030 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:10.030 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:10.031 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:10.031 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:10.031 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:10.031 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:10.031 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:10.031 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:10.031 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:10.031 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:10.031 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:10.031 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:10.031 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:10.031 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:10.031 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:10.031 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:10.031 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:10.032 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:10.032 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:10.032 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:10.032 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:10.032 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:10.032 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:10.032 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:10.032 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:10.032 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:10.032 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:10.032 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:10.032 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:10.032 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:10.032 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:10.033 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:10.033 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:10.033 INFO:teuthology.orchestra.run.smithi072.stdout: "organization": { 2024-03-22T01:47:10.033 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "organization", 2024-03-22T01:47:10.033 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:10.033 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:10.033 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:10.033 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:10.033 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:10.033 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:10.033 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:10.033 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:10.033 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:10.033 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:10.033 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:10.033 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:10.034 INFO:teuthology.orchestra.run.smithi072.stdout: "proxy": { 2024-03-22T01:47:10.034 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "proxy", 2024-03-22T01:47:10.034 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:10.034 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:10.034 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:10.034 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:10.034 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:10.034 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:10.034 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:10.034 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:10.034 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:10.034 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:10.034 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:10.034 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:10.034 INFO:teuthology.orchestra.run.smithi072.stdout: "url": { 2024-03-22T01:47:10.035 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "url", 2024-03-22T01:47:10.035 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:10.035 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:10.035 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:10.035 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-22T01:47:10.035 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:10.035 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:10.035 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:10.035 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:10.035 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:10.035 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:10.035 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:10.035 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:10.035 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:10.035 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:10.036 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:10.036 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "test_orchestrator", 2024-03-22T01:47:10.036 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:10.036 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:10.036 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:10.036 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:10.036 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:10.036 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:10.036 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:10.036 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:10.036 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:10.036 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:10.036 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:10.036 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:10.036 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:10.036 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:10.037 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:10.037 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:10.037 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:10.037 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:10.037 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:10.037 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:10.037 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:10.037 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:10.037 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:10.037 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:10.037 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:10.037 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:10.037 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:10.038 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:10.038 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:10.038 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:10.038 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:10.038 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:10.038 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:10.038 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:10.038 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:10.038 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:10.038 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:10.038 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:10.038 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:10.038 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:10.038 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:10.038 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:10.039 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:10.039 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:10.039 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:10.039 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:10.039 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:10.039 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:10.039 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:10.039 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:10.039 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:10.039 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:10.039 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:10.039 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:10.039 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:10.039 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:10.039 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:10.039 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:10.040 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:10.040 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:10.040 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:10.040 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:10.040 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:10.040 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:10.040 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:10.040 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:10.040 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:10.040 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:10.040 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:10.040 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:10.040 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:10.040 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:10.040 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:10.041 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:10.041 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:10.041 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:10.041 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "volumes", 2024-03-22T01:47:10.041 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:10.041 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:10.041 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:10.041 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:10.041 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:10.041 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:10.041 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:10.041 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:10.041 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:10.041 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:10.041 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:10.042 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:10.042 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:10.042 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:10.042 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:10.042 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:10.042 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:10.042 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:10.042 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:10.042 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:10.042 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:10.042 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:10.042 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:10.042 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:10.042 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:10.043 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:10.043 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:10.043 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:10.043 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:10.043 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:10.043 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:10.043 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:10.043 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:10.043 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:10.043 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:10.043 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:10.043 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:10.043 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:10.043 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:10.043 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:10.043 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:10.044 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:10.044 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:10.044 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:10.044 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:10.044 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:10.044 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:10.044 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:10.044 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:10.044 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:10.044 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:10.044 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:10.044 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:10.044 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:10.044 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:10.044 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:10.045 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:10.045 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:10.045 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:10.045 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:10.045 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:10.045 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:10.045 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:10.045 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:10.045 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:10.045 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:10.045 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:10.045 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:10.045 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:10.045 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:10.045 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:10.046 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:10.046 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:10.046 INFO:teuthology.orchestra.run.smithi072.stdout: "max_concurrent_clones": { 2024-03-22T01:47:10.046 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "max_concurrent_clones", 2024-03-22T01:47:10.046 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:10.046 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:10.046 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:10.046 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "4", 2024-03-22T01:47:10.046 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:10.046 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:10.046 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:10.046 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-22T01:47:10.046 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:10.046 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:10.047 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:10.047 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:10.047 INFO:teuthology.orchestra.run.smithi072.stdout: "periodic_async_work": { 2024-03-22T01:47:10.047 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "periodic_async_work", 2024-03-22T01:47:10.047 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:10.047 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:10.047 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:10.047 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:10.047 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:10.047 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:10.047 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:10.047 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Periodically check for async work", 2024-03-22T01:47:10.047 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:10.047 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:10.048 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:10.048 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:10.048 INFO:teuthology.orchestra.run.smithi072.stdout: "snapshot_clone_delay": { 2024-03-22T01:47:10.048 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "snapshot_clone_delay", 2024-03-22T01:47:10.048 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:10.048 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:10.048 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:10.048 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "0", 2024-03-22T01:47:10.048 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:10.048 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:10.048 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:10.048 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-22T01:47:10.048 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:10.048 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:10.048 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:10.048 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:10.049 INFO:teuthology.orchestra.run.smithi072.stdout: "snapshot_clone_no_wait": { 2024-03-22T01:47:10.049 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "snapshot_clone_no_wait", 2024-03-22T01:47:10.049 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:10.049 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:10.049 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:10.049 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "True", 2024-03-22T01:47:10.049 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:10.049 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:10.049 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:10.049 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-22T01:47:10.049 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:10.049 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:10.049 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:10.049 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:10.049 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:10.050 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:10.050 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:10.050 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "zabbix", 2024-03-22T01:47:10.050 INFO:teuthology.orchestra.run.smithi072.stdout: "can_run": true, 2024-03-22T01:47:10.050 INFO:teuthology.orchestra.run.smithi072.stdout: "error_string": "", 2024-03-22T01:47:10.050 INFO:teuthology.orchestra.run.smithi072.stdout: "module_options": { 2024-03-22T01:47:10.050 INFO:teuthology.orchestra.run.smithi072.stdout: "discovery_interval": { 2024-03-22T01:47:10.050 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "discovery_interval", 2024-03-22T01:47:10.050 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "uint", 2024-03-22T01:47:10.050 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:10.050 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:10.050 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "100", 2024-03-22T01:47:10.050 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:10.050 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:10.051 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:10.051 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:10.051 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:10.051 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:10.051 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:10.051 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:10.051 INFO:teuthology.orchestra.run.smithi072.stdout: "identifier": { 2024-03-22T01:47:10.051 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "identifier", 2024-03-22T01:47:10.051 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:10.051 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:10.051 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:10.051 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:10.051 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:10.051 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:10.052 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:10.052 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:10.052 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:10.052 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:10.052 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:10.052 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:10.052 INFO:teuthology.orchestra.run.smithi072.stdout: "interval": { 2024-03-22T01:47:10.052 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "interval", 2024-03-22T01:47:10.052 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "secs", 2024-03-22T01:47:10.052 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:10.052 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:10.052 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "60", 2024-03-22T01:47:10.052 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:10.052 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:10.052 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:10.052 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:10.052 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:10.053 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:10.053 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:10.053 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:10.053 INFO:teuthology.orchestra.run.smithi072.stdout: "log_level": { 2024-03-22T01:47:10.053 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_level", 2024-03-22T01:47:10.053 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:10.053 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:10.053 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:10.053 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:10.053 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:10.053 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:10.053 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:10.053 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:10.053 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:10.054 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:10.054 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:10.054 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:10.054 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:10.054 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:10.054 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:10.054 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:10.054 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:10.054 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:10.054 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:10.054 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster": { 2024-03-22T01:47:10.054 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster", 2024-03-22T01:47:10.055 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:10.055 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:10.055 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:10.055 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:10.055 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:10.055 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:10.055 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:10.055 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:10.055 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:10.055 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:10.055 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:10.055 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:10.055 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_cluster_level": { 2024-03-22T01:47:10.055 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_cluster_level", 2024-03-22T01:47:10.056 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:10.056 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:10.056 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:10.056 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "info", 2024-03-22T01:47:10.056 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:10.056 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:10.056 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [ 2024-03-22T01:47:10.056 INFO:teuthology.orchestra.run.smithi072.stdout: "", 2024-03-22T01:47:10.056 INFO:teuthology.orchestra.run.smithi072.stdout: "critical", 2024-03-22T01:47:10.056 INFO:teuthology.orchestra.run.smithi072.stdout: "debug", 2024-03-22T01:47:10.056 INFO:teuthology.orchestra.run.smithi072.stdout: "error", 2024-03-22T01:47:10.056 INFO:teuthology.orchestra.run.smithi072.stdout: "info", 2024-03-22T01:47:10.056 INFO:teuthology.orchestra.run.smithi072.stdout: "warning" 2024-03-22T01:47:10.057 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:10.057 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:10.057 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:10.057 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:10.057 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:10.057 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:10.057 INFO:teuthology.orchestra.run.smithi072.stdout: "log_to_file": { 2024-03-22T01:47:10.057 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "log_to_file", 2024-03-22T01:47:10.057 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "bool", 2024-03-22T01:47:10.057 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:10.057 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 1, 2024-03-22T01:47:10.057 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "False", 2024-03-22T01:47:10.057 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:10.057 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:10.058 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:10.058 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:10.058 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:10.058 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:10.058 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:10.058 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:10.058 INFO:teuthology.orchestra.run.smithi072.stdout: "zabbix_host": { 2024-03-22T01:47:10.058 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "zabbix_host", 2024-03-22T01:47:10.058 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:10.058 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:10.058 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:10.058 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "", 2024-03-22T01:47:10.058 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:10.058 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:10.058 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:10.059 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:10.059 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:10.059 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:10.059 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:10.059 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:10.059 INFO:teuthology.orchestra.run.smithi072.stdout: "zabbix_port": { 2024-03-22T01:47:10.059 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "zabbix_port", 2024-03-22T01:47:10.059 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "int", 2024-03-22T01:47:10.059 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:10.059 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:10.059 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "10051", 2024-03-22T01:47:10.059 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:10.059 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:10.059 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:10.060 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:10.060 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:10.060 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:10.060 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:10.060 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:10.060 INFO:teuthology.orchestra.run.smithi072.stdout: "zabbix_sender": { 2024-03-22T01:47:10.060 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "zabbix_sender", 2024-03-22T01:47:10.060 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "str", 2024-03-22T01:47:10.060 INFO:teuthology.orchestra.run.smithi072.stdout: "level": "advanced", 2024-03-22T01:47:10.060 INFO:teuthology.orchestra.run.smithi072.stdout: "flags": 0, 2024-03-22T01:47:10.060 INFO:teuthology.orchestra.run.smithi072.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-22T01:47:10.060 INFO:teuthology.orchestra.run.smithi072.stdout: "min": "", 2024-03-22T01:47:10.060 INFO:teuthology.orchestra.run.smithi072.stdout: "max": "", 2024-03-22T01:47:10.060 INFO:teuthology.orchestra.run.smithi072.stdout: "enum_allowed": [], 2024-03-22T01:47:10.061 INFO:teuthology.orchestra.run.smithi072.stdout: "desc": "", 2024-03-22T01:47:10.061 INFO:teuthology.orchestra.run.smithi072.stdout: "long_desc": "", 2024-03-22T01:47:10.061 INFO:teuthology.orchestra.run.smithi072.stdout: "tags": [], 2024-03-22T01:47:10.061 INFO:teuthology.orchestra.run.smithi072.stdout: "see_also": [] 2024-03-22T01:47:10.061 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:10.061 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:10.061 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:10.061 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:10.061 INFO:teuthology.orchestra.run.smithi072.stdout: "services": { 2024-03-22T01:47:10.061 INFO:teuthology.orchestra.run.smithi072.stdout: "dashboard": "https://172.21.15.72:8443/" 2024-03-22T01:47:10.061 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:10.061 INFO:teuthology.orchestra.run.smithi072.stdout: "always_on_modules": { 2024-03-22T01:47:10.061 INFO:teuthology.orchestra.run.smithi072.stdout: "octopus": [ 2024-03-22T01:47:10.061 INFO:teuthology.orchestra.run.smithi072.stdout: "balancer", 2024-03-22T01:47:10.061 INFO:teuthology.orchestra.run.smithi072.stdout: "crash", 2024-03-22T01:47:10.062 INFO:teuthology.orchestra.run.smithi072.stdout: "devicehealth", 2024-03-22T01:47:10.062 INFO:teuthology.orchestra.run.smithi072.stdout: "orchestrator", 2024-03-22T01:47:10.062 INFO:teuthology.orchestra.run.smithi072.stdout: "pg_autoscaler", 2024-03-22T01:47:10.062 INFO:teuthology.orchestra.run.smithi072.stdout: "progress", 2024-03-22T01:47:10.062 INFO:teuthology.orchestra.run.smithi072.stdout: "rbd_support", 2024-03-22T01:47:10.062 INFO:teuthology.orchestra.run.smithi072.stdout: "status", 2024-03-22T01:47:10.062 INFO:teuthology.orchestra.run.smithi072.stdout: "telemetry", 2024-03-22T01:47:10.062 INFO:teuthology.orchestra.run.smithi072.stdout: "volumes" 2024-03-22T01:47:10.062 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:10.062 INFO:teuthology.orchestra.run.smithi072.stdout: "pacific": [ 2024-03-22T01:47:10.062 INFO:teuthology.orchestra.run.smithi072.stdout: "balancer", 2024-03-22T01:47:10.062 INFO:teuthology.orchestra.run.smithi072.stdout: "crash", 2024-03-22T01:47:10.062 INFO:teuthology.orchestra.run.smithi072.stdout: "devicehealth", 2024-03-22T01:47:10.062 INFO:teuthology.orchestra.run.smithi072.stdout: "orchestrator", 2024-03-22T01:47:10.062 INFO:teuthology.orchestra.run.smithi072.stdout: "pg_autoscaler", 2024-03-22T01:47:10.063 INFO:teuthology.orchestra.run.smithi072.stdout: "progress", 2024-03-22T01:47:10.063 INFO:teuthology.orchestra.run.smithi072.stdout: "rbd_support", 2024-03-22T01:47:10.063 INFO:teuthology.orchestra.run.smithi072.stdout: "status", 2024-03-22T01:47:10.063 INFO:teuthology.orchestra.run.smithi072.stdout: "telemetry", 2024-03-22T01:47:10.063 INFO:teuthology.orchestra.run.smithi072.stdout: "volumes" 2024-03-22T01:47:10.063 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:10.063 INFO:teuthology.orchestra.run.smithi072.stdout: "quincy": [ 2024-03-22T01:47:10.063 INFO:teuthology.orchestra.run.smithi072.stdout: "balancer", 2024-03-22T01:47:10.063 INFO:teuthology.orchestra.run.smithi072.stdout: "crash", 2024-03-22T01:47:10.063 INFO:teuthology.orchestra.run.smithi072.stdout: "devicehealth", 2024-03-22T01:47:10.063 INFO:teuthology.orchestra.run.smithi072.stdout: "orchestrator", 2024-03-22T01:47:10.063 INFO:teuthology.orchestra.run.smithi072.stdout: "pg_autoscaler", 2024-03-22T01:47:10.063 INFO:teuthology.orchestra.run.smithi072.stdout: "progress", 2024-03-22T01:47:10.063 INFO:teuthology.orchestra.run.smithi072.stdout: "rbd_support", 2024-03-22T01:47:10.063 INFO:teuthology.orchestra.run.smithi072.stdout: "status", 2024-03-22T01:47:10.064 INFO:teuthology.orchestra.run.smithi072.stdout: "telemetry", 2024-03-22T01:47:10.064 INFO:teuthology.orchestra.run.smithi072.stdout: "volumes" 2024-03-22T01:47:10.064 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:10.064 INFO:teuthology.orchestra.run.smithi072.stdout: "reef": [ 2024-03-22T01:47:10.064 INFO:teuthology.orchestra.run.smithi072.stdout: "balancer", 2024-03-22T01:47:10.064 INFO:teuthology.orchestra.run.smithi072.stdout: "crash", 2024-03-22T01:47:10.064 INFO:teuthology.orchestra.run.smithi072.stdout: "devicehealth", 2024-03-22T01:47:10.064 INFO:teuthology.orchestra.run.smithi072.stdout: "orchestrator", 2024-03-22T01:47:10.064 INFO:teuthology.orchestra.run.smithi072.stdout: "pg_autoscaler", 2024-03-22T01:47:10.064 INFO:teuthology.orchestra.run.smithi072.stdout: "progress", 2024-03-22T01:47:10.064 INFO:teuthology.orchestra.run.smithi072.stdout: "rbd_support", 2024-03-22T01:47:10.064 INFO:teuthology.orchestra.run.smithi072.stdout: "status", 2024-03-22T01:47:10.064 INFO:teuthology.orchestra.run.smithi072.stdout: "telemetry", 2024-03-22T01:47:10.064 INFO:teuthology.orchestra.run.smithi072.stdout: "volumes" 2024-03-22T01:47:10.064 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-22T01:47:10.064 INFO:teuthology.orchestra.run.smithi072.stdout: "squid": [ 2024-03-22T01:47:10.065 INFO:teuthology.orchestra.run.smithi072.stdout: "balancer", 2024-03-22T01:47:10.065 INFO:teuthology.orchestra.run.smithi072.stdout: "crash", 2024-03-22T01:47:10.065 INFO:teuthology.orchestra.run.smithi072.stdout: "devicehealth", 2024-03-22T01:47:10.065 INFO:teuthology.orchestra.run.smithi072.stdout: "orchestrator", 2024-03-22T01:47:10.065 INFO:teuthology.orchestra.run.smithi072.stdout: "pg_autoscaler", 2024-03-22T01:47:10.065 INFO:teuthology.orchestra.run.smithi072.stdout: "progress", 2024-03-22T01:47:10.065 INFO:teuthology.orchestra.run.smithi072.stdout: "rbd_support", 2024-03-22T01:47:10.065 INFO:teuthology.orchestra.run.smithi072.stdout: "status", 2024-03-22T01:47:10.065 INFO:teuthology.orchestra.run.smithi072.stdout: "telemetry", 2024-03-22T01:47:10.065 INFO:teuthology.orchestra.run.smithi072.stdout: "volumes" 2024-03-22T01:47:10.065 INFO:teuthology.orchestra.run.smithi072.stdout: ] 2024-03-22T01:47:10.065 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:10.065 INFO:teuthology.orchestra.run.smithi072.stdout: "last_failure_osd_epoch": 96, 2024-03-22T01:47:10.065 INFO:teuthology.orchestra.run.smithi072.stdout: "active_clients": [ 2024-03-22T01:47:10.066 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:10.066 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "devicehealth", 2024-03-22T01:47:10.066 INFO:teuthology.orchestra.run.smithi072.stdout: "addrvec": [ 2024-03-22T01:47:10.066 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:10.066 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "v2", 2024-03-22T01:47:10.066 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": "172.21.15.72:0", 2024-03-22T01:47:10.066 INFO:teuthology.orchestra.run.smithi072.stdout: "nonce": 2843199028 2024-03-22T01:47:10.066 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:10.066 INFO:teuthology.orchestra.run.smithi072.stdout: ] 2024-03-22T01:47:10.066 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:10.066 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:10.066 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "libcephsqlite", 2024-03-22T01:47:10.066 INFO:teuthology.orchestra.run.smithi072.stdout: "addrvec": [ 2024-03-22T01:47:10.066 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:10.067 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "v2", 2024-03-22T01:47:10.067 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": "172.21.15.72:0", 2024-03-22T01:47:10.067 INFO:teuthology.orchestra.run.smithi072.stdout: "nonce": 4096411805 2024-03-22T01:47:10.067 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:10.067 INFO:teuthology.orchestra.run.smithi072.stdout: ] 2024-03-22T01:47:10.067 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:10.067 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:10.067 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "rbd_support", 2024-03-22T01:47:10.067 INFO:teuthology.orchestra.run.smithi072.stdout: "addrvec": [ 2024-03-22T01:47:10.067 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:10.067 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "v2", 2024-03-22T01:47:10.067 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": "172.21.15.72:0", 2024-03-22T01:47:10.067 INFO:teuthology.orchestra.run.smithi072.stdout: "nonce": 3139418193 2024-03-22T01:47:10.067 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:10.067 INFO:teuthology.orchestra.run.smithi072.stdout: ] 2024-03-22T01:47:10.067 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:10.068 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:10.068 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "snap_schedule", 2024-03-22T01:47:10.068 INFO:teuthology.orchestra.run.smithi072.stdout: "addrvec": [ 2024-03-22T01:47:10.068 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:10.068 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "v2", 2024-03-22T01:47:10.068 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": "172.21.15.72:0", 2024-03-22T01:47:10.068 INFO:teuthology.orchestra.run.smithi072.stdout: "nonce": 1181479797 2024-03-22T01:47:10.068 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:10.068 INFO:teuthology.orchestra.run.smithi072.stdout: ] 2024-03-22T01:47:10.068 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:10.068 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:10.068 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "snap_schedule", 2024-03-22T01:47:10.068 INFO:teuthology.orchestra.run.smithi072.stdout: "addrvec": [ 2024-03-22T01:47:10.068 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:10.068 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "v2", 2024-03-22T01:47:10.068 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": "172.21.15.72:0", 2024-03-22T01:47:10.069 INFO:teuthology.orchestra.run.smithi072.stdout: "nonce": 2255957642 2024-03-22T01:47:10.069 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:10.069 INFO:teuthology.orchestra.run.smithi072.stdout: ] 2024-03-22T01:47:10.069 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:10.069 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:10.069 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "volumes", 2024-03-22T01:47:10.069 INFO:teuthology.orchestra.run.smithi072.stdout: "addrvec": [ 2024-03-22T01:47:10.069 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-22T01:47:10.069 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "v2", 2024-03-22T01:47:10.069 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": "172.21.15.72:0", 2024-03-22T01:47:10.069 INFO:teuthology.orchestra.run.smithi072.stdout: "nonce": 3385033360 2024-03-22T01:47:10.069 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:10.069 INFO:teuthology.orchestra.run.smithi072.stdout: ] 2024-03-22T01:47:10.069 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:10.069 INFO:teuthology.orchestra.run.smithi072.stdout: ] 2024-03-22T01:47:10.069 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-22T01:47:10.071 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:09 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/1991817475' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T01:47:10.072 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:09 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/1227938334' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-22T01:47:10.072 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:09 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/1991817475' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T01:47:10.072 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:09 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/1227938334' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-22T01:47:10.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.777+0000 7fd4ae7a0640 1 -- 172.21.15.72:0/1227938334 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fd4940a3100 msgr2=0x7fd4940a55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:10.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.777+0000 7fd4ae7a0640 1 --2- 172.21.15.72:0/1227938334 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fd4940a3100 0x7fd4940a55c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd4a4005fd0 tx=0x7fd4a4009680 comp rx=0 tx=0).stop 2024-03-22T01:47:10.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.777+0000 7fd4ae7a0640 1 -- 172.21.15.72:0/1227938334 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd4a810c020 msgr2=0x7fd4a819cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:10.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.777+0000 7fd4ae7a0640 1 --2- 172.21.15.72:0/1227938334 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd4a810c020 0x7fd4a819cbb0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fd4a0005940 tx=0x7fd4a002cc40 comp rx=0 tx=0).stop 2024-03-22T01:47:10.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.777+0000 7fd4ae7a0640 1 -- 172.21.15.72:0/1227938334 shutdown_connections 2024-03-22T01:47:10.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.777+0000 7fd4ae7a0640 1 --2- 172.21.15.72:0/1227938334 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fd4940a3100 0x7fd4940a55c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:10.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.777+0000 7fd4ae7a0640 1 --2- 172.21.15.72:0/1227938334 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd4a810c020 0x7fd4a819cbb0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:10.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.777+0000 7fd4ae7a0640 1 --2- 172.21.15.72:0/1227938334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4a8103570 0x7fd4a8198880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:10.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.777+0000 7fd4ae7a0640 1 --2- 172.21.15.72:0/1227938334 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd4a8102c50 0x7fd4a8198340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:10.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.777+0000 7fd4ae7a0640 1 -- 172.21.15.72:0/1227938334 >> 172.21.15.72:0/1227938334 conn(0x7fd4a80fc3f0 msgr2=0x7fd4a80fd100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:47:10.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.777+0000 7fd4ae7a0640 1 -- 172.21.15.72:0/1227938334 shutdown_connections 2024-03-22T01:47:10.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:08.777+0000 7fd4ae7a0640 1 -- 172.21.15.72:0/1227938334 wait complete. 2024-03-22T01:47:10.080 DEBUG:tasks.check_counter:Getting stats from a 2024-03-22T01:47:10.080 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph daemon mds.a perf dump 2024-03-22T01:47:10.081 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:09 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/1991817475' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T01:47:10.081 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:09 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/1227938334' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-22T01:47:10.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:10 smithi072 ceph-mon[32414]: pgmap v318: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 19 op/s 2024-03-22T01:47:10.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:10 smithi089 ceph-mon[35770]: pgmap v318: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 19 op/s 2024-03-22T01:47:10.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:10 smithi098 ceph-mon[36468]: pgmap v318: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 19 op/s 2024-03-22T01:47:12.522 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:47:12.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:12 smithi089 ceph-mon[35770]: pgmap v319: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 20 op/s 2024-03-22T01:47:12.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:12 smithi072 ceph-mon[32414]: pgmap v319: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 20 op/s 2024-03-22T01:47:12.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:12 smithi098 ceph-mon[36468]: pgmap v319: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 20 op/s 2024-03-22T01:47:14.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:14 smithi089 ceph-mon[35770]: pgmap v320: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 20 op/s 2024-03-22T01:47:14.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:14 smithi072 ceph-mon[32414]: pgmap v320: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 20 op/s 2024-03-22T01:47:14.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:14 smithi098 ceph-mon[36468]: pgmap v320: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 20 op/s 2024-03-22T01:47:16.463 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:16 smithi072 ceph-mon[32414]: pgmap v321: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 255 B/s wr, 2 op/s 2024-03-22T01:47:16.609 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:16 smithi089 ceph-mon[35770]: pgmap v321: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 255 B/s wr, 2 op/s 2024-03-22T01:47:16.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:16 smithi098 ceph-mon[36468]: pgmap v321: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 255 B/s wr, 2 op/s 2024-03-22T01:47:18.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:18 smithi089 ceph-mon[35770]: pgmap v322: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-22T01:47:18.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:18 smithi072 ceph-mon[32414]: pgmap v322: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-22T01:47:18.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:18 smithi098 ceph-mon[36468]: pgmap v322: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-22T01:47:19.099 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-22T01:47:19.099 INFO:teuthology.orchestra.run.smithi072.stdout: "AsyncMessenger::Worker-0": { 2024-03-22T01:47:19.099 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_recv_messages": 228, 2024-03-22T01:47:19.099 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_send_messages": 216, 2024-03-22T01:47:19.099 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_recv_bytes": 1446864, 2024-03-22T01:47:19.100 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_send_bytes": 131712, 2024-03-22T01:47:19.100 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_created_connections": 4, 2024-03-22T01:47:19.100 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_active_connections": 0, 2024-03-22T01:47:19.100 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_running_total_time": 0.042336921, 2024-03-22T01:47:19.100 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_running_send_time": 0.013928674, 2024-03-22T01:47:19.100 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_running_recv_time": 0.031233682, 2024-03-22T01:47:19.100 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_running_fast_dispatch_time": 0.004189460, 2024-03-22T01:47:19.100 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T01:47:19.100 INFO:teuthology.orchestra.run.smithi072.stdout: "avgcount": 216, 2024-03-22T01:47:19.100 INFO:teuthology.orchestra.run.smithi072.stdout: "sum": 0.015026622, 2024-03-22T01:47:19.100 INFO:teuthology.orchestra.run.smithi072.stdout: "avgtime": 0.000069567 2024-03-22T01:47:19.100 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:19.100 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_handle_ack_lat": { 2024-03-22T01:47:19.100 INFO:teuthology.orchestra.run.smithi072.stdout: "avgcount": 0, 2024-03-22T01:47:19.101 INFO:teuthology.orchestra.run.smithi072.stdout: "sum": 0.000000000, 2024-03-22T01:47:19.101 INFO:teuthology.orchestra.run.smithi072.stdout: "avgtime": 0.000000000 2024-03-22T01:47:19.101 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:19.101 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_recv_encrypted_bytes": 1446864, 2024-03-22T01:47:19.101 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_send_encrypted_bytes": 131712 2024-03-22T01:47:19.101 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:19.101 INFO:teuthology.orchestra.run.smithi072.stdout: "AsyncMessenger::Worker-1": { 2024-03-22T01:47:19.101 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_recv_messages": 2, 2024-03-22T01:47:19.101 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_send_messages": 162, 2024-03-22T01:47:19.101 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_recv_bytes": 320, 2024-03-22T01:47:19.101 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_send_bytes": 184112, 2024-03-22T01:47:19.101 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_created_connections": 5, 2024-03-22T01:47:19.101 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_active_connections": 3, 2024-03-22T01:47:19.101 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_running_total_time": 0.012205235, 2024-03-22T01:47:19.101 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_running_send_time": 0.008032835, 2024-03-22T01:47:19.101 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_running_recv_time": 0.001784611, 2024-03-22T01:47:19.102 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-22T01:47:19.102 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T01:47:19.102 INFO:teuthology.orchestra.run.smithi072.stdout: "avgcount": 159, 2024-03-22T01:47:19.102 INFO:teuthology.orchestra.run.smithi072.stdout: "sum": 0.011767210, 2024-03-22T01:47:19.102 INFO:teuthology.orchestra.run.smithi072.stdout: "avgtime": 0.000074007 2024-03-22T01:47:19.102 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:19.102 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_handle_ack_lat": { 2024-03-22T01:47:19.102 INFO:teuthology.orchestra.run.smithi072.stdout: "avgcount": 0, 2024-03-22T01:47:19.102 INFO:teuthology.orchestra.run.smithi072.stdout: "sum": 0.000000000, 2024-03-22T01:47:19.102 INFO:teuthology.orchestra.run.smithi072.stdout: "avgtime": 0.000000000 2024-03-22T01:47:19.102 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:19.102 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-22T01:47:19.102 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_send_encrypted_bytes": 184112 2024-03-22T01:47:19.103 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:19.103 INFO:teuthology.orchestra.run.smithi072.stdout: "AsyncMessenger::Worker-2": { 2024-03-22T01:47:19.103 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_recv_messages": 0, 2024-03-22T01:47:19.103 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_send_messages": 2, 2024-03-22T01:47:19.103 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_recv_bytes": 0, 2024-03-22T01:47:19.103 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_send_bytes": 0, 2024-03-22T01:47:19.103 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_created_connections": 2, 2024-03-22T01:47:19.103 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_active_connections": 1, 2024-03-22T01:47:19.103 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_running_total_time": 0.000362099, 2024-03-22T01:47:19.103 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_running_send_time": 0.000000000, 2024-03-22T01:47:19.103 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_running_recv_time": 0.000305109, 2024-03-22T01:47:19.103 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-22T01:47:19.103 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T01:47:19.103 INFO:teuthology.orchestra.run.smithi072.stdout: "avgcount": 0, 2024-03-22T01:47:19.103 INFO:teuthology.orchestra.run.smithi072.stdout: "sum": 0.000000000, 2024-03-22T01:47:19.104 INFO:teuthology.orchestra.run.smithi072.stdout: "avgtime": 0.000000000 2024-03-22T01:47:19.104 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:19.104 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_handle_ack_lat": { 2024-03-22T01:47:19.104 INFO:teuthology.orchestra.run.smithi072.stdout: "avgcount": 0, 2024-03-22T01:47:19.104 INFO:teuthology.orchestra.run.smithi072.stdout: "sum": 0.000000000, 2024-03-22T01:47:19.104 INFO:teuthology.orchestra.run.smithi072.stdout: "avgtime": 0.000000000 2024-03-22T01:47:19.104 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:19.104 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-22T01:47:19.104 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_send_encrypted_bytes": 0 2024-03-22T01:47:19.104 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:19.104 INFO:teuthology.orchestra.run.smithi072.stdout: "cct": { 2024-03-22T01:47:19.104 INFO:teuthology.orchestra.run.smithi072.stdout: "total_workers": 0, 2024-03-22T01:47:19.104 INFO:teuthology.orchestra.run.smithi072.stdout: "unhealthy_workers": 0 2024-03-22T01:47:19.104 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:19.104 INFO:teuthology.orchestra.run.smithi072.stdout: "mempool": { 2024-03-22T01:47:19.104 INFO:teuthology.orchestra.run.smithi072.stdout: "bloom_filter_bytes": 0, 2024-03-22T01:47:19.105 INFO:teuthology.orchestra.run.smithi072.stdout: "bloom_filter_items": 0, 2024-03-22T01:47:19.105 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_alloc_bytes": 0, 2024-03-22T01:47:19.105 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_alloc_items": 0, 2024-03-22T01:47:19.105 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_cache_data_bytes": 0, 2024-03-22T01:47:19.105 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_cache_data_items": 0, 2024-03-22T01:47:19.105 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-22T01:47:19.105 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_cache_onode_items": 0, 2024-03-22T01:47:19.105 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-22T01:47:19.105 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_cache_meta_items": 0, 2024-03-22T01:47:19.105 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_cache_other_bytes": 0, 2024-03-22T01:47:19.105 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_cache_other_items": 0, 2024-03-22T01:47:19.105 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-22T01:47:19.105 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_cache_buffer_items": 0, 2024-03-22T01:47:19.105 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_extent_bytes": 0, 2024-03-22T01:47:19.105 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_extent_items": 0, 2024-03-22T01:47:19.106 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_blob_bytes": 0, 2024-03-22T01:47:19.106 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_blob_items": 0, 2024-03-22T01:47:19.106 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-22T01:47:19.106 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_shared_blob_items": 0, 2024-03-22T01:47:19.106 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-22T01:47:19.106 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_inline_bl_items": 0, 2024-03-22T01:47:19.106 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_fsck_bytes": 0, 2024-03-22T01:47:19.106 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_fsck_items": 0, 2024-03-22T01:47:19.106 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_txc_bytes": 0, 2024-03-22T01:47:19.106 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_txc_items": 0, 2024-03-22T01:47:19.106 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-22T01:47:19.106 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_writing_deferred_items": 0, 2024-03-22T01:47:19.106 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_writing_bytes": 0, 2024-03-22T01:47:19.107 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_writing_items": 0, 2024-03-22T01:47:19.107 INFO:teuthology.orchestra.run.smithi072.stdout: "bluefs_bytes": 0, 2024-03-22T01:47:19.107 INFO:teuthology.orchestra.run.smithi072.stdout: "bluefs_items": 0, 2024-03-22T01:47:19.107 INFO:teuthology.orchestra.run.smithi072.stdout: "bluefs_file_reader_bytes": 0, 2024-03-22T01:47:19.107 INFO:teuthology.orchestra.run.smithi072.stdout: "bluefs_file_reader_items": 0, 2024-03-22T01:47:19.107 INFO:teuthology.orchestra.run.smithi072.stdout: "bluefs_file_writer_bytes": 0, 2024-03-22T01:47:19.107 INFO:teuthology.orchestra.run.smithi072.stdout: "bluefs_file_writer_items": 0, 2024-03-22T01:47:19.107 INFO:teuthology.orchestra.run.smithi072.stdout: "buffer_anon_bytes": 149017, 2024-03-22T01:47:19.107 INFO:teuthology.orchestra.run.smithi072.stdout: "buffer_anon_items": 31, 2024-03-22T01:47:19.107 INFO:teuthology.orchestra.run.smithi072.stdout: "buffer_meta_bytes": 0, 2024-03-22T01:47:19.107 INFO:teuthology.orchestra.run.smithi072.stdout: "buffer_meta_items": 0, 2024-03-22T01:47:19.107 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_bytes": 0, 2024-03-22T01:47:19.107 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_items": 0, 2024-03-22T01:47:19.107 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_mapbl_bytes": 0, 2024-03-22T01:47:19.107 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_mapbl_items": 0, 2024-03-22T01:47:19.108 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_pglog_bytes": 0, 2024-03-22T01:47:19.108 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_pglog_items": 0, 2024-03-22T01:47:19.108 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_bytes": 0, 2024-03-22T01:47:19.108 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_items": 0, 2024-03-22T01:47:19.108 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_mapping_bytes": 0, 2024-03-22T01:47:19.108 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_mapping_items": 0, 2024-03-22T01:47:19.108 INFO:teuthology.orchestra.run.smithi072.stdout: "pgmap_bytes": 0, 2024-03-22T01:47:19.108 INFO:teuthology.orchestra.run.smithi072.stdout: "pgmap_items": 0, 2024-03-22T01:47:19.108 INFO:teuthology.orchestra.run.smithi072.stdout: "mds_co_bytes": 928, 2024-03-22T01:47:19.108 INFO:teuthology.orchestra.run.smithi072.stdout: "mds_co_items": 2, 2024-03-22T01:47:19.108 INFO:teuthology.orchestra.run.smithi072.stdout: "unittest_1_bytes": 0, 2024-03-22T01:47:19.108 INFO:teuthology.orchestra.run.smithi072.stdout: "unittest_1_items": 0, 2024-03-22T01:47:19.108 INFO:teuthology.orchestra.run.smithi072.stdout: "unittest_2_bytes": 0, 2024-03-22T01:47:19.108 INFO:teuthology.orchestra.run.smithi072.stdout: "unittest_2_items": 0 2024-03-22T01:47:19.108 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:19.108 INFO:teuthology.orchestra.run.smithi072.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-22T01:47:19.109 INFO:teuthology.orchestra.run.smithi072.stdout: "val": 0, 2024-03-22T01:47:19.109 INFO:teuthology.orchestra.run.smithi072.stdout: "max": 104857600, 2024-03-22T01:47:19.109 INFO:teuthology.orchestra.run.smithi072.stdout: "get_started": 0, 2024-03-22T01:47:19.109 INFO:teuthology.orchestra.run.smithi072.stdout: "get": 226, 2024-03-22T01:47:19.109 INFO:teuthology.orchestra.run.smithi072.stdout: "get_sum": 1412270, 2024-03-22T01:47:19.109 INFO:teuthology.orchestra.run.smithi072.stdout: "get_or_fail_fail": 0, 2024-03-22T01:47:19.109 INFO:teuthology.orchestra.run.smithi072.stdout: "get_or_fail_success": 226, 2024-03-22T01:47:19.109 INFO:teuthology.orchestra.run.smithi072.stdout: "take": 0, 2024-03-22T01:47:19.109 INFO:teuthology.orchestra.run.smithi072.stdout: "take_sum": 0, 2024-03-22T01:47:19.109 INFO:teuthology.orchestra.run.smithi072.stdout: "put": 226, 2024-03-22T01:47:19.109 INFO:teuthology.orchestra.run.smithi072.stdout: "put_sum": 1412270, 2024-03-22T01:47:19.109 INFO:teuthology.orchestra.run.smithi072.stdout: "wait": { 2024-03-22T01:47:19.109 INFO:teuthology.orchestra.run.smithi072.stdout: "avgcount": 0, 2024-03-22T01:47:19.109 INFO:teuthology.orchestra.run.smithi072.stdout: "sum": 0.000000000, 2024-03-22T01:47:19.109 INFO:teuthology.orchestra.run.smithi072.stdout: "avgtime": 0.000000000 2024-03-22T01:47:19.109 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:19.110 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:19.110 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-22T01:47:19.808 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.a 2024-03-22T01:47:19.808 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.a 2024-03-22T01:47:19.808 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.a 2024-03-22T01:47:19.809 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.a 2024-03-22T01:47:19.809 DEBUG:tasks.check_counter:Getting stats from d 2024-03-22T01:47:19.809 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph daemon mds.d perf dump 2024-03-22T01:47:20.064 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:47:20.455 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:20 smithi072 ceph-mon[32414]: pgmap v323: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 85 B/s wr, 1 op/s 2024-03-22T01:47:20.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:20 smithi089 ceph-mon[35770]: pgmap v323: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 85 B/s wr, 1 op/s 2024-03-22T01:47:20.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:20 smithi098 ceph-mon[36468]: pgmap v323: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 85 B/s wr, 1 op/s 2024-03-22T01:47:21.827 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-22T01:47:21.828 INFO:teuthology.orchestra.run.smithi072.stdout: "AsyncMessenger::Worker-0": { 2024-03-22T01:47:21.828 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_recv_messages": 0, 2024-03-22T01:47:21.828 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_send_messages": 2, 2024-03-22T01:47:21.828 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_recv_bytes": 0, 2024-03-22T01:47:21.828 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_send_bytes": 0, 2024-03-22T01:47:21.828 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_created_connections": 4, 2024-03-22T01:47:21.828 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_active_connections": 0, 2024-03-22T01:47:21.828 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_running_total_time": 0.001222444, 2024-03-22T01:47:21.828 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_running_send_time": 0.000000000, 2024-03-22T01:47:21.828 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_running_recv_time": 0.000353389, 2024-03-22T01:47:21.828 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-22T01:47:21.828 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T01:47:21.829 INFO:teuthology.orchestra.run.smithi072.stdout: "avgcount": 0, 2024-03-22T01:47:21.829 INFO:teuthology.orchestra.run.smithi072.stdout: "sum": 0.000000000, 2024-03-22T01:47:21.829 INFO:teuthology.orchestra.run.smithi072.stdout: "avgtime": 0.000000000 2024-03-22T01:47:21.829 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:21.829 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_handle_ack_lat": { 2024-03-22T01:47:21.829 INFO:teuthology.orchestra.run.smithi072.stdout: "avgcount": 0, 2024-03-22T01:47:21.829 INFO:teuthology.orchestra.run.smithi072.stdout: "sum": 0.000000000, 2024-03-22T01:47:21.829 INFO:teuthology.orchestra.run.smithi072.stdout: "avgtime": 0.000000000 2024-03-22T01:47:21.829 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:21.829 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-22T01:47:21.829 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_send_encrypted_bytes": 0 2024-03-22T01:47:21.829 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:21.829 INFO:teuthology.orchestra.run.smithi072.stdout: "AsyncMessenger::Worker-1": { 2024-03-22T01:47:21.829 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_recv_messages": 6, 2024-03-22T01:47:21.830 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_send_messages": 169, 2024-03-22T01:47:21.830 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_recv_bytes": 3312, 2024-03-22T01:47:21.830 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_send_bytes": 185440, 2024-03-22T01:47:21.830 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_created_connections": 5, 2024-03-22T01:47:21.830 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_active_connections": 3, 2024-03-22T01:47:21.830 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_running_total_time": 0.013151205, 2024-03-22T01:47:21.830 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_running_send_time": 0.008239071, 2024-03-22T01:47:21.830 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_running_recv_time": 0.002589438, 2024-03-22T01:47:21.830 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-22T01:47:21.830 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T01:47:21.830 INFO:teuthology.orchestra.run.smithi072.stdout: "avgcount": 167, 2024-03-22T01:47:21.830 INFO:teuthology.orchestra.run.smithi072.stdout: "sum": 0.021038088, 2024-03-22T01:47:21.830 INFO:teuthology.orchestra.run.smithi072.stdout: "avgtime": 0.000125976 2024-03-22T01:47:21.830 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:21.831 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_handle_ack_lat": { 2024-03-22T01:47:21.831 INFO:teuthology.orchestra.run.smithi072.stdout: "avgcount": 0, 2024-03-22T01:47:21.831 INFO:teuthology.orchestra.run.smithi072.stdout: "sum": 0.000000000, 2024-03-22T01:47:21.831 INFO:teuthology.orchestra.run.smithi072.stdout: "avgtime": 0.000000000 2024-03-22T01:47:21.831 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:21.831 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-03-22T01:47:21.831 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_send_encrypted_bytes": 185440 2024-03-22T01:47:21.831 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:21.831 INFO:teuthology.orchestra.run.smithi072.stdout: "AsyncMessenger::Worker-2": { 2024-03-22T01:47:21.831 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_recv_messages": 229, 2024-03-22T01:47:21.831 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_send_messages": 219, 2024-03-22T01:47:21.831 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_recv_bytes": 1445232, 2024-03-22T01:47:21.832 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_send_bytes": 134464, 2024-03-22T01:47:21.832 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_created_connections": 2, 2024-03-22T01:47:21.832 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_active_connections": 1, 2024-03-22T01:47:21.832 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_running_total_time": 0.041521378, 2024-03-22T01:47:21.832 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_running_send_time": 0.013518019, 2024-03-22T01:47:21.832 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_running_recv_time": 0.030286430, 2024-03-22T01:47:21.832 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_running_fast_dispatch_time": 0.005099256, 2024-03-22T01:47:21.832 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T01:47:21.832 INFO:teuthology.orchestra.run.smithi072.stdout: "avgcount": 218, 2024-03-22T01:47:21.832 INFO:teuthology.orchestra.run.smithi072.stdout: "sum": 0.011495798, 2024-03-22T01:47:21.832 INFO:teuthology.orchestra.run.smithi072.stdout: "avgtime": 0.000052733 2024-03-22T01:47:21.833 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:21.833 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_handle_ack_lat": { 2024-03-22T01:47:21.833 INFO:teuthology.orchestra.run.smithi072.stdout: "avgcount": 0, 2024-03-22T01:47:21.833 INFO:teuthology.orchestra.run.smithi072.stdout: "sum": 0.000000000, 2024-03-22T01:47:21.833 INFO:teuthology.orchestra.run.smithi072.stdout: "avgtime": 0.000000000 2024-03-22T01:47:21.833 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:21.833 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_recv_encrypted_bytes": 1445232, 2024-03-22T01:47:21.833 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_send_encrypted_bytes": 134464 2024-03-22T01:47:21.833 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:21.833 INFO:teuthology.orchestra.run.smithi072.stdout: "cct": { 2024-03-22T01:47:21.833 INFO:teuthology.orchestra.run.smithi072.stdout: "total_workers": 0, 2024-03-22T01:47:21.833 INFO:teuthology.orchestra.run.smithi072.stdout: "unhealthy_workers": 0 2024-03-22T01:47:21.833 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:21.833 INFO:teuthology.orchestra.run.smithi072.stdout: "mempool": { 2024-03-22T01:47:21.834 INFO:teuthology.orchestra.run.smithi072.stdout: "bloom_filter_bytes": 0, 2024-03-22T01:47:21.834 INFO:teuthology.orchestra.run.smithi072.stdout: "bloom_filter_items": 0, 2024-03-22T01:47:21.834 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_alloc_bytes": 0, 2024-03-22T01:47:21.834 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_alloc_items": 0, 2024-03-22T01:47:21.834 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_cache_data_bytes": 0, 2024-03-22T01:47:21.834 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_cache_data_items": 0, 2024-03-22T01:47:21.834 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-22T01:47:21.834 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_cache_onode_items": 0, 2024-03-22T01:47:21.834 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-22T01:47:21.837 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_cache_meta_items": 0, 2024-03-22T01:47:21.838 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_cache_other_bytes": 0, 2024-03-22T01:47:21.838 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_cache_other_items": 0, 2024-03-22T01:47:21.838 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-22T01:47:21.838 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_cache_buffer_items": 0, 2024-03-22T01:47:21.838 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_extent_bytes": 0, 2024-03-22T01:47:21.838 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_extent_items": 0, 2024-03-22T01:47:21.838 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_blob_bytes": 0, 2024-03-22T01:47:21.838 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_blob_items": 0, 2024-03-22T01:47:21.838 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-22T01:47:21.838 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_shared_blob_items": 0, 2024-03-22T01:47:21.838 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-22T01:47:21.838 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_inline_bl_items": 0, 2024-03-22T01:47:21.838 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_fsck_bytes": 0, 2024-03-22T01:47:21.838 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_fsck_items": 0, 2024-03-22T01:47:21.839 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_txc_bytes": 0, 2024-03-22T01:47:21.839 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_txc_items": 0, 2024-03-22T01:47:21.839 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-22T01:47:21.839 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_writing_deferred_items": 0, 2024-03-22T01:47:21.839 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_writing_bytes": 0, 2024-03-22T01:47:21.839 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_writing_items": 0, 2024-03-22T01:47:21.839 INFO:teuthology.orchestra.run.smithi072.stdout: "bluefs_bytes": 0, 2024-03-22T01:47:21.839 INFO:teuthology.orchestra.run.smithi072.stdout: "bluefs_items": 0, 2024-03-22T01:47:21.839 INFO:teuthology.orchestra.run.smithi072.stdout: "bluefs_file_reader_bytes": 0, 2024-03-22T01:47:21.839 INFO:teuthology.orchestra.run.smithi072.stdout: "bluefs_file_reader_items": 0, 2024-03-22T01:47:21.839 INFO:teuthology.orchestra.run.smithi072.stdout: "bluefs_file_writer_bytes": 0, 2024-03-22T01:47:21.839 INFO:teuthology.orchestra.run.smithi072.stdout: "bluefs_file_writer_items": 0, 2024-03-22T01:47:21.839 INFO:teuthology.orchestra.run.smithi072.stdout: "buffer_anon_bytes": 149057, 2024-03-22T01:47:21.839 INFO:teuthology.orchestra.run.smithi072.stdout: "buffer_anon_items": 32, 2024-03-22T01:47:21.840 INFO:teuthology.orchestra.run.smithi072.stdout: "buffer_meta_bytes": 0, 2024-03-22T01:47:21.840 INFO:teuthology.orchestra.run.smithi072.stdout: "buffer_meta_items": 0, 2024-03-22T01:47:21.840 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_bytes": 0, 2024-03-22T01:47:21.840 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_items": 0, 2024-03-22T01:47:21.840 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_mapbl_bytes": 0, 2024-03-22T01:47:21.840 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_mapbl_items": 0, 2024-03-22T01:47:21.840 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_pglog_bytes": 0, 2024-03-22T01:47:21.840 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_pglog_items": 0, 2024-03-22T01:47:21.840 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_bytes": 0, 2024-03-22T01:47:21.840 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_items": 0, 2024-03-22T01:47:21.840 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_mapping_bytes": 0, 2024-03-22T01:47:21.840 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_mapping_items": 0, 2024-03-22T01:47:21.840 INFO:teuthology.orchestra.run.smithi072.stdout: "pgmap_bytes": 0, 2024-03-22T01:47:21.840 INFO:teuthology.orchestra.run.smithi072.stdout: "pgmap_items": 0, 2024-03-22T01:47:21.840 INFO:teuthology.orchestra.run.smithi072.stdout: "mds_co_bytes": 928, 2024-03-22T01:47:21.841 INFO:teuthology.orchestra.run.smithi072.stdout: "mds_co_items": 2, 2024-03-22T01:47:21.841 INFO:teuthology.orchestra.run.smithi072.stdout: "unittest_1_bytes": 0, 2024-03-22T01:47:21.841 INFO:teuthology.orchestra.run.smithi072.stdout: "unittest_1_items": 0, 2024-03-22T01:47:21.841 INFO:teuthology.orchestra.run.smithi072.stdout: "unittest_2_bytes": 0, 2024-03-22T01:47:21.841 INFO:teuthology.orchestra.run.smithi072.stdout: "unittest_2_items": 0 2024-03-22T01:47:21.841 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:21.841 INFO:teuthology.orchestra.run.smithi072.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-22T01:47:21.841 INFO:teuthology.orchestra.run.smithi072.stdout: "val": 0, 2024-03-22T01:47:21.841 INFO:teuthology.orchestra.run.smithi072.stdout: "max": 104857600, 2024-03-22T01:47:21.841 INFO:teuthology.orchestra.run.smithi072.stdout: "get_started": 0, 2024-03-22T01:47:21.841 INFO:teuthology.orchestra.run.smithi072.stdout: "get": 231, 2024-03-22T01:47:21.841 INFO:teuthology.orchestra.run.smithi072.stdout: "get_sum": 1412920, 2024-03-22T01:47:21.841 INFO:teuthology.orchestra.run.smithi072.stdout: "get_or_fail_fail": 0, 2024-03-22T01:47:21.841 INFO:teuthology.orchestra.run.smithi072.stdout: "get_or_fail_success": 231, 2024-03-22T01:47:21.842 INFO:teuthology.orchestra.run.smithi072.stdout: "take": 0, 2024-03-22T01:47:21.842 INFO:teuthology.orchestra.run.smithi072.stdout: "take_sum": 0, 2024-03-22T01:47:21.842 INFO:teuthology.orchestra.run.smithi072.stdout: "put": 231, 2024-03-22T01:47:21.843 INFO:teuthology.orchestra.run.smithi072.stdout: "put_sum": 1412920, 2024-03-22T01:47:21.843 INFO:teuthology.orchestra.run.smithi072.stdout: "wait": { 2024-03-22T01:47:21.843 INFO:teuthology.orchestra.run.smithi072.stdout: "avgcount": 0, 2024-03-22T01:47:21.843 INFO:teuthology.orchestra.run.smithi072.stdout: "sum": 0.000000000, 2024-03-22T01:47:21.843 INFO:teuthology.orchestra.run.smithi072.stdout: "avgtime": 0.000000000 2024-03-22T01:47:21.843 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:21.843 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:21.843 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-22T01:47:22.498 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:22 smithi072 ceph-mon[32414]: pgmap v324: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 85 B/s wr, 2 op/s 2024-03-22T01:47:22.524 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-03-22T01:47:22.524 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-03-22T01:47:22.524 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-03-22T01:47:22.524 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.d 2024-03-22T01:47:22.524 DEBUG:tasks.check_counter:Getting stats from g 2024-03-22T01:47:22.525 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph daemon mds.g perf dump 2024-03-22T01:47:22.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:22 smithi089 ceph-mon[35770]: pgmap v324: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 85 B/s wr, 2 op/s 2024-03-22T01:47:22.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:22 smithi098 ceph-mon[36468]: pgmap v324: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 85 B/s wr, 2 op/s 2024-03-22T01:47:22.764 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:47:24.477 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:24 smithi072 ceph-mon[32414]: pgmap v325: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 85 B/s wr, 2 op/s 2024-03-22T01:47:24.477 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-22T01:47:24.478 INFO:teuthology.orchestra.run.smithi072.stdout: "AsyncMessenger::Worker-0": { 2024-03-22T01:47:24.478 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_recv_messages": 0, 2024-03-22T01:47:24.478 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_send_messages": 2, 2024-03-22T01:47:24.478 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_recv_bytes": 0, 2024-03-22T01:47:24.478 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_send_bytes": 0, 2024-03-22T01:47:24.478 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_created_connections": 4, 2024-03-22T01:47:24.478 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_active_connections": 0, 2024-03-22T01:47:24.478 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_running_total_time": 0.001182374, 2024-03-22T01:47:24.479 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_running_send_time": 0.000000000, 2024-03-22T01:47:24.479 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_running_recv_time": 0.000193674, 2024-03-22T01:47:24.479 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-22T01:47:24.479 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T01:47:24.479 INFO:teuthology.orchestra.run.smithi072.stdout: "avgcount": 0, 2024-03-22T01:47:24.479 INFO:teuthology.orchestra.run.smithi072.stdout: "sum": 0.000000000, 2024-03-22T01:47:24.479 INFO:teuthology.orchestra.run.smithi072.stdout: "avgtime": 0.000000000 2024-03-22T01:47:24.479 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:24.479 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_handle_ack_lat": { 2024-03-22T01:47:24.479 INFO:teuthology.orchestra.run.smithi072.stdout: "avgcount": 0, 2024-03-22T01:47:24.479 INFO:teuthology.orchestra.run.smithi072.stdout: "sum": 0.000000000, 2024-03-22T01:47:24.479 INFO:teuthology.orchestra.run.smithi072.stdout: "avgtime": 0.000000000 2024-03-22T01:47:24.479 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:24.479 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-22T01:47:24.480 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_send_encrypted_bytes": 0 2024-03-22T01:47:24.480 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:24.480 INFO:teuthology.orchestra.run.smithi072.stdout: "AsyncMessenger::Worker-1": { 2024-03-22T01:47:24.480 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_recv_messages": 230, 2024-03-22T01:47:24.480 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_send_messages": 218, 2024-03-22T01:47:24.480 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_recv_bytes": 1447408, 2024-03-22T01:47:24.480 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_send_bytes": 132992, 2024-03-22T01:47:24.480 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_created_connections": 2, 2024-03-22T01:47:24.480 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_active_connections": 1, 2024-03-22T01:47:24.480 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_running_total_time": 0.040680720, 2024-03-22T01:47:24.480 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_running_send_time": 0.013672452, 2024-03-22T01:47:24.480 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_running_recv_time": 0.030761842, 2024-03-22T01:47:24.480 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_running_fast_dispatch_time": 0.004226039, 2024-03-22T01:47:24.481 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T01:47:24.481 INFO:teuthology.orchestra.run.smithi072.stdout: "avgcount": 218, 2024-03-22T01:47:24.481 INFO:teuthology.orchestra.run.smithi072.stdout: "sum": 0.012750196, 2024-03-22T01:47:24.481 INFO:teuthology.orchestra.run.smithi072.stdout: "avgtime": 0.000058487 2024-03-22T01:47:24.481 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:24.481 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_handle_ack_lat": { 2024-03-22T01:47:24.481 INFO:teuthology.orchestra.run.smithi072.stdout: "avgcount": 0, 2024-03-22T01:47:24.481 INFO:teuthology.orchestra.run.smithi072.stdout: "sum": 0.000000000, 2024-03-22T01:47:24.481 INFO:teuthology.orchestra.run.smithi072.stdout: "avgtime": 0.000000000 2024-03-22T01:47:24.482 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:24.482 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_recv_encrypted_bytes": 1447408, 2024-03-22T01:47:24.482 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_send_encrypted_bytes": 132992 2024-03-22T01:47:24.482 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:24.482 INFO:teuthology.orchestra.run.smithi072.stdout: "AsyncMessenger::Worker-2": { 2024-03-22T01:47:24.482 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_recv_messages": 2, 2024-03-22T01:47:24.482 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_send_messages": 165, 2024-03-22T01:47:24.482 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_recv_bytes": 320, 2024-03-22T01:47:24.482 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_send_bytes": 184640, 2024-03-22T01:47:24.482 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_created_connections": 5, 2024-03-22T01:47:24.482 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_active_connections": 3, 2024-03-22T01:47:24.483 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_running_total_time": 0.012583502, 2024-03-22T01:47:24.483 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_running_send_time": 0.008169552, 2024-03-22T01:47:24.483 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_running_recv_time": 0.002041886, 2024-03-22T01:47:24.483 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-22T01:47:24.483 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T01:47:24.483 INFO:teuthology.orchestra.run.smithi072.stdout: "avgcount": 162, 2024-03-22T01:47:24.483 INFO:teuthology.orchestra.run.smithi072.stdout: "sum": 0.011086519, 2024-03-22T01:47:24.483 INFO:teuthology.orchestra.run.smithi072.stdout: "avgtime": 0.000068435 2024-03-22T01:47:24.483 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:24.483 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_handle_ack_lat": { 2024-03-22T01:47:24.483 INFO:teuthology.orchestra.run.smithi072.stdout: "avgcount": 0, 2024-03-22T01:47:24.483 INFO:teuthology.orchestra.run.smithi072.stdout: "sum": 0.000000000, 2024-03-22T01:47:24.483 INFO:teuthology.orchestra.run.smithi072.stdout: "avgtime": 0.000000000 2024-03-22T01:47:24.483 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:24.484 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-22T01:47:24.484 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_send_encrypted_bytes": 184640 2024-03-22T01:47:24.484 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:24.484 INFO:teuthology.orchestra.run.smithi072.stdout: "cct": { 2024-03-22T01:47:24.484 INFO:teuthology.orchestra.run.smithi072.stdout: "total_workers": 0, 2024-03-22T01:47:24.484 INFO:teuthology.orchestra.run.smithi072.stdout: "unhealthy_workers": 0 2024-03-22T01:47:24.484 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:24.484 INFO:teuthology.orchestra.run.smithi072.stdout: "mempool": { 2024-03-22T01:47:24.484 INFO:teuthology.orchestra.run.smithi072.stdout: "bloom_filter_bytes": 0, 2024-03-22T01:47:24.484 INFO:teuthology.orchestra.run.smithi072.stdout: "bloom_filter_items": 0, 2024-03-22T01:47:24.484 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_alloc_bytes": 0, 2024-03-22T01:47:24.484 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_alloc_items": 0, 2024-03-22T01:47:24.484 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_cache_data_bytes": 0, 2024-03-22T01:47:24.484 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_cache_data_items": 0, 2024-03-22T01:47:24.484 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-22T01:47:24.485 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_cache_onode_items": 0, 2024-03-22T01:47:24.485 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-22T01:47:24.485 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_cache_meta_items": 0, 2024-03-22T01:47:24.485 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_cache_other_bytes": 0, 2024-03-22T01:47:24.485 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_cache_other_items": 0, 2024-03-22T01:47:24.485 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-22T01:47:24.485 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_cache_buffer_items": 0, 2024-03-22T01:47:24.485 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_extent_bytes": 0, 2024-03-22T01:47:24.485 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_extent_items": 0, 2024-03-22T01:47:24.485 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_blob_bytes": 0, 2024-03-22T01:47:24.485 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_blob_items": 0, 2024-03-22T01:47:24.485 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-22T01:47:24.485 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_shared_blob_items": 0, 2024-03-22T01:47:24.485 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-22T01:47:24.486 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_inline_bl_items": 0, 2024-03-22T01:47:24.486 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_fsck_bytes": 0, 2024-03-22T01:47:24.486 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_fsck_items": 0, 2024-03-22T01:47:24.486 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_txc_bytes": 0, 2024-03-22T01:47:24.486 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_txc_items": 0, 2024-03-22T01:47:24.486 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-22T01:47:24.486 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_writing_deferred_items": 0, 2024-03-22T01:47:24.486 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_writing_bytes": 0, 2024-03-22T01:47:24.486 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_writing_items": 0, 2024-03-22T01:47:24.486 INFO:teuthology.orchestra.run.smithi072.stdout: "bluefs_bytes": 0, 2024-03-22T01:47:24.486 INFO:teuthology.orchestra.run.smithi072.stdout: "bluefs_items": 0, 2024-03-22T01:47:24.486 INFO:teuthology.orchestra.run.smithi072.stdout: "bluefs_file_reader_bytes": 0, 2024-03-22T01:47:24.486 INFO:teuthology.orchestra.run.smithi072.stdout: "bluefs_file_reader_items": 0, 2024-03-22T01:47:24.487 INFO:teuthology.orchestra.run.smithi072.stdout: "bluefs_file_writer_bytes": 0, 2024-03-22T01:47:24.487 INFO:teuthology.orchestra.run.smithi072.stdout: "bluefs_file_writer_items": 0, 2024-03-22T01:47:24.487 INFO:teuthology.orchestra.run.smithi072.stdout: "buffer_anon_bytes": 149017, 2024-03-22T01:47:24.487 INFO:teuthology.orchestra.run.smithi072.stdout: "buffer_anon_items": 31, 2024-03-22T01:47:24.487 INFO:teuthology.orchestra.run.smithi072.stdout: "buffer_meta_bytes": 0, 2024-03-22T01:47:24.487 INFO:teuthology.orchestra.run.smithi072.stdout: "buffer_meta_items": 0, 2024-03-22T01:47:24.487 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_bytes": 0, 2024-03-22T01:47:24.487 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_items": 0, 2024-03-22T01:47:24.487 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_mapbl_bytes": 0, 2024-03-22T01:47:24.487 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_mapbl_items": 0, 2024-03-22T01:47:24.487 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_pglog_bytes": 0, 2024-03-22T01:47:24.487 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_pglog_items": 0, 2024-03-22T01:47:24.487 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_bytes": 0, 2024-03-22T01:47:24.487 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_items": 0, 2024-03-22T01:47:24.488 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_mapping_bytes": 0, 2024-03-22T01:47:24.488 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_mapping_items": 0, 2024-03-22T01:47:24.488 INFO:teuthology.orchestra.run.smithi072.stdout: "pgmap_bytes": 0, 2024-03-22T01:47:24.488 INFO:teuthology.orchestra.run.smithi072.stdout: "pgmap_items": 0, 2024-03-22T01:47:24.488 INFO:teuthology.orchestra.run.smithi072.stdout: "mds_co_bytes": 928, 2024-03-22T01:47:24.488 INFO:teuthology.orchestra.run.smithi072.stdout: "mds_co_items": 2, 2024-03-22T01:47:24.488 INFO:teuthology.orchestra.run.smithi072.stdout: "unittest_1_bytes": 0, 2024-03-22T01:47:24.488 INFO:teuthology.orchestra.run.smithi072.stdout: "unittest_1_items": 0, 2024-03-22T01:47:24.488 INFO:teuthology.orchestra.run.smithi072.stdout: "unittest_2_bytes": 0, 2024-03-22T01:47:24.488 INFO:teuthology.orchestra.run.smithi072.stdout: "unittest_2_items": 0 2024-03-22T01:47:24.488 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:24.488 INFO:teuthology.orchestra.run.smithi072.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-22T01:47:24.488 INFO:teuthology.orchestra.run.smithi072.stdout: "val": 0, 2024-03-22T01:47:24.488 INFO:teuthology.orchestra.run.smithi072.stdout: "max": 104857600, 2024-03-22T01:47:24.488 INFO:teuthology.orchestra.run.smithi072.stdout: "get_started": 0, 2024-03-22T01:47:24.489 INFO:teuthology.orchestra.run.smithi072.stdout: "get": 228, 2024-03-22T01:47:24.489 INFO:teuthology.orchestra.run.smithi072.stdout: "get_sum": 1412530, 2024-03-22T01:47:24.489 INFO:teuthology.orchestra.run.smithi072.stdout: "get_or_fail_fail": 0, 2024-03-22T01:47:24.489 INFO:teuthology.orchestra.run.smithi072.stdout: "get_or_fail_success": 228, 2024-03-22T01:47:24.489 INFO:teuthology.orchestra.run.smithi072.stdout: "take": 0, 2024-03-22T01:47:24.489 INFO:teuthology.orchestra.run.smithi072.stdout: "take_sum": 0, 2024-03-22T01:47:24.489 INFO:teuthology.orchestra.run.smithi072.stdout: "put": 228, 2024-03-22T01:47:24.489 INFO:teuthology.orchestra.run.smithi072.stdout: "put_sum": 1412530, 2024-03-22T01:47:24.489 INFO:teuthology.orchestra.run.smithi072.stdout: "wait": { 2024-03-22T01:47:24.489 INFO:teuthology.orchestra.run.smithi072.stdout: "avgcount": 0, 2024-03-22T01:47:24.489 INFO:teuthology.orchestra.run.smithi072.stdout: "sum": 0.000000000, 2024-03-22T01:47:24.489 INFO:teuthology.orchestra.run.smithi072.stdout: "avgtime": 0.000000000 2024-03-22T01:47:24.489 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:24.489 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:24.490 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-22T01:47:24.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:24 smithi089 ceph-mon[35770]: pgmap v325: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 85 B/s wr, 2 op/s 2024-03-22T01:47:24.675 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:24 smithi098 ceph-mon[36468]: pgmap v325: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 85 B/s wr, 2 op/s 2024-03-22T01:47:25.217 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.g 2024-03-22T01:47:25.217 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.g 2024-03-22T01:47:25.217 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.g 2024-03-22T01:47:25.217 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.g 2024-03-22T01:47:25.217 DEBUG:tasks.check_counter:Getting stats from j 2024-03-22T01:47:25.218 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph daemon mds.j perf dump 2024-03-22T01:47:25.459 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T01:47:26.592 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:26 smithi072 ceph-mon[32414]: pgmap v326: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-22T01:47:26.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:26 smithi089 ceph-mon[35770]: pgmap v326: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-22T01:47:26.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:26 smithi098 ceph-mon[36468]: pgmap v326: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-22T01:47:27.176 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-22T01:47:27.176 INFO:teuthology.orchestra.run.smithi072.stdout: "AsyncMessenger::Worker-0": { 2024-03-22T01:47:27.176 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_recv_messages": 4, 2024-03-22T01:47:27.176 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_send_messages": 4, 2024-03-22T01:47:27.176 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_recv_bytes": 2992, 2024-03-22T01:47:27.177 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_send_bytes": 448, 2024-03-22T01:47:27.177 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_created_connections": 4, 2024-03-22T01:47:27.177 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_active_connections": 0, 2024-03-22T01:47:27.177 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_running_total_time": 0.001559442, 2024-03-22T01:47:27.177 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_running_send_time": 0.000048378, 2024-03-22T01:47:27.177 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_running_recv_time": 0.000707893, 2024-03-22T01:47:27.177 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-22T01:47:27.177 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T01:47:27.177 INFO:teuthology.orchestra.run.smithi072.stdout: "avgcount": 3, 2024-03-22T01:47:27.177 INFO:teuthology.orchestra.run.smithi072.stdout: "sum": 0.000707135, 2024-03-22T01:47:27.177 INFO:teuthology.orchestra.run.smithi072.stdout: "avgtime": 0.000235711 2024-03-22T01:47:27.177 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:27.177 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_handle_ack_lat": { 2024-03-22T01:47:27.177 INFO:teuthology.orchestra.run.smithi072.stdout: "avgcount": 0, 2024-03-22T01:47:27.177 INFO:teuthology.orchestra.run.smithi072.stdout: "sum": 0.000000000, 2024-03-22T01:47:27.178 INFO:teuthology.orchestra.run.smithi072.stdout: "avgtime": 0.000000000 2024-03-22T01:47:27.178 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:27.178 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-22T01:47:27.178 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_send_encrypted_bytes": 448 2024-03-22T01:47:27.178 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:27.178 INFO:teuthology.orchestra.run.smithi072.stdout: "AsyncMessenger::Worker-1": { 2024-03-22T01:47:27.178 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_recv_messages": 2, 2024-03-22T01:47:27.178 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_send_messages": 169, 2024-03-22T01:47:27.178 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_recv_bytes": 320, 2024-03-22T01:47:27.178 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_send_bytes": 185344, 2024-03-22T01:47:27.178 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_created_connections": 5, 2024-03-22T01:47:27.178 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_active_connections": 3, 2024-03-22T01:47:27.178 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_running_total_time": 0.012278327, 2024-03-22T01:47:27.178 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_running_send_time": 0.008072538, 2024-03-22T01:47:27.179 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_running_recv_time": 0.001802162, 2024-03-22T01:47:27.179 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-22T01:47:27.179 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T01:47:27.179 INFO:teuthology.orchestra.run.smithi072.stdout: "avgcount": 166, 2024-03-22T01:47:27.179 INFO:teuthology.orchestra.run.smithi072.stdout: "sum": 0.009937981, 2024-03-22T01:47:27.179 INFO:teuthology.orchestra.run.smithi072.stdout: "avgtime": 0.000059867 2024-03-22T01:47:27.179 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:27.179 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_handle_ack_lat": { 2024-03-22T01:47:27.179 INFO:teuthology.orchestra.run.smithi072.stdout: "avgcount": 0, 2024-03-22T01:47:27.179 INFO:teuthology.orchestra.run.smithi072.stdout: "sum": 0.000000000, 2024-03-22T01:47:27.179 INFO:teuthology.orchestra.run.smithi072.stdout: "avgtime": 0.000000000 2024-03-22T01:47:27.179 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:27.179 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-22T01:47:27.179 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_send_encrypted_bytes": 185344 2024-03-22T01:47:27.179 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:27.180 INFO:teuthology.orchestra.run.smithi072.stdout: "AsyncMessenger::Worker-2": { 2024-03-22T01:47:27.180 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_recv_messages": 231, 2024-03-22T01:47:27.180 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_send_messages": 221, 2024-03-22T01:47:27.180 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_recv_bytes": 1445776, 2024-03-22T01:47:27.180 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_send_bytes": 135744, 2024-03-22T01:47:27.180 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_created_connections": 2, 2024-03-22T01:47:27.180 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_active_connections": 1, 2024-03-22T01:47:27.180 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_running_total_time": 0.042917568, 2024-03-22T01:47:27.180 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_running_send_time": 0.013358099, 2024-03-22T01:47:27.180 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_running_recv_time": 0.034084468, 2024-03-22T01:47:27.180 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_running_fast_dispatch_time": 0.004328782, 2024-03-22T01:47:27.180 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T01:47:27.180 INFO:teuthology.orchestra.run.smithi072.stdout: "avgcount": 220, 2024-03-22T01:47:27.180 INFO:teuthology.orchestra.run.smithi072.stdout: "sum": 0.011372792, 2024-03-22T01:47:27.180 INFO:teuthology.orchestra.run.smithi072.stdout: "avgtime": 0.000051694 2024-03-22T01:47:27.181 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:27.181 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_handle_ack_lat": { 2024-03-22T01:47:27.181 INFO:teuthology.orchestra.run.smithi072.stdout: "avgcount": 0, 2024-03-22T01:47:27.181 INFO:teuthology.orchestra.run.smithi072.stdout: "sum": 0.000000000, 2024-03-22T01:47:27.181 INFO:teuthology.orchestra.run.smithi072.stdout: "avgtime": 0.000000000 2024-03-22T01:47:27.181 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:27.181 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_recv_encrypted_bytes": 1445776, 2024-03-22T01:47:27.181 INFO:teuthology.orchestra.run.smithi072.stdout: "msgr_send_encrypted_bytes": 135744 2024-03-22T01:47:27.181 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:27.181 INFO:teuthology.orchestra.run.smithi072.stdout: "cct": { 2024-03-22T01:47:27.181 INFO:teuthology.orchestra.run.smithi072.stdout: "total_workers": 0, 2024-03-22T01:47:27.181 INFO:teuthology.orchestra.run.smithi072.stdout: "unhealthy_workers": 0 2024-03-22T01:47:27.181 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:27.181 INFO:teuthology.orchestra.run.smithi072.stdout: "mempool": { 2024-03-22T01:47:27.181 INFO:teuthology.orchestra.run.smithi072.stdout: "bloom_filter_bytes": 0, 2024-03-22T01:47:27.182 INFO:teuthology.orchestra.run.smithi072.stdout: "bloom_filter_items": 0, 2024-03-22T01:47:27.182 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_alloc_bytes": 0, 2024-03-22T01:47:27.182 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_alloc_items": 0, 2024-03-22T01:47:27.182 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_cache_data_bytes": 0, 2024-03-22T01:47:27.182 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_cache_data_items": 0, 2024-03-22T01:47:27.182 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-22T01:47:27.182 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_cache_onode_items": 0, 2024-03-22T01:47:27.182 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-22T01:47:27.182 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_cache_meta_items": 0, 2024-03-22T01:47:27.182 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_cache_other_bytes": 0, 2024-03-22T01:47:27.182 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_cache_other_items": 0, 2024-03-22T01:47:27.182 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-22T01:47:27.182 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_cache_buffer_items": 0, 2024-03-22T01:47:27.182 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_extent_bytes": 0, 2024-03-22T01:47:27.182 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_extent_items": 0, 2024-03-22T01:47:27.183 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_blob_bytes": 0, 2024-03-22T01:47:27.183 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_blob_items": 0, 2024-03-22T01:47:27.183 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-22T01:47:27.183 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_shared_blob_items": 0, 2024-03-22T01:47:27.183 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-22T01:47:27.183 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_inline_bl_items": 0, 2024-03-22T01:47:27.183 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_fsck_bytes": 0, 2024-03-22T01:47:27.183 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_fsck_items": 0, 2024-03-22T01:47:27.183 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_txc_bytes": 0, 2024-03-22T01:47:27.183 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_txc_items": 0, 2024-03-22T01:47:27.183 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-22T01:47:27.183 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_writing_deferred_items": 0, 2024-03-22T01:47:27.183 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_writing_bytes": 0, 2024-03-22T01:47:27.183 INFO:teuthology.orchestra.run.smithi072.stdout: "bluestore_writing_items": 0, 2024-03-22T01:47:27.183 INFO:teuthology.orchestra.run.smithi072.stdout: "bluefs_bytes": 0, 2024-03-22T01:47:27.184 INFO:teuthology.orchestra.run.smithi072.stdout: "bluefs_items": 0, 2024-03-22T01:47:27.184 INFO:teuthology.orchestra.run.smithi072.stdout: "bluefs_file_reader_bytes": 0, 2024-03-22T01:47:27.184 INFO:teuthology.orchestra.run.smithi072.stdout: "bluefs_file_reader_items": 0, 2024-03-22T01:47:27.184 INFO:teuthology.orchestra.run.smithi072.stdout: "bluefs_file_writer_bytes": 0, 2024-03-22T01:47:27.184 INFO:teuthology.orchestra.run.smithi072.stdout: "bluefs_file_writer_items": 0, 2024-03-22T01:47:27.184 INFO:teuthology.orchestra.run.smithi072.stdout: "buffer_anon_bytes": 151029, 2024-03-22T01:47:27.184 INFO:teuthology.orchestra.run.smithi072.stdout: "buffer_anon_items": 31, 2024-03-22T01:47:27.184 INFO:teuthology.orchestra.run.smithi072.stdout: "buffer_meta_bytes": 0, 2024-03-22T01:47:27.184 INFO:teuthology.orchestra.run.smithi072.stdout: "buffer_meta_items": 0, 2024-03-22T01:47:27.184 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_bytes": 0, 2024-03-22T01:47:27.184 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_items": 0, 2024-03-22T01:47:27.184 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_mapbl_bytes": 0, 2024-03-22T01:47:27.184 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_mapbl_items": 0, 2024-03-22T01:47:27.184 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_pglog_bytes": 0, 2024-03-22T01:47:27.184 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_pglog_items": 0, 2024-03-22T01:47:27.185 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_bytes": 0, 2024-03-22T01:47:27.185 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_items": 0, 2024-03-22T01:47:27.185 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_mapping_bytes": 0, 2024-03-22T01:47:27.185 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_mapping_items": 0, 2024-03-22T01:47:27.185 INFO:teuthology.orchestra.run.smithi072.stdout: "pgmap_bytes": 0, 2024-03-22T01:47:27.185 INFO:teuthology.orchestra.run.smithi072.stdout: "pgmap_items": 0, 2024-03-22T01:47:27.185 INFO:teuthology.orchestra.run.smithi072.stdout: "mds_co_bytes": 928, 2024-03-22T01:47:27.185 INFO:teuthology.orchestra.run.smithi072.stdout: "mds_co_items": 2, 2024-03-22T01:47:27.185 INFO:teuthology.orchestra.run.smithi072.stdout: "unittest_1_bytes": 0, 2024-03-22T01:47:27.185 INFO:teuthology.orchestra.run.smithi072.stdout: "unittest_1_items": 0, 2024-03-22T01:47:27.185 INFO:teuthology.orchestra.run.smithi072.stdout: "unittest_2_bytes": 0, 2024-03-22T01:47:27.185 INFO:teuthology.orchestra.run.smithi072.stdout: "unittest_2_items": 0 2024-03-22T01:47:27.185 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-22T01:47:27.185 INFO:teuthology.orchestra.run.smithi072.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-22T01:47:27.185 INFO:teuthology.orchestra.run.smithi072.stdout: "val": 0, 2024-03-22T01:47:27.186 INFO:teuthology.orchestra.run.smithi072.stdout: "max": 104857600, 2024-03-22T01:47:27.186 INFO:teuthology.orchestra.run.smithi072.stdout: "get_started": 0, 2024-03-22T01:47:27.186 INFO:teuthology.orchestra.run.smithi072.stdout: "get": 233, 2024-03-22T01:47:27.186 INFO:teuthology.orchestra.run.smithi072.stdout: "get_sum": 1413180, 2024-03-22T01:47:27.186 INFO:teuthology.orchestra.run.smithi072.stdout: "get_or_fail_fail": 0, 2024-03-22T01:47:27.186 INFO:teuthology.orchestra.run.smithi072.stdout: "get_or_fail_success": 233, 2024-03-22T01:47:27.186 INFO:teuthology.orchestra.run.smithi072.stdout: "take": 0, 2024-03-22T01:47:27.186 INFO:teuthology.orchestra.run.smithi072.stdout: "take_sum": 0, 2024-03-22T01:47:27.186 INFO:teuthology.orchestra.run.smithi072.stdout: "put": 233, 2024-03-22T01:47:27.186 INFO:teuthology.orchestra.run.smithi072.stdout: "put_sum": 1413180, 2024-03-22T01:47:27.186 INFO:teuthology.orchestra.run.smithi072.stdout: "wait": { 2024-03-22T01:47:27.186 INFO:teuthology.orchestra.run.smithi072.stdout: "avgcount": 0, 2024-03-22T01:47:27.186 INFO:teuthology.orchestra.run.smithi072.stdout: "sum": 0.000000000, 2024-03-22T01:47:27.186 INFO:teuthology.orchestra.run.smithi072.stdout: "avgtime": 0.000000000 2024-03-22T01:47:27.186 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:27.187 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-22T01:47:27.187 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-22T01:47:27.828 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-03-22T01:47:27.828 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-03-22T01:47:27.828 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-03-22T01:47:27.828 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.j 2024-03-22T01:47:27.828 DEBUG:tasks.check_counter:Getting stats from b 2024-03-22T01:47:27.829 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph daemon mds.b perf dump 2024-03-22T01:47:28.194 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.b/config 2024-03-22T01:47:28.508 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:28 smithi089 ceph-mon[35770]: pgmap v327: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-22T01:47:28.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:28 smithi072 ceph-mon[32414]: pgmap v327: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-22T01:47:28.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:28 smithi098 ceph-mon[36468]: pgmap v327: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-22T01:47:30.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:30 smithi089 ceph-mon[35770]: pgmap v328: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-22T01:47:30.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:30 smithi072 ceph-mon[32414]: pgmap v328: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-22T01:47:30.667 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-03-22T01:47:30.667 INFO:teuthology.orchestra.run.smithi089.stdout: "AsyncMessenger::Worker-0": { 2024-03-22T01:47:30.667 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_messages": 4937, 2024-03-22T01:47:30.667 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages": 4219, 2024-03-22T01:47:30.668 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_bytes": 3089609, 2024-03-22T01:47:30.668 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_bytes": 10240468, 2024-03-22T01:47:30.668 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_created_connections": 15, 2024-03-22T01:47:30.668 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_active_connections": 5, 2024-03-22T01:47:30.668 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_total_time": 0.397894087, 2024-03-22T01:47:30.668 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_send_time": 0.112323182, 2024-03-22T01:47:30.669 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_recv_time": 0.130397286, 2024-03-22T01:47:30.669 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_fast_dispatch_time": 0.138232606, 2024-03-22T01:47:30.669 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T01:47:30.669 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 4218, 2024-03-22T01:47:30.669 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 1.221094138, 2024-03-22T01:47:30.669 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000289496 2024-03-22T01:47:30.669 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:30.669 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_handle_ack_lat": { 2024-03-22T01:47:30.669 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 744, 2024-03-22T01:47:30.669 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000100446, 2024-03-22T01:47:30.670 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000135 2024-03-22T01:47:30.670 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:30.670 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_encrypted_bytes": 1954272, 2024-03-22T01:47:30.670 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_encrypted_bytes": 205696 2024-03-22T01:47:30.670 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:30.670 INFO:teuthology.orchestra.run.smithi089.stdout: "AsyncMessenger::Worker-1": { 2024-03-22T01:47:30.670 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_messages": 3511, 2024-03-22T01:47:30.670 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages": 3373, 2024-03-22T01:47:30.670 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_bytes": 1163275, 2024-03-22T01:47:30.670 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_bytes": 26575967, 2024-03-22T01:47:30.671 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_created_connections": 92, 2024-03-22T01:47:30.671 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_active_connections": 5, 2024-03-22T01:47:30.671 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_total_time": 0.340797003, 2024-03-22T01:47:30.671 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_send_time": 0.121517103, 2024-03-22T01:47:30.671 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_recv_time": 0.080636971, 2024-03-22T01:47:30.671 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_fast_dispatch_time": 0.114969592, 2024-03-22T01:47:30.671 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T01:47:30.672 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 3370, 2024-03-22T01:47:30.672 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 1.480296780, 2024-03-22T01:47:30.672 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000439257 2024-03-22T01:47:30.672 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:30.672 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_handle_ack_lat": { 2024-03-22T01:47:30.672 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 294, 2024-03-22T01:47:30.672 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000032474, 2024-03-22T01:47:30.672 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000110 2024-03-22T01:47:30.672 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:30.672 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_encrypted_bytes": 86448, 2024-03-22T01:47:30.672 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_encrypted_bytes": 143424 2024-03-22T01:47:30.672 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:30.672 INFO:teuthology.orchestra.run.smithi089.stdout: "AsyncMessenger::Worker-2": { 2024-03-22T01:47:30.672 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_messages": 26020, 2024-03-22T01:47:30.673 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages": 30243, 2024-03-22T01:47:30.673 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_bytes": 9461596, 2024-03-22T01:47:30.673 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_bytes": 26819540, 2024-03-22T01:47:30.673 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_created_connections": 16, 2024-03-22T01:47:30.673 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_active_connections": 6, 2024-03-22T01:47:30.673 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_total_time": 1.622160787, 2024-03-22T01:47:30.673 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_send_time": 0.652838320, 2024-03-22T01:47:30.673 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_recv_time": 1.529561246, 2024-03-22T01:47:30.673 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_fast_dispatch_time": 0.144512771, 2024-03-22T01:47:30.673 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T01:47:30.673 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 30242, 2024-03-22T01:47:30.673 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 3.882753921, 2024-03-22T01:47:30.673 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000128389 2024-03-22T01:47:30.673 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:30.673 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_handle_ack_lat": { 2024-03-22T01:47:30.674 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 46118, 2024-03-22T01:47:30.674 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.006984789, 2024-03-22T01:47:30.674 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000151 2024-03-22T01:47:30.674 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:30.674 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_encrypted_bytes": 8365424, 2024-03-22T01:47:30.674 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_encrypted_bytes": 13542288 2024-03-22T01:47:30.674 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:30.674 INFO:teuthology.orchestra.run.smithi089.stdout: "cct": { 2024-03-22T01:47:30.674 INFO:teuthology.orchestra.run.smithi089.stdout: "total_workers": 1, 2024-03-22T01:47:30.674 INFO:teuthology.orchestra.run.smithi089.stdout: "unhealthy_workers": 0 2024-03-22T01:47:30.674 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:30.674 INFO:teuthology.orchestra.run.smithi089.stdout: "finisher-MDSRank": { 2024-03-22T01:47:30.675 INFO:teuthology.orchestra.run.smithi089.stdout: "queue_len": 0, 2024-03-22T01:47:30.675 INFO:teuthology.orchestra.run.smithi089.stdout: "complete_latency": { 2024-03-22T01:47:30.675 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 4567, 2024-03-22T01:47:30.675 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 7.189444495, 2024-03-22T01:47:30.675 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.001574216 2024-03-22T01:47:30.675 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-03-22T01:47:30.675 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:30.675 INFO:teuthology.orchestra.run.smithi089.stdout: "finisher-PurgeQueue": { 2024-03-22T01:47:30.675 INFO:teuthology.orchestra.run.smithi089.stdout: "queue_len": 0, 2024-03-22T01:47:30.675 INFO:teuthology.orchestra.run.smithi089.stdout: "complete_latency": { 2024-03-22T01:47:30.675 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 4156, 2024-03-22T01:47:30.676 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.270581462, 2024-03-22T01:47:30.676 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000065106 2024-03-22T01:47:30.676 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-03-22T01:47:30.676 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:30.676 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": { 2024-03-22T01:47:30.676 INFO:teuthology.orchestra.run.smithi089.stdout: "request": 2415, 2024-03-22T01:47:30.676 INFO:teuthology.orchestra.run.smithi089.stdout: "reply": 2415, 2024-03-22T01:47:30.676 INFO:teuthology.orchestra.run.smithi089.stdout: "reply_latency": { 2024-03-22T01:47:30.676 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 2415, 2024-03-22T01:47:30.676 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 399.122420215, 2024-03-22T01:47:30.677 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.165268082 2024-03-22T01:47:30.677 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:30.677 INFO:teuthology.orchestra.run.smithi089.stdout: "slow_reply": 1, 2024-03-22T01:47:30.677 INFO:teuthology.orchestra.run.smithi089.stdout: "forward": 0, 2024-03-22T01:47:30.677 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_fetch_complete": 0, 2024-03-22T01:47:30.677 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_fetch_keys": 0, 2024-03-22T01:47:30.677 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_commit": 371, 2024-03-22T01:47:30.677 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_split": 1, 2024-03-22T01:47:30.677 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_merge": 0, 2024-03-22T01:47:30.677 INFO:teuthology.orchestra.run.smithi089.stdout: "inodes": 986, 2024-03-22T01:47:30.677 INFO:teuthology.orchestra.run.smithi089.stdout: "inodes_top": 622, 2024-03-22T01:47:30.678 INFO:teuthology.orchestra.run.smithi089.stdout: "inodes_bottom": 364, 2024-03-22T01:47:30.678 INFO:teuthology.orchestra.run.smithi089.stdout: "inodes_pin_tail": 0, 2024-03-22T01:47:30.678 INFO:teuthology.orchestra.run.smithi089.stdout: "inodes_pinned": 97, 2024-03-22T01:47:30.678 INFO:teuthology.orchestra.run.smithi089.stdout: "inodes_expired": 0, 2024-03-22T01:47:30.678 INFO:teuthology.orchestra.run.smithi089.stdout: "inodes_with_caps": 3, 2024-03-22T01:47:30.678 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": 4, 2024-03-22T01:47:30.679 INFO:teuthology.orchestra.run.smithi089.stdout: "subtrees": 2, 2024-03-22T01:47:30.679 INFO:teuthology.orchestra.run.smithi089.stdout: "traverse": 3018, 2024-03-22T01:47:30.679 INFO:teuthology.orchestra.run.smithi089.stdout: "traverse_hit": 2900, 2024-03-22T01:47:30.679 INFO:teuthology.orchestra.run.smithi089.stdout: "traverse_forward": 0, 2024-03-22T01:47:30.679 INFO:teuthology.orchestra.run.smithi089.stdout: "traverse_discover": 0, 2024-03-22T01:47:30.679 INFO:teuthology.orchestra.run.smithi089.stdout: "traverse_dir_fetch": 0, 2024-03-22T01:47:30.679 INFO:teuthology.orchestra.run.smithi089.stdout: "traverse_remote_ino": 0, 2024-03-22T01:47:30.679 INFO:teuthology.orchestra.run.smithi089.stdout: "traverse_lock": 51, 2024-03-22T01:47:30.679 INFO:teuthology.orchestra.run.smithi089.stdout: "load_cent": 0, 2024-03-22T01:47:30.679 INFO:teuthology.orchestra.run.smithi089.stdout: "q": 0, 2024-03-22T01:47:30.679 INFO:teuthology.orchestra.run.smithi089.stdout: "exported": 0, 2024-03-22T01:47:30.679 INFO:teuthology.orchestra.run.smithi089.stdout: "exported_inodes": 0, 2024-03-22T01:47:30.680 INFO:teuthology.orchestra.run.smithi089.stdout: "imported": 0, 2024-03-22T01:47:30.680 INFO:teuthology.orchestra.run.smithi089.stdout: "imported_inodes": 0, 2024-03-22T01:47:30.680 INFO:teuthology.orchestra.run.smithi089.stdout: "openino_dir_fetch": 0, 2024-03-22T01:47:30.680 INFO:teuthology.orchestra.run.smithi089.stdout: "openino_backtrace_fetch": 1212, 2024-03-22T01:47:30.680 INFO:teuthology.orchestra.run.smithi089.stdout: "openino_peer_discover": 0, 2024-03-22T01:47:30.680 INFO:teuthology.orchestra.run.smithi089.stdout: "root_rfiles": 4, 2024-03-22T01:47:30.680 INFO:teuthology.orchestra.run.smithi089.stdout: "root_rbytes": 234, 2024-03-22T01:47:30.680 INFO:teuthology.orchestra.run.smithi089.stdout: "root_rsnaps": 5, 2024-03-22T01:47:30.680 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_backtrace_fetch": 1212, 2024-03-22T01:47:30.680 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_set_tag": 0, 2024-03-22T01:47:30.680 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_backtrace_repaired": 0, 2024-03-22T01:47:30.680 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_inotable_repaired": 0, 2024-03-22T01:47:30.681 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_dir_inodes": 301, 2024-03-22T01:47:30.681 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_dir_base_inodes": 7, 2024-03-22T01:47:30.681 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_dirfrag_rstats": 294, 2024-03-22T01:47:30.681 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_file_inodes": 911, 2024-03-22T01:47:30.681 INFO:teuthology.orchestra.run.smithi089.stdout: "handle_inode_file_caps": 0, 2024-03-22T01:47:30.681 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph_cap_op_revoke": 337, 2024-03-22T01:47:30.681 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph_cap_op_grant": 3738, 2024-03-22T01:47:30.682 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph_cap_op_trunc": 6, 2024-03-22T01:47:30.682 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-22T01:47:30.682 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-22T01:47:30.682 INFO:teuthology.orchestra.run.smithi089.stdout: "handle_client_caps": 19389, 2024-03-22T01:47:30.682 INFO:teuthology.orchestra.run.smithi089.stdout: "handle_client_caps_dirty": 18244, 2024-03-22T01:47:30.682 INFO:teuthology.orchestra.run.smithi089.stdout: "handle_client_cap_release": 112, 2024-03-22T01:47:30.682 INFO:teuthology.orchestra.run.smithi089.stdout: "process_request_cap_release": 1336 2024-03-22T01:47:30.682 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:30.682 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_cache": { 2024-03-22T01:47:30.682 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_update": 0, 2024-03-22T01:47:30.682 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_update_receipt": 0, 2024-03-22T01:47:30.683 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_try_discover": 0, 2024-03-22T01:47:30.683 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_send_discover": 0, 2024-03-22T01:47:30.683 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_handle_discover": 0, 2024-03-22T01:47:30.683 INFO:teuthology.orchestra.run.smithi089.stdout: "num_strays": 83, 2024-03-22T01:47:30.683 INFO:teuthology.orchestra.run.smithi089.stdout: "num_strays_delayed": 0, 2024-03-22T01:47:30.683 INFO:teuthology.orchestra.run.smithi089.stdout: "num_strays_enqueuing": 0, 2024-03-22T01:47:30.683 INFO:teuthology.orchestra.run.smithi089.stdout: "strays_created": 933, 2024-03-22T01:47:30.683 INFO:teuthology.orchestra.run.smithi089.stdout: "strays_enqueued": 789, 2024-03-22T01:47:30.683 INFO:teuthology.orchestra.run.smithi089.stdout: "strays_reintegrated": 61, 2024-03-22T01:47:30.683 INFO:teuthology.orchestra.run.smithi089.stdout: "strays_migrated": 0, 2024-03-22T01:47:30.683 INFO:teuthology.orchestra.run.smithi089.stdout: "num_recovering_processing": 0, 2024-03-22T01:47:30.683 INFO:teuthology.orchestra.run.smithi089.stdout: "num_recovering_enqueued": 0, 2024-03-22T01:47:30.683 INFO:teuthology.orchestra.run.smithi089.stdout: "num_recovering_prioritized": 0, 2024-03-22T01:47:30.683 INFO:teuthology.orchestra.run.smithi089.stdout: "recovery_started": 0, 2024-03-22T01:47:30.683 INFO:teuthology.orchestra.run.smithi089.stdout: "recovery_completed": 0, 2024-03-22T01:47:30.684 INFO:teuthology.orchestra.run.smithi089.stdout: "ireq_quiesce_path": 0, 2024-03-22T01:47:30.684 INFO:teuthology.orchestra.run.smithi089.stdout: "ireq_quiesce_inode": 0, 2024-03-22T01:47:30.684 INFO:teuthology.orchestra.run.smithi089.stdout: "ireq_enqueue_scrub": 7, 2024-03-22T01:47:30.684 INFO:teuthology.orchestra.run.smithi089.stdout: "ireq_exportdir": 0, 2024-03-22T01:47:30.684 INFO:teuthology.orchestra.run.smithi089.stdout: "ireq_flush": 0, 2024-03-22T01:47:30.684 INFO:teuthology.orchestra.run.smithi089.stdout: "ireq_fragmentdir": 1, 2024-03-22T01:47:30.684 INFO:teuthology.orchestra.run.smithi089.stdout: "ireq_fragstats": 0, 2024-03-22T01:47:30.684 INFO:teuthology.orchestra.run.smithi089.stdout: "ireq_inodestats": 0 2024-03-22T01:47:30.684 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:30.684 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_log": { 2024-03-22T01:47:30.684 INFO:teuthology.orchestra.run.smithi089.stdout: "evlrg": 0, 2024-03-22T01:47:30.685 INFO:teuthology.orchestra.run.smithi089.stdout: "evadd": 22002, 2024-03-22T01:47:30.685 INFO:teuthology.orchestra.run.smithi089.stdout: "evex": 21996, 2024-03-22T01:47:30.685 INFO:teuthology.orchestra.run.smithi089.stdout: "evtrm": 21996, 2024-03-22T01:47:30.685 INFO:teuthology.orchestra.run.smithi089.stdout: "ev": 6, 2024-03-22T01:47:30.685 INFO:teuthology.orchestra.run.smithi089.stdout: "evexg": 0, 2024-03-22T01:47:30.685 INFO:teuthology.orchestra.run.smithi089.stdout: "evexd": 1, 2024-03-22T01:47:30.685 INFO:teuthology.orchestra.run.smithi089.stdout: "segadd": 41, 2024-03-22T01:47:30.685 INFO:teuthology.orchestra.run.smithi089.stdout: "segex": 40, 2024-03-22T01:47:30.685 INFO:teuthology.orchestra.run.smithi089.stdout: "segtrm": 40, 2024-03-22T01:47:30.686 INFO:teuthology.orchestra.run.smithi089.stdout: "seg": 1, 2024-03-22T01:47:30.686 INFO:teuthology.orchestra.run.smithi089.stdout: "segmjr": 1, 2024-03-22T01:47:30.686 INFO:teuthology.orchestra.run.smithi089.stdout: "segexg": 0, 2024-03-22T01:47:30.686 INFO:teuthology.orchestra.run.smithi089.stdout: "segexd": 1, 2024-03-22T01:47:30.686 INFO:teuthology.orchestra.run.smithi089.stdout: "expos": 52273579, 2024-03-22T01:47:30.686 INFO:teuthology.orchestra.run.smithi089.stdout: "wrpos": 52297594, 2024-03-22T01:47:30.686 INFO:teuthology.orchestra.run.smithi089.stdout: "rdpos": 0, 2024-03-22T01:47:30.686 INFO:teuthology.orchestra.run.smithi089.stdout: "jlat": { 2024-03-22T01:47:30.687 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 2079, 2024-03-22T01:47:30.687 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 3925.691912550, 2024-03-22T01:47:30.687 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 1.888259698 2024-03-22T01:47:30.687 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:30.687 INFO:teuthology.orchestra.run.smithi089.stdout: "replayed": 0 2024-03-22T01:47:30.687 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:30.687 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_mem": { 2024-03-22T01:47:30.687 INFO:teuthology.orchestra.run.smithi089.stdout: "ino": 426, 2024-03-22T01:47:30.687 INFO:teuthology.orchestra.run.smithi089.stdout: "ino+": 1222, 2024-03-22T01:47:30.687 INFO:teuthology.orchestra.run.smithi089.stdout: "ino-": 796, 2024-03-22T01:47:30.687 INFO:teuthology.orchestra.run.smithi089.stdout: "dir": 108, 2024-03-22T01:47:30.687 INFO:teuthology.orchestra.run.smithi089.stdout: "dir+": 125, 2024-03-22T01:47:30.687 INFO:teuthology.orchestra.run.smithi089.stdout: "dir-": 17, 2024-03-22T01:47:30.687 INFO:teuthology.orchestra.run.smithi089.stdout: "dn": 986, 2024-03-22T01:47:30.688 INFO:teuthology.orchestra.run.smithi089.stdout: "dn+": 1984, 2024-03-22T01:47:30.688 INFO:teuthology.orchestra.run.smithi089.stdout: "dn-": 998, 2024-03-22T01:47:30.688 INFO:teuthology.orchestra.run.smithi089.stdout: "cap": 4, 2024-03-22T01:47:30.688 INFO:teuthology.orchestra.run.smithi089.stdout: "cap+": 892, 2024-03-22T01:47:30.688 INFO:teuthology.orchestra.run.smithi089.stdout: "cap-": 888, 2024-03-22T01:47:30.688 INFO:teuthology.orchestra.run.smithi089.stdout: "rss": 83092, 2024-03-22T01:47:30.688 INFO:teuthology.orchestra.run.smithi089.stdout: "heap": 231800 2024-03-22T01:47:30.688 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:30.688 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_server": { 2024-03-22T01:47:30.688 INFO:teuthology.orchestra.run.smithi089.stdout: "dispatch_client_request": 3005, 2024-03-22T01:47:30.690 INFO:teuthology.orchestra.run.smithi089.stdout: "dispatch_server_request": 0, 2024-03-22T01:47:30.690 INFO:teuthology.orchestra.run.smithi089.stdout: "handle_client_request": 2415, 2024-03-22T01:47:30.690 INFO:teuthology.orchestra.run.smithi089.stdout: "handle_client_session": 36, 2024-03-22T01:47:30.690 INFO:teuthology.orchestra.run.smithi089.stdout: "handle_peer_request": 0, 2024-03-22T01:47:30.690 INFO:teuthology.orchestra.run.smithi089.stdout: "req_create_latency": { 2024-03-22T01:47:30.690 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 764, 2024-03-22T01:47:30.690 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.911693098, 2024-03-22T01:47:30.690 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.001193315 2024-03-22T01:47:30.690 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:30.690 INFO:teuthology.orchestra.run.smithi089.stdout: "req_getattr_latency": { 2024-03-22T01:47:30.690 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 123, 2024-03-22T01:47:30.690 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.021759099, 2024-03-22T01:47:30.691 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000176903 2024-03-22T01:47:30.691 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:30.691 INFO:teuthology.orchestra.run.smithi089.stdout: "req_getfilelock_latency": { 2024-03-22T01:47:30.691 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-03-22T01:47:30.691 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-03-22T01:47:30.691 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-03-22T01:47:30.691 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:30.691 INFO:teuthology.orchestra.run.smithi089.stdout: "req_link_latency": { 2024-03-22T01:47:30.691 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 61, 2024-03-22T01:47:30.691 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.056333697, 2024-03-22T01:47:30.691 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000923503 2024-03-22T01:47:30.691 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:30.692 INFO:teuthology.orchestra.run.smithi089.stdout: "req_lookup_latency": { 2024-03-22T01:47:30.692 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 80, 2024-03-22T01:47:30.692 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.022135918, 2024-03-22T01:47:30.692 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000276698 2024-03-22T01:47:30.692 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:30.692 INFO:teuthology.orchestra.run.smithi089.stdout: "req_lookuphash_latency": { 2024-03-22T01:47:30.692 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-03-22T01:47:30.692 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-03-22T01:47:30.692 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-03-22T01:47:30.692 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:30.692 INFO:teuthology.orchestra.run.smithi089.stdout: "req_lookupino_latency": { 2024-03-22T01:47:30.692 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 2, 2024-03-22T01:47:30.693 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000354953, 2024-03-22T01:47:30.693 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000177476 2024-03-22T01:47:30.693 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:30.693 INFO:teuthology.orchestra.run.smithi089.stdout: "req_lookupname_latency": { 2024-03-22T01:47:30.693 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-03-22T01:47:30.693 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-03-22T01:47:30.693 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-03-22T01:47:30.693 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:30.693 INFO:teuthology.orchestra.run.smithi089.stdout: "req_lookupparent_latency": { 2024-03-22T01:47:30.693 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-03-22T01:47:30.693 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-03-22T01:47:30.693 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-03-22T01:47:30.693 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:30.693 INFO:teuthology.orchestra.run.smithi089.stdout: "req_lookupsnap_latency": { 2024-03-22T01:47:30.694 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-03-22T01:47:30.694 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-03-22T01:47:30.694 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-03-22T01:47:30.694 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:30.694 INFO:teuthology.orchestra.run.smithi089.stdout: "req_lssnap_latency": { 2024-03-22T01:47:30.694 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 5, 2024-03-22T01:47:30.694 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000586912, 2024-03-22T01:47:30.694 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000117382 2024-03-22T01:47:30.694 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:30.694 INFO:teuthology.orchestra.run.smithi089.stdout: "req_mkdir_latency": { 2024-03-22T01:47:30.694 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 105, 2024-03-22T01:47:30.695 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.113035557, 2024-03-22T01:47:30.695 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.001076529 2024-03-22T01:47:30.695 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:30.695 INFO:teuthology.orchestra.run.smithi089.stdout: "req_mknod_latency": { 2024-03-22T01:47:30.695 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-03-22T01:47:30.695 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-03-22T01:47:30.695 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-03-22T01:47:30.695 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:30.695 INFO:teuthology.orchestra.run.smithi089.stdout: "req_mksnap_latency": { 2024-03-22T01:47:30.695 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 5, 2024-03-22T01:47:30.695 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 381.168926269, 2024-03-22T01:47:30.695 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 76.233785253 2024-03-22T01:47:30.695 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:30.695 INFO:teuthology.orchestra.run.smithi089.stdout: "req_open_latency": { 2024-03-22T01:47:30.695 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 49, 2024-03-22T01:47:30.696 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.010166312, 2024-03-22T01:47:30.696 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000207475 2024-03-22T01:47:30.696 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:30.696 INFO:teuthology.orchestra.run.smithi089.stdout: "req_readdir_latency": { 2024-03-22T01:47:30.696 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 175, 2024-03-22T01:47:30.696 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 1.093802455, 2024-03-22T01:47:30.696 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.006250299 2024-03-22T01:47:30.696 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:30.696 INFO:teuthology.orchestra.run.smithi089.stdout: "req_rename_latency": { 2024-03-22T01:47:30.696 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 113, 2024-03-22T01:47:30.696 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 13.009998711, 2024-03-22T01:47:30.696 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.115132731 2024-03-22T01:47:30.696 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:30.696 INFO:teuthology.orchestra.run.smithi089.stdout: "req_renamesnap_latency": { 2024-03-22T01:47:30.697 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-03-22T01:47:30.697 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-03-22T01:47:30.697 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-03-22T01:47:30.697 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:30.697 INFO:teuthology.orchestra.run.smithi089.stdout: "req_snapdiff_latency": { 2024-03-22T01:47:30.697 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-03-22T01:47:30.697 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-03-22T01:47:30.697 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-03-22T01:47:30.697 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:30.697 INFO:teuthology.orchestra.run.smithi089.stdout: "req_rmdir_latency": { 2024-03-22T01:47:30.697 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 99, 2024-03-22T01:47:30.697 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 1.301178773, 2024-03-22T01:47:30.697 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.013143219 2024-03-22T01:47:30.697 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:30.698 INFO:teuthology.orchestra.run.smithi089.stdout: "req_rmsnap_latency": { 2024-03-22T01:47:30.698 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-03-22T01:47:30.698 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-03-22T01:47:30.698 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-03-22T01:47:30.698 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:30.698 INFO:teuthology.orchestra.run.smithi089.stdout: "req_rmxattr_latency": { 2024-03-22T01:47:30.698 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-03-22T01:47:30.698 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-03-22T01:47:30.698 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-03-22T01:47:30.698 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:30.698 INFO:teuthology.orchestra.run.smithi089.stdout: "req_setattr_latency": { 2024-03-22T01:47:30.699 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 8, 2024-03-22T01:47:30.699 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.055326560, 2024-03-22T01:47:30.699 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.006915820 2024-03-22T01:47:30.699 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:30.699 INFO:teuthology.orchestra.run.smithi089.stdout: "req_setdirlayout_latency": { 2024-03-22T01:47:30.699 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-03-22T01:47:30.699 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-03-22T01:47:30.699 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-03-22T01:47:30.700 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:30.700 INFO:teuthology.orchestra.run.smithi089.stdout: "req_setfilelock_latency": { 2024-03-22T01:47:30.700 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 4, 2024-03-22T01:47:30.700 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.001286092, 2024-03-22T01:47:30.700 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000321523 2024-03-22T01:47:30.700 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:30.700 INFO:teuthology.orchestra.run.smithi089.stdout: "req_setlayout_latency": { 2024-03-22T01:47:30.700 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-03-22T01:47:30.700 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-03-22T01:47:30.700 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-03-22T01:47:30.700 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:30.700 INFO:teuthology.orchestra.run.smithi089.stdout: "req_setxattr_latency": { 2024-03-22T01:47:30.700 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 8, 2024-03-22T01:47:30.700 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.020951008, 2024-03-22T01:47:30.701 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.002618876 2024-03-22T01:47:30.701 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:30.701 INFO:teuthology.orchestra.run.smithi089.stdout: "req_symlink_latency": { 2024-03-22T01:47:30.701 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 13, 2024-03-22T01:47:30.701 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.010903309, 2024-03-22T01:47:30.701 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000838716 2024-03-22T01:47:30.701 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:30.701 INFO:teuthology.orchestra.run.smithi089.stdout: "req_unlink_latency": { 2024-03-22T01:47:30.701 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 801, 2024-03-22T01:47:30.701 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 1.323981492, 2024-03-22T01:47:30.701 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.001652910 2024-03-22T01:47:30.702 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:30.702 INFO:teuthology.orchestra.run.smithi089.stdout: "cap_revoke_eviction": 0, 2024-03-22T01:47:30.702 INFO:teuthology.orchestra.run.smithi089.stdout: "cap_acquisition_throttle": 0, 2024-03-22T01:47:30.702 INFO:teuthology.orchestra.run.smithi089.stdout: "req_getvxattr_latency": { 2024-03-22T01:47:30.702 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-03-22T01:47:30.702 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-03-22T01:47:30.702 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-03-22T01:47:30.703 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-03-22T01:47:30.703 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:30.703 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_sessions": { 2024-03-22T01:47:30.703 INFO:teuthology.orchestra.run.smithi089.stdout: "session_count": 2, 2024-03-22T01:47:30.703 INFO:teuthology.orchestra.run.smithi089.stdout: "session_add": 6, 2024-03-22T01:47:30.703 INFO:teuthology.orchestra.run.smithi089.stdout: "session_remove": 4, 2024-03-22T01:47:30.704 INFO:teuthology.orchestra.run.smithi089.stdout: "sessions_open": 2, 2024-03-22T01:47:30.704 INFO:teuthology.orchestra.run.smithi089.stdout: "sessions_stale": 0, 2024-03-22T01:47:30.704 INFO:teuthology.orchestra.run.smithi089.stdout: "total_load": 270, 2024-03-22T01:47:30.704 INFO:teuthology.orchestra.run.smithi089.stdout: "average_load": 135, 2024-03-22T01:47:30.704 INFO:teuthology.orchestra.run.smithi089.stdout: "avg_session_uptime": 640, 2024-03-22T01:47:30.704 INFO:teuthology.orchestra.run.smithi089.stdout: "mdthresh_evicted": 0 2024-03-22T01:47:30.704 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:30.705 INFO:teuthology.orchestra.run.smithi089.stdout: "mempool": { 2024-03-22T01:47:30.705 INFO:teuthology.orchestra.run.smithi089.stdout: "bloom_filter_bytes": 0, 2024-03-22T01:47:30.705 INFO:teuthology.orchestra.run.smithi089.stdout: "bloom_filter_items": 0, 2024-03-22T01:47:30.705 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_alloc_bytes": 0, 2024-03-22T01:47:30.705 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_alloc_items": 0, 2024-03-22T01:47:30.705 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_data_bytes": 0, 2024-03-22T01:47:30.706 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_data_items": 0, 2024-03-22T01:47:30.706 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-22T01:47:30.706 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_onode_items": 0, 2024-03-22T01:47:30.706 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-22T01:47:30.706 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_meta_items": 0, 2024-03-22T01:47:30.706 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_other_bytes": 0, 2024-03-22T01:47:30.706 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_other_items": 0, 2024-03-22T01:47:30.707 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-22T01:47:30.707 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_buffer_items": 0, 2024-03-22T01:47:30.707 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_extent_bytes": 0, 2024-03-22T01:47:30.707 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_extent_items": 0, 2024-03-22T01:47:30.707 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_blob_bytes": 0, 2024-03-22T01:47:30.707 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_blob_items": 0, 2024-03-22T01:47:30.707 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-22T01:47:30.708 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_shared_blob_items": 0, 2024-03-22T01:47:30.708 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-22T01:47:30.708 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_inline_bl_items": 0, 2024-03-22T01:47:30.708 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_fsck_bytes": 0, 2024-03-22T01:47:30.708 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_fsck_items": 0, 2024-03-22T01:47:30.708 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_txc_bytes": 0, 2024-03-22T01:47:30.708 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_txc_items": 0, 2024-03-22T01:47:30.708 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-22T01:47:30.708 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_deferred_items": 0, 2024-03-22T01:47:30.708 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_bytes": 0, 2024-03-22T01:47:30.708 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_items": 0, 2024-03-22T01:47:30.708 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_bytes": 0, 2024-03-22T01:47:30.708 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_items": 0, 2024-03-22T01:47:30.708 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_reader_bytes": 0, 2024-03-22T01:47:30.709 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_reader_items": 0, 2024-03-22T01:47:30.709 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_writer_bytes": 0, 2024-03-22T01:47:30.709 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_writer_items": 0, 2024-03-22T01:47:30.709 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_anon_bytes": 983998, 2024-03-22T01:47:30.709 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_anon_items": 699, 2024-03-22T01:47:30.709 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_meta_bytes": 1144, 2024-03-22T01:47:30.709 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_meta_items": 13, 2024-03-22T01:47:30.709 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_bytes": 0, 2024-03-22T01:47:30.709 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_items": 0, 2024-03-22T01:47:30.709 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mapbl_bytes": 0, 2024-03-22T01:47:30.710 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mapbl_items": 0, 2024-03-22T01:47:30.710 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pglog_bytes": 0, 2024-03-22T01:47:30.710 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pglog_items": 0, 2024-03-22T01:47:30.710 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_bytes": 9464, 2024-03-22T01:47:30.710 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_items": 180, 2024-03-22T01:47:30.710 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_mapping_bytes": 0, 2024-03-22T01:47:30.710 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_mapping_items": 0, 2024-03-22T01:47:30.710 INFO:teuthology.orchestra.run.smithi089.stdout: "pgmap_bytes": 0, 2024-03-22T01:47:30.710 INFO:teuthology.orchestra.run.smithi089.stdout: "pgmap_items": 0, 2024-03-22T01:47:30.710 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_co_bytes": 1973520, 2024-03-22T01:47:30.710 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_co_items": 38577, 2024-03-22T01:47:30.711 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_1_bytes": 0, 2024-03-22T01:47:30.711 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_1_items": 0, 2024-03-22T01:47:30.711 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_2_bytes": 0, 2024-03-22T01:47:30.711 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_2_items": 0 2024-03-22T01:47:30.711 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:30.711 INFO:teuthology.orchestra.run.smithi089.stdout: "objecter": { 2024-03-22T01:47:30.711 INFO:teuthology.orchestra.run.smithi089.stdout: "op_active": 0, 2024-03-22T01:47:30.711 INFO:teuthology.orchestra.run.smithi089.stdout: "op_laggy": 0, 2024-03-22T01:47:30.711 INFO:teuthology.orchestra.run.smithi089.stdout: "op_send": 10533, 2024-03-22T01:47:30.711 INFO:teuthology.orchestra.run.smithi089.stdout: "op_send_bytes": 50094016, 2024-03-22T01:47:30.711 INFO:teuthology.orchestra.run.smithi089.stdout: "op_resend": 37, 2024-03-22T01:47:30.711 INFO:teuthology.orchestra.run.smithi089.stdout: "op_reply": 10533, 2024-03-22T01:47:30.711 INFO:teuthology.orchestra.run.smithi089.stdout: "op_latency": { 2024-03-22T01:47:30.711 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 10533, 2024-03-22T01:47:30.712 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 4307.410243060, 2024-03-22T01:47:30.712 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.408944293 2024-03-22T01:47:30.712 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:30.712 INFO:teuthology.orchestra.run.smithi089.stdout: "op_inflight": 0, 2024-03-22T01:47:30.712 INFO:teuthology.orchestra.run.smithi089.stdout: "oplen_avg": { 2024-03-22T01:47:30.712 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 10533, 2024-03-22T01:47:30.712 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 13479 2024-03-22T01:47:30.712 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:30.712 INFO:teuthology.orchestra.run.smithi089.stdout: "op": 10533, 2024-03-22T01:47:30.712 INFO:teuthology.orchestra.run.smithi089.stdout: "op_r": 1241, 2024-03-22T01:47:30.712 INFO:teuthology.orchestra.run.smithi089.stdout: "op_w": 9292, 2024-03-22T01:47:30.712 INFO:teuthology.orchestra.run.smithi089.stdout: "op_rmw": 0, 2024-03-22T01:47:30.712 INFO:teuthology.orchestra.run.smithi089.stdout: "op_pg": 0, 2024-03-22T01:47:30.712 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_stat": 262, 2024-03-22T01:47:30.712 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_create": 1225, 2024-03-22T01:47:30.713 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_read": 22, 2024-03-22T01:47:30.713 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_write": 2094, 2024-03-22T01:47:30.713 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_writefull": 99, 2024-03-22T01:47:30.713 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_writesame": 0, 2024-03-22T01:47:30.713 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_append": 0, 2024-03-22T01:47:30.713 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_zero": 0, 2024-03-22T01:47:30.713 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_truncate": 1, 2024-03-22T01:47:30.713 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_delete": 5259, 2024-03-22T01:47:30.713 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_mapext": 0, 2024-03-22T01:47:30.713 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_sparse_read": 0, 2024-03-22T01:47:30.714 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_clonerange": 0, 2024-03-22T01:47:30.714 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_getxattr": 1219, 2024-03-22T01:47:30.714 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_setxattr": 1955, 2024-03-22T01:47:30.714 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_cmpxattr": 0, 2024-03-22T01:47:30.714 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_rmxattr": 0, 2024-03-22T01:47:30.714 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_resetxattrs": 0, 2024-03-22T01:47:30.714 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_call": 0, 2024-03-22T01:47:30.714 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_watch": 0, 2024-03-22T01:47:30.714 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_notify": 0, 2024-03-22T01:47:30.714 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_src_cmpxattr": 0, 2024-03-22T01:47:30.714 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_pgls": 0, 2024-03-22T01:47:30.715 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_pgls_filter": 0, 2024-03-22T01:47:30.715 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_other": 6, 2024-03-22T01:47:30.715 INFO:teuthology.orchestra.run.smithi089.stdout: "linger_active": 0, 2024-03-22T01:47:30.715 INFO:teuthology.orchestra.run.smithi089.stdout: "linger_send": 0, 2024-03-22T01:47:30.715 INFO:teuthology.orchestra.run.smithi089.stdout: "linger_resend": 0, 2024-03-22T01:47:30.715 INFO:teuthology.orchestra.run.smithi089.stdout: "linger_ping": 0, 2024-03-22T01:47:30.715 INFO:teuthology.orchestra.run.smithi089.stdout: "poolop_active": 0, 2024-03-22T01:47:30.715 INFO:teuthology.orchestra.run.smithi089.stdout: "poolop_send": 0, 2024-03-22T01:47:30.716 INFO:teuthology.orchestra.run.smithi089.stdout: "poolop_resend": 0, 2024-03-22T01:47:30.716 INFO:teuthology.orchestra.run.smithi089.stdout: "poolstat_active": 0, 2024-03-22T01:47:30.716 INFO:teuthology.orchestra.run.smithi089.stdout: "poolstat_send": 0, 2024-03-22T01:47:30.716 INFO:teuthology.orchestra.run.smithi089.stdout: "poolstat_resend": 0, 2024-03-22T01:47:30.716 INFO:teuthology.orchestra.run.smithi089.stdout: "statfs_active": 0, 2024-03-22T01:47:30.716 INFO:teuthology.orchestra.run.smithi089.stdout: "statfs_send": 0, 2024-03-22T01:47:30.716 INFO:teuthology.orchestra.run.smithi089.stdout: "statfs_resend": 0, 2024-03-22T01:47:30.717 INFO:teuthology.orchestra.run.smithi089.stdout: "command_active": 0, 2024-03-22T01:47:30.717 INFO:teuthology.orchestra.run.smithi089.stdout: "command_send": 0, 2024-03-22T01:47:30.717 INFO:teuthology.orchestra.run.smithi089.stdout: "command_resend": 0, 2024-03-22T01:47:30.717 INFO:teuthology.orchestra.run.smithi089.stdout: "map_epoch": 96, 2024-03-22T01:47:30.717 INFO:teuthology.orchestra.run.smithi089.stdout: "map_full": 0, 2024-03-22T01:47:30.717 INFO:teuthology.orchestra.run.smithi089.stdout: "map_inc": 10, 2024-03-22T01:47:30.717 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_sessions": 12, 2024-03-22T01:47:30.718 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_session_open": 12, 2024-03-22T01:47:30.718 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_session_close": 0, 2024-03-22T01:47:30.718 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_laggy": 0, 2024-03-22T01:47:30.718 INFO:teuthology.orchestra.run.smithi089.stdout: "omap_wr": 1050, 2024-03-22T01:47:30.718 INFO:teuthology.orchestra.run.smithi089.stdout: "omap_rd": 0, 2024-03-22T01:47:30.718 INFO:teuthology.orchestra.run.smithi089.stdout: "omap_del": 287 2024-03-22T01:47:30.718 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:30.719 INFO:teuthology.orchestra.run.smithi089.stdout: "oft": { 2024-03-22T01:47:30.719 INFO:teuthology.orchestra.run.smithi089.stdout: "omap_total_objs": 1, 2024-03-22T01:47:30.719 INFO:teuthology.orchestra.run.smithi089.stdout: "omap_total_kv_pairs": 0, 2024-03-22T01:47:30.719 INFO:teuthology.orchestra.run.smithi089.stdout: "omap_total_updates": 0, 2024-03-22T01:47:30.719 INFO:teuthology.orchestra.run.smithi089.stdout: "omap_total_removes": 0 2024-03-22T01:47:30.719 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:30.720 INFO:teuthology.orchestra.run.smithi089.stdout: "osd-slow-ops": { 2024-03-22T01:47:30.720 INFO:teuthology.orchestra.run.smithi089.stdout: "slow_ops_count": 1 2024-03-22T01:47:30.720 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:30.720 INFO:teuthology.orchestra.run.smithi089.stdout: "purge_queue": { 2024-03-22T01:47:30.720 INFO:teuthology.orchestra.run.smithi089.stdout: "pq_executing_ops": 0, 2024-03-22T01:47:30.720 INFO:teuthology.orchestra.run.smithi089.stdout: "pq_executing_ops_high_water": 78, 2024-03-22T01:47:30.720 INFO:teuthology.orchestra.run.smithi089.stdout: "pq_executing": 0, 2024-03-22T01:47:30.721 INFO:teuthology.orchestra.run.smithi089.stdout: "pq_executing_high_water": 29, 2024-03-22T01:47:30.721 INFO:teuthology.orchestra.run.smithi089.stdout: "pq_executed_ops": 4725, 2024-03-22T01:47:30.721 INFO:teuthology.orchestra.run.smithi089.stdout: "pq_executed": 789, 2024-03-22T01:47:30.721 INFO:teuthology.orchestra.run.smithi089.stdout: "pq_item_in_journal": 0 2024-03-22T01:47:30.721 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:30.721 INFO:teuthology.orchestra.run.smithi089.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-22T01:47:30.721 INFO:teuthology.orchestra.run.smithi089.stdout: "val": 0, 2024-03-22T01:47:30.722 INFO:teuthology.orchestra.run.smithi089.stdout: "max": 104857600, 2024-03-22T01:47:30.722 INFO:teuthology.orchestra.run.smithi089.stdout: "get_started": 0, 2024-03-22T01:47:30.722 INFO:teuthology.orchestra.run.smithi089.stdout: "get": 34464, 2024-03-22T01:47:30.722 INFO:teuthology.orchestra.run.smithi089.stdout: "get_sum": 9578825, 2024-03-22T01:47:30.722 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_fail": 0, 2024-03-22T01:47:30.722 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_success": 34464, 2024-03-22T01:47:30.722 INFO:teuthology.orchestra.run.smithi089.stdout: "take": 0, 2024-03-22T01:47:30.722 INFO:teuthology.orchestra.run.smithi089.stdout: "take_sum": 0, 2024-03-22T01:47:30.723 INFO:teuthology.orchestra.run.smithi089.stdout: "put": 34464, 2024-03-22T01:47:30.723 INFO:teuthology.orchestra.run.smithi089.stdout: "put_sum": 9578825, 2024-03-22T01:47:30.723 INFO:teuthology.orchestra.run.smithi089.stdout: "wait": { 2024-03-22T01:47:30.723 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-03-22T01:47:30.723 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-03-22T01:47:30.723 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-03-22T01:47:30.723 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-03-22T01:47:30.723 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:30.724 INFO:teuthology.orchestra.run.smithi089.stdout: "throttle-objecter_bytes": { 2024-03-22T01:47:30.724 INFO:teuthology.orchestra.run.smithi089.stdout: "val": 0, 2024-03-22T01:47:30.724 INFO:teuthology.orchestra.run.smithi089.stdout: "max": 104857600, 2024-03-22T01:47:30.724 INFO:teuthology.orchestra.run.smithi089.stdout: "get_started": 0, 2024-03-22T01:47:30.724 INFO:teuthology.orchestra.run.smithi089.stdout: "get": 0, 2024-03-22T01:47:30.724 INFO:teuthology.orchestra.run.smithi089.stdout: "get_sum": 0, 2024-03-22T01:47:30.725 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_fail": 0, 2024-03-22T01:47:30.725 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_success": 0, 2024-03-22T01:47:30.725 INFO:teuthology.orchestra.run.smithi089.stdout: "take": 10533, 2024-03-22T01:47:30.725 INFO:teuthology.orchestra.run.smithi089.stdout: "take_sum": 50184905, 2024-03-22T01:47:30.725 INFO:teuthology.orchestra.run.smithi089.stdout: "put": 10533, 2024-03-22T01:47:30.725 INFO:teuthology.orchestra.run.smithi089.stdout: "put_sum": 50184905, 2024-03-22T01:47:30.725 INFO:teuthology.orchestra.run.smithi089.stdout: "wait": { 2024-03-22T01:47:30.726 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-03-22T01:47:30.726 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-03-22T01:47:30.726 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-03-22T01:47:30.726 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-03-22T01:47:30.726 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:30.726 INFO:teuthology.orchestra.run.smithi089.stdout: "throttle-objecter_ops": { 2024-03-22T01:47:30.727 INFO:teuthology.orchestra.run.smithi089.stdout: "val": 0, 2024-03-22T01:47:30.727 INFO:teuthology.orchestra.run.smithi089.stdout: "max": 1024, 2024-03-22T01:47:30.727 INFO:teuthology.orchestra.run.smithi089.stdout: "get_started": 0, 2024-03-22T01:47:30.727 INFO:teuthology.orchestra.run.smithi089.stdout: "get": 0, 2024-03-22T01:47:30.727 INFO:teuthology.orchestra.run.smithi089.stdout: "get_sum": 0, 2024-03-22T01:47:30.727 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_fail": 0, 2024-03-22T01:47:30.728 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_success": 0, 2024-03-22T01:47:30.728 INFO:teuthology.orchestra.run.smithi089.stdout: "take": 10533, 2024-03-22T01:47:30.728 INFO:teuthology.orchestra.run.smithi089.stdout: "take_sum": 10533, 2024-03-22T01:47:30.728 INFO:teuthology.orchestra.run.smithi089.stdout: "put": 10533, 2024-03-22T01:47:30.728 INFO:teuthology.orchestra.run.smithi089.stdout: "put_sum": 10533, 2024-03-22T01:47:30.728 INFO:teuthology.orchestra.run.smithi089.stdout: "wait": { 2024-03-22T01:47:30.728 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-03-22T01:47:30.728 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-03-22T01:47:30.728 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-03-22T01:47:30.729 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-03-22T01:47:30.729 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:30.729 INFO:teuthology.orchestra.run.smithi089.stdout: "throttle-write_buf_throttle": { 2024-03-22T01:47:30.729 INFO:teuthology.orchestra.run.smithi089.stdout: "val": 0, 2024-03-22T01:47:30.729 INFO:teuthology.orchestra.run.smithi089.stdout: "max": 3758096384, 2024-03-22T01:47:30.729 INFO:teuthology.orchestra.run.smithi089.stdout: "get_started": 0, 2024-03-22T01:47:30.729 INFO:teuthology.orchestra.run.smithi089.stdout: "get": 789, 2024-03-22T01:47:30.729 INFO:teuthology.orchestra.run.smithi089.stdout: "get_sum": 90889, 2024-03-22T01:47:30.729 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_fail": 0, 2024-03-22T01:47:30.729 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_success": 789, 2024-03-22T01:47:30.729 INFO:teuthology.orchestra.run.smithi089.stdout: "take": 0, 2024-03-22T01:47:30.729 INFO:teuthology.orchestra.run.smithi089.stdout: "take_sum": 0, 2024-03-22T01:47:30.730 INFO:teuthology.orchestra.run.smithi089.stdout: "put": 15, 2024-03-22T01:47:30.730 INFO:teuthology.orchestra.run.smithi089.stdout: "put_sum": 90889, 2024-03-22T01:47:30.730 INFO:teuthology.orchestra.run.smithi089.stdout: "wait": { 2024-03-22T01:47:30.730 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-03-22T01:47:30.730 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-03-22T01:47:30.730 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-03-22T01:47:30.730 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-03-22T01:47:30.730 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:30.730 INFO:teuthology.orchestra.run.smithi089.stdout: "throttle-write_buf_throttle-0x55b875910460": { 2024-03-22T01:47:30.730 INFO:teuthology.orchestra.run.smithi089.stdout: "val": 0, 2024-03-22T01:47:30.730 INFO:teuthology.orchestra.run.smithi089.stdout: "max": 3758096384, 2024-03-22T01:47:30.730 INFO:teuthology.orchestra.run.smithi089.stdout: "get_started": 0, 2024-03-22T01:47:30.731 INFO:teuthology.orchestra.run.smithi089.stdout: "get": 22002, 2024-03-22T01:47:30.731 INFO:teuthology.orchestra.run.smithi089.stdout: "get_sum": 48103290, 2024-03-22T01:47:30.731 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_fail": 0, 2024-03-22T01:47:30.731 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_success": 22002, 2024-03-22T01:47:30.731 INFO:teuthology.orchestra.run.smithi089.stdout: "take": 0, 2024-03-22T01:47:30.731 INFO:teuthology.orchestra.run.smithi089.stdout: "take_sum": 0, 2024-03-22T01:47:30.731 INFO:teuthology.orchestra.run.smithi089.stdout: "put": 2079, 2024-03-22T01:47:30.731 INFO:teuthology.orchestra.run.smithi089.stdout: "put_sum": 48103290, 2024-03-22T01:47:30.731 INFO:teuthology.orchestra.run.smithi089.stdout: "wait": { 2024-03-22T01:47:30.731 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-03-22T01:47:30.732 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-03-22T01:47:30.732 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-03-22T01:47:30.732 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-03-22T01:47:30.732 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-03-22T01:47:30.732 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-03-22T01:47:30.732 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:30 smithi098 ceph-mon[36468]: pgmap v328: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-22T01:47:32.101 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=5 2024-03-22T01:47:32.101 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=5 2024-03-22T01:47:32.101 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=0 2024-03-22T01:47:32.101 INFO:tasks.check_counter:Daemon mds.b mds.dir_split=1 2024-03-22T01:47:32.101 DEBUG:tasks.check_counter:Getting stats from e 2024-03-22T01:47:32.102 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph daemon mds.e perf dump 2024-03-22T01:47:32.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:32 smithi089 ceph-mon[35770]: pgmap v329: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-22T01:47:32.415 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.b/config 2024-03-22T01:47:32.473 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:32 smithi098 ceph-mon[36468]: pgmap v329: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-22T01:47:32.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:32 smithi072 ceph-mon[32414]: pgmap v329: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-22T01:47:34.277 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:34 smithi089 ceph-mon[35770]: pgmap v330: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 2 op/s 2024-03-22T01:47:34.590 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-03-22T01:47:34.590 INFO:teuthology.orchestra.run.smithi089.stdout: "AsyncMessenger::Worker-0": { 2024-03-22T01:47:34.590 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_messages": 4, 2024-03-22T01:47:34.590 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages": 4, 2024-03-22T01:47:34.590 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_bytes": 2992, 2024-03-22T01:47:34.590 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_bytes": 448, 2024-03-22T01:47:34.590 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_created_connections": 4, 2024-03-22T01:47:34.590 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_active_connections": 0, 2024-03-22T01:47:34.590 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_total_time": 0.001448855, 2024-03-22T01:47:34.591 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_send_time": 0.000032147, 2024-03-22T01:47:34.591 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_recv_time": 0.000639544, 2024-03-22T01:47:34.591 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-22T01:47:34.591 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T01:47:34.591 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 3, 2024-03-22T01:47:34.591 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000813083, 2024-03-22T01:47:34.591 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000271027 2024-03-22T01:47:34.591 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:34.591 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_handle_ack_lat": { 2024-03-22T01:47:34.591 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-03-22T01:47:34.591 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-03-22T01:47:34.591 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-03-22T01:47:34.591 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:34.591 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-22T01:47:34.591 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_encrypted_bytes": 448 2024-03-22T01:47:34.592 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:34.592 INFO:teuthology.orchestra.run.smithi089.stdout: "AsyncMessenger::Worker-1": { 2024-03-22T01:47:34.592 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_messages": 2, 2024-03-22T01:47:34.592 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages": 170, 2024-03-22T01:47:34.592 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_bytes": 320, 2024-03-22T01:47:34.592 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_bytes": 185520, 2024-03-22T01:47:34.592 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_created_connections": 5, 2024-03-22T01:47:34.592 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_active_connections": 3, 2024-03-22T01:47:34.592 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_total_time": 0.008341791, 2024-03-22T01:47:34.592 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_send_time": 0.004926604, 2024-03-22T01:47:34.592 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_recv_time": 0.001898537, 2024-03-22T01:47:34.592 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-22T01:47:34.592 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T01:47:34.592 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 167, 2024-03-22T01:47:34.593 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.010957582, 2024-03-22T01:47:34.593 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000065614 2024-03-22T01:47:34.593 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:34.593 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_handle_ack_lat": { 2024-03-22T01:47:34.593 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-03-22T01:47:34.593 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-03-22T01:47:34.593 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-03-22T01:47:34.593 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:34.593 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-22T01:47:34.593 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_encrypted_bytes": 185520 2024-03-22T01:47:34.593 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:34.593 INFO:teuthology.orchestra.run.smithi089.stdout: "AsyncMessenger::Worker-2": { 2024-03-22T01:47:34.593 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_messages": 232, 2024-03-22T01:47:34.593 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages": 222, 2024-03-22T01:47:34.594 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_bytes": 1446048, 2024-03-22T01:47:34.594 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_bytes": 136384, 2024-03-22T01:47:34.594 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_created_connections": 2, 2024-03-22T01:47:34.594 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_active_connections": 1, 2024-03-22T01:47:34.594 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_total_time": 0.034605489, 2024-03-22T01:47:34.594 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_send_time": 0.009082058, 2024-03-22T01:47:34.594 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_recv_time": 0.030491952, 2024-03-22T01:47:34.594 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_fast_dispatch_time": 0.004158856, 2024-03-22T01:47:34.594 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T01:47:34.594 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 221, 2024-03-22T01:47:34.594 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.006800359, 2024-03-22T01:47:34.594 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000030770 2024-03-22T01:47:34.594 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:34.594 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_handle_ack_lat": { 2024-03-22T01:47:34.595 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-03-22T01:47:34.595 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-03-22T01:47:34.595 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-03-22T01:47:34.595 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:34.595 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_encrypted_bytes": 1446048, 2024-03-22T01:47:34.595 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_encrypted_bytes": 136384 2024-03-22T01:47:34.595 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:34.595 INFO:teuthology.orchestra.run.smithi089.stdout: "cct": { 2024-03-22T01:47:34.595 INFO:teuthology.orchestra.run.smithi089.stdout: "total_workers": 0, 2024-03-22T01:47:34.595 INFO:teuthology.orchestra.run.smithi089.stdout: "unhealthy_workers": 0 2024-03-22T01:47:34.595 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:34.595 INFO:teuthology.orchestra.run.smithi089.stdout: "mempool": { 2024-03-22T01:47:34.595 INFO:teuthology.orchestra.run.smithi089.stdout: "bloom_filter_bytes": 0, 2024-03-22T01:47:34.595 INFO:teuthology.orchestra.run.smithi089.stdout: "bloom_filter_items": 0, 2024-03-22T01:47:34.595 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_alloc_bytes": 0, 2024-03-22T01:47:34.596 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_alloc_items": 0, 2024-03-22T01:47:34.596 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_data_bytes": 0, 2024-03-22T01:47:34.596 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_data_items": 0, 2024-03-22T01:47:34.596 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-22T01:47:34.596 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_onode_items": 0, 2024-03-22T01:47:34.596 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-22T01:47:34.596 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_meta_items": 0, 2024-03-22T01:47:34.596 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_other_bytes": 0, 2024-03-22T01:47:34.596 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_other_items": 0, 2024-03-22T01:47:34.596 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-22T01:47:34.596 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_buffer_items": 0, 2024-03-22T01:47:34.596 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_extent_bytes": 0, 2024-03-22T01:47:34.596 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_extent_items": 0, 2024-03-22T01:47:34.596 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_blob_bytes": 0, 2024-03-22T01:47:34.596 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_blob_items": 0, 2024-03-22T01:47:34.596 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-22T01:47:34.597 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_shared_blob_items": 0, 2024-03-22T01:47:34.597 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-22T01:47:34.597 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_inline_bl_items": 0, 2024-03-22T01:47:34.597 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_fsck_bytes": 0, 2024-03-22T01:47:34.597 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_fsck_items": 0, 2024-03-22T01:47:34.597 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_txc_bytes": 0, 2024-03-22T01:47:34.597 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_txc_items": 0, 2024-03-22T01:47:34.597 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-22T01:47:34.597 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_deferred_items": 0, 2024-03-22T01:47:34.597 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_bytes": 0, 2024-03-22T01:47:34.597 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_items": 0, 2024-03-22T01:47:34.597 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_bytes": 0, 2024-03-22T01:47:34.597 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_items": 0, 2024-03-22T01:47:34.597 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_reader_bytes": 0, 2024-03-22T01:47:34.597 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_reader_items": 0, 2024-03-22T01:47:34.598 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_writer_bytes": 0, 2024-03-22T01:47:34.598 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_writer_items": 0, 2024-03-22T01:47:34.598 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_anon_bytes": 149017, 2024-03-22T01:47:34.598 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_anon_items": 31, 2024-03-22T01:47:34.598 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_meta_bytes": 0, 2024-03-22T01:47:34.598 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_meta_items": 0, 2024-03-22T01:47:34.598 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_bytes": 0, 2024-03-22T01:47:34.598 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_items": 0, 2024-03-22T01:47:34.598 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mapbl_bytes": 0, 2024-03-22T01:47:34.598 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mapbl_items": 0, 2024-03-22T01:47:34.598 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pglog_bytes": 0, 2024-03-22T01:47:34.598 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pglog_items": 0, 2024-03-22T01:47:34.598 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_bytes": 0, 2024-03-22T01:47:34.598 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_items": 0, 2024-03-22T01:47:34.598 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_mapping_bytes": 0, 2024-03-22T01:47:34.599 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_mapping_items": 0, 2024-03-22T01:47:34.599 INFO:teuthology.orchestra.run.smithi089.stdout: "pgmap_bytes": 0, 2024-03-22T01:47:34.599 INFO:teuthology.orchestra.run.smithi089.stdout: "pgmap_items": 0, 2024-03-22T01:47:34.599 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_co_bytes": 928, 2024-03-22T01:47:34.599 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_co_items": 2, 2024-03-22T01:47:34.599 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_1_bytes": 0, 2024-03-22T01:47:34.599 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_1_items": 0, 2024-03-22T01:47:34.599 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_2_bytes": 0, 2024-03-22T01:47:34.599 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_2_items": 0 2024-03-22T01:47:34.599 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:34.599 INFO:teuthology.orchestra.run.smithi089.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-22T01:47:34.599 INFO:teuthology.orchestra.run.smithi089.stdout: "val": 0, 2024-03-22T01:47:34.599 INFO:teuthology.orchestra.run.smithi089.stdout: "max": 104857600, 2024-03-22T01:47:34.599 INFO:teuthology.orchestra.run.smithi089.stdout: "get_started": 0, 2024-03-22T01:47:34.599 INFO:teuthology.orchestra.run.smithi089.stdout: "get": 234, 2024-03-22T01:47:34.599 INFO:teuthology.orchestra.run.smithi089.stdout: "get_sum": 1413310, 2024-03-22T01:47:34.600 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_fail": 0, 2024-03-22T01:47:34.600 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_success": 234, 2024-03-22T01:47:34.600 INFO:teuthology.orchestra.run.smithi089.stdout: "take": 0, 2024-03-22T01:47:34.600 INFO:teuthology.orchestra.run.smithi089.stdout: "take_sum": 0, 2024-03-22T01:47:34.600 INFO:teuthology.orchestra.run.smithi089.stdout: "put": 234, 2024-03-22T01:47:34.600 INFO:teuthology.orchestra.run.smithi089.stdout: "put_sum": 1413310, 2024-03-22T01:47:34.600 INFO:teuthology.orchestra.run.smithi089.stdout: "wait": { 2024-03-22T01:47:34.600 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-03-22T01:47:34.600 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-03-22T01:47:34.600 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-03-22T01:47:34.600 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-03-22T01:47:34.601 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-03-22T01:47:34.601 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-03-22T01:47:34.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:34 smithi072 ceph-mon[32414]: pgmap v330: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 2 op/s 2024-03-22T01:47:34.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:34 smithi098 ceph-mon[36468]: pgmap v330: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 2 op/s 2024-03-22T01:47:35.358 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:35 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:47:35.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:35 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:47:35.643 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.e 2024-03-22T01:47:35.643 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.e 2024-03-22T01:47:35.643 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.e 2024-03-22T01:47:35.643 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.e 2024-03-22T01:47:35.643 DEBUG:tasks.check_counter:Getting stats from h 2024-03-22T01:47:35.643 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph daemon mds.h perf dump 2024-03-22T01:47:35.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:35 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T01:47:35.946 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.b/config 2024-03-22T01:47:36.324 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:36 smithi089 ceph-mon[35770]: pgmap v331: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-22T01:47:36.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:36 smithi072 ceph-mon[32414]: pgmap v331: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-22T01:47:36.431 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:36 smithi098 ceph-mon[36468]: pgmap v331: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-22T01:47:37.916 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-03-22T01:47:37.916 INFO:teuthology.orchestra.run.smithi089.stdout: "AsyncMessenger::Worker-0": { 2024-03-22T01:47:37.916 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_messages": 4, 2024-03-22T01:47:37.916 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages": 4, 2024-03-22T01:47:37.916 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_bytes": 2992, 2024-03-22T01:47:37.917 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_bytes": 448, 2024-03-22T01:47:37.917 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_created_connections": 4, 2024-03-22T01:47:37.917 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_active_connections": 0, 2024-03-22T01:47:37.917 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_total_time": 0.001390533, 2024-03-22T01:47:37.917 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_send_time": 0.000032957, 2024-03-22T01:47:37.917 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_recv_time": 0.000535050, 2024-03-22T01:47:37.917 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-22T01:47:37.917 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T01:47:37.917 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 3, 2024-03-22T01:47:37.917 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000671644, 2024-03-22T01:47:37.917 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000223881 2024-03-22T01:47:37.917 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:37.918 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_handle_ack_lat": { 2024-03-22T01:47:37.918 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-03-22T01:47:37.918 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-03-22T01:47:37.918 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-03-22T01:47:37.918 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:37.918 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-22T01:47:37.918 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_encrypted_bytes": 448 2024-03-22T01:47:37.918 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:37.918 INFO:teuthology.orchestra.run.smithi089.stdout: "AsyncMessenger::Worker-1": { 2024-03-22T01:47:37.918 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_messages": 2, 2024-03-22T01:47:37.918 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages": 168, 2024-03-22T01:47:37.918 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_bytes": 320, 2024-03-22T01:47:37.919 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_bytes": 185168, 2024-03-22T01:47:37.919 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_created_connections": 5, 2024-03-22T01:47:37.919 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_active_connections": 3, 2024-03-22T01:47:37.919 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_total_time": 0.007862635, 2024-03-22T01:47:37.919 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_send_time": 0.004602512, 2024-03-22T01:47:37.919 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_recv_time": 0.001647361, 2024-03-22T01:47:37.919 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-22T01:47:37.919 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T01:47:37.919 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 165, 2024-03-22T01:47:37.919 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.014083490, 2024-03-22T01:47:37.919 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000085354 2024-03-22T01:47:37.919 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:37.920 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_handle_ack_lat": { 2024-03-22T01:47:37.920 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-03-22T01:47:37.920 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-03-22T01:47:37.920 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-03-22T01:47:37.920 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:37.920 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-22T01:47:37.920 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_encrypted_bytes": 185168 2024-03-22T01:47:37.920 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:37.920 INFO:teuthology.orchestra.run.smithi089.stdout: "AsyncMessenger::Worker-2": { 2024-03-22T01:47:37.920 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_messages": 231, 2024-03-22T01:47:37.920 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages": 221, 2024-03-22T01:47:37.920 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_bytes": 1445776, 2024-03-22T01:47:37.920 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_bytes": 135744, 2024-03-22T01:47:37.921 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_created_connections": 2, 2024-03-22T01:47:37.921 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_active_connections": 1, 2024-03-22T01:47:37.921 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_total_time": 0.035381952, 2024-03-22T01:47:37.921 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_send_time": 0.008965886, 2024-03-22T01:47:37.921 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_recv_time": 0.031430810, 2024-03-22T01:47:37.921 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_fast_dispatch_time": 0.003994960, 2024-03-22T01:47:37.921 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T01:47:37.921 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 220, 2024-03-22T01:47:37.921 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.008967516, 2024-03-22T01:47:37.921 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000040761 2024-03-22T01:47:37.921 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:37.921 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_handle_ack_lat": { 2024-03-22T01:47:37.922 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-03-22T01:47:37.922 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-03-22T01:47:37.922 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-03-22T01:47:37.922 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:37.922 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_encrypted_bytes": 1445776, 2024-03-22T01:47:37.922 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_encrypted_bytes": 135744 2024-03-22T01:47:37.922 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:37.922 INFO:teuthology.orchestra.run.smithi089.stdout: "cct": { 2024-03-22T01:47:37.922 INFO:teuthology.orchestra.run.smithi089.stdout: "total_workers": 0, 2024-03-22T01:47:37.922 INFO:teuthology.orchestra.run.smithi089.stdout: "unhealthy_workers": 0 2024-03-22T01:47:37.922 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:37.922 INFO:teuthology.orchestra.run.smithi089.stdout: "mempool": { 2024-03-22T01:47:37.923 INFO:teuthology.orchestra.run.smithi089.stdout: "bloom_filter_bytes": 0, 2024-03-22T01:47:37.923 INFO:teuthology.orchestra.run.smithi089.stdout: "bloom_filter_items": 0, 2024-03-22T01:47:37.923 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_alloc_bytes": 0, 2024-03-22T01:47:37.923 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_alloc_items": 0, 2024-03-22T01:47:37.923 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_data_bytes": 0, 2024-03-22T01:47:37.923 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_data_items": 0, 2024-03-22T01:47:37.923 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-22T01:47:37.923 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_onode_items": 0, 2024-03-22T01:47:37.923 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-22T01:47:37.923 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_meta_items": 0, 2024-03-22T01:47:37.923 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_other_bytes": 0, 2024-03-22T01:47:37.923 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_other_items": 0, 2024-03-22T01:47:37.923 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-22T01:47:37.924 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_buffer_items": 0, 2024-03-22T01:47:37.924 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_extent_bytes": 0, 2024-03-22T01:47:37.924 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_extent_items": 0, 2024-03-22T01:47:37.924 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_blob_bytes": 0, 2024-03-22T01:47:37.924 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_blob_items": 0, 2024-03-22T01:47:37.924 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-22T01:47:37.924 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_shared_blob_items": 0, 2024-03-22T01:47:37.924 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-22T01:47:37.924 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_inline_bl_items": 0, 2024-03-22T01:47:37.924 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_fsck_bytes": 0, 2024-03-22T01:47:37.924 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_fsck_items": 0, 2024-03-22T01:47:37.924 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_txc_bytes": 0, 2024-03-22T01:47:37.924 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_txc_items": 0, 2024-03-22T01:47:37.925 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-22T01:47:37.925 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_deferred_items": 0, 2024-03-22T01:47:37.925 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_bytes": 0, 2024-03-22T01:47:37.925 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_items": 0, 2024-03-22T01:47:37.925 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_bytes": 0, 2024-03-22T01:47:37.925 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_items": 0, 2024-03-22T01:47:37.925 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_reader_bytes": 0, 2024-03-22T01:47:37.925 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_reader_items": 0, 2024-03-22T01:47:37.925 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_writer_bytes": 0, 2024-03-22T01:47:37.925 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_writer_items": 0, 2024-03-22T01:47:37.925 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_anon_bytes": 148977, 2024-03-22T01:47:37.925 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_anon_items": 30, 2024-03-22T01:47:37.926 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_meta_bytes": 0, 2024-03-22T01:47:37.926 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_meta_items": 0, 2024-03-22T01:47:37.926 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_bytes": 0, 2024-03-22T01:47:37.926 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_items": 0, 2024-03-22T01:47:37.926 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mapbl_bytes": 0, 2024-03-22T01:47:37.926 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mapbl_items": 0, 2024-03-22T01:47:37.926 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pglog_bytes": 0, 2024-03-22T01:47:37.926 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pglog_items": 0, 2024-03-22T01:47:37.926 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_bytes": 0, 2024-03-22T01:47:37.926 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_items": 0, 2024-03-22T01:47:37.926 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_mapping_bytes": 0, 2024-03-22T01:47:37.926 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_mapping_items": 0, 2024-03-22T01:47:37.927 INFO:teuthology.orchestra.run.smithi089.stdout: "pgmap_bytes": 0, 2024-03-22T01:47:37.927 INFO:teuthology.orchestra.run.smithi089.stdout: "pgmap_items": 0, 2024-03-22T01:47:37.927 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_co_bytes": 928, 2024-03-22T01:47:37.927 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_co_items": 2, 2024-03-22T01:47:37.927 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_1_bytes": 0, 2024-03-22T01:47:37.927 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_1_items": 0, 2024-03-22T01:47:37.927 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_2_bytes": 0, 2024-03-22T01:47:37.927 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_2_items": 0 2024-03-22T01:47:37.927 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:37.927 INFO:teuthology.orchestra.run.smithi089.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-22T01:47:37.927 INFO:teuthology.orchestra.run.smithi089.stdout: "val": 0, 2024-03-22T01:47:37.927 INFO:teuthology.orchestra.run.smithi089.stdout: "max": 104857600, 2024-03-22T01:47:37.927 INFO:teuthology.orchestra.run.smithi089.stdout: "get_started": 0, 2024-03-22T01:47:37.928 INFO:teuthology.orchestra.run.smithi089.stdout: "get": 233, 2024-03-22T01:47:37.928 INFO:teuthology.orchestra.run.smithi089.stdout: "get_sum": 1413180, 2024-03-22T01:47:37.928 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_fail": 0, 2024-03-22T01:47:37.928 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_success": 233, 2024-03-22T01:47:37.928 INFO:teuthology.orchestra.run.smithi089.stdout: "take": 0, 2024-03-22T01:47:37.928 INFO:teuthology.orchestra.run.smithi089.stdout: "take_sum": 0, 2024-03-22T01:47:37.928 INFO:teuthology.orchestra.run.smithi089.stdout: "put": 233, 2024-03-22T01:47:37.928 INFO:teuthology.orchestra.run.smithi089.stdout: "put_sum": 1413180, 2024-03-22T01:47:37.928 INFO:teuthology.orchestra.run.smithi089.stdout: "wait": { 2024-03-22T01:47:37.928 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-03-22T01:47:37.928 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-03-22T01:47:37.928 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-03-22T01:47:37.928 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-03-22T01:47:37.929 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-03-22T01:47:37.929 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-03-22T01:47:38.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:37 smithi089 ceph-mon[35770]: pgmap v332: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-22T01:47:38.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:37 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:47:38.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:37 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:47:38.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:37 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:47:38.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:37 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:47:38.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:37 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:47:38.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:37 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:47:38.109 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:37 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:47:38.109 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:37 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:47:38.109 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:37 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:47:38.109 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:37 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:47:38.109 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:37 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:47:38.109 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:37 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:47:38.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:37 smithi072 ceph-mon[32414]: pgmap v332: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-22T01:47:38.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:37 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:47:38.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:37 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:47:38.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:37 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:47:38.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:37 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:47:38.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:37 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:47:38.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:37 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:47:38.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:37 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:47:38.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:37 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:47:38.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:37 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:47:38.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:37 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:47:38.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:37 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:47:38.123 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:37 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:47:38.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:37 smithi098 ceph-mon[36468]: pgmap v332: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-22T01:47:38.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:37 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:47:38.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:37 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:47:38.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:37 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:47:38.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:37 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:47:38.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:37 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:47:38.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:37 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:47:38.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:37 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:47:38.173 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:37 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:47:38.173 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:37 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:47:38.173 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:37 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:47:38.173 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:37 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:47:38.173 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:37 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:47:38.780 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.h 2024-03-22T01:47:38.780 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.h 2024-03-22T01:47:38.780 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.h 2024-03-22T01:47:38.780 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.h 2024-03-22T01:47:38.780 DEBUG:tasks.check_counter:Getting stats from k 2024-03-22T01:47:38.781 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph daemon mds.k perf dump 2024-03-22T01:47:39.027 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:38 smithi089 ceph-mon[35770]: Adjusting osd_memory_target on smithi072 to 176.1M 2024-03-22T01:47:39.027 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:38 smithi089 ceph-mon[35770]: Unable to set osd_memory_target on smithi072 to 184725811: error parsing value: Value '184725811' is below minimum 939524096 2024-03-22T01:47:39.027 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:38 smithi089 ceph-mon[35770]: Adjusting osd_memory_target on smithi098 to 174.1M 2024-03-22T01:47:39.027 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:38 smithi089 ceph-mon[35770]: Unable to set osd_memory_target on smithi098 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-22T01:47:39.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:38 smithi072 ceph-mon[32414]: Adjusting osd_memory_target on smithi072 to 176.1M 2024-03-22T01:47:39.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:38 smithi072 ceph-mon[32414]: Unable to set osd_memory_target on smithi072 to 184725811: error parsing value: Value '184725811' is below minimum 939524096 2024-03-22T01:47:39.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:38 smithi072 ceph-mon[32414]: Adjusting osd_memory_target on smithi098 to 174.1M 2024-03-22T01:47:39.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:38 smithi072 ceph-mon[32414]: Unable to set osd_memory_target on smithi098 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-22T01:47:39.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:38 smithi098 ceph-mon[36468]: Adjusting osd_memory_target on smithi072 to 176.1M 2024-03-22T01:47:39.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:38 smithi098 ceph-mon[36468]: Unable to set osd_memory_target on smithi072 to 184725811: error parsing value: Value '184725811' is below minimum 939524096 2024-03-22T01:47:39.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:38 smithi098 ceph-mon[36468]: Adjusting osd_memory_target on smithi098 to 174.1M 2024-03-22T01:47:39.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:38 smithi098 ceph-mon[36468]: Unable to set osd_memory_target on smithi098 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-22T01:47:39.715 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.b/config 2024-03-22T01:47:40.106 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:39 smithi089 ceph-mon[35770]: pgmap v333: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-22T01:47:40.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:39 smithi072 ceph-mon[32414]: pgmap v333: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-22T01:47:40.171 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:39 smithi098 ceph-mon[36468]: pgmap v333: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-22T01:47:41.448 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-03-22T01:47:41.448 INFO:teuthology.orchestra.run.smithi089.stdout: "AsyncMessenger::Worker-0": { 2024-03-22T01:47:41.448 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_messages": 0, 2024-03-22T01:47:41.448 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages": 2, 2024-03-22T01:47:41.449 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_bytes": 0, 2024-03-22T01:47:41.449 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_bytes": 0, 2024-03-22T01:47:41.449 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_created_connections": 4, 2024-03-22T01:47:41.449 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_active_connections": 0, 2024-03-22T01:47:41.449 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_total_time": 0.000957025, 2024-03-22T01:47:41.449 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_send_time": 0.000000000, 2024-03-22T01:47:41.449 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_recv_time": 0.000142072, 2024-03-22T01:47:41.449 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-22T01:47:41.449 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T01:47:41.449 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-03-22T01:47:41.449 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-03-22T01:47:41.449 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-03-22T01:47:41.449 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:41.449 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_handle_ack_lat": { 2024-03-22T01:47:41.450 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-03-22T01:47:41.450 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-03-22T01:47:41.450 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-03-22T01:47:41.450 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:41.450 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-22T01:47:41.450 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_encrypted_bytes": 0 2024-03-22T01:47:41.450 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:41.450 INFO:teuthology.orchestra.run.smithi089.stdout: "AsyncMessenger::Worker-1": { 2024-03-22T01:47:41.450 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_messages": 236, 2024-03-22T01:47:41.450 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages": 226, 2024-03-22T01:47:41.450 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_bytes": 1447136, 2024-03-22T01:47:41.450 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_bytes": 138944, 2024-03-22T01:47:41.450 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_created_connections": 2, 2024-03-22T01:47:41.450 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_active_connections": 1, 2024-03-22T01:47:41.451 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_total_time": 0.035746135, 2024-03-22T01:47:41.451 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_send_time": 0.009378552, 2024-03-22T01:47:41.451 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_recv_time": 0.031821056, 2024-03-22T01:47:41.451 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_fast_dispatch_time": 0.004262310, 2024-03-22T01:47:41.451 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T01:47:41.451 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 225, 2024-03-22T01:47:41.451 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.008234348, 2024-03-22T01:47:41.451 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000036597 2024-03-22T01:47:41.451 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:41.451 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_handle_ack_lat": { 2024-03-22T01:47:41.451 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-03-22T01:47:41.451 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-03-22T01:47:41.451 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-03-22T01:47:41.451 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:41.451 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_encrypted_bytes": 1447136, 2024-03-22T01:47:41.452 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_encrypted_bytes": 138944 2024-03-22T01:47:41.452 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:41.452 INFO:teuthology.orchestra.run.smithi089.stdout: "AsyncMessenger::Worker-2": { 2024-03-22T01:47:41.452 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_messages": 6, 2024-03-22T01:47:41.452 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages": 174, 2024-03-22T01:47:41.452 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_bytes": 3312, 2024-03-22T01:47:41.452 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_bytes": 186320, 2024-03-22T01:47:41.452 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_created_connections": 5, 2024-03-22T01:47:41.452 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_active_connections": 3, 2024-03-22T01:47:41.452 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_total_time": 0.010149040, 2024-03-22T01:47:41.452 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_send_time": 0.004959932, 2024-03-22T01:47:41.452 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_recv_time": 0.003642373, 2024-03-22T01:47:41.452 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-22T01:47:41.452 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T01:47:41.452 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 172, 2024-03-22T01:47:41.453 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.014532150, 2024-03-22T01:47:41.453 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000084489 2024-03-22T01:47:41.453 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:41.453 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_handle_ack_lat": { 2024-03-22T01:47:41.453 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-03-22T01:47:41.453 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-03-22T01:47:41.453 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-03-22T01:47:41.453 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:41.453 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-03-22T01:47:41.453 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_encrypted_bytes": 186320 2024-03-22T01:47:41.453 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:41.453 INFO:teuthology.orchestra.run.smithi089.stdout: "cct": { 2024-03-22T01:47:41.453 INFO:teuthology.orchestra.run.smithi089.stdout: "total_workers": 0, 2024-03-22T01:47:41.453 INFO:teuthology.orchestra.run.smithi089.stdout: "unhealthy_workers": 0 2024-03-22T01:47:41.454 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:41.454 INFO:teuthology.orchestra.run.smithi089.stdout: "mempool": { 2024-03-22T01:47:41.454 INFO:teuthology.orchestra.run.smithi089.stdout: "bloom_filter_bytes": 0, 2024-03-22T01:47:41.454 INFO:teuthology.orchestra.run.smithi089.stdout: "bloom_filter_items": 0, 2024-03-22T01:47:41.454 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_alloc_bytes": 0, 2024-03-22T01:47:41.454 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_alloc_items": 0, 2024-03-22T01:47:41.454 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_data_bytes": 0, 2024-03-22T01:47:41.454 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_data_items": 0, 2024-03-22T01:47:41.454 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-22T01:47:41.454 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_onode_items": 0, 2024-03-22T01:47:41.454 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-22T01:47:41.454 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_meta_items": 0, 2024-03-22T01:47:41.454 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_other_bytes": 0, 2024-03-22T01:47:41.454 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_other_items": 0, 2024-03-22T01:47:41.455 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-22T01:47:41.455 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_buffer_items": 0, 2024-03-22T01:47:41.455 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_extent_bytes": 0, 2024-03-22T01:47:41.455 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_extent_items": 0, 2024-03-22T01:47:41.455 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_blob_bytes": 0, 2024-03-22T01:47:41.455 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_blob_items": 0, 2024-03-22T01:47:41.455 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-22T01:47:41.455 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_shared_blob_items": 0, 2024-03-22T01:47:41.455 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-22T01:47:41.455 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_inline_bl_items": 0, 2024-03-22T01:47:41.455 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_fsck_bytes": 0, 2024-03-22T01:47:41.455 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_fsck_items": 0, 2024-03-22T01:47:41.455 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_txc_bytes": 0, 2024-03-22T01:47:41.455 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_txc_items": 0, 2024-03-22T01:47:41.456 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-22T01:47:41.456 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_deferred_items": 0, 2024-03-22T01:47:41.456 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_bytes": 0, 2024-03-22T01:47:41.456 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_items": 0, 2024-03-22T01:47:41.456 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_bytes": 0, 2024-03-22T01:47:41.456 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_items": 0, 2024-03-22T01:47:41.456 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_reader_bytes": 0, 2024-03-22T01:47:41.456 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_reader_items": 0, 2024-03-22T01:47:41.456 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_writer_bytes": 0, 2024-03-22T01:47:41.456 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_writer_items": 0, 2024-03-22T01:47:41.456 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_anon_bytes": 148977, 2024-03-22T01:47:41.456 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_anon_items": 30, 2024-03-22T01:47:41.456 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_meta_bytes": 0, 2024-03-22T01:47:41.456 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_meta_items": 0, 2024-03-22T01:47:41.456 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_bytes": 0, 2024-03-22T01:47:41.457 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_items": 0, 2024-03-22T01:47:41.457 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mapbl_bytes": 0, 2024-03-22T01:47:41.457 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mapbl_items": 0, 2024-03-22T01:47:41.457 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pglog_bytes": 0, 2024-03-22T01:47:41.457 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pglog_items": 0, 2024-03-22T01:47:41.457 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_bytes": 0, 2024-03-22T01:47:41.457 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_items": 0, 2024-03-22T01:47:41.457 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_mapping_bytes": 0, 2024-03-22T01:47:41.457 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_mapping_items": 0, 2024-03-22T01:47:41.457 INFO:teuthology.orchestra.run.smithi089.stdout: "pgmap_bytes": 0, 2024-03-22T01:47:41.457 INFO:teuthology.orchestra.run.smithi089.stdout: "pgmap_items": 0, 2024-03-22T01:47:41.457 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_co_bytes": 928, 2024-03-22T01:47:41.457 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_co_items": 2, 2024-03-22T01:47:41.457 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_1_bytes": 0, 2024-03-22T01:47:41.458 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_1_items": 0, 2024-03-22T01:47:41.458 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_2_bytes": 0, 2024-03-22T01:47:41.458 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_2_items": 0 2024-03-22T01:47:41.458 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-03-22T01:47:41.458 INFO:teuthology.orchestra.run.smithi089.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-22T01:47:41.458 INFO:teuthology.orchestra.run.smithi089.stdout: "val": 0, 2024-03-22T01:47:41.458 INFO:teuthology.orchestra.run.smithi089.stdout: "max": 104857600, 2024-03-22T01:47:41.458 INFO:teuthology.orchestra.run.smithi089.stdout: "get_started": 0, 2024-03-22T01:47:41.458 INFO:teuthology.orchestra.run.smithi089.stdout: "get": 238, 2024-03-22T01:47:41.458 INFO:teuthology.orchestra.run.smithi089.stdout: "get_sum": 1413830, 2024-03-22T01:47:41.458 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_fail": 0, 2024-03-22T01:47:41.458 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_success": 238, 2024-03-22T01:47:41.458 INFO:teuthology.orchestra.run.smithi089.stdout: "take": 0, 2024-03-22T01:47:41.458 INFO:teuthology.orchestra.run.smithi089.stdout: "take_sum": 0, 2024-03-22T01:47:41.459 INFO:teuthology.orchestra.run.smithi089.stdout: "put": 238, 2024-03-22T01:47:41.459 INFO:teuthology.orchestra.run.smithi089.stdout: "put_sum": 1413830, 2024-03-22T01:47:41.459 INFO:teuthology.orchestra.run.smithi089.stdout: "wait": { 2024-03-22T01:47:41.459 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-03-22T01:47:41.459 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-03-22T01:47:41.459 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-03-22T01:47:41.459 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-03-22T01:47:41.459 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-03-22T01:47:41.459 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-03-22T01:47:42.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:41 smithi089 ceph-mon[35770]: pgmap v334: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-22T01:47:42.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:41 smithi072 ceph-mon[32414]: pgmap v334: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-22T01:47:42.168 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.k 2024-03-22T01:47:42.168 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.k 2024-03-22T01:47:42.168 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.k 2024-03-22T01:47:42.168 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.k 2024-03-22T01:47:42.168 DEBUG:tasks.check_counter:Getting stats from c 2024-03-22T01:47:42.169 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph daemon mds.c perf dump 2024-03-22T01:47:42.172 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:41 smithi098 ceph-mon[36468]: pgmap v334: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-22T01:47:42.493 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.c/config 2024-03-22T01:47:43.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:43 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:47:43.609 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:43 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:47:43.609 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:43 smithi089 ceph-mon[35770]: pgmap v335: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 2 op/s 2024-03-22T01:47:43.609 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:43 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:47:43.609 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:43 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:47:43.609 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:43 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:47:43.609 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:43 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:47:43.609 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:43 smithi089 ceph-mon[35770]: Adjusting osd_memory_target on smithi089 to 174.1M 2024-03-22T01:47:43.609 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:43 smithi089 ceph-mon[35770]: Unable to set osd_memory_target on smithi089 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-22T01:47:43.610 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:43 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:47:43.610 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:43 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:47:43.610 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:43 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:47:43.610 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:43 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:47:43.610 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:43 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:47:43.610 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:43 smithi098 ceph-mon[36468]: pgmap v335: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 2 op/s 2024-03-22T01:47:43.610 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:43 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:47:43.611 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:43 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:47:43.611 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:43 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:47:43.611 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:43 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:47:43.611 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:43 smithi098 ceph-mon[36468]: Adjusting osd_memory_target on smithi089 to 174.1M 2024-03-22T01:47:43.611 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:43 smithi098 ceph-mon[36468]: Unable to set osd_memory_target on smithi089 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-22T01:47:43.611 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:43 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:47:43.611 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:43 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:47:43.611 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:43 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:47:43.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:43 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:47:43.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:43 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:47:43.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:43 smithi072 ceph-mon[32414]: pgmap v335: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 2 op/s 2024-03-22T01:47:43.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:43 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:47:43.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:43 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:47:43.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:43 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:47:43.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:43 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-22T01:47:43.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:43 smithi072 ceph-mon[32414]: Adjusting osd_memory_target on smithi089 to 174.1M 2024-03-22T01:47:43.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:43 smithi072 ceph-mon[32414]: Unable to set osd_memory_target on smithi089 to 182574694: error parsing value: Value '182574694' is below minimum 939524096 2024-03-22T01:47:43.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:43 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T01:47:43.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:43 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T01:47:43.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:43 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' 2024-03-22T01:47:44.162 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-22T01:47:44.162 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-0": { 2024-03-22T01:47:44.162 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 0, 2024-03-22T01:47:44.162 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 2, 2024-03-22T01:47:44.162 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 0, 2024-03-22T01:47:44.162 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 0, 2024-03-22T01:47:44.162 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 4, 2024-03-22T01:47:44.162 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 0, 2024-03-22T01:47:44.162 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.001149648, 2024-03-22T01:47:44.163 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.000000000, 2024-03-22T01:47:44.163 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.000298247, 2024-03-22T01:47:44.163 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-22T01:47:44.163 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T01:47:44.163 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-22T01:47:44.163 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-22T01:47:44.163 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-22T01:47:44.163 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:44.163 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-03-22T01:47:44.163 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-22T01:47:44.163 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-22T01:47:44.163 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-22T01:47:44.163 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:44.164 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-22T01:47:44.164 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 0 2024-03-22T01:47:44.164 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:44.164 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-1": { 2024-03-22T01:47:44.164 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 2, 2024-03-22T01:47:44.164 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 170, 2024-03-22T01:47:44.164 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 320, 2024-03-22T01:47:44.164 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 185520, 2024-03-22T01:47:44.164 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 5, 2024-03-22T01:47:44.164 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 3, 2024-03-22T01:47:44.165 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.010375553, 2024-03-22T01:47:44.165 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.006709667, 2024-03-22T01:47:44.165 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.001679881, 2024-03-22T01:47:44.165 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-22T01:47:44.165 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T01:47:44.165 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 167, 2024-03-22T01:47:44.165 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.015425632, 2024-03-22T01:47:44.165 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000092369 2024-03-22T01:47:44.165 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:44.165 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-03-22T01:47:44.165 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-22T01:47:44.165 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-22T01:47:44.165 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-22T01:47:44.165 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:44.166 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-22T01:47:44.166 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 185520 2024-03-22T01:47:44.166 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:44.166 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-2": { 2024-03-22T01:47:44.166 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 237, 2024-03-22T01:47:44.166 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 225, 2024-03-22T01:47:44.166 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 1449312, 2024-03-22T01:47:44.166 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 137472, 2024-03-22T01:47:44.166 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 2, 2024-03-22T01:47:44.166 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 1, 2024-03-22T01:47:44.166 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.042304772, 2024-03-22T01:47:44.166 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.013167896, 2024-03-22T01:47:44.166 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.031502561, 2024-03-22T01:47:44.167 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.005103604, 2024-03-22T01:47:44.167 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T01:47:44.167 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 225, 2024-03-22T01:47:44.167 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.011985171, 2024-03-22T01:47:44.167 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000053267 2024-03-22T01:47:44.167 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:44.167 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-03-22T01:47:44.167 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-22T01:47:44.167 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-22T01:47:44.168 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-22T01:47:44.168 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:44.169 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 1449312, 2024-03-22T01:47:44.169 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 137472 2024-03-22T01:47:44.169 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:44.169 INFO:teuthology.orchestra.run.smithi098.stdout: "cct": { 2024-03-22T01:47:44.169 INFO:teuthology.orchestra.run.smithi098.stdout: "total_workers": 0, 2024-03-22T01:47:44.169 INFO:teuthology.orchestra.run.smithi098.stdout: "unhealthy_workers": 0 2024-03-22T01:47:44.169 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:44.169 INFO:teuthology.orchestra.run.smithi098.stdout: "mempool": { 2024-03-22T01:47:44.169 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_bytes": 0, 2024-03-22T01:47:44.169 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_items": 0, 2024-03-22T01:47:44.169 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_bytes": 0, 2024-03-22T01:47:44.169 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_items": 0, 2024-03-22T01:47:44.169 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_bytes": 0, 2024-03-22T01:47:44.169 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_items": 0, 2024-03-22T01:47:44.169 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-22T01:47:44.170 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_items": 0, 2024-03-22T01:47:44.170 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-22T01:47:44.170 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_items": 0, 2024-03-22T01:47:44.170 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_bytes": 0, 2024-03-22T01:47:44.170 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_items": 0, 2024-03-22T01:47:44.170 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-22T01:47:44.170 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_items": 0, 2024-03-22T01:47:44.170 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_bytes": 0, 2024-03-22T01:47:44.170 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_items": 0, 2024-03-22T01:47:44.170 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_bytes": 0, 2024-03-22T01:47:44.170 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_items": 0, 2024-03-22T01:47:44.170 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-22T01:47:44.170 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_items": 0, 2024-03-22T01:47:44.170 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-22T01:47:44.174 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_items": 0, 2024-03-22T01:47:44.174 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_bytes": 0, 2024-03-22T01:47:44.174 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_items": 0, 2024-03-22T01:47:44.174 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_bytes": 0, 2024-03-22T01:47:44.175 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_items": 0, 2024-03-22T01:47:44.175 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-22T01:47:44.175 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_items": 0, 2024-03-22T01:47:44.175 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_bytes": 0, 2024-03-22T01:47:44.175 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_items": 0, 2024-03-22T01:47:44.175 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_bytes": 0, 2024-03-22T01:47:44.175 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_items": 0, 2024-03-22T01:47:44.175 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_bytes": 0, 2024-03-22T01:47:44.175 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_items": 0, 2024-03-22T01:47:44.175 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_bytes": 0, 2024-03-22T01:47:44.175 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_items": 0, 2024-03-22T01:47:44.175 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_bytes": 148977, 2024-03-22T01:47:44.175 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_items": 30, 2024-03-22T01:47:44.175 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_bytes": 0, 2024-03-22T01:47:44.175 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_items": 0, 2024-03-22T01:47:44.176 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_bytes": 0, 2024-03-22T01:47:44.176 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_items": 0, 2024-03-22T01:47:44.176 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_bytes": 0, 2024-03-22T01:47:44.176 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_items": 0, 2024-03-22T01:47:44.176 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_bytes": 0, 2024-03-22T01:47:44.176 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_items": 0, 2024-03-22T01:47:44.176 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_bytes": 0, 2024-03-22T01:47:44.176 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_items": 0, 2024-03-22T01:47:44.176 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_bytes": 0, 2024-03-22T01:47:44.176 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_items": 0, 2024-03-22T01:47:44.176 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_bytes": 0, 2024-03-22T01:47:44.176 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_items": 0, 2024-03-22T01:47:44.176 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_bytes": 928, 2024-03-22T01:47:44.176 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_items": 2, 2024-03-22T01:47:44.177 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_bytes": 0, 2024-03-22T01:47:44.177 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_items": 0, 2024-03-22T01:47:44.177 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_bytes": 0, 2024-03-22T01:47:44.177 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_items": 0 2024-03-22T01:47:44.177 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:44.177 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-22T01:47:44.177 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-03-22T01:47:44.177 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 104857600, 2024-03-22T01:47:44.177 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-03-22T01:47:44.177 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 235, 2024-03-22T01:47:44.177 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 1413440, 2024-03-22T01:47:44.177 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-03-22T01:47:44.177 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 235, 2024-03-22T01:47:44.177 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-03-22T01:47:44.178 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-03-22T01:47:44.178 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 235, 2024-03-22T01:47:44.178 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 1413440, 2024-03-22T01:47:44.178 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-03-22T01:47:44.178 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-22T01:47:44.178 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-22T01:47:44.178 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-22T01:47:44.178 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-22T01:47:44.178 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-22T01:47:44.178 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-22T01:47:45.202 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.c 2024-03-22T01:47:45.202 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.c 2024-03-22T01:47:45.202 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.c 2024-03-22T01:47:45.202 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.c 2024-03-22T01:47:45.203 DEBUG:tasks.check_counter:Getting stats from f 2024-03-22T01:47:45.203 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph daemon mds.f perf dump 2024-03-22T01:47:45.480 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.c/config 2024-03-22T01:47:45.798 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:45 smithi098 ceph-mon[36468]: pgmap v336: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-22T01:47:45.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:45 smithi089 ceph-mon[35770]: pgmap v336: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-22T01:47:45.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:45 smithi072 ceph-mon[32414]: pgmap v336: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-22T01:47:47.266 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-22T01:47:47.267 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-0": { 2024-03-22T01:47:47.267 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 0, 2024-03-22T01:47:47.267 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 2, 2024-03-22T01:47:47.267 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 0, 2024-03-22T01:47:47.267 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 0, 2024-03-22T01:47:47.267 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 4, 2024-03-22T01:47:47.267 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 0, 2024-03-22T01:47:47.267 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.001071533, 2024-03-22T01:47:47.267 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.000000000, 2024-03-22T01:47:47.267 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.000243429, 2024-03-22T01:47:47.267 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-22T01:47:47.267 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T01:47:47.267 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-22T01:47:47.267 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-22T01:47:47.268 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-22T01:47:47.268 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:47.268 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-03-22T01:47:47.268 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-22T01:47:47.268 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-22T01:47:47.268 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-22T01:47:47.268 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:47.268 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-22T01:47:47.268 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 0 2024-03-22T01:47:47.268 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:47.268 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-1": { 2024-03-22T01:47:47.268 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 238, 2024-03-22T01:47:47.268 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 226, 2024-03-22T01:47:47.268 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 1449584, 2024-03-22T01:47:47.268 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 138096, 2024-03-22T01:47:47.269 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 2, 2024-03-22T01:47:47.269 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 1, 2024-03-22T01:47:47.269 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.043232712, 2024-03-22T01:47:47.269 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.013046779, 2024-03-22T01:47:47.269 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.031881603, 2024-03-22T01:47:47.269 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.005596347, 2024-03-22T01:47:47.269 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T01:47:47.269 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 226, 2024-03-22T01:47:47.269 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.014444142, 2024-03-22T01:47:47.269 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000063912 2024-03-22T01:47:47.269 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:47.269 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-03-22T01:47:47.269 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-22T01:47:47.269 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-22T01:47:47.269 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-22T01:47:47.270 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:47.270 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 1449584, 2024-03-22T01:47:47.270 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 138096 2024-03-22T01:47:47.270 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:47.270 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-2": { 2024-03-22T01:47:47.270 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 2, 2024-03-22T01:47:47.270 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 171, 2024-03-22T01:47:47.270 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 320, 2024-03-22T01:47:47.270 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 185696, 2024-03-22T01:47:47.270 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 5, 2024-03-22T01:47:47.270 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 3, 2024-03-22T01:47:47.270 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.010411918, 2024-03-22T01:47:47.270 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.006760227, 2024-03-22T01:47:47.270 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.001483503, 2024-03-22T01:47:47.270 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-22T01:47:47.271 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T01:47:47.271 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 168, 2024-03-22T01:47:47.271 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.017505441, 2024-03-22T01:47:47.271 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000104199 2024-03-22T01:47:47.271 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:47.271 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-03-22T01:47:47.271 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-22T01:47:47.271 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-22T01:47:47.271 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-22T01:47:47.271 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:47.271 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-22T01:47:47.271 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 185696 2024-03-22T01:47:47.271 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:47.271 INFO:teuthology.orchestra.run.smithi098.stdout: "cct": { 2024-03-22T01:47:47.271 INFO:teuthology.orchestra.run.smithi098.stdout: "total_workers": 0, 2024-03-22T01:47:47.272 INFO:teuthology.orchestra.run.smithi098.stdout: "unhealthy_workers": 0 2024-03-22T01:47:47.272 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:47.272 INFO:teuthology.orchestra.run.smithi098.stdout: "mempool": { 2024-03-22T01:47:47.272 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_bytes": 0, 2024-03-22T01:47:47.272 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_items": 0, 2024-03-22T01:47:47.272 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_bytes": 0, 2024-03-22T01:47:47.272 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_items": 0, 2024-03-22T01:47:47.272 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_bytes": 0, 2024-03-22T01:47:47.272 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_items": 0, 2024-03-22T01:47:47.273 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-22T01:47:47.273 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_items": 0, 2024-03-22T01:47:47.273 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-22T01:47:47.273 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_items": 0, 2024-03-22T01:47:47.273 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_bytes": 0, 2024-03-22T01:47:47.273 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_items": 0, 2024-03-22T01:47:47.273 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-22T01:47:47.273 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_items": 0, 2024-03-22T01:47:47.273 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_bytes": 0, 2024-03-22T01:47:47.274 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_items": 0, 2024-03-22T01:47:47.274 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_bytes": 0, 2024-03-22T01:47:47.274 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_items": 0, 2024-03-22T01:47:47.274 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-22T01:47:47.274 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_items": 0, 2024-03-22T01:47:47.274 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-22T01:47:47.274 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_items": 0, 2024-03-22T01:47:47.274 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_bytes": 0, 2024-03-22T01:47:47.274 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_items": 0, 2024-03-22T01:47:47.274 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_bytes": 0, 2024-03-22T01:47:47.274 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_items": 0, 2024-03-22T01:47:47.274 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-22T01:47:47.274 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_items": 0, 2024-03-22T01:47:47.274 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_bytes": 0, 2024-03-22T01:47:47.275 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_items": 0, 2024-03-22T01:47:47.275 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_bytes": 0, 2024-03-22T01:47:47.275 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_items": 0, 2024-03-22T01:47:47.275 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_bytes": 0, 2024-03-22T01:47:47.275 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_items": 0, 2024-03-22T01:47:47.275 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_bytes": 0, 2024-03-22T01:47:47.275 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_items": 0, 2024-03-22T01:47:47.275 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_bytes": 150038, 2024-03-22T01:47:47.275 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_items": 31, 2024-03-22T01:47:47.275 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_bytes": 0, 2024-03-22T01:47:47.275 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_items": 0, 2024-03-22T01:47:47.275 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_bytes": 0, 2024-03-22T01:47:47.275 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_items": 0, 2024-03-22T01:47:47.275 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_bytes": 0, 2024-03-22T01:47:47.275 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_items": 0, 2024-03-22T01:47:47.276 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_bytes": 0, 2024-03-22T01:47:47.276 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_items": 0, 2024-03-22T01:47:47.276 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_bytes": 0, 2024-03-22T01:47:47.276 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_items": 0, 2024-03-22T01:47:47.276 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_bytes": 0, 2024-03-22T01:47:47.276 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_items": 0, 2024-03-22T01:47:47.276 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_bytes": 0, 2024-03-22T01:47:47.276 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_items": 0, 2024-03-22T01:47:47.276 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_bytes": 928, 2024-03-22T01:47:47.276 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_items": 2, 2024-03-22T01:47:47.276 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_bytes": 0, 2024-03-22T01:47:47.276 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_items": 0, 2024-03-22T01:47:47.276 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_bytes": 0, 2024-03-22T01:47:47.276 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_items": 0 2024-03-22T01:47:47.276 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:47.276 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-22T01:47:47.277 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-03-22T01:47:47.277 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 104857600, 2024-03-22T01:47:47.277 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-03-22T01:47:47.277 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 236, 2024-03-22T01:47:47.277 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 1413570, 2024-03-22T01:47:47.277 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-03-22T01:47:47.277 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 236, 2024-03-22T01:47:47.277 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-03-22T01:47:47.277 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-03-22T01:47:47.277 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 236, 2024-03-22T01:47:47.277 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 1413570, 2024-03-22T01:47:47.277 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-03-22T01:47:47.277 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-22T01:47:47.277 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-22T01:47:47.277 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-22T01:47:47.278 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-22T01:47:47.278 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-22T01:47:47.278 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-22T01:47:47.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:47 smithi098 ceph-mon[36468]: pgmap v337: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-22T01:47:47.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:47 smithi089 ceph-mon[35770]: pgmap v337: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-22T01:47:47.873 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:47 smithi072 ceph-mon[32414]: pgmap v337: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-22T01:47:48.016 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.f 2024-03-22T01:47:48.016 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.f 2024-03-22T01:47:48.016 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.f 2024-03-22T01:47:48.017 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.f 2024-03-22T01:47:48.017 DEBUG:tasks.check_counter:Getting stats from i 2024-03-22T01:47:48.017 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph daemon mds.i perf dump 2024-03-22T01:47:48.260 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.c/config 2024-03-22T01:47:49.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:49 smithi098 ceph-mon[36468]: pgmap v338: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-22T01:47:49.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:49 smithi089 ceph-mon[35770]: pgmap v338: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-22T01:47:49.875 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:49 smithi072 ceph-mon[32414]: pgmap v338: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-22T01:47:49.979 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-22T01:47:49.980 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-0": { 2024-03-22T01:47:49.980 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 1190, 2024-03-22T01:47:49.980 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 1097, 2024-03-22T01:47:49.980 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 20620686, 2024-03-22T01:47:49.980 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 418710, 2024-03-22T01:47:49.980 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 6, 2024-03-22T01:47:49.980 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 2, 2024-03-22T01:47:49.980 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.141000919, 2024-03-22T01:47:49.980 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.039450030, 2024-03-22T01:47:49.980 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.071457787, 2024-03-22T01:47:49.980 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.031845585, 2024-03-22T01:47:49.981 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T01:47:49.981 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 1096, 2024-03-22T01:47:49.981 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.048565552, 2024-03-22T01:47:49.981 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000044311 2024-03-22T01:47:49.981 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:49.981 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-03-22T01:47:49.981 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-22T01:47:49.981 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-22T01:47:49.981 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-22T01:47:49.981 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:49.981 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 1624096, 2024-03-22T01:47:49.981 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 144272 2024-03-22T01:47:49.981 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:49.982 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-1": { 2024-03-22T01:47:49.982 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 50, 2024-03-22T01:47:49.982 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 294, 2024-03-22T01:47:49.982 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 16792264, 2024-03-22T01:47:49.982 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 478304, 2024-03-22T01:47:49.982 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 7, 2024-03-22T01:47:49.982 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 5, 2024-03-22T01:47:49.982 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.028482696, 2024-03-22T01:47:49.982 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.008882465, 2024-03-22T01:47:49.982 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.015599187, 2024-03-22T01:47:49.982 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.001459429, 2024-03-22T01:47:49.982 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T01:47:49.983 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 292, 2024-03-22T01:47:49.983 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.020079993, 2024-03-22T01:47:49.983 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000068767 2024-03-22T01:47:49.983 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:49.983 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-03-22T01:47:49.983 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-22T01:47:49.983 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-22T01:47:49.983 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-22T01:47:49.983 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:49.983 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-03-22T01:47:49.983 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 458240 2024-03-22T01:47:49.983 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:49.984 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-2": { 2024-03-22T01:47:49.984 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 344, 2024-03-22T01:47:49.984 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 346, 2024-03-22T01:47:49.984 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 12667978, 2024-03-22T01:47:49.984 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 109399, 2024-03-22T01:47:49.984 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 5, 2024-03-22T01:47:49.984 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 4, 2024-03-22T01:47:49.984 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.042045159, 2024-03-22T01:47:49.984 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.011181019, 2024-03-22T01:47:49.984 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.016901188, 2024-03-22T01:47:49.984 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.010950683, 2024-03-22T01:47:49.984 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T01:47:49.985 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 344, 2024-03-22T01:47:49.985 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.014844719, 2024-03-22T01:47:49.985 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000043153 2024-03-22T01:47:49.985 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:49.985 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-03-22T01:47:49.985 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-22T01:47:49.985 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-22T01:47:49.985 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-22T01:47:49.985 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:49.985 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-22T01:47:49.985 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 0 2024-03-22T01:47:49.985 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:49.986 INFO:teuthology.orchestra.run.smithi098.stdout: "cct": { 2024-03-22T01:47:49.986 INFO:teuthology.orchestra.run.smithi098.stdout: "total_workers": 1, 2024-03-22T01:47:49.986 INFO:teuthology.orchestra.run.smithi098.stdout: "unhealthy_workers": 0 2024-03-22T01:47:49.986 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:49.986 INFO:teuthology.orchestra.run.smithi098.stdout: "finisher-MDSRank": { 2024-03-22T01:47:49.986 INFO:teuthology.orchestra.run.smithi098.stdout: "queue_len": 0, 2024-03-22T01:47:49.986 INFO:teuthology.orchestra.run.smithi098.stdout: "complete_latency": { 2024-03-22T01:47:49.986 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 2060, 2024-03-22T01:47:49.986 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.075324676, 2024-03-22T01:47:49.986 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000036565 2024-03-22T01:47:49.986 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-22T01:47:49.987 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:49.987 INFO:teuthology.orchestra.run.smithi098.stdout: "finisher-PurgeQueue": { 2024-03-22T01:47:49.987 INFO:teuthology.orchestra.run.smithi098.stdout: "queue_len": 0, 2024-03-22T01:47:49.987 INFO:teuthology.orchestra.run.smithi098.stdout: "complete_latency": { 2024-03-22T01:47:49.987 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-22T01:47:49.987 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-22T01:47:49.987 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-22T01:47:49.987 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-22T01:47:49.987 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:49.987 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": { 2024-03-22T01:47:49.987 INFO:teuthology.orchestra.run.smithi098.stdout: "request": 0, 2024-03-22T01:47:49.987 INFO:teuthology.orchestra.run.smithi098.stdout: "reply": 0, 2024-03-22T01:47:49.988 INFO:teuthology.orchestra.run.smithi098.stdout: "reply_latency": { 2024-03-22T01:47:49.988 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-22T01:47:49.988 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-22T01:47:49.988 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-22T01:47:49.988 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:49.988 INFO:teuthology.orchestra.run.smithi098.stdout: "slow_reply": 0, 2024-03-22T01:47:49.988 INFO:teuthology.orchestra.run.smithi098.stdout: "forward": 0, 2024-03-22T01:47:49.988 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_fetch_complete": 0, 2024-03-22T01:47:49.988 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_fetch_keys": 0, 2024-03-22T01:47:49.988 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_commit": 0, 2024-03-22T01:47:49.988 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_split": 0, 2024-03-22T01:47:49.988 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_merge": 0, 2024-03-22T01:47:49.989 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes": 1892, 2024-03-22T01:47:49.989 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_top": 1262, 2024-03-22T01:47:49.989 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_bottom": 630, 2024-03-22T01:47:49.989 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_pin_tail": 0, 2024-03-22T01:47:49.989 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_pinned": 91, 2024-03-22T01:47:49.989 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_expired": 0, 2024-03-22T01:47:49.989 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_with_caps": 0, 2024-03-22T01:47:49.989 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": 0, 2024-03-22T01:47:49.989 INFO:teuthology.orchestra.run.smithi098.stdout: "subtrees": 2, 2024-03-22T01:47:49.989 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse": 0, 2024-03-22T01:47:49.989 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_hit": 0, 2024-03-22T01:47:49.989 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_forward": 0, 2024-03-22T01:47:49.989 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_discover": 0, 2024-03-22T01:47:49.990 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_dir_fetch": 0, 2024-03-22T01:47:49.990 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_remote_ino": 0, 2024-03-22T01:47:49.990 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_lock": 0, 2024-03-22T01:47:49.990 INFO:teuthology.orchestra.run.smithi098.stdout: "load_cent": 0, 2024-03-22T01:47:49.990 INFO:teuthology.orchestra.run.smithi098.stdout: "q": 0, 2024-03-22T01:47:49.990 INFO:teuthology.orchestra.run.smithi098.stdout: "exported": 0, 2024-03-22T01:47:49.990 INFO:teuthology.orchestra.run.smithi098.stdout: "exported_inodes": 0, 2024-03-22T01:47:49.990 INFO:teuthology.orchestra.run.smithi098.stdout: "imported": 0, 2024-03-22T01:47:49.990 INFO:teuthology.orchestra.run.smithi098.stdout: "imported_inodes": 0, 2024-03-22T01:47:49.990 INFO:teuthology.orchestra.run.smithi098.stdout: "openino_dir_fetch": 0, 2024-03-22T01:47:49.990 INFO:teuthology.orchestra.run.smithi098.stdout: "openino_backtrace_fetch": 0, 2024-03-22T01:47:49.990 INFO:teuthology.orchestra.run.smithi098.stdout: "openino_peer_discover": 0, 2024-03-22T01:47:49.991 INFO:teuthology.orchestra.run.smithi098.stdout: "root_rfiles": 4, 2024-03-22T01:47:49.991 INFO:teuthology.orchestra.run.smithi098.stdout: "root_rbytes": 234, 2024-03-22T01:47:49.991 INFO:teuthology.orchestra.run.smithi098.stdout: "root_rsnaps": 5, 2024-03-22T01:47:49.991 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_backtrace_fetch": 0, 2024-03-22T01:47:49.991 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_set_tag": 0, 2024-03-22T01:47:49.991 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_backtrace_repaired": 0, 2024-03-22T01:47:49.991 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_inotable_repaired": 0, 2024-03-22T01:47:49.991 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_dir_inodes": 0, 2024-03-22T01:47:49.991 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_dir_base_inodes": 0, 2024-03-22T01:47:49.991 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_dirfrag_rstats": 0, 2024-03-22T01:47:49.991 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_file_inodes": 0, 2024-03-22T01:47:49.991 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_inode_file_caps": 0, 2024-03-22T01:47:49.992 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_revoke": 0, 2024-03-22T01:47:49.992 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_grant": 0, 2024-03-22T01:47:49.992 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_trunc": 0, 2024-03-22T01:47:49.992 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-22T01:47:49.992 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-22T01:47:49.992 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_caps": 0, 2024-03-22T01:47:49.992 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_caps_dirty": 0, 2024-03-22T01:47:49.992 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_cap_release": 0, 2024-03-22T01:47:49.992 INFO:teuthology.orchestra.run.smithi098.stdout: "process_request_cap_release": 0 2024-03-22T01:47:49.992 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:49.992 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_cache": { 2024-03-22T01:47:49.992 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_update": 0, 2024-03-22T01:47:49.993 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_update_receipt": 0, 2024-03-22T01:47:49.993 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_try_discover": 0, 2024-03-22T01:47:49.993 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_send_discover": 0, 2024-03-22T01:47:49.993 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_handle_discover": 0, 2024-03-22T01:47:49.993 INFO:teuthology.orchestra.run.smithi098.stdout: "num_strays": 0, 2024-03-22T01:47:49.993 INFO:teuthology.orchestra.run.smithi098.stdout: "num_strays_delayed": 0, 2024-03-22T01:47:49.993 INFO:teuthology.orchestra.run.smithi098.stdout: "num_strays_enqueuing": 0, 2024-03-22T01:47:49.993 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_created": 0, 2024-03-22T01:47:49.993 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_enqueued": 0, 2024-03-22T01:47:49.993 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_reintegrated": 0, 2024-03-22T01:47:49.993 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_migrated": 0, 2024-03-22T01:47:49.993 INFO:teuthology.orchestra.run.smithi098.stdout: "num_recovering_processing": 0, 2024-03-22T01:47:49.993 INFO:teuthology.orchestra.run.smithi098.stdout: "num_recovering_enqueued": 0, 2024-03-22T01:47:49.994 INFO:teuthology.orchestra.run.smithi098.stdout: "num_recovering_prioritized": 0, 2024-03-22T01:47:49.994 INFO:teuthology.orchestra.run.smithi098.stdout: "recovery_started": 0, 2024-03-22T01:47:49.994 INFO:teuthology.orchestra.run.smithi098.stdout: "recovery_completed": 0, 2024-03-22T01:47:49.994 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_quiesce_path": 0, 2024-03-22T01:47:49.994 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_quiesce_inode": 0, 2024-03-22T01:47:49.994 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_enqueue_scrub": 0, 2024-03-22T01:47:49.994 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_exportdir": 0, 2024-03-22T01:47:49.994 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_flush": 0, 2024-03-22T01:47:49.994 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_fragmentdir": 0, 2024-03-22T01:47:49.994 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_fragstats": 0, 2024-03-22T01:47:49.994 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_inodestats": 0 2024-03-22T01:47:49.994 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:49.995 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_log": { 2024-03-22T01:47:49.995 INFO:teuthology.orchestra.run.smithi098.stdout: "evlrg": 0, 2024-03-22T01:47:49.995 INFO:teuthology.orchestra.run.smithi098.stdout: "evadd": 0, 2024-03-22T01:47:49.995 INFO:teuthology.orchestra.run.smithi098.stdout: "evex": 0, 2024-03-22T01:47:49.995 INFO:teuthology.orchestra.run.smithi098.stdout: "evtrm": 0, 2024-03-22T01:47:49.995 INFO:teuthology.orchestra.run.smithi098.stdout: "ev": 22002, 2024-03-22T01:47:49.995 INFO:teuthology.orchestra.run.smithi098.stdout: "evexg": 0, 2024-03-22T01:47:49.995 INFO:teuthology.orchestra.run.smithi098.stdout: "evexd": 0, 2024-03-22T01:47:49.995 INFO:teuthology.orchestra.run.smithi098.stdout: "segadd": 0, 2024-03-22T01:47:49.995 INFO:teuthology.orchestra.run.smithi098.stdout: "segex": 0, 2024-03-22T01:47:49.995 INFO:teuthology.orchestra.run.smithi098.stdout: "segtrm": 0, 2024-03-22T01:47:49.995 INFO:teuthology.orchestra.run.smithi098.stdout: "seg": 2, 2024-03-22T01:47:49.995 INFO:teuthology.orchestra.run.smithi098.stdout: "segmjr": 19, 2024-03-22T01:47:49.996 INFO:teuthology.orchestra.run.smithi098.stdout: "segexg": 0, 2024-03-22T01:47:49.996 INFO:teuthology.orchestra.run.smithi098.stdout: "segexd": 0, 2024-03-22T01:47:49.996 INFO:teuthology.orchestra.run.smithi098.stdout: "expos": 52273579, 2024-03-22T01:47:49.996 INFO:teuthology.orchestra.run.smithi098.stdout: "wrpos": 52297594, 2024-03-22T01:47:49.996 INFO:teuthology.orchestra.run.smithi098.stdout: "rdpos": 52297534, 2024-03-22T01:47:49.996 INFO:teuthology.orchestra.run.smithi098.stdout: "jlat": { 2024-03-22T01:47:49.996 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-22T01:47:49.996 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-22T01:47:49.996 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-22T01:47:49.996 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:49.996 INFO:teuthology.orchestra.run.smithi098.stdout: "replayed": 22002 2024-03-22T01:47:49.996 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:49.997 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_mem": { 2024-03-22T01:47:49.997 INFO:teuthology.orchestra.run.smithi098.stdout: "ino": 422, 2024-03-22T01:47:49.997 INFO:teuthology.orchestra.run.smithi098.stdout: "ino+": 1211, 2024-03-22T01:47:49.997 INFO:teuthology.orchestra.run.smithi098.stdout: "ino-": 789, 2024-03-22T01:47:49.997 INFO:teuthology.orchestra.run.smithi098.stdout: "dir": 104, 2024-03-22T01:47:49.997 INFO:teuthology.orchestra.run.smithi098.stdout: "dir+": 121, 2024-03-22T01:47:49.997 INFO:teuthology.orchestra.run.smithi098.stdout: "dir-": 17, 2024-03-22T01:47:49.997 INFO:teuthology.orchestra.run.smithi098.stdout: "dn": 1892, 2024-03-22T01:47:49.997 INFO:teuthology.orchestra.run.smithi098.stdout: "dn+": 1940, 2024-03-22T01:47:49.997 INFO:teuthology.orchestra.run.smithi098.stdout: "dn-": 48, 2024-03-22T01:47:49.997 INFO:teuthology.orchestra.run.smithi098.stdout: "cap": 0, 2024-03-22T01:47:49.997 INFO:teuthology.orchestra.run.smithi098.stdout: "cap+": 0, 2024-03-22T01:47:49.997 INFO:teuthology.orchestra.run.smithi098.stdout: "cap-": 0, 2024-03-22T01:47:49.998 INFO:teuthology.orchestra.run.smithi098.stdout: "rss": 54996, 2024-03-22T01:47:49.998 INFO:teuthology.orchestra.run.smithi098.stdout: "heap": 215416 2024-03-22T01:47:49.998 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:49.998 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_server": { 2024-03-22T01:47:49.998 INFO:teuthology.orchestra.run.smithi098.stdout: "dispatch_client_request": 0, 2024-03-22T01:47:49.998 INFO:teuthology.orchestra.run.smithi098.stdout: "dispatch_server_request": 0, 2024-03-22T01:47:49.998 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_request": 0, 2024-03-22T01:47:49.998 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_session": 0, 2024-03-22T01:47:49.998 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_peer_request": 0, 2024-03-22T01:47:49.998 INFO:teuthology.orchestra.run.smithi098.stdout: "req_create_latency": { 2024-03-22T01:47:49.998 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-22T01:47:49.999 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-22T01:47:49.999 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-22T01:47:49.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:49.999 INFO:teuthology.orchestra.run.smithi098.stdout: "req_getattr_latency": { 2024-03-22T01:47:49.999 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-22T01:47:49.999 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-22T01:47:49.999 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-22T01:47:49.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:49.999 INFO:teuthology.orchestra.run.smithi098.stdout: "req_getfilelock_latency": { 2024-03-22T01:47:49.999 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-22T01:47:49.999 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-22T01:47:50.000 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-22T01:47:50.000 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:50.000 INFO:teuthology.orchestra.run.smithi098.stdout: "req_link_latency": { 2024-03-22T01:47:50.000 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-22T01:47:50.000 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-22T01:47:50.000 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-22T01:47:50.000 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:50.000 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookup_latency": { 2024-03-22T01:47:50.000 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-22T01:47:50.000 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-22T01:47:50.000 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-22T01:47:50.000 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:50.001 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookuphash_latency": { 2024-03-22T01:47:50.001 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-22T01:47:50.001 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-22T01:47:50.001 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-22T01:47:50.001 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:50.001 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupino_latency": { 2024-03-22T01:47:50.001 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-22T01:47:50.001 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-22T01:47:50.001 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-22T01:47:50.001 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:50.001 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupname_latency": { 2024-03-22T01:47:50.002 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-22T01:47:50.002 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-22T01:47:50.002 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-22T01:47:50.002 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:50.002 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupparent_latency": { 2024-03-22T01:47:50.002 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-22T01:47:50.002 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-22T01:47:50.002 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-22T01:47:50.002 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:50.002 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupsnap_latency": { 2024-03-22T01:47:50.002 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-22T01:47:50.002 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-22T01:47:50.003 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-22T01:47:50.003 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:50.003 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lssnap_latency": { 2024-03-22T01:47:50.003 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-22T01:47:50.003 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-22T01:47:50.003 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-22T01:47:50.003 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:50.003 INFO:teuthology.orchestra.run.smithi098.stdout: "req_mkdir_latency": { 2024-03-22T01:47:50.004 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-22T01:47:50.004 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-22T01:47:50.004 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-22T01:47:50.005 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:50.005 INFO:teuthology.orchestra.run.smithi098.stdout: "req_mknod_latency": { 2024-03-22T01:47:50.005 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-22T01:47:50.005 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-22T01:47:50.005 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-22T01:47:50.005 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:50.005 INFO:teuthology.orchestra.run.smithi098.stdout: "req_mksnap_latency": { 2024-03-22T01:47:50.005 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-22T01:47:50.005 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-22T01:47:50.005 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-22T01:47:50.005 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:50.006 INFO:teuthology.orchestra.run.smithi098.stdout: "req_open_latency": { 2024-03-22T01:47:50.006 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-22T01:47:50.006 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-22T01:47:50.006 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-22T01:47:50.006 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:50.006 INFO:teuthology.orchestra.run.smithi098.stdout: "req_readdir_latency": { 2024-03-22T01:47:50.006 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-22T01:47:50.006 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-22T01:47:50.006 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-22T01:47:50.006 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:50.006 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rename_latency": { 2024-03-22T01:47:50.007 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-22T01:47:50.007 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-22T01:47:50.007 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-22T01:47:50.007 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:50.007 INFO:teuthology.orchestra.run.smithi098.stdout: "req_renamesnap_latency": { 2024-03-22T01:47:50.007 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-22T01:47:50.007 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-22T01:47:50.007 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-22T01:47:50.007 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:50.007 INFO:teuthology.orchestra.run.smithi098.stdout: "req_snapdiff_latency": { 2024-03-22T01:47:50.007 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-22T01:47:50.007 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-22T01:47:50.008 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-22T01:47:50.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:50.008 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rmdir_latency": { 2024-03-22T01:47:50.008 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-22T01:47:50.008 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-22T01:47:50.008 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-22T01:47:50.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:50.008 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rmsnap_latency": { 2024-03-22T01:47:50.008 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-22T01:47:50.008 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-22T01:47:50.008 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-22T01:47:50.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:50.009 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rmxattr_latency": { 2024-03-22T01:47:50.009 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-22T01:47:50.009 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-22T01:47:50.009 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-22T01:47:50.009 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:50.009 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setattr_latency": { 2024-03-22T01:47:50.009 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-22T01:47:50.009 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-22T01:47:50.009 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-22T01:47:50.009 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:50.009 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setdirlayout_latency": { 2024-03-22T01:47:50.009 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-22T01:47:50.010 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-22T01:47:50.010 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-22T01:47:50.010 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:50.010 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setfilelock_latency": { 2024-03-22T01:47:50.010 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-22T01:47:50.010 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-22T01:47:50.010 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-22T01:47:50.010 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:50.010 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setlayout_latency": { 2024-03-22T01:47:50.010 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-22T01:47:50.010 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-22T01:47:50.010 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-22T01:47:50.011 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:50.011 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setxattr_latency": { 2024-03-22T01:47:50.011 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-22T01:47:50.011 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-22T01:47:50.011 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-22T01:47:50.011 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:50.011 INFO:teuthology.orchestra.run.smithi098.stdout: "req_symlink_latency": { 2024-03-22T01:47:50.011 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-22T01:47:50.011 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-22T01:47:50.011 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-22T01:47:50.011 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:50.012 INFO:teuthology.orchestra.run.smithi098.stdout: "req_unlink_latency": { 2024-03-22T01:47:50.012 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-22T01:47:50.012 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-22T01:47:50.012 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-22T01:47:50.012 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:50.012 INFO:teuthology.orchestra.run.smithi098.stdout: "cap_revoke_eviction": 0, 2024-03-22T01:47:50.012 INFO:teuthology.orchestra.run.smithi098.stdout: "cap_acquisition_throttle": 0, 2024-03-22T01:47:50.012 INFO:teuthology.orchestra.run.smithi098.stdout: "req_getvxattr_latency": { 2024-03-22T01:47:50.012 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-22T01:47:50.012 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-22T01:47:50.012 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-22T01:47:50.013 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-22T01:47:50.013 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:50.013 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_sessions": { 2024-03-22T01:47:50.013 INFO:teuthology.orchestra.run.smithi098.stdout: "session_count": 2, 2024-03-22T01:47:50.013 INFO:teuthology.orchestra.run.smithi098.stdout: "session_add": 6, 2024-03-22T01:47:50.013 INFO:teuthology.orchestra.run.smithi098.stdout: "session_remove": 4, 2024-03-22T01:47:50.013 INFO:teuthology.orchestra.run.smithi098.stdout: "sessions_open": 4, 2024-03-22T01:47:50.013 INFO:teuthology.orchestra.run.smithi098.stdout: "sessions_stale": 0, 2024-03-22T01:47:50.013 INFO:teuthology.orchestra.run.smithi098.stdout: "total_load": 0, 2024-03-22T01:47:50.013 INFO:teuthology.orchestra.run.smithi098.stdout: "average_load": 0, 2024-03-22T01:47:50.013 INFO:teuthology.orchestra.run.smithi098.stdout: "avg_session_uptime": 3596, 2024-03-22T01:47:50.013 INFO:teuthology.orchestra.run.smithi098.stdout: "mdthresh_evicted": 0 2024-03-22T01:47:50.014 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:50.014 INFO:teuthology.orchestra.run.smithi098.stdout: "mempool": { 2024-03-22T01:47:50.014 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_bytes": 0, 2024-03-22T01:47:50.014 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_items": 0, 2024-03-22T01:47:50.014 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_bytes": 0, 2024-03-22T01:47:50.014 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_items": 0, 2024-03-22T01:47:50.014 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_bytes": 0, 2024-03-22T01:47:50.014 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_items": 0, 2024-03-22T01:47:50.014 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-22T01:47:50.014 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_items": 0, 2024-03-22T01:47:50.014 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-22T01:47:50.014 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_items": 0, 2024-03-22T01:47:50.015 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_bytes": 0, 2024-03-22T01:47:50.015 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_items": 0, 2024-03-22T01:47:50.015 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-22T01:47:50.015 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_items": 0, 2024-03-22T01:47:50.015 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_bytes": 0, 2024-03-22T01:47:50.015 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_items": 0, 2024-03-22T01:47:50.015 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_bytes": 0, 2024-03-22T01:47:50.015 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_items": 0, 2024-03-22T01:47:50.015 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-22T01:47:50.015 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_items": 0, 2024-03-22T01:47:50.015 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-22T01:47:50.016 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_items": 0, 2024-03-22T01:47:50.016 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_bytes": 0, 2024-03-22T01:47:50.016 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_items": 0, 2024-03-22T01:47:50.016 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_bytes": 0, 2024-03-22T01:47:50.016 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_items": 0, 2024-03-22T01:47:50.016 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-22T01:47:50.016 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_items": 0, 2024-03-22T01:47:50.016 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_bytes": 0, 2024-03-22T01:47:50.016 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_items": 0, 2024-03-22T01:47:50.016 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_bytes": 0, 2024-03-22T01:47:50.016 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_items": 0, 2024-03-22T01:47:50.016 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_bytes": 0, 2024-03-22T01:47:50.016 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_items": 0, 2024-03-22T01:47:50.017 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_bytes": 0, 2024-03-22T01:47:50.017 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_items": 0, 2024-03-22T01:47:50.017 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_bytes": 224447, 2024-03-22T01:47:50.017 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_items": 551, 2024-03-22T01:47:50.017 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_bytes": 0, 2024-03-22T01:47:50.017 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_items": 0, 2024-03-22T01:47:50.017 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_bytes": 0, 2024-03-22T01:47:50.017 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_items": 0, 2024-03-22T01:47:50.017 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_bytes": 0, 2024-03-22T01:47:50.017 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_items": 0, 2024-03-22T01:47:50.017 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_bytes": 0, 2024-03-22T01:47:50.017 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_items": 0, 2024-03-22T01:47:50.017 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_bytes": 9464, 2024-03-22T01:47:50.018 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_items": 180, 2024-03-22T01:47:50.018 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_bytes": 0, 2024-03-22T01:47:50.018 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_items": 0, 2024-03-22T01:47:50.018 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_bytes": 0, 2024-03-22T01:47:50.018 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_items": 0, 2024-03-22T01:47:50.018 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_bytes": 2560880, 2024-03-22T01:47:50.018 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_items": 38188, 2024-03-22T01:47:50.018 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_bytes": 0, 2024-03-22T01:47:50.018 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_items": 0, 2024-03-22T01:47:50.018 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_bytes": 0, 2024-03-22T01:47:50.018 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_items": 0 2024-03-22T01:47:50.019 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:50.019 INFO:teuthology.orchestra.run.smithi098.stdout: "objecter": { 2024-03-22T01:47:50.019 INFO:teuthology.orchestra.run.smithi098.stdout: "op_active": 0, 2024-03-22T01:47:50.019 INFO:teuthology.orchestra.run.smithi098.stdout: "op_laggy": 0, 2024-03-22T01:47:50.019 INFO:teuthology.orchestra.run.smithi098.stdout: "op_send": 1251, 2024-03-22T01:47:50.019 INFO:teuthology.orchestra.run.smithi098.stdout: "op_send_bytes": 26, 2024-03-22T01:47:50.019 INFO:teuthology.orchestra.run.smithi098.stdout: "op_resend": 4, 2024-03-22T01:47:50.019 INFO:teuthology.orchestra.run.smithi098.stdout: "op_reply": 1251, 2024-03-22T01:47:50.019 INFO:teuthology.orchestra.run.smithi098.stdout: "op_latency": { 2024-03-22T01:47:50.019 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 1251, 2024-03-22T01:47:50.019 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 405.512970453, 2024-03-22T01:47:50.019 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.324151055 2024-03-22T01:47:50.020 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:50.020 INFO:teuthology.orchestra.run.smithi098.stdout: "op_inflight": 0, 2024-03-22T01:47:50.020 INFO:teuthology.orchestra.run.smithi098.stdout: "oplen_avg": { 2024-03-22T01:47:50.020 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 1251, 2024-03-22T01:47:50.020 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 1252 2024-03-22T01:47:50.020 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:50.020 INFO:teuthology.orchestra.run.smithi098.stdout: "op": 1251, 2024-03-22T01:47:50.020 INFO:teuthology.orchestra.run.smithi098.stdout: "op_r": 1251, 2024-03-22T01:47:50.020 INFO:teuthology.orchestra.run.smithi098.stdout: "op_w": 0, 2024-03-22T01:47:50.020 INFO:teuthology.orchestra.run.smithi098.stdout: "op_rmw": 0, 2024-03-22T01:47:50.020 INFO:teuthology.orchestra.run.smithi098.stdout: "op_pg": 0, 2024-03-22T01:47:50.020 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_stat": 775, 2024-03-22T01:47:50.021 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_create": 0, 2024-03-22T01:47:50.021 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_read": 473, 2024-03-22T01:47:50.021 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_write": 0, 2024-03-22T01:47:50.021 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_writefull": 0, 2024-03-22T01:47:50.021 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_writesame": 0, 2024-03-22T01:47:50.021 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_append": 0, 2024-03-22T01:47:50.021 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_zero": 0, 2024-03-22T01:47:50.021 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_truncate": 0, 2024-03-22T01:47:50.021 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_delete": 0, 2024-03-22T01:47:50.021 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_mapext": 0, 2024-03-22T01:47:50.021 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_sparse_read": 0, 2024-03-22T01:47:50.022 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_clonerange": 0, 2024-03-22T01:47:50.022 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_getxattr": 2, 2024-03-22T01:47:50.022 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_setxattr": 0, 2024-03-22T01:47:50.022 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_cmpxattr": 0, 2024-03-22T01:47:50.022 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_rmxattr": 0, 2024-03-22T01:47:50.022 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_resetxattrs": 0, 2024-03-22T01:47:50.022 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_call": 0, 2024-03-22T01:47:50.022 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_watch": 0, 2024-03-22T01:47:50.022 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_notify": 0, 2024-03-22T01:47:50.022 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_src_cmpxattr": 0, 2024-03-22T01:47:50.022 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_pgls": 0, 2024-03-22T01:47:50.023 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_pgls_filter": 0, 2024-03-22T01:47:50.023 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_other": 0, 2024-03-22T01:47:50.023 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_active": 0, 2024-03-22T01:47:50.023 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_send": 0, 2024-03-22T01:47:50.023 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_resend": 0, 2024-03-22T01:47:50.023 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_ping": 0, 2024-03-22T01:47:50.023 INFO:teuthology.orchestra.run.smithi098.stdout: "poolop_active": 0, 2024-03-22T01:47:50.023 INFO:teuthology.orchestra.run.smithi098.stdout: "poolop_send": 0, 2024-03-22T01:47:50.023 INFO:teuthology.orchestra.run.smithi098.stdout: "poolop_resend": 0, 2024-03-22T01:47:50.023 INFO:teuthology.orchestra.run.smithi098.stdout: "poolstat_active": 0, 2024-03-22T01:47:50.023 INFO:teuthology.orchestra.run.smithi098.stdout: "poolstat_send": 0, 2024-03-22T01:47:50.023 INFO:teuthology.orchestra.run.smithi098.stdout: "poolstat_resend": 0, 2024-03-22T01:47:50.023 INFO:teuthology.orchestra.run.smithi098.stdout: "statfs_active": 0, 2024-03-22T01:47:50.024 INFO:teuthology.orchestra.run.smithi098.stdout: "statfs_send": 0, 2024-03-22T01:47:50.024 INFO:teuthology.orchestra.run.smithi098.stdout: "statfs_resend": 0, 2024-03-22T01:47:50.024 INFO:teuthology.orchestra.run.smithi098.stdout: "command_active": 0, 2024-03-22T01:47:50.024 INFO:teuthology.orchestra.run.smithi098.stdout: "command_send": 0, 2024-03-22T01:47:50.024 INFO:teuthology.orchestra.run.smithi098.stdout: "command_resend": 0, 2024-03-22T01:47:50.024 INFO:teuthology.orchestra.run.smithi098.stdout: "map_epoch": 96, 2024-03-22T01:47:50.024 INFO:teuthology.orchestra.run.smithi098.stdout: "map_full": 0, 2024-03-22T01:47:50.024 INFO:teuthology.orchestra.run.smithi098.stdout: "map_inc": 5, 2024-03-22T01:47:50.024 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_sessions": 7, 2024-03-22T01:47:50.024 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_session_open": 7, 2024-03-22T01:47:50.024 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_session_close": 0, 2024-03-22T01:47:50.024 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_laggy": 0, 2024-03-22T01:47:50.025 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_wr": 0, 2024-03-22T01:47:50.025 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_rd": 2, 2024-03-22T01:47:50.025 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_del": 0 2024-03-22T01:47:50.025 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:50.025 INFO:teuthology.orchestra.run.smithi098.stdout: "oft": { 2024-03-22T01:47:50.025 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_objs": 0, 2024-03-22T01:47:50.025 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_kv_pairs": 0, 2024-03-22T01:47:50.025 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_updates": 0, 2024-03-22T01:47:50.025 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_removes": 0 2024-03-22T01:47:50.025 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:50.026 INFO:teuthology.orchestra.run.smithi098.stdout: "osd-slow-ops": { 2024-03-22T01:47:50.026 INFO:teuthology.orchestra.run.smithi098.stdout: "slow_ops_count": 0 2024-03-22T01:47:50.026 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:50.026 INFO:teuthology.orchestra.run.smithi098.stdout: "purge_queue": { 2024-03-22T01:47:50.026 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing_ops": 0, 2024-03-22T01:47:50.026 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing_ops_high_water": 0, 2024-03-22T01:47:50.026 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing": 0, 2024-03-22T01:47:50.026 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing_high_water": 0, 2024-03-22T01:47:50.026 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executed_ops": 0, 2024-03-22T01:47:50.026 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executed": 0, 2024-03-22T01:47:50.026 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_item_in_journal": 0 2024-03-22T01:47:50.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:50.027 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-22T01:47:50.027 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-03-22T01:47:50.027 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 104857600, 2024-03-22T01:47:50.027 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-03-22T01:47:50.027 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 1580, 2024-03-22T01:47:50.027 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 49919096, 2024-03-22T01:47:50.027 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-03-22T01:47:50.027 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 1580, 2024-03-22T01:47:50.027 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-03-22T01:47:50.028 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-03-22T01:47:50.028 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 1580, 2024-03-22T01:47:50.028 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 49919096, 2024-03-22T01:47:50.028 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-03-22T01:47:50.028 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-22T01:47:50.028 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-22T01:47:50.028 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-22T01:47:50.028 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-22T01:47:50.028 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:50.028 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-objecter_bytes": { 2024-03-22T01:47:50.028 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-03-22T01:47:50.029 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 104857600, 2024-03-22T01:47:50.029 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-03-22T01:47:50.029 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 0, 2024-03-22T01:47:50.029 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 0, 2024-03-22T01:47:50.029 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-03-22T01:47:50.029 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 0, 2024-03-22T01:47:50.029 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 1251, 2024-03-22T01:47:50.029 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 48103300, 2024-03-22T01:47:50.029 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 1251, 2024-03-22T01:47:50.029 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 48103300, 2024-03-22T01:47:50.029 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-03-22T01:47:50.030 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-22T01:47:50.030 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-22T01:47:50.030 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-22T01:47:50.030 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-22T01:47:50.030 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:50.030 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-objecter_ops": { 2024-03-22T01:47:50.030 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-03-22T01:47:50.030 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 1024, 2024-03-22T01:47:50.030 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-03-22T01:47:50.030 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 0, 2024-03-22T01:47:50.030 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 0, 2024-03-22T01:47:50.031 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-03-22T01:47:50.031 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 0, 2024-03-22T01:47:50.031 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 1251, 2024-03-22T01:47:50.031 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 1251, 2024-03-22T01:47:50.031 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 1251, 2024-03-22T01:47:50.031 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 1251, 2024-03-22T01:47:50.031 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-03-22T01:47:50.031 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-22T01:47:50.031 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-22T01:47:50.031 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-22T01:47:50.031 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-22T01:47:50.031 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:50.032 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-write_buf_throttle": { 2024-03-22T01:47:50.032 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-03-22T01:47:50.032 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 3758096384, 2024-03-22T01:47:50.032 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-03-22T01:47:50.032 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 0, 2024-03-22T01:47:50.032 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 0, 2024-03-22T01:47:50.032 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-03-22T01:47:50.032 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 0, 2024-03-22T01:47:50.032 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-03-22T01:47:50.032 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-03-22T01:47:50.032 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 0, 2024-03-22T01:47:50.033 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 0, 2024-03-22T01:47:50.033 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-03-22T01:47:50.033 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-22T01:47:50.033 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-22T01:47:50.033 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-22T01:47:50.033 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-22T01:47:50.033 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:50.033 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-write_buf_throttle-0x55e4e26cc6e0": { 2024-03-22T01:47:50.033 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-03-22T01:47:50.033 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 3758096384, 2024-03-22T01:47:50.033 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-03-22T01:47:50.033 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 0, 2024-03-22T01:47:50.034 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 0, 2024-03-22T01:47:50.034 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-03-22T01:47:50.034 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 0, 2024-03-22T01:47:50.034 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-03-22T01:47:50.034 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-03-22T01:47:50.034 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 0, 2024-03-22T01:47:50.034 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 0, 2024-03-22T01:47:50.034 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-03-22T01:47:50.034 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-22T01:47:50.034 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-22T01:47:50.034 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-22T01:47:50.034 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-22T01:47:50.035 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-22T01:47:50.035 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-22T01:47:50.751 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=5 2024-03-22T01:47:50.752 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-03-22T01:47:50.752 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-03-22T01:47:50.752 INFO:tasks.check_counter:Daemon mds.i mds.dir_split=0 2024-03-22T01:47:50.752 DEBUG:tasks.check_counter:Getting stats from l 2024-03-22T01:47:50.752 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 -- ceph daemon mds.l perf dump 2024-03-22T01:47:50.988 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/mon.c/config 2024-03-22T01:47:51.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:51 smithi098 ceph-mon[36468]: pgmap v339: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-22T01:47:51.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:51 smithi089 ceph-mon[35770]: pgmap v339: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-22T01:47:51.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:51 smithi072 ceph-mon[32414]: pgmap v339: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-22T01:47:52.680 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-22T01:47:52.681 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-0": { 2024-03-22T01:47:52.681 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 0, 2024-03-22T01:47:52.681 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 2, 2024-03-22T01:47:52.681 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 0, 2024-03-22T01:47:52.681 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 0, 2024-03-22T01:47:52.681 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 4, 2024-03-22T01:47:52.681 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 0, 2024-03-22T01:47:52.681 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.000979459, 2024-03-22T01:47:52.681 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.000000000, 2024-03-22T01:47:52.681 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.000173446, 2024-03-22T01:47:52.681 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-22T01:47:52.681 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T01:47:52.681 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-22T01:47:52.681 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-22T01:47:52.682 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-22T01:47:52.682 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:52.682 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-03-22T01:47:52.682 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-22T01:47:52.682 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-22T01:47:52.682 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-22T01:47:52.682 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:52.682 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-22T01:47:52.682 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 0 2024-03-22T01:47:52.682 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:52.682 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-1": { 2024-03-22T01:47:52.682 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 243, 2024-03-22T01:47:52.682 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 231, 2024-03-22T01:47:52.682 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 1450944, 2024-03-22T01:47:52.682 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 141312, 2024-03-22T01:47:52.683 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 2, 2024-03-22T01:47:52.683 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 1, 2024-03-22T01:47:52.683 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.047605841, 2024-03-22T01:47:52.683 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.014570376, 2024-03-22T01:47:52.683 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.035512011, 2024-03-22T01:47:52.683 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.005402269, 2024-03-22T01:47:52.683 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T01:47:52.683 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 231, 2024-03-22T01:47:52.683 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.013974069, 2024-03-22T01:47:52.683 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000060493 2024-03-22T01:47:52.683 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:52.683 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-03-22T01:47:52.683 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-22T01:47:52.683 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-22T01:47:52.684 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-22T01:47:52.684 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:52.684 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 1450944, 2024-03-22T01:47:52.684 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 141312 2024-03-22T01:47:52.684 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:52.684 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-2": { 2024-03-22T01:47:52.684 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 2, 2024-03-22T01:47:52.684 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 175, 2024-03-22T01:47:52.684 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 320, 2024-03-22T01:47:52.684 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 186400, 2024-03-22T01:47:52.684 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 5, 2024-03-22T01:47:52.684 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 3, 2024-03-22T01:47:52.684 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.010575691, 2024-03-22T01:47:52.684 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.007108562, 2024-03-22T01:47:52.685 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.001300528, 2024-03-22T01:47:52.685 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-22T01:47:52.685 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T01:47:52.685 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 172, 2024-03-22T01:47:52.685 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.014365886, 2024-03-22T01:47:52.685 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000083522 2024-03-22T01:47:52.685 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:52.685 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-03-22T01:47:52.685 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-22T01:47:52.685 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-22T01:47:52.686 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-22T01:47:52.686 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:52.686 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-22T01:47:52.686 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 186400 2024-03-22T01:47:52.686 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:52.686 INFO:teuthology.orchestra.run.smithi098.stdout: "cct": { 2024-03-22T01:47:52.686 INFO:teuthology.orchestra.run.smithi098.stdout: "total_workers": 0, 2024-03-22T01:47:52.686 INFO:teuthology.orchestra.run.smithi098.stdout: "unhealthy_workers": 0 2024-03-22T01:47:52.686 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:52.686 INFO:teuthology.orchestra.run.smithi098.stdout: "mempool": { 2024-03-22T01:47:52.686 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_bytes": 0, 2024-03-22T01:47:52.686 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_items": 0, 2024-03-22T01:47:52.686 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_bytes": 0, 2024-03-22T01:47:52.686 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_items": 0, 2024-03-22T01:47:52.687 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_bytes": 0, 2024-03-22T01:47:52.687 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_items": 0, 2024-03-22T01:47:52.687 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-22T01:47:52.687 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_items": 0, 2024-03-22T01:47:52.687 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-22T01:47:52.687 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_items": 0, 2024-03-22T01:47:52.687 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_bytes": 0, 2024-03-22T01:47:52.687 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_items": 0, 2024-03-22T01:47:52.687 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-22T01:47:52.687 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_items": 0, 2024-03-22T01:47:52.689 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_bytes": 0, 2024-03-22T01:47:52.689 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_items": 0, 2024-03-22T01:47:52.689 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_bytes": 0, 2024-03-22T01:47:52.689 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_items": 0, 2024-03-22T01:47:52.689 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-22T01:47:52.689 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_items": 0, 2024-03-22T01:47:52.689 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-22T01:47:52.689 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_items": 0, 2024-03-22T01:47:52.690 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_bytes": 0, 2024-03-22T01:47:52.690 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_items": 0, 2024-03-22T01:47:52.690 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_bytes": 0, 2024-03-22T01:47:52.690 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_items": 0, 2024-03-22T01:47:52.690 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-22T01:47:52.690 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_items": 0, 2024-03-22T01:47:52.690 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_bytes": 0, 2024-03-22T01:47:52.690 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_items": 0, 2024-03-22T01:47:52.690 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_bytes": 0, 2024-03-22T01:47:52.691 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_items": 0, 2024-03-22T01:47:52.691 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_bytes": 0, 2024-03-22T01:47:52.691 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_items": 0, 2024-03-22T01:47:52.691 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_bytes": 0, 2024-03-22T01:47:52.691 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_items": 0, 2024-03-22T01:47:52.691 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_bytes": 148660, 2024-03-22T01:47:52.691 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_items": 29, 2024-03-22T01:47:52.691 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_bytes": 0, 2024-03-22T01:47:52.691 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_items": 0, 2024-03-22T01:47:52.691 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_bytes": 0, 2024-03-22T01:47:52.692 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_items": 0, 2024-03-22T01:47:52.692 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_bytes": 0, 2024-03-22T01:47:52.692 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_items": 0, 2024-03-22T01:47:52.692 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_bytes": 0, 2024-03-22T01:47:52.692 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_items": 0, 2024-03-22T01:47:52.692 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_bytes": 0, 2024-03-22T01:47:52.692 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_items": 0, 2024-03-22T01:47:52.692 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_bytes": 0, 2024-03-22T01:47:52.692 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_items": 0, 2024-03-22T01:47:52.692 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_bytes": 0, 2024-03-22T01:47:52.693 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_items": 0, 2024-03-22T01:47:52.693 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_bytes": 928, 2024-03-22T01:47:52.693 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_items": 2, 2024-03-22T01:47:52.693 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_bytes": 0, 2024-03-22T01:47:52.693 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_items": 0, 2024-03-22T01:47:52.693 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_bytes": 0, 2024-03-22T01:47:52.693 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_items": 0 2024-03-22T01:47:52.693 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-22T01:47:52.693 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-22T01:47:52.694 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-03-22T01:47:52.694 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 104857600, 2024-03-22T01:47:52.694 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-03-22T01:47:52.694 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 241, 2024-03-22T01:47:52.694 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 1414220, 2024-03-22T01:47:52.694 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-03-22T01:47:52.694 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 241, 2024-03-22T01:47:52.694 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-03-22T01:47:52.694 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-03-22T01:47:52.694 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 241, 2024-03-22T01:47:52.694 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 1414220, 2024-03-22T01:47:52.694 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-03-22T01:47:52.694 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-22T01:47:52.694 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-22T01:47:52.695 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-22T01:47:52.695 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-22T01:47:52.695 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-22T01:47:52.695 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-22T01:47:53.374 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.l 2024-03-22T01:47:53.375 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.l 2024-03-22T01:47:53.375 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.l 2024-03-22T01:47:53.375 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.l 2024-03-22T01:47:53.375 ERROR:teuthology.run_tasks:Manager failed: check-counter Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/git.ceph.com_ceph-c_c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/qa/tasks/check_counter.py", line 125, in end raise RuntimeError("The following counters failed to be set " RuntimeError: The following counters failed to be set on mds daemons: {'mds_server.req_rmsnap_latency.avgcount'} 2024-03-22T01:47:53.376 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-03-22T01:47:53.387 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-22T01:47:53.387 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-22T01:47:53.387 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-22T01:47:53.418 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-22T01:47:53.419 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-22T01:47:53.474 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T01:47:53.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.619+0000 7f61b5088640 1 -- 172.21.15.72:0/613428110 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f61b01013f0 msgr2=0x7f61b010d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:53.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.619+0000 7f61b5088640 1 --2- 172.21.15.72:0/613428110 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f61b01013f0 0x7f61b010d900 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f61a8009cd0 tx=0x7f61a802c0f0 comp rx=0 tx=0).stop 2024-03-22T01:47:53.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.620+0000 7f61b5088640 1 -- 172.21.15.72:0/613428110 shutdown_connections 2024-03-22T01:47:53.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.620+0000 7f61b5088640 1 --2- 172.21.15.72:0/613428110 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f61b01013f0 0x7f61b010d900 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:53.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.620+0000 7f61b5088640 1 --2- 172.21.15.72:0/613428110 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61b0100a30 0x7f61b0100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:53.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.620+0000 7f61b5088640 1 --2- 172.21.15.72:0/613428110 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61b0106a40 0x7f61b0106e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:53.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.620+0000 7f61b5088640 1 -- 172.21.15.72:0/613428110 >> 172.21.15.72:0/613428110 conn(0x7f61b0077b80 msgr2=0x7f61b0077f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:47:53.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.620+0000 7f61b5088640 1 -- 172.21.15.72:0/613428110 shutdown_connections 2024-03-22T01:47:53.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.620+0000 7f61b5088640 1 -- 172.21.15.72:0/613428110 wait complete. 2024-03-22T01:47:53.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.620+0000 7f61b5088640 1 Processor -- start 2024-03-22T01:47:53.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.620+0000 7f61b5088640 1 -- start start 2024-03-22T01:47:53.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.621+0000 7f61b5088640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61b0100a30 0x7f61b0198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:53.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.621+0000 7f61b5088640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f61b01013f0 0x7f61b0198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:53.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.621+0000 7f61b5088640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61b0106a40 0x7f61b019cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:53.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.621+0000 7f61b5088640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f61b0111f50 con 0x7f61b01013f0 2024-03-22T01:47:53.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.621+0000 7f61b5088640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f61b0111dd0 con 0x7f61b0100a30 2024-03-22T01:47:53.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.621+0000 7f61b5088640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f61b01120d0 con 0x7f61b0106a40 2024-03-22T01:47:53.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.621+0000 7f61ae59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f61b01013f0 0x7f61b0198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:53.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.621+0000 7f61aed9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61b0100a30 0x7f61b0198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:53.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.621+0000 7f61af59e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61b0106a40 0x7f61b019cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:53.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.621+0000 7f61ae59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f61b01013f0 0x7f61b0198950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:35906/0 (socket says 172.21.15.72:35906) 2024-03-22T01:47:53.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.621+0000 7f61ae59c640 1 -- 172.21.15.72:0/3077825787 learned_addr learned my addr 172.21.15.72:0/3077825787 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:47:53.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.621+0000 7f61ae59c640 1 -- 172.21.15.72:0/3077825787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61b0106a40 msgr2=0x7f61b019cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:53.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.621+0000 7f61ae59c640 1 --2- 172.21.15.72:0/3077825787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61b0106a40 0x7f61b019cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:53.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.621+0000 7f61ae59c640 1 -- 172.21.15.72:0/3077825787 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61b0100a30 msgr2=0x7f61b0198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:53.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.621+0000 7f61ae59c640 1 --2- 172.21.15.72:0/3077825787 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61b0100a30 0x7f61b0198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:53.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.621+0000 7f61ae59c640 1 -- 172.21.15.72:0/3077825787 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61b019d3b0 con 0x7f61b01013f0 2024-03-22T01:47:53.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.621+0000 7f61aed9d640 1 --2- 172.21.15.72:0/3077825787 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61b0100a30 0x7f61b0198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:47:53.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.621+0000 7f61ae59c640 1 --2- 172.21.15.72:0/3077825787 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f61b01013f0 0x7f61b0198950 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f61a000c890 tx=0x7f61a000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:47:53.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.621+0000 7f6197fff640 1 -- 172.21.15.72:0/3077825787 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61a0004220 con 0x7f61b01013f0 2024-03-22T01:47:53.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.621+0000 7f61b5088640 1 -- 172.21.15.72:0/3077825787 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61b019d6a0 con 0x7f61b01013f0 2024-03-22T01:47:53.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.622+0000 7f6197fff640 1 -- 172.21.15.72:0/3077825787 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61a00043c0 con 0x7f61b01013f0 2024-03-22T01:47:53.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.622+0000 7f6197fff640 1 -- 172.21.15.72:0/3077825787 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61a0004a00 con 0x7f61b01013f0 2024-03-22T01:47:53.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.622+0000 7f61b5088640 1 -- 172.21.15.72:0/3077825787 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61b01a4f30 con 0x7f61b01013f0 2024-03-22T01:47:53.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.623+0000 7f61b5088640 1 -- 172.21.15.72:0/3077825787 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61b0068180 con 0x7f61b01013f0 2024-03-22T01:47:53.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.624+0000 7f6197fff640 1 -- 172.21.15.72:0/3077825787 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 25) ==== 137985+0+0 (secure 0 0 0) 0x7f61a00079b0 con 0x7f61b01013f0 2024-03-22T01:47:53.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.625+0000 7f6197fff640 1 --2- 172.21.15.72:0/3077825787 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f61980a3090 0x7f61980a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:53.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.625+0000 7f61aed9d640 1 --2- 172.21.15.72:0/3077825787 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f61980a3090 0x7f61980a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:53.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.625+0000 7f6197fff640 1 -- 172.21.15.72:0/3077825787 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f61a00cf7e0 con 0x7f61b01013f0 2024-03-22T01:47:53.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.625+0000 7f61aed9d640 1 --2- 172.21.15.72:0/3077825787 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f61980a3090 0x7f61980a5550 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f61a4009f40 tx=0x7f61a4005f90 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:47:53.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.630+0000 7f6197fff640 1 -- 172.21.15.72:0/3077825787 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f61a0014030 con 0x7f61b01013f0 2024-03-22T01:47:53.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:53 smithi098 ceph-mon[36468]: pgmap v340: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 2 op/s 2024-03-22T01:47:53.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.812+0000 7f61b5088640 1 -- 172.21.15.72:0/3077825787 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f61b0108a10 con 0x7f61b01013f0 2024-03-22T01:47:53.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.813+0000 7f6197fff640 1 -- 172.21.15.72:0/3077825787 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) ==== 74+0+25134 (secure 0 0 0) 0x7f61a009b8d0 con 0x7f61b01013f0 2024-03-22T01:47:53.814 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:47:53.814 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":96,"fsid":"022a99bc-e7eb-11ee-95cd-87774f69a715","created":"2024-03-22T01:26:23.893672+0000","modified":"2024-03-22T01:36:38.183837+0000","last_up_change":"2024-03-22T01:33:25.223684+0000","last_in_change":"2024-03-22T01:33:05.010331+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T01:29:37.438503+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-22T01:34:12.610422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-22T01:34:13.761686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-22T01:34:17.507039+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"16c15bb0-db92-4191-9063-38b4890e81e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6803","nonce":661906791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6805","nonce":661906791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6809","nonce":661906791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6807","nonce":661906791}]},"public_addr":"172.21.15.72:6803/661906791","cluster_addr":"172.21.15.72:6805/661906791","heartbeat_back_addr":"172.21.15.72:6809/661906791","heartbeat_front_addr":"172.21.15.72:6807/661906791","state":["exists","up"]},{"osd":1,"uuid":"981290ba-97dc-496b-aa22-d51704184b10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6801","nonce":2870590550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6803","nonce":2870590550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6807","nonce":2870590550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6805","nonce":2870590550}]},"public_addr":"172.21.15.89:6801/2870590550","cluster_addr":"172.21.15.89:6803/2870590550","heartbeat_back_addr":"172.21.15.89:6807/2870590550","heartbeat_front_addr":"172.21.15.89:6805/2870590550","state":["exists","up"]},{"osd":2,"uuid":"f07ed658-f545-46a1-8edd-8b1d5d49885c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6801","nonce":3865778118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6803","nonce":3865778118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6807","nonce":3865778118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6805","nonce":3865778118}]},"public_addr":"172.21.15.98:6801/3865778118","cluster_addr":"172.21.15.98:6803/3865778118","heartbeat_back_addr":"172.21.15.98:6807/3865778118","heartbeat_front_addr":"172.21.15.98:6805/3865778118","state":["exists","up"]},{"osd":3,"uuid":"ccc4bc40-2c48-46d2-8548-0c05b44f9f1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6811","nonce":1865951395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6813","nonce":1865951395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6817","nonce":1865951395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6815","nonce":1865951395}]},"public_addr":"172.21.15.72:6811/1865951395","cluster_addr":"172.21.15.72:6813/1865951395","heartbeat_back_addr":"172.21.15.72:6817/1865951395","heartbeat_front_addr":"172.21.15.72:6815/1865951395","state":["exists","up"]},{"osd":4,"uuid":"1812439f-810a-4ed2-9395-c00c9a085a13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6809","nonce":2683760795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6811","nonce":2683760795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6815","nonce":2683760795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6813","nonce":2683760795}]},"public_addr":"172.21.15.89:6809/2683760795","cluster_addr":"172.21.15.89:6811/2683760795","heartbeat_back_addr":"172.21.15.89:6815/2683760795","heartbeat_front_addr":"172.21.15.89:6813/2683760795","state":["exists","up"]},{"osd":5,"uuid":"e78954d8-f4e3-4d58-8536-d2d59c843386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6809","nonce":2635838572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6811","nonce":2635838572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6815","nonce":2635838572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6813","nonce":2635838572}]},"public_addr":"172.21.15.98:6809/2635838572","cluster_addr":"172.21.15.98:6811/2635838572","heartbeat_back_addr":"172.21.15.98:6815/2635838572","heartbeat_front_addr":"172.21.15.98:6813/2635838572","state":["exists","up"]},{"osd":6,"uuid":"dc6c8be2-b64c-4959-8015-7b0ea6d25599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6819","nonce":3158982661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6821","nonce":3158982661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6825","nonce":3158982661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6823","nonce":3158982661}]},"public_addr":"172.21.15.72:6819/3158982661","cluster_addr":"172.21.15.72:6821/3158982661","heartbeat_back_addr":"172.21.15.72:6825/3158982661","heartbeat_front_addr":"172.21.15.72:6823/3158982661","state":["exists","up"]},{"osd":7,"uuid":"7a73469f-cf3a-4306-ae93-ea46e21ebbbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6817","nonce":3075913234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6819","nonce":3075913234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6823","nonce":3075913234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6821","nonce":3075913234}]},"public_addr":"172.21.15.89:6817/3075913234","cluster_addr":"172.21.15.89:6819/3075913234","heartbeat_back_addr":"172.21.15.89:6823/3075913234","heartbeat_front_addr":"172.21.15.89:6821/3075913234","state":["exists","up"]},{"osd":8,"uuid":"c4b13bee-ea4d-4344-95fb-2622ecbb8196","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6817","nonce":1813255577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6819","nonce":1813255577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6823","nonce":1813255577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6821","nonce":1813255577}]},"public_addr":"172.21.15.98:6817/1813255577","cluster_addr":"172.21.15.98:6819/1813255577","heartbeat_back_addr":"172.21.15.98:6823/1813255577","heartbeat_front_addr":"172.21.15.98:6821/1813255577","state":["exists","up"]},{"osd":9,"uuid":"eefb0a44-4a36-4e55-9d04-0501bf718db6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6827","nonce":3377749206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6829","nonce":3377749206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6832","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6833","nonce":3377749206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6831","nonce":3377749206}]},"public_addr":"172.21.15.72:6827/3377749206","cluster_addr":"172.21.15.72:6829/3377749206","heartbeat_back_addr":"172.21.15.72:6833/3377749206","heartbeat_front_addr":"172.21.15.72:6831/3377749206","state":["exists","up"]},{"osd":10,"uuid":"8741d25f-40dd-40d7-b040-5cc5e411eb98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6825","nonce":885058786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6827","nonce":885058786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6831","nonce":885058786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6829","nonce":885058786}]},"public_addr":"172.21.15.89:6825/885058786","cluster_addr":"172.21.15.89:6827/885058786","heartbeat_back_addr":"172.21.15.89:6831/885058786","heartbeat_front_addr":"172.21.15.89:6829/885058786","state":["exists","up"]},{"osd":11,"uuid":"8d02e957-eda4-4184-b5c6-5eb3b8b38060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6825","nonce":1517095838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6827","nonce":1517095838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6831","nonce":1517095838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6829","nonce":1517095838}]},"public_addr":"172.21.15.98:6825/1517095838","cluster_addr":"172.21.15.98:6827/1517095838","heartbeat_back_addr":"172.21.15.98:6831/1517095838","heartbeat_front_addr":"172.21.15.98:6829/1517095838","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:28:47.307652+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:11.037135+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:34.619188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:59.350738+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:30:23.787337+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:30:48.994154+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:31:13.690350+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:31:39.628116+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:04.723912+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:30.689741+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:57.229508+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:33:23.154537+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":7}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:0/2032770149":"2024-03-23T01:36:38.183806+0000","172.21.15.72:6801/401403200":"2024-03-23T01:36:38.183806+0000","172.21.15.72:0/1161486989":"2024-03-23T01:36:38.183806+0000","172.21.15.72:6800/401403200":"2024-03-23T01:36:38.183806+0000","172.21.15.72:0/121087196":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/2451857761":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/1228477995":"2024-03-23T01:27:15.339809+0000","172.21.15.72:6801/1513232913":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/4223447038":"2024-03-23T01:36:38.183806+0000","172.21.15.72:0/3558958264":"2024-03-23T01:26:50.074871+0000","172.21.15.72:0/1510187198":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6800/3518012916":"2024-03-23T01:26:50.074871+0000","172.21.15.72:0/608566205":"2024-03-23T01:36:38.183806+0000","172.21.15.72:6801/3518012916":"2024-03-23T01:26:50.074871+0000","172.21.15.72:0/1372801041":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6800/1513232913":"2024-03-23T01:27:15.339809+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T01:47:53.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.815+0000 7f61b5088640 1 -- 172.21.15.72:0/3077825787 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f61980a3090 msgr2=0x7f61980a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:53.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.815+0000 7f61b5088640 1 --2- 172.21.15.72:0/3077825787 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f61980a3090 0x7f61980a5550 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f61a4009f40 tx=0x7f61a4005f90 comp rx=0 tx=0).stop 2024-03-22T01:47:53.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.815+0000 7f61b5088640 1 -- 172.21.15.72:0/3077825787 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f61b01013f0 msgr2=0x7f61b0198950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:53.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.815+0000 7f61b5088640 1 --2- 172.21.15.72:0/3077825787 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f61b01013f0 0x7f61b0198950 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f61a000c890 tx=0x7f61a000cd50 comp rx=0 tx=0).stop 2024-03-22T01:47:53.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.815+0000 7f61b5088640 1 -- 172.21.15.72:0/3077825787 shutdown_connections 2024-03-22T01:47:53.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.815+0000 7f61b5088640 1 --2- 172.21.15.72:0/3077825787 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f61980a3090 0x7f61980a5550 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:53.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.815+0000 7f61b5088640 1 --2- 172.21.15.72:0/3077825787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61b0106a40 0x7f61b019cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:53.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.815+0000 7f61b5088640 1 --2- 172.21.15.72:0/3077825787 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f61b01013f0 0x7f61b0198950 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:53.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.815+0000 7f61b5088640 1 --2- 172.21.15.72:0/3077825787 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61b0100a30 0x7f61b0198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:53.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.815+0000 7f61b5088640 1 -- 172.21.15.72:0/3077825787 >> 172.21.15.72:0/3077825787 conn(0x7f61b0077b80 msgr2=0x7f61b0103270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:47:53.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.816+0000 7f61b5088640 1 -- 172.21.15.72:0/3077825787 shutdown_connections 2024-03-22T01:47:53.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.816+0000 7f61b5088640 1 -- 172.21.15.72:0/3077825787 wait complete. 2024-03-22T01:47:53.824 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:53 smithi072 ceph-mon[32414]: pgmap v340: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 2 op/s 2024-03-22T01:47:53.826 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-22T01:29:37.438503+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-22T01:34:12.610422+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-22T01:34:13.761686+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-22T01:34:17.507039+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-22T01:47:53.827 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-22T01:47:53.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:53 smithi089 ceph-mon[35770]: pgmap v340: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 2 op/s 2024-03-22T01:47:53.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.935+0000 7ff63aef4640 1 -- 172.21.15.72:0/796220808 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff634101b40 msgr2=0x7ff634101fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:53.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.935+0000 7ff63aef4640 1 --2- 172.21.15.72:0/796220808 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff634101b40 0x7ff634101fc0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7ff628009900 tx=0x7ff62802bc30 comp rx=0 tx=0).stop 2024-03-22T01:47:53.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.935+0000 7ff63aef4640 1 -- 172.21.15.72:0/796220808 shutdown_connections 2024-03-22T01:47:53.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.935+0000 7ff63aef4640 1 --2- 172.21.15.72:0/796220808 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff634102500 0x7ff63410e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:53.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.935+0000 7ff63aef4640 1 --2- 172.21.15.72:0/796220808 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff634101b40 0x7ff634101fc0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:53.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.935+0000 7ff63aef4640 1 --2- 172.21.15.72:0/796220808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff634107c00 0x7ff634107fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:53.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.935+0000 7ff63aef4640 1 -- 172.21.15.72:0/796220808 >> 172.21.15.72:0/796220808 conn(0x7ff6340fd630 msgr2=0x7ff6340ffa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:47:53.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.935+0000 7ff63aef4640 1 -- 172.21.15.72:0/796220808 shutdown_connections 2024-03-22T01:47:53.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.935+0000 7ff63aef4640 1 -- 172.21.15.72:0/796220808 wait complete. 2024-03-22T01:47:53.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.935+0000 7ff63aef4640 1 Processor -- start 2024-03-22T01:47:53.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.935+0000 7ff63aef4640 1 -- start start 2024-03-22T01:47:53.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.935+0000 7ff63aef4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff634101b40 0x7ff634073e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:53.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.935+0000 7ff63aef4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff634102500 0x7ff6340743d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:53.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.935+0000 7ff63aef4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff634107c00 0x7ff634074910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:53.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.935+0000 7ff63aef4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7ff634111f00 con 0x7ff634102500 2024-03-22T01:47:53.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.935+0000 7ff63aef4640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff634111d80 con 0x7ff634107c00 2024-03-22T01:47:53.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.935+0000 7ff63aef4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff634112080 con 0x7ff634101b40 2024-03-22T01:47:53.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.935+0000 7ff633fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff634102500 0x7ff6340743d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:53.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.935+0000 7ff633fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff634102500 0x7ff6340743d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:35922/0 (socket says 172.21.15.72:35922) 2024-03-22T01:47:53.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.936+0000 7ff633fff640 1 -- 172.21.15.72:0/3459975124 learned_addr learned my addr 172.21.15.72:0/3459975124 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:47:53.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.936+0000 7ff639491640 1 --2- 172.21.15.72:0/3459975124 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff634107c00 0x7ff634074910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:53.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.936+0000 7ff638c90640 1 --2- 172.21.15.72:0/3459975124 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff634101b40 0x7ff634073e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:53.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.936+0000 7ff639491640 1 -- 172.21.15.72:0/3459975124 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff634101b40 msgr2=0x7ff634073e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:53.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.936+0000 7ff639491640 1 --2- 172.21.15.72:0/3459975124 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff634101b40 0x7ff634073e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:53.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.936+0000 7ff639491640 1 -- 172.21.15.72:0/3459975124 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff634102500 msgr2=0x7ff6340743d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:53.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.936+0000 7ff639491640 1 --2- 172.21.15.72:0/3459975124 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff634102500 0x7ff6340743d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:53.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.936+0000 7ff639491640 1 -- 172.21.15.72:0/3459975124 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6341a6900 con 0x7ff634107c00 2024-03-22T01:47:53.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.936+0000 7ff633fff640 1 --2- 172.21.15.72:0/3459975124 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff634102500 0x7ff6340743d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:47:53.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.936+0000 7ff639491640 1 --2- 172.21.15.72:0/3459975124 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff634107c00 0x7ff634074910 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff62c00ca30 tx=0x7ff62c00cef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:47:53.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.936+0000 7ff631ffb640 1 -- 172.21.15.72:0/3459975124 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff62c007d90 con 0x7ff634107c00 2024-03-22T01:47:53.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.936+0000 7ff63aef4640 1 -- 172.21.15.72:0/3459975124 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6341a6bf0 con 0x7ff634107c00 2024-03-22T01:47:53.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.936+0000 7ff631ffb640 1 -- 172.21.15.72:0/3459975124 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff62c005020 con 0x7ff634107c00 2024-03-22T01:47:53.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.936+0000 7ff631ffb640 1 -- 172.21.15.72:0/3459975124 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff62c00f780 con 0x7ff634107c00 2024-03-22T01:47:53.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.936+0000 7ff63aef4640 1 -- 172.21.15.72:0/3459975124 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6341a70c0 con 0x7ff634107c00 2024-03-22T01:47:53.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.937+0000 7ff63aef4640 1 -- 172.21.15.72:0/3459975124 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff634068e60 con 0x7ff634107c00 2024-03-22T01:47:53.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.938+0000 7ff631ffb640 1 -- 172.21.15.72:0/3459975124 <== mon.2 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 25) ==== 137985+0+0 (secure 0 0 0) 0x7ff62c0051d0 con 0x7ff634107c00 2024-03-22T01:47:53.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.939+0000 7ff631ffb640 1 --2- 172.21.15.72:0/3459975124 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7ff61c0a3090 0x7ff61c0a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:53.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.939+0000 7ff638c90640 1 --2- 172.21.15.72:0/3459975124 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7ff61c0a3090 0x7ff61c0a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:53.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.939+0000 7ff631ffb640 1 -- 172.21.15.72:0/3459975124 <== mon.2 v2:172.21.15.89:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7ff62c0cfda0 con 0x7ff634107c00 2024-03-22T01:47:53.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.939+0000 7ff638c90640 1 --2- 172.21.15.72:0/3459975124 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7ff61c0a3090 0x7ff61c0a5550 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ff624009b50 tx=0x7ff624005f90 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:47:53.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:53.944+0000 7ff631ffb640 1 -- 172.21.15.72:0/3459975124 <== mon.2 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff62c014070 con 0x7ff634107c00 2024-03-22T01:47:54.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.118+0000 7ff63aef4640 1 -- 172.21.15.72:0/3459975124 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff634109a80 con 0x7ff634107c00 2024-03-22T01:47:54.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.119+0000 7ff631ffb640 1 -- 172.21.15.72:0/3459975124 <== mon.2 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v96) ==== 93+0+10 (secure 0 0 0) 0x7ff62c09be90 con 0x7ff634107c00 2024-03-22T01:47:54.118 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-22T01:47:54.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.121+0000 7ff63aef4640 1 -- 172.21.15.72:0/3459975124 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7ff61c0a3090 msgr2=0x7ff61c0a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:54.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.121+0000 7ff63aef4640 1 --2- 172.21.15.72:0/3459975124 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7ff61c0a3090 0x7ff61c0a5550 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ff624009b50 tx=0x7ff624005f90 comp rx=0 tx=0).stop 2024-03-22T01:47:54.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.121+0000 7ff63aef4640 1 -- 172.21.15.72:0/3459975124 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff634107c00 msgr2=0x7ff634074910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:54.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.121+0000 7ff63aef4640 1 --2- 172.21.15.72:0/3459975124 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff634107c00 0x7ff634074910 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff62c00ca30 tx=0x7ff62c00cef0 comp rx=0 tx=0).stop 2024-03-22T01:47:54.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.121+0000 7ff63aef4640 1 -- 172.21.15.72:0/3459975124 shutdown_connections 2024-03-22T01:47:54.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.121+0000 7ff63aef4640 1 --2- 172.21.15.72:0/3459975124 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7ff61c0a3090 0x7ff61c0a5550 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:54.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.121+0000 7ff63aef4640 1 --2- 172.21.15.72:0/3459975124 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff634107c00 0x7ff634074910 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:54.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.121+0000 7ff63aef4640 1 --2- 172.21.15.72:0/3459975124 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff634102500 0x7ff6340743d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:54.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.121+0000 7ff63aef4640 1 --2- 172.21.15.72:0/3459975124 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff634101b40 0x7ff634073e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:54.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.121+0000 7ff63aef4640 1 -- 172.21.15.72:0/3459975124 >> 172.21.15.72:0/3459975124 conn(0x7ff6340fd630 msgr2=0x7ff6340fee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:47:54.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.121+0000 7ff63aef4640 1 -- 172.21.15.72:0/3459975124 shutdown_connections 2024-03-22T01:47:54.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.121+0000 7ff63aef4640 1 -- 172.21.15.72:0/3459975124 wait complete. 2024-03-22T01:47:54.131 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-22T01:47:54.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.237+0000 7f8fe9c9a640 1 -- 172.21.15.72:0/3586414083 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8fe4100a30 msgr2=0x7f8fe4100eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:54.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.237+0000 7f8fe9c9a640 1 --2- 172.21.15.72:0/3586414083 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8fe4100a30 0x7f8fe4100eb0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc009930 tx=0x7f8fcc02bc30 comp rx=0 tx=0).stop 2024-03-22T01:47:54.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.237+0000 7f8fe9c9a640 1 -- 172.21.15.72:0/3586414083 shutdown_connections 2024-03-22T01:47:54.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.237+0000 7f8fe9c9a640 1 --2- 172.21.15.72:0/3586414083 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fe41013f0 0x7f8fe410d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:54.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.237+0000 7f8fe9c9a640 1 --2- 172.21.15.72:0/3586414083 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8fe4100a30 0x7f8fe4100eb0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:54.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.237+0000 7f8fe9c9a640 1 --2- 172.21.15.72:0/3586414083 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8fe4106a40 0x7f8fe4106e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:54.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.237+0000 7f8fe9c9a640 1 -- 172.21.15.72:0/3586414083 >> 172.21.15.72:0/3586414083 conn(0x7f8fe4077c40 msgr2=0x7f8fe4078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:47:54.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.237+0000 7f8fe9c9a640 1 -- 172.21.15.72:0/3586414083 shutdown_connections 2024-03-22T01:47:54.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.237+0000 7f8fe9c9a640 1 -- 172.21.15.72:0/3586414083 wait complete. 2024-03-22T01:47:54.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.238+0000 7f8fe9c9a640 1 Processor -- start 2024-03-22T01:47:54.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.238+0000 7f8fe9c9a640 1 -- start start 2024-03-22T01:47:54.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.238+0000 7f8fe9c9a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8fe4100a30 0x7f8fe4199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:54.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.238+0000 7f8fe9c9a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fe41013f0 0x7f8fe4199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:54.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.238+0000 7f8fe9c9a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8fe4106a40 0x7f8fe4199df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:54.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.238+0000 7f8fe9c9a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f8fe4110e50 con 0x7f8fe4106a40 2024-03-22T01:47:54.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.238+0000 7f8fe9c9a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8fe4110cd0 con 0x7f8fe41013f0 2024-03-22T01:47:54.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.238+0000 7f8fe9c9a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8fe4110fd0 con 0x7f8fe4100a30 2024-03-22T01:47:54.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.239+0000 7f8fe3fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8fe4106a40 0x7f8fe4199df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:54.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.239+0000 7f8fdaffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fe41013f0 0x7f8fe4199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:54.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.239+0000 7f8fe3fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8fe4106a40 0x7f8fe4199df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:35946/0 (socket says 172.21.15.72:35946) 2024-03-22T01:47:54.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.239+0000 7f8fe3fff640 1 -- 172.21.15.72:0/3575940520 learned_addr learned my addr 172.21.15.72:0/3575940520 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:47:54.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.239+0000 7f8fe37fe640 1 --2- 172.21.15.72:0/3575940520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8fe4100a30 0x7f8fe4199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:54.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.239+0000 7f8fe3fff640 1 -- 172.21.15.72:0/3575940520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8fe4100a30 msgr2=0x7f8fe4199330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:54.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.239+0000 7f8fe3fff640 1 --2- 172.21.15.72:0/3575940520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8fe4100a30 0x7f8fe4199330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:54.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.239+0000 7f8fe3fff640 1 -- 172.21.15.72:0/3575940520 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fe41013f0 msgr2=0x7f8fe4199870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:54.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.239+0000 7f8fe3fff640 1 --2- 172.21.15.72:0/3575940520 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fe41013f0 0x7f8fe4199870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:54.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.239+0000 7f8fe3fff640 1 -- 172.21.15.72:0/3575940520 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fe41a4730 con 0x7f8fe4106a40 2024-03-22T01:47:54.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.239+0000 7f8fdaffd640 1 --2- 172.21.15.72:0/3575940520 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fe41013f0 0x7f8fe4199870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:47:54.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.239+0000 7f8fe3fff640 1 --2- 172.21.15.72:0/3575940520 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8fe4106a40 0x7f8fe4199df0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc00ea60 tx=0x7f8fdc00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:47:54.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.239+0000 7f8fe17fa640 1 -- 172.21.15.72:0/3575940520 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fdc00ce40 con 0x7f8fe4106a40 2024-03-22T01:47:54.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.239+0000 7f8fe17fa640 1 -- 172.21.15.72:0/3575940520 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8fdc005020 con 0x7f8fe4106a40 2024-03-22T01:47:54.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.239+0000 7f8fe9c9a640 1 -- 172.21.15.72:0/3575940520 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fe41a4a20 con 0x7f8fe4106a40 2024-03-22T01:47:54.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.239+0000 7f8fe17fa640 1 -- 172.21.15.72:0/3575940520 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fdc010690 con 0x7f8fe4106a40 2024-03-22T01:47:54.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.239+0000 7f8fe9c9a640 1 -- 172.21.15.72:0/3575940520 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fe41a4ef0 con 0x7f8fe4106a40 2024-03-22T01:47:54.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.240+0000 7f8fe9c9a640 1 -- 172.21.15.72:0/3575940520 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fe4102170 con 0x7f8fe4106a40 2024-03-22T01:47:54.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.241+0000 7f8fe17fa640 1 -- 172.21.15.72:0/3575940520 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 25) ==== 137985+0+0 (secure 0 0 0) 0x7f8fdc005520 con 0x7f8fe4106a40 2024-03-22T01:47:54.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.242+0000 7f8fe17fa640 1 --2- 172.21.15.72:0/3575940520 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f8fd00a3090 0x7f8fd00a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:54.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.242+0000 7f8fe37fe640 1 --2- 172.21.15.72:0/3575940520 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f8fd00a3090 0x7f8fd00a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:54.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.242+0000 7f8fe17fa640 1 -- 172.21.15.72:0/3575940520 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f8fdc0d0100 con 0x7f8fe4106a40 2024-03-22T01:47:54.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.242+0000 7f8fe37fe640 1 --2- 172.21.15.72:0/3575940520 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f8fd00a3090 0x7f8fd00a5550 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f8fd4009ae0 tx=0x7f8fd4005f90 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:47:54.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.247+0000 7f8fe17fa640 1 -- 172.21.15.72:0/3575940520 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8fdc09c840 con 0x7f8fe4106a40 2024-03-22T01:47:54.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.421+0000 7f8fe9c9a640 1 -- 172.21.15.72:0/3575940520 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f8fe402d3b0 con 0x7f8fe4106a40 2024-03-22T01:47:54.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.422+0000 7f8fe17fa640 1 -- 172.21.15.72:0/3575940520 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v96) ==== 104+0+11 (secure 0 0 0) 0x7f8fdc09c140 con 0x7f8fe4106a40 2024-03-22T01:47:54.421 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-03-22T01:47:54.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.424+0000 7f8fe9c9a640 1 -- 172.21.15.72:0/3575940520 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f8fd00a3090 msgr2=0x7f8fd00a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:54.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.424+0000 7f8fe9c9a640 1 --2- 172.21.15.72:0/3575940520 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f8fd00a3090 0x7f8fd00a5550 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f8fd4009ae0 tx=0x7f8fd4005f90 comp rx=0 tx=0).stop 2024-03-22T01:47:54.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.424+0000 7f8fe9c9a640 1 -- 172.21.15.72:0/3575940520 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8fe4106a40 msgr2=0x7f8fe4199df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:54.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.424+0000 7f8fe9c9a640 1 --2- 172.21.15.72:0/3575940520 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8fe4106a40 0x7f8fe4199df0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc00ea60 tx=0x7f8fdc00ef20 comp rx=0 tx=0).stop 2024-03-22T01:47:54.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.424+0000 7f8fe9c9a640 1 -- 172.21.15.72:0/3575940520 shutdown_connections 2024-03-22T01:47:54.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.424+0000 7f8fe9c9a640 1 --2- 172.21.15.72:0/3575940520 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f8fd00a3090 0x7f8fd00a5550 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:54.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.424+0000 7f8fe9c9a640 1 --2- 172.21.15.72:0/3575940520 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fe41013f0 0x7f8fe4199870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:54.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.424+0000 7f8fe9c9a640 1 --2- 172.21.15.72:0/3575940520 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8fe4106a40 0x7f8fe4199df0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:54.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.424+0000 7f8fe9c9a640 1 --2- 172.21.15.72:0/3575940520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8fe4100a30 0x7f8fe4199330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:54.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.424+0000 7f8fe9c9a640 1 -- 172.21.15.72:0/3575940520 >> 172.21.15.72:0/3575940520 conn(0x7f8fe4077c40 msgr2=0x7f8fe4103600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:47:54.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.424+0000 7f8fe9c9a640 1 -- 172.21.15.72:0/3575940520 shutdown_connections 2024-03-22T01:47:54.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.424+0000 7f8fe9c9a640 1 -- 172.21.15.72:0/3575940520 wait complete. 2024-03-22T01:47:54.434 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-22T01:47:54.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.542+0000 7fabfe13a640 1 -- 172.21.15.72:0/1442065537 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fabf8101e80 msgr2=0x7fabf8102260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:54.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.542+0000 7fabfe13a640 1 --2- 172.21.15.72:0/1442065537 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fabf8101e80 0x7fabf8102260 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fabe0009900 tx=0x7fabe002bcd0 comp rx=0 tx=0).stop 2024-03-22T01:47:54.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.542+0000 7fabfe13a640 1 -- 172.21.15.72:0/1442065537 shutdown_connections 2024-03-22T01:47:54.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.542+0000 7fabfe13a640 1 --2- 172.21.15.72:0/1442065537 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabf8103e40 0x7fabf810ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:54.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.542+0000 7fabfe13a640 1 --2- 172.21.15.72:0/1442065537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fabf81027a0 0x7fabf8103710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:54.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.542+0000 7fabfe13a640 1 --2- 172.21.15.72:0/1442065537 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fabf8101e80 0x7fabf8102260 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:54.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.542+0000 7fabfe13a640 1 -- 172.21.15.72:0/1442065537 >> 172.21.15.72:0/1442065537 conn(0x7fabf80fd620 msgr2=0x7fabf80ffa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:47:54.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.542+0000 7fabfe13a640 1 -- 172.21.15.72:0/1442065537 shutdown_connections 2024-03-22T01:47:54.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.542+0000 7fabfe13a640 1 -- 172.21.15.72:0/1442065537 wait complete. 2024-03-22T01:47:54.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.543+0000 7fabfe13a640 1 Processor -- start 2024-03-22T01:47:54.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.543+0000 7fabfe13a640 1 -- start start 2024-03-22T01:47:54.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.543+0000 7fabfe13a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fabf8101e80 0x7fabf81983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:54.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.543+0000 7fabfe13a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fabf8103e40 0x7fabf8198930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:54.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.543+0000 7fabfe13a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabf819ccb0 0x7fabf819d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:54.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.543+0000 7fabfe13a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fabf8112030 con 0x7fabf8101e80 2024-03-22T01:47:54.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.543+0000 7fabfe13a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fabf8111eb0 con 0x7fabf819ccb0 2024-03-22T01:47:54.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.543+0000 7fabfe13a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fabf81121b0 con 0x7fabf8103e40 2024-03-22T01:47:54.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.543+0000 7fabf77fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fabf8101e80 0x7fabf81983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:54.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.543+0000 7fabf77fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fabf8101e80 0x7fabf81983f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:35964/0 (socket says 172.21.15.72:35964) 2024-03-22T01:47:54.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.543+0000 7fabf77fe640 1 -- 172.21.15.72:0/2432592393 learned_addr learned my addr 172.21.15.72:0/2432592393 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:47:54.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.543+0000 7fabf7fff640 1 --2- 172.21.15.72:0/2432592393 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabf819ccb0 0x7fabf819d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:54.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.544+0000 7fabf6ffd640 1 --2- 172.21.15.72:0/2432592393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fabf8103e40 0x7fabf8198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:54.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.544+0000 7fabf77fe640 1 -- 172.21.15.72:0/2432592393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fabf8103e40 msgr2=0x7fabf8198930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:54.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.544+0000 7fabf77fe640 1 --2- 172.21.15.72:0/2432592393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fabf8103e40 0x7fabf8198930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:54.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.544+0000 7fabf77fe640 1 -- 172.21.15.72:0/2432592393 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabf819ccb0 msgr2=0x7fabf819d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:54.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.544+0000 7fabf77fe640 1 --2- 172.21.15.72:0/2432592393 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabf819ccb0 0x7fabf819d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:54.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.544+0000 7fabf77fe640 1 -- 172.21.15.72:0/2432592393 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabf819d7a0 con 0x7fabf8101e80 2024-03-22T01:47:54.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.544+0000 7fabf6ffd640 1 --2- 172.21.15.72:0/2432592393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fabf8103e40 0x7fabf8198930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T01:47:54.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.544+0000 7fabf77fe640 1 --2- 172.21.15.72:0/2432592393 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fabf8101e80 0x7fabf81983f0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fabe0006010 tx=0x7fabe00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:47:54.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.544+0000 7fabf4ff9640 1 -- 172.21.15.72:0/2432592393 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabe0042020 con 0x7fabf8101e80 2024-03-22T01:47:54.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.544+0000 7fabfe13a640 1 -- 172.21.15.72:0/2432592393 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabf819da30 con 0x7fabf8101e80 2024-03-22T01:47:54.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.544+0000 7fabf4ff9640 1 -- 172.21.15.72:0/2432592393 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fabe00054e0 con 0x7fabf8101e80 2024-03-22T01:47:54.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.544+0000 7fabf4ff9640 1 -- 172.21.15.72:0/2432592393 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabe0003af0 con 0x7fabf8101e80 2024-03-22T01:47:54.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.544+0000 7fabfe13a640 1 -- 172.21.15.72:0/2432592393 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabf81a51f0 con 0x7fabf8101e80 2024-03-22T01:47:54.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.545+0000 7fabfe13a640 1 -- 172.21.15.72:0/2432592393 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabf8068e60 con 0x7fabf8101e80 2024-03-22T01:47:54.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.546+0000 7fabf4ff9640 1 -- 172.21.15.72:0/2432592393 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 25) ==== 137985+0+0 (secure 0 0 0) 0x7fabe0045020 con 0x7fabf8101e80 2024-03-22T01:47:54.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.547+0000 7fabf4ff9640 1 --2- 172.21.15.72:0/2432592393 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fabe40a3090 0x7fabe40a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:54.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.547+0000 7fabf6ffd640 1 --2- 172.21.15.72:0/2432592393 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fabe40a3090 0x7fabe40a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:54.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.547+0000 7fabf4ff9640 1 -- 172.21.15.72:0/2432592393 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fabe00f1e60 con 0x7fabf8101e80 2024-03-22T01:47:54.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.547+0000 7fabf6ffd640 1 --2- 172.21.15.72:0/2432592393 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fabe40a3090 0x7fabe40a5550 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fabf8108410 tx=0x7fabe8005d90 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:47:54.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.550+0000 7fabf4ff9640 1 -- 172.21.15.72:0/2432592393 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fabe004c020 con 0x7fabf8101e80 2024-03-22T01:47:54.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.727+0000 7fabfe13a640 1 -- 172.21.15.72:0/2432592393 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fabf810a150 con 0x7fabf8101e80 2024-03-22T01:47:54.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.727+0000 7fabf4ff9640 1 -- 172.21.15.72:0/2432592393 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v96) ==== 100+0+11 (secure 0 0 0) 0x7fabe00be690 con 0x7fabf8101e80 2024-03-22T01:47:54.727 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-03-22T01:47:54.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.730+0000 7fabfe13a640 1 -- 172.21.15.72:0/2432592393 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fabe40a3090 msgr2=0x7fabe40a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:54.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.730+0000 7fabfe13a640 1 --2- 172.21.15.72:0/2432592393 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fabe40a3090 0x7fabe40a5550 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fabf8108410 tx=0x7fabe8005d90 comp rx=0 tx=0).stop 2024-03-22T01:47:54.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.730+0000 7fabfe13a640 1 -- 172.21.15.72:0/2432592393 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fabf8101e80 msgr2=0x7fabf81983f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:54.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.730+0000 7fabfe13a640 1 --2- 172.21.15.72:0/2432592393 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fabf8101e80 0x7fabf81983f0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fabe0006010 tx=0x7fabe00042f0 comp rx=0 tx=0).stop 2024-03-22T01:47:54.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.730+0000 7fabfe13a640 1 -- 172.21.15.72:0/2432592393 shutdown_connections 2024-03-22T01:47:54.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.730+0000 7fabfe13a640 1 --2- 172.21.15.72:0/2432592393 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fabe40a3090 0x7fabe40a5550 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:54.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.730+0000 7fabfe13a640 1 --2- 172.21.15.72:0/2432592393 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fabf819ccb0 0x7fabf819d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:54.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.730+0000 7fabfe13a640 1 --2- 172.21.15.72:0/2432592393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fabf8103e40 0x7fabf8198930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:54.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.730+0000 7fabfe13a640 1 --2- 172.21.15.72:0/2432592393 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fabf8101e80 0x7fabf81983f0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:54.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.730+0000 7fabfe13a640 1 -- 172.21.15.72:0/2432592393 >> 172.21.15.72:0/2432592393 conn(0x7fabf80fd620 msgr2=0x7fabf80fed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:47:54.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.730+0000 7fabfe13a640 1 -- 172.21.15.72:0/2432592393 shutdown_connections 2024-03-22T01:47:54.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.730+0000 7fabfe13a640 1 -- 172.21.15.72:0/2432592393 wait complete. 2024-03-22T01:47:54.739 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:54 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/3077825787' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:47:54.739 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:54 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/3459975124' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T01:47:54.739 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:54 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/3575940520' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T01:47:54.740 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-22T01:47:54.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.844+0000 7f1c253f6640 1 -- 172.21.15.72:0/1316368893 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c20100a30 msgr2=0x7f1c20100eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:54.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.844+0000 7f1c253f6640 1 --2- 172.21.15.72:0/1316368893 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c20100a30 0x7f1c20100eb0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f1c14009930 tx=0x7f1c1402bc30 comp rx=0 tx=0).stop 2024-03-22T01:47:54.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.844+0000 7f1c253f6640 1 -- 172.21.15.72:0/1316368893 shutdown_connections 2024-03-22T01:47:54.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.844+0000 7f1c253f6640 1 --2- 172.21.15.72:0/1316368893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c201013f0 0x7f1c2010d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:54.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.844+0000 7f1c253f6640 1 --2- 172.21.15.72:0/1316368893 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c20100a30 0x7f1c20100eb0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:54.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.844+0000 7f1c253f6640 1 --2- 172.21.15.72:0/1316368893 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c20106a40 0x7f1c20106e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:54.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.844+0000 7f1c253f6640 1 -- 172.21.15.72:0/1316368893 >> 172.21.15.72:0/1316368893 conn(0x7f1c20077c40 msgr2=0x7f1c20078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:47:54.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.844+0000 7f1c253f6640 1 -- 172.21.15.72:0/1316368893 shutdown_connections 2024-03-22T01:47:54.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.845+0000 7f1c253f6640 1 -- 172.21.15.72:0/1316368893 wait complete. 2024-03-22T01:47:54.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.845+0000 7f1c253f6640 1 Processor -- start 2024-03-22T01:47:54.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.845+0000 7f1c253f6640 1 -- start start 2024-03-22T01:47:54.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.845+0000 7f1c253f6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c20100a30 0x7f1c20198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:54.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.845+0000 7f1c253f6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c201013f0 0x7f1c201989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:54.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.845+0000 7f1c253f6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c20106a40 0x7f1c2019cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:54.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.845+0000 7f1c253f6640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f1c20111f10 con 0x7f1c201013f0 2024-03-22T01:47:54.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.845+0000 7f1c253f6640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1c20111d90 con 0x7f1c20100a30 2024-03-22T01:47:54.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.845+0000 7f1c253f6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1c20112090 con 0x7f1c20106a40 2024-03-22T01:47:54.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.846+0000 7f1c1e7fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c201013f0 0x7f1c201989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:54.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.846+0000 7f1c1effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c20100a30 0x7f1c20198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:54.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.846+0000 7f1c1e7fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c201013f0 0x7f1c201989d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:35986/0 (socket says 172.21.15.72:35986) 2024-03-22T01:47:54.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.846+0000 7f1c1e7fc640 1 -- 172.21.15.72:0/4175373319 learned_addr learned my addr 172.21.15.72:0/4175373319 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:47:54.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.846+0000 7f1c1f7fe640 1 --2- 172.21.15.72:0/4175373319 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c20106a40 0x7f1c2019cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:54.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.846+0000 7f1c1effd640 1 -- 172.21.15.72:0/4175373319 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c20106a40 msgr2=0x7f1c2019cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:54.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.846+0000 7f1c1effd640 1 --2- 172.21.15.72:0/4175373319 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c20106a40 0x7f1c2019cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:54.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.846+0000 7f1c1effd640 1 -- 172.21.15.72:0/4175373319 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c201013f0 msgr2=0x7f1c201989d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:54.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.846+0000 7f1c1effd640 1 --2- 172.21.15.72:0/4175373319 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c201013f0 0x7f1c201989d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:54.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.846+0000 7f1c1effd640 1 -- 172.21.15.72:0/4175373319 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c2019d430 con 0x7f1c20100a30 2024-03-22T01:47:54.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.846+0000 7f1c1e7fc640 1 --2- 172.21.15.72:0/4175373319 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c201013f0 0x7f1c201989d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:47:54.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.846+0000 7f1c1effd640 1 --2- 172.21.15.72:0/4175373319 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c20100a30 0x7f1c20198490 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1c1000ec80 tx=0x7f1c1000c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:47:54.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.846+0000 7f1c07fff640 1 -- 172.21.15.72:0/4175373319 <== mon.2 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c10003b70 con 0x7f1c20100a30 2024-03-22T01:47:54.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.846+0000 7f1c253f6640 1 -- 172.21.15.72:0/4175373319 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c2019d720 con 0x7f1c20100a30 2024-03-22T01:47:54.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.846+0000 7f1c07fff640 1 -- 172.21.15.72:0/4175373319 <== mon.2 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c10003d10 con 0x7f1c20100a30 2024-03-22T01:47:54.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.846+0000 7f1c07fff640 1 -- 172.21.15.72:0/4175373319 <== mon.2 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c10010a30 con 0x7f1c20100a30 2024-03-22T01:47:54.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.846+0000 7f1c253f6640 1 -- 172.21.15.72:0/4175373319 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c201a4fb0 con 0x7f1c20100a30 2024-03-22T01:47:54.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.847+0000 7f1c253f6640 1 -- 172.21.15.72:0/4175373319 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c20068a10 con 0x7f1c20100a30 2024-03-22T01:47:54.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.848+0000 7f1c07fff640 1 -- 172.21.15.72:0/4175373319 <== mon.2 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 25) ==== 137985+0+0 (secure 0 0 0) 0x7f1c10003ec0 con 0x7f1c20100a30 2024-03-22T01:47:54.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.849+0000 7f1c07fff640 1 --2- 172.21.15.72:0/4175373319 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f1c080a3090 0x7f1c080a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:54.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.849+0000 7f1c1e7fc640 1 --2- 172.21.15.72:0/4175373319 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f1c080a3090 0x7f1c080a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:54.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.849+0000 7f1c07fff640 1 -- 172.21.15.72:0/4175373319 <== mon.2 v2:172.21.15.89:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f1c100d02b0 con 0x7f1c20100a30 2024-03-22T01:47:54.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.849+0000 7f1c1e7fc640 1 --2- 172.21.15.72:0/4175373319 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f1c080a3090 0x7f1c080a5550 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1c14000c00 tx=0x7f1c140058d0 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:47:54.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:54.853+0000 7f1c07fff640 1 -- 172.21.15.72:0/4175373319 <== mon.2 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1c1009c9f0 con 0x7f1c20100a30 2024-03-22T01:47:54.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:54 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/3077825787' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:47:54.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:54 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/3459975124' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T01:47:54.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:54 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/3575940520' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T01:47:54.921 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:54 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/3077825787' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:47:54.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:54 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/3459975124' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T01:47:54.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:54 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/3575940520' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T01:47:55.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.027+0000 7f1c253f6640 1 -- 172.21.15.72:0/4175373319 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f1c2002d3b0 con 0x7f1c20100a30 2024-03-22T01:47:55.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.027+0000 7f1c07fff640 1 -- 172.21.15.72:0/4175373319 <== mon.2 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v96) ==== 103+0+11 (secure 0 0 0) 0x7f1c1009c2f0 con 0x7f1c20100a30 2024-03-22T01:47:55.027 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-03-22T01:47:55.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.029+0000 7f1c253f6640 1 -- 172.21.15.72:0/4175373319 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f1c080a3090 msgr2=0x7f1c080a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:55.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.029+0000 7f1c253f6640 1 --2- 172.21.15.72:0/4175373319 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f1c080a3090 0x7f1c080a5550 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1c14000c00 tx=0x7f1c140058d0 comp rx=0 tx=0).stop 2024-03-22T01:47:55.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.029+0000 7f1c253f6640 1 -- 172.21.15.72:0/4175373319 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c20100a30 msgr2=0x7f1c20198490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:55.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.029+0000 7f1c253f6640 1 --2- 172.21.15.72:0/4175373319 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c20100a30 0x7f1c20198490 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1c1000ec80 tx=0x7f1c1000c6a0 comp rx=0 tx=0).stop 2024-03-22T01:47:55.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.030+0000 7f1c253f6640 1 -- 172.21.15.72:0/4175373319 shutdown_connections 2024-03-22T01:47:55.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.030+0000 7f1c253f6640 1 --2- 172.21.15.72:0/4175373319 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f1c080a3090 0x7f1c080a5550 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:55.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.030+0000 7f1c253f6640 1 --2- 172.21.15.72:0/4175373319 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c20106a40 0x7f1c2019cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:55.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.030+0000 7f1c253f6640 1 --2- 172.21.15.72:0/4175373319 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c201013f0 0x7f1c201989d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:55.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.030+0000 7f1c253f6640 1 --2- 172.21.15.72:0/4175373319 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c20100a30 0x7f1c20198490 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:55.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.030+0000 7f1c253f6640 1 -- 172.21.15.72:0/4175373319 >> 172.21.15.72:0/4175373319 conn(0x7f1c20077c40 msgr2=0x7f1c20103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:47:55.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.030+0000 7f1c253f6640 1 -- 172.21.15.72:0/4175373319 shutdown_connections 2024-03-22T01:47:55.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.030+0000 7f1c253f6640 1 -- 172.21.15.72:0/4175373319 wait complete. 2024-03-22T01:47:55.040 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-22T01:47:55.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.143+0000 7fd9e03f7640 1 -- 172.21.15.72:0/2754101066 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9d8106a40 msgr2=0x7fd9d8106e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:55.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.143+0000 7fd9e03f7640 1 --2- 172.21.15.72:0/2754101066 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9d8106a40 0x7fd9d8106e20 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fd9d4009930 tx=0x7fd9d402bc30 comp rx=0 tx=0).stop 2024-03-22T01:47:55.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.144+0000 7fd9e03f7640 1 -- 172.21.15.72:0/2754101066 shutdown_connections 2024-03-22T01:47:55.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.144+0000 7fd9e03f7640 1 --2- 172.21.15.72:0/2754101066 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9d8100a30 0x7fd9d8100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:55.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.144+0000 7fd9e03f7640 1 --2- 172.21.15.72:0/2754101066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9d81013f0 0x7fd9d810d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:55.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.144+0000 7fd9e03f7640 1 --2- 172.21.15.72:0/2754101066 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9d8106a40 0x7fd9d8106e20 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:55.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.144+0000 7fd9e03f7640 1 -- 172.21.15.72:0/2754101066 >> 172.21.15.72:0/2754101066 conn(0x7fd9d8077c40 msgr2=0x7fd9d8078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:47:55.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.144+0000 7fd9e03f7640 1 -- 172.21.15.72:0/2754101066 shutdown_connections 2024-03-22T01:47:55.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.144+0000 7fd9e03f7640 1 -- 172.21.15.72:0/2754101066 wait complete. 2024-03-22T01:47:55.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.144+0000 7fd9e03f7640 1 Processor -- start 2024-03-22T01:47:55.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.144+0000 7fd9e03f7640 1 -- start start 2024-03-22T01:47:55.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.144+0000 7fd9e03f7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9d8100a30 0x7fd9d81984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:55.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.144+0000 7fd9e03f7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9d81013f0 0x7fd9d8198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:55.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.144+0000 7fd9e03f7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9d8106a40 0x7fd9d819cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:55.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.144+0000 7fd9e03f7640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fd9d8110e50 con 0x7fd9d81013f0 2024-03-22T01:47:55.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.144+0000 7fd9e03f7640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd9d8110cd0 con 0x7fd9d8100a30 2024-03-22T01:47:55.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.144+0000 7fd9e03f7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd9d8110fd0 con 0x7fd9d8106a40 2024-03-22T01:47:55.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.145+0000 7fd9dd992640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9d81013f0 0x7fd9d8198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:55.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.145+0000 7fd9dd992640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9d81013f0 0x7fd9d8198a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36000/0 (socket says 172.21.15.72:36000) 2024-03-22T01:47:55.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.145+0000 7fd9dd992640 1 -- 172.21.15.72:0/1126877609 learned_addr learned my addr 172.21.15.72:0/1126877609 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:47:55.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.145+0000 7fd9de193640 1 --2- 172.21.15.72:0/1126877609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9d8100a30 0x7fd9d81984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:55.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.145+0000 7fd9de994640 1 --2- 172.21.15.72:0/1126877609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9d8106a40 0x7fd9d819cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:55.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.145+0000 7fd9dd992640 1 -- 172.21.15.72:0/1126877609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9d8106a40 msgr2=0x7fd9d819cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:55.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.145+0000 7fd9dd992640 1 --2- 172.21.15.72:0/1126877609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9d8106a40 0x7fd9d819cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:55.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.145+0000 7fd9dd992640 1 -- 172.21.15.72:0/1126877609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9d8100a30 msgr2=0x7fd9d81984e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:55.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.145+0000 7fd9dd992640 1 --2- 172.21.15.72:0/1126877609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9d8100a30 0x7fd9d81984e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:55.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.145+0000 7fd9dd992640 1 -- 172.21.15.72:0/1126877609 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9d819d480 con 0x7fd9d81013f0 2024-03-22T01:47:55.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.145+0000 7fd9dd992640 1 --2- 172.21.15.72:0/1126877609 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9d81013f0 0x7fd9d8198a20 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fd9c800ea60 tx=0x7fd9c800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:47:55.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.145+0000 7fd9cf7fe640 1 -- 172.21.15.72:0/1126877609 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9c800cde0 con 0x7fd9d81013f0 2024-03-22T01:47:55.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.145+0000 7fd9e03f7640 1 -- 172.21.15.72:0/1126877609 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9d819d770 con 0x7fd9d81013f0 2024-03-22T01:47:55.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.145+0000 7fd9cf7fe640 1 -- 172.21.15.72:0/1126877609 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd9c80035f0 con 0x7fd9d81013f0 2024-03-22T01:47:55.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.145+0000 7fd9cf7fe640 1 -- 172.21.15.72:0/1126877609 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9c8010670 con 0x7fd9d81013f0 2024-03-22T01:47:55.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.145+0000 7fd9e03f7640 1 -- 172.21.15.72:0/1126877609 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9d81a5000 con 0x7fd9d81013f0 2024-03-22T01:47:55.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.146+0000 7fd9e03f7640 1 -- 172.21.15.72:0/1126877609 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9d8068a10 con 0x7fd9d81013f0 2024-03-22T01:47:55.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.147+0000 7fd9cf7fe640 1 -- 172.21.15.72:0/1126877609 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 25) ==== 137985+0+0 (secure 0 0 0) 0x7fd9c80037a0 con 0x7fd9d81013f0 2024-03-22T01:47:55.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.148+0000 7fd9cf7fe640 1 --2- 172.21.15.72:0/1126877609 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fd9c40a3090 0x7fd9c40a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:55.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.148+0000 7fd9de193640 1 --2- 172.21.15.72:0/1126877609 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fd9c40a3090 0x7fd9c40a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:55.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.148+0000 7fd9cf7fe640 1 -- 172.21.15.72:0/1126877609 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fd9c80cff20 con 0x7fd9d81013f0 2024-03-22T01:47:55.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.148+0000 7fd9de193640 1 --2- 172.21.15.72:0/1126877609 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fd9c40a3090 0x7fd9c40a5550 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd9d4005ab0 tx=0x7fd9d4005fd0 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:47:55.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.152+0000 7fd9cf7fe640 1 -- 172.21.15.72:0/1126877609 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd9c809c660 con 0x7fd9d81013f0 2024-03-22T01:47:55.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.339+0000 7fd9e03f7640 1 -- 172.21.15.72:0/1126877609 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fd9d8109fe0 con 0x7fd9d81013f0 2024-03-22T01:47:55.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.340+0000 7fd9cf7fe640 1 -- 172.21.15.72:0/1126877609 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v96) ==== 81+0+942 (secure 0 0 0) 0x7fd9c809bf60 con 0x7fd9d81013f0 2024-03-22T01:47:55.339 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:0/2032770149 2024-03-23T01:36:38.183806+0000 2024-03-22T01:47:55.339 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:6801/401403200 2024-03-23T01:36:38.183806+0000 2024-03-22T01:47:55.339 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:0/1161486989 2024-03-23T01:36:38.183806+0000 2024-03-22T01:47:55.340 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:6800/401403200 2024-03-23T01:36:38.183806+0000 2024-03-22T01:47:55.340 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:0/121087196 2024-03-23T01:27:15.339809+0000 2024-03-22T01:47:55.340 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:0/2451857761 2024-03-23T01:27:15.339809+0000 2024-03-22T01:47:55.340 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:0/1228477995 2024-03-23T01:27:15.339809+0000 2024-03-22T01:47:55.340 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:6801/1513232913 2024-03-23T01:27:15.339809+0000 2024-03-22T01:47:55.340 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:0/4223447038 2024-03-23T01:36:38.183806+0000 2024-03-22T01:47:55.340 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:0/3558958264 2024-03-23T01:26:50.074871+0000 2024-03-22T01:47:55.340 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:0/1510187198 2024-03-23T01:26:50.074871+0000 2024-03-22T01:47:55.340 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:6800/3518012916 2024-03-23T01:26:50.074871+0000 2024-03-22T01:47:55.340 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:0/608566205 2024-03-23T01:36:38.183806+0000 2024-03-22T01:47:55.340 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:6801/3518012916 2024-03-23T01:26:50.074871+0000 2024-03-22T01:47:55.340 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:0/1372801041 2024-03-23T01:26:50.074871+0000 2024-03-22T01:47:55.340 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:6800/1513232913 2024-03-23T01:27:15.339809+0000 2024-03-22T01:47:55.341 INFO:teuthology.orchestra.run.smithi072.stderr:listed 16 entries 2024-03-22T01:47:55.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.342+0000 7fd9e03f7640 1 -- 172.21.15.72:0/1126877609 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fd9c40a3090 msgr2=0x7fd9c40a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:55.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.342+0000 7fd9e03f7640 1 --2- 172.21.15.72:0/1126877609 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fd9c40a3090 0x7fd9c40a5550 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd9d4005ab0 tx=0x7fd9d4005fd0 comp rx=0 tx=0).stop 2024-03-22T01:47:55.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.342+0000 7fd9e03f7640 1 -- 172.21.15.72:0/1126877609 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9d81013f0 msgr2=0x7fd9d8198a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:55.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.342+0000 7fd9e03f7640 1 --2- 172.21.15.72:0/1126877609 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9d81013f0 0x7fd9d8198a20 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fd9c800ea60 tx=0x7fd9c800ef20 comp rx=0 tx=0).stop 2024-03-22T01:47:55.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.342+0000 7fd9e03f7640 1 -- 172.21.15.72:0/1126877609 shutdown_connections 2024-03-22T01:47:55.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.342+0000 7fd9e03f7640 1 --2- 172.21.15.72:0/1126877609 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fd9c40a3090 0x7fd9c40a5550 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:55.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.342+0000 7fd9e03f7640 1 --2- 172.21.15.72:0/1126877609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9d8106a40 0x7fd9d819cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:55.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.342+0000 7fd9e03f7640 1 --2- 172.21.15.72:0/1126877609 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9d81013f0 0x7fd9d8198a20 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:55.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.342+0000 7fd9e03f7640 1 --2- 172.21.15.72:0/1126877609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9d8100a30 0x7fd9d81984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:55.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.342+0000 7fd9e03f7640 1 -- 172.21.15.72:0/1126877609 >> 172.21.15.72:0/1126877609 conn(0x7fd9d8077c40 msgr2=0x7fd9d8069240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:47:55.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.342+0000 7fd9e03f7640 1 -- 172.21.15.72:0/1126877609 shutdown_connections 2024-03-22T01:47:55.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:55.342+0000 7fd9e03f7640 1 -- 172.21.15.72:0/1126877609 wait complete. 2024-03-22T01:47:55.352 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-22T01:47:55.352 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T01:47:55.352 DEBUG:teuthology.orchestra.run.smithi072:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-22T01:47:55.413 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi072.front.sepia.ceph.com 2024-03-22T01:47:55.413 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T01:47:55.413 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-22T01:47:55.451 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T01:47:55.451 DEBUG:teuthology.orchestra.run.smithi072:> ip netns list 2024-03-22T01:47:55.555 INFO:teuthology.orchestra.run.smithi072.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-22T01:47:55.556 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T01:47:55.556 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-22T01:47:55.583 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T01:47:55.583 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link delete ceph-brx 2024-03-22T01:47:55.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:55 smithi072 ceph-mon[32414]: pgmap v341: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-22T01:47:55.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:55 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/2432592393' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T01:47:55.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:55 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/4175373319' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T01:47:55.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:55 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/1126877609' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-22T01:47:55.671 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-22T01:47:55.671 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-22T01:47:55.671 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-22T01:47:55.686 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17560 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248808k,nr_inodes=812202,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/sda1 /var/lib/containers/storage/overlay ext4 rw,seclabel,relatime 0 0 overlay /var/lib/containers/storage/overlay/069117ffe870aadaee94635f35b7cf02aae0f606d9587cb0907b5d4dab63e946/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/6EVIYLPG3PEXFADAE7QMML2Y32:/var/lib/containers/storage/overlay/l/OVGHZFZRIHU47D4EJOBFH4L3P2,upperdir=/var/lib/containers/storage/overlay/069117ffe870aadaee94635f35b7cf02aae0f606d9587cb0907b5d4dab63e946/diff,workdir=/var/lib/containers/storage/overlay/069117ffe870aadaee94635f35b7cf02aae0f606d9587cb0907b5d4dab63e946/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/6b1cf7b589b79c37eeeeb6b78f58c056fee85807d776163ee22a7a89de3ddf75/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/6EVIYLPG3PEXFADAE7QMML2Y32:/var/lib/containers/storage/overlay/l/OVGHZFZRIHU47D4EJOBFH4L3P2,upperdir=/var/lib/containers/storage/overlay/6b1cf7b589b79c37eeeeb6b78f58c056fee85807d776163ee22a7a89de3ddf75/diff,workdir=/var/lib/containers/storage/overlay/6b1cf7b589b79c37eeeeb6b78f58c056fee85807d776163ee22a7a89de3ddf75/work,metacopy=on,volatile 0 0 tmpfs /run/user/0 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248808k,nr_inodes=812202,mode=700,inode64 0 0 binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,nosuid,nodev,noexec,relatime 0 0 overlay /var/lib/containers/storage/overlay/b069fa7d57b8b76bc3d1c42abf6d7c1b5fb54ad1a0a29aa2dac69e9ab9916978/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/6EVIYLPG3PEXFADAE7QMML2Y32:/var/lib/containers/storage/overlay/l/OVGHZFZRIHU47D4EJOBFH4L3P2,upperdir=/var/lib/containers/storage/overlay/b069fa7d57b8b76bc3d1c42abf6d7c1b5fb54ad1a0a29aa2dac69e9ab9916978/diff,workdir=/var/lib/containers/storage/overlay/b069fa7d57b8b76bc3d1c42abf6d7c1b5fb54ad1a0a29aa2dac69e9ab9916978/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/2f007650bd719ee2dd3b735bbae33dbcc3561e2865d342d06379848b1e7b80fe/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/6EVIYLPG3PEXFADAE7QMML2Y32:/var/lib/containers/storage/overlay/l/OVGHZFZRIHU47D4EJOBFH4L3P2,upperdir=/var/lib/containers/storage/overlay/2f007650bd719ee2dd3b735bbae33dbcc3561e2865d342d06379848b1e7b80fe/diff,workdir=/var/lib/containers/storage/overlay/2f007650bd719ee2dd3b735bbae33dbcc3561e2865d342d06379848b1e7b80fe/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/ed2330155c50a791c4a8b49015124397c44acf96b694bce6b3035eaad27ceadf/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/6EVIYLPG3PEXFADAE7QMML2Y32:/var/lib/containers/storage/overlay/l/OVGHZFZRIHU47D4EJOBFH4L3P2,upperdir=/var/lib/containers/storage/overlay/ed2330155c50a791c4a8b49015124397c44acf96b694bce6b3035eaad27ceadf/diff,workdir=/var/lib/containers/storage/overlay/ed2330155c50a791c4a8b49015124397c44acf96b694bce6b3035eaad27ceadf/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/de768fa5dc235fe12656be652916a87a4b0dddaca31df687197622b510116874/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/6EVIYLPG3PEXFADAE7QMML2Y32:/var/lib/containers/storage/overlay/l/OVGHZFZRIHU47D4EJOBFH4L3P2,upperdir=/var/lib/containers/storage/overlay/de768fa5dc235fe12656be652916a87a4b0dddaca31df687197622b510116874/diff,workdir=/var/lib/containers/storage/overlay/de768fa5dc235fe12656be652916a87a4b0dddaca31df687197622b510116874/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/f2cf671e5b1e0026e223fe02882b788143825624e5b2f5c4fffc473f47f5a9c8/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/6EVIYLPG3PEXFADAE7QMML2Y32:/var/lib/containers/storage/overlay/l/OVGHZFZRIHU47D4EJOBFH4L3P2,upperdir=/var/lib/containers/storage/overlay/f2cf671e5b1e0026e223fe02882b788143825624e5b2f5c4fffc473f47f5a9c8/diff,workdir=/var/lib/containers/storage/overlay/f2cf671e5b1e0026e223fe02882b788143825624e5b2f5c4fffc473f47f5a9c8/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/a2ca6ec688349bbe337aff3deb37a4c2706108c32e7b2f1dfa5b65f67ceaecb1/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/6EVIYLPG3PEXFADAE7QMML2Y32:/var/lib/containers/storage/overlay/l/OVGHZFZRIHU47D4EJOBFH4L3P2,upperdir=/var/lib/containers/storage/overlay/a2ca6ec688349bbe337aff3deb37a4c2706108c32e7b2f1dfa5b65f67ceaecb1/diff,workdir=/var/lib/containers/storage/overlay/a2ca6ec688349bbe337aff3deb37a4c2706108c32e7b2f1dfa5b65f67ceaecb1/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/7e5b20051723b5937b23547234cd57ff1644844cd2cf7cf7e7dbe5939ac1f3f8/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/6EVIYLPG3PEXFADAE7QMML2Y32:/var/lib/containers/storage/overlay/l/OVGHZFZRIHU47D4EJOBFH4L3P2,upperdir=/var/lib/containers/storage/overlay/7e5b20051723b5937b23547234cd57ff1644844cd2cf7cf7e7dbe5939ac1f3f8/diff,workdir=/var/lib/containers/storage/overlay/7e5b20051723b5937b23547234cd57ff1644844cd2cf7cf7e7dbe5939ac1f3f8/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/6337ea23c964add57bc86d4a987aed22ce9f35bfc1e9b2d0d153922d7017419d/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/6EVIYLPG3PEXFADAE7QMML2Y32:/var/lib/containers/storage/overlay/l/OVGHZFZRIHU47D4EJOBFH4L3P2,upperdir=/var/lib/containers/storage/overlay/6337ea23c964add57bc86d4a987aed22ce9f35bfc1e9b2d0d153922d7017419d/diff,workdir=/var/lib/containers/storage/overlay/6337ea23c964add57bc86d4a987aed22ce9f35bfc1e9b2d0d153922d7017419d/work,metacopy=on,volatile 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 2024-03-22T01:47:55.687 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T01:47:55.687 DEBUG:teuthology.orchestra.run.smithi072:> ip netns list 2024-03-22T01:47:55.742 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T01:47:55.742 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link delete ceph-brx 2024-03-22T01:47:55.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:55 smithi089 ceph-mon[35770]: pgmap v341: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-22T01:47:55.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:55 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/2432592393' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T01:47:55.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:55 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/4175373319' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T01:47:55.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:55 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/1126877609' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-22T01:47:55.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:55 smithi098 ceph-mon[36468]: pgmap v341: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-22T01:47:55.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:55 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/2432592393' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T01:47:55.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:55 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/4175373319' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T01:47:55.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:55 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/1126877609' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-22T01:47:55.933 INFO:teuthology.orchestra.run.smithi072.stderr:Cannot find device "ceph-brx" 2024-03-22T01:47:55.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-22T01:47:55.935 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-22T01:47:55.946 INFO:tasks.cephadm:Teardown begin 2024-03-22T01:47:55.946 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-22T01:47:55.947 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T01:47:56.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.049+0000 7fc2c6f3e640 1 -- 172.21.15.72:0/1667501871 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc2c01054f0 msgr2=0x7fc2c010d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:56.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.049+0000 7fc2c6f3e640 1 --2- 172.21.15.72:0/1667501871 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc2c01054f0 0x7fc2c010d0c0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fc2b800adb0 tx=0x7fc2b802cff0 comp rx=0 tx=0).stop 2024-03-22T01:47:56.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.049+0000 7fc2c6f3e640 1 -- 172.21.15.72:0/1667501871 shutdown_connections 2024-03-22T01:47:56.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.049+0000 7fc2c6f3e640 1 --2- 172.21.15.72:0/1667501871 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2c0100dd0 0x7fc2c0104e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:56.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.049+0000 7fc2c6f3e640 1 --2- 172.21.15.72:0/1667501871 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc2c01054f0 0x7fc2c010d0c0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:56.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.049+0000 7fc2c6f3e640 1 --2- 172.21.15.72:0/1667501871 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2c0100420 0x7fc2c0100800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:56.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.049+0000 7fc2c6f3e640 1 -- 172.21.15.72:0/1667501871 >> 172.21.15.72:0/1667501871 conn(0x7fc2c0076fa0 msgr2=0x7fc2c00fe330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:47:56.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.049+0000 7fc2c6f3e640 1 -- 172.21.15.72:0/1667501871 shutdown_connections 2024-03-22T01:47:56.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.049+0000 7fc2c6f3e640 1 -- 172.21.15.72:0/1667501871 wait complete. 2024-03-22T01:47:56.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.050+0000 7fc2c6f3e640 1 Processor -- start 2024-03-22T01:47:56.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.050+0000 7fc2c6f3e640 1 -- start start 2024-03-22T01:47:56.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.050+0000 7fc2c6f3e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2c0100420 0x7fc2c0198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:56.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.050+0000 7fc2c6f3e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2c01054f0 0x7fc2c0198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:56.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.050+0000 7fc2c6f3e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc2c019cd70 0x7fc2c019d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:56.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.050+0000 7fc2c6f3e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fc2c010fd20 con 0x7fc2c019cd70 2024-03-22T01:47:56.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.050+0000 7fc2c6f3e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc2c010fba0 con 0x7fc2c01054f0 2024-03-22T01:47:56.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.050+0000 7fc2c6f3e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc2c010fea0 con 0x7fc2c0100420 2024-03-22T01:47:56.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.050+0000 7fc2c54db640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc2c019cd70 0x7fc2c019d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:56.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.050+0000 7fc2c54db640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc2c019cd70 0x7fc2c019d220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36024/0 (socket says 172.21.15.72:36024) 2024-03-22T01:47:56.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.050+0000 7fc2c54db640 1 -- 172.21.15.72:0/272212675 learned_addr learned my addr 172.21.15.72:0/272212675 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:47:56.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.050+0000 7fc2bffff640 1 --2- 172.21.15.72:0/272212675 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2c01054f0 0x7fc2c0198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:56.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.050+0000 7fc2c4cda640 1 --2- 172.21.15.72:0/272212675 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2c0100420 0x7fc2c0198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:56.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.050+0000 7fc2c54db640 1 -- 172.21.15.72:0/272212675 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2c0100420 msgr2=0x7fc2c0198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:56.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.050+0000 7fc2c54db640 1 --2- 172.21.15.72:0/272212675 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2c0100420 0x7fc2c0198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:56.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.050+0000 7fc2c54db640 1 -- 172.21.15.72:0/272212675 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2c01054f0 msgr2=0x7fc2c0198a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:56.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.050+0000 7fc2c54db640 1 --2- 172.21.15.72:0/272212675 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2c01054f0 0x7fc2c0198a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:56.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.050+0000 7fc2c54db640 1 -- 172.21.15.72:0/272212675 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2c019d860 con 0x7fc2c019cd70 2024-03-22T01:47:56.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.050+0000 7fc2bffff640 1 --2- 172.21.15.72:0/272212675 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2c01054f0 0x7fc2c0198a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:47:56.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.051+0000 7fc2c54db640 1 --2- 172.21.15.72:0/272212675 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc2c019cd70 0x7fc2c019d220 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fc2b802d600 tx=0x7fc2b800a4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:47:56.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.051+0000 7fc2bdffb640 1 -- 172.21.15.72:0/272212675 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2b803a040 con 0x7fc2c019cd70 2024-03-22T01:47:56.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.051+0000 7fc2bdffb640 1 -- 172.21.15.72:0/272212675 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2b8043070 con 0x7fc2c019cd70 2024-03-22T01:47:56.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.051+0000 7fc2c6f3e640 1 -- 172.21.15.72:0/272212675 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2c019daf0 con 0x7fc2c019cd70 2024-03-22T01:47:56.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.051+0000 7fc2bdffb640 1 -- 172.21.15.72:0/272212675 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2b80356a0 con 0x7fc2c019cd70 2024-03-22T01:47:56.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.051+0000 7fc2c6f3e640 1 -- 172.21.15.72:0/272212675 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2c01a5250 con 0x7fc2c019cd70 2024-03-22T01:47:56.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.052+0000 7fc2c6f3e640 1 -- 172.21.15.72:0/272212675 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2c0068e60 con 0x7fc2c019cd70 2024-03-22T01:47:56.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.053+0000 7fc2bdffb640 1 -- 172.21.15.72:0/272212675 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 25) ==== 137985+0+0 (secure 0 0 0) 0x7fc2b8003b70 con 0x7fc2c019cd70 2024-03-22T01:47:56.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.054+0000 7fc2bdffb640 1 --2- 172.21.15.72:0/272212675 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fc2a80a3090 0x7fc2a80a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:56.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.054+0000 7fc2c4cda640 1 --2- 172.21.15.72:0/272212675 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fc2a80a3090 0x7fc2a80a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:56.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.054+0000 7fc2bdffb640 1 -- 172.21.15.72:0/272212675 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fc2b80f17f0 con 0x7fc2c019cd70 2024-03-22T01:47:56.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.054+0000 7fc2c4cda640 1 --2- 172.21.15.72:0/272212675 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fc2a80a3090 0x7fc2a80a5550 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc2b00058c0 tx=0x7fc2b000a4c0 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:47:56.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.058+0000 7fc2bdffb640 1 -- 172.21.15.72:0/272212675 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc2b80be0d0 con 0x7fc2c019cd70 2024-03-22T01:47:56.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.269+0000 7fc2c6f3e640 1 -- 172.21.15.72:0/272212675 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc2c0100dd0 con 0x7fc2c019cd70 2024-03-22T01:47:56.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.270+0000 7fc2bdffb640 1 -- 172.21.15.72:0/272212675 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) ==== 95+0+11791 (secure 0 0 0) 0x7fc2b8004820 con 0x7fc2c019cd70 2024-03-22T01:47:56.269 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:47:56.269 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":103,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6835/3077216308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3077216308},{"type":"v1","addr":"172.21.15.72:6835","nonce":3077216308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2460254143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":2460254143},{"type":"v1","addr":"172.21.15.72:6837","nonce":2460254143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6839/2026432819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6838","nonce":2026432819},{"type":"v1","addr":"172.21.15.72:6839","nonce":2026432819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6841/400094545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6840","nonce":400094545},{"type":"v1","addr":"172.21.15.72:6841","nonce":400094545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6833/1578701171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":1578701171},{"type":"v1","addr":"172.21.15.89:6833","nonce":1578701171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/2703047213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2703047213},{"type":"v1","addr":"172.21.15.98:6833","nonce":2703047213}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2589468787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2589468787},{"type":"v1","addr":"172.21.15.89:6835","nonce":2589468787}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/92657636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":92657636},{"type":"v1","addr":"172.21.15.98:6835","nonce":92657636}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1662941782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1662941782},{"type":"v1","addr":"172.21.15.89:6837","nonce":1662941782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24490,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/781860371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":781860371},{"type":"v1","addr":"172.21.15.98:6837","nonce":781860371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":103,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-22T01:34:14.815677+0000","modified":"2024-03-22T01:45:27.515143+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24524},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":171,"addr":"172.21.15.98:6839/3192173650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3192173650},{"type":"v1","addr":"172.21.15.98:6839","nonce":3192173650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":171,"addr":"172.21.15.89:6839/149200280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":149200280},{"type":"v1","addr":"172.21.15.89:6839","nonce":149200280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24524]},"id":1}]} 2024-03-22T01:47:56.269 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 103 2024-03-22T01:47:56.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.272+0000 7fc2c6f3e640 1 -- 172.21.15.72:0/272212675 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fc2a80a3090 msgr2=0x7fc2a80a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:56.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.272+0000 7fc2c6f3e640 1 --2- 172.21.15.72:0/272212675 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fc2a80a3090 0x7fc2a80a5550 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc2b00058c0 tx=0x7fc2b000a4c0 comp rx=0 tx=0).stop 2024-03-22T01:47:56.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.272+0000 7fc2c6f3e640 1 -- 172.21.15.72:0/272212675 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc2c019cd70 msgr2=0x7fc2c019d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:56.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.272+0000 7fc2c6f3e640 1 --2- 172.21.15.72:0/272212675 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc2c019cd70 0x7fc2c019d220 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fc2b802d600 tx=0x7fc2b800a4e0 comp rx=0 tx=0).stop 2024-03-22T01:47:56.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.272+0000 7fc2c6f3e640 1 -- 172.21.15.72:0/272212675 shutdown_connections 2024-03-22T01:47:56.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.272+0000 7fc2c6f3e640 1 --2- 172.21.15.72:0/272212675 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fc2a80a3090 0x7fc2a80a5550 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:56.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.272+0000 7fc2c6f3e640 1 --2- 172.21.15.72:0/272212675 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc2c01054f0 0x7fc2c0198a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:56.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.272+0000 7fc2c6f3e640 1 --2- 172.21.15.72:0/272212675 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc2c019cd70 0x7fc2c019d220 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:56.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.272+0000 7fc2c6f3e640 1 --2- 172.21.15.72:0/272212675 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2c0100420 0x7fc2c0198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:56.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.272+0000 7fc2c6f3e640 1 -- 172.21.15.72:0/272212675 >> 172.21.15.72:0/272212675 conn(0x7fc2c0076fa0 msgr2=0x7fc2c0103bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:47:56.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.272+0000 7fc2c6f3e640 1 -- 172.21.15.72:0/272212675 shutdown_connections 2024-03-22T01:47:56.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.272+0000 7fc2c6f3e640 1 -- 172.21.15.72:0/272212675 wait complete. 2024-03-22T01:47:56.282 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T01:47:56.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.384+0000 7f6a51365640 1 -- 172.21.15.72:0/2224616265 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6a4c107c10 msgr2=0x7f6a4c107ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:56.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.384+0000 7f6a51365640 1 --2- 172.21.15.72:0/2224616265 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6a4c107c10 0x7f6a4c107ff0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c009900 tx=0x7f6a3c02bc30 comp rx=0 tx=0).stop 2024-03-22T01:47:56.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.384+0000 7f6a51365640 1 -- 172.21.15.72:0/2224616265 shutdown_connections 2024-03-22T01:47:56.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.384+0000 7f6a51365640 1 --2- 172.21.15.72:0/2224616265 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a4c101b50 0x7f6a4c101fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:56.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.384+0000 7f6a51365640 1 --2- 172.21.15.72:0/2224616265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a4c102510 0x7f6a4c10e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:56.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.384+0000 7f6a51365640 1 --2- 172.21.15.72:0/2224616265 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6a4c107c10 0x7f6a4c107ff0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:56.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.384+0000 7f6a51365640 1 -- 172.21.15.72:0/2224616265 >> 172.21.15.72:0/2224616265 conn(0x7f6a4c0fd640 msgr2=0x7f6a4c0ffa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:47:56.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.384+0000 7f6a51365640 1 -- 172.21.15.72:0/2224616265 shutdown_connections 2024-03-22T01:47:56.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.384+0000 7f6a51365640 1 -- 172.21.15.72:0/2224616265 wait complete. 2024-03-22T01:47:56.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.385+0000 7f6a51365640 1 Processor -- start 2024-03-22T01:47:56.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.385+0000 7f6a51365640 1 -- start start 2024-03-22T01:47:56.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.385+0000 7f6a51365640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6a4c101b50 0x7f6a4c1983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:56.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.385+0000 7f6a51365640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a4c102510 0x7f6a4c198930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:56.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.385+0000 7f6a51365640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a4c107c10 0x7f6a4c19ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:56.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.385+0000 7f6a51365640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f6a4c111f50 con 0x7f6a4c101b50 2024-03-22T01:47:56.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.385+0000 7f6a51365640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f6a4c111dd0 con 0x7f6a4c102510 2024-03-22T01:47:56.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.385+0000 7f6a51365640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6a4c1120d0 con 0x7f6a4c107c10 2024-03-22T01:47:56.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.385+0000 7f6a4affd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6a4c101b50 0x7f6a4c1983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:56.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.385+0000 7f6a4affd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6a4c101b50 0x7f6a4c1983f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36048/0 (socket says 172.21.15.72:36048) 2024-03-22T01:47:56.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.385+0000 7f6a4affd640 1 -- 172.21.15.72:0/2929120188 learned_addr learned my addr 172.21.15.72:0/2929120188 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:47:56.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.385+0000 7f6a43fff640 1 --2- 172.21.15.72:0/2929120188 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a4c102510 0x7f6a4c198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:56.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.385+0000 7f6a4b7fe640 1 --2- 172.21.15.72:0/2929120188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a4c107c10 0x7f6a4c19ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:56.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.385+0000 7f6a4affd640 1 -- 172.21.15.72:0/2929120188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a4c107c10 msgr2=0x7f6a4c19ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:56.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.385+0000 7f6a4affd640 1 --2- 172.21.15.72:0/2929120188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a4c107c10 0x7f6a4c19ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:56.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.385+0000 7f6a4affd640 1 -- 172.21.15.72:0/2929120188 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a4c102510 msgr2=0x7f6a4c198930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:56.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.385+0000 7f6a4affd640 1 --2- 172.21.15.72:0/2929120188 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a4c102510 0x7f6a4c198930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:56.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.385+0000 7f6a4affd640 1 -- 172.21.15.72:0/2929120188 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a4c19d3e0 con 0x7f6a4c101b50 2024-03-22T01:47:56.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.385+0000 7f6a43fff640 1 --2- 172.21.15.72:0/2929120188 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a4c102510 0x7f6a4c198930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:47:56.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.386+0000 7f6a4affd640 1 --2- 172.21.15.72:0/2929120188 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6a4c101b50 0x7f6a4c1983f0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c005940 tx=0x7f6a3c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:47:56.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.386+0000 7f6a48ff9640 1 -- 172.21.15.72:0/2929120188 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a3c041070 con 0x7f6a4c101b50 2024-03-22T01:47:56.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.386+0000 7f6a48ff9640 1 -- 172.21.15.72:0/2929120188 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a3c02ccc0 con 0x7f6a4c101b50 2024-03-22T01:47:56.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.386+0000 7f6a51365640 1 -- 172.21.15.72:0/2929120188 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a4c19d670 con 0x7f6a4c101b50 2024-03-22T01:47:56.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.386+0000 7f6a48ff9640 1 -- 172.21.15.72:0/2929120188 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a3c003d80 con 0x7f6a4c101b50 2024-03-22T01:47:56.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.386+0000 7f6a51365640 1 -- 172.21.15.72:0/2929120188 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a4c1a4f60 con 0x7f6a4c101b50 2024-03-22T01:47:56.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.387+0000 7f6a51365640 1 -- 172.21.15.72:0/2929120188 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a4c103200 con 0x7f6a4c101b50 2024-03-22T01:47:56.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.388+0000 7f6a48ff9640 1 -- 172.21.15.72:0/2929120188 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 25) ==== 137985+0+0 (secure 0 0 0) 0x7f6a3c044070 con 0x7f6a4c101b50 2024-03-22T01:47:56.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.388+0000 7f6a48ff9640 1 --2- 172.21.15.72:0/2929120188 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f6a380a3090 0x7f6a380a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:56.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.389+0000 7f6a43fff640 1 --2- 172.21.15.72:0/2929120188 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f6a380a3090 0x7f6a380a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:56.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.389+0000 7f6a48ff9640 1 -- 172.21.15.72:0/2929120188 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f6a3c0f1cb0 con 0x7f6a4c101b50 2024-03-22T01:47:56.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.389+0000 7f6a43fff640 1 --2- 172.21.15.72:0/2929120188 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f6a380a3090 0x7f6a380a5550 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c107290 tx=0x7f6a34005f90 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:47:56.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.392+0000 7f6a48ff9640 1 -- 172.21.15.72:0/2929120188 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6a3c0be500 con 0x7f6a4c101b50 2024-03-22T01:47:56.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.610+0000 7f6a51365640 1 -- 172.21.15.72:0/2929120188 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6a4c068e60 con 0x7f6a4c101b50 2024-03-22T01:47:56.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.610+0000 7f6a48ff9640 1 -- 172.21.15.72:0/2929120188 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) ==== 95+0+11791 (secure 0 0 0) 0x7f6a4c068e60 con 0x7f6a4c101b50 2024-03-22T01:47:56.610 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:47:56.610 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":103,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6835/3077216308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3077216308},{"type":"v1","addr":"172.21.15.72:6835","nonce":3077216308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2460254143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":2460254143},{"type":"v1","addr":"172.21.15.72:6837","nonce":2460254143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6839/2026432819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6838","nonce":2026432819},{"type":"v1","addr":"172.21.15.72:6839","nonce":2026432819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6841/400094545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6840","nonce":400094545},{"type":"v1","addr":"172.21.15.72:6841","nonce":400094545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6833/1578701171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":1578701171},{"type":"v1","addr":"172.21.15.89:6833","nonce":1578701171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/2703047213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2703047213},{"type":"v1","addr":"172.21.15.98:6833","nonce":2703047213}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2589468787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":2589468787},{"type":"v1","addr":"172.21.15.89:6835","nonce":2589468787}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/92657636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":92657636},{"type":"v1","addr":"172.21.15.98:6835","nonce":92657636}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1662941782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1662941782},{"type":"v1","addr":"172.21.15.89:6837","nonce":1662941782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24490,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/781860371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":781860371},{"type":"v1","addr":"172.21.15.98:6837","nonce":781860371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":103,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-22T01:34:14.815677+0000","modified":"2024-03-22T01:45:27.515143+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24524},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":171,"addr":"172.21.15.98:6839/3192173650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3192173650},{"type":"v1","addr":"172.21.15.98:6839","nonce":3192173650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":171,"addr":"172.21.15.89:6839/149200280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":149200280},{"type":"v1","addr":"172.21.15.89:6839","nonce":149200280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24524]},"id":1}]} 2024-03-22T01:47:56.610 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 103 2024-03-22T01:47:56.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.612+0000 7f6a51365640 1 -- 172.21.15.72:0/2929120188 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f6a380a3090 msgr2=0x7f6a380a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:56.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.612+0000 7f6a51365640 1 --2- 172.21.15.72:0/2929120188 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f6a380a3090 0x7f6a380a5550 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c107290 tx=0x7f6a34005f90 comp rx=0 tx=0).stop 2024-03-22T01:47:56.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.612+0000 7f6a51365640 1 -- 172.21.15.72:0/2929120188 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6a4c101b50 msgr2=0x7f6a4c1983f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:56.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.612+0000 7f6a51365640 1 --2- 172.21.15.72:0/2929120188 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6a4c101b50 0x7f6a4c1983f0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c005940 tx=0x7f6a3c0042f0 comp rx=0 tx=0).stop 2024-03-22T01:47:56.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.612+0000 7f6a51365640 1 -- 172.21.15.72:0/2929120188 shutdown_connections 2024-03-22T01:47:56.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.612+0000 7f6a51365640 1 --2- 172.21.15.72:0/2929120188 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f6a380a3090 0x7f6a380a5550 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:56.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.612+0000 7f6a51365640 1 --2- 172.21.15.72:0/2929120188 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a4c102510 0x7f6a4c198930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:56.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.612+0000 7f6a51365640 1 --2- 172.21.15.72:0/2929120188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a4c107c10 0x7f6a4c19ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:56.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.612+0000 7f6a51365640 1 --2- 172.21.15.72:0/2929120188 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6a4c101b50 0x7f6a4c1983f0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:56.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.612+0000 7f6a51365640 1 -- 172.21.15.72:0/2929120188 >> 172.21.15.72:0/2929120188 conn(0x7f6a4c0fd640 msgr2=0x7f6a4c0fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:47:56.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.613+0000 7f6a51365640 1 -- 172.21.15.72:0/2929120188 shutdown_connections 2024-03-22T01:47:56.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.613+0000 7f6a51365640 1 -- 172.21.15.72:0/2929120188 wait complete. 2024-03-22T01:47:56.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:56 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/272212675' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:47:56.623 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T01:47:56.671 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:56 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/272212675' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:47:56.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.725+0000 7f148f6f4640 1 -- 172.21.15.72:0/351964210 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14881013e0 msgr2=0x7f148810d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:56.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.725+0000 7f148f6f4640 1 --2- 172.21.15.72:0/351964210 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14881013e0 0x7f148810d8f0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f1484009cd0 tx=0x7f148402c0f0 comp rx=0 tx=0).stop 2024-03-22T01:47:56.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.725+0000 7f148f6f4640 1 -- 172.21.15.72:0/351964210 shutdown_connections 2024-03-22T01:47:56.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.725+0000 7f148f6f4640 1 --2- 172.21.15.72:0/351964210 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1488100a20 0x7f1488100ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:56.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.725+0000 7f148f6f4640 1 --2- 172.21.15.72:0/351964210 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14881013e0 0x7f148810d8f0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:56.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.725+0000 7f148f6f4640 1 --2- 172.21.15.72:0/351964210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1488106a30 0x7f1488106e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:56.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.725+0000 7f148f6f4640 1 -- 172.21.15.72:0/351964210 >> 172.21.15.72:0/351964210 conn(0x7f1488077ca0 msgr2=0x7f14880780b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:47:56.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.725+0000 7f148f6f4640 1 -- 172.21.15.72:0/351964210 shutdown_connections 2024-03-22T01:47:56.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.725+0000 7f148f6f4640 1 -- 172.21.15.72:0/351964210 wait complete. 2024-03-22T01:47:56.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.725+0000 7f148f6f4640 1 Processor -- start 2024-03-22T01:47:56.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.725+0000 7f148f6f4640 1 -- start start 2024-03-22T01:47:56.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.726+0000 7f148f6f4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1488100a20 0x7f1488197270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:56.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.726+0000 7f148f6f4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14881013e0 0x7f14881977b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:56.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.726+0000 7f148f6f4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1488106a30 0x7f148819bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:56.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.726+0000 7f148f6f4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f1488110e10 con 0x7f1488100a20 2024-03-22T01:47:56.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.726+0000 7f148f6f4640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1488110c90 con 0x7f1488106a30 2024-03-22T01:47:56.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.726+0000 7f148f6f4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1488110f90 con 0x7f14881013e0 2024-03-22T01:47:56.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.726+0000 7f148d490640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1488100a20 0x7f1488197270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:56.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.726+0000 7f148d490640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1488100a20 0x7f1488197270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36072/0 (socket says 172.21.15.72:36072) 2024-03-22T01:47:56.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.726+0000 7f148d490640 1 -- 172.21.15.72:0/1545535021 learned_addr learned my addr 172.21.15.72:0/1545535021 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:47:56.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.726+0000 7f148dc91640 1 --2- 172.21.15.72:0/1545535021 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1488106a30 0x7f148819bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:56.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.726+0000 7f148cc8f640 1 --2- 172.21.15.72:0/1545535021 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14881013e0 0x7f14881977b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:56.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.726+0000 7f148d490640 1 -- 172.21.15.72:0/1545535021 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14881013e0 msgr2=0x7f14881977b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:56.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.726+0000 7f148d490640 1 --2- 172.21.15.72:0/1545535021 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14881013e0 0x7f14881977b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:56.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.726+0000 7f148d490640 1 -- 172.21.15.72:0/1545535021 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1488106a30 msgr2=0x7f148819bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:56.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.726+0000 7f148d490640 1 --2- 172.21.15.72:0/1545535021 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1488106a30 0x7f148819bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:56.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.726+0000 7f148d490640 1 -- 172.21.15.72:0/1545535021 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f148819c210 con 0x7f1488100a20 2024-03-22T01:47:56.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.726+0000 7f148cc8f640 1 --2- 172.21.15.72:0/1545535021 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14881013e0 0x7f14881977b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T01:47:56.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.726+0000 7f148d490640 1 --2- 172.21.15.72:0/1545535021 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1488100a20 0x7f1488197270 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f148000c920 tx=0x7f148000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:47:56.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.726+0000 7f147e7fc640 1 -- 172.21.15.72:0/1545535021 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1480007c70 con 0x7f1488100a20 2024-03-22T01:47:56.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.726+0000 7f148f6f4640 1 -- 172.21.15.72:0/1545535021 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f148819c500 con 0x7f1488100a20 2024-03-22T01:47:56.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.726+0000 7f147e7fc640 1 -- 172.21.15.72:0/1545535021 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1480007e10 con 0x7f1488100a20 2024-03-22T01:47:56.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.726+0000 7f147e7fc640 1 -- 172.21.15.72:0/1545535021 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1480018890 con 0x7f1488100a20 2024-03-22T01:47:56.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.726+0000 7f148f6f4640 1 -- 172.21.15.72:0/1545535021 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14881a3d90 con 0x7f1488100a20 2024-03-22T01:47:56.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.727+0000 7f148f6f4640 1 -- 172.21.15.72:0/1545535021 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1488102160 con 0x7f1488100a20 2024-03-22T01:47:56.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.728+0000 7f147e7fc640 1 -- 172.21.15.72:0/1545535021 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 25) ==== 137985+0+0 (secure 0 0 0) 0x7f1480018a70 con 0x7f1488100a20 2024-03-22T01:47:56.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.729+0000 7f147e7fc640 1 --2- 172.21.15.72:0/1545535021 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f14740a3090 0x7f14740a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:56.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.729+0000 7f147e7fc640 1 -- 172.21.15.72:0/1545535021 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f14800cfc70 con 0x7f1488100a20 2024-03-22T01:47:56.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.729+0000 7f148cc8f640 1 --2- 172.21.15.72:0/1545535021 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f14740a3090 0x7f14740a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:56.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.729+0000 7f148cc8f640 1 --2- 172.21.15.72:0/1545535021 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f14740a3090 0x7f14740a5550 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1488105010 tx=0x7f1478005f90 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:47:56.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.733+0000 7f147e7fc640 1 -- 172.21.15.72:0/1545535021 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1480014030 con 0x7f1488100a20 2024-03-22T01:47:56.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:56 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/272212675' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:47:56.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.911+0000 7f148f6f4640 1 -- 172.21.15.72:0/1545535021 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1488198570 con 0x7f1488100a20 2024-03-22T01:47:56.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.912+0000 7f147e7fc640 1 -- 172.21.15.72:0/1545535021 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) ==== 74+0+25134 (secure 0 0 0) 0x7f148009c3b0 con 0x7f1488100a20 2024-03-22T01:47:56.912 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-22T01:47:56.912 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":96,"fsid":"022a99bc-e7eb-11ee-95cd-87774f69a715","created":"2024-03-22T01:26:23.893672+0000","modified":"2024-03-22T01:36:38.183837+0000","last_up_change":"2024-03-22T01:33:25.223684+0000","last_in_change":"2024-03-22T01:33:05.010331+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T01:29:37.438503+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-22T01:34:12.610422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-22T01:34:13.761686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-22T01:34:17.507039+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"16c15bb0-db92-4191-9063-38b4890e81e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6803","nonce":661906791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6805","nonce":661906791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6809","nonce":661906791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":661906791},{"type":"v1","addr":"172.21.15.72:6807","nonce":661906791}]},"public_addr":"172.21.15.72:6803/661906791","cluster_addr":"172.21.15.72:6805/661906791","heartbeat_back_addr":"172.21.15.72:6809/661906791","heartbeat_front_addr":"172.21.15.72:6807/661906791","state":["exists","up"]},{"osd":1,"uuid":"981290ba-97dc-496b-aa22-d51704184b10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6801","nonce":2870590550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6803","nonce":2870590550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6807","nonce":2870590550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":2870590550},{"type":"v1","addr":"172.21.15.89:6805","nonce":2870590550}]},"public_addr":"172.21.15.89:6801/2870590550","cluster_addr":"172.21.15.89:6803/2870590550","heartbeat_back_addr":"172.21.15.89:6807/2870590550","heartbeat_front_addr":"172.21.15.89:6805/2870590550","state":["exists","up"]},{"osd":2,"uuid":"f07ed658-f545-46a1-8edd-8b1d5d49885c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6801","nonce":3865778118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6803","nonce":3865778118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6807","nonce":3865778118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3865778118},{"type":"v1","addr":"172.21.15.98:6805","nonce":3865778118}]},"public_addr":"172.21.15.98:6801/3865778118","cluster_addr":"172.21.15.98:6803/3865778118","heartbeat_back_addr":"172.21.15.98:6807/3865778118","heartbeat_front_addr":"172.21.15.98:6805/3865778118","state":["exists","up"]},{"osd":3,"uuid":"ccc4bc40-2c48-46d2-8548-0c05b44f9f1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6811","nonce":1865951395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6813","nonce":1865951395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6817","nonce":1865951395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":1865951395},{"type":"v1","addr":"172.21.15.72:6815","nonce":1865951395}]},"public_addr":"172.21.15.72:6811/1865951395","cluster_addr":"172.21.15.72:6813/1865951395","heartbeat_back_addr":"172.21.15.72:6817/1865951395","heartbeat_front_addr":"172.21.15.72:6815/1865951395","state":["exists","up"]},{"osd":4,"uuid":"1812439f-810a-4ed2-9395-c00c9a085a13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6809","nonce":2683760795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6811","nonce":2683760795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6815","nonce":2683760795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":2683760795},{"type":"v1","addr":"172.21.15.89:6813","nonce":2683760795}]},"public_addr":"172.21.15.89:6809/2683760795","cluster_addr":"172.21.15.89:6811/2683760795","heartbeat_back_addr":"172.21.15.89:6815/2683760795","heartbeat_front_addr":"172.21.15.89:6813/2683760795","state":["exists","up"]},{"osd":5,"uuid":"e78954d8-f4e3-4d58-8536-d2d59c843386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6809","nonce":2635838572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6811","nonce":2635838572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6815","nonce":2635838572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2635838572},{"type":"v1","addr":"172.21.15.98:6813","nonce":2635838572}]},"public_addr":"172.21.15.98:6809/2635838572","cluster_addr":"172.21.15.98:6811/2635838572","heartbeat_back_addr":"172.21.15.98:6815/2635838572","heartbeat_front_addr":"172.21.15.98:6813/2635838572","state":["exists","up"]},{"osd":6,"uuid":"dc6c8be2-b64c-4959-8015-7b0ea6d25599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6819","nonce":3158982661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6821","nonce":3158982661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6825","nonce":3158982661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":3158982661},{"type":"v1","addr":"172.21.15.72:6823","nonce":3158982661}]},"public_addr":"172.21.15.72:6819/3158982661","cluster_addr":"172.21.15.72:6821/3158982661","heartbeat_back_addr":"172.21.15.72:6825/3158982661","heartbeat_front_addr":"172.21.15.72:6823/3158982661","state":["exists","up"]},{"osd":7,"uuid":"7a73469f-cf3a-4306-ae93-ea46e21ebbbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6817","nonce":3075913234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6819","nonce":3075913234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6823","nonce":3075913234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":3075913234},{"type":"v1","addr":"172.21.15.89:6821","nonce":3075913234}]},"public_addr":"172.21.15.89:6817/3075913234","cluster_addr":"172.21.15.89:6819/3075913234","heartbeat_back_addr":"172.21.15.89:6823/3075913234","heartbeat_front_addr":"172.21.15.89:6821/3075913234","state":["exists","up"]},{"osd":8,"uuid":"c4b13bee-ea4d-4344-95fb-2622ecbb8196","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6817","nonce":1813255577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6819","nonce":1813255577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6823","nonce":1813255577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1813255577},{"type":"v1","addr":"172.21.15.98:6821","nonce":1813255577}]},"public_addr":"172.21.15.98:6817/1813255577","cluster_addr":"172.21.15.98:6819/1813255577","heartbeat_back_addr":"172.21.15.98:6823/1813255577","heartbeat_front_addr":"172.21.15.98:6821/1813255577","state":["exists","up"]},{"osd":9,"uuid":"eefb0a44-4a36-4e55-9d04-0501bf718db6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6827","nonce":3377749206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6829","nonce":3377749206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6832","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6833","nonce":3377749206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3377749206},{"type":"v1","addr":"172.21.15.72:6831","nonce":3377749206}]},"public_addr":"172.21.15.72:6827/3377749206","cluster_addr":"172.21.15.72:6829/3377749206","heartbeat_back_addr":"172.21.15.72:6833/3377749206","heartbeat_front_addr":"172.21.15.72:6831/3377749206","state":["exists","up"]},{"osd":10,"uuid":"8741d25f-40dd-40d7-b040-5cc5e411eb98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6825","nonce":885058786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6827","nonce":885058786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6831","nonce":885058786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":885058786},{"type":"v1","addr":"172.21.15.89:6829","nonce":885058786}]},"public_addr":"172.21.15.89:6825/885058786","cluster_addr":"172.21.15.89:6827/885058786","heartbeat_back_addr":"172.21.15.89:6831/885058786","heartbeat_front_addr":"172.21.15.89:6829/885058786","state":["exists","up"]},{"osd":11,"uuid":"8d02e957-eda4-4184-b5c6-5eb3b8b38060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6825","nonce":1517095838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6827","nonce":1517095838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6831","nonce":1517095838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1517095838},{"type":"v1","addr":"172.21.15.98:6829","nonce":1517095838}]},"public_addr":"172.21.15.98:6825/1517095838","cluster_addr":"172.21.15.98:6827/1517095838","heartbeat_back_addr":"172.21.15.98:6831/1517095838","heartbeat_front_addr":"172.21.15.98:6829/1517095838","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:28:47.307652+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:11.037135+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:34.619188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:29:59.350738+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:30:23.787337+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:30:48.994154+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:31:13.690350+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:31:39.628116+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:04.723912+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:30.689741+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:32:57.229508+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T01:33:23.154537+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":7}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:0/2032770149":"2024-03-23T01:36:38.183806+0000","172.21.15.72:6801/401403200":"2024-03-23T01:36:38.183806+0000","172.21.15.72:0/1161486989":"2024-03-23T01:36:38.183806+0000","172.21.15.72:6800/401403200":"2024-03-23T01:36:38.183806+0000","172.21.15.72:0/121087196":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/2451857761":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/1228477995":"2024-03-23T01:27:15.339809+0000","172.21.15.72:6801/1513232913":"2024-03-23T01:27:15.339809+0000","172.21.15.72:0/4223447038":"2024-03-23T01:36:38.183806+0000","172.21.15.72:0/3558958264":"2024-03-23T01:26:50.074871+0000","172.21.15.72:0/1510187198":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6800/3518012916":"2024-03-23T01:26:50.074871+0000","172.21.15.72:0/608566205":"2024-03-23T01:36:38.183806+0000","172.21.15.72:6801/3518012916":"2024-03-23T01:26:50.074871+0000","172.21.15.72:0/1372801041":"2024-03-23T01:26:50.074871+0000","172.21.15.72:6800/1513232913":"2024-03-23T01:27:15.339809+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T01:47:56.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.914+0000 7f148f6f4640 1 -- 172.21.15.72:0/1545535021 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f14740a3090 msgr2=0x7f14740a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:56.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.914+0000 7f148f6f4640 1 --2- 172.21.15.72:0/1545535021 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f14740a3090 0x7f14740a5550 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1488105010 tx=0x7f1478005f90 comp rx=0 tx=0).stop 2024-03-22T01:47:56.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.914+0000 7f148f6f4640 1 -- 172.21.15.72:0/1545535021 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1488100a20 msgr2=0x7f1488197270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:56.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.914+0000 7f148f6f4640 1 --2- 172.21.15.72:0/1545535021 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1488100a20 0x7f1488197270 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f148000c920 tx=0x7f148000cde0 comp rx=0 tx=0).stop 2024-03-22T01:47:56.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.914+0000 7f148f6f4640 1 -- 172.21.15.72:0/1545535021 shutdown_connections 2024-03-22T01:47:56.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.914+0000 7f148f6f4640 1 --2- 172.21.15.72:0/1545535021 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f14740a3090 0x7f14740a5550 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:56.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.914+0000 7f148f6f4640 1 --2- 172.21.15.72:0/1545535021 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1488106a30 0x7f148819bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:56.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.914+0000 7f148f6f4640 1 --2- 172.21.15.72:0/1545535021 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14881013e0 0x7f14881977b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:56.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.914+0000 7f148f6f4640 1 --2- 172.21.15.72:0/1545535021 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1488100a20 0x7f1488197270 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:56.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.914+0000 7f148f6f4640 1 -- 172.21.15.72:0/1545535021 >> 172.21.15.72:0/1545535021 conn(0x7f1488077ca0 msgr2=0x7f1488103130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:47:56.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.914+0000 7f148f6f4640 1 -- 172.21.15.72:0/1545535021 shutdown_connections 2024-03-22T01:47:56.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:56.914+0000 7f148f6f4640 1 -- 172.21.15.72:0/1545535021 wait complete. 2024-03-22T01:47:56.925 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-22T01:47:57.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.028+0000 7f2426155640 1 -- 172.21.15.72:0/2138983560 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2420107c00 msgr2=0x7f2420107fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:57.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.028+0000 7f2426155640 1 --2- 172.21.15.72:0/2138983560 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2420107c00 0x7f2420107fe0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f2410009900 tx=0x7f241002bc30 comp rx=0 tx=0).stop 2024-03-22T01:47:57.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.028+0000 7f2426155640 1 -- 172.21.15.72:0/2138983560 shutdown_connections 2024-03-22T01:47:57.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.028+0000 7f2426155640 1 --2- 172.21.15.72:0/2138983560 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2420102500 0x7f242010e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:57.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.028+0000 7f2426155640 1 --2- 172.21.15.72:0/2138983560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2420101b40 0x7f2420101fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:57.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.028+0000 7f2426155640 1 --2- 172.21.15.72:0/2138983560 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2420107c00 0x7f2420107fe0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:57.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.028+0000 7f2426155640 1 -- 172.21.15.72:0/2138983560 >> 172.21.15.72:0/2138983560 conn(0x7f24200fd630 msgr2=0x7f24200ffa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:47:57.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.028+0000 7f2426155640 1 -- 172.21.15.72:0/2138983560 shutdown_connections 2024-03-22T01:47:57.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.028+0000 7f2426155640 1 -- 172.21.15.72:0/2138983560 wait complete. 2024-03-22T01:47:57.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.028+0000 7f2426155640 1 Processor -- start 2024-03-22T01:47:57.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.029+0000 7f2426155640 1 -- start start 2024-03-22T01:47:57.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.029+0000 7f2426155640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2420101b40 0x7f2420198360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:57.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.029+0000 7f2426155640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2420102500 0x7f24201988a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:57.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.029+0000 7f2426155640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2420107c00 0x7f242019cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:57.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.029+0000 7f2426155640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f2420111f00 con 0x7f2420107c00 2024-03-22T01:47:57.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.029+0000 7f2426155640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2420111d80 con 0x7f2420102500 2024-03-22T01:47:57.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.029+0000 7f2426155640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2420112080 con 0x7f2420101b40 2024-03-22T01:47:57.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.029+0000 7f241ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2420107c00 0x7f242019cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:57.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.029+0000 7f241ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2420107c00 0x7f242019cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36076/0 (socket says 172.21.15.72:36076) 2024-03-22T01:47:57.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.029+0000 7f241ffff640 1 -- 172.21.15.72:0/450222743 learned_addr learned my addr 172.21.15.72:0/450222743 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:47:57.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.029+0000 7f241f7fe640 1 --2- 172.21.15.72:0/450222743 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2420101b40 0x7f2420198360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:57.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.029+0000 7f241effd640 1 --2- 172.21.15.72:0/450222743 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2420102500 0x7f24201988a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:57.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.029+0000 7f241ffff640 1 -- 172.21.15.72:0/450222743 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2420101b40 msgr2=0x7f2420198360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:57.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.029+0000 7f241ffff640 1 --2- 172.21.15.72:0/450222743 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2420101b40 0x7f2420198360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:57.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.029+0000 7f241ffff640 1 -- 172.21.15.72:0/450222743 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2420102500 msgr2=0x7f24201988a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:57.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.029+0000 7f241ffff640 1 --2- 172.21.15.72:0/450222743 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2420102500 0x7f24201988a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:57.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.029+0000 7f241ffff640 1 -- 172.21.15.72:0/450222743 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f242019d350 con 0x7f2420107c00 2024-03-22T01:47:57.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.029+0000 7f241effd640 1 --2- 172.21.15.72:0/450222743 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2420102500 0x7f24201988a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T01:47:57.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.029+0000 7f241ffff640 1 --2- 172.21.15.72:0/450222743 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2420107c00 0x7f242019cc20 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f241800eb10 tx=0x7f241800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:47:57.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.030+0000 7f241cff9640 1 -- 172.21.15.72:0/450222743 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2418004b20 con 0x7f2420107c00 2024-03-22T01:47:57.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.030+0000 7f2426155640 1 -- 172.21.15.72:0/450222743 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f242019d640 con 0x7f2420107c00 2024-03-22T01:47:57.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.030+0000 7f241cff9640 1 -- 172.21.15.72:0/450222743 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2418004cc0 con 0x7f2420107c00 2024-03-22T01:47:57.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.030+0000 7f241cff9640 1 -- 172.21.15.72:0/450222743 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2418010760 con 0x7f2420107c00 2024-03-22T01:47:57.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.030+0000 7f2426155640 1 -- 172.21.15.72:0/450222743 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24201a4ed0 con 0x7f2420107c00 2024-03-22T01:47:57.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.031+0000 7f2426155640 1 -- 172.21.15.72:0/450222743 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2420068e60 con 0x7f2420107c00 2024-03-22T01:47:57.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.032+0000 7f241cff9640 1 -- 172.21.15.72:0/450222743 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 25) ==== 137985+0+0 (secure 0 0 0) 0x7f2418005020 con 0x7f2420107c00 2024-03-22T01:47:57.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.032+0000 7f241cff9640 1 --2- 172.21.15.72:0/450222743 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f24080a3090 0x7f24080a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:57.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.032+0000 7f241f7fe640 1 --2- 172.21.15.72:0/450222743 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f24080a3090 0x7f24080a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:57.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.032+0000 7f241cff9640 1 -- 172.21.15.72:0/450222743 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f24180d0110 con 0x7f2420107c00 2024-03-22T01:47:57.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.033+0000 7f241f7fe640 1 --2- 172.21.15.72:0/450222743 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f24080a3090 0x7f24080a5550 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2410009230 tx=0x7f2410005fd0 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:47:57.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.037+0000 7f241cff9640 1 -- 172.21.15.72:0/450222743 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f241809c850 con 0x7f2420107c00 2024-03-22T01:47:57.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.249+0000 7f2426155640 1 -- 172.21.15.72:0/450222743 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f2420109a60 con 0x7f2420107c00 2024-03-22T01:47:57.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.251+0000 7f241cff9640 1 -- 172.21.15.72:0/450222743 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v104) ==== 161+0+0 (secure 0 0 0) 0x7f241809c150 con 0x7f2420107c00 2024-03-22T01:47:57.250 INFO:teuthology.orchestra.run.smithi072.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-22T01:47:57.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.253+0000 7f2426155640 1 -- 172.21.15.72:0/450222743 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f24080a3090 msgr2=0x7f24080a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:57.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.253+0000 7f2426155640 1 --2- 172.21.15.72:0/450222743 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f24080a3090 0x7f24080a5550 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2410009230 tx=0x7f2410005fd0 comp rx=0 tx=0).stop 2024-03-22T01:47:57.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.253+0000 7f2426155640 1 -- 172.21.15.72:0/450222743 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2420107c00 msgr2=0x7f242019cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:57.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.253+0000 7f2426155640 1 --2- 172.21.15.72:0/450222743 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2420107c00 0x7f242019cc20 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f241800eb10 tx=0x7f241800c2d0 comp rx=0 tx=0).stop 2024-03-22T01:47:57.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.254+0000 7f2426155640 1 -- 172.21.15.72:0/450222743 shutdown_connections 2024-03-22T01:47:57.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.254+0000 7f2426155640 1 --2- 172.21.15.72:0/450222743 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f24080a3090 0x7f24080a5550 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:57.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.254+0000 7f2426155640 1 --2- 172.21.15.72:0/450222743 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2420102500 0x7f24201988a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:57.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.254+0000 7f2426155640 1 --2- 172.21.15.72:0/450222743 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2420107c00 0x7f242019cc20 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:57.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.254+0000 7f2426155640 1 --2- 172.21.15.72:0/450222743 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2420101b40 0x7f2420198360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:57.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.254+0000 7f2426155640 1 -- 172.21.15.72:0/450222743 >> 172.21.15.72:0/450222743 conn(0x7f24200fd630 msgr2=0x7f24200fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:47:57.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.254+0000 7f2426155640 1 -- 172.21.15.72:0/450222743 shutdown_connections 2024-03-22T01:47:57.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.254+0000 7f2426155640 1 -- 172.21.15.72:0/450222743 wait complete. 2024-03-22T01:47:57.265 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-22T01:47:57.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.373+0000 7f79f15a0640 1 -- 172.21.15.72:0/3444839711 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79ec106840 msgr2=0x7f79ec106c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:57.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.373+0000 7f79f15a0640 1 --2- 172.21.15.72:0/3444839711 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79ec106840 0x7f79ec106c20 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f79dc009930 tx=0x7f79dc02bc30 comp rx=0 tx=0).stop 2024-03-22T01:47:57.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.373+0000 7f79f15a0640 1 -- 172.21.15.72:0/3444839711 shutdown_connections 2024-03-22T01:47:57.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.373+0000 7f79f15a0640 1 --2- 172.21.15.72:0/3444839711 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79ec1011f0 0x7f79ec10d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:57.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.373+0000 7f79f15a0640 1 --2- 172.21.15.72:0/3444839711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79ec100830 0x7f79ec100cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:57.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.373+0000 7f79f15a0640 1 --2- 172.21.15.72:0/3444839711 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79ec106840 0x7f79ec106c20 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:57.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.373+0000 7f79f15a0640 1 -- 172.21.15.72:0/3444839711 >> 172.21.15.72:0/3444839711 conn(0x7f79ec0fc340 msgr2=0x7f79ec0fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:47:57.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.373+0000 7f79f15a0640 1 -- 172.21.15.72:0/3444839711 shutdown_connections 2024-03-22T01:47:57.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.373+0000 7f79f15a0640 1 -- 172.21.15.72:0/3444839711 wait complete. 2024-03-22T01:47:57.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.373+0000 7f79f15a0640 1 Processor -- start 2024-03-22T01:47:57.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.373+0000 7f79f15a0640 1 -- start start 2024-03-22T01:47:57.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.374+0000 7f79f15a0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79ec100830 0x7f79ec1990b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:57.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.374+0000 7f79f15a0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79ec1011f0 0x7f79ec1995f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:57.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.374+0000 7f79f15a0640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79ec106840 0x7f79ec1a4130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:57.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.374+0000 7f79f15a0640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f79ec110c50 con 0x7f79ec106840 2024-03-22T01:47:57.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.374+0000 7f79f15a0640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f79ec110ad0 con 0x7f79ec100830 2024-03-22T01:47:57.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.374+0000 7f79f15a0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f79ec110dd0 con 0x7f79ec1011f0 2024-03-22T01:47:57.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.374+0000 7f79f0d9f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79ec106840 0x7f79ec1a4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:57.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.374+0000 7f79f0d9f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79ec106840 0x7f79ec1a4130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36096/0 (socket says 172.21.15.72:36096) 2024-03-22T01:47:57.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.374+0000 7f79f0d9f640 1 -- 172.21.15.72:0/3116348588 learned_addr learned my addr 172.21.15.72:0/3116348588 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:47:57.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.374+0000 7f79eb7fe640 1 --2- 172.21.15.72:0/3116348588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79ec1011f0 0x7f79ec1995f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:57.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.374+0000 7f79ebfff640 1 --2- 172.21.15.72:0/3116348588 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79ec100830 0x7f79ec1990b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:57.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.374+0000 7f79f0d9f640 1 -- 172.21.15.72:0/3116348588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79ec1011f0 msgr2=0x7f79ec1995f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:57.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.374+0000 7f79f0d9f640 1 --2- 172.21.15.72:0/3116348588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79ec1011f0 0x7f79ec1995f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:57.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.374+0000 7f79f0d9f640 1 -- 172.21.15.72:0/3116348588 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79ec100830 msgr2=0x7f79ec1990b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:57.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.374+0000 7f79f0d9f640 1 --2- 172.21.15.72:0/3116348588 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79ec100830 0x7f79ec1990b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:57.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.374+0000 7f79f0d9f640 1 -- 172.21.15.72:0/3116348588 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79ec1a4670 con 0x7f79ec106840 2024-03-22T01:47:57.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.374+0000 7f79f0d9f640 1 --2- 172.21.15.72:0/3116348588 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79ec106840 0x7f79ec1a4130 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f79e400eb10 tx=0x7f79e400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:47:57.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.374+0000 7f79e97fa640 1 -- 172.21.15.72:0/3116348588 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e4004b20 con 0x7f79ec106840 2024-03-22T01:47:57.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.375+0000 7f79e97fa640 1 -- 172.21.15.72:0/3116348588 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79e4004cc0 con 0x7f79ec106840 2024-03-22T01:47:57.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.375+0000 7f79f15a0640 1 -- 172.21.15.72:0/3116348588 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79ec1a4840 con 0x7f79ec106840 2024-03-22T01:47:57.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.375+0000 7f79e97fa640 1 -- 172.21.15.72:0/3116348588 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e4010760 con 0x7f79ec106840 2024-03-22T01:47:57.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.375+0000 7f79f15a0640 1 -- 172.21.15.72:0/3116348588 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79ec1a4d10 con 0x7f79ec106840 2024-03-22T01:47:57.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.375+0000 7f79f15a0640 1 -- 172.21.15.72:0/3116348588 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79ec04f8f0 con 0x7f79ec106840 2024-03-22T01:47:57.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.376+0000 7f79e97fa640 1 -- 172.21.15.72:0/3116348588 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 25) ==== 137985+0+0 (secure 0 0 0) 0x7f79e4005020 con 0x7f79ec106840 2024-03-22T01:47:57.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.377+0000 7f79e97fa640 1 --2- 172.21.15.72:0/3116348588 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f79d40a3090 0x7f79d40a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:57.379 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.377+0000 7f79ebfff640 1 --2- 172.21.15.72:0/3116348588 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f79d40a3090 0x7f79d40a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:57.379 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.377+0000 7f79e97fa640 1 -- 172.21.15.72:0/3116348588 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9552+0+0 (secure 0 0 0) 0x7f79e40d01b0 con 0x7f79ec106840 2024-03-22T01:47:57.379 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.377+0000 7f79ebfff640 1 --2- 172.21.15.72:0/3116348588 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f79d40a3090 0x7f79d40a5550 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f79dc006010 tx=0x7f79dc000bc0 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:47:57.380 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.381+0000 7f79e97fa640 1 -- 172.21.15.72:0/3116348588 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f79e409c8a0 con 0x7f79ec106840 2024-03-22T01:47:57.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:57.595+0000 7f79f15a0640 1 -- 172.21.15.72:0/3116348588 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f79ec02d3b0 con 0x7f79ec106840 2024-03-22T01:47:57.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:57 smithi089 ceph-mon[35770]: pgmap v342: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 170 B/s wr, 2 op/s 2024-03-22T01:47:57.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:57 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/2929120188' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:47:57.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:57 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/1545535021' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:47:57.609 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:57 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/450222743' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-22T01:47:57.609 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:57 smithi089 ceph-mon[35770]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-22T01:47:57.609 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:57 smithi089 ceph-mon[35770]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-22T01:47:57.609 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:57 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/450222743' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-22T01:47:57.609 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:57 smithi089 ceph-mon[35770]: osdmap e97: 12 total, 12 up, 12 in 2024-03-22T01:47:57.609 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:57 smithi089 ceph-mon[35770]: fsmap cephfs:0/1 10 up:standby, 1 failed 2024-03-22T01:47:57.610 INFO:journalctl@ceph.mds.b.smithi089.stdout:Mar 22 01:47:57 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-b[66811]: ignoring --setuser ceph since I am not root 2024-03-22T01:47:57.610 INFO:journalctl@ceph.mds.b.smithi089.stdout:Mar 22 01:47:57 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-b[66811]: ignoring --setgroup ceph since I am not root 2024-03-22T01:47:57.610 INFO:journalctl@ceph.mds.b.smithi089.stdout:Mar 22 01:47:57 smithi089 ceph-mds[66815]: -- 172.21.15.89:0/898036686 <== mon.2 v2:172.21.15.89:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5588d0133a40 con 0x5588d0106800 2024-03-22T01:47:57.610 INFO:journalctl@ceph.mds.b.smithi089.stdout:Mar 22 01:47:57 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-b[66811]: starting mds.b at 2024-03-22T01:47:57.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:57 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mon-a[32410]: 2024-03-22T01:47:57.250+0000 7f4d524e0700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-22T01:47:57.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:57 smithi072 ceph-mon[32414]: pgmap v342: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 170 B/s wr, 2 op/s 2024-03-22T01:47:57.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:57 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/2929120188' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:47:57.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:57 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/1545535021' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:47:57.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:57 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/450222743' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-22T01:47:57.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:57 smithi072 ceph-mon[32414]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-22T01:47:57.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:57 smithi072 ceph-mon[32414]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-22T01:47:57.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:57 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/450222743' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-22T01:47:57.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:57 smithi072 ceph-mon[32414]: osdmap e97: 12 total, 12 up, 12 in 2024-03-22T01:47:57.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:57 smithi072 ceph-mon[32414]: fsmap cephfs:0/1 10 up:standby, 1 failed 2024-03-22T01:47:57.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:57 smithi098 ceph-mon[36468]: pgmap v342: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 170 B/s wr, 2 op/s 2024-03-22T01:47:57.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:57 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/2929120188' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T01:47:57.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:57 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/1545535021' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T01:47:57.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:57 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/450222743' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-22T01:47:57.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:57 smithi098 ceph-mon[36468]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-22T01:47:57.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:57 smithi098 ceph-mon[36468]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-22T01:47:57.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:57 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/450222743' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-22T01:47:57.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:57 smithi098 ceph-mon[36468]: osdmap e97: 12 total, 12 up, 12 in 2024-03-22T01:47:57.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:57 smithi098 ceph-mon[36468]: fsmap cephfs:0/1 10 up:standby, 1 failed 2024-03-22T01:47:57.672 INFO:journalctl@ceph.mds.i.smithi098.stdout:Mar 22 01:47:57 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-i[68331]: ignoring --setuser ceph since I am not root 2024-03-22T01:47:57.673 INFO:journalctl@ceph.mds.i.smithi098.stdout:Mar 22 01:47:57 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-i[68331]: ignoring --setgroup ceph since I am not root 2024-03-22T01:47:57.673 INFO:journalctl@ceph.mds.i.smithi098.stdout:Mar 22 01:47:57 smithi098 ceph-mds[68335]: -- 172.21.15.98:0/283587158 <== mon.2 v2:172.21.15.89:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5595fc7bda40 con 0x5595fc790c00 2024-03-22T01:47:57.673 INFO:journalctl@ceph.mds.i.smithi098.stdout:Mar 22 01:47:57 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-i[68331]: starting mds.i at 2024-03-22T01:47:58.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:58.253+0000 7f79e97fa640 1 -- 172.21.15.72:0/3116348588 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v105) ==== 104+0+0 (secure 0 0 0) 0x7f79e409c1a0 con 0x7f79ec106840 2024-03-22T01:47:58.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:58.256+0000 7f79f15a0640 1 -- 172.21.15.72:0/3116348588 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f79d40a3090 msgr2=0x7f79d40a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:58.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:58.256+0000 7f79f15a0640 1 --2- 172.21.15.72:0/3116348588 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f79d40a3090 0x7f79d40a5550 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f79dc006010 tx=0x7f79dc000bc0 comp rx=0 tx=0).stop 2024-03-22T01:47:58.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:58.256+0000 7f79f15a0640 1 -- 172.21.15.72:0/3116348588 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79ec106840 msgr2=0x7f79ec1a4130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:58.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:58.256+0000 7f79f15a0640 1 --2- 172.21.15.72:0/3116348588 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79ec106840 0x7f79ec1a4130 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f79e400eb10 tx=0x7f79e400c2d0 comp rx=0 tx=0).stop 2024-03-22T01:47:58.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:58.256+0000 7f79f15a0640 1 -- 172.21.15.72:0/3116348588 shutdown_connections 2024-03-22T01:47:58.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:58.256+0000 7f79f15a0640 1 --2- 172.21.15.72:0/3116348588 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f79d40a3090 0x7f79d40a5550 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:58.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:58.256+0000 7f79f15a0640 1 --2- 172.21.15.72:0/3116348588 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79ec106840 0x7f79ec1a4130 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:58.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:58.256+0000 7f79f15a0640 1 --2- 172.21.15.72:0/3116348588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79ec1011f0 0x7f79ec1995f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:58.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:58.256+0000 7f79f15a0640 1 --2- 172.21.15.72:0/3116348588 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79ec100830 0x7f79ec1990b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:58.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:58.256+0000 7f79f15a0640 1 -- 172.21.15.72:0/3116348588 >> 172.21.15.72:0/3116348588 conn(0x7f79ec0fc340 msgr2=0x7f79ec0fcb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:47:58.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:58.256+0000 7f79f15a0640 1 -- 172.21.15.72:0/3116348588 shutdown_connections 2024-03-22T01:47:58.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:58.256+0000 7f79f15a0640 1 -- 172.21.15.72:0/3116348588 wait complete. 2024-03-22T01:47:58.270 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-22T01:47:58.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:58.383+0000 7f34f489a640 1 -- 172.21.15.72:0/3864259410 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f34f0102500 msgr2=0x7f34f010e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:58.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:58.383+0000 7f34f489a640 1 --2- 172.21.15.72:0/3864259410 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f34f0102500 0x7f34f010e9c0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f34e8009ca0 tx=0x7f34e802c0f0 comp rx=0 tx=0).stop 2024-03-22T01:47:58.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:58.383+0000 7f34f489a640 1 -- 172.21.15.72:0/3864259410 shutdown_connections 2024-03-22T01:47:58.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:58.383+0000 7f34f489a640 1 --2- 172.21.15.72:0/3864259410 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34f0101b40 0x7f34f0101fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:58.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:58.383+0000 7f34f489a640 1 --2- 172.21.15.72:0/3864259410 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f34f0102500 0x7f34f010e9c0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:58.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:58.383+0000 7f34f489a640 1 --2- 172.21.15.72:0/3864259410 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34f0107c00 0x7f34f0107fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:58.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:58.383+0000 7f34f489a640 1 -- 172.21.15.72:0/3864259410 >> 172.21.15.72:0/3864259410 conn(0x7f34f00fd630 msgr2=0x7f34f00ffa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:47:58.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:58.383+0000 7f34f489a640 1 -- 172.21.15.72:0/3864259410 shutdown_connections 2024-03-22T01:47:58.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:58.383+0000 7f34f489a640 1 -- 172.21.15.72:0/3864259410 wait complete. 2024-03-22T01:47:58.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:58.384+0000 7f34f489a640 1 Processor -- start 2024-03-22T01:47:58.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:58.384+0000 7f34f489a640 1 -- start start 2024-03-22T01:47:58.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:58.384+0000 7f34f489a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f34f0101b40 0x7f34f019a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:58.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:58.384+0000 7f34f489a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34f0102500 0x7f34f019aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:58.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:58.384+0000 7f34f489a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34f0107c00 0x7f34f019ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:58.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:58.384+0000 7f34f489a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f34f0111f40 con 0x7f34f0101b40 2024-03-22T01:47:58.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:58.384+0000 7f34f489a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f34f0111dc0 con 0x7f34f0107c00 2024-03-22T01:47:58.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:58.384+0000 7f34f489a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f34f01120c0 con 0x7f34f0102500 2024-03-22T01:47:58.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:58.384+0000 7f34ee59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f34f0101b40 0x7f34f019a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:58.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:58.384+0000 7f34ee59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f34f0101b40 0x7f34f019a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36108/0 (socket says 172.21.15.72:36108) 2024-03-22T01:47:58.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:58.384+0000 7f34ee59c640 1 -- 172.21.15.72:0/1107710108 learned_addr learned my addr 172.21.15.72:0/1107710108 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:47:58.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:58.384+0000 7f34ee59c640 1 -- 172.21.15.72:0/1107710108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34f0102500 msgr2=0x7f34f019aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:58.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:58.384+0000 7f34ee59c640 1 --2- 172.21.15.72:0/1107710108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34f0102500 0x7f34f019aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:58.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:58.384+0000 7f34ee59c640 1 -- 172.21.15.72:0/1107710108 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34f0107c00 msgr2=0x7f34f019ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:58.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:58.384+0000 7f34ee59c640 1 --2- 172.21.15.72:0/1107710108 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34f0107c00 0x7f34f019ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:58.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:58.384+0000 7f34ee59c640 1 -- 172.21.15.72:0/1107710108 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34f019f590 con 0x7f34f0101b40 2024-03-22T01:47:58.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:58.385+0000 7f34ee59c640 1 --2- 172.21.15.72:0/1107710108 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f34f0101b40 0x7f34f019a5a0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f34e000d770 tx=0x7f34e000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:47:58.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:58.385+0000 7f34e77fe640 1 -- 172.21.15.72:0/1107710108 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34e0017070 con 0x7f34f0101b40 2024-03-22T01:47:58.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:58.385+0000 7f34e77fe640 1 -- 172.21.15.72:0/1107710108 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34e0005500 con 0x7f34f0101b40 2024-03-22T01:47:58.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:58.385+0000 7f34f489a640 1 -- 172.21.15.72:0/1107710108 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34f019f880 con 0x7f34f0101b40 2024-03-22T01:47:58.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:58.385+0000 7f34e77fe640 1 -- 172.21.15.72:0/1107710108 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34e0004e00 con 0x7f34f0101b40 2024-03-22T01:47:58.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:58.385+0000 7f34f489a640 1 -- 172.21.15.72:0/1107710108 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34f01a7110 con 0x7f34f0101b40 2024-03-22T01:47:58.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:58.386+0000 7f34f489a640 1 -- 172.21.15.72:0/1107710108 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34f0068e60 con 0x7f34f0101b40 2024-03-22T01:47:58.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:58.387+0000 7f34e77fe640 1 -- 172.21.15.72:0/1107710108 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 25) ==== 137985+0+0 (secure 0 0 0) 0x7f34e0004fa0 con 0x7f34f0101b40 2024-03-22T01:47:58.393 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:58.387+0000 7f34e77fe640 1 --2- 172.21.15.72:0/1107710108 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f34dc0a3090 0x7f34dc0a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:58.393 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:58.387+0000 7f34e7fff640 1 --2- 172.21.15.72:0/1107710108 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f34dc0a3090 0x7f34dc0a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:58.393 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:58.387+0000 7f34e77fe640 1 -- 172.21.15.72:0/1107710108 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9552+0+0 (secure 0 0 0) 0x7f34e00cf650 con 0x7f34f0101b40 2024-03-22T01:47:58.393 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:58.388+0000 7f34e7fff640 1 --2- 172.21.15.72:0/1107710108 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f34dc0a3090 0x7f34dc0a5550 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f34f010e650 tx=0x7f34d8005c70 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:47:58.393 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:58.392+0000 7f34e77fe640 1 -- 172.21.15.72:0/1107710108 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f34e009bd40 con 0x7f34f0101b40 2024-03-22T01:47:58.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:58.568+0000 7f34f489a640 1 -- 172.21.15.72:0/1107710108 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f34f01a7440 con 0x7f34f0101b40 2024-03-22T01:47:58.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:58 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/3116348588' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-22T01:47:58.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:58 smithi072 ceph-mon[32414]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-22T01:47:58.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:58 smithi072 ceph-mon[32414]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-22T01:47:58.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:58 smithi072 ceph-mon[32414]: Cluster is now healthy 2024-03-22T01:47:58.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:58 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/3116348588' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-22T01:47:58.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:58 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.89:6838/792476396,v1:172.21.15.89:6839/792476396] up:boot 2024-03-22T01:47:58.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:58 smithi072 ceph-mon[32414]: mds.? [v2:172.21.15.98:6838/2390014404,v1:172.21.15.98:6839/2390014404] up:boot 2024-03-22T01:47:58.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:58 smithi072 ceph-mon[32414]: fsmap 12 up:standby 2024-03-22T01:47:58.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:58 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-22T01:47:58.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:58 smithi072 ceph-mon[32414]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-22T01:47:58.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:58 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/3116348588' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-22T01:47:58.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:58 smithi089 ceph-mon[35770]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-22T01:47:58.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:58 smithi089 ceph-mon[35770]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-22T01:47:58.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:58 smithi089 ceph-mon[35770]: Cluster is now healthy 2024-03-22T01:47:58.859 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:58 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/3116348588' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-22T01:47:58.859 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:58 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.89:6838/792476396,v1:172.21.15.89:6839/792476396] up:boot 2024-03-22T01:47:58.859 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:58 smithi089 ceph-mon[35770]: mds.? [v2:172.21.15.98:6838/2390014404,v1:172.21.15.98:6839/2390014404] up:boot 2024-03-22T01:47:58.859 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:58 smithi089 ceph-mon[35770]: fsmap 12 up:standby 2024-03-22T01:47:58.859 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:58 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-22T01:47:58.859 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:58 smithi089 ceph-mon[35770]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-22T01:47:58.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:58 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/3116348588' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-22T01:47:58.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:58 smithi098 ceph-mon[36468]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-22T01:47:58.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:58 smithi098 ceph-mon[36468]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-22T01:47:58.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:58 smithi098 ceph-mon[36468]: Cluster is now healthy 2024-03-22T01:47:58.923 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:58 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/3116348588' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-22T01:47:58.923 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:58 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.89:6838/792476396,v1:172.21.15.89:6839/792476396] up:boot 2024-03-22T01:47:58.923 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:58 smithi098 ceph-mon[36468]: mds.? [v2:172.21.15.98:6838/2390014404,v1:172.21.15.98:6839/2390014404] up:boot 2024-03-22T01:47:58.923 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:58 smithi098 ceph-mon[36468]: fsmap 12 up:standby 2024-03-22T01:47:58.923 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:58 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-22T01:47:58.923 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:58 smithi098 ceph-mon[36468]: from='mgr.15381 172.21.15.72:0/1510714844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-22T01:47:59.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:59.486+0000 7f34e77fe640 1 -- 172.21.15.72:0/1107710108 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v98) ==== 181+0+0 (secure 0 0 0) 0x7f34e0097020 con 0x7f34f0101b40 2024-03-22T01:47:59.486 INFO:teuthology.orchestra.run.smithi072.stderr:pool 'cephfs_metadata' removed 2024-03-22T01:47:59.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:59.490+0000 7f34f489a640 1 -- 172.21.15.72:0/1107710108 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f34dc0a3090 msgr2=0x7f34dc0a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:59.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:59.490+0000 7f34f489a640 1 --2- 172.21.15.72:0/1107710108 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f34dc0a3090 0x7f34dc0a5550 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f34f010e650 tx=0x7f34d8005c70 comp rx=0 tx=0).stop 2024-03-22T01:47:59.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:59.490+0000 7f34f489a640 1 -- 172.21.15.72:0/1107710108 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f34f0101b40 msgr2=0x7f34f019a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:59.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:59.490+0000 7f34f489a640 1 --2- 172.21.15.72:0/1107710108 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f34f0101b40 0x7f34f019a5a0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f34e000d770 tx=0x7f34e000dc30 comp rx=0 tx=0).stop 2024-03-22T01:47:59.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:59.490+0000 7f34f489a640 1 -- 172.21.15.72:0/1107710108 shutdown_connections 2024-03-22T01:47:59.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:59.490+0000 7f34f489a640 1 --2- 172.21.15.72:0/1107710108 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f34dc0a3090 0x7f34dc0a5550 secure :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f34f010e650 tx=0x7f34d8005c70 comp rx=0 tx=0).stop 2024-03-22T01:47:59.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:59.490+0000 7f34f489a640 1 --2- 172.21.15.72:0/1107710108 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34f0107c00 0x7f34f019ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:59.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:59.490+0000 7f34f489a640 1 --2- 172.21.15.72:0/1107710108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34f0102500 0x7f34f019aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:59.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:59.490+0000 7f34f489a640 1 --2- 172.21.15.72:0/1107710108 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f34f0101b40 0x7f34f019a5a0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:59.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:59.490+0000 7f34f489a640 1 -- 172.21.15.72:0/1107710108 >> 172.21.15.72:0/1107710108 conn(0x7f34f00fd630 msgr2=0x7f34f00fea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:47:59.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:59.491+0000 7f34f489a640 1 -- 172.21.15.72:0/1107710108 shutdown_connections 2024-03-22T01:47:59.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:59.491+0000 7f34f489a640 1 -- 172.21.15.72:0/1107710108 wait complete. 2024-03-22T01:47:59.505 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-22T01:47:59.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:59.617+0000 7fb0a0193640 1 -- 172.21.15.72:0/1389288582 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb098106a30 msgr2=0x7fb098106e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:59.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:59.617+0000 7fb0a0193640 1 --2- 172.21.15.72:0/1389288582 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb098106a30 0x7fb098106e10 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fb094009930 tx=0x7fb09402bc30 comp rx=0 tx=0).stop 2024-03-22T01:47:59.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:59.617+0000 7fb0a0193640 1 -- 172.21.15.72:0/1389288582 shutdown_connections 2024-03-22T01:47:59.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:59.617+0000 7fb0a0193640 1 --2- 172.21.15.72:0/1389288582 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0981013e0 0x7fb09810d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:59.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:59.617+0000 7fb0a0193640 1 --2- 172.21.15.72:0/1389288582 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb098100a20 0x7fb098100ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:59.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:59.617+0000 7fb0a0193640 1 --2- 172.21.15.72:0/1389288582 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb098106a30 0x7fb098106e10 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:59.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:59.617+0000 7fb0a0193640 1 -- 172.21.15.72:0/1389288582 >> 172.21.15.72:0/1389288582 conn(0x7fb098077bb0 msgr2=0x7fb098077fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:47:59.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:59.617+0000 7fb0a0193640 1 -- 172.21.15.72:0/1389288582 shutdown_connections 2024-03-22T01:47:59.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:59.617+0000 7fb0a0193640 1 -- 172.21.15.72:0/1389288582 wait complete. 2024-03-22T01:47:59.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:59.618+0000 7fb0a0193640 1 Processor -- start 2024-03-22T01:47:59.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:59.618+0000 7fb0a0193640 1 -- start start 2024-03-22T01:47:59.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:59.618+0000 7fb0a0193640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb098100a20 0x7fb098197410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:59.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:59.618+0000 7fb0a0193640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0981013e0 0x7fb098197950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:59.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:59.618+0000 7fb0a0193640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb098106a30 0x7fb09819bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:59.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:59.618+0000 7fb0a0193640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fb098110e40 con 0x7fb098100a20 2024-03-22T01:47:59.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:59.618+0000 7fb0a0193640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb098110cc0 con 0x7fb0981013e0 2024-03-22T01:47:59.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:59.618+0000 7fb0a0193640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb098110fc0 con 0x7fb098106a30 2024-03-22T01:47:59.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:59.618+0000 7fb09df2f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb098100a20 0x7fb098197410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:59.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:59.618+0000 7fb09df2f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb098100a20 0x7fb098197410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36136/0 (socket says 172.21.15.72:36136) 2024-03-22T01:47:59.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:59.618+0000 7fb09df2f640 1 -- 172.21.15.72:0/1720395647 learned_addr learned my addr 172.21.15.72:0/1720395647 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:47:59.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:59.618+0000 7fb09e730640 1 --2- 172.21.15.72:0/1720395647 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb098106a30 0x7fb09819bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:59.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:59.618+0000 7fb09df2f640 1 -- 172.21.15.72:0/1720395647 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb098106a30 msgr2=0x7fb09819bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:59.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:59.618+0000 7fb09df2f640 1 --2- 172.21.15.72:0/1720395647 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb098106a30 0x7fb09819bcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:59.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:59.618+0000 7fb09df2f640 1 -- 172.21.15.72:0/1720395647 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0981013e0 msgr2=0x7fb098197950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:47:59.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:59.618+0000 7fb09d72e640 1 --2- 172.21.15.72:0/1720395647 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0981013e0 0x7fb098197950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:59.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:59.618+0000 7fb09df2f640 1 --2- 172.21.15.72:0/1720395647 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0981013e0 0x7fb098197950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:47:59.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:59.618+0000 7fb09df2f640 1 -- 172.21.15.72:0/1720395647 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb09819c400 con 0x7fb098100a20 2024-03-22T01:47:59.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:59.618+0000 7fb09d72e640 1 --2- 172.21.15.72:0/1720395647 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0981013e0 0x7fb098197950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T01:47:59.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:59.618+0000 7fb09df2f640 1 --2- 172.21.15.72:0/1720395647 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb098100a20 0x7fb098197410 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fb094009230 tx=0x7fb09402cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:47:59.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:59.619+0000 7fb08effd640 1 -- 172.21.15.72:0/1720395647 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb094041070 con 0x7fb098100a20 2024-03-22T01:47:59.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:59.619+0000 7fb08effd640 1 -- 172.21.15.72:0/1720395647 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb094004450 con 0x7fb098100a20 2024-03-22T01:47:59.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:59.619+0000 7fb0a0193640 1 -- 172.21.15.72:0/1720395647 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb09819c690 con 0x7fb098100a20 2024-03-22T01:47:59.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:59.619+0000 7fb08effd640 1 -- 172.21.15.72:0/1720395647 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb094034440 con 0x7fb098100a20 2024-03-22T01:47:59.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:59.619+0000 7fb0a0193640 1 -- 172.21.15.72:0/1720395647 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0981a3f80 con 0x7fb098100a20 2024-03-22T01:47:59.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:59.620+0000 7fb0a0193640 1 -- 172.21.15.72:0/1720395647 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb098102160 con 0x7fb098100a20 2024-03-22T01:47:59.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:59.621+0000 7fb08effd640 1 -- 172.21.15.72:0/1720395647 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 25) ==== 137985+0+0 (secure 0 0 0) 0x7fb094044070 con 0x7fb098100a20 2024-03-22T01:47:59.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:59.622+0000 7fb08effd640 1 --2- 172.21.15.72:0/1720395647 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fb0840a3090 0x7fb0840a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:47:59.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:59.622+0000 7fb09d72e640 1 --2- 172.21.15.72:0/1720395647 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fb0840a3090 0x7fb0840a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:47:59.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:59.622+0000 7fb08effd640 1 -- 172.21.15.72:0/1720395647 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8998+0+0 (secure 0 0 0) 0x7fb0940f17b0 con 0x7fb098100a20 2024-03-22T01:47:59.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:59.622+0000 7fb09d72e640 1 --2- 172.21.15.72:0/1720395647 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fb0840a3090 0x7fb0840a5550 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb088006fd0 tx=0x7fb088008040 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:47:59.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:59.626+0000 7fb08effd640 1 -- 172.21.15.72:0/1720395647 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb0940be1d0 con 0x7fb098100a20 2024-03-22T01:47:59.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:47:59.800+0000 7fb0a0193640 1 -- 172.21.15.72:0/1720395647 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fb098108f90 con 0x7fb098100a20 2024-03-22T01:47:59.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:59 smithi089 ceph-mon[35770]: pgmap v344: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-22T01:47:59.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:47:59 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/1107710108' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-22T01:47:59.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:59 smithi072 ceph-mon[32414]: pgmap v344: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-22T01:47:59.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:47:59 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/1107710108' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-22T01:47:59.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:59 smithi098 ceph-mon[36468]: pgmap v344: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-22T01:47:59.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:47:59 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/1107710108' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-22T01:48:00.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:00.493+0000 7fb08effd640 1 -- 172.21.15.72:0/1720395647 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v99) ==== 169+0+0 (secure 0 0 0) 0x7fb0940bdad0 con 0x7fb098100a20 2024-03-22T01:48:00.493 INFO:teuthology.orchestra.run.smithi072.stderr:pool 'cephfs_data' removed 2024-03-22T01:48:00.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:00.496+0000 7fb0a0193640 1 -- 172.21.15.72:0/1720395647 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fb0840a3090 msgr2=0x7fb0840a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:48:00.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:00.496+0000 7fb0a0193640 1 --2- 172.21.15.72:0/1720395647 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fb0840a3090 0x7fb0840a5550 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb088006fd0 tx=0x7fb088008040 comp rx=0 tx=0).stop 2024-03-22T01:48:00.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:00.496+0000 7fb0a0193640 1 -- 172.21.15.72:0/1720395647 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb098100a20 msgr2=0x7fb098197410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:48:00.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:00.496+0000 7fb0a0193640 1 --2- 172.21.15.72:0/1720395647 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb098100a20 0x7fb098197410 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fb094009230 tx=0x7fb09402cc70 comp rx=0 tx=0).stop 2024-03-22T01:48:00.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:00.496+0000 7fb0a0193640 1 -- 172.21.15.72:0/1720395647 shutdown_connections 2024-03-22T01:48:00.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:00.496+0000 7fb0a0193640 1 --2- 172.21.15.72:0/1720395647 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7fb0840a3090 0x7fb0840a5550 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:48:00.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:00.496+0000 7fb0a0193640 1 --2- 172.21.15.72:0/1720395647 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0981013e0 0x7fb098197950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:48:00.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:00.496+0000 7fb0a0193640 1 --2- 172.21.15.72:0/1720395647 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb098106a30 0x7fb09819bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:48:00.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:00.496+0000 7fb0a0193640 1 --2- 172.21.15.72:0/1720395647 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb098100a20 0x7fb098197410 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:48:00.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:00.496+0000 7fb0a0193640 1 -- 172.21.15.72:0/1720395647 >> 172.21.15.72:0/1720395647 conn(0x7fb098077bb0 msgr2=0x7fb098103400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:48:00.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:00.496+0000 7fb0a0193640 1 -- 172.21.15.72:0/1720395647 shutdown_connections 2024-03-22T01:48:00.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:00.496+0000 7fb0a0193640 1 -- 172.21.15.72:0/1720395647 wait complete. 2024-03-22T01:48:00.513 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-03-22T01:48:00.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:00.622+0000 7f5fc05d7640 1 -- 172.21.15.72:0/2340119403 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5fb8100a50 msgr2=0x7f5fb8100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:48:00.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:00.622+0000 7f5fc05d7640 1 --2- 172.21.15.72:0/2340119403 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5fb8100a50 0x7f5fb8100ed0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8009930 tx=0x7f5fa802bc30 comp rx=0 tx=0).stop 2024-03-22T01:48:00.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:00.622+0000 7f5fc05d7640 1 -- 172.21.15.72:0/2340119403 shutdown_connections 2024-03-22T01:48:00.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:00.622+0000 7f5fc05d7640 1 --2- 172.21.15.72:0/2340119403 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fb8101410 0x7f5fb810da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:48:00.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:00.622+0000 7f5fc05d7640 1 --2- 172.21.15.72:0/2340119403 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5fb8100a50 0x7f5fb8100ed0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:48:00.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:00.622+0000 7f5fc05d7640 1 --2- 172.21.15.72:0/2340119403 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fb8106b70 0x7f5fb8106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:48:00.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:00.622+0000 7f5fc05d7640 1 -- 172.21.15.72:0/2340119403 >> 172.21.15.72:0/2340119403 conn(0x7f5fb8077c00 msgr2=0x7f5fb8078010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:48:00.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:00.622+0000 7f5fc05d7640 1 -- 172.21.15.72:0/2340119403 shutdown_connections 2024-03-22T01:48:00.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:00.622+0000 7f5fc05d7640 1 -- 172.21.15.72:0/2340119403 wait complete. 2024-03-22T01:48:00.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:00.623+0000 7f5fc05d7640 1 Processor -- start 2024-03-22T01:48:00.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:00.623+0000 7f5fc05d7640 1 -- start start 2024-03-22T01:48:00.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:00.623+0000 7f5fc05d7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fb8100a50 0x7f5fb8198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:48:00.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:00.623+0000 7f5fc05d7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fb8101410 0x7f5fb8198b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:48:00.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:00.623+0000 7f5fc05d7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5fb8106b70 0x7f5fb819cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:48:00.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:00.623+0000 7f5fc05d7640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb8112100 con 0x7f5fb8106b70 2024-03-22T01:48:00.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:00.623+0000 7f5fc05d7640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb8111f80 con 0x7f5fb8100a50 2024-03-22T01:48:00.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:00.623+0000 7f5fc05d7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb8112280 con 0x7f5fb8101410 2024-03-22T01:48:00.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:00.623+0000 7f5fbeb74640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5fb8106b70 0x7f5fb819cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:48:00.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:00.623+0000 7f5fbeb74640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5fb8106b70 0x7f5fb819cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36164/0 (socket says 172.21.15.72:36164) 2024-03-22T01:48:00.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:00.623+0000 7f5fbeb74640 1 -- 172.21.15.72:0/358641068 learned_addr learned my addr 172.21.15.72:0/358641068 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-22T01:48:00.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:00.623+0000 7f5fbdb72640 1 --2- 172.21.15.72:0/358641068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fb8101410 0x7f5fb8198b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:48:00.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:00.623+0000 7f5fbe373640 1 --2- 172.21.15.72:0/358641068 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fb8100a50 0x7f5fb8198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:48:00.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:00.623+0000 7f5fbeb74640 1 -- 172.21.15.72:0/358641068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fb8101410 msgr2=0x7f5fb8198b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:48:00.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:00.623+0000 7f5fbeb74640 1 --2- 172.21.15.72:0/358641068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fb8101410 0x7f5fb8198b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:48:00.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:00.623+0000 7f5fbeb74640 1 -- 172.21.15.72:0/358641068 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fb8100a50 msgr2=0x7f5fb8198620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:48:00.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:00.623+0000 7f5fbeb74640 1 --2- 172.21.15.72:0/358641068 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fb8100a50 0x7f5fb8198620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:48:00.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:00.623+0000 7f5fbeb74640 1 -- 172.21.15.72:0/358641068 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fb819d610 con 0x7f5fb8106b70 2024-03-22T01:48:00.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:00.623+0000 7f5fbe373640 1 --2- 172.21.15.72:0/358641068 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fb8100a50 0x7f5fb8198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T01:48:00.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:00.623+0000 7f5fbeb74640 1 --2- 172.21.15.72:0/358641068 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5fb8106b70 0x7f5fb819cee0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f5fb000b9e0 tx=0x7f5fb000bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:48:00.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:00.623+0000 7f5faf7fe640 1 -- 172.21.15.72:0/358641068 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fb0019070 con 0x7f5fb8106b70 2024-03-22T01:48:00.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:00.623+0000 7f5faf7fe640 1 -- 172.21.15.72:0/358641068 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5fb0014440 con 0x7f5fb8106b70 2024-03-22T01:48:00.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:00.624+0000 7f5faf7fe640 1 -- 172.21.15.72:0/358641068 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fb00075d0 con 0x7f5fb8106b70 2024-03-22T01:48:00.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:00.624+0000 7f5fc05d7640 1 -- 172.21.15.72:0/358641068 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fb819d900 con 0x7f5fb8106b70 2024-03-22T01:48:00.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:00.624+0000 7f5fc05d7640 1 -- 172.21.15.72:0/358641068 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fb81a5190 con 0x7f5fb8106b70 2024-03-22T01:48:00.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:00.624+0000 7f5fc05d7640 1 -- 172.21.15.72:0/358641068 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fb8068e60 con 0x7f5fb8106b70 2024-03-22T01:48:00.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:00.625+0000 7f5faf7fe640 1 -- 172.21.15.72:0/358641068 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 25) ==== 137985+0+0 (secure 0 0 0) 0x7f5fb001c070 con 0x7f5fb8106b70 2024-03-22T01:48:00.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:00.626+0000 7f5faf7fe640 1 --2- 172.21.15.72:0/358641068 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f5fa40a3090 0x7f5fa40a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T01:48:00.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:00.626+0000 7f5fbe373640 1 --2- 172.21.15.72:0/358641068 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f5fa40a3090 0x7f5fa40a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T01:48:00.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:00.626+0000 7f5faf7fe640 1 -- 172.21.15.72:0/358641068 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8410+0+0 (secure 0 0 0) 0x7f5fb00cf5d0 con 0x7f5fb8106b70 2024-03-22T01:48:00.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:00.627+0000 7f5fbe373640 1 --2- 172.21.15.72:0/358641068 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f5fa40a3090 0x7f5fa40a5550 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5fb4006fd0 tx=0x7f5fb4008040 comp rx=0 tx=0).ready entity=mgr.15381 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T01:48:00.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:00.630+0000 7f5faf7fe640 1 -- 172.21.15.72:0/358641068 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5fb0023020 con 0x7f5fb8106b70 2024-03-22T01:48:00.776 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:48:00 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/1107710108' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-22T01:48:00.776 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:48:00 smithi089 ceph-mon[35770]: osdmap e98: 12 total, 12 up, 12 in 2024-03-22T01:48:00.776 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:48:00 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/1720395647' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-22T01:48:00.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:00.803+0000 7f5fc05d7640 1 -- 172.21.15.72:0/358641068 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7f5fb8199970 con 0x7f5fb8106b70 2024-03-22T01:48:00.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:48:00 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/1107710108' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-22T01:48:00.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:48:00 smithi072 ceph-mon[32414]: osdmap e98: 12 total, 12 up, 12 in 2024-03-22T01:48:00.872 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:48:00 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/1720395647' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-22T01:48:00.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:48:00 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/1107710108' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-22T01:48:00.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:48:00 smithi098 ceph-mon[36468]: osdmap e98: 12 total, 12 up, 12 in 2024-03-22T01:48:00.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:48:00 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/1720395647' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-22T01:48:01.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:01.495+0000 7f5faf7fe640 1 -- 172.21.15.72:0/358641068 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v100) ==== 178+0+0 (secure 0 0 0) 0x7f5fb8199970 con 0x7f5fb8106b70 2024-03-22T01:48:01.495 INFO:teuthology.orchestra.run.smithi072.stderr:pool 'cephfs_data_ec' removed 2024-03-22T01:48:01.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:01.498+0000 7f5fc05d7640 1 -- 172.21.15.72:0/358641068 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f5fa40a3090 msgr2=0x7f5fa40a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:48:01.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:01.498+0000 7f5fc05d7640 1 --2- 172.21.15.72:0/358641068 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f5fa40a3090 0x7f5fa40a5550 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5fb4006fd0 tx=0x7f5fb4008040 comp rx=0 tx=0).stop 2024-03-22T01:48:01.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:01.498+0000 7f5fc05d7640 1 -- 172.21.15.72:0/358641068 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5fb8106b70 msgr2=0x7f5fb819cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T01:48:01.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:01.498+0000 7f5fc05d7640 1 --2- 172.21.15.72:0/358641068 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5fb8106b70 0x7f5fb819cee0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f5fb000b9e0 tx=0x7f5fb000bea0 comp rx=0 tx=0).stop 2024-03-22T01:48:01.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:01.498+0000 7f5fc05d7640 1 -- 172.21.15.72:0/358641068 shutdown_connections 2024-03-22T01:48:01.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:01.498+0000 7f5fc05d7640 1 --2- 172.21.15.72:0/358641068 >> [v2:172.21.15.72:6800/1379139045,v1:172.21.15.72:6801/1379139045] conn(0x7f5fa40a3090 0x7f5fa40a5550 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:48:01.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:01.498+0000 7f5fc05d7640 1 --2- 172.21.15.72:0/358641068 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5fb8106b70 0x7f5fb819cee0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:48:01.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:01.498+0000 7f5fc05d7640 1 --2- 172.21.15.72:0/358641068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fb8101410 0x7f5fb8198b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:48:01.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:01.498+0000 7f5fc05d7640 1 --2- 172.21.15.72:0/358641068 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fb8100a50 0x7f5fb8198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T01:48:01.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:01.498+0000 7f5fc05d7640 1 -- 172.21.15.72:0/358641068 >> 172.21.15.72:0/358641068 conn(0x7f5fb8077c00 msgr2=0x7f5fb8103300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T01:48:01.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:01.498+0000 7f5fc05d7640 1 -- 172.21.15.72:0/358641068 shutdown_connections 2024-03-22T01:48:01.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-22T01:48:01.498+0000 7f5fc05d7640 1 -- 172.21.15.72:0/358641068 wait complete. 2024-03-22T01:48:01.517 DEBUG:teuthology.orchestra.run.smithi072:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-22T01:48:01.575 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-22T01:48:01.610 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-22T01:48:01.654 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-22T01:48:01.655 DEBUG:teuthology.orchestra.run.smithi072:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-22T01:48:01.677 DEBUG:teuthology.orchestra.run.smithi089:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-22T01:48:01.695 DEBUG:teuthology.orchestra.run.smithi098:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-22T01:48:01.712 INFO:tasks.cephadm:Stopping all daemons... 2024-03-22T01:48:01.712 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-22T01:48:01.712 DEBUG:teuthology.orchestra.run.smithi072:> sudo systemctl stop ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mon.a 2024-03-22T01:48:01.750 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:48:01 smithi072 ceph-mon[32414]: pgmap v346: 97 pgs: 97 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:48:01.750 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:48:01 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/1720395647' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-22T01:48:01.751 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:48:01 smithi072 ceph-mon[32414]: osdmap e99: 12 total, 12 up, 12 in 2024-03-22T01:48:01.751 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:48:01 smithi072 ceph-mon[32414]: from='client.? 172.21.15.72:0/358641068' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-22T01:48:01.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:48:01 smithi089 ceph-mon[35770]: pgmap v346: 97 pgs: 97 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:48:01.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:48:01 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/1720395647' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-22T01:48:01.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:48:01 smithi089 ceph-mon[35770]: osdmap e99: 12 total, 12 up, 12 in 2024-03-22T01:48:01.858 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:48:01 smithi089 ceph-mon[35770]: from='client.? 172.21.15.72:0/358641068' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-22T01:48:01.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:48:01 smithi098 ceph-mon[36468]: pgmap v346: 97 pgs: 97 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T01:48:01.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:48:01 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/1720395647' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-22T01:48:01.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:48:01 smithi098 ceph-mon[36468]: osdmap e99: 12 total, 12 up, 12 in 2024-03-22T01:48:01.922 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:48:01 smithi098 ceph-mon[36468]: from='client.? 172.21.15.72:0/358641068' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-22T01:48:02.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:48:01 smithi072 systemd[1]: Stopping Ceph mon.a for 022a99bc-e7eb-11ee-95cd-87774f69a715... 2024-03-22T01:48:02.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:48:02 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mon-a[32410]: 2024-03-22T01:48:02.157+0000 7f4d57ceb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T01:48:02.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:48:02 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mon-a[32410]: 2024-03-22T01:48:02.157+0000 7f4d57ceb700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-22T01:48:02.623 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:48:02 smithi072 podman[96048]: 2024-03-22 01:48:02.269785556 +0000 UTC m=+0.237392360 container died c91280b7adc3cb78e5ecaa1c015a5fce371dddbe8c964b3c10de41107ae3c499 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mon-a, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-22T01:48:03.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:48:03 smithi072 podman[96048]: 2024-03-22 01:48:03.296802911 +0000 UTC m=+1.264409694 container cleanup c91280b7adc3cb78e5ecaa1c015a5fce371dddbe8c964b3c10de41107ae3c499 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mon-a, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240319) 2024-03-22T01:48:03.622 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:48:03 smithi072 bash[96048]: ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mon-a 2024-03-22T01:48:04.000 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 22 01:48:03 smithi072 podman[96063]: 2024-03-22 01:48:03.625637873 +0000 UTC m=+1.348136248 container remove c91280b7adc3cb78e5ecaa1c015a5fce371dddbe8c964b3c10de41107ae3c499 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mon-a, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-22T01:48:04.025 DEBUG:teuthology.orchestra.run.smithi072:> sudo pkill -f 'journalctl -f -n 0 -u ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mon.a.service' 2024-03-22T01:48:04.176 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T01:48:04.176 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-22T01:48:04.176 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-22T01:48:04.176 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mon.b 2024-03-22T01:48:04.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:48:04 smithi089 systemd[1]: Stopping Ceph mon.b for 022a99bc-e7eb-11ee-95cd-87774f69a715... 2024-03-22T01:48:05.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:48:04 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mon-b[35766]: 2024-03-22T01:48:04.718+0000 7f50bea4f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T01:48:05.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:48:04 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mon-b[35766]: 2024-03-22T01:48:04.718+0000 7f50bea4f700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-22T01:48:05.108 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:48:04 smithi089 podman[72167]: 2024-03-22 01:48:04.739864498 +0000 UTC m=+0.166277219 container died b9ecbcab627969172d76b51739bb6f24eb5e3fabd3eb23fe9481281ccef7053a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mon-b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-22T01:48:05.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:48:05 smithi089 podman[72167]: 2024-03-22 01:48:05.374838367 +0000 UTC m=+0.801251085 container cleanup b9ecbcab627969172d76b51739bb6f24eb5e3fabd3eb23fe9481281ccef7053a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mon-b, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-22T01:48:05.608 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:48:05 smithi089 bash[72167]: ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mon-b 2024-03-22T01:48:05.925 INFO:journalctl@ceph.mon.b.smithi089.stdout:Mar 22 01:48:05 smithi089 podman[72181]: 2024-03-22 01:48:05.668366377 +0000 UTC m=+0.913952109 container remove b9ecbcab627969172d76b51739bb6f24eb5e3fabd3eb23fe9481281ccef7053a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mon-b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-22T01:48:05.937 DEBUG:teuthology.orchestra.run.smithi089:> sudo pkill -f 'journalctl -f -n 0 -u ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mon.b.service' 2024-03-22T01:48:05.994 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T01:48:05.995 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-22T01:48:05.995 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-22T01:48:05.995 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mon.c 2024-03-22T01:48:06.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:48:06 smithi098 systemd[1]: Stopping Ceph mon.c for 022a99bc-e7eb-11ee-95cd-87774f69a715... 2024-03-22T01:48:07.410 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:48:07 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mon-c[36451]: 2024-03-22T01:48:07.052+0000 7f17b64c9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T01:48:07.410 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:48:07 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mon-c[36451]: 2024-03-22T01:48:07.052+0000 7f17b64c9700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-22T01:48:07.410 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:48:07 smithi098 podman[73686]: 2024-03-22 01:48:07.185499414 +0000 UTC m=+0.741869388 container died b5e529ba5a4296bf14a25142dd4f126dcdd96dd755499a28aedab1163efd5fdf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mon-c, org.label-schema.build-date=20240319, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-22T01:48:07.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:48:07 smithi098 podman[73686]: 2024-03-22 01:48:07.571758806 +0000 UTC m=+1.128128781 container cleanup b5e529ba5a4296bf14a25142dd4f126dcdd96dd755499a28aedab1163efd5fdf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mon-c, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-22T01:48:07.672 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 22 01:48:07 smithi098 bash[73686]: ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mon-c 2024-03-22T01:48:07.994 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mon.c.service' 2024-03-22T01:48:08.046 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T01:48:08.046 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-22T01:48:08.047 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-22T01:48:08.047 DEBUG:teuthology.orchestra.run.smithi072:> sudo systemctl stop ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mgr.x 2024-03-22T01:48:08.373 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:48:08 smithi072 systemd[1]: Stopping Ceph mgr.x for 022a99bc-e7eb-11ee-95cd-87774f69a715... 2024-03-22T01:48:10.372 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:48:10 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x[32643]: 2024-03-22T01:48:10.040+0000 7fa50bb34700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T01:48:10.373 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:48:10 smithi072 podman[96244]: 2024-03-22 01:48:10.081865683 +0000 UTC m=+1.818355735 container died 8322249439be5b27b6af56646022c3966777c1eb0f4f46a617dc035fcd8237fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240321.172515-c3a04c0) 2024-03-22T01:48:19.373 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:48:18 smithi072 podman[96244]: 2024-03-22 01:48:18.921941563 +0000 UTC m=+10.658431610 container cleanup 8322249439be5b27b6af56646022c3966777c1eb0f4f46a617dc035fcd8237fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-22T01:48:19.373 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:48:18 smithi072 bash[96244]: ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x 2024-03-22T01:48:25.745 DEBUG:teuthology.orchestra.run.smithi072:> sudo pkill -f 'journalctl -f -n 0 -u ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mgr.x.service' 2024-03-22T01:48:25.770 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:48:25 smithi072 podman[96258]: 2024-03-22 01:48:25.518799358 +0000 UTC m=+15.279101745 container remove 8322249439be5b27b6af56646022c3966777c1eb0f4f46a617dc035fcd8237fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-x, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-22T01:48:25.771 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:48:25 smithi072 systemd[1]: ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mgr.x.service: Deactivated successfully. 2024-03-22T01:48:25.771 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:48:25 smithi072 systemd[1]: Stopped Ceph mgr.x for 022a99bc-e7eb-11ee-95cd-87774f69a715. 2024-03-22T01:48:25.771 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Mar 22 01:48:25 smithi072 systemd[1]: ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mgr.x.service: Consumed 1min 20.300s CPU time. 2024-03-22T01:48:25.790 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T01:48:25.790 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-22T01:48:25.790 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-22T01:48:25.790 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mgr.y 2024-03-22T01:48:26.108 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:48:25 smithi089 systemd[1]: Stopping Ceph mgr.y for 022a99bc-e7eb-11ee-95cd-87774f69a715... 2024-03-22T01:48:26.562 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:48:26 smithi089 podman[72293]: 2024-03-22 01:48:26.158989983 +0000 UTC m=+0.174103295 container died 87cbf62f31f74cc9ac89d59c8eb9d251dee549f75efb912369ab06c22519f1d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-y, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0) 2024-03-22T01:48:26.858 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:48:26 smithi089 podman[72293]: 2024-03-22 01:48:26.663762373 +0000 UTC m=+0.678875673 container cleanup 87cbf62f31f74cc9ac89d59c8eb9d251dee549f75efb912369ab06c22519f1d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-y, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0) 2024-03-22T01:48:26.858 INFO:journalctl@ceph.mgr.y.smithi089.stdout:Mar 22 01:48:26 smithi089 bash[72293]: ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-y 2024-03-22T01:48:27.130 DEBUG:teuthology.orchestra.run.smithi089:> sudo pkill -f 'journalctl -f -n 0 -u ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mgr.y.service' 2024-03-22T01:48:27.168 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T01:48:27.168 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-22T01:48:27.168 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-22T01:48:27.168 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mgr.z 2024-03-22T01:48:27.498 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:48:27 smithi098 systemd[1]: Stopping Ceph mgr.z for 022a99bc-e7eb-11ee-95cd-87774f69a715... 2024-03-22T01:48:27.783 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:48:27 smithi098 podman[73823]: 2024-03-22 01:48:27.499038585 +0000 UTC m=+0.139795929 container died 63922caaa5007c1d8aae8b8b237d96785f7ac051bb1cb0e4be13f40c31bc6e58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-z, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319) 2024-03-22T01:48:28.163 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:48:27 smithi098 podman[73823]: 2024-03-22 01:48:27.888179934 +0000 UTC m=+0.528937270 container cleanup 63922caaa5007c1d8aae8b8b237d96785f7ac051bb1cb0e4be13f40c31bc6e58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-z, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.build-date=20240319, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-22T01:48:28.163 INFO:journalctl@ceph.mgr.z.smithi098.stdout:Mar 22 01:48:27 smithi098 bash[73823]: ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mgr-z 2024-03-22T01:48:28.277 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mgr.z.service' 2024-03-22T01:48:28.316 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T01:48:28.317 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-22T01:48:28.317 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-22T01:48:28.317 DEBUG:teuthology.orchestra.run.smithi072:> sudo systemctl stop ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@osd.0 2024-03-22T01:48:28.622 INFO:journalctl@ceph.osd.0.smithi072.stdout:Mar 22 01:48:28 smithi072 systemd[1]: Stopping Ceph osd.0 for 022a99bc-e7eb-11ee-95cd-87774f69a715... 2024-03-22T01:48:30.873 INFO:journalctl@ceph.osd.0.smithi072.stdout:Mar 22 01:48:30 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-0[44792]: 2024-03-22T01:48:30.411+0000 7ff0514c5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T01:48:30.873 INFO:journalctl@ceph.osd.0.smithi072.stdout:Mar 22 01:48:30 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-0[44792]: 2024-03-22T01:48:30.411+0000 7ff0514c5700 -1 osd.0 100 *** Got signal Terminated *** 2024-03-22T01:48:30.873 INFO:journalctl@ceph.osd.0.smithi072.stdout:Mar 22 01:48:30 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-0[44792]: 2024-03-22T01:48:30.411+0000 7ff0514c5700 -1 osd.0 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-22T01:48:36.372 INFO:journalctl@ceph.osd.0.smithi072.stdout:Mar 22 01:48:36 smithi072 podman[96380]: 2024-03-22 01:48:36.003708077 +0000 UTC m=+7.485703040 container died 7df82e628826319cbd667fefd74c77b8d210c7c73cf151c3f8e6781e2cbc9eb9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-22T01:48:41.622 INFO:journalctl@ceph.osd.0.smithi072.stdout:Mar 22 01:48:41 smithi072 podman[96380]: 2024-03-22 01:48:41.371546113 +0000 UTC m=+12.853541070 container cleanup 7df82e628826319cbd667fefd74c77b8d210c7c73cf151c3f8e6781e2cbc9eb9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.build-date=20240319) 2024-03-22T01:48:41.622 INFO:journalctl@ceph.osd.0.smithi072.stdout:Mar 22 01:48:41 smithi072 bash[96380]: ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-0 2024-03-22T01:48:43.333 INFO:journalctl@ceph.osd.0.smithi072.stdout:Mar 22 01:48:42 smithi072 podman[96396]: 2024-03-22 01:48:42.893770165 +0000 UTC m=+6.884642209 container remove 7df82e628826319cbd667fefd74c77b8d210c7c73cf151c3f8e6781e2cbc9eb9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, ceph=True, org.label-schema.schema-version=1.0) 2024-03-22T01:48:43.622 INFO:journalctl@ceph.osd.0.smithi072.stdout:Mar 22 01:48:43 smithi072 podman[96484]: 2024-03-22 01:48:43.284933363 +0000 UTC m=+0.018212400 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T01:48:43.623 INFO:journalctl@ceph.osd.0.smithi072.stdout:Mar 22 01:48:43 smithi072 podman[96484]: 2024-03-22 01:48:43.412799587 +0000 UTC m=+0.146078607 container create 8ba45271b09d08274dad65c2d35cc3bc55f78e436d33648aa98a8063cd8c7739 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD) 2024-03-22T01:48:44.122 INFO:journalctl@ceph.osd.0.smithi072.stdout:Mar 22 01:48:43 smithi072 podman[96484]: 2024-03-22 01:48:43.667499627 +0000 UTC m=+0.400778660 container init 8ba45271b09d08274dad65c2d35cc3bc55f78e436d33648aa98a8063cd8c7739 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-22T01:48:44.122 INFO:journalctl@ceph.osd.0.smithi072.stdout:Mar 22 01:48:43 smithi072 podman[96484]: 2024-03-22 01:48:43.67075836 +0000 UTC m=+0.404037392 container start 8ba45271b09d08274dad65c2d35cc3bc55f78e436d33648aa98a8063cd8c7739 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-0-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-22T01:48:44.122 INFO:journalctl@ceph.osd.0.smithi072.stdout:Mar 22 01:48:43 smithi072 podman[96484]: 2024-03-22 01:48:43.844950116 +0000 UTC m=+0.578229148 container attach 8ba45271b09d08274dad65c2d35cc3bc55f78e436d33648aa98a8063cd8c7739 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.build-date=20240319) 2024-03-22T01:48:46.074 INFO:journalctl@ceph.osd.0.smithi072.stdout:Mar 22 01:48:45 smithi072 podman[96484]: 2024-03-22 01:48:45.773339578 +0000 UTC m=+2.506618591 container died 8ba45271b09d08274dad65c2d35cc3bc55f78e436d33648aa98a8063cd8c7739 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-0-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True) 2024-03-22T01:48:46.391 DEBUG:teuthology.orchestra.run.smithi072:> sudo pkill -f 'journalctl -f -n 0 -u ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@osd.0.service' 2024-03-22T01:48:46.439 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T01:48:46.440 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-22T01:48:46.440 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-22T01:48:46.440 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@osd.1 2024-03-22T01:48:46.749 INFO:journalctl@ceph.osd.1.smithi089.stdout:Mar 22 01:48:46 smithi089 systemd[1]: Stopping Ceph osd.1 for 022a99bc-e7eb-11ee-95cd-87774f69a715... 2024-03-22T01:48:47.108 INFO:journalctl@ceph.osd.1.smithi089.stdout:Mar 22 01:48:46 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-1[40372]: 2024-03-22T01:48:46.750+0000 7fc2b75ed700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T01:48:47.108 INFO:journalctl@ceph.osd.1.smithi089.stdout:Mar 22 01:48:46 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-1[40372]: 2024-03-22T01:48:46.750+0000 7fc2b75ed700 -1 osd.1 100 *** Got signal Terminated *** 2024-03-22T01:48:47.108 INFO:journalctl@ceph.osd.1.smithi089.stdout:Mar 22 01:48:46 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-1[40372]: 2024-03-22T01:48:46.750+0000 7fc2b75ed700 -1 osd.1 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-22T01:48:52.518 INFO:journalctl@ceph.osd.1.smithi089.stdout:Mar 22 01:48:52 smithi089 podman[72415]: 2024-03-22 01:48:52.202504778 +0000 UTC m=+5.574880905 container died 68400e4cebe908a181e9ca3bd97a4726b7f57f96a6c955a3bd02d9970ca43ddf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-1, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-22T01:48:52.858 INFO:journalctl@ceph.osd.1.smithi089.stdout:Mar 22 01:48:52 smithi089 podman[72415]: 2024-03-22 01:48:52.634557168 +0000 UTC m=+6.006933294 container cleanup 68400e4cebe908a181e9ca3bd97a4726b7f57f96a6c955a3bd02d9970ca43ddf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-1, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-22T01:48:52.858 INFO:journalctl@ceph.osd.1.smithi089.stdout:Mar 22 01:48:52 smithi089 bash[72415]: ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-1 2024-03-22T01:48:53.281 INFO:journalctl@ceph.osd.1.smithi089.stdout:Mar 22 01:48:52 smithi089 podman[72445]: 2024-03-22 01:48:52.877676075 +0000 UTC m=+0.675951992 container remove 68400e4cebe908a181e9ca3bd97a4726b7f57f96a6c955a3bd02d9970ca43ddf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-1, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-22T01:48:53.589 INFO:journalctl@ceph.osd.1.smithi089.stdout:Mar 22 01:48:53 smithi089 podman[72538]: 2024-03-22 01:48:53.185506759 +0000 UTC m=+0.017633726 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T01:48:53.589 INFO:journalctl@ceph.osd.1.smithi089.stdout:Mar 22 01:48:53 smithi089 podman[72538]: 2024-03-22 01:48:53.320414758 +0000 UTC m=+0.152541721 container create 8d9819b5933d9c1b04b63b9a677ef11822cc8f80255daf17cf8f2410bfca0bf0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-22T01:48:53.858 INFO:journalctl@ceph.osd.1.smithi089.stdout:Mar 22 01:48:53 smithi089 podman[72538]: 2024-03-22 01:48:53.590223347 +0000 UTC m=+0.422350319 container init 8d9819b5933d9c1b04b63b9a677ef11822cc8f80255daf17cf8f2410bfca0bf0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-1-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-22T01:48:53.858 INFO:journalctl@ceph.osd.1.smithi089.stdout:Mar 22 01:48:53 smithi089 podman[72538]: 2024-03-22 01:48:53.593639398 +0000 UTC m=+0.425766372 container start 8d9819b5933d9c1b04b63b9a677ef11822cc8f80255daf17cf8f2410bfca0bf0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-1-deactivate, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-22T01:48:53.858 INFO:journalctl@ceph.osd.1.smithi089.stdout:Mar 22 01:48:53 smithi089 podman[72538]: 2024-03-22 01:48:53.698315988 +0000 UTC m=+0.530442960 container attach 8d9819b5933d9c1b04b63b9a677ef11822cc8f80255daf17cf8f2410bfca0bf0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-1-deactivate, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-22T01:48:54.275 INFO:journalctl@ceph.osd.1.smithi089.stdout:Mar 22 01:48:53 smithi089 podman[72538]: 2024-03-22 01:48:53.965455685 +0000 UTC m=+0.797582650 container died 8d9819b5933d9c1b04b63b9a677ef11822cc8f80255daf17cf8f2410bfca0bf0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-1-deactivate, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-22T01:48:54.601 DEBUG:teuthology.orchestra.run.smithi089:> sudo pkill -f 'journalctl -f -n 0 -u ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@osd.1.service' 2024-03-22T01:48:54.645 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T01:48:54.645 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-22T01:48:54.645 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-22T01:48:54.645 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@osd.2 2024-03-22T01:48:54.958 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 22 01:48:54 smithi098 systemd[1]: Stopping Ceph osd.2 for 022a99bc-e7eb-11ee-95cd-87774f69a715... 2024-03-22T01:48:55.222 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 22 01:48:54 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-2[41603]: 2024-03-22T01:48:54.958+0000 7f7a60c08700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T01:48:55.223 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 22 01:48:54 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-2[41603]: 2024-03-22T01:48:54.958+0000 7f7a60c08700 -1 osd.2 100 *** Got signal Terminated *** 2024-03-22T01:48:55.223 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 22 01:48:54 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-2[41603]: 2024-03-22T01:48:54.958+0000 7f7a60c08700 -1 osd.2 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-22T01:49:00.792 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 22 01:49:00 smithi098 podman[73945]: 2024-03-22 01:49:00.49326197 +0000 UTC m=+5.649326974 container died b1acd598834721e40fda4bbd8215641f58e99edf4e218e624ef7fc87801fd06e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-22T01:49:01.172 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 22 01:49:00 smithi098 podman[73945]: 2024-03-22 01:49:00.892820503 +0000 UTC m=+6.048885520 container cleanup b1acd598834721e40fda4bbd8215641f58e99edf4e218e624ef7fc87801fd06e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-22T01:49:01.172 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 22 01:49:00 smithi098 bash[73945]: ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-2 2024-03-22T01:49:01.541 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 22 01:49:01 smithi098 podman[73976]: 2024-03-22 01:49:01.190578853 +0000 UTC m=+0.697778548 container remove b1acd598834721e40fda4bbd8215641f58e99edf4e218e624ef7fc87801fd06e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-2, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS) 2024-03-22T01:49:01.797 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 22 01:49:01 smithi098 podman[74062]: 2024-03-22 01:49:01.44571858 +0000 UTC m=+0.017651738 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T01:49:01.798 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 22 01:49:01 smithi098 podman[74062]: 2024-03-22 01:49:01.56264506 +0000 UTC m=+0.134578238 container create 2142ee4fa8e4cfc8e4399673390c59b251808c8a863fda3bb309b385dba139b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-22T01:49:01.798 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 22 01:49:01 smithi098 podman[74062]: 2024-03-22 01:49:01.798703097 +0000 UTC m=+0.370636268 container init 2142ee4fa8e4cfc8e4399673390c59b251808c8a863fda3bb309b385dba139b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-2-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-22T01:49:02.172 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 22 01:49:01 smithi098 podman[74062]: 2024-03-22 01:49:01.80236684 +0000 UTC m=+0.374300012 container start 2142ee4fa8e4cfc8e4399673390c59b251808c8a863fda3bb309b385dba139b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-22T01:49:02.172 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 22 01:49:01 smithi098 podman[74062]: 2024-03-22 01:49:01.898939195 +0000 UTC m=+0.470872369 container attach 2142ee4fa8e4cfc8e4399673390c59b251808c8a863fda3bb309b385dba139b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-22T01:49:02.467 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 22 01:49:02 smithi098 podman[74062]: 2024-03-22 01:49:02.199697244 +0000 UTC m=+0.771630407 container died 2142ee4fa8e4cfc8e4399673390c59b251808c8a863fda3bb309b385dba139b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240319, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-22T01:49:02.793 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@osd.2.service' 2024-03-22T01:49:02.834 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T01:49:02.834 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-22T01:49:02.835 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-22T01:49:02.835 DEBUG:teuthology.orchestra.run.smithi072:> sudo systemctl stop ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@osd.3 2024-03-22T01:49:03.122 INFO:journalctl@ceph.osd.3.smithi072.stdout:Mar 22 01:49:02 smithi072 systemd[1]: Stopping Ceph osd.3 for 022a99bc-e7eb-11ee-95cd-87774f69a715... 2024-03-22T01:49:03.423 INFO:journalctl@ceph.osd.3.smithi072.stdout:Mar 22 01:49:03 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-3[51749]: 2024-03-22T01:49:03.159+0000 7f461ccdb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T01:49:03.424 INFO:journalctl@ceph.osd.3.smithi072.stdout:Mar 22 01:49:03 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-3[51749]: 2024-03-22T01:49:03.159+0000 7f461ccdb700 -1 osd.3 100 *** Got signal Terminated *** 2024-03-22T01:49:03.424 INFO:journalctl@ceph.osd.3.smithi072.stdout:Mar 22 01:49:03 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-3[51749]: 2024-03-22T01:49:03.159+0000 7f461ccdb700 -1 osd.3 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-22T01:49:09.006 INFO:journalctl@ceph.osd.3.smithi072.stdout:Mar 22 01:49:08 smithi072 podman[96595]: 2024-03-22 01:49:08.727752044 +0000 UTC m=+5.690202412 container died 0b373b9fd134d57b0255c5cbc7b3863ffcabcceddbad1af8c0b0f8e3815967da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-3, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-22T01:49:09.372 INFO:journalctl@ceph.osd.3.smithi072.stdout:Mar 22 01:49:09 smithi072 podman[96595]: 2024-03-22 01:49:09.098548259 +0000 UTC m=+6.060998637 container cleanup 0b373b9fd134d57b0255c5cbc7b3863ffcabcceddbad1af8c0b0f8e3815967da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-3, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True) 2024-03-22T01:49:09.372 INFO:journalctl@ceph.osd.3.smithi072.stdout:Mar 22 01:49:09 smithi072 bash[96595]: ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-3 2024-03-22T01:49:09.759 INFO:journalctl@ceph.osd.3.smithi072.stdout:Mar 22 01:49:09 smithi072 podman[96624]: 2024-03-22 01:49:09.435276675 +0000 UTC m=+0.695788772 container remove 0b373b9fd134d57b0255c5cbc7b3863ffcabcceddbad1af8c0b0f8e3815967da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-3, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-22T01:49:10.033 INFO:journalctl@ceph.osd.3.smithi072.stdout:Mar 22 01:49:09 smithi072 podman[96717]: 2024-03-22 01:49:09.663904299 +0000 UTC m=+0.017285676 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T01:49:10.033 INFO:journalctl@ceph.osd.3.smithi072.stdout:Mar 22 01:49:09 smithi072 podman[96717]: 2024-03-22 01:49:09.772130822 +0000 UTC m=+0.125512189 container create 19ffd6f2f6f9da32d400b8bc589634e1ae3eafa9a77113b90c7f2d82b0c69a47 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-3-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-22T01:49:10.331 INFO:journalctl@ceph.osd.3.smithi072.stdout:Mar 22 01:49:10 smithi072 podman[96717]: 2024-03-22 01:49:10.035024151 +0000 UTC m=+0.388405519 container init 19ffd6f2f6f9da32d400b8bc589634e1ae3eafa9a77113b90c7f2d82b0c69a47 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-3-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-22T01:49:10.332 INFO:journalctl@ceph.osd.3.smithi072.stdout:Mar 22 01:49:10 smithi072 podman[96717]: 2024-03-22 01:49:10.03816929 +0000 UTC m=+0.391550655 container start 19ffd6f2f6f9da32d400b8bc589634e1ae3eafa9a77113b90c7f2d82b0c69a47 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-3-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-22T01:49:10.332 INFO:journalctl@ceph.osd.3.smithi072.stdout:Mar 22 01:49:10 smithi072 podman[96717]: 2024-03-22 01:49:10.144076526 +0000 UTC m=+0.497457919 container attach 19ffd6f2f6f9da32d400b8bc589634e1ae3eafa9a77113b90c7f2d82b0c69a47 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-3-deactivate, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-22T01:49:10.622 INFO:journalctl@ceph.osd.3.smithi072.stdout:Mar 22 01:49:10 smithi072 podman[96717]: 2024-03-22 01:49:10.333115315 +0000 UTC m=+0.686496684 container died 19ffd6f2f6f9da32d400b8bc589634e1ae3eafa9a77113b90c7f2d82b0c69a47 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-3-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-22T01:49:10.938 DEBUG:teuthology.orchestra.run.smithi072:> sudo pkill -f 'journalctl -f -n 0 -u ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@osd.3.service' 2024-03-22T01:49:10.981 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T01:49:10.981 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-22T01:49:10.981 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-22T01:49:10.981 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@osd.4 2024-03-22T01:49:11.303 INFO:journalctl@ceph.osd.4.smithi089.stdout:Mar 22 01:49:11 smithi089 systemd[1]: Stopping Ceph osd.4 for 022a99bc-e7eb-11ee-95cd-87774f69a715... 2024-03-22T01:49:11.608 INFO:journalctl@ceph.osd.4.smithi089.stdout:Mar 22 01:49:11 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-4[47352]: 2024-03-22T01:49:11.303+0000 7fda49868700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T01:49:11.608 INFO:journalctl@ceph.osd.4.smithi089.stdout:Mar 22 01:49:11 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-4[47352]: 2024-03-22T01:49:11.303+0000 7fda49868700 -1 osd.4 100 *** Got signal Terminated *** 2024-03-22T01:49:11.608 INFO:journalctl@ceph.osd.4.smithi089.stdout:Mar 22 01:49:11 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-4[47352]: 2024-03-22T01:49:11.303+0000 7fda49868700 -1 osd.4 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-22T01:49:17.096 INFO:journalctl@ceph.osd.4.smithi089.stdout:Mar 22 01:49:16 smithi089 podman[72645]: 2024-03-22 01:49:16.829704667 +0000 UTC m=+5.650393521 container died ad5594f41a0d8d844c7f93a9c929137de196ef864d0defd54365398e8a0c35b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-4, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240319) 2024-03-22T01:49:17.358 INFO:journalctl@ceph.osd.4.smithi089.stdout:Mar 22 01:49:17 smithi089 podman[72645]: 2024-03-22 01:49:17.204813527 +0000 UTC m=+6.025502380 container cleanup ad5594f41a0d8d844c7f93a9c929137de196ef864d0defd54365398e8a0c35b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-4, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-22T01:49:17.358 INFO:journalctl@ceph.osd.4.smithi089.stdout:Mar 22 01:49:17 smithi089 bash[72645]: ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-4 2024-03-22T01:49:17.849 INFO:journalctl@ceph.osd.4.smithi089.stdout:Mar 22 01:49:17 smithi089 podman[72661]: 2024-03-22 01:49:17.506427922 +0000 UTC m=+0.674895323 container remove ad5594f41a0d8d844c7f93a9c929137de196ef864d0defd54365398e8a0c35b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-4, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.vendor=CentOS) 2024-03-22T01:49:18.108 INFO:journalctl@ceph.osd.4.smithi089.stdout:Mar 22 01:49:17 smithi089 podman[72760]: 2024-03-22 01:49:17.753392964 +0000 UTC m=+0.017763691 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T01:49:18.108 INFO:journalctl@ceph.osd.4.smithi089.stdout:Mar 22 01:49:17 smithi089 podman[72760]: 2024-03-22 01:49:17.86549219 +0000 UTC m=+0.129862899 container create cf8c21bfe7b7185468d5c1bcf4b12a2b3b6358821caec847fe3ec5407d59dfef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0) 2024-03-22T01:49:18.108 INFO:journalctl@ceph.osd.4.smithi089.stdout:Mar 22 01:49:18 smithi089 podman[72760]: 2024-03-22 01:49:18.093284558 +0000 UTC m=+0.357655282 container init cf8c21bfe7b7185468d5c1bcf4b12a2b3b6358821caec847fe3ec5407d59dfef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-4-deactivate, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-22T01:49:18.108 INFO:journalctl@ceph.osd.4.smithi089.stdout:Mar 22 01:49:18 smithi089 podman[72760]: 2024-03-22 01:49:18.096809222 +0000 UTC m=+0.361179953 container start cf8c21bfe7b7185468d5c1bcf4b12a2b3b6358821caec847fe3ec5407d59dfef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-4-deactivate, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-22T01:49:18.608 INFO:journalctl@ceph.osd.4.smithi089.stdout:Mar 22 01:49:18 smithi089 podman[72760]: 2024-03-22 01:49:18.193626923 +0000 UTC m=+0.457997658 container attach cf8c21bfe7b7185468d5c1bcf4b12a2b3b6358821caec847fe3ec5407d59dfef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-22T01:49:18.608 INFO:journalctl@ceph.osd.4.smithi089.stdout:Mar 22 01:49:18 smithi089 podman[72760]: 2024-03-22 01:49:18.402952681 +0000 UTC m=+0.667323398 container died cf8c21bfe7b7185468d5c1bcf4b12a2b3b6358821caec847fe3ec5407d59dfef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-4-deactivate, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-22T01:49:19.041 DEBUG:teuthology.orchestra.run.smithi089:> sudo pkill -f 'journalctl -f -n 0 -u ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@osd.4.service' 2024-03-22T01:49:19.085 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T01:49:19.086 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-22T01:49:19.086 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-22T01:49:19.086 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@osd.5 2024-03-22T01:49:19.382 INFO:journalctl@ceph.osd.5.smithi098.stdout:Mar 22 01:49:19 smithi098 systemd[1]: Stopping Ceph osd.5 for 022a99bc-e7eb-11ee-95cd-87774f69a715... 2024-03-22T01:49:19.671 INFO:journalctl@ceph.osd.5.smithi098.stdout:Mar 22 01:49:19 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-5[48584]: 2024-03-22T01:49:19.382+0000 7f9331893700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T01:49:19.672 INFO:journalctl@ceph.osd.5.smithi098.stdout:Mar 22 01:49:19 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-5[48584]: 2024-03-22T01:49:19.382+0000 7f9331893700 -1 osd.5 100 *** Got signal Terminated *** 2024-03-22T01:49:19.672 INFO:journalctl@ceph.osd.5.smithi098.stdout:Mar 22 01:49:19 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-5[48584]: 2024-03-22T01:49:19.382+0000 7f9331893700 -1 osd.5 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-22T01:49:25.275 INFO:journalctl@ceph.osd.5.smithi098.stdout:Mar 22 01:49:24 smithi098 podman[74167]: 2024-03-22 01:49:24.961495368 +0000 UTC m=+5.685193756 container died d00090d52f75580ba148f030c718d50811cddf90aceeed38e18bbe0a533d0f73 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True) 2024-03-22T01:49:25.626 INFO:journalctl@ceph.osd.5.smithi098.stdout:Mar 22 01:49:25 smithi098 podman[74167]: 2024-03-22 01:49:25.400288121 +0000 UTC m=+6.123986511 container cleanup d00090d52f75580ba148f030c718d50811cddf90aceeed38e18bbe0a533d0f73 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-5, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, CEPH_POINT_RELEASE=) 2024-03-22T01:49:25.626 INFO:journalctl@ceph.osd.5.smithi098.stdout:Mar 22 01:49:25 smithi098 bash[74167]: ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-5 2024-03-22T01:49:25.922 INFO:journalctl@ceph.osd.5.smithi098.stdout:Mar 22 01:49:25 smithi098 podman[74185]: 2024-03-22 01:49:25.626817952 +0000 UTC m=+0.661287515 container remove d00090d52f75580ba148f030c718d50811cddf90aceeed38e18bbe0a533d0f73 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True) 2024-03-22T01:49:26.388 INFO:journalctl@ceph.osd.5.smithi098.stdout:Mar 22 01:49:26 smithi098 podman[74285]: 2024-03-22 01:49:25.994658106 +0000 UTC m=+0.024108097 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T01:49:26.388 INFO:journalctl@ceph.osd.5.smithi098.stdout:Mar 22 01:49:26 smithi098 podman[74285]: 2024-03-22 01:49:26.127981084 +0000 UTC m=+0.157431056 container create 3e4cdfb2d69b2f93e622991775a79625f4c95accd43ccf9ba4fd45107a606312 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-5-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS) 2024-03-22T01:49:26.388 INFO:journalctl@ceph.osd.5.smithi098.stdout:Mar 22 01:49:26 smithi098 podman[74285]: 2024-03-22 01:49:26.389155471 +0000 UTC m=+0.418605454 container init 3e4cdfb2d69b2f93e622991775a79625f4c95accd43ccf9ba4fd45107a606312 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-5-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.build-date=20240319, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-22T01:49:26.672 INFO:journalctl@ceph.osd.5.smithi098.stdout:Mar 22 01:49:26 smithi098 podman[74285]: 2024-03-22 01:49:26.392748587 +0000 UTC m=+0.422198553 container start 3e4cdfb2d69b2f93e622991775a79625f4c95accd43ccf9ba4fd45107a606312 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-5-deactivate, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-22T01:49:26.672 INFO:journalctl@ceph.osd.5.smithi098.stdout:Mar 22 01:49:26 smithi098 podman[74285]: 2024-03-22 01:49:26.48908598 +0000 UTC m=+0.518535956 container attach 3e4cdfb2d69b2f93e622991775a79625f4c95accd43ccf9ba4fd45107a606312 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-22T01:49:27.040 INFO:journalctl@ceph.osd.5.smithi098.stdout:Mar 22 01:49:26 smithi098 podman[74285]: 2024-03-22 01:49:26.682640216 +0000 UTC m=+0.712090189 container died 3e4cdfb2d69b2f93e622991775a79625f4c95accd43ccf9ba4fd45107a606312 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-5-deactivate, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-22T01:49:27.377 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@osd.5.service' 2024-03-22T01:49:27.420 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T01:49:27.420 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-22T01:49:27.421 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-22T01:49:27.421 DEBUG:teuthology.orchestra.run.smithi072:> sudo systemctl stop ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@osd.6 2024-03-22T01:49:27.724 INFO:journalctl@ceph.osd.6.smithi072.stdout:Mar 22 01:49:27 smithi072 systemd[1]: Stopping Ceph osd.6 for 022a99bc-e7eb-11ee-95cd-87774f69a715... 2024-03-22T01:49:28.123 INFO:journalctl@ceph.osd.6.smithi072.stdout:Mar 22 01:49:27 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-6[58711]: 2024-03-22T01:49:27.725+0000 7fa4235ba700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T01:49:28.124 INFO:journalctl@ceph.osd.6.smithi072.stdout:Mar 22 01:49:27 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-6[58711]: 2024-03-22T01:49:27.725+0000 7fa4235ba700 -1 osd.6 100 *** Got signal Terminated *** 2024-03-22T01:49:28.124 INFO:journalctl@ceph.osd.6.smithi072.stdout:Mar 22 01:49:27 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-6[58711]: 2024-03-22T01:49:27.725+0000 7fa4235ba700 -1 osd.6 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-22T01:49:33.489 INFO:journalctl@ceph.osd.6.smithi072.stdout:Mar 22 01:49:33 smithi072 podman[96823]: 2024-03-22 01:49:33.211334695 +0000 UTC m=+5.596199766 container died 3b7c37f0ddfba07951e82cc08b60f7471bee98fefb02a57f52289d127e308950 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240319) 2024-03-22T01:49:33.872 INFO:journalctl@ceph.osd.6.smithi072.stdout:Mar 22 01:49:33 smithi072 podman[96823]: 2024-03-22 01:49:33.569717971 +0000 UTC m=+5.954583055 container cleanup 3b7c37f0ddfba07951e82cc08b60f7471bee98fefb02a57f52289d127e308950 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-6, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2) 2024-03-22T01:49:33.872 INFO:journalctl@ceph.osd.6.smithi072.stdout:Mar 22 01:49:33 smithi072 bash[96823]: ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-6 2024-03-22T01:49:34.186 INFO:journalctl@ceph.osd.6.smithi072.stdout:Mar 22 01:49:33 smithi072 podman[96840]: 2024-03-22 01:49:33.898986236 +0000 UTC m=+0.680617558 container remove 3b7c37f0ddfba07951e82cc08b60f7471bee98fefb02a57f52289d127e308950 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-6, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-22T01:49:34.438 INFO:journalctl@ceph.osd.6.smithi072.stdout:Mar 22 01:49:34 smithi072 podman[96930]: 2024-03-22 01:49:34.090306761 +0000 UTC m=+0.018362221 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T01:49:34.439 INFO:journalctl@ceph.osd.6.smithi072.stdout:Mar 22 01:49:34 smithi072 podman[96930]: 2024-03-22 01:49:34.209352023 +0000 UTC m=+0.137407469 container create 4652c93533f1e8f6dd0f986cbe3758ba9570b3bfbd44996a108f3b315ee23bd2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-6-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-22T01:49:34.439 INFO:journalctl@ceph.osd.6.smithi072.stdout:Mar 22 01:49:34 smithi072 podman[96930]: 2024-03-22 01:49:34.439800224 +0000 UTC m=+0.367855688 container init 4652c93533f1e8f6dd0f986cbe3758ba9570b3bfbd44996a108f3b315ee23bd2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-22T01:49:34.734 INFO:journalctl@ceph.osd.6.smithi072.stdout:Mar 22 01:49:34 smithi072 podman[96930]: 2024-03-22 01:49:34.443447243 +0000 UTC m=+0.371502693 container start 4652c93533f1e8f6dd0f986cbe3758ba9570b3bfbd44996a108f3b315ee23bd2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-6-deactivate, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-22T01:49:34.734 INFO:journalctl@ceph.osd.6.smithi072.stdout:Mar 22 01:49:34 smithi072 podman[96930]: 2024-03-22 01:49:34.549842785 +0000 UTC m=+0.477898247 container attach 4652c93533f1e8f6dd0f986cbe3758ba9570b3bfbd44996a108f3b315ee23bd2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-6-deactivate, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-22T01:49:35.036 INFO:journalctl@ceph.osd.6.smithi072.stdout:Mar 22 01:49:34 smithi072 podman[96930]: 2024-03-22 01:49:34.732914464 +0000 UTC m=+0.660969918 container died 4652c93533f1e8f6dd0f986cbe3758ba9570b3bfbd44996a108f3b315ee23bd2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-6-deactivate, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-22T01:49:35.337 DEBUG:teuthology.orchestra.run.smithi072:> sudo pkill -f 'journalctl -f -n 0 -u ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@osd.6.service' 2024-03-22T01:49:35.381 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T01:49:35.381 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-22T01:49:35.381 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-22T01:49:35.381 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@osd.7 2024-03-22T01:49:35.749 INFO:journalctl@ceph.osd.7.smithi089.stdout:Mar 22 01:49:35 smithi089 systemd[1]: Stopping Ceph osd.7 for 022a99bc-e7eb-11ee-95cd-87774f69a715... 2024-03-22T01:49:36.108 INFO:journalctl@ceph.osd.7.smithi089.stdout:Mar 22 01:49:35 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-7[54352]: 2024-03-22T01:49:35.749+0000 7f09d7682700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T01:49:36.108 INFO:journalctl@ceph.osd.7.smithi089.stdout:Mar 22 01:49:35 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-7[54352]: 2024-03-22T01:49:35.749+0000 7f09d7682700 -1 osd.7 100 *** Got signal Terminated *** 2024-03-22T01:49:36.108 INFO:journalctl@ceph.osd.7.smithi089.stdout:Mar 22 01:49:35 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-7[54352]: 2024-03-22T01:49:35.749+0000 7f09d7682700 -1 osd.7 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-22T01:49:36.922 INFO:journalctl@ceph.osd.8.smithi098.stdout:Mar 22 01:49:36 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-8[55604]: 2024-03-22T01:49:36.489+0000 7f987d6aa700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:34.540633+0000 front 2024-03-22T01:48:34.540612+0000 (oldest deadline 2024-03-22T01:49:36.240597+0000) 2024-03-22T01:49:37.922 INFO:journalctl@ceph.osd.8.smithi098.stdout:Mar 22 01:49:37 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-8[55604]: 2024-03-22T01:49:37.520+0000 7f987d6aa700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:34.540633+0000 front 2024-03-22T01:48:34.540612+0000 (oldest deadline 2024-03-22T01:49:36.240597+0000) 2024-03-22T01:49:38.858 INFO:journalctl@ceph.osd.10.smithi089.stdout:Mar 22 01:49:38 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-10[61378]: 2024-03-22T01:49:38.502+0000 7f8b187bf700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:32.441146+0000 front 2024-03-22T01:48:32.441205+0000 (oldest deadline 2024-03-22T01:49:37.741167+0000) 2024-03-22T01:49:38.922 INFO:journalctl@ceph.osd.8.smithi098.stdout:Mar 22 01:49:38 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-8[55604]: 2024-03-22T01:49:38.493+0000 7f987d6aa700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:34.540633+0000 front 2024-03-22T01:48:34.540612+0000 (oldest deadline 2024-03-22T01:49:36.240597+0000) 2024-03-22T01:49:39.858 INFO:journalctl@ceph.osd.10.smithi089.stdout:Mar 22 01:49:39 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-10[61378]: 2024-03-22T01:49:39.520+0000 7f8b187bf700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:32.441146+0000 front 2024-03-22T01:48:32.441205+0000 (oldest deadline 2024-03-22T01:49:37.741167+0000) 2024-03-22T01:49:39.922 INFO:journalctl@ceph.osd.8.smithi098.stdout:Mar 22 01:49:39 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-8[55604]: 2024-03-22T01:49:39.454+0000 7f987d6aa700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:34.540633+0000 front 2024-03-22T01:48:34.540612+0000 (oldest deadline 2024-03-22T01:49:36.240597+0000) 2024-03-22T01:49:40.421 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:49:40 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:49:40.019+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:33.677632+0000 front 2024-03-22T01:48:33.677592+0000 (oldest deadline 2024-03-22T01:49:39.577549+0000) 2024-03-22T01:49:40.858 INFO:journalctl@ceph.osd.10.smithi089.stdout:Mar 22 01:49:40 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-10[61378]: 2024-03-22T01:49:40.488+0000 7f8b187bf700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:32.441146+0000 front 2024-03-22T01:48:32.441205+0000 (oldest deadline 2024-03-22T01:49:37.741167+0000) 2024-03-22T01:49:40.922 INFO:journalctl@ceph.osd.8.smithi098.stdout:Mar 22 01:49:40 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-8[55604]: 2024-03-22T01:49:40.430+0000 7f987d6aa700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:34.540633+0000 front 2024-03-22T01:48:34.540612+0000 (oldest deadline 2024-03-22T01:49:36.240597+0000) 2024-03-22T01:49:41.373 INFO:journalctl@ceph.osd.9.smithi072.stdout:Mar 22 01:49:41 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-9[65321]: 2024-03-22T01:49:41.115+0000 7fcc0a003700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:33.784567+0000 front 2024-03-22T01:48:33.784520+0000 (oldest deadline 2024-03-22T01:49:40.185335+0000) 2024-03-22T01:49:41.383 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:49:40 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:49:40.970+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:33.677632+0000 front 2024-03-22T01:48:33.677592+0000 (oldest deadline 2024-03-22T01:49:39.577549+0000) 2024-03-22T01:49:41.672 INFO:journalctl@ceph.osd.8.smithi098.stdout:Mar 22 01:49:41 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-8[55604]: 2024-03-22T01:49:41.384+0000 7f987d6aa700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:34.540633+0000 front 2024-03-22T01:48:34.540612+0000 (oldest deadline 2024-03-22T01:49:36.240597+0000) 2024-03-22T01:49:41.718 INFO:journalctl@ceph.osd.10.smithi089.stdout:Mar 22 01:49:41 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-10[61378]: 2024-03-22T01:49:41.511+0000 7f8b187bf700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:32.441146+0000 front 2024-03-22T01:48:32.441205+0000 (oldest deadline 2024-03-22T01:49:37.741167+0000) 2024-03-22T01:49:41.718 INFO:journalctl@ceph.osd.7.smithi089.stdout:Mar 22 01:49:41 smithi089 podman[72869]: 2024-03-22 01:49:41.407834169 +0000 UTC m=+5.833973100 container died 385ef9dff0824a989d1c8880399b38f336736eb6122a822e88824f2612f480d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-7, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-22T01:49:42.077 INFO:journalctl@ceph.osd.7.smithi089.stdout:Mar 22 01:49:41 smithi089 podman[72869]: 2024-03-22 01:49:41.818925803 +0000 UTC m=+6.245064727 container cleanup 385ef9dff0824a989d1c8880399b38f336736eb6122a822e88824f2612f480d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-7, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.vendor=CentOS) 2024-03-22T01:49:42.077 INFO:journalctl@ceph.osd.7.smithi089.stdout:Mar 22 01:49:41 smithi089 bash[72869]: ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-7 2024-03-22T01:49:42.358 INFO:journalctl@ceph.osd.7.smithi089.stdout:Mar 22 01:49:42 smithi089 podman[72886]: 2024-03-22 01:49:42.078349731 +0000 UTC m=+0.667666359 container remove 385ef9dff0824a989d1c8880399b38f336736eb6122a822e88824f2612f480d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-7, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-22T01:49:42.372 INFO:journalctl@ceph.osd.9.smithi072.stdout:Mar 22 01:49:42 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-9[65321]: 2024-03-22T01:49:42.068+0000 7fcc0a003700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:33.784567+0000 front 2024-03-22T01:48:33.784520+0000 (oldest deadline 2024-03-22T01:49:40.185335+0000) 2024-03-22T01:49:42.389 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:49:41 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:49:41.940+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:33.677632+0000 front 2024-03-22T01:48:33.677592+0000 (oldest deadline 2024-03-22T01:49:39.577549+0000) 2024-03-22T01:49:42.672 INFO:journalctl@ceph.osd.8.smithi098.stdout:Mar 22 01:49:42 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-8[55604]: 2024-03-22T01:49:42.389+0000 7f987d6aa700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:34.540633+0000 front 2024-03-22T01:48:34.540612+0000 (oldest deadline 2024-03-22T01:49:36.240597+0000) 2024-03-22T01:49:42.763 INFO:journalctl@ceph.osd.10.smithi089.stdout:Mar 22 01:49:42 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-10[61378]: 2024-03-22T01:49:42.539+0000 7f8b187bf700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:32.441146+0000 front 2024-03-22T01:48:32.441205+0000 (oldest deadline 2024-03-22T01:49:37.741167+0000) 2024-03-22T01:49:42.763 INFO:journalctl@ceph.osd.7.smithi089.stdout:Mar 22 01:49:42 smithi089 podman[72974]: 2024-03-22 01:49:42.389794581 +0000 UTC m=+0.017631875 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T01:49:42.764 INFO:journalctl@ceph.osd.7.smithi089.stdout:Mar 22 01:49:42 smithi089 podman[72974]: 2024-03-22 01:49:42.529288162 +0000 UTC m=+0.157125440 container create a07bf4ce17c6602c0225ef8e1150d07941b02a3ba9fb858cfb6b5845db249109 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-22T01:49:43.065 INFO:journalctl@ceph.osd.7.smithi089.stdout:Mar 22 01:49:42 smithi089 podman[72974]: 2024-03-22 01:49:42.764917185 +0000 UTC m=+0.392754465 container init a07bf4ce17c6602c0225ef8e1150d07941b02a3ba9fb858cfb6b5845db249109 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-7-deactivate, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, ceph=True) 2024-03-22T01:49:43.066 INFO:journalctl@ceph.osd.7.smithi089.stdout:Mar 22 01:49:42 smithi089 podman[72974]: 2024-03-22 01:49:42.768238591 +0000 UTC m=+0.396075875 container start a07bf4ce17c6602c0225ef8e1150d07941b02a3ba9fb858cfb6b5845db249109 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-7-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-22T01:49:43.066 INFO:journalctl@ceph.osd.7.smithi089.stdout:Mar 22 01:49:42 smithi089 podman[72974]: 2024-03-22 01:49:42.86494264 +0000 UTC m=+0.492779935 container attach a07bf4ce17c6602c0225ef8e1150d07941b02a3ba9fb858cfb6b5845db249109 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-22T01:49:43.171 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:49:42 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:49:42.917+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:33.677632+0000 front 2024-03-22T01:48:33.677592+0000 (oldest deadline 2024-03-22T01:49:39.577549+0000) 2024-03-22T01:49:43.330 INFO:journalctl@ceph.osd.7.smithi089.stdout:Mar 22 01:49:43 smithi089 podman[72974]: 2024-03-22 01:49:43.067068345 +0000 UTC m=+0.694905638 container died a07bf4ce17c6602c0225ef8e1150d07941b02a3ba9fb858cfb6b5845db249109 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-7-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-22T01:49:43.373 INFO:journalctl@ceph.osd.9.smithi072.stdout:Mar 22 01:49:43 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-9[65321]: 2024-03-22T01:49:43.094+0000 7fcc0a003700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:33.784567+0000 front 2024-03-22T01:48:33.784520+0000 (oldest deadline 2024-03-22T01:49:40.185335+0000) 2024-03-22T01:49:43.608 INFO:journalctl@ceph.osd.10.smithi089.stdout:Mar 22 01:49:43 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-10[61378]: 2024-03-22T01:49:43.570+0000 7f8b187bf700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:32.441146+0000 front 2024-03-22T01:48:32.441205+0000 (oldest deadline 2024-03-22T01:49:37.741167+0000) 2024-03-22T01:49:43.672 INFO:journalctl@ceph.osd.8.smithi098.stdout:Mar 22 01:49:43 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-8[55604]: 2024-03-22T01:49:43.363+0000 7f987d6aa700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:34.540633+0000 front 2024-03-22T01:48:34.540612+0000 (oldest deadline 2024-03-22T01:49:36.240597+0000) 2024-03-22T01:49:43.727 DEBUG:teuthology.orchestra.run.smithi089:> sudo pkill -f 'journalctl -f -n 0 -u ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@osd.7.service' 2024-03-22T01:49:43.768 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T01:49:43.768 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-22T01:49:43.768 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-03-22T01:49:43.768 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@osd.8 2024-03-22T01:49:44.094 INFO:journalctl@ceph.osd.8.smithi098.stdout:Mar 22 01:49:43 smithi098 systemd[1]: Stopping Ceph osd.8 for 022a99bc-e7eb-11ee-95cd-87774f69a715... 2024-03-22T01:49:44.095 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:49:43 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:49:43.915+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:33.677632+0000 front 2024-03-22T01:48:33.677592+0000 (oldest deadline 2024-03-22T01:49:39.577549+0000) 2024-03-22T01:49:44.366 INFO:journalctl@ceph.osd.8.smithi098.stdout:Mar 22 01:49:44 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-8[55604]: 2024-03-22T01:49:44.094+0000 7f98850be700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T01:49:44.366 INFO:journalctl@ceph.osd.8.smithi098.stdout:Mar 22 01:49:44 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-8[55604]: 2024-03-22T01:49:44.094+0000 7f98850be700 -1 osd.8 100 *** Got signal Terminated *** 2024-03-22T01:49:44.366 INFO:journalctl@ceph.osd.8.smithi098.stdout:Mar 22 01:49:44 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-8[55604]: 2024-03-22T01:49:44.094+0000 7f98850be700 -1 osd.8 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-22T01:49:44.366 INFO:journalctl@ceph.osd.8.smithi098.stdout:Mar 22 01:49:44 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-8[55604]: 2024-03-22T01:49:44.366+0000 7f987d6aa700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:34.540633+0000 front 2024-03-22T01:48:34.540612+0000 (oldest deadline 2024-03-22T01:49:36.240597+0000) 2024-03-22T01:49:44.372 INFO:journalctl@ceph.osd.9.smithi072.stdout:Mar 22 01:49:44 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-9[65321]: 2024-03-22T01:49:44.046+0000 7fcc0a003700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:33.784567+0000 front 2024-03-22T01:48:33.784520+0000 (oldest deadline 2024-03-22T01:49:40.185335+0000) 2024-03-22T01:49:44.858 INFO:journalctl@ceph.osd.10.smithi089.stdout:Mar 22 01:49:44 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-10[61378]: 2024-03-22T01:49:44.536+0000 7f8b187bf700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:32.441146+0000 front 2024-03-22T01:48:32.441205+0000 (oldest deadline 2024-03-22T01:49:37.741167+0000) 2024-03-22T01:49:45.172 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:49:44 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:49:44.869+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:33.677632+0000 front 2024-03-22T01:48:33.677592+0000 (oldest deadline 2024-03-22T01:49:39.577549+0000) 2024-03-22T01:49:45.372 INFO:journalctl@ceph.osd.9.smithi072.stdout:Mar 22 01:49:45 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-9[65321]: 2024-03-22T01:49:45.019+0000 7fcc0a003700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:33.784567+0000 front 2024-03-22T01:48:33.784520+0000 (oldest deadline 2024-03-22T01:49:40.185335+0000) 2024-03-22T01:49:45.671 INFO:journalctl@ceph.osd.8.smithi098.stdout:Mar 22 01:49:45 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-8[55604]: 2024-03-22T01:49:45.398+0000 7f987d6aa700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:34.540633+0000 front 2024-03-22T01:48:34.540612+0000 (oldest deadline 2024-03-22T01:49:36.240597+0000) 2024-03-22T01:49:45.858 INFO:journalctl@ceph.osd.10.smithi089.stdout:Mar 22 01:49:45 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-10[61378]: 2024-03-22T01:49:45.565+0000 7f8b187bf700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:32.441146+0000 front 2024-03-22T01:48:32.441205+0000 (oldest deadline 2024-03-22T01:49:37.741167+0000) 2024-03-22T01:49:46.172 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:49:45 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:49:45.828+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:33.677632+0000 front 2024-03-22T01:48:33.677592+0000 (oldest deadline 2024-03-22T01:49:39.577549+0000) 2024-03-22T01:49:46.372 INFO:journalctl@ceph.osd.9.smithi072.stdout:Mar 22 01:49:45 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-9[65321]: 2024-03-22T01:49:45.993+0000 7fcc0a003700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:33.784567+0000 front 2024-03-22T01:48:33.784520+0000 (oldest deadline 2024-03-22T01:49:40.185335+0000) 2024-03-22T01:49:46.850 INFO:journalctl@ceph.osd.8.smithi098.stdout:Mar 22 01:49:46 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-8[55604]: 2024-03-22T01:49:46.436+0000 7f987d6aa700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:34.540633+0000 front 2024-03-22T01:48:34.540612+0000 (oldest deadline 2024-03-22T01:49:36.240597+0000) 2024-03-22T01:49:46.858 INFO:journalctl@ceph.osd.10.smithi089.stdout:Mar 22 01:49:46 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-10[61378]: 2024-03-22T01:49:46.604+0000 7f8b187bf700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:32.441146+0000 front 2024-03-22T01:48:32.441205+0000 (oldest deadline 2024-03-22T01:49:37.741167+0000) 2024-03-22T01:49:47.172 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:49:46 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:49:46.851+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:33.677632+0000 front 2024-03-22T01:48:33.677592+0000 (oldest deadline 2024-03-22T01:49:39.577549+0000) 2024-03-22T01:49:47.372 INFO:journalctl@ceph.osd.9.smithi072.stdout:Mar 22 01:49:47 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-9[65321]: 2024-03-22T01:49:47.043+0000 7fcc0a003700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:33.784567+0000 front 2024-03-22T01:48:33.784520+0000 (oldest deadline 2024-03-22T01:49:40.185335+0000) 2024-03-22T01:49:47.854 INFO:journalctl@ceph.osd.8.smithi098.stdout:Mar 22 01:49:47 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-8[55604]: 2024-03-22T01:49:47.475+0000 7f987d6aa700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:34.540633+0000 front 2024-03-22T01:48:34.540612+0000 (oldest deadline 2024-03-22T01:49:36.240597+0000) 2024-03-22T01:49:48.108 INFO:journalctl@ceph.osd.10.smithi089.stdout:Mar 22 01:49:47 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-10[61378]: 2024-03-22T01:49:47.635+0000 7f8b187bf700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:32.441146+0000 front 2024-03-22T01:48:32.441205+0000 (oldest deadline 2024-03-22T01:49:37.741167+0000) 2024-03-22T01:49:48.171 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:49:47 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:49:47.855+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:33.677632+0000 front 2024-03-22T01:48:33.677592+0000 (oldest deadline 2024-03-22T01:49:39.577549+0000) 2024-03-22T01:49:48.372 INFO:journalctl@ceph.osd.9.smithi072.stdout:Mar 22 01:49:48 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-9[65321]: 2024-03-22T01:49:48.011+0000 7fcc0a003700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:33.784567+0000 front 2024-03-22T01:48:33.784520+0000 (oldest deadline 2024-03-22T01:49:40.185335+0000) 2024-03-22T01:49:48.858 INFO:journalctl@ceph.osd.10.smithi089.stdout:Mar 22 01:49:48 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-10[61378]: 2024-03-22T01:49:48.602+0000 7f8b187bf700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:32.441146+0000 front 2024-03-22T01:48:32.441205+0000 (oldest deadline 2024-03-22T01:49:37.741167+0000) 2024-03-22T01:49:48.869 INFO:journalctl@ceph.osd.8.smithi098.stdout:Mar 22 01:49:48 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-8[55604]: 2024-03-22T01:49:48.466+0000 7f987d6aa700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:34.540633+0000 front 2024-03-22T01:48:34.540612+0000 (oldest deadline 2024-03-22T01:49:36.240597+0000) 2024-03-22T01:49:49.172 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:49:48 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:49:48.870+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:33.677632+0000 front 2024-03-22T01:48:33.677592+0000 (oldest deadline 2024-03-22T01:49:39.577549+0000) 2024-03-22T01:49:49.372 INFO:journalctl@ceph.osd.9.smithi072.stdout:Mar 22 01:49:48 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-9[65321]: 2024-03-22T01:49:48.976+0000 7fcc0a003700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:33.784567+0000 front 2024-03-22T01:48:33.784520+0000 (oldest deadline 2024-03-22T01:49:40.185335+0000) 2024-03-22T01:49:49.762 INFO:journalctl@ceph.osd.8.smithi098.stdout:Mar 22 01:49:49 smithi098 podman[74391]: 2024-03-22 01:49:49.476585707 +0000 UTC m=+5.511432931 container died 1696af3747dc1178e5906d155bd56ecd0599ac9b0a95d15be900306f80ef2a7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-8, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-22T01:49:49.858 INFO:journalctl@ceph.osd.10.smithi089.stdout:Mar 22 01:49:49 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-10[61378]: 2024-03-22T01:49:49.603+0000 7f8b187bf700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:32.441146+0000 front 2024-03-22T01:48:32.441205+0000 (oldest deadline 2024-03-22T01:49:37.741167+0000) 2024-03-22T01:49:50.154 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:49:49 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:49:49.847+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:33.677632+0000 front 2024-03-22T01:48:33.677592+0000 (oldest deadline 2024-03-22T01:49:39.577549+0000) 2024-03-22T01:49:50.154 INFO:journalctl@ceph.osd.8.smithi098.stdout:Mar 22 01:49:49 smithi098 podman[74391]: 2024-03-22 01:49:49.862442079 +0000 UTC m=+5.897289290 container cleanup 1696af3747dc1178e5906d155bd56ecd0599ac9b0a95d15be900306f80ef2a7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-8, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-22T01:49:50.155 INFO:journalctl@ceph.osd.8.smithi098.stdout:Mar 22 01:49:49 smithi098 bash[74391]: ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-8 2024-03-22T01:49:50.372 INFO:journalctl@ceph.osd.9.smithi072.stdout:Mar 22 01:49:49 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-9[65321]: 2024-03-22T01:49:49.939+0000 7fcc0a003700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:33.784567+0000 front 2024-03-22T01:48:33.784520+0000 (oldest deadline 2024-03-22T01:49:40.185335+0000) 2024-03-22T01:49:50.422 INFO:journalctl@ceph.osd.8.smithi098.stdout:Mar 22 01:49:50 smithi098 podman[74409]: 2024-03-22 01:49:50.155540731 +0000 UTC m=+0.677230785 container remove 1696af3747dc1178e5906d155bd56ecd0599ac9b0a95d15be900306f80ef2a7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-8, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-22T01:49:50.741 INFO:journalctl@ceph.osd.8.smithi098.stdout:Mar 22 01:49:50 smithi098 podman[74503]: 2024-03-22 01:49:50.368421601 +0000 UTC m=+0.018242969 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T01:49:50.741 INFO:journalctl@ceph.osd.8.smithi098.stdout:Mar 22 01:49:50 smithi098 podman[74503]: 2024-03-22 01:49:50.481534368 +0000 UTC m=+0.131355791 container create 419502b61e4539914a177e59cdb8849c481e1c79176e9a103ed00777ac1d5e33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-8-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, maintainer=Guillaume Abrioux ) 2024-03-22T01:49:50.858 INFO:journalctl@ceph.osd.10.smithi089.stdout:Mar 22 01:49:50 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-10[61378]: 2024-03-22T01:49:50.569+0000 7f8b187bf700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:32.441146+0000 front 2024-03-22T01:48:32.441205+0000 (oldest deadline 2024-03-22T01:49:37.741167+0000) 2024-03-22T01:49:51.035 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:49:50 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:49:50.798+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:33.677632+0000 front 2024-03-22T01:48:33.677592+0000 (oldest deadline 2024-03-22T01:49:39.577549+0000) 2024-03-22T01:49:51.036 INFO:journalctl@ceph.osd.8.smithi098.stdout:Mar 22 01:49:50 smithi098 podman[74503]: 2024-03-22 01:49:50.74301592 +0000 UTC m=+0.392837295 container init 419502b61e4539914a177e59cdb8849c481e1c79176e9a103ed00777ac1d5e33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-8-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-22T01:49:51.036 INFO:journalctl@ceph.osd.8.smithi098.stdout:Mar 22 01:49:50 smithi098 podman[74503]: 2024-03-22 01:49:50.746172918 +0000 UTC m=+0.395994285 container start 419502b61e4539914a177e59cdb8849c481e1c79176e9a103ed00777ac1d5e33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-8-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, ceph=True, org.label-schema.build-date=20240319, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-22T01:49:51.036 INFO:journalctl@ceph.osd.8.smithi098.stdout:Mar 22 01:49:50 smithi098 podman[74503]: 2024-03-22 01:49:50.851671499 +0000 UTC m=+0.501492862 container attach 419502b61e4539914a177e59cdb8849c481e1c79176e9a103ed00777ac1d5e33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-8-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-22T01:49:51.303 INFO:journalctl@ceph.osd.8.smithi098.stdout:Mar 22 01:49:51 smithi098 podman[74503]: 2024-03-22 01:49:51.037135456 +0000 UTC m=+0.686956821 container died 419502b61e4539914a177e59cdb8849c481e1c79176e9a103ed00777ac1d5e33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-8-deactivate, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-22T01:49:51.372 INFO:journalctl@ceph.osd.9.smithi072.stdout:Mar 22 01:49:50 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-9[65321]: 2024-03-22T01:49:50.945+0000 7fcc0a003700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:33.784567+0000 front 2024-03-22T01:48:33.784520+0000 (oldest deadline 2024-03-22T01:49:40.185335+0000) 2024-03-22T01:49:51.670 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@osd.8.service' 2024-03-22T01:49:51.712 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T01:49:51.712 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-03-22T01:49:51.712 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-03-22T01:49:51.712 DEBUG:teuthology.orchestra.run.smithi072:> sudo systemctl stop ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@osd.9 2024-03-22T01:49:51.858 INFO:journalctl@ceph.osd.10.smithi089.stdout:Mar 22 01:49:51 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-10[61378]: 2024-03-22T01:49:51.567+0000 7f8b187bf700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:32.441146+0000 front 2024-03-22T01:48:32.441205+0000 (oldest deadline 2024-03-22T01:49:37.741167+0000) 2024-03-22T01:49:51.922 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:49:51 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:49:51.805+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:33.677632+0000 front 2024-03-22T01:48:33.677592+0000 (oldest deadline 2024-03-22T01:49:39.577549+0000) 2024-03-22T01:49:52.021 INFO:journalctl@ceph.osd.9.smithi072.stdout:Mar 22 01:49:51 smithi072 systemd[1]: Stopping Ceph osd.9 for 022a99bc-e7eb-11ee-95cd-87774f69a715... 2024-03-22T01:49:52.021 INFO:journalctl@ceph.osd.9.smithi072.stdout:Mar 22 01:49:51 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-9[65321]: 2024-03-22T01:49:51.937+0000 7fcc0a003700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:33.784567+0000 front 2024-03-22T01:48:33.784520+0000 (oldest deadline 2024-03-22T01:49:40.185335+0000) 2024-03-22T01:49:52.372 INFO:journalctl@ceph.osd.9.smithi072.stdout:Mar 22 01:49:52 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-9[65321]: 2024-03-22T01:49:52.021+0000 7fcc0da0f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T01:49:52.373 INFO:journalctl@ceph.osd.9.smithi072.stdout:Mar 22 01:49:52 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-9[65321]: 2024-03-22T01:49:52.021+0000 7fcc0da0f700 -1 osd.9 100 *** Got signal Terminated *** 2024-03-22T01:49:52.373 INFO:journalctl@ceph.osd.9.smithi072.stdout:Mar 22 01:49:52 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-9[65321]: 2024-03-22T01:49:52.021+0000 7fcc0da0f700 -1 osd.9 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-22T01:49:52.858 INFO:journalctl@ceph.osd.10.smithi089.stdout:Mar 22 01:49:52 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-10[61378]: 2024-03-22T01:49:52.520+0000 7f8b187bf700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:32.441146+0000 front 2024-03-22T01:48:32.441205+0000 (oldest deadline 2024-03-22T01:49:37.741167+0000) 2024-03-22T01:49:53.172 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:49:52 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:49:52.836+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:33.677632+0000 front 2024-03-22T01:48:33.677592+0000 (oldest deadline 2024-03-22T01:49:39.577549+0000) 2024-03-22T01:49:53.372 INFO:journalctl@ceph.osd.9.smithi072.stdout:Mar 22 01:49:52 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-9[65321]: 2024-03-22T01:49:52.967+0000 7fcc0a003700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:33.784567+0000 front 2024-03-22T01:48:33.784520+0000 (oldest deadline 2024-03-22T01:49:40.185335+0000) 2024-03-22T01:49:53.858 INFO:journalctl@ceph.osd.10.smithi089.stdout:Mar 22 01:49:53 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-10[61378]: 2024-03-22T01:49:53.520+0000 7f8b187bf700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:32.441146+0000 front 2024-03-22T01:48:32.441205+0000 (oldest deadline 2024-03-22T01:49:37.741167+0000) 2024-03-22T01:49:53.858 INFO:journalctl@ceph.osd.10.smithi089.stdout:Mar 22 01:49:53 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-10[61378]: 2024-03-22T01:49:53.520+0000 7f8b187bf700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.89:6804 osd.1 since back 2024-03-22T01:48:49.343691+0000 front 2024-03-22T01:48:49.343729+0000 (oldest deadline 2024-03-22T01:49:53.443772+0000) 2024-03-22T01:49:54.172 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:49:53 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:49:53.821+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:33.677632+0000 front 2024-03-22T01:48:33.677592+0000 (oldest deadline 2024-03-22T01:49:39.577549+0000) 2024-03-22T01:49:54.372 INFO:journalctl@ceph.osd.9.smithi072.stdout:Mar 22 01:49:54 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-9[65321]: 2024-03-22T01:49:54.002+0000 7fcc0a003700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:33.784567+0000 front 2024-03-22T01:48:33.784520+0000 (oldest deadline 2024-03-22T01:49:40.185335+0000) 2024-03-22T01:49:54.608 INFO:journalctl@ceph.osd.10.smithi089.stdout:Mar 22 01:49:54 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-10[61378]: 2024-03-22T01:49:54.568+0000 7f8b187bf700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:32.441146+0000 front 2024-03-22T01:48:32.441205+0000 (oldest deadline 2024-03-22T01:49:37.741167+0000) 2024-03-22T01:49:54.608 INFO:journalctl@ceph.osd.10.smithi089.stdout:Mar 22 01:49:54 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-10[61378]: 2024-03-22T01:49:54.568+0000 7f8b187bf700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.89:6804 osd.1 since back 2024-03-22T01:48:49.343691+0000 front 2024-03-22T01:48:49.343729+0000 (oldest deadline 2024-03-22T01:49:53.443772+0000) 2024-03-22T01:49:55.073 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:49:54 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:49:54.800+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:33.677632+0000 front 2024-03-22T01:48:33.677592+0000 (oldest deadline 2024-03-22T01:49:39.577549+0000) 2024-03-22T01:49:55.372 INFO:journalctl@ceph.osd.9.smithi072.stdout:Mar 22 01:49:54 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-9[65321]: 2024-03-22T01:49:54.979+0000 7fcc0a003700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:33.784567+0000 front 2024-03-22T01:48:33.784520+0000 (oldest deadline 2024-03-22T01:49:40.185335+0000) 2024-03-22T01:49:55.373 INFO:journalctl@ceph.osd.9.smithi072.stdout:Mar 22 01:49:54 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-9[65321]: 2024-03-22T01:49:54.979+0000 7fcc0a003700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.89:6804 osd.1 since back 2024-03-22T01:48:51.688880+0000 front 2024-03-22T01:48:51.688678+0000 (oldest deadline 2024-03-22T01:49:54.588680+0000) 2024-03-22T01:49:55.858 INFO:journalctl@ceph.osd.10.smithi089.stdout:Mar 22 01:49:55 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-10[61378]: 2024-03-22T01:49:55.564+0000 7f8b187bf700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:32.441146+0000 front 2024-03-22T01:48:32.441205+0000 (oldest deadline 2024-03-22T01:49:37.741167+0000) 2024-03-22T01:49:55.858 INFO:journalctl@ceph.osd.10.smithi089.stdout:Mar 22 01:49:55 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-10[61378]: 2024-03-22T01:49:55.564+0000 7f8b187bf700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.89:6804 osd.1 since back 2024-03-22T01:48:49.343691+0000 front 2024-03-22T01:48:49.343729+0000 (oldest deadline 2024-03-22T01:49:53.443772+0000) 2024-03-22T01:49:56.096 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:49:55 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:49:55.839+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:33.677632+0000 front 2024-03-22T01:48:33.677592+0000 (oldest deadline 2024-03-22T01:49:39.577549+0000) 2024-03-22T01:49:56.372 INFO:journalctl@ceph.osd.9.smithi072.stdout:Mar 22 01:49:55 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-9[65321]: 2024-03-22T01:49:55.985+0000 7fcc0a003700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:33.784567+0000 front 2024-03-22T01:48:33.784520+0000 (oldest deadline 2024-03-22T01:49:40.185335+0000) 2024-03-22T01:49:56.372 INFO:journalctl@ceph.osd.9.smithi072.stdout:Mar 22 01:49:55 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-9[65321]: 2024-03-22T01:49:55.985+0000 7fcc0a003700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.89:6804 osd.1 since back 2024-03-22T01:48:51.688880+0000 front 2024-03-22T01:48:51.688678+0000 (oldest deadline 2024-03-22T01:49:54.588680+0000) 2024-03-22T01:49:56.858 INFO:journalctl@ceph.osd.10.smithi089.stdout:Mar 22 01:49:56 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-10[61378]: 2024-03-22T01:49:56.583+0000 7f8b187bf700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:32.441146+0000 front 2024-03-22T01:48:32.441205+0000 (oldest deadline 2024-03-22T01:49:37.741167+0000) 2024-03-22T01:49:56.858 INFO:journalctl@ceph.osd.10.smithi089.stdout:Mar 22 01:49:56 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-10[61378]: 2024-03-22T01:49:56.583+0000 7f8b187bf700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.89:6804 osd.1 since back 2024-03-22T01:48:49.343691+0000 front 2024-03-22T01:48:49.343729+0000 (oldest deadline 2024-03-22T01:49:53.443772+0000) 2024-03-22T01:49:57.172 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:49:56 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:49:56.880+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:33.677632+0000 front 2024-03-22T01:48:33.677592+0000 (oldest deadline 2024-03-22T01:49:39.577549+0000) 2024-03-22T01:49:57.172 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:49:56 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:49:56.880+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.89:6804 osd.1 since back 2024-03-22T01:48:50.779479+0000 front 2024-03-22T01:48:50.779435+0000 (oldest deadline 2024-03-22T01:49:56.579980+0000) 2024-03-22T01:49:57.372 INFO:journalctl@ceph.osd.9.smithi072.stdout:Mar 22 01:49:56 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-9[65321]: 2024-03-22T01:49:56.992+0000 7fcc0a003700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:33.784567+0000 front 2024-03-22T01:48:33.784520+0000 (oldest deadline 2024-03-22T01:49:40.185335+0000) 2024-03-22T01:49:57.372 INFO:journalctl@ceph.osd.9.smithi072.stdout:Mar 22 01:49:56 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-9[65321]: 2024-03-22T01:49:56.992+0000 7fcc0a003700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.89:6804 osd.1 since back 2024-03-22T01:48:51.688880+0000 front 2024-03-22T01:48:51.688678+0000 (oldest deadline 2024-03-22T01:49:54.588680+0000) 2024-03-22T01:49:57.858 INFO:journalctl@ceph.osd.10.smithi089.stdout:Mar 22 01:49:57 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-10[61378]: 2024-03-22T01:49:57.559+0000 7f8b187bf700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:32.441146+0000 front 2024-03-22T01:48:32.441205+0000 (oldest deadline 2024-03-22T01:49:37.741167+0000) 2024-03-22T01:49:57.858 INFO:journalctl@ceph.osd.10.smithi089.stdout:Mar 22 01:49:57 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-10[61378]: 2024-03-22T01:49:57.559+0000 7f8b187bf700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.89:6804 osd.1 since back 2024-03-22T01:48:49.343691+0000 front 2024-03-22T01:48:49.343729+0000 (oldest deadline 2024-03-22T01:49:53.443772+0000) 2024-03-22T01:49:57.895 INFO:journalctl@ceph.osd.9.smithi072.stdout:Mar 22 01:49:57 smithi072 podman[97047]: 2024-03-22 01:49:57.624729045 +0000 UTC m=+5.715159864 container died 72f019a34cdb7fb73ea601a7f6897d6b3a170a2bd7adb0b4dbced7c9f488df33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-9, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, ceph=True) 2024-03-22T01:49:58.172 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:49:57 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:49:57.834+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:33.677632+0000 front 2024-03-22T01:48:33.677592+0000 (oldest deadline 2024-03-22T01:49:39.577549+0000) 2024-03-22T01:49:58.172 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:49:57 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:49:57.834+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.89:6804 osd.1 since back 2024-03-22T01:48:50.779479+0000 front 2024-03-22T01:48:50.779435+0000 (oldest deadline 2024-03-22T01:49:56.579980+0000) 2024-03-22T01:49:58.250 INFO:journalctl@ceph.osd.9.smithi072.stdout:Mar 22 01:49:57 smithi072 podman[97047]: 2024-03-22 01:49:57.990821109 +0000 UTC m=+6.081251934 container cleanup 72f019a34cdb7fb73ea601a7f6897d6b3a170a2bd7adb0b4dbced7c9f488df33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-9, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0) 2024-03-22T01:49:58.250 INFO:journalctl@ceph.osd.9.smithi072.stdout:Mar 22 01:49:57 smithi072 bash[97047]: ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-9 2024-03-22T01:49:58.604 INFO:journalctl@ceph.osd.9.smithi072.stdout:Mar 22 01:49:58 smithi072 podman[97062]: 2024-03-22 01:49:58.251349709 +0000 UTC m=+0.622168781 container remove 72f019a34cdb7fb73ea601a7f6897d6b3a170a2bd7adb0b4dbced7c9f488df33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-9, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-22T01:49:58.858 INFO:journalctl@ceph.osd.10.smithi089.stdout:Mar 22 01:49:58 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-10[61378]: 2024-03-22T01:49:58.560+0000 7f8b187bf700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:32.441146+0000 front 2024-03-22T01:48:32.441205+0000 (oldest deadline 2024-03-22T01:49:37.741167+0000) 2024-03-22T01:49:58.858 INFO:journalctl@ceph.osd.10.smithi089.stdout:Mar 22 01:49:58 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-10[61378]: 2024-03-22T01:49:58.560+0000 7f8b187bf700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.89:6804 osd.1 since back 2024-03-22T01:48:49.343691+0000 front 2024-03-22T01:48:49.343729+0000 (oldest deadline 2024-03-22T01:49:53.443772+0000) 2024-03-22T01:49:58.872 INFO:journalctl@ceph.osd.9.smithi072.stdout:Mar 22 01:49:58 smithi072 podman[97157]: 2024-03-22 01:49:58.509127143 +0000 UTC m=+0.018436160 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T01:49:58.873 INFO:journalctl@ceph.osd.9.smithi072.stdout:Mar 22 01:49:58 smithi072 podman[97157]: 2024-03-22 01:49:58.611378945 +0000 UTC m=+0.120687950 container create 8700397e5a6256962ab3a2e8ff4531aa56ac3ca092194eb694f368e1da0043bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-9-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.build-date=20240319, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-22T01:49:58.873 INFO:journalctl@ceph.osd.9.smithi072.stdout:Mar 22 01:49:58 smithi072 podman[97157]: 2024-03-22 01:49:58.825507459 +0000 UTC m=+0.334816456 container init 8700397e5a6256962ab3a2e8ff4531aa56ac3ca092194eb694f368e1da0043bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-9-deactivate, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-22T01:49:58.873 INFO:journalctl@ceph.osd.9.smithi072.stdout:Mar 22 01:49:58 smithi072 podman[97157]: 2024-03-22 01:49:58.828484373 +0000 UTC m=+0.337793376 container start 8700397e5a6256962ab3a2e8ff4531aa56ac3ca092194eb694f368e1da0043bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-9-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0) 2024-03-22T01:49:59.172 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:49:58 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:49:58.791+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:33.677632+0000 front 2024-03-22T01:48:33.677592+0000 (oldest deadline 2024-03-22T01:49:39.577549+0000) 2024-03-22T01:49:59.172 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:49:58 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:49:58.791+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.89:6804 osd.1 since back 2024-03-22T01:48:50.779479+0000 front 2024-03-22T01:48:50.779435+0000 (oldest deadline 2024-03-22T01:49:56.579980+0000) 2024-03-22T01:49:59.372 INFO:journalctl@ceph.osd.9.smithi072.stdout:Mar 22 01:49:58 smithi072 podman[97157]: 2024-03-22 01:49:58.900861509 +0000 UTC m=+0.410170524 container attach 8700397e5a6256962ab3a2e8ff4531aa56ac3ca092194eb694f368e1da0043bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-9-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-22T01:49:59.372 INFO:journalctl@ceph.osd.9.smithi072.stdout:Mar 22 01:49:59 smithi072 podman[97157]: 2024-03-22 01:49:59.119364059 +0000 UTC m=+0.628673057 container died 8700397e5a6256962ab3a2e8ff4531aa56ac3ca092194eb694f368e1da0043bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-9-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-22T01:49:59.679 INFO:journalctl@ceph.osd.9.smithi072.stdout:Mar 22 01:49:59 smithi072 podman[97183]: 2024-03-22 01:49:59.667487939 +0000 UTC m=+0.542940260 container remove 8700397e5a6256962ab3a2e8ff4531aa56ac3ca092194eb694f368e1da0043bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-9-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, maintainer=Guillaume Abrioux ) 2024-03-22T01:49:59.685 DEBUG:teuthology.orchestra.run.smithi072:> sudo pkill -f 'journalctl -f -n 0 -u ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@osd.9.service' 2024-03-22T01:49:59.728 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T01:49:59.728 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-03-22T01:49:59.729 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-03-22T01:49:59.729 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@osd.10 2024-03-22T01:49:59.774 INFO:journalctl@ceph.osd.10.smithi089.stdout:Mar 22 01:49:59 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-10[61378]: 2024-03-22T01:49:59.524+0000 7f8b187bf700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:32.441146+0000 front 2024-03-22T01:48:32.441205+0000 (oldest deadline 2024-03-22T01:49:37.741167+0000) 2024-03-22T01:49:59.774 INFO:journalctl@ceph.osd.10.smithi089.stdout:Mar 22 01:49:59 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-10[61378]: 2024-03-22T01:49:59.524+0000 7f8b187bf700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.89:6804 osd.1 since back 2024-03-22T01:48:49.343691+0000 front 2024-03-22T01:48:49.343729+0000 (oldest deadline 2024-03-22T01:49:53.443772+0000) 2024-03-22T01:50:00.032 INFO:journalctl@ceph.osd.10.smithi089.stdout:Mar 22 01:49:59 smithi089 systemd[1]: Stopping Ceph osd.10 for 022a99bc-e7eb-11ee-95cd-87774f69a715... 2024-03-22T01:50:00.033 INFO:journalctl@ceph.osd.10.smithi089.stdout:Mar 22 01:50:00 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-10[61378]: 2024-03-22T01:50:00.020+0000 7f8b201d3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T01:50:00.033 INFO:journalctl@ceph.osd.10.smithi089.stdout:Mar 22 01:50:00 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-10[61378]: 2024-03-22T01:50:00.020+0000 7f8b201d3700 -1 osd.10 100 *** Got signal Terminated *** 2024-03-22T01:50:00.033 INFO:journalctl@ceph.osd.10.smithi089.stdout:Mar 22 01:50:00 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-10[61378]: 2024-03-22T01:50:00.020+0000 7f8b201d3700 -1 osd.10 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-22T01:50:00.172 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:49:59 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:49:59.815+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:33.677632+0000 front 2024-03-22T01:48:33.677592+0000 (oldest deadline 2024-03-22T01:49:39.577549+0000) 2024-03-22T01:50:00.172 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:49:59 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:49:59.815+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.89:6804 osd.1 since back 2024-03-22T01:48:50.779479+0000 front 2024-03-22T01:48:50.779435+0000 (oldest deadline 2024-03-22T01:49:56.579980+0000) 2024-03-22T01:50:00.858 INFO:journalctl@ceph.osd.10.smithi089.stdout:Mar 22 01:50:00 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-10[61378]: 2024-03-22T01:50:00.516+0000 7f8b187bf700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:32.441146+0000 front 2024-03-22T01:48:32.441205+0000 (oldest deadline 2024-03-22T01:49:37.741167+0000) 2024-03-22T01:50:00.858 INFO:journalctl@ceph.osd.10.smithi089.stdout:Mar 22 01:50:00 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-10[61378]: 2024-03-22T01:50:00.516+0000 7f8b187bf700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.89:6804 osd.1 since back 2024-03-22T01:48:49.343691+0000 front 2024-03-22T01:48:49.343729+0000 (oldest deadline 2024-03-22T01:49:53.443772+0000) 2024-03-22T01:50:01.172 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:50:00 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:50:00.861+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:33.677632+0000 front 2024-03-22T01:48:33.677592+0000 (oldest deadline 2024-03-22T01:49:39.577549+0000) 2024-03-22T01:50:01.172 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:50:00 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:50:00.861+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.89:6804 osd.1 since back 2024-03-22T01:48:50.779479+0000 front 2024-03-22T01:48:50.779435+0000 (oldest deadline 2024-03-22T01:49:56.579980+0000) 2024-03-22T01:50:01.858 INFO:journalctl@ceph.osd.10.smithi089.stdout:Mar 22 01:50:01 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-10[61378]: 2024-03-22T01:50:01.517+0000 7f8b187bf700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:32.441146+0000 front 2024-03-22T01:48:32.441205+0000 (oldest deadline 2024-03-22T01:49:37.741167+0000) 2024-03-22T01:50:01.858 INFO:journalctl@ceph.osd.10.smithi089.stdout:Mar 22 01:50:01 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-10[61378]: 2024-03-22T01:50:01.517+0000 7f8b187bf700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.89:6804 osd.1 since back 2024-03-22T01:48:49.343691+0000 front 2024-03-22T01:48:49.343729+0000 (oldest deadline 2024-03-22T01:49:53.443772+0000) 2024-03-22T01:50:02.172 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:50:01 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:50:01.848+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:33.677632+0000 front 2024-03-22T01:48:33.677592+0000 (oldest deadline 2024-03-22T01:49:39.577549+0000) 2024-03-22T01:50:02.172 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:50:01 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:50:01.848+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.89:6804 osd.1 since back 2024-03-22T01:48:50.779479+0000 front 2024-03-22T01:48:50.779435+0000 (oldest deadline 2024-03-22T01:49:56.579980+0000) 2024-03-22T01:50:02.172 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:50:01 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:50:01.848+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.98:6804 osd.2 since back 2024-03-22T01:48:56.580387+0000 front 2024-03-22T01:48:56.580350+0000 (oldest deadline 2024-03-22T01:50:01.181087+0000) 2024-03-22T01:50:02.858 INFO:journalctl@ceph.osd.10.smithi089.stdout:Mar 22 01:50:02 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-10[61378]: 2024-03-22T01:50:02.525+0000 7f8b187bf700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:32.441146+0000 front 2024-03-22T01:48:32.441205+0000 (oldest deadline 2024-03-22T01:49:37.741167+0000) 2024-03-22T01:50:02.858 INFO:journalctl@ceph.osd.10.smithi089.stdout:Mar 22 01:50:02 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-10[61378]: 2024-03-22T01:50:02.525+0000 7f8b187bf700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.89:6804 osd.1 since back 2024-03-22T01:48:49.343691+0000 front 2024-03-22T01:48:49.343729+0000 (oldest deadline 2024-03-22T01:49:53.443772+0000) 2024-03-22T01:50:02.858 INFO:journalctl@ceph.osd.10.smithi089.stdout:Mar 22 01:50:02 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-10[61378]: 2024-03-22T01:50:02.525+0000 7f8b187bf700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.98:6804 osd.2 since back 2024-03-22T01:48:58.045740+0000 front 2024-03-22T01:48:58.045645+0000 (oldest deadline 2024-03-22T01:50:02.145572+0000) 2024-03-22T01:50:03.172 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:50:02 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:50:02.846+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:33.677632+0000 front 2024-03-22T01:48:33.677592+0000 (oldest deadline 2024-03-22T01:49:39.577549+0000) 2024-03-22T01:50:03.172 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:50:02 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:50:02.846+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.89:6804 osd.1 since back 2024-03-22T01:48:50.779479+0000 front 2024-03-22T01:48:50.779435+0000 (oldest deadline 2024-03-22T01:49:56.579980+0000) 2024-03-22T01:50:03.172 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:50:02 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:50:02.846+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.98:6804 osd.2 since back 2024-03-22T01:48:56.580387+0000 front 2024-03-22T01:48:56.580350+0000 (oldest deadline 2024-03-22T01:50:01.181087+0000) 2024-03-22T01:50:03.858 INFO:journalctl@ceph.osd.10.smithi089.stdout:Mar 22 01:50:03 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-10[61378]: 2024-03-22T01:50:03.537+0000 7f8b187bf700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:32.441146+0000 front 2024-03-22T01:48:32.441205+0000 (oldest deadline 2024-03-22T01:49:37.741167+0000) 2024-03-22T01:50:03.858 INFO:journalctl@ceph.osd.10.smithi089.stdout:Mar 22 01:50:03 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-10[61378]: 2024-03-22T01:50:03.537+0000 7f8b187bf700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.89:6804 osd.1 since back 2024-03-22T01:48:49.343691+0000 front 2024-03-22T01:48:49.343729+0000 (oldest deadline 2024-03-22T01:49:53.443772+0000) 2024-03-22T01:50:03.858 INFO:journalctl@ceph.osd.10.smithi089.stdout:Mar 22 01:50:03 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-10[61378]: 2024-03-22T01:50:03.537+0000 7f8b187bf700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.98:6804 osd.2 since back 2024-03-22T01:48:58.045740+0000 front 2024-03-22T01:48:58.045645+0000 (oldest deadline 2024-03-22T01:50:02.145572+0000) 2024-03-22T01:50:04.172 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:50:03 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:50:03.800+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:33.677632+0000 front 2024-03-22T01:48:33.677592+0000 (oldest deadline 2024-03-22T01:49:39.577549+0000) 2024-03-22T01:50:04.172 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:50:03 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:50:03.800+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.89:6804 osd.1 since back 2024-03-22T01:48:50.779479+0000 front 2024-03-22T01:48:50.779435+0000 (oldest deadline 2024-03-22T01:49:56.579980+0000) 2024-03-22T01:50:04.172 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:50:03 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:50:03.800+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.98:6804 osd.2 since back 2024-03-22T01:48:56.580387+0000 front 2024-03-22T01:48:56.580350+0000 (oldest deadline 2024-03-22T01:50:01.181087+0000) 2024-03-22T01:50:04.858 INFO:journalctl@ceph.osd.10.smithi089.stdout:Mar 22 01:50:04 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-10[61378]: 2024-03-22T01:50:04.509+0000 7f8b187bf700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:32.441146+0000 front 2024-03-22T01:48:32.441205+0000 (oldest deadline 2024-03-22T01:49:37.741167+0000) 2024-03-22T01:50:04.858 INFO:journalctl@ceph.osd.10.smithi089.stdout:Mar 22 01:50:04 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-10[61378]: 2024-03-22T01:50:04.509+0000 7f8b187bf700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.89:6804 osd.1 since back 2024-03-22T01:48:49.343691+0000 front 2024-03-22T01:48:49.343729+0000 (oldest deadline 2024-03-22T01:49:53.443772+0000) 2024-03-22T01:50:04.858 INFO:journalctl@ceph.osd.10.smithi089.stdout:Mar 22 01:50:04 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-10[61378]: 2024-03-22T01:50:04.509+0000 7f8b187bf700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.98:6804 osd.2 since back 2024-03-22T01:48:58.045740+0000 front 2024-03-22T01:48:58.045645+0000 (oldest deadline 2024-03-22T01:50:02.145572+0000) 2024-03-22T01:50:05.172 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:50:04 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:50:04.754+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:33.677632+0000 front 2024-03-22T01:48:33.677592+0000 (oldest deadline 2024-03-22T01:49:39.577549+0000) 2024-03-22T01:50:05.172 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:50:04 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:50:04.754+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.89:6804 osd.1 since back 2024-03-22T01:48:50.779479+0000 front 2024-03-22T01:48:50.779435+0000 (oldest deadline 2024-03-22T01:49:56.579980+0000) 2024-03-22T01:50:05.172 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:50:04 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:50:04.754+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.98:6804 osd.2 since back 2024-03-22T01:48:56.580387+0000 front 2024-03-22T01:48:56.580350+0000 (oldest deadline 2024-03-22T01:50:01.181087+0000) 2024-03-22T01:50:05.779 INFO:journalctl@ceph.osd.10.smithi089.stdout:Mar 22 01:50:05 smithi089 podman[73091]: 2024-03-22 01:50:05.515693719 +0000 UTC m=+5.597891462 container died 55ccb426eabe040f744a3bf75741092607a085d76538732cd403e6e7f08fe9cf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-10, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319) 2024-03-22T01:50:06.038 INFO:journalctl@ceph.osd.10.smithi089.stdout:Mar 22 01:50:05 smithi089 podman[73091]: 2024-03-22 01:50:05.862837316 +0000 UTC m=+5.945035037 container cleanup 55ccb426eabe040f744a3bf75741092607a085d76538732cd403e6e7f08fe9cf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-10, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-22T01:50:06.039 INFO:journalctl@ceph.osd.10.smithi089.stdout:Mar 22 01:50:05 smithi089 bash[73091]: ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-10 2024-03-22T01:50:06.135 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:50:05 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:50:05.742+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:33.677632+0000 front 2024-03-22T01:48:33.677592+0000 (oldest deadline 2024-03-22T01:49:39.577549+0000) 2024-03-22T01:50:06.136 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:50:05 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:50:05.742+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.89:6804 osd.1 since back 2024-03-22T01:48:50.779479+0000 front 2024-03-22T01:48:50.779435+0000 (oldest deadline 2024-03-22T01:49:56.579980+0000) 2024-03-22T01:50:06.136 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:50:05 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:50:05.742+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.98:6804 osd.2 since back 2024-03-22T01:48:56.580387+0000 front 2024-03-22T01:48:56.580350+0000 (oldest deadline 2024-03-22T01:50:01.181087+0000) 2024-03-22T01:50:06.358 INFO:journalctl@ceph.osd.10.smithi089.stdout:Mar 22 01:50:06 smithi089 podman[73110]: 2024-03-22 01:50:06.040169129 +0000 UTC m=+0.520130066 container remove 55ccb426eabe040f744a3bf75741092607a085d76538732cd403e6e7f08fe9cf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-10, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, maintainer=Guillaume Abrioux ) 2024-03-22T01:50:06.783 INFO:journalctl@ceph.osd.10.smithi089.stdout:Mar 22 01:50:06 smithi089 podman[73200]: 2024-03-22 01:50:06.372792101 +0000 UTC m=+0.017759131 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T01:50:06.783 INFO:journalctl@ceph.osd.10.smithi089.stdout:Mar 22 01:50:06 smithi089 podman[73200]: 2024-03-22 01:50:06.474048466 +0000 UTC m=+0.119015487 container create 5c3149c8a6ed8add55c7d60c827e96fd1ba962e520f89ab8a4decdd699966858 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-10-deactivate, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-22T01:50:06.783 INFO:journalctl@ceph.osd.10.smithi089.stdout:Mar 22 01:50:06 smithi089 podman[73200]: 2024-03-22 01:50:06.693257974 +0000 UTC m=+0.338225007 container init 5c3149c8a6ed8add55c7d60c827e96fd1ba962e520f89ab8a4decdd699966858 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-10-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD) 2024-03-22T01:50:06.783 INFO:journalctl@ceph.osd.10.smithi089.stdout:Mar 22 01:50:06 smithi089 podman[73200]: 2024-03-22 01:50:06.696787644 +0000 UTC m=+0.341754662 container start 5c3149c8a6ed8add55c7d60c827e96fd1ba962e520f89ab8a4decdd699966858 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-10-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-22T01:50:07.108 INFO:journalctl@ceph.osd.10.smithi089.stdout:Mar 22 01:50:06 smithi089 podman[73200]: 2024-03-22 01:50:06.784534624 +0000 UTC m=+0.429501659 container attach 5c3149c8a6ed8add55c7d60c827e96fd1ba962e520f89ab8a4decdd699966858 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-10-deactivate, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-22T01:50:07.108 INFO:journalctl@ceph.osd.10.smithi089.stdout:Mar 22 01:50:06 smithi089 podman[73200]: 2024-03-22 01:50:06.988000854 +0000 UTC m=+0.632967866 container died 5c3149c8a6ed8add55c7d60c827e96fd1ba962e520f89ab8a4decdd699966858 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-10-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-22T01:50:07.172 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:50:06 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:50:06.740+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:33.677632+0000 front 2024-03-22T01:48:33.677592+0000 (oldest deadline 2024-03-22T01:49:39.577549+0000) 2024-03-22T01:50:07.172 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:50:06 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:50:06.740+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.89:6804 osd.1 since back 2024-03-22T01:48:50.779479+0000 front 2024-03-22T01:48:50.779435+0000 (oldest deadline 2024-03-22T01:49:56.579980+0000) 2024-03-22T01:50:07.172 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:50:06 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:50:06.740+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.98:6804 osd.2 since back 2024-03-22T01:48:56.580387+0000 front 2024-03-22T01:48:56.580350+0000 (oldest deadline 2024-03-22T01:50:01.181087+0000) 2024-03-22T01:50:07.602 DEBUG:teuthology.orchestra.run.smithi089:> sudo pkill -f 'journalctl -f -n 0 -u ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@osd.10.service' 2024-03-22T01:50:07.643 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T01:50:07.644 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-03-22T01:50:07.644 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-03-22T01:50:07.644 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@osd.11 2024-03-22T01:50:07.941 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:50:07 smithi098 systemd[1]: Stopping Ceph osd.11 for 022a99bc-e7eb-11ee-95cd-87774f69a715... 2024-03-22T01:50:07.941 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:50:07 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:50:07.768+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:33.677632+0000 front 2024-03-22T01:48:33.677592+0000 (oldest deadline 2024-03-22T01:49:39.577549+0000) 2024-03-22T01:50:07.941 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:50:07 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:50:07.768+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.89:6804 osd.1 since back 2024-03-22T01:48:50.779479+0000 front 2024-03-22T01:48:50.779435+0000 (oldest deadline 2024-03-22T01:49:56.579980+0000) 2024-03-22T01:50:07.941 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:50:07 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:50:07.768+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.98:6804 osd.2 since back 2024-03-22T01:48:56.580387+0000 front 2024-03-22T01:48:56.580350+0000 (oldest deadline 2024-03-22T01:50:01.181087+0000) 2024-03-22T01:50:07.941 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:50:07 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:50:07.941+0000 7f6446fda700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T01:50:07.941 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:50:07 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:50:07.941+0000 7f6446fda700 -1 osd.11 100 *** Got signal Terminated *** 2024-03-22T01:50:07.941 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:50:07 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:50:07.941+0000 7f6446fda700 -1 osd.11 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-22T01:50:09.172 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:50:08 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:50:08.733+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:33.677632+0000 front 2024-03-22T01:48:33.677592+0000 (oldest deadline 2024-03-22T01:49:39.577549+0000) 2024-03-22T01:50:09.172 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:50:08 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:50:08.733+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.89:6804 osd.1 since back 2024-03-22T01:48:50.779479+0000 front 2024-03-22T01:48:50.779435+0000 (oldest deadline 2024-03-22T01:49:56.579980+0000) 2024-03-22T01:50:09.172 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:50:08 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:50:08.733+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.98:6804 osd.2 since back 2024-03-22T01:48:56.580387+0000 front 2024-03-22T01:48:56.580350+0000 (oldest deadline 2024-03-22T01:50:01.181087+0000) 2024-03-22T01:50:10.172 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:50:09 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:50:09.766+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:33.677632+0000 front 2024-03-22T01:48:33.677592+0000 (oldest deadline 2024-03-22T01:49:39.577549+0000) 2024-03-22T01:50:10.172 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:50:09 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:50:09.767+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.89:6804 osd.1 since back 2024-03-22T01:48:50.779479+0000 front 2024-03-22T01:48:50.779435+0000 (oldest deadline 2024-03-22T01:49:56.579980+0000) 2024-03-22T01:50:10.172 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:50:09 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:50:09.767+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.98:6804 osd.2 since back 2024-03-22T01:48:56.580387+0000 front 2024-03-22T01:48:56.580350+0000 (oldest deadline 2024-03-22T01:50:01.181087+0000) 2024-03-22T01:50:11.172 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:50:10 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:50:10.797+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:33.677632+0000 front 2024-03-22T01:48:33.677592+0000 (oldest deadline 2024-03-22T01:49:39.577549+0000) 2024-03-22T01:50:11.172 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:50:10 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:50:10.797+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.89:6804 osd.1 since back 2024-03-22T01:48:50.779479+0000 front 2024-03-22T01:48:50.779435+0000 (oldest deadline 2024-03-22T01:49:56.579980+0000) 2024-03-22T01:50:11.172 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:50:10 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:50:10.797+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.98:6804 osd.2 since back 2024-03-22T01:48:56.580387+0000 front 2024-03-22T01:48:56.580350+0000 (oldest deadline 2024-03-22T01:50:01.181087+0000) 2024-03-22T01:50:12.171 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:50:11 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:50:11.820+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:33.677632+0000 front 2024-03-22T01:48:33.677592+0000 (oldest deadline 2024-03-22T01:49:39.577549+0000) 2024-03-22T01:50:12.172 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:50:11 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:50:11.820+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.89:6804 osd.1 since back 2024-03-22T01:48:50.779479+0000 front 2024-03-22T01:48:50.779435+0000 (oldest deadline 2024-03-22T01:49:56.579980+0000) 2024-03-22T01:50:12.172 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:50:11 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:50:11.820+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.98:6804 osd.2 since back 2024-03-22T01:48:56.580387+0000 front 2024-03-22T01:48:56.580350+0000 (oldest deadline 2024-03-22T01:50:01.181087+0000) 2024-03-22T01:50:13.172 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:50:12 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:50:12.770+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-03-22T01:48:33.677632+0000 front 2024-03-22T01:48:33.677592+0000 (oldest deadline 2024-03-22T01:49:39.577549+0000) 2024-03-22T01:50:13.172 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:50:12 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:50:12.770+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.89:6804 osd.1 since back 2024-03-22T01:48:50.779479+0000 front 2024-03-22T01:48:50.779435+0000 (oldest deadline 2024-03-22T01:49:56.579980+0000) 2024-03-22T01:50:13.172 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:50:12 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11[62548]: 2024-03-22T01:50:12.770+0000 7f643f5c6700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.98:6804 osd.2 since back 2024-03-22T01:48:56.580387+0000 front 2024-03-22T01:48:56.580350+0000 (oldest deadline 2024-03-22T01:50:01.181087+0000) 2024-03-22T01:50:13.874 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:50:13 smithi098 podman[74623]: 2024-03-22 01:50:13.562132872 +0000 UTC m=+5.715912675 container died f324c17e1d0ed49a99ad76cf1f11852032856e8abdf242af752378137e2c5f32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-22T01:50:14.141 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:50:13 smithi098 podman[74623]: 2024-03-22 01:50:13.974465077 +0000 UTC m=+6.128244863 container cleanup f324c17e1d0ed49a99ad76cf1f11852032856e8abdf242af752378137e2c5f32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-22T01:50:14.142 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:50:13 smithi098 bash[74623]: ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11 2024-03-22T01:50:14.422 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:50:14 smithi098 podman[74640]: 2024-03-22 01:50:14.142264133 +0000 UTC m=+0.578975865 container remove f324c17e1d0ed49a99ad76cf1f11852032856e8abdf242af752378137e2c5f32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_CLEAN=True, ceph=True) 2024-03-22T01:50:14.836 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:50:14 smithi098 podman[74725]: 2024-03-22 01:50:14.526440983 +0000 UTC m=+0.108903562 container create 6cfc48a1463a53600967dd2bc1077b3776b29fe5f385abdbc215394dc0f7c247 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11-deactivate, ceph=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-22T01:50:14.836 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:50:14 smithi098 podman[74725]: 2024-03-22 01:50:14.435644835 +0000 UTC m=+0.018107422 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T01:50:14.836 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:50:14 smithi098 podman[74725]: 2024-03-22 01:50:14.762547571 +0000 UTC m=+0.345010171 container init 6cfc48a1463a53600967dd2bc1077b3776b29fe5f385abdbc215394dc0f7c247 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-22T01:50:14.836 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:50:14 smithi098 podman[74725]: 2024-03-22 01:50:14.765846626 +0000 UTC m=+0.348309220 container start 6cfc48a1463a53600967dd2bc1077b3776b29fe5f385abdbc215394dc0f7c247 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11-deactivate, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-22T01:50:14.836 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:50:14 smithi098 podman[74725]: 2024-03-22 01:50:14.837229527 +0000 UTC m=+0.419692114 container attach 6cfc48a1463a53600967dd2bc1077b3776b29fe5f385abdbc215394dc0f7c247 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-22T01:50:15.172 INFO:journalctl@ceph.osd.11.smithi098.stdout:Mar 22 01:50:15 smithi098 podman[74725]: 2024-03-22 01:50:15.051212289 +0000 UTC m=+0.633674876 container died 6cfc48a1463a53600967dd2bc1077b3776b29fe5f385abdbc215394dc0f7c247 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-osd-11-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_CLEAN=True) 2024-03-22T01:50:15.632 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@osd.11.service' 2024-03-22T01:50:15.674 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T01:50:15.674 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-03-22T01:50:15.675 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-03-22T01:50:15.675 DEBUG:teuthology.orchestra.run.smithi072:> sudo systemctl stop ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mds.a 2024-03-22T01:50:15.962 INFO:journalctl@ceph.mds.a.smithi072.stdout:Mar 22 01:50:15 smithi072 systemd[1]: Stopping Ceph mds.a for 022a99bc-e7eb-11ee-95cd-87774f69a715... 2024-03-22T01:50:16.235 INFO:journalctl@ceph.mds.a.smithi072.stdout:Mar 22 01:50:15 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-a[71838]: 2024-03-22T01:50:15.962+0000 7ff49c1cc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T01:50:16.236 INFO:journalctl@ceph.mds.a.smithi072.stdout:Mar 22 01:50:15 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-a[71838]: 2024-03-22T01:50:15.962+0000 7ff49c1cc700 -1 mds.a *** got signal Terminated *** 2024-03-22T01:50:18.189 INFO:journalctl@ceph.mds.a.smithi072.stdout:Mar 22 01:50:17 smithi072 podman[97262]: 2024-03-22 01:50:17.921275262 +0000 UTC m=+2.052750485 container died 88b4b3dad62127331e96eda429708ef2847ec7f742f33bf573b8b17b560913a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-22T01:50:18.566 INFO:journalctl@ceph.mds.a.smithi072.stdout:Mar 22 01:50:18 smithi072 podman[97262]: 2024-03-22 01:50:18.29091425 +0000 UTC m=+2.422389475 container cleanup 88b4b3dad62127331e96eda429708ef2847ec7f742f33bf573b8b17b560913a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-a, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-22T01:50:18.567 INFO:journalctl@ceph.mds.a.smithi072.stdout:Mar 22 01:50:18 smithi072 bash[97262]: ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-a 2024-03-22T01:50:18.669 DEBUG:teuthology.orchestra.run.smithi072:> sudo pkill -f 'journalctl -f -n 0 -u ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mds.a.service' 2024-03-22T01:50:18.706 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T01:50:18.707 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-03-22T01:50:18.707 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-03-22T01:50:18.707 DEBUG:teuthology.orchestra.run.smithi072:> sudo systemctl stop ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mds.d 2024-03-22T01:50:18.872 INFO:journalctl@ceph.mds.d.smithi072.stdout:Mar 22 01:50:18 smithi072 systemd[1]: Stopping Ceph mds.d for 022a99bc-e7eb-11ee-95cd-87774f69a715... 2024-03-22T01:50:19.372 INFO:journalctl@ceph.mds.d.smithi072.stdout:Mar 22 01:50:19 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-d[70871]: 2024-03-22T01:50:19.021+0000 7f03c83c9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T01:50:19.372 INFO:journalctl@ceph.mds.d.smithi072.stdout:Mar 22 01:50:19 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-d[70871]: 2024-03-22T01:50:19.021+0000 7f03c83c9700 -1 mds.d *** got signal Terminated *** 2024-03-22T01:50:22.372 INFO:journalctl@ceph.mds.d.smithi072.stdout:Mar 22 01:50:22 smithi072 podman[97396]: 2024-03-22 01:50:22.099788198 +0000 UTC m=+3.172637106 container died bf8e80142e1c689e5edf05bba9698b82c27c6c662c25af2faef2c910b2979380 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-d, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.vendor=CentOS) 2024-03-22T01:50:22.718 INFO:journalctl@ceph.mds.d.smithi072.stdout:Mar 22 01:50:22 smithi072 podman[97396]: 2024-03-22 01:50:22.466728074 +0000 UTC m=+3.539576978 container cleanup bf8e80142e1c689e5edf05bba9698b82c27c6c662c25af2faef2c910b2979380 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-d, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-22T01:50:22.718 INFO:journalctl@ceph.mds.d.smithi072.stdout:Mar 22 01:50:22 smithi072 bash[97396]: ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-d 2024-03-22T01:50:22.850 DEBUG:teuthology.orchestra.run.smithi072:> sudo pkill -f 'journalctl -f -n 0 -u ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mds.d.service' 2024-03-22T01:50:22.886 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T01:50:22.886 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-03-22T01:50:22.887 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-03-22T01:50:22.887 DEBUG:teuthology.orchestra.run.smithi072:> sudo systemctl stop ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mds.g 2024-03-22T01:50:23.122 INFO:journalctl@ceph.mds.g.smithi072.stdout:Mar 22 01:50:22 smithi072 systemd[1]: Stopping Ceph mds.g for 022a99bc-e7eb-11ee-95cd-87774f69a715... 2024-03-22T01:50:23.622 INFO:journalctl@ceph.mds.g.smithi072.stdout:Mar 22 01:50:23 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-g[71477]: 2024-03-22T01:50:23.230+0000 7f1be1c3d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T01:50:23.623 INFO:journalctl@ceph.mds.g.smithi072.stdout:Mar 22 01:50:23 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-g[71477]: 2024-03-22T01:50:23.230+0000 7f1be1c3d700 -1 mds.g *** got signal Terminated *** 2024-03-22T01:50:27.515 INFO:journalctl@ceph.mds.g.smithi072.stdout:Mar 22 01:50:27 smithi072 podman[97516]: 2024-03-22 01:50:27.225476489 +0000 UTC m=+4.118874345 container died d15911c82385e48400369753629635b07c0dcae5f53f9bcd0e828289c984b562 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-g, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-22T01:50:27.860 INFO:journalctl@ceph.mds.g.smithi072.stdout:Mar 22 01:50:27 smithi072 podman[97516]: 2024-03-22 01:50:27.626382951 +0000 UTC m=+4.519780799 container cleanup d15911c82385e48400369753629635b07c0dcae5f53f9bcd0e828289c984b562 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-g, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-22T01:50:27.861 INFO:journalctl@ceph.mds.g.smithi072.stdout:Mar 22 01:50:27 smithi072 bash[97516]: ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-g 2024-03-22T01:50:27.968 DEBUG:teuthology.orchestra.run.smithi072:> sudo pkill -f 'journalctl -f -n 0 -u ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mds.g.service' 2024-03-22T01:50:28.004 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T01:50:28.004 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-03-22T01:50:28.005 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-03-22T01:50:28.005 DEBUG:teuthology.orchestra.run.smithi072:> sudo systemctl stop ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mds.j 2024-03-22T01:50:28.122 INFO:journalctl@ceph.mds.j.smithi072.stdout:Mar 22 01:50:28 smithi072 systemd[1]: Stopping Ceph mds.j for 022a99bc-e7eb-11ee-95cd-87774f69a715... 2024-03-22T01:50:28.622 INFO:journalctl@ceph.mds.j.smithi072.stdout:Mar 22 01:50:28 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-j[70522]: 2024-03-22T01:50:28.322+0000 7f2a98308700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T01:50:28.622 INFO:journalctl@ceph.mds.j.smithi072.stdout:Mar 22 01:50:28 smithi072 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-j[70522]: 2024-03-22T01:50:28.322+0000 7f2a98308700 -1 mds.j *** got signal Terminated *** 2024-03-22T01:50:31.622 INFO:journalctl@ceph.mds.j.smithi072.stdout:Mar 22 01:50:31 smithi072 podman[97637]: 2024-03-22 01:50:31.332146911 +0000 UTC m=+3.109530515 container died 46300af5390d83c9f44f92b394f000ba7a1268cba3c990a9473b5641345c63c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-22T01:50:31.941 INFO:journalctl@ceph.mds.j.smithi072.stdout:Mar 22 01:50:31 smithi072 podman[97637]: 2024-03-22 01:50:31.641632587 +0000 UTC m=+3.419016200 container cleanup 46300af5390d83c9f44f92b394f000ba7a1268cba3c990a9473b5641345c63c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-j, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-22T01:50:31.941 INFO:journalctl@ceph.mds.j.smithi072.stdout:Mar 22 01:50:31 smithi072 bash[97637]: ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-j 2024-03-22T01:50:31.941 INFO:journalctl@ceph.mds.j.smithi072.stdout:Mar 22 01:50:31 smithi072 podman[97651]: 2024-03-22 01:50:31.85358341 +0000 UTC m=+0.517293337 container remove 46300af5390d83c9f44f92b394f000ba7a1268cba3c990a9473b5641345c63c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-j, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-22T01:50:31.960 DEBUG:teuthology.orchestra.run.smithi072:> sudo pkill -f 'journalctl -f -n 0 -u ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mds.j.service' 2024-03-22T01:50:31.994 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T01:50:31.994 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-03-22T01:50:31.994 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-03-22T01:50:31.994 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mds.b 2024-03-22T01:50:32.280 INFO:journalctl@ceph.mds.b.smithi089.stdout:Mar 22 01:50:32 smithi089 systemd[1]: Stopping Ceph mds.b for 022a99bc-e7eb-11ee-95cd-87774f69a715... 2024-03-22T01:50:32.608 INFO:journalctl@ceph.mds.b.smithi089.stdout:Mar 22 01:50:32 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-b[66811]: 2024-03-22T01:50:32.280+0000 7f0e8a9bc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T01:50:32.609 INFO:journalctl@ceph.mds.b.smithi089.stdout:Mar 22 01:50:32 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-b[66811]: 2024-03-22T01:50:32.280+0000 7f0e8a9bc700 -1 mds.b *** got signal Terminated *** 2024-03-22T01:50:33.608 INFO:journalctl@ceph.mds.b.smithi089.stdout:Mar 22 01:50:33 smithi089 podman[73306]: 2024-03-22 01:50:33.317007139 +0000 UTC m=+1.127451476 container died a7ed85c8dcb6a6b159f4fa17f3f7b297477ee90fc20e4c492a4f1ddae8278581 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-22T01:50:33.956 INFO:journalctl@ceph.mds.b.smithi089.stdout:Mar 22 01:50:33 smithi089 podman[73306]: 2024-03-22 01:50:33.69777211 +0000 UTC m=+1.508216465 container cleanup a7ed85c8dcb6a6b159f4fa17f3f7b297477ee90fc20e4c492a4f1ddae8278581 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-22T01:50:33.956 INFO:journalctl@ceph.mds.b.smithi089.stdout:Mar 22 01:50:33 smithi089 bash[73306]: ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-b 2024-03-22T01:50:34.072 DEBUG:teuthology.orchestra.run.smithi089:> sudo pkill -f 'journalctl -f -n 0 -u ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mds.b.service' 2024-03-22T01:50:34.111 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T01:50:34.111 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-03-22T01:50:34.111 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-03-22T01:50:34.111 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mds.e 2024-03-22T01:50:34.358 INFO:journalctl@ceph.mds.e.smithi089.stdout:Mar 22 01:50:34 smithi089 systemd[1]: Stopping Ceph mds.e for 022a99bc-e7eb-11ee-95cd-87774f69a715... 2024-03-22T01:50:34.858 INFO:journalctl@ceph.mds.e.smithi089.stdout:Mar 22 01:50:34 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-e[66183]: 2024-03-22T01:50:34.442+0000 7f718ec41700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T01:50:34.858 INFO:journalctl@ceph.mds.e.smithi089.stdout:Mar 22 01:50:34 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-e[66183]: 2024-03-22T01:50:34.443+0000 7f718ec41700 -1 mds.e *** got signal Terminated *** 2024-03-22T01:50:37.330 INFO:journalctl@ceph.mds.e.smithi089.stdout:Mar 22 01:50:37 smithi089 podman[73439]: 2024-03-22 01:50:37.055648352 +0000 UTC m=+2.733632984 container died 1dd0843af24de83ee38691d02b232f28b1f4481a442127c7bc4621e3a25e0cd4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-e, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-22T01:50:37.608 INFO:journalctl@ceph.mds.e.smithi089.stdout:Mar 22 01:50:37 smithi089 podman[73439]: 2024-03-22 01:50:37.434722472 +0000 UTC m=+3.112707101 container cleanup 1dd0843af24de83ee38691d02b232f28b1f4481a442127c7bc4621e3a25e0cd4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-e, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, ceph=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-22T01:50:37.608 INFO:journalctl@ceph.mds.e.smithi089.stdout:Mar 22 01:50:37 smithi089 bash[73439]: ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-e 2024-03-22T01:50:38.004 DEBUG:teuthology.orchestra.run.smithi089:> sudo pkill -f 'journalctl -f -n 0 -u ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mds.e.service' 2024-03-22T01:50:38.040 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T01:50:38.040 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-03-22T01:50:38.040 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-03-22T01:50:38.040 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mds.h 2024-03-22T01:50:38.354 INFO:journalctl@ceph.mds.h.smithi089.stdout:Mar 22 01:50:38 smithi089 systemd[1]: Stopping Ceph mds.h for 022a99bc-e7eb-11ee-95cd-87774f69a715... 2024-03-22T01:50:38.608 INFO:journalctl@ceph.mds.h.smithi089.stdout:Mar 22 01:50:38 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-h[66505]: 2024-03-22T01:50:38.355+0000 7fe993539700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T01:50:38.608 INFO:journalctl@ceph.mds.h.smithi089.stdout:Mar 22 01:50:38 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-h[66505]: 2024-03-22T01:50:38.355+0000 7fe993539700 -1 mds.h *** got signal Terminated *** 2024-03-22T01:50:41.858 INFO:journalctl@ceph.mds.h.smithi089.stdout:Mar 22 01:50:41 smithi089 podman[73561]: 2024-03-22 01:50:41.58293794 +0000 UTC m=+3.331098979 container died 608794ace6ce8cb33d0b3d112f490c6d148e6c6677cc873f1ce596f34ba3319a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-h, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-22T01:50:42.171 INFO:journalctl@ceph.mds.h.smithi089.stdout:Mar 22 01:50:41 smithi089 podman[73561]: 2024-03-22 01:50:41.963572276 +0000 UTC m=+3.711733316 container cleanup 608794ace6ce8cb33d0b3d112f490c6d148e6c6677cc873f1ce596f34ba3319a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-h, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-22T01:50:42.172 INFO:journalctl@ceph.mds.h.smithi089.stdout:Mar 22 01:50:41 smithi089 bash[73561]: ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-h 2024-03-22T01:50:42.377 DEBUG:teuthology.orchestra.run.smithi089:> sudo pkill -f 'journalctl -f -n 0 -u ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mds.h.service' 2024-03-22T01:50:42.411 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T01:50:42.412 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-03-22T01:50:42.412 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-03-22T01:50:42.412 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mds.k 2024-03-22T01:50:42.858 INFO:journalctl@ceph.mds.k.smithi089.stdout:Mar 22 01:50:42 smithi089 systemd[1]: Stopping Ceph mds.k for 022a99bc-e7eb-11ee-95cd-87774f69a715... 2024-03-22T01:50:42.858 INFO:journalctl@ceph.mds.k.smithi089.stdout:Mar 22 01:50:42 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-k[65863]: 2024-03-22T01:50:42.724+0000 7f00778e8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T01:50:42.858 INFO:journalctl@ceph.mds.k.smithi089.stdout:Mar 22 01:50:42 smithi089 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-k[65863]: 2024-03-22T01:50:42.724+0000 7f00778e8700 -1 mds.k *** got signal Terminated *** 2024-03-22T01:50:44.858 INFO:journalctl@ceph.mds.k.smithi089.stdout:Mar 22 01:50:44 smithi089 podman[73682]: 2024-03-22 01:50:44.59980709 +0000 UTC m=+1.971059405 container died de4cb18e4d6407484e1ef0e2e27bbc47b8dacbac71dfc800b2ac7d510a3b2fb5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-k, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-22T01:50:45.116 INFO:journalctl@ceph.mds.k.smithi089.stdout:Mar 22 01:50:44 smithi089 podman[73682]: 2024-03-22 01:50:44.941971974 +0000 UTC m=+2.313224275 container cleanup de4cb18e4d6407484e1ef0e2e27bbc47b8dacbac71dfc800b2ac7d510a3b2fb5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-k, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240319) 2024-03-22T01:50:45.116 INFO:journalctl@ceph.mds.k.smithi089.stdout:Mar 22 01:50:44 smithi089 bash[73682]: ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-k 2024-03-22T01:50:45.351 DEBUG:teuthology.orchestra.run.smithi089:> sudo pkill -f 'journalctl -f -n 0 -u ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mds.k.service' 2024-03-22T01:50:45.376 INFO:journalctl@ceph.mds.k.smithi089.stdout:Mar 22 01:50:45 smithi089 podman[73696]: 2024-03-22 01:50:45.118070286 +0000 UTC m=+0.509802094 container remove de4cb18e4d6407484e1ef0e2e27bbc47b8dacbac71dfc800b2ac7d510a3b2fb5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-k, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0) 2024-03-22T01:50:45.376 INFO:journalctl@ceph.mds.k.smithi089.stdout:Mar 22 01:50:45 smithi089 systemd[1]: ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mds.k.service: Deactivated successfully. 2024-03-22T01:50:45.376 INFO:journalctl@ceph.mds.k.smithi089.stdout:Mar 22 01:50:45 smithi089 systemd[1]: Stopped Ceph mds.k for 022a99bc-e7eb-11ee-95cd-87774f69a715. 2024-03-22T01:50:45.387 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T01:50:45.387 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-03-22T01:50:45.387 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-03-22T01:50:45.387 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mds.c 2024-03-22T01:50:45.667 INFO:journalctl@ceph.mds.c.smithi098.stdout:Mar 22 01:50:45 smithi098 systemd[1]: Stopping Ceph mds.c for 022a99bc-e7eb-11ee-95cd-87774f69a715... 2024-03-22T01:50:45.922 INFO:journalctl@ceph.mds.c.smithi098.stdout:Mar 22 01:50:45 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-c[68007]: 2024-03-22T01:50:45.673+0000 7f0972d02700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T01:50:45.922 INFO:journalctl@ceph.mds.c.smithi098.stdout:Mar 22 01:50:45 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-c[68007]: 2024-03-22T01:50:45.673+0000 7f0972d02700 -1 mds.c *** got signal Terminated *** 2024-03-22T01:50:46.922 INFO:journalctl@ceph.mds.c.smithi098.stdout:Mar 22 01:50:46 smithi098 podman[74830]: 2024-03-22 01:50:46.669783136 +0000 UTC m=+1.090779157 container died df75b211444a6295b2516116e51af97874cf8e763d1577df437b20c4dbf94012 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-c, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-22T01:50:47.191 INFO:journalctl@ceph.mds.c.smithi098.stdout:Mar 22 01:50:47 smithi098 podman[74830]: 2024-03-22 01:50:47.007754722 +0000 UTC m=+1.428750733 container cleanup df75b211444a6295b2516116e51af97874cf8e763d1577df437b20c4dbf94012 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-22T01:50:47.191 INFO:journalctl@ceph.mds.c.smithi098.stdout:Mar 22 01:50:47 smithi098 bash[74830]: ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-c 2024-03-22T01:50:47.411 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mds.c.service' 2024-03-22T01:50:47.451 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T01:50:47.451 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-03-22T01:50:47.451 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-03-22T01:50:47.451 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mds.f 2024-03-22T01:50:47.769 INFO:journalctl@ceph.mds.f.smithi098.stdout:Mar 22 01:50:47 smithi098 systemd[1]: Stopping Ceph mds.f for 022a99bc-e7eb-11ee-95cd-87774f69a715... 2024-03-22T01:50:48.172 INFO:journalctl@ceph.mds.f.smithi098.stdout:Mar 22 01:50:47 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-f[67699]: 2024-03-22T01:50:47.769+0000 7f8b8bbc9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T01:50:48.172 INFO:journalctl@ceph.mds.f.smithi098.stdout:Mar 22 01:50:47 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-f[67699]: 2024-03-22T01:50:47.769+0000 7f8b8bbc9700 -1 mds.f *** got signal Terminated *** 2024-03-22T01:50:52.111 INFO:journalctl@ceph.mds.f.smithi098.stdout:Mar 22 01:50:51 smithi098 podman[74962]: 2024-03-22 01:50:51.779627965 +0000 UTC m=+4.116135896 container died 882fad7c69f457aa75d4d208150d69b9e03d721d8948e63079f7ab89f860027b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-f, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-22T01:50:52.403 INFO:journalctl@ceph.mds.f.smithi098.stdout:Mar 22 01:50:52 smithi098 podman[74962]: 2024-03-22 01:50:52.211867892 +0000 UTC m=+4.548375812 container cleanup 882fad7c69f457aa75d4d208150d69b9e03d721d8948e63079f7ab89f860027b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-22T01:50:52.403 INFO:journalctl@ceph.mds.f.smithi098.stdout:Mar 22 01:50:52 smithi098 bash[74962]: ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-f 2024-03-22T01:50:52.627 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mds.f.service' 2024-03-22T01:50:52.662 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T01:50:52.662 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-03-22T01:50:52.663 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-03-22T01:50:52.663 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mds.i 2024-03-22T01:50:52.922 INFO:journalctl@ceph.mds.i.smithi098.stdout:Mar 22 01:50:52 smithi098 systemd[1]: Stopping Ceph mds.i for 022a99bc-e7eb-11ee-95cd-87774f69a715... 2024-03-22T01:50:53.422 INFO:journalctl@ceph.mds.i.smithi098.stdout:Mar 22 01:50:52 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-i[68331]: 2024-03-22T01:50:52.981+0000 7fb13e5ed700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T01:50:53.422 INFO:journalctl@ceph.mds.i.smithi098.stdout:Mar 22 01:50:52 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-i[68331]: 2024-03-22T01:50:52.981+0000 7fb13e5ed700 -1 mds.i *** got signal Terminated *** 2024-03-22T01:50:57.172 INFO:journalctl@ceph.mds.i.smithi098.stdout:Mar 22 01:50:56 smithi098 podman[75084]: 2024-03-22 01:50:56.994955664 +0000 UTC m=+4.120082852 container died 9ad857dd4375eb65cb9ab97e9e54c9d3028bbe3ea87193faf157ab633cdf5574 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-i, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS) 2024-03-22T01:50:57.507 INFO:journalctl@ceph.mds.i.smithi098.stdout:Mar 22 01:50:57 smithi098 podman[75084]: 2024-03-22 01:50:57.331790083 +0000 UTC m=+4.456917246 container cleanup 9ad857dd4375eb65cb9ab97e9e54c9d3028bbe3ea87193faf157ab633cdf5574 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-i, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0) 2024-03-22T01:50:57.507 INFO:journalctl@ceph.mds.i.smithi098.stdout:Mar 22 01:50:57 smithi098 bash[75084]: ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-i 2024-03-22T01:50:57.736 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mds.i.service' 2024-03-22T01:50:57.760 INFO:journalctl@ceph.mds.i.smithi098.stdout:Mar 22 01:50:57 smithi098 podman[75096]: 2024-03-22 01:50:57.508294119 +0000 UTC m=+0.516258127 container remove 9ad857dd4375eb65cb9ab97e9e54c9d3028bbe3ea87193faf157ab633cdf5574 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-i, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240319) 2024-03-22T01:50:57.760 INFO:journalctl@ceph.mds.i.smithi098.stdout:Mar 22 01:50:57 smithi098 systemd[1]: ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mds.i.service: Deactivated successfully. 2024-03-22T01:50:57.760 INFO:journalctl@ceph.mds.i.smithi098.stdout:Mar 22 01:50:57 smithi098 systemd[1]: Stopped Ceph mds.i for 022a99bc-e7eb-11ee-95cd-87774f69a715. 2024-03-22T01:50:57.760 INFO:journalctl@ceph.mds.i.smithi098.stdout:Mar 22 01:50:57 smithi098 systemd[1]: ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mds.i.service: Consumed 4.871s CPU time. 2024-03-22T01:50:57.772 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T01:50:57.772 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-03-22T01:50:57.772 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-03-22T01:50:57.772 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mds.l 2024-03-22T01:50:58.101 INFO:journalctl@ceph.mds.l.smithi098.stdout:Mar 22 01:50:57 smithi098 systemd[1]: Stopping Ceph mds.l for 022a99bc-e7eb-11ee-95cd-87774f69a715... 2024-03-22T01:50:58.421 INFO:journalctl@ceph.mds.l.smithi098.stdout:Mar 22 01:50:58 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-l[67387]: 2024-03-22T01:50:58.102+0000 7fe90d15f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T01:50:58.422 INFO:journalctl@ceph.mds.l.smithi098.stdout:Mar 22 01:50:58 smithi098 ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-l[67387]: 2024-03-22T01:50:58.102+0000 7fe90d15f700 -1 mds.l *** got signal Terminated *** 2024-03-22T01:51:01.956 INFO:journalctl@ceph.mds.l.smithi098.stdout:Mar 22 01:51:01 smithi098 podman[75204]: 2024-03-22 01:51:01.682292908 +0000 UTC m=+3.692109597 container died f5b0a765eaa3ada11ad05731813a5a84b01d60ad12699c16f0403bd50f1d2deb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-l, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-22T01:51:02.422 INFO:journalctl@ceph.mds.l.smithi098.stdout:Mar 22 01:51:02 smithi098 podman[75204]: 2024-03-22 01:51:02.044175039 +0000 UTC m=+4.053991727 container cleanup f5b0a765eaa3ada11ad05731813a5a84b01d60ad12699c16f0403bd50f1d2deb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-22T01:51:02.422 INFO:journalctl@ceph.mds.l.smithi098.stdout:Mar 22 01:51:02 smithi098 bash[75204]: ceph-022a99bc-e7eb-11ee-95cd-87774f69a715-mds-l 2024-03-22T01:51:02.612 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-022a99bc-e7eb-11ee-95cd-87774f69a715@mds.l.service' 2024-03-22T01:51:02.647 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T01:51:02.647 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-03-22T01:51:02.648 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-22T01:51:02.838 INFO:teuthology.orchestra.run.smithi072.stdout:Deleting cluster with fsid: 022a99bc-e7eb-11ee-95cd-87774f69a715 2024-03-22T01:51:06.073 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-22T01:51:06.263 INFO:teuthology.orchestra.run.smithi089.stdout:Deleting cluster with fsid: 022a99bc-e7eb-11ee-95cd-87774f69a715 2024-03-22T01:51:08.850 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-22T01:51:09.040 INFO:teuthology.orchestra.run.smithi098.stdout:Deleting cluster with fsid: 022a99bc-e7eb-11ee-95cd-87774f69a715 2024-03-22T01:51:11.844 DEBUG:teuthology.orchestra.run.smithi072:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-22T01:51:11.880 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-22T01:51:11.916 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-22T01:51:11.944 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-22T01:51:11.948 DEBUG:teuthology.misc:Transferring archived files from smithi072:/var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-22_01:06:30-fs-wip-batrick-testing-20240321.172515-distro-default-smithi/7615721/remote/smithi072/crash 2024-03-22T01:51:11.949 DEBUG:teuthology.orchestra.run.smithi072:> sudo tar c -f - -C /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/crash -- . 2024-03-22T01:51:11.999 INFO:teuthology.orchestra.run.smithi072.stderr:tar: /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-22T01:51:11.999 INFO:teuthology.orchestra.run.smithi072.stderr:tar: Error is not recoverable: exiting now 2024-03-22T01:51:12.002 DEBUG:teuthology.misc:Transferring archived files from smithi089:/var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-22_01:06:30-fs-wip-batrick-testing-20240321.172515-distro-default-smithi/7615721/remote/smithi089/crash 2024-03-22T01:51:12.006 DEBUG:teuthology.orchestra.run.smithi089:> sudo tar c -f - -C /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/crash -- . 2024-03-22T01:51:12.050 INFO:teuthology.orchestra.run.smithi089.stderr:tar: /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-22T01:51:12.050 INFO:teuthology.orchestra.run.smithi089.stderr:tar: Error is not recoverable: exiting now 2024-03-22T01:51:12.052 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-22_01:06:30-fs-wip-batrick-testing-20240321.172515-distro-default-smithi/7615721/remote/smithi098/crash 2024-03-22T01:51:12.053 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/crash -- . 2024-03-22T01:51:12.088 INFO:teuthology.orchestra.run.smithi098.stderr:tar: /var/lib/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-22T01:51:12.088 INFO:teuthology.orchestra.run.smithi098.stderr:tar: Error is not recoverable: exiting now 2024-03-22T01:51:12.089 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-22T01:51:12.089 DEBUG:teuthology.orchestra.run.smithi072:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-22T01:51:12.188 INFO:tasks.cephadm:Compressing logs... 2024-03-22T01:51:12.188 DEBUG:teuthology.orchestra.run.smithi072:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-22T01:51:12.191 DEBUG:teuthology.orchestra.run.smithi089:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-22T01:51:12.193 DEBUG:teuthology.orchestra.run.smithi098:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-22T01:51:12.229 INFO:teuthology.orchestra.run.smithi098.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-22T01:51:12.229 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-22T01:51:12.229 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-mds.l.log 2024-03-22T01:51:12.229 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-22T01:51:12.231 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-mds.l.log: gzip -5 --verbose -- /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-mds.i.log 2024-03-22T01:51:12.231 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-osd.11.log 2024-03-22T01:51:12.231 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-mgr.z.log 2024-03-22T01:51:12.231 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-osd.8.log 2024-03-22T01:51:12.233 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-osd.5.log 2024-03-22T01:51:12.233 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-mds.f.log 2024-03-22T01:51:12.233 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-22T01:51:12.233 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph.log 2024-03-22T01:51:12.234 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-osd.2.log 2024-03-22T01:51:12.234 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-mds.c.log 2024-03-22T01:51:12.234 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph.log: /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-22T01:51:12.234 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-mon.c.log 2024-03-22T01:51:12.234 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-mds.c.log: /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-volume.log: /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-mon.c.log: 94.5% -- replaced with /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-mds.l.log.gz 2024-03-22T01:51:12.235 INFO:teuthology.orchestra.run.smithi089.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-22T01:51:12.235 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-22T01:51:12.235 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-osd.10.log 2024-03-22T01:51:12.235 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-22T01:51:12.235 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-mds.b.log 2024-03-22T01:51:12.236 INFO:teuthology.orchestra.run.smithi098.stderr: 94.4% -- replaced with /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-mds.f.log.gz 2024-03-22T01:51:12.239 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-osd.10.log: /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-osd.4.log 2024-03-22T01:51:12.239 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-osd.1.log 2024-03-22T01:51:12.239 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-mds.k.log 2024-03-22T01:51:12.239 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-mds.h.log 2024-03-22T01:51:12.239 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-osd.7.log 2024-03-22T01:51:12.239 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-mds.e.log 2024-03-22T01:51:12.239 INFO:teuthology.orchestra.run.smithi089.stderr:gzip/var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-osd.7.log: -5 --verbose -- /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-mgr.y.log 2024-03-22T01:51:12.239 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-22T01:51:12.239 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph.log 2024-03-22T01:51:12.240 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-22T01:51:12.240 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-mon.b.log 2024-03-22T01:51:12.240 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-volume.log: /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-mon.b.log: 94.6% -- replaced with /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-mds.k.log.gz 2024-03-22T01:51:12.240 INFO:teuthology.orchestra.run.smithi089.stderr: 94.7% -- replaced with /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-mds.h.log.gz 2024-03-22T01:51:12.240 INFO:teuthology.orchestra.run.smithi089.stderr: 94.6% -- replaced with /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-mds.e.log.gz 2024-03-22T01:51:12.240 INFO:teuthology.orchestra.run.smithi098.stderr: 94.5% -- replaced with /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-mds.c.log.gz 2024-03-22T01:51:12.240 INFO:teuthology.orchestra.run.smithi098.stderr: 92.1% -- replaced with /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-mgr.z.log.gz 2024-03-22T01:51:12.242 INFO:teuthology.orchestra.run.smithi072.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose: No such file or directory -- 2024-03-22T01:51:12.242 INFO:teuthology.orchestra.run.smithi072.stderr: /var/log/ceph/cephadm.log 2024-03-22T01:51:12.242 INFO:teuthology.orchestra.run.smithi089.stderr: 92.0% -- replaced with /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-mgr.y.log.gz 2024-03-22T01:51:12.252 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-osd.6.log 2024-03-22T01:51:12.252 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-mon.a.log 2024-03-22T01:51:12.252 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-22T01:51:12.252 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-mds.g.log 2024-03-22T01:51:12.252 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-mds.j.log 2024-03-22T01:51:12.252 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-osd.3.log 2024-03-22T01:51:12.252 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-mgr.x.log 2024-03-22T01:51:12.252 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-osd.0.log 2024-03-22T01:51:12.252 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-22T01:51:12.252 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph.log 2024-03-22T01:51:12.253 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-22T01:51:12.253 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-osd.9.log 2024-03-22T01:51:12.253 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-mds.d.log 2024-03-22T01:51:12.253 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-mds.a.log 2024-03-22T01:51:12.253 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-03-22T01:51:12.253 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-mds.a.log: /var/log/ceph/ceph-client.admin.log: 87.5% -- replaced with /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-22T01:51:12.288 INFO:teuthology.orchestra.run.smithi098.stderr: 83.7% -- replaced with /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-22T01:51:12.324 INFO:teuthology.orchestra.run.smithi098.stderr: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-22T01:51:12.353 INFO:teuthology.orchestra.run.smithi098.stderr: 90.4% -- replaced with /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-22T01:51:12.369 INFO:teuthology.orchestra.run.smithi098.stderr: 90.9% -- replaced with /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-22T01:51:12.404 INFO:teuthology.orchestra.run.smithi098.stderr: 87.6% -- replaced with /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-22T01:51:12.419 INFO:teuthology.orchestra.run.smithi072.stderr: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-03-22T01:51:12.426 INFO:teuthology.orchestra.run.smithi072.stderr: 84.1% -- replaced with /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-22T01:51:12.447 INFO:teuthology.orchestra.run.smithi089.stderr: 91.0% -- replaced with /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-22T01:51:12.474 INFO:teuthology.orchestra.run.smithi089.stderr: 83.7% -- replaced with /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-22T01:51:12.475 INFO:teuthology.orchestra.run.smithi089.stderr: 90.4% -- replaced with /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-22T01:51:12.498 INFO:teuthology.orchestra.run.smithi089.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-22T01:51:12.499 INFO:teuthology.orchestra.run.smithi089.stderr: 87.6% -- replaced with /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-22T01:51:12.572 INFO:teuthology.orchestra.run.smithi072.stderr: 94.6% -- replaced with /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-mds.g.log.gz 2024-03-22T01:51:12.619 INFO:teuthology.orchestra.run.smithi072.stderr: 90.3% -- replaced with /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-22T01:51:12.679 INFO:teuthology.orchestra.run.smithi072.stderr: 94.6% -- replaced with /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-mds.a.log.gz 2024-03-22T01:51:12.748 INFO:teuthology.orchestra.run.smithi072.stderr: 94.7% -- replaced with /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-mds.j.log.gz 2024-03-22T01:51:12.756 INFO:teuthology.orchestra.run.smithi072.stderr: 94.7% -- replaced with /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-mds.d.log.gz 2024-03-22T01:51:12.960 INFO:teuthology.orchestra.run.smithi072.stderr: 93.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-22T01:51:13.104 INFO:teuthology.orchestra.run.smithi072.stderr: 91.0% -- replaced with /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-22T01:51:13.641 INFO:teuthology.orchestra.run.smithi098.stderr: 91.9% -- replaced with /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-mon.c.log.gz 2024-03-22T01:51:14.449 INFO:teuthology.orchestra.run.smithi089.stderr: 91.9% -- replaced with /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-mon.b.log.gz 2024-03-22T01:51:15.230 INFO:teuthology.orchestra.run.smithi098.stderr: 96.4% -- replaced with /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-mds.i.log.gz 2024-03-22T01:51:16.478 INFO:teuthology.orchestra.run.smithi072.stderr: 90.1% -- replaced with /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-mgr.x.log.gz 2024-03-22T01:51:19.240 INFO:teuthology.orchestra.run.smithi072.stderr: 90.4% -- replaced with /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-mon.a.log.gz 2024-03-22T01:51:45.394 INFO:teuthology.orchestra.run.smithi089.stderr: 95.1% -- replaced with /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-mds.b.log.gz 2024-03-22T01:54:08.508 INFO:teuthology.orchestra.run.smithi089.stderr: 93.0% -- replaced with /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-osd.10.log.gz 2024-03-22T01:54:46.293 INFO:teuthology.orchestra.run.smithi098.stderr: 92.9% -- replaced with /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-osd.8.log.gz 2024-03-22T01:55:03.823 INFO:teuthology.orchestra.run.smithi089.stderr: 92.7% -- replaced with /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-osd.1.log.gz 2024-03-22T01:55:27.549 INFO:teuthology.orchestra.run.smithi098.stderr: 92.6% -- replaced with /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-osd.11.log.gz 2024-03-22T01:55:37.898 INFO:teuthology.orchestra.run.smithi098.stderr: 92.4% -- replaced with /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-osd.2.log.gz 2024-03-22T01:55:44.810 INFO:teuthology.orchestra.run.smithi089.stderr: 92.6% -- replaced with /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-osd.7.log.gz 2024-03-22T01:55:56.474 INFO:teuthology.orchestra.run.smithi089.stderr: 92.0% -- replaced with /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-osd.4.log.gz 2024-03-22T01:55:56.529 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-22T01:55:56.529 INFO:teuthology.orchestra.run.smithi089.stderr:real 4m44.301s 2024-03-22T01:55:56.529 INFO:teuthology.orchestra.run.smithi089.stderr:user 4m48.957s 2024-03-22T01:55:56.529 INFO:teuthology.orchestra.run.smithi089.stderr:sys 0m10.408s 2024-03-22T01:56:17.336 INFO:teuthology.orchestra.run.smithi098.stderr: 92.4% -- replaced with /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-osd.5.log.gz 2024-03-22T01:56:17.458 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-22T01:56:17.458 INFO:teuthology.orchestra.run.smithi098.stderr:real 5m5.242s 2024-03-22T01:56:17.458 INFO:teuthology.orchestra.run.smithi098.stderr:user 4m48.460s 2024-03-22T01:56:17.458 INFO:teuthology.orchestra.run.smithi098.stderr:sys 0m10.246s 2024-03-22T01:58:31.163 INFO:teuthology.orchestra.run.smithi072.stderr: 92.7% -- replaced with /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-osd.3.log.gz 2024-03-22T02:00:05.024 INFO:teuthology.orchestra.run.smithi072.stderr: 91.8% -- replaced with /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-osd.6.log.gz 2024-03-22T02:01:51.293 INFO:teuthology.orchestra.run.smithi072.stderr: 92.5% -- replaced with /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-osd.0.log.gz 2024-03-22T02:02:34.936 INFO:teuthology.orchestra.run.smithi072.stderr: 90.9% -- replaced with /var/log/ceph/022a99bc-e7eb-11ee-95cd-87774f69a715/ceph-osd.9.log.gz 2024-03-22T02:02:34.938 INFO:teuthology.orchestra.run.smithi072.stderr: 2024-03-22T02:02:34.938 INFO:teuthology.orchestra.run.smithi072.stderr:real 11m22.734s 2024-03-22T02:02:34.938 INFO:teuthology.orchestra.run.smithi072.stderr:user 6m10.233s 2024-03-22T02:02:34.938 INFO:teuthology.orchestra.run.smithi072.stderr:sys 0m14.691s 2024-03-22T02:02:34.939 INFO:tasks.cephadm:Archiving logs... 2024-03-22T02:02:34.940 DEBUG:teuthology.misc:Transferring archived files from smithi072:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-22_01:06:30-fs-wip-batrick-testing-20240321.172515-distro-default-smithi/7615721/remote/smithi072/log 2024-03-22T02:02:34.941 DEBUG:teuthology.orchestra.run.smithi072:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-22T02:03:36.291 DEBUG:teuthology.misc:Transferring archived files from smithi089:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-22_01:06:30-fs-wip-batrick-testing-20240321.172515-distro-default-smithi/7615721/remote/smithi089/log 2024-03-22T02:03:36.292 DEBUG:teuthology.orchestra.run.smithi089:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-22T02:04:23.230 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-22_01:06:30-fs-wip-batrick-testing-20240321.172515-distro-default-smithi/7615721/remote/smithi098/log 2024-03-22T02:04:23.231 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-22T02:05:09.875 INFO:tasks.cephadm:Removing cluster... 2024-03-22T02:05:09.876 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 --force 2024-03-22T02:05:10.087 INFO:teuthology.orchestra.run.smithi072.stdout:Deleting cluster with fsid: 022a99bc-e7eb-11ee-95cd-87774f69a715 2024-03-22T02:05:10.775 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 --force 2024-03-22T02:05:11.073 INFO:teuthology.orchestra.run.smithi089.stdout:Deleting cluster with fsid: 022a99bc-e7eb-11ee-95cd-87774f69a715 2024-03-22T02:05:11.676 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 022a99bc-e7eb-11ee-95cd-87774f69a715 --force 2024-03-22T02:05:12.190 INFO:teuthology.orchestra.run.smithi098.stdout:Deleting cluster with fsid: 022a99bc-e7eb-11ee-95cd-87774f69a715 2024-03-22T02:05:12.772 INFO:tasks.cephadm:Removing cephadm ... 2024-03-22T02:05:12.772 DEBUG:teuthology.orchestra.run.smithi072:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-22T02:05:12.790 DEBUG:teuthology.orchestra.run.smithi089:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-22T02:05:12.807 DEBUG:teuthology.orchestra.run.smithi098:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-22T02:05:12.822 INFO:tasks.cephadm:Teardown complete 2024-03-22T02:05:12.823 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-22T02:05:12.834 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-22T02:05:12.834 DEBUG:teuthology.orchestra.run.smithi072:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-22T02:05:12.837 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-22T02:05:12.849 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-22T02:05:12.978 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-22T02:05:12.989 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-22T02:05:12.989 DEBUG:teuthology.orchestra.run.smithi072:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-22T02:05:12.992 DEBUG:teuthology.orchestra.run.smithi089:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-22T02:05:12.994 DEBUG:teuthology.orchestra.run.smithi098:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-22T02:05:13.006 INFO:teuthology.orchestra.run.smithi089.stderr:bash: line 1: ntpq: command not found 2024-03-22T02:05:13.008 INFO:teuthology.orchestra.run.smithi098.stderr:bash: line 1: ntpq: command not found 2024-03-22T02:05:13.013 INFO:teuthology.orchestra.run.smithi072.stderr:bash: line 1: ntpq: command not found 2024-03-22T02:05:13.041 INFO:teuthology.orchestra.run.smithi072.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-22T02:05:13.042 INFO:teuthology.orchestra.run.smithi072.stdout:=============================================================================== 2024-03-22T02:05:13.042 INFO:teuthology.orchestra.run.smithi072.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 11 +315us[ +315us] +/- 88ms 2024-03-22T02:05:13.042 INFO:teuthology.orchestra.run.smithi072.stdout:^* hv02.front.sepia.ceph.com 3 6 377 11 -633us[ -675us] +/- 49ms 2024-03-22T02:05:13.042 INFO:teuthology.orchestra.run.smithi072.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 12 +749us[ +706us] +/- 93ms 2024-03-22T02:05:13.042 INFO:teuthology.orchestra.run.smithi072.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-22T02:05:13.064 INFO:teuthology.orchestra.run.smithi098.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-22T02:05:13.065 INFO:teuthology.orchestra.run.smithi098.stdout:=============================================================================== 2024-03-22T02:05:13.065 INFO:teuthology.orchestra.run.smithi098.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 13 +353us[ +353us] +/- 88ms 2024-03-22T02:05:13.065 INFO:teuthology.orchestra.run.smithi098.stdout:^* hv02.front.sepia.ceph.com 3 6 377 14 -602us[ -627us] +/- 49ms 2024-03-22T02:05:13.065 INFO:teuthology.orchestra.run.smithi098.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 13 +779us[ +779us] +/- 93ms 2024-03-22T02:05:13.065 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-22T02:05:13.082 INFO:teuthology.orchestra.run.smithi089.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-22T02:05:13.083 INFO:teuthology.orchestra.run.smithi089.stdout:=============================================================================== 2024-03-22T02:05:13.083 INFO:teuthology.orchestra.run.smithi089.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 9 +347us[ +347us] +/- 89ms 2024-03-22T02:05:13.083 INFO:teuthology.orchestra.run.smithi089.stdout:^* hv02.front.sepia.ceph.com 3 6 377 12 -612us[ -695us] +/- 49ms 2024-03-22T02:05:13.083 INFO:teuthology.orchestra.run.smithi089.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 8 +777us[ +777us] +/- 94ms 2024-03-22T02:05:13.083 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-22T02:05:13.091 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-22T02:05:13.102 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-22T02:05:13.102 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-22T02:05:13.113 DEBUG:teuthology.orchestra.run.smithi072:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-22T02:05:13.323 DEBUG:teuthology.orchestra.run.smithi089:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-22T02:05:13.414 DEBUG:teuthology.orchestra.run.smithi098:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-22T02:05:13.512 DEBUG:teuthology.orchestra.run.smithi072:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-22T02:05:13.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-22T02:05:13.547 DEBUG:teuthology.orchestra.run.smithi089:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-22T02:05:13.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-22T02:05:13.579 DEBUG:teuthology.orchestra.run.smithi098:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-22T02:05:13.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-22T02:05:13.609 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-22T02:05:13.621 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-22T02:05:13.935 INFO:teuthology.task.internal:Duration was 2832.562750 seconds 2024-03-22T02:05:13.935 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-22T02:05:14.240 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-22T02:05:14.240 DEBUG:teuthology.orchestra.run.smithi072:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-22T02:05:14.244 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-22T02:05:14.247 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-22T02:05:14.319 INFO:teuthology.orchestra.run.smithi098.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-22T02:05:14.333 INFO:teuthology.orchestra.run.smithi072.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-22T02:05:14.334 INFO:teuthology.orchestra.run.smithi089.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-22T02:05:14.739 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-22T02:05:14.739 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi072.front.sepia.ceph.com 2024-03-22T02:05:14.739 DEBUG:teuthology.orchestra.run.smithi072:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-22T02:05:14.768 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi089.front.sepia.ceph.com 2024-03-22T02:05:14.768 DEBUG:teuthology.orchestra.run.smithi089:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-22T02:05:14.801 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi098.front.sepia.ceph.com 2024-03-22T02:05:14.801 DEBUG:teuthology.orchestra.run.smithi098:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-22T02:05:14.833 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-22T02:05:14.833 DEBUG:teuthology.orchestra.run.smithi072:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-22T02:05:14.836 DEBUG:teuthology.orchestra.run.smithi089:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-22T02:05:14.843 DEBUG:teuthology.orchestra.run.smithi098:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-22T02:05:14.905 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-22T02:05:14.905 DEBUG:teuthology.orchestra.run.smithi072:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-22T02:05:14.908 DEBUG:teuthology.orchestra.run.smithi089:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-22T02:05:14.917 DEBUG:teuthology.orchestra.run.smithi098:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-22T02:05:15.291 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-22T02:05:15.304 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-22T02:05:15.304 DEBUG:teuthology.orchestra.run.smithi072:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-22T02:05:15.332 DEBUG:teuthology.orchestra.run.smithi089:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-22T02:05:15.361 DEBUG:teuthology.orchestra.run.smithi098:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-22T02:05:15.389 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-22T02:05:15.401 DEBUG:teuthology.orchestra.run.smithi072:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-22T02:05:15.404 DEBUG:teuthology.orchestra.run.smithi089:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-22T02:05:15.406 DEBUG:teuthology.orchestra.run.smithi098:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-22T02:05:15.435 INFO:teuthology.orchestra.run.smithi072.stdout:kernel.core_pattern = core 2024-03-22T02:05:15.443 INFO:teuthology.orchestra.run.smithi089.stdout:kernel.core_pattern = core 2024-03-22T02:05:15.469 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = core 2024-03-22T02:05:15.493 DEBUG:teuthology.orchestra.run.smithi072:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-22T02:05:15.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-22T02:05:15.515 DEBUG:teuthology.orchestra.run.smithi089:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-22T02:05:15.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-22T02:05:15.532 DEBUG:teuthology.orchestra.run.smithi098:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-22T02:05:15.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-22T02:05:15.547 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-22T02:05:15.559 INFO:teuthology.task.internal:Transferring archived files... 2024-03-22T02:05:15.561 DEBUG:teuthology.misc:Transferring archived files from smithi072:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-22_01:06:30-fs-wip-batrick-testing-20240321.172515-distro-default-smithi/7615721/remote/smithi072 2024-03-22T02:05:15.561 DEBUG:teuthology.orchestra.run.smithi072:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-22T02:05:15.609 DEBUG:teuthology.misc:Transferring archived files from smithi089:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-22_01:06:30-fs-wip-batrick-testing-20240321.172515-distro-default-smithi/7615721/remote/smithi089 2024-03-22T02:05:15.609 DEBUG:teuthology.orchestra.run.smithi089:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-22T02:05:15.648 DEBUG:teuthology.misc:Transferring archived files from smithi098:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-22_01:06:30-fs-wip-batrick-testing-20240321.172515-distro-default-smithi/7615721/remote/smithi098 2024-03-22T02:05:15.648 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-22T02:05:15.692 INFO:teuthology.task.internal:Removing archive directory... 2024-03-22T02:05:15.692 DEBUG:teuthology.orchestra.run.smithi072:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-22T02:05:15.694 DEBUG:teuthology.orchestra.run.smithi089:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-22T02:05:15.697 DEBUG:teuthology.orchestra.run.smithi098:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-22T02:05:15.740 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-22T02:05:15.754 INFO:teuthology.task.internal:Not uploading archives. 2024-03-22T02:05:15.754 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-22T02:05:15.766 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-22T02:05:15.766 DEBUG:teuthology.orchestra.run.smithi072:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-22T02:05:15.768 DEBUG:teuthology.orchestra.run.smithi089:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-22T02:05:15.770 DEBUG:teuthology.orchestra.run.smithi098:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-22T02:05:15.780 INFO:teuthology.orchestra.run.smithi072.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 22 02:05 /home/ubuntu/cephtest 2024-03-22T02:05:15.782 INFO:teuthology.orchestra.run.smithi089.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 22 02:05 /home/ubuntu/cephtest 2024-03-22T02:05:15.794 INFO:teuthology.orchestra.run.smithi098.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 22 02:05 /home/ubuntu/cephtest 2024-03-22T02:05:15.795 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-22T02:05:15.808 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-22T02:05:15.834 INFO:teuthology.nuke:Checking targets against current locks 2024-03-22T02:05:15.862 DEBUG:teuthology.nuke:shortname: smithi072 2024-03-22T02:05:15.862 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-22T02:05:15.888 DEBUG:teuthology.nuke:shortname: smithi089 2024-03-22T02:05:15.888 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-22T02:05:15.893 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi072.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-22_01:06:30-fs-wip-batrick-testing-20240321.172515-distro-default-smithi/7615721', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-22 01:08:20.963144', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKo+S2CXMsjpVcPBFgBBxourTVomk7ZnFioj/wZwp2FQnk+p+7YpjlIcrH0cdg8qzENQcY7AY/FnSyIKX1KDZzE='} 2024-03-22T02:05:15.914 DEBUG:teuthology.nuke:shortname: smithi098 2024-03-22T02:05:15.914 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-22T02:05:15.922 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi089.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-22_01:06:30-fs-wip-batrick-testing-20240321.172515-distro-default-smithi/7615721', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-22 01:08:20.964332', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLWzifK/OZv43YSON/Qai1svwB4TjTJhvPiWpjHja5YIoIpzEAE5MhByrLzxAxpJXLlwFKb3r5VESzqMTnqdIrI='} 2024-03-22T02:05:15.947 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi098.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-22_01:06:30-fs-wip-batrick-testing-20240321.172515-distro-default-smithi/7615721', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-22 01:08:20.961732', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKWQYuZbn6ocySjHj2r0rq7Kuu00Mey6QK1jgdp35wO6acMUarg2s3WLQn35c9XdEpKW6IZmQNfpVpOPnNZxD74='} 2024-03-22T02:05:15.958 INFO:teuthology.orchestra.console.smithi072:Power off 2024-03-22T02:05:15.958 DEBUG:teuthology.orchestra.console.smithi072:pexpect command: ipmitool -H smithi072.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-22T02:05:15.983 DEBUG:teuthology.orchestra.console.smithi072:power off output: Chassis Power Control: Down/Off 2024-03-22T02:05:15.984 DEBUG:teuthology.orchestra.console.smithi072:pexpect command: ipmitool -H smithi072.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-22T02:05:16.003 INFO:teuthology.orchestra.console.smithi089:Power off 2024-03-22T02:05:16.003 DEBUG:teuthology.orchestra.console.smithi089:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-22T02:05:16.014 DEBUG:teuthology.orchestra.console.smithi072:check power output: Chassis Power is on 2024-03-22T02:05:16.028 DEBUG:teuthology.orchestra.console.smithi089:power off output: Chassis Power Control: Down/Off 2024-03-22T02:05:16.028 DEBUG:teuthology.orchestra.console.smithi089:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-22T02:05:16.043 INFO:teuthology.orchestra.console.smithi098:Power off 2024-03-22T02:05:16.043 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-22T02:05:16.056 DEBUG:teuthology.orchestra.console.smithi089:check power output: Chassis Power is on 2024-03-22T02:05:16.068 DEBUG:teuthology.orchestra.console.smithi098:power off output: Chassis Power Control: Down/Off 2024-03-22T02:05:16.068 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-22T02:05:16.092 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is on 2024-03-22T02:05:20.015 DEBUG:teuthology.orchestra.console.smithi072:pexpect command: ipmitool -H smithi072.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-22T02:05:20.058 DEBUG:teuthology.orchestra.console.smithi089:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-22T02:05:20.094 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-22T02:05:20.128 DEBUG:teuthology.orchestra.console.smithi072:check power output: Chassis Power is on 2024-03-22T02:05:20.168 DEBUG:teuthology.orchestra.console.smithi089:check power output: Chassis Power is on 2024-03-22T02:05:20.204 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is on 2024-03-22T02:05:24.133 DEBUG:teuthology.orchestra.console.smithi072:pexpect command: ipmitool -H smithi072.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-22T02:05:24.169 DEBUG:teuthology.orchestra.console.smithi089:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-22T02:05:24.205 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-22T02:05:24.245 DEBUG:teuthology.orchestra.console.smithi072:check power output: Chassis Power is on 2024-03-22T02:05:24.279 DEBUG:teuthology.orchestra.console.smithi089:check power output: Chassis Power is off 2024-03-22T02:05:24.316 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is off 2024-03-22T02:05:24.381 INFO:teuthology.orchestra.console.smithi089:Power off completed 2024-03-22T02:05:24.417 INFO:teuthology.orchestra.console.smithi098:Power off completed 2024-03-22T02:05:28.250 DEBUG:teuthology.orchestra.console.smithi072:pexpect command: ipmitool -H smithi072.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-22T02:05:28.360 DEBUG:teuthology.orchestra.console.smithi072:check power output: Chassis Power is off 2024-03-22T02:05:28.461 INFO:teuthology.orchestra.console.smithi072:Power off completed 2024-03-22T02:05:28.567 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/ffsb}} duration: 2832.5627501010895 failure_reason: 'The following counters failed to be set on mds daemons: {''mds_server.req_rmsnap_latency.avgcount''}' flavor: default owner: scheduled_pdonnell@teuthology status: fail success: false 2024-03-22T02:05:28.567 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-22T02:05:28.657 INFO:teuthology.run:FAIL